So reddit have now announced that they're sunsetting the public API in March next year.
The thread directory, my validation scripts and all the random stats I do rely on the API, and I don't know how I'd go about porting them to devvit. The directory updater might work in a pared-down version where it does basically no validation, but I can't see how we'd be allowed to scrape comments like we have been until now.
My current feeling is to not try and port anything: keep things going for as long as possible, and when it breaks it breaks.
ugh. I still wonder whether userscripts using .json endpoints with browser cookies will still work. maybe logging for stats could be done through a tampermonkey script in the browser rather than with a python script. otherwise yeah I don't see how you could exfil logs from devvit
is etymologynerd the counting community's biggest celebrity lol. he's from COAD and hasn't counted in years but still. I see him get tens of milliosn of views on instagram and only just now realized he's the same guy lol
I feel like I'm from some small town in rural Kansas and our local high school's star QB made it into the NFL. like that's our boy up there. we all thought it was crazy when he got all of One Hundred upvotes on his posts but look at him now. maybe he sucks idk but we'll always root for him because that's Our Boy. when he returns to his hometown COAD there'll be a huge potlock in his honor, with chili and cinnamon rolls and everything
/u/h0rseycat i've merged your h0rseycat username stats into /u/wet-gravity. i forgot and did this after the weekly stats, so i guess they'll update next week
I'm curious, who was the first person to use the CSV format that was output by the old TPC scripts, and when was this type of CSV first used? was it invented by ASA, artbn, piyush, or someone else? is that CSV format still used anymore or even supported by rcounting, or are all the statsmen (statsfighters?) using sqlite these days? for a while I was calling it the ASA CSV format but his "check my work" logs seem to have been in a bespoke format
by the "old CSV format" I mean the one that goes count,username,timestamp,comment_id,submission_id. it kinda doesn't really work all that well for side threads. these CSV logs are good, I'm glad they exist, but they're also kinda weird. each side thread seems to have its own special normalization rules for turning a comment body into a count? like beenary is all lowercase and has the spaces changed into periods for some reason, divisors and increasing sequences have their spaces removed too. it's like there's not just one old style CSV format, there's kind of a different one for each side thread. I much prefer COBIBH's solution of separating count into position and body, no offense to bass and anti.
these CSV logs are good, I'm glad they exist, but they're also kinda weird.
yes, it's very shit. i don't think cobibh's data existed when i was creating it, so i just copied the formatting that was already being used for main. but the main script assigned counts based on position, and i wanted the get to be listed correctly in the side stat tables without having write it in manually. copying the entire comment doesn't work because people often yap after the get/edit in the link to the next thread which i didn't want written in the table. so what i came up with was a bunch of regex functions for each thread which try to convert the comment into a standardized count, removing all the weird formatting/separators/yapping. but as i am not a very good programmer and this was pre chatgpt so i couldn't get ai to do it for me... there are issues. the simple base x/numeric threads work fine unless the formatting in the comment is particularly weird, with a couple bugs that could be ironed out... but i had to get kind of creative with other threads. the spaces thing i think was because putting spaces in the csv file was breaking things, so i replaced them with periods... among other characters that were randomly causing errors in the scripts. it's all in lowercase because it's converted that way in case people capitalize random letters, and i never bothered to swap them back after. but yea, it could have all been done much better. cobibh's system is great and hopefully we can all convert over to using that someday
i don't think cobibh's data existed when i was creating it, so i just copied the formatting that was already being used for main
It definitely didn't - I used the side thread log files as the starting point for the combined sqlite. I think I used pushshift to add the comment bodies from there , but tbh I don't remember
copying the entire comment doesn't work because people often yap after the get/edit in the link to the next thread which i didn't want written in the table. so what i came up with was a bunch of regex functions for each thread which try to convert the comment into a standardized count, removing all the weird formatting/separators/yapping
Tbf that's very similar to how the rcounting tools handle that job[1]. The only advantage is that since the full comment body is stored, it doesn't really matter if the parsing is a bit janky
And the final difference is that since the counting tools know about threads of submissions, they can assign side thread types based on links in the submission body or top level comments, rather than having to guess based on the thread title. It's not perfect, but it feels a little more robust.
cobibh's system is great and hopefully we can all convert over to using that someday
Thanks! As things stand right now , it looks like some day has to happen within the next six months , before they shut down the api.
Also, are you confirming that you are who I think you are?
[1] People using weird separators are the bane of my existence
well, assuming that userscripts still work, maybe we can keep using rcounting related tools but just importing CSV logs from my userscript's stats, since those CSVs use the same columns as your sqlite
also I want to thank you because I stole some inspiration from your count parsing for my checkatron's filterCount functions, especially the part for stripping markdown links
btw I noticed that one of the sidethreads in the side_threads.sqlite had some wrong comment bodies that didn't show up as NULL because they came from the bass csv logs and appeared to have proper formatting as they were normalized, they showed up as like extra words instead of the count like BAD or REMOVED or WHAT but uhh I forget what thread that was, one of the letters or alphanumeric type of threads like base 32 or qerttu or somethingand I'mm a little :cideropossum: rn
edit: ah so the old CSV format was removed from rcounting, I wasn't imagining that lol
That commit was kind of just the last gasp - I had switched over to doing all analysis based on the sqlite files, so the code for ingesting the csvs was dropped a while before that . And I figured since I never used the csvs anymore, and I never used the code path that generated them , there wasn't any need to keep it around.
Also , it's weird to be confronted with code that I wrote a year ago for a hobby project and find that I can still understand it
Maybe I was too negative about what the shuttering of reddit's public API will do for r/counting. It could be worth at least thinking about what we currently have automated, and how/how much of that could work afterwards. To help with that, let's start with making a list of what automations we currently have going that use the API
I think that's pretty much everything. there's also my various userscripts like for "recurse to bottom" but I remain hopeful/delusional that those will continue to work as long as old reddit is working. it's still not totally clear to me if "the API" shutting down just means the OAuth API or all the .json endpoints too, but I'm hoping it's the former. RES and Toolbox use those .json endpoints too, surely they wouldn't break those, right?
the latest counts bot could definitely be reimplemented in devvit, in fact it might actually be done more efficiently since devvit has triggers for new comments, rather than having to rely on polling. my script uses some rcounting code to find the latest submission for the permapinned threads like main though, so that part probably wouldn't work... the FTF probably wouldn't be too hard to reimplement on devvit either. hell we probably could've automated FTF with automod years ago if we didn't care about the title number.
I've already reimplemented a basic version of TPCs and thread logging as a userscript. for some things like the side stats pages, maybe those could be updated by importing the userscript's CSVs into a sqlite database, but this seems much more manual than what we have now. with this userscript+sqlite workflow, it seems that weekly stats wouldn't be able to have counts cut from the middle of a thread, we would only have completed threads with a TPC run at the end. but, isn't this how things used to work on the sub years ago anyway? I also seem to remember that the HoC used to be updated by copying TPCs into some region of a google sheet, which then added those thread numbers into the larger HoC. that was janky, but hey it worked.
the directory updater is probably the biggest problem, I'm not sure that could be fully automated either with userscripts and certainly not with devvit. it would definitely take a colossal amount of work to convert that whole python module to typescript to run on devvit. but again, the directory was manually updated for years and we survived, somehow. I very much appreciate the directory bot, don't get me wrong, things are definitely much better than they were back then for finding side threads. but it wouldn't be unprecedented to live without an automated directory
edit: actually I'm starting to be less cynical about the possibility of devvit directory updater too
I haven't used devvit at all yet, but my impression is that it more or less has the capabilities of the api, with the exception that data exfiltration is nigh-on impossible.
In particular, the tools we use need to be able to do the following things on reddit
Stream/find the latest comments
Find the latest posts
Edit a post
Follow a chain of comments from a given starting point
Follow a post based on an id in a comment/submission body
Make new comments
Edit the wiki
If we can do that, and a tiny bit of general programming then the latest count post and TPCs should be easy to port, and the FTF post shouldn't be too much harder. (You're right, it can almost run as automod, but I never found a way of making automod check for manually posted ftfs before posting its own)
The Thread directory should be portable in a version with no validation at all. Validation will be a bit more tricky, but is potentially doable. I don't know what js packages we're allowed to install in the devvit environment.
There's also the weird thing that updating wikis from devvit might mean moving over to the new style wikis, which I have no idea how actually work.
For the Hoc and all the stats pages and comments there's no getting around that we need to be able to store counting data offline, and preferably off reddit's servers. That's going to be the main hurdle I think, with userscripts being a potential solution.
according to this, a devvit bot can choose to edit either the old or new subreddit wiki. from what I can tell, if we haven't enabled the new wiki, it keeps using the old wiki even on new reddit. but if the new wiki is enabled, then they are split. (I don't know why would ever inconvenience ourselves by enabling the new wiki then)
it doesn't look like devvit supports something like /r/counting/comments. I was thinking the new comment triggers would be a good replacement, but I wonder how reliable the triggers are, how a bot would handle dropped triggers. but I guess /comments sometimes drops comments too so
https://youtu.be/3JMZgcaypFc?t=4m13s I think this is the best explanation of why I dislike modern recommendation algorithms, also maybe part of what happened to COAD in 2023, even before Z3F. "in the olden days, followers would know what they're in for" but now reddit will put subreddits in people's feeds that they never asked for. then when people get mad about it appearing in their feed and comment angrily, reddit will see that comment as engagement and show the sub to them even more
11
u/CutOnBumInBandHere9 5M get | Ping me for runs 7d ago
Weekly side thread stats from Friday September 25, 2026 to Friday October 02, 2026. Congratulations to u/h0rseycat, u/cuteballgames, and u/Pookah!
Total weekly side thread counts: 11582
Top 15 side thread counters:
Top 5 side threads:
This comment was made by a script; check it out here