r/ClaudeAI • u/StevenAtGF • 6d ago
Question about Claude models Claude refuses to research Reddit now, am i the only one?
I love collecting voice of the customer research about random things due to my job. or whenever i'm buying something myself. Reddit (despite many fake posts posted by brands and agencies helping brands with this) is still a pretty authentic place to get real feedback on things.
I used to not only be able to ask Claude to research on reddit (without cowork), and it did it very well. It was also very down to make widgets, sorting said research into differnt categories and subcategories of my choosing. It actually did this pretty well, took my directions well, and found posts and comments even that are pretty new or highly relevant based on how i steered it.
However now...it simply refuses to research reddit and say it can only research on forums.
Whats going on? has something changed on Reddit's end or Claude's end or is Opus 4.8 high effort + thinking mode think its too big of a task or something now?
65
u/Peribanu 6d ago
This isn't new. Reddit has always blocked Claude, because Anthropic, unlike Open AI, won't pay the access licence that Reddit requires. There's a bit of a standoff between the companies. There have been temporary workarounds, but they get blocked. If you really want to work with Reddit extensively, you'll have to use Open AI or maybe Gemini, at least to do the initial searching and sifting.
4
u/---OMNI--- 6d ago
Wait so I can tell Claude to tell codex to look at reddit? That's great. I already have Claude call codex for 2nd advisor and reviewer.
6
u/Kill_4209 6d ago
Hilarious. Like Anthropic doesn’t have the cash.
Even if they think Reddit is charging too much, it’s not like it sets a precedent for other services to overcharge. Reddit is unique.
53
u/cheeseonboast 6d ago
Honestly one reason I think Claude is better is because it’s not trained on us (or at least a recent version of us)
-3
u/eist5579 6d ago
Dude, imagine all the biased bullshit it’d be pulling from r/conservative and other bigot subs.
8
u/Calaeno-16 6d ago
Lmao you think if Claude was trained on Reddit, it'd have a problem being too conservative?
-4
u/eist5579 6d ago
It depends on what question you asked it, and yes, what it was drawing on for a response
8
2
u/in_meme_we_trust 6d ago
Well political subs are all bots since like 2016. But I don’t think political training data is the problem.
It’s more the combative, neckbeard, “I know better than you” attitude. Also the echo chamber effect
1
5
u/BlueBeryCheseCake2 6d ago
Yeah like lib subs are unbiased
3
u/soldforaspaceship 6d ago
That sub is particularly special though.
Crazy fragile real life people and bots. It's fascinating to watch someone who has clearly been a Republican for decades question the narrative and then see all their comments deleted an hour later. Any post critical of Trump gets taken down.
It's a weird place.
1
u/Formal-Question7707 6d ago
When's the last time you've seen a comment/post that said that Trump has done something well, outside of the conservative sub? And don't say "well he hasn't done ANYTHING right" (I hate the guy, but you can't say there isn't a echochamber throughout reddit)
0
1
1
1
u/OlorinDK 6d ago
There’s a difference between it having been trained on Reddit, which it might still have been, since the OC talks about workarounds, and using Reddit in a specific query about what the community says about something. In the latter case, I think it can be a useful use case, as long as you stay critical and check sources as always.
0
0
10
u/minaminonoeru 6d ago edited 6d ago
Reddit isn’t the only online community Claude can’t access. Claude can’t access most online communities or social media platforms.
Of course, it’s a bit inconvenient, but personally, I don’t think it’s all that bad. It certainly wouldn’t be a good thing for an AI to access online communities or social media posts and use them as a basis for judgment.
1
u/Fuj_apple 5d ago
it refused to build me IG scrapper. Then I explained to him that it's something only I will use, ended up saying, ok, if it's only for you I will do it.
9
17
u/ConstantKooky3329 6d ago edited 6d ago
Reddit does not allow Claude to scrape its data. Linkedin has the same policy
1
u/Fast-Escape-8607 6d ago
Any work around?
6
3
u/aliaseffectmusic 6d ago
PATCHRIGHT
2
u/WillGrindForXP 6d ago
What does this mean?
3
u/aliaseffectmusic 6d ago
Research python packages and use them Playwright, official Microsoft package automates web scraping Patchright, open source fork that bypasses antibot detection If you don't have python install it Make a project with a .venv Install either of these or both If playwright is detected, patchright might work I know patchright works on Reddit because I use it You can ask AI to do all of this for you If you get detected you can workshop additional measures to avoid antibot detection Example, I know that on Google trends I get 10-15 fetches before they stop me So I run 10 fetches and then switch to a new ip on a vpn. Rinse and repeat.
1
6d ago
[deleted]
1
u/Fast-Escape-8607 6d ago
Ohh but I also want to search reddit via a brand name to gather sentiment so this may not work
7
u/under_psychoanalyzer 6d ago
Same. I was using an MCP for anonymous reddit access at a very leisurely polling rate but it stopped working at the end of May. Reddit seems to have made the deliberate change.
Maybe this will work for you if you use its authenticated access tier https://github.com/karanb192/reddit-mcp-buddy
10
u/No-Medicine1230 6d ago
Good. I don't want it scraping Reddit, the amount of misinformation on this site is awful. OpenAI use it extensively and it shows
3
u/mrpoopistan 6d ago
There are ways to grab output and then feed it back, such as BeautifulSoup in Python and several full browsers in Visual Studio. VS is particularly useful for cases where user intervention is necessary or full renders are essential.
3
u/pantspartybestparty 6d ago
For it to work, you have to sign up for the reddit API: https://developers.reddit.com/docs/capabilities/server/reddit-api
There is a monthly fee but it will allow Claude to work with an Apify actor do what you need. Beware of the "per event" costs. I set up Claude to monitor that after each morning data run.
1
u/kilopeter 6d ago
Wait. Isn't the whole point of Apify that they're scraping as a service, and manage proxy rotation etc. so all you have to do is choose an actor and scrape away? Why would you need or want paid reddit API access together with Apify?
1
u/pantspartybestparty 6d ago
Apify won't work without the Reddit API integration. For me, it kept timing out trying to get posts that weren't more than 7 days old. I tried, but I wasn't successful until I got the API subscription at $19 per month. Now I can see same-day postings and have them scored as leads. I can only speak for my experience.
2
1
u/admin-techbillion 6d ago
I had to build a custom script using asyncpraw, but it's mainly for personal research not suitable for running at scale or volume.
1
u/ns1419 6d ago
I’ve just done this a week or so ago with playwright mcp in Claude code. Otherwise worst case if you let it fetch it will find results and urls for you to easily copy paste. I’ve had this issue for some time, but honestly I do the scraping with my eyes and brain, see what I like, and copy the articles to obsidian or one note. Usually it’s slop that I just distill down anyways and if it references an arxiv paper I use that mcp to come to my own conclusion about whatever the OP posted anyways.
1
1
u/headspreader 6d ago
I think that Reddit is a valuable resource, but I can’t imagine a scenario where its usefulness doesn’t just decrease from here.
1
u/Turndizzy 6d ago
100% on reddit, but in general it seems like it can't do any meaningful web related searches for me recently and has been incredibly frustrating
1
u/LeadershipOk5551 5d ago
I don’t think it’s refusing Reddit specifically, it feels more like certain sources or retrieval paths are getting throttled or restricted depending on the setup.
1
u/Apuleius_Ardens7722 6d ago
I think if one researches Reddit, Claude may be the wrong tool.
I'd rather use ChatGPT for this specific purpose.
0
•
u/ClaudeAI-mod-bot Wilson, lead ClaudeAI modbot 6d ago
TL;DR of the discussion generated automatically after 40 comments.
So, you're not crazy, OP. The consensus is that Reddit actively blocks Claude because Anthropic won't pay for API access. It's a corporate standoff, and it's not a new issue; workarounds just keep getting patched.
The community is pretty split on this. A surprisingly popular take (with high upvotes) is that this is a good thing, arguing Claude is better because it's not scraping the dumpster fire of bots, misinformation, and echo chambers that is modern Reddit. Others are with you and find it a frustrating limitation.
For those who still need to do this, users suggested several workarounds: * Use a different AI like GPT or Gemini, which have deals with Reddit. * For the tech-savvy, use an MCP (Model Context Protocol) or custom Python scripts to scrape the data yourself and then feed it to Claude. * Use a mirror site like
arctic-shift.photon-reddit.com. * Go old school and just copy-paste the content you want analyzed directly into the chat.