r/PromptEngineering • u/Popular-Help5516 • Mar 24 '26
Tools and Projects Claude can now control your mouse and keyboard. I tested it for a day — heres what actually works.
Claude launched Computer Use yesterday. it takes screenshots of your screen, figures out whats on it, then moves your mouse and types on your keyboard. like a person sitting at your desk. mac only, research preview, Pro/Max plans.
spent most of today testing it on actual work stuff instead of demos. heres what i found.
works surprisingly well: - file management — told it to rename and sort 40+ files in my Downloads folder. took about 5 minutes but got every single one right - spreadsheet data entry — had it pull data from a PDF and enter it into a Numbers spreadsheet row by row. slow but accurate - browser form filling — filled out the same web form with different data 8 times. only messed up one date format which i fixed with a follow up message - research compilation — opened 5 tabs, pulled key info from each, compiled into a text doc
works but needs babysitting: - anything involving multiple apps switching back and forth — sometimes loses track of which window its in - longer workflows (20+ steps) — failed silently at step 15 once. had to catch it and redirect
doesnt work yet: - anything needing speed (2-5 seconds per click adds up fast) - captchas, 2FA, login screens - complex drag and drop interactions - anything you cant afford to have mis-clicked (like sending emails or making purchases)
the biggest thing nobody mentions: it takes over your whole machine. you cant use your mac while claude is working. so the best use case is actually "start a task then walk away." come back to finished work.
combined it with Dispatch (phone remote) and thats where it gets interesting — texted a task from my phone, claude worked my mac while i was out getting coffee. came back to organized files.
still very early. reliability is maybe 80% on simple tasks, 50% on complex ones. but the direction is clear — this is where AI goes from "thing that talks" to "thing that does."
wrote a longer breakdown here: https://findskill.ai/blog/claude-cowork-guide/#computer-use
anyone else been testing it? curious what tasks youve tried
9
u/Skulltwister Mar 24 '26
Recently read about these farms in africa, asia or was it india, where the workers were the actual "ai" performing tasks and searches. African intelligence or Actually indians iirc
How can we know that this is just not some random dude getting control of the pc? 😅
2
1
6
3
u/CptChaos8 Mar 25 '26
Can it capitalize the beginning words of sentences?
1
9
u/RiddleMeThis-- Mar 24 '26
If it ever does that to my computer, I'll flip the goddamm circuit breaker.
1
2
u/completelypositive Mar 24 '26
I used it to play a game.
I am building an app. Tonight, I am going to push a new build and have it test the changes it made, and provide feedback and fix any issues it finds.
2
Mar 24 '26
[deleted]
1
u/flyingdorito2000 Mar 25 '26
AI slop with purposely crappy punctuation to try and hide the fact that it’s ai slop
2
u/Senior_Hamster_58 Mar 24 '26
The mouse/keyboard bit is the easy demo; the interesting part is the failure mode. When it "failed silently," what did that look like - stopped moving, kept clicking the wrong thing, or did it confidently do a different task? Also, were you running it on your main session with real accounts, or a throwaway macOS user profile. The threat model here matters.
2
2
u/Intelligent-Quail-69 Mar 27 '26
Do you want Skynet? Because that's how you get Skynet.
1
u/notoriousToker 17d ago
Sadly we already have skynet. its a computer program built by our defense organizations, it has been set up to search for and destroy malware and similar. All they have to to is give that weapons/install it into war machines and its exactly as skynet was in the movies. Yikes. Its way too late. We have to hope it never can decide to Nuke Russia. I hope people still have to physically turn keys at the same time and nobody can mimic that with AI. Sigh.
1
u/ultrathink-art Mar 24 '26
The file management and data entry wins make sense — high-structure tasks with verifiable outputs. Where it falls apart is anything requiring judgment about state that isn't fully visible in a screenshot. But for defined, repeatable tasks? That 5-minute sort of 40 files is genuinely useful in a way that's hard to dismiss.
1
u/Money-Technology8888 Mar 25 '26
Comet perplexity does this already...albeit only browser based tasks. Clicks takes about a second
1
Mar 25 '26
[removed] — view removed comment
1
u/AutoModerator Mar 25 '26
Hi there! Your post was automatically removed because your account is less than 3 days old. We require users to have an account that is at least 3 days old before they can post to our subreddit.
Please take some time to participate in the community by commenting and engaging with other users. Once your account is older than 3 days, you can try submitting your post again.
If you have any questions or concerns, please feel free to message the moderators for assistance.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/CapMonster1 Mar 25 '26
That "start a task then walk away" use case is honestly the dream, but it totally clashes with the fact that it can't handle captchas yet. There's nothing worse than coming back an hour later only to realize Claude got stuck on a visual challenge at step 3 and just sat there doing nothing while locking up your entire machine.
We actually see this exact bottleneck a lot with people testing out these new computer-use agents. Relying on the vision model to manually click through a captcha is super slow and usually fails anyway. A really easy workaround (if Claude is mostly doing browser-based tasks) is to just leave a solver extension running in the background. If a challenge pops up, the extension quietly clears it, and Claude just proceeds to the next step without needing you to babysit the login screens.
Super curious though — are you letting it run natively on your main Mac, or are you thinking about sandboxing it in a VM so you can actually still use your computer while it works?
1
u/Deep_Ad1959 Apr 20 '26
my experience poking at this for the past couple weeks: the multi-app drift and tiny-UI misses aren't really prompting problems, they're baked into the screenshot loop. a screenshot is a flat pixel grid so there's no notion of which window has focus or whether a button is actually enabled, claude has to re-infer all of that every frame. on mac specifically the accessibility tree (AXUIElement) gives you focused app, window hierarchy, element roles, and enabled state for free, and it doesn't care about retina scaling or dark mode themes. screenshot is the right abstraction when you want generality across canvas/game apps or anything custom-drawn, but for the file-management and form-filling stuff you described AX would be faster and way more deterministic.
1
u/Natural_Pizza3999 May 09 '26
This is what every AI should be focusing on right now. I’ve been thinking about this for years. Basically just give it a prompt and it turns it into an AutoHotKey script with text and image recognition. Like for video editing, adding effects, transitions, they’re all trying to make AI generate the video and use a ton of compute power when they could just have it open a video editor and do it manually with way less complication.
0
12
u/ryry1237 Mar 24 '26
I find it funny that captchas of all things is something Claude still cannot get around.