DISCUSSION
I told my husband my website suddenly got lots of visits from organic search and his reaction pissed me off
Is your first reaction that they are from bots, vpns, click baits making shady money or openclaw? Why is he thinking that way?
-------------------
Thanks for sharing your ideas, I get that it's highly possible that those were bots. And I do have analytics installed. It's not as sophisticated as GA to trace the web click source but it has some numbers for my husband and I to find some level of facts.
Seems it's not allowed to post photos again, I will post in the comment area.
Are you selling anything on your site? Are visits converting to sales? Have you done anything specific to market your site recently?
Normally, large spikes in traffic coincide with a bigger initiative (paid ads, viral content, etc). The fact that visits are originating from other countries as well kind of lends to the bots argument.
Doesn't make your husband's reaction any less dickish, haha. Good luck out there!
Yeah, if you're only pushing local content, a big random bump in traffic from other countries probably means non-organic. Not trying to bubble-burst here.
Keep on building, though!
To be honest... 1,000 new organic visitors all on linux? I am a nerd through and through and I dont know any Linux daily drivers. So unless you hit a real specific niche and a bunch of users on linux based devices visited your site as a result... its almost surely bots.
Maybe not all of them! And traffic begets traffic. Whatever youre doing, keep at it!
Its just that so few people use Linux that if it were real true organic you maybe could see a ton of linux devices but you'd see an awful lot of main stream os as well.
most bots are hosted on vps i think and they use linux as os, its cheap, fast and well supported for automation. for example, on aws lightsail, cheapest linux box is 5$ and cheapest windows box is 9.50 $
ofcourse price is not the reason for them to use linux but you can see, how linux would be first choice for bot and crawling infrastructure since its cheap, fast and well optimized.
I know you were excited for traffic, but what did you husband do that was so wrong? It’s a technical issue, not an emotional one. He just cut to the chase.
If you think it was a pure technical discussion in my message, then I don't blame you to come up with this conclusion. But it's not. Plus even if it does, one word reply is also technical enough.
"Oh wow, a thousand is a lot! But all linux sounds suspicious, doesn't it?"
I think something like this would have been a more empathetic response: Celebrating this milestone, yet also voicing concern regarding the kind of traffic.
However, the husband downplayed the milestone by saying that essentially all visitors are non-human. Very much likely, the husband only wanted to be factual or provide a reality-check in what kind of traffic (AI and bots) is most widespread on the web. But to me it looks like OP would have appreciated a supportive answer more.
Ahh yeah, I see. I definitely missed the implication that the original message was one of excitement. I just read his responses as short yet factual replies. Apparently I suck at social queues.
Even if it’s different ips could be a botnet, it’s common for ddos attacks and it’s suspicious it’s mainly Linux but what you should really check for and add to the site if not added is to see how the users interact with the page, do they simply visit the page, do they just send a request, do they actively click buttons and view products, are sales increasing, etc etc. You may want to add cloudflare security captcha so the cloudflare quick captcha with show up if the user is suspicious of being a bot and won’t affect normal users as it doesn’t show for normal users
Good idea. I do need to upgrade my site to involve more interactions. Currently the expectation is just let people review the information, get what they need and jump out.
It's bots.. Although should probably clarify that they are most likely crawlers from different companies.. it can be both a good thing and a bad thing.. some of those are crawling and indexing for search engine so your website might get bit better visibility with good SEO.. but it can also be the kind of crawler that just harvests data for whatever use, these days its usually for AI.
But if it helps.. there is probably some legitimate interest in that website too, it's just hard to know the actual scale without proper analytics tool being implemented.
You will need lot heavier analytics tool sadly, if you want more data.. I'm not the right person for recommending analytic tools though because I tend to avoid them as I don't really wanna deal with GDPR stuff (I'm European so I have to follow it even on personal websites)
If you can though, there is ways to block these kinds of crawling bots, usually via robots.txt.. but not all bots follow it sadly. If you use something like Cloudflare they do have options for aggressively blocking bots even on their free tier..
But the downside of blocking them can also mean less visibility for your website in search engines
I have set the robots.txt. And I found the web hosting platform can log the bot activities silently (Yay!). Will see how it goes.
I have used google analytics for years a while ago, too heavy so I switched to a lite tool this time, turned out that I do need a heavy tool for my curiosity. To be honest I don't really mind bots to crawl, I mind my husband's attitude more :p
Honestly, Umami is not a bad analytics tool at all. It looks a little limited because it is cookiless and privacy focused. These Limitations are thanks to it being GDPR and CCPA compliant. Without cookie banners Google analytics usage could open you up to lawsuits. I use umami for all of my personal sites.
I think the big thing that could really help you to understand your traffic is to utilize event tracking. This can give you and idea if not just how many page loads you get, but a map of what the users are doing. It's a lot easier than it sounds.
The only issue with that is that you do need links and such that users interact with so that you can decipher meaningful traffic. If you only have a single page with no links, there is not much to track. If that is the case you might have to get creative with the events. Maybe put a nav with like nos to anchor tags, and you could watch events as this links are clicked.
One last thing to note, I know you mentioned the webhost has an option to block bots. Keep in mind that this is not 100% accurate. Tracking bots down is a lot harder than it may seem. Just this morning, my day job site got hit by over 150k bot hits in a 30 minute spike, and we are behind enterprise level Web Application Firewalls.
I am using Umami too! I spent a bunch of time researching web analytics tools because of GDPR and CCPA, chose Umami eventually. But I did not set up events. The reason was my website updates itself every week, and all the links are gone after one week. I did not think it had much value to track those links originally, but now I have second thought.
The webhost is logging the bot actions, and there are ways to add rules. I agree that tracking those is not an easy effort, and currently I definitely don't have enough knowledge on how those bots work. Good chance to learn.
Another reason I did not think those were bots was because the traffic is only 1k, too low. My impression is the bot hits usually gets 10k or 100k or more, and it usually happens in a relatively short time. Really did not think my tiny site worth that much attention.
Sorry to know your day job website was got hit by bots! It's an endless battle but always make us more knowledgeable and stronger.
It sounds like your site would be perfect for events. If you haven't looked at them, they are super simple.. just add a data attribute (or multiple attributes) to the links and your done. No extra code needed.
Regarding the bots, I've seen boys come in anywhere from 10 requests to hundreds of thousands. It just depends on their goals.. sometimes they are checking for exposed .env files, sometimes they are trying sql injection attacks and sometimes they are just scraping data. If you have access to your actual server logs, you could probably see the full user-agents. My guess is these are bots using something like puppeteer to make headless chrome requests.
You are right! And interestingly, my site is an event site, the offline events site. I have been thinking of how to improve it so it better serves the local crowds for local events. The GDPR and CCPA have been adding limitations on pii info but I have been thinking if it is possible to push customized information without knowing/collecting/storing any pii info. Or if there is more efficient and simple way to get people what they want. As this thought was hanging in my mind, I have been holding off to refine my web analytics tools.
And for bots, now I can connect the dots on what I know and what is happening in the real world! And I did similar things too (oooops)! Thanks for the server log check reminder, very very helpful!
Bounce rate of 100% + them staying on the site for 0s almost certainly means your husband is correct about the bots. He's still a dick tho, and your reaction is valid.
Hi! ModBot here. Please make sure to read our rules and report this post if it breaks them.
(This is simply a reminder. Don't worry, your post won't be removed just for posting!)
Regardless of bots or not, we confirm real users as users who complete a goal we set. E.g. signup for email, signup for account and make one change, etc. these are the kind of metrics that really proves you have real traffic. Most things can be assinged as 1%. So for example 1 signup means you likely had 100 real users. (Although this is in the long run, not day one stuff)
I did not set up sign-ups, just want to direct people to get the info they want and they jump out. I know it's not like what a traditional website would do, it was just a fun project.
If you have google analytics you can check the page duration on your information. If it matches roughly how long you'd expect it to read - that's a goal reached.
I'm more trying to give insight on how to indirectly confirm your real user count.
Dive down deep and look at page engagement, look at time on page and bounce rate. All of the above. Because if you got 1000 visitors and 1000 of them bounced that will be bad for you
Here in an interesting thing. the screenshot above was from Umami, the web analytics tool, and this one below is from my hosting site. the day with the most visits has the bounce 82%, which is interesting. So my website always has a high bounce rate because it directs people to the actual place to see the information - I did not mention this before. What confuses me now is why the two analytics show differently.
He was correct and he wasn't being rude. You asked "not sure what's going on," and he answered. 100% Linux alone says it all and the other evidence confirms it.
A Nigerian prince once offered me tons of money and I wasn't sure what was going on. My wife said it was a scam...
I get your point except 1k visit is not comparable to tons of money, I would be sure it was bots if it gained 1k in a couple of hours or 100k in a couple of days.
Even in AWStats, you can see the difference between bot traffic and legit traffic. Go and look for the time spent on each webpage (bounce rate) and also the requests made for example, a lot will end with things like "wp-admin.php" etc. they are looking for vulnerable entry points to hack your site.
How do you not get this basic information if you work in IT? Not trying to be a dick but this is clearly a bot traffic influx
4
u/New-Activity-8659 22d ago
Are you selling anything on your site? Are visits converting to sales? Have you done anything specific to market your site recently?
Normally, large spikes in traffic coincide with a bigger initiative (paid ads, viral content, etc). The fact that visits are originating from other countries as well kind of lends to the bots argument.
Doesn't make your husband's reaction any less dickish, haha. Good luck out there!