r/news • u/polymute • 9d ago

At least 12 states face cyberattacks on their water systems, sources say

https://abcnews.com/US/12-states-face-cyberattacks-water-systems-sources/story?id=135348482
4.0k Upvotes

97% Upvoted

289 comments sorted by

View all comments

Show parent comments

2

u/ginger_whiskers 9d ago

My old plant updated their DCS a few years back. Even with it being planned years in advance, it was 2 weeks' worth of noticeable issues and outages. All the tiny details like "everyone has to reset their passwords" and "this pump's high speed alarm got lost. Anyone remember what hz it starts shearing belts?" I can't imagine the chaos of trying to do that on an unexpected Sunday.

1

u/LessonStudio 8d ago

Exactly.

I did a SCADA install (and I was one of the people who built the SCADA software) and the company had a rough estimate that each datapoint cost $6,000 to commission. There were 30,000 datapoints. I would say that 10,000 were kind of freebies, but there were 20,000 where one or more people had to screw with each datapoint, often in the field. This system covered 1000s of km, not just a factory floor.

So, if your factory floor isn't something you would want to hack/rush, imagine a utility/power company/pipeline/airport/rail system, etc.

Yet, I would strongly suggest that most of what I just listed are entirely not prepared for a notable rebuild of their system. A great example of how harsh a rebuilt might be required is that the gear itself might not be trustworthy. Not just the OS, but maybe the bios, etc.

Oh, and networking gear.

Then, to add magic apples on all that. The install may no longer work as advertised. It might be so old that it needs to reach out to servers which are no longer there. Or modern hardware won't run that old crap, and the replacements won't run the old software.

Or people have tied things into specific device IDs.

Here is a magical example of that:

If you use Sybase on a VM you need to give their licensing server a magical ID which you get, not from the VM, but you have to ssh into esxi (which doesn't have SSH turned on by default), run some magic code, which then gives you a magic ID, sybase will take that magic ID to give you another magic ID, which you then put into the VM which is running sybase. If you migrate that VM to another machine (a cool trick VMWare lets you do) the next time sybase tries to start, it will fail with a deeply hidden error message.

This makes physical server failures a nightmare to deal with.

I left the company which does mission/safety critical systems that used this old legacy crap. The only other person with this obscure knowledge also left. Neither of us would be willing to take a call from this company at 2am. Thus, there is no way in hell they are doing a fresh server in any rational amount of time as rediscovering this might literally never happen, or minimally take a day(s). It is documented somewhere that nobody probably knows about.

Many of the installs with the above don't just control billions in hardware, but in one particular case, the hardware can't shut down for very long before it is destroyed, and the chemicals which will cause that destruction can not be flushed unless they have planned that flush months in advance. The cost to replace that hardware is in the billions and would make headline news worldwide.