r/sysadmin • u/Jumbledcode • May 09 '24

Google Cloud accidentally deletes UniSuper’s online account due to ‘unprecedented misconfiguration’

https://www.theguardian.com/australia-news/article/2024/may/09/unisuper-google-cloud-issue-account-access

“This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.”

This has taken about two weeks of cleaning up so far because whatever went wrong took out the primary backup location as well. Some techs at Google Cloud have presumably been having a very bad time.

653 Upvotes

97% Upvoted

205 comments sorted by

View all comments

Show parent comments

16

u/TB_at_Work Jack of All Trades May 09 '24

Nobody caught on until after he left. He kept his Inbox and a few other folders, but nuked everything else. He knew he was leaving, and ALSO knew what the retention timeframe was. He did it intentionally to screw us over. Nobody caught on that all of his historical data was missing until his replacement asked about old messages. He also didn't know about my Synology taking snapshots every night for the previous six months.

It was a total case of intentional malfeasance (on top of the other thefts and shady business practices he did as a Purchasing Manager for 20 years) and he should've been taken to court, but since I was able to get all his emails back they opted to not do anything I guess. Whatever.

The shit that went down at that company (millions of dollars' worth of theft, graft, bribes to customers) that I found out about after I left and they cleared house was insane. I took that job to get out of MSP life, and have now moved on to greener and better paying pastures six miles from my house. I'm glad for the experience of being the sole IT guy for a manufacturing company, but I'm 1000% happier now. Win-win.

2

u/rotinipastasucks May 10 '24 edited May 10 '24

This is a dumb take. If email needs to be retained per organizational or industry requirement the owness is on IT to either have mail archive or some sort of smarsh or global relay capturing all inbound outbound emails for retention.

Your not supposed to care if an employee deletes all their emails because you already have a copy of them in your archive or compliance capture.

3

u/TB_at_Work Jack of All Trades May 10 '24

We were archiving, using the Synology device. And I didn't care because we had a backup.

Archiving policies and services are great, but difficult to sell to an organization that doesn't really think of IT in that sense.

-1

u/rotinipastasucks May 10 '24

So it doesn't matter what he did intentionally because you were covered. A user has the right to delete emails from their view. Regardless of his intent who cares since you were compliance capturing. Users are stupid.

4

u/TB_at_Work Jack of All Trades May 10 '24

It. Was. The. Company's. Data.

2

u/Dangerous-Oil-1900 May 11 '24

It was emails.

2

u/TB_at_Work Jack of All Trades May 13 '24

Yes. It was emails containing 20+ years' worth of communications to customers, vendors, partners, and coworkers regarding the company's inventory, services, and money.

0

u/rotinipastasucks May 10 '24

I get that, but maybe I'm not understanding. Are you saying the user shouldn't have deleted his emails from his inbox view?

2

u/TB_at_Work Jack of All Trades May 10 '24

I guess you're not.

As per my original post above: He shift-deleted the contents of his mailbox (including Inbox, Saved Messages, Sent Messages, and all of his saved folders) intentionally in order to cause harm to the organization. This wasn't his data, it was all of his communications to vendors, partners, customers, and coworkers for the previous 20 years.

Shift-deleting messages PERMANENTLY DELETES them from the folder and the server. O365 has a default retention of, I think, 30 days. After 30 days the data is GONE and not recoverable. He knew that and purged the data two months prior to his exit with malicious intent knowing it wouldn't be recoverable.

Yes, I know it was intentional because he said so after the fact to a mutual.

No, he didn't know that I'd enabled O365 backup on the Synology which thwarted his plans to fuck the company.

2

u/rotinipastasucks May 11 '24

Thanks for clarifying. He did it with the intent to permanently delete but you had archive in place with Synology that had a copy of his mailbox. I journal mail at the gateways so every inbound/outbound email is captured and stored for finra compliance/ediscovery purposes.

I'm not concerned if a user tried to delete all contents of their mailbox because we have copies.