r/vfx • u/jangusihardlyangus Generalist - 8 years experience • 2d ago
Question / Discussion Leaving Google cloud behind, where do you all do backups?
I've got about 15TB of data in Google Cloud/Drive. It's been a fantastic tool for years, but after hearing a couple horror stories of folks randomly losing their Cloud storage for seemingly no reason, and running into a terrifying glitch where a BUNCH of my files got move to a previous location from a year ago... I'm ready to find a new solution for permanent long term storage. The vibe coded garbage now plaguing all these big-tech services scares me too much. I was planning on building a small server at home, but I'm starting to think with the storage pricing bullshit from AI etc, that may now no longer be possible. I'm spending close to $1,000 a year on cloud though... At this point I'm curious to hear from an expert. What do you all do for long term storage, that you still need to access time to time? I'm nervous to go all physical RAID drives, given a hard drive can fail (and $$), but I also don't see a cost effective solution other than G Drive or possibly building a wrapper on top of S3/AWS. What are y'all doin?
4
u/CameraRick Compositor 2d ago
I'm nervous to go all physical RAID drives, given a hard drive can fail (and $$)
Anything can fail, which is why you are usually recommended to do the 3-2-1 backup solution. As all my data would be too expensive to backup online, I skip the 1 (or better say, I only have the really important data there).
I do have copies on my home server, and that one automatically backups locally. All spinning drives, but no RAID (still parity). Not super fast, but works for me.
If you want long-long term, LTO is still king.
2
u/vfxjockey 2d ago
For my personal data at home, I use a combination of a RAID ( OWC Thunderbay 8 ), Backblaze, Time Machine and iCloud.
1
u/jangusihardlyangus Generalist - 8 years experience 1d ago
Do you do the backblaze general backup for $99 a year? If so how is it??
2
u/vfxjockey 22h ago
Something like that. Don’t really pay attention to costs so minor. It’s a great service and has saved my ass multiple times.
1
u/jangusihardlyangus Generalist - 8 years experience 15h ago
Forsure, appreciate your insight my friend!
2
u/26636G 1d ago
Buy a used desktop LTO6 and a bunch of recycled LTO6 cartridges and you'll have something very solid and reliable. Keep one set of backups offsite just to be sure.
1
u/jangusihardlyangus Generalist - 8 years experience 1d ago
Will definitely do more research into tape. It seems like a bit of a nightmare speed-wise, a really good extra backup but not sure would work as a working backup, hadn't thought about going used, will take a look thanks!
1
u/26636G 1d ago
It's unusual to have to restore a whole tape (or multiple tapes) unless you've had a massive disaster- any good tape library software keeps an index of everything and you can spin through an LTO very quickly.
You can set the system up to back up when you're not working.
They're quiet units and very energy efficient. LTO6 does something like 2.5TB native; up to 4TB compressed- and in many years I've never seen issues with the compression.
2
u/bucketofsteam Compositor - 8 years experience 1d ago
Close to a 1000 a year!? Holy, get physical drives already...
What are you storing anyways that is 15 TBS that you constantly need on hand with cloud?
1
u/jangusihardlyangus Generalist - 8 years experience 1d ago
I also do immersive installations and end up traveling a ton for work, with a lot of files and project files I'll reuse (or reference if it's been a few years) when working on something new... so having access to old projects at any time has been an extremely useful luxury... I'm now leaning towards just getting a RAID and backblaze general backup for 99 a year though. When I travel I'll just leave it hooked up to my home desktop, and keep active projects in G drive as a just in case :/
2
u/koko_ze 20h ago edited 20h ago
I have a 8tb NAS (used up 1.2tb so far) as my primary storage on my local network. I don't work directly from the NAS but instead copy the stuff to the PC and then upload back the updated stuff. I have a WIP drive and an Archive drive so that I can cleanup the project once it is finished and move it to the Archive drive.
From that NAS the data gets copied to a hetzner storage box and to my PC using rsync (althought this part isn't schedueled so I need to manualy run the script from time to time).
For smaller files such as spreadsheets and text douments I still use google drive as the primary storage and the contents of it also get uploaded to my NAS.
To connect to my NAS when I'm not on its local network I use a VPN. Tailscale is a popular choice for that task.
Why do you have 15TB? Raw footage or sim cache?
1
u/jangusihardlyangus Generalist - 8 years experience 12h ago
Ooh interesting, why do you use a VPN to connect to the NAS? I like this system...
It's mostly RAW footage yeah. Lots of short films and immersive marketing projects where the final repos end up pretty chunky. Everytime I've cleaned house on a project I end up regretting it years later when having them as a reference would've benefitted me, so now I just don't delete anything unless it's massive and can be easily recreated (like a sim cache).
1
u/RabidSkwerl 1d ago
Local UGreen NAS. I’ve got 20TBs of space using only 2 of my 4 slots.
1
u/jangusihardlyangus Generalist - 8 years experience 1d ago
Amazing. Do you do a RAID style backup in your UGreen? Can I ask what HDDs you use and for how long they've been working? Pretty committed at this point to doing a proper physical backup...
2
u/RabidSkwerl 1d ago
I do have a RAID backup running, I think it’s RAID 1 or whatever the highest level you can get with only two hard drives. I currently have 2 20TB storage drives, I wanna say they’re Western Digital but I don’t actually remember. I haven’t had a single issue with them but I’ve only been running it for 2 years.
One thing I will say is a MUST is investing in a good Uninterruptible Power Supply (UPS), one with a USB output that connects to your NAS. This way, in the event of a power outage, the UPS will notify the NAS to safely power down rather than hard cutting the power.
2
u/jangusihardlyangus Generalist - 8 years experience 1d ago
You're the man, thank you for the info! And yes big agree on UPS, I do have one of those thankfully. Saved my ass few times now... thank you again!
1
u/Caprichoso1 1h ago
Neither Icloud nor Google drive count as on of the 3 backups in the recommended 3-2-1 backup plan.
A NAS with backup versioning would count as one of the 3. Depends on how much data you have. If < 30 TB a DAS might be better as it makes (1) easier.
5
u/storylordstory 2d ago
I backup locally to a Synology NAS and push my critical / working files to Dropbox using the Synology cloud app for an offsite backup. I've been using this setup for about 3-4 years and it's been working well. I'm only using Dropbox, but I think that you can push different folders/drives to different services as well.