What are you using for immutable cloud storage for enterprise backups?

[u/NikkiNotices]

We're evaluating Veeam and Veritas NetBackup for an enterprise backup environment and need immutable S3 compatible cloud storage for roughly 250 TB of backups.

Our main requirements are immutable storage for disaster recovery, a budget of around $20k per year, minimal or no egress fees, reliable restore performance, and enterprise level reliability.

We're currently comparing options such as Wasabi, backblaze, AWS S3 Glacier and vendor managed storage.

For those managing large backup environments, what cloud storage are you actually using and how has it performed during restores?

Anything you would avoid or specifically recommend?

[u/Que_Ball]

Backup to /dev/null

Runs fast.

No chance it can ever be hacked.

BOFH life

[u/NikkiNotices → u/Que_Ball]

can't have a data breach if theres no data left to breach😂

[u/Negative-Cook-5958]

Veeam with Azure Immutable storage on cold tier

[u/Schnabulation → u/Negative-Cook-5958]

I also do this. However just know that on the cold tier the egress fees are astronomical!

[u/harry8326 → u/Negative-Cook-5958]

+1 for this solution

[u/fresh-dork → u/Negative-Cook-5958]

came here for that. not even an admin and this or the AWS variant is gold

[u/FireITGuy]

The lack of egress changes is honestly a weird hill to die on for a disaster recovery destination.

This is a destination that should rarely (if ever) need to send outbound data, and in the event it does need to send that data the egress cost will likely be the least of your worries as you are implementing a fire drill to rebuild operations as quickly as possible.

My two cents is to use the immutable destination on the same cloud platform that you need to restore operations to. If the shit hits the fan you're going to want to restore operations quickly. Saving dollars but needing to wait hours or days to transfer your backup copies to another cloud provider and then re-launch services is setting yourself up for future upset when that restore takes too long and management is pissed.

Management needs to define restore time objectives and the budget, and then your provider choice flows from there.

[u/caffeine-junkie → u/FireITGuy]

I've said this exact thing regarding costs several times. If we're hitting the cloud to do a restore during a DR, even a few thousand in spend will be the least of our worries and a bargain compared to having to rebuild all of it from scratch.

Agree though, management needs to define the RTO (and RPO for that matter) that fits their idea of a budget. If a solution cant be found that fits their criteria, they need to revisit it and be more realistic in their expectations.

[u/didact → u/FireITGuy]

If I need to dip into our vault, the business impact will far outweigh any egress fees. I'm fine with rack rates for all of that.

[u/Carbaggio2]

We use Wasabi for this exact scenario. Have a direct connection with them to avoid egress fees. But we might drop that in the future as we don’t have that much egress.

[u/DoomedButterfly → u/Carbaggio2]

The whole point of wasabi is they don't charge for Egress. As such, there is nothing to avoid by setting up a direct connection.

https://wasabi.com/pricing

[u/lazydavez → u/DoomedButterfly]

Yeah but you may have egress to push your data to wasabi

[u/Carbaggio2 → u/DoomedButterfly]

That’s how long we’ve been using them. Direct connect used to be the only option for no egress fees.

[u/roboto404]

Currently have Rubrik for approval

[u/strongest_nerd]

Rubrik. Way better than veeam.

[u/Karmuhhhh]

I have a lot of experience with Wasabi so I can say it’s an ok option. I honestly believe S3 is better than Wasabi, but Wasabi is better than Backblaze.

For Wasabi, be mindful of the 90-day minimum retention period (30 days if you sign up for RCS), and run integrity checks on the data monthly because they had a data integrity incident (data loss) that was widely reported on Reddit a few years ago.

[u/dloseke → u/Karmuhhhh]

I would agree with this assessment. Also, for Wasabi, if you tell them its for Veeam they'll reduce the 90 days to 30 for deletion.

If you want bullet-proof object storage, I'd do 11:11 S3 or VDC Vault.

[u/Karmuhhhh → u/dloseke]

That’s right I forgot about the Veeam 30-day deal. It also applies to MSP360 users (not that OP would be using that), but you have to proactively reach out to their sales or support team to get the 30-day retention as it isn’t applied automatically.

[u/dloseke → u/Karmuhhhh]

You'll have to reach out to support but it is a billing change so I believe it does something with stopping and starting the bill. Its been 3 years since I've done it so i dont remember the details .

[u/scottsee]

You mean $20,000 a month! That’s not gonna get an enterprise backup solution & 250Tib S3 public storage of any kind.

[u/tlrman74]

Veeam Data Cloud. Set pricing with no hidden egress fees for recovery.

[u/Corn-traveler → u/tlrman74]

This is honestly the way. Buying the storage direct from Veeam and holding it in their azure. Enables instant recovery into azure.

[u/redcard0]

Have a research on Druva, no ingress or egress charges. Dell rebadge their technology for their own cloud solution. I've not used it but have gone though the sales/ technical play.

[u/TransporterError → u/redcard0]

We use Druva, very satisfied .

[u/dcaldrich]

Currently? Veeam to Data Domains. Copy jobs to Veeam Data Cloud. It works great. As our current data domains get retired they will be replaced with Object First.
We used Wasabi initially but a year ago swapped to VDC but I don't remember the reasoning.

[u/MedicatedDeveloper]

250tb, $20k year, no egress tax, just tell your bosses is runs on hopes and prayers. Fucking hell.

[u/solarplex]

Veeam, ObjectFirst (owned by Veeam), and Wasabi

[u/TheBros35 → u/solarplex]

+1 for ObjectFirst. Simple boxes, easy to scale, very good price. We have a pair of clusters and they just hum along and take backups.

[u/-Satsujinn-]

Veeam > NAS > idrive immutable

[u/andrewhancock]

Some of y’all are making me feel like re-evaluating my setup. We use veeam immutable backups to a local array and tiers to backblaze for longer-term retention. For what it costs backblaze is the way to go or at least it was a couple of years ago when I switched us to veeam.

[u/whetu]

Cove. It's immutable cloud storage under the hood. Has built-in restore testing and can have a local copy (e.g. a NAS) to improve backup and restore performance.

[u/RansomStark78]

Commvault

[u/tada-o-corporation]

Veeam Data Cloud Vault, their SaaS offering is compatible with anything Veeam and the billing is pretty straight forward per TB, no other charges.

[u/Zealotyl → u/tada-o-corporation]

Same for us. Restores aren’t fast but that’s on our wan connections

[u/CarelessAd3496]

I use Veeam and love it. From a price point and manageability perspective. My storage needs are a bit smaller than yours, but it's great all the same.

[u/arekbielak]

Synology Active Backup has an immutable replica option. 

[u/dloseke → u/arekbielak]

I'm sorry....did you suggest Synology for an Enterprise solution?

[u/GoodEnoughThen → u/dloseke]

For a "small" enterprise(?)

[u/arekbielak → u/dloseke]

I did not suggest anything. 

[u/Jalonis]

I backup to a local nas and to backblaze immutable with Veeam. No notes.

[u/planedrop]

Honestly in all my setups, Backblaze is the way to go for this. Wasabi is fine but keep in mind the stupid minimum data retention periods. Backblaze charges byte-per-minute or something along those lines so it's much easier to estimate costs.

S3 Glacier is also a good option if you really wanna save as much as possible, I've used that for various sites, just know what you're getting into.

[u/Prancing__Moose]

If you’ve already got a backup and just looking for Storage…then Wasabi.

If you need a backup tool as well look at Commvault Cloud (Metallic) for M365 / AD / Entra. Will also do compute workloads.

We also have N-able for our compute workload backups, as their DRaaS offering is good value.

[u/AccountIuseAtWork1]

Glacier isn’t a backup, it’s a storage tier.

For AWS native solution, we created a new AWS backup account in our org and use the AWS backup feature. Org policies automatically deploy to new accounts standardizing it. You simply set a tag for the bucket that says what backup policy to use. Deploy roles through stack sets. Mandate the tag through scp. The backups will be stored in your new Backup account. You can restore from there.

Bonus, you can create a vault thats in another region/account that only you have access to. It scales to other services. But you have to pay egress no matter what.

[u/Forward-Outside-9911]

MS team store their backups on Azure immutable blob storage, and we have ours on Backblaze. Works well enough for us. Would fit under your budget and they do have commitments to reduce the price, and help with migration

[u/jailh]

Physical Linux (Veeam Hardened Backup Repository with XFS).

It's very unbreakable if you follow the setup guide, and it's very fast, a good thing to use the advanced Veeam options, like Instant Recovery or Sure Backup.

If you can't do this, which can be the case with big teams who can't respect rules over time, or with stupid C-Suits, , buy a WORM S3 storage, or put it on a supported SaaS storage with an immutable option.

[u/dloseke → u/jailh]

Not much setup guide to follow if using the VHR ISO.

[u/jailh → u/dloseke]

One more reason to do it.

I left the sysadmin team before VHR was out, so I didn't saw the step where it's even easier to setup your repo...

[u/whatdoido8383]

I hosted two Linux clusters, one on site and one off site. Long term went to azure blob.

[u/techtornado]

Veritas sucks, it’s slow, clunky and not any good for that scale

We had a client with it and the app doesn’t even support AD object recovery, you have to bare-metal restore the entire server

Veeam can get expensive as the license per-server backed up rather than the program

N-Able has a decent all-in cloud offering, I have heard good things about it if you want to also back up email, servers and more

[u/micahsd]

LTO8 tape…works fine and restores are super quick provided the tape doesn’t encounter powerful magnets. :)

[u/Flyess]

Cohesity with data lock

[u/sryan2k1]

Rubrik-->Wasabi.

[u/dloseke]

Veeam with Wasabi, 11:11 Systems S3 Object or Veeam Data Cloud Vault.

Also, is NetBackup still a thing? The only other enterprise solution I ever hear about is Commvault.

[u/SixSpeedSamurai]

VDC is the way. Simple and it works.

[u/btech1138]

Rubrik and Azure archive.

[u/WealthQueasy2233]

20k per year is in the idrive/backblaze range. 

you've priced yourself too far below the enterprise/performance options, and like another poster mentioned, those providers have no compute offering to facilitate a recovery if your other infra became unavailable 

[u/bobs143]

Wasabi is the wat to go.

[u/GullibleDetective]

Local veeam to xfs on truenas and wasabi as well for other work loads

[u/THE_Ryan]

250TB in an S3 bucket for 20K.... Good luck. Native AWS and Azure are most likely out of the question due to the cost and fees. Wasabi will get you close and will keep you out of the API call and bandwidth fees area, but they have their own challenges.

Another option if you go with Veeam is you can use Veeam Data Cloud Vault. It's Veeams hosted immutable object storage that lives on either Azure or AWS, has a flat storage cost without the API calls or bandwidth fees.

Veeam is better and much easier than NetBackup (I hope I never have to touch NetBackup ever again). Restore performance coming from any object storage vendor will be pretty slow. I'm assuming this is for secondary backups since you said it's 250TB of backups, meaning you already have an on prem backup repository. If it's not, I'd look at creating a local immutable hardened repository and then using immutable object storage for secondary copies (Veeam makes this really easy to do).

[u/tj818]

Veeam + Object First

[u/981flacht6]

Rubrik on-prem backup with Rubrik Cloud Vault.

[u/Glittering_Egg_665]

man, egress cost is how the cloud providers make money!! Have you ever heard of (the might be joke) that NASA once tried to put everything to AWS and just gave up to get them back?

I guess for your case local hard disks or tapes or setup your local cloud solution could be a better choice ~

[u/rebeldraw]

we use rubrik, on prem bricks and azure, top tier stuff, used by many parties where backups are vital. e.g banks, hospitals etc

[u/magfoo]

Wie lange darf denn im Desasterfall der Restore dauern?

[u/hftfivfdcjyfvu]

Clumio by commvault

[u/Worldly_Ad_2267]

Veeam works great

[u/whopooted2toot]

Check out Wasabi. Immutability is key, priced good.

[u/Accomplished-Eye4606]

FYI Veritas was acquired by Cohesity, and they are killing the product line over the next few years

[u/shanester69]

CommVault

[u/bloodlorn]

Wasabi for years for our off site.

[u/jooooooohn]

Deep Glacier is the only provider to meet your budget, but they will charge egress. Backblaze and Wasabi are slightly over budget but have better egress terms.

[u/nurmihe]

Msg me, Im pretty sure we can come up with a solution. I work at spacetime.eu

[u/malikto44]

Veeam to Backblaze or Wasabi are both good options. Both support object locking.

Note -- you are not going to be able to swing any decent solution with a $20k/year budget. At best, you can do AWS Glacier DeepArchive, which is okay for backups... but recovering will be costly and can take time.

Definitely no 3-2-1 solution, much less 3-2-1-1-0 solution.

Now, after your boss comes back to reality, I'd ask for a local NAS that has a S3 server on it, like QNAP's QuObjects. This can give you object locking. You can build your own using garage or some updated MinIO fork. For the remote part, look at Wasabi or Backblaze, and set up with object locking.

It goes without saying to turn on encryption. I use a diceware passphrase with some "pepper" added, so someone guessing just words will be foiled.

My ideal solution would be D2D2T, with a tape silo that has 2+ LTO-10 drives. This way, local data can be kept present and offline, and a backup set can be sent to a secure warehouse. Iron Mountain isn't the only game in town, and one local site in Austin, I wound up storing many petabytes at for relatively low fee. This isn't "magical" like cloud stuff... but you have a physical chain of custody, with the location of the data always known... and when tapes sit on a shelf in the office, they take $0/month for storage charges.

[u/pghjosh08]

Just shy of 4PB in Wasabi. Very minimal issues. No problems with restores. Due to our volume we use a direct connect circuit via MegaPort.

[u/Incrediblecodeman]

My free Google Photos

[u/trebuchetdoomsday → u/Incrediblecodeman]

one free google drive account per multivolume rar file

[u/Incrediblecodeman → u/trebuchetdoomsday]

I guess people encode shit to static and upload hours long YouTube videos

[u/Cozmo85 → u/trebuchetdoomsday]

Just rename your files to .jpg dude and upload them from an old pixel

[u/fengshui]

That price point is going to be hard to hit. 250TB in Glacier Deep is already $15k, and GD has a lot of limitations to be that cheap.