I once set an S3 lifecycle setting that accidentally affected 3 years worth of logs to Glacier. The next morning I woke up to a billing alert and an AWS bill with an extra $250k in charges (our normal run rate was $30k/month at the time). Basically I spent my entire add annual cloud budget for the year overnight.
Thankfully after an email to our account rep and a bunch of back and forth I was able to get the charges reduced to $4,300.
The problem is having a competent team to manage your infrastructure. You can do a lot with a handful of people - but you need competences spanning a lot of areas, and finding that is pretty hard.
If you can get a competent team the only advantage cloud still has is the ability to quickly scale up and down - but if there might be a need for that it’d still be better to go hybrid, most on your own hardware, and just the prepared ability to quickly bring up cloud workers if needed. The cost savings of properly doing it yourself are so huge that it still might be cheaper to just have some pre-provisioned standby hardware for that, though.
If I never have to buzz into another colo and stand in the exhaust of hundreds of servers again, it’s worth every single penny. If I never have to plan for capacity weeks to years in advance again, its worth every penny.
Depends on your needs. If you expect to grow fast and unpredictably, or have extreme burst workloads (at my company it fluctuates between requiring ~10 cpus to ~50,000, and between 0 GPUs and dozens) or if you need several complex types of services and no people at hand who can manage them, it can be way cheaper. If you just need a few servers, a tape backup and a database, actual hardware has always been cheaper.
I made this mistake but honestly? AWS is the most confusing clusterfuck of all time. I can’t stand it and refuse to use it for personal projects.
For me the problem came down to four conflicting sources on AWS regarding tiers and then another problem with the SDK. The SDK didn’t match the tiers at all so “archive” meant Glacier for some reason. 👎
Deleting from. We move logstores and I added an ageout policy for anything over 1 day, to “easily” empty a bucket overnight. I forgot that I had been cycling stuff to glacier after 6 months, and there were 3 years of logs in there.
I once set an S3 lifecycle setting that accidentally affected 3 years worth of logs to Glacier. The next morning I woke up to a billing alert and an AWS bill with an extra $250k in charges (our normal run rate was $30k/month at the time). Basically I spent my entire add annual cloud budget for the year overnight.
Thankfully after an email to our account rep and a bunch of back and forth I was able to get the charges reduced to $4,300.
Is cloud even cheaper than managing your own infrastructure anymore?
The problem is having a competent team to manage your infrastructure. You can do a lot with a handful of people - but you need competences spanning a lot of areas, and finding that is pretty hard.
If you can get a competent team the only advantage cloud still has is the ability to quickly scale up and down - but if there might be a need for that it’d still be better to go hybrid, most on your own hardware, and just the prepared ability to quickly bring up cloud workers if needed. The cost savings of properly doing it yourself are so huge that it still might be cheaper to just have some pre-provisioned standby hardware for that, though.
If I never have to buzz into another colo and stand in the exhaust of hundreds of servers again, it’s worth every single penny. If I never have to plan for capacity weeks to years in advance again, its worth every penny.
Depends on your needs. If you expect to grow fast and unpredictably, or have extreme burst workloads (at my company it fluctuates between requiring ~10 cpus to ~50,000, and between 0 GPUs and dozens) or if you need several complex types of services and no people at hand who can manage them, it can be way cheaper. If you just need a few servers, a tape backup and a database, actual hardware has always been cheaper.
It depends on the workload. Some workloads do well on other people’s computers, some are better on your own computers. One size does not fit all.
I made this mistake but honestly? AWS is the most confusing clusterfuck of all time. I can’t stand it and refuse to use it for personal projects.
For me the problem came down to four conflicting sources on AWS regarding tiers and then another problem with the SDK. The SDK didn’t match the tiers at all so “archive” meant Glacier for some reason. 👎
Yeah luckily Amazon is good about mess-ups that are one-time like this. Was the cost because you were pushing to, or retrieving from Glacier?
Deleting from. We move logstores and I added an ageout policy for anything over 1 day, to “easily” empty a bucket overnight. I forgot that I had been cycling stuff to glacier after 6 months, and there were 3 years of logs in there.