How I Racked Up $14,000 in AWS Bills with My Side Project

I've been building my side project easyCDN for over a year. I've always been drawn towards sleek architecture and infrastructure. I love the idea of serverless and infinite scaling, as well as high performance asset distribution. However, all of this had to be simple. I've set up quite a bit of infrastructure over the years, across AWS, Azure and Google Cloud. All of them offer a vast variety of infrastructure, but it comes with a learning curve. Also, a couple of years back, we built Assetizr (now discontinued), which actually gained some traction. The 2nd version of Assetizr was basically a precursor to easyCDN.

Now, before you jump to the conclusion that the serverless setup caused some infinite calls to the serverless function, I am happy to announce my setup was flawless in that regard (until proven otherwise).

Before I can explain what the flaw was, I need to explain a bit more what easyCDN actually did. Also, as expected, AWS did not make any mistake, the charges were justified, albeit unexpected. I had a bill of a couple of dollars per month on AWS beforehand, which jumped to $14k suddenly.

Now, let's have a look at how things got out of control. I built easyCDN with the premise of offering a simpler solution to hosting assets, serving through a CDN, as well as offering more functionality out of the box, all crafted with ease of use as first priority. After signing up for easyCDN (even the free tier), you'd be able to drag'n'drop files and upload them to easyCDN or integrate upload functionality into your existing application in a couple of minutes. For every file, you'd get a CDN url instantly to embed in your app or wherever.

Sure, you get drag'n'drop uploads with S3 as well, but setting up the CDN can be a bit more tricky, especially if you're not well versed. But even the drag'n'drop is cumbersome. Then comes integrating it in your app. Server-side uploads are usually fairly simple. However, integrating into a frontend application brings less joy. Handling presigned uploads, building a drag'n'drop component, parallel uploads, graceful error handling, retries and so on. With easyCDN, I tried to bring everything together and offer the whole thing in a couple of clicks. For that, I had to host all assets on my own AWS account and route the traffic through my own Cloudfront distribution. Sure, that is the easiest solution (not without drawbacks) for new users, but it means I have to actually pay for storage / bandwidth. To achieve positive cashflow, tight guardrails had to be put in place.

As I mentioned before, I did offer a free tier. The free tier included 2GB of storage, as well as 2GB of bandwidth. This should have prevented me from getting any crazy bills, at least unless I have a lot of users. I also correctly implemented upload storage limits, as well as bandwidth monitoring. Upload storage limits are fairly easy to implement, you just check against the existing uploads and their storage usage and possibly throw an error. For bandwidth monitoring, it gets a bit more complicated. By default, Cloudfront (AWS' CDN solution) does not show any bandwidth. However, you can enable Cloudfront logs, which will write the requests to a text file in an S3 bucket. Fair enough, but then you need to parse that text file, extract and aggregate any usage based on period. All in all, not rocket science, but not trivial either. Once again, it proves my original thought that building a wrapper around AWS to make it easier for users is a valid business model (hello Vercel).

So if I had all this in place, why the massive, unexpected bill? Well, there are multiple factors. For one, my bandwidth monitoring stopped working at a certain point. I actually did notice it, but decided to postpone the fix: usage was limited and the potential for abuse seemed low. My thinking (now proven wrong) was that bandwidth usually comes from many files being served a moderate number of times. If individual assets were served a lot (e.g. a million times a day), the venture would most likely be more sophisticated and already have its own CDN set up. Second, I stopped checking my AWS account (billing) for a while, for the same reason. I always had single digit dollar invoices per month.

But wait, I still had the storage limit! If users weren't able to upload many/large files, it must have been the amount of requests. Exactly... For the cost to be incurred, I had one file requested 37 million times in a single month, meaning >1 million times per day, resulting in 186.37TB of bandwidth. Now, that's a lot of traffic. If any of my projects had 10 million visits per day, I guess I wouldn't have to worry about a side project not having a sufficient user base.

What/Who caused this massive traffic? Turns out it was some GTA V modder. I still don't know how exactly this traffic came to be, but I assume he posted it to a modding community and, in addition, many clients repeatedly tried to download the file (a video). I assume there was a bug on their side as well, causing the repeated downloads. I can't imagine a modding community has >1 million visits per day...

This went on for a full month, until I woke up one morning to a new email from AWS, with a billing statement of ~$14k attached.

What's the silver lining? Well, I had active users and my side project was actually useful. The downside? The combination of errors/glitches turned my side project into a money pit real quick.

Props to Amazon

After finding out about the cost, I immediately turned off the Cloudfront distribution and investigated where the usage came from. I also contacted AWS support right afterwards. Luckily, after a couple of messages back and forth and 2-3 weeks of uncertainty, AWS decided to waive the bill and thus the whole story turned into a lesson paid in cortisol rather than hard cash. Thanks AWS.

What about easyCDN?

As you can imagine, I wouldn't be able to sleep well turning easyCDN back on as is, knowing something like this could happen again. I spent a couple of weeks refactoring easyCDN to work with users' own infrastructure. This means users can now connect their bucket/CDN and use easyCDN's management layer, SDKs and extended functionality (resize, scheduled asset deletion, etc.). That way users keep full control over their assets and their infrastructure bill, and so do I. Even if users decide to no longer use easyCDN, their assets and CDN urls will continue to work. It currently works with Cloudflare. I am planning to offer AWS support as well in the near future. If you're curious, feel free to check it out at easycdn.co and give me some feedback!

Hope you liked to hear about my mishaps with my side project. Let me know if you encountered anything similar.

Keep building!