In the middle of August we noticed something weird started happening to our bandwidth usage (how many people are accessing our site). Our first reaction was, of course, "cool, people are using the site more!" But there was something suspicious going on. When we looked at some graphs, this is what we saw:
That wasn't a steady increase, that was more like something found us and then started bombarding our server. When we looked at our logs, we saw that we were getting hit every couple of seconds by a variety of sources. It was a regular pattern. We went from roughly 100-200mb of bandwidth every hour to 2 GB, in the span of one hour! At this rate we would have an astronomical bill coming at the end of August. At the end of this, we got a graph from our server provider Render that confirmed what we suspected: most of this was automated.
We went from roughly 100-200mb of bandwidth every hour to 2 GB, in the span of one hour!
With the help of the wonderful folks at Ampwall (who know so much about devops and these kinds of things, we're incredibly grateful), we had already been doing some thinking about this, so we had a pretty good idea of what we had to do. First, we turned up our security levels, and implemented some very heavy rate limiting on specific endpoints (that's where you see the traffic decrease in the first graph, back to normal levels). Sadly, this made the site almost unusable for logged out users, but it gave us time to put in place other systems. We switched our DNS provider over to allow for proxying by Cloudflare, who have some good systems in place for preventing bot (and especially AI!) traffic. We had some good communication with our server provider who is now putting features in place to tackle this kind of thing. It looks like all of this has scared away whoever was barraging our site.
Some other things we'll likely work on is user trust levels, where users gain trust as they use the site, and more functionality becomes available to them. This is fairly common in larger enterprise software. It'll also let us degrade experiences for specific sets of users if we come under attack again, keeping our more loyal users happily using our site.
It's an indictment of the current structure of the internet that content is seen nothing more than a farm to mine as AI fodder.
Of course, this kind of stuff detracts from the things we love doing: building usable software. It's an indictment of the current structure of the internet that content is seen nothing more than a farm to mine as AI fodder. That a profit motive encourages scammers to take advantage of small sites and producers. That AI scams are running amok. This is, of course, part of the process of enshittification. By putting in features to protect ourselves from scammers, we've made the experience worse for everyone.
A note about Cloudflare: using Cloudflare for managing incoming data had some side effects. One is that because of a misunderstanding of how the internet works, Spanish users now can't access our site during La Liga games. Absurd. And we don't quite know yet how we'll circumvent this. Cloudflare also throttles uploads over 150mb, and we're working our way around that.
It's free, so sometimes you get a weird e-mail.
Mirlo is run by a collective of people squeezing minutes out of their days to make the site and everything around it real. We're entirely bootstrapped and crowd funded. But we don't make enough money on a monthly basis to pay anyone even part time to work on this project. This, of course, means that when shit hits the fan, we're stressing out, working late nights, and trying to handle all of the communication associated with it. It means that sometimes we accidentally send out a cryptic e-mail to 3000 users.
When we set out to build Mirlo, we thought even having 20 artists use it to make sales would have been a success. This summer we surpassed 1000 artists who trust us to upload their music, make sales, and advocate for us across the Internet. However, we've also noticed a plateau in how much people are able or willing to support us. While we can cover our operating expenses and pay for some niceties (like social media management accounts), we do not make enough money to cover the cost of working on Mirlo. This means that every communication that we do, every e-mail we respond to, every graphic we make, every post, every feature implemented, and every bug fixed, is done in our spare time.
This means that every communication that we do, every e-mail we respond to, every graphic we make, every post, every feature implemented, and every bug fixed, is done in our spare time.
We currently make just under $500 a month in supporting subscriptions. This is enough money to run a hobby site, but it is not enough money to make sure we have someone on call when things go wrong. We're incredibly grateful to the 34 supporters who pitch in monthly or yearly to help us out. But to make Mirlo sustainable, we need people to either pitch in financially or volunteer to help keep things working.
This is a companion discussion topic for the original entry at https://mirlo.space/team/posts/381
