Notes on serving images and a brief Mastodon server outage

I’ve been running my own Mastodon server on Amazon Web Services for less than a week. The first challenge was to get anything working at all. This took me a solid two days to work out, but once I did, I was in business and having fun learning all about the “fediverse.”

Shortly after launch, I set up Amazon CloudWatch to give myself a nice dashboard for all my critical server metrics, as well as alerting, so I’d know when my little EC2 instance was struggling. And, I’m glad I did because shortly after setting up my alarms, I had a server outage. These are my notes:

1. The server outage was caused by my t4g.small instance running out of resources for both CPU and Memory. It lasted for about 30 minutes and I wound up doing a full stop and start to bring it all back online.
2. I’m not entirely sure what caused this to happen, but it looks more and more like a long-running process was to blame. After looking through logs and charts, I’ve narrowed the issue down to a periodic job that runs when you have “Trends” turned on. I’ve since switched off that feature temporarily while I do more research.
3. This also prompted me to change the threshold for my alerts, since they didn’t trigger, and I only figured out the server was offline when I tried to visit the site and wasn’t able to.
4. When running a single user Mastodon instance, it actually doesn’t really matter if you go offline for periods of time.

All of this also prompted me to get back to figuring out how to properly serve images from Amazon Simple Storage Service (S3) using Amazon CloudFront as a Content Delivery Network (CDN). This was harder than it needed to be. Here are my notes:

1. I had my CDN and S3 bucket properly configured and serving images I manually uploaded as a test. I had used an Origin Access Identity to keep the bucket itself locked down, but to allow CloudFront to fetch and serve the images via the CDN.
2. I found several write-ups on how to configure Mastodon to use S3, but they were mostly predicated on serving the images directly from S3 using S3’s Static Website Hosting feature, which was not what I wanted to do.

In the end, I needed to turn off the Block Public Access setting on my bucket. Doing this doesn’t mean my bucket is open to the world, but it does allow the Mastodon web service to correctly talk to the S3 API. I’m not entirely sure why it has to be this way, but after looking through the code where Mastodon uses a library called kt-paperclip, it became apparent that this was the issue.

Once I made this change, everything started to work. I synced up the existing folder of static assets with my S3 bucket, restarted the mastodon-web service and everything came back to life.

Mastodon appears to be a really good copy machine, and makes cache copies of all kinds of image files, videos, and avatars based on the accounts you follow. There is a setting to clean these up after a certain number of days.

Here is a chart of CPU and Memory for my last three days running a Mastodon server on a single t4g.small instance. (The group of spikes on Thanksgiving are the server outage.)

For the next month I am going to keep things as is and continue to monitor and learn about how the server operates. Then (when my t4g.small Free Trial ends) I am going to decided if this is the best instance type and either buy a Savings Plan or switch to a different type based on my findings.

Wish me luck!