Hi, I'm Courtney! I'm a North American wife, mom and UX engineer.

I love video games where you farm stuff.*


All opinions are my own, and do not reflect the attitudes of the company I work for. They know better.

* No, not Farming Simulator, sorry.

© 2009 – 2023 Courtney Cloudman

Lessons Learned Hosting and Maintaining a Mastodon Instance

When Elon Musk bought Twitter, gutted its staff, and welcomed back its most poisonous users, it triggered an exodus. Many if not most of Twitter's users still remain, but a migration of this scale hasn't really been witnessed since Digg unleashed version 4 to a revolted userbase in 2010.

Years ago, already seeking an alternative, I joined a Mastodon instance. I didn't find it much fun in 2018, when relatively few people were there. When the migration brought a huge wave of users to the platform, it didn't take long for me to find my people. I locked down my Twitter account and stopped posting there.

Soon after that, I began to consider hosting my own Mastodon instance. I started reading a lot about other people's experiences doing so, and I realized that this would not be nearly as simple as spinning up a simple Wordpress blog. It made me hesitate - yet for some reason, I was drawn to the challenge. I'm not a sysadmin, and this was a great learning opportunity in that respect.

This post isn't about selling you on Mastodon, or teaching you how to run your own instance. Instead, I'm sharing what I've found and what I've learned. There's a lot of important information scattered across the Internet, difficult to find if you don't know where to look. My goal is to compile more of that information in one place, and help point others thinking of running instances in the right direction.

Table of Contents

A Simplified Overview of Mastodon & the Fediverse

Mastodon, unlike Twitter or Instagram, is not a single web service owned by a global corporation. It's an open-source software platform that connects to a global network of federated social media servers, popularly referred to as the "Fediverse." Mastodon is merely the most common platform among many, implementing common communication standards (notably ActivityPub). When a user on one instance makes a post, that instance pushes that post to all the instances where someone is following that user.

If you want to run your own instance of Mastodon, you'll need to know about the components that make up a functioning installation. A typical, smaller instance will be made up of these services:

  • Mastodon (Web): The core web application
  • Mastodon (Streaming API): Provides the streaming connections for users to get real-time updates
  • Mastodon (Sidekiq): Runs & manages scheduled background tasks, like processing incoming messages
  • Postgres: The primary database
  • Redis: A key-value store used for caching, as well as tracking timelines and Sidekiq tasks
  • Reverse Proxy/Web Server: Something like NGINX or Caddy, which handles incoming HTTP requests and forwards to web/streaming as needed
  • Media Store: Could be a folder on the server, but more likely a third-party service like Amazon S3 or Backblaze B2

I set up my instance about a month and a half ago. Now that the dust has settled, these are some of my take-aways from the experience:

Start by Joining Someone Else's Instance

If you jump straight into running your own instance, then you'll find yourself pretty lonely. If you don't already know who to follow, then the best way to find people is through the local timeline, where you see posts from other users on the server. On your own instance, however, you're the only one in there.

Same with the federated timeline, which shows posts from everyone who users on the server follow. On your own instance, it'll just be a partial copy of your timeline - unless you add a relay, which we'll get to.

I highly recommend being on another server to start, especially one aimed at a particular crowd that you identify with. You can find people you'll like following, and they can find you on that local timeline too. When you're happy with who you're following & interacting with, you can migrate without losing that community.

It might also be a good idea to follow other instance administrators. [1] Some of them talk about their job, and they can provide valuable information on occasion! [2]

Hosting With Docker Compose

The official documentation provides instructions for hosting Mastodon, and there are Docker packages available. I wanted to use Docker Compose to organize & control the services, however, and the Mastodon documentation doesn't cover that scenario very well.

I found Ben Tasker's guide to running a Mastodon Instance using docker-compose very helpful! You can pretty much follow it step-by-step and it covered the vast majority of what I needed to do.

I made a tweak to those directions in the name of ease-of-maintenance, by opting to use Nginx Proxy Manager to manage the reverse proxy and SSL. If you do so, make sure that the locations /api/v1/streaming and api/v2/streaming both point to your streaming server, and not the web server.

Relays Are Useful, but Watch Out...

To add content to your instance beyond yourself and the people you follow, you can connect it to one or more relays. When someone posts on an instance connected to a relay, that post gets relayed to all other connected instances.

These relays can be a general purpose, un-moderated free-for-all, where any instance can connect - and if you do, who knows what you're getting. Other relays are moderated communities, specific to a certain demographic. They may have less content, but more of it may be meaningful to you.

You won't be able to see all of the posts out there, no matter how many relays you join. The largest instances have little need for relays, since just about everyone is followed by somebody there, so they're not connected to any.

You may find a high volume relay with open sign-ups. This can do a lot to fill your federated timeline, of course, but you're also likely to find out just how robust your setup is.

Sidekiq Issues Come at You Fast

I found one of those higher volume relays a month ago. I was excited to see posts pour into my federated timeline... but then I realized those posts were getting older and older, and so were the posts on my home timeline.

In the administration interface, you'll find a link to the Sidekiq dashboard. Sidekiq, you'll recall, is the service for processing background tasks. Long story short: if you go there, and see the Enqueued job number growing and growing, then it's being asked to process more jobs than it can handle.

This seems to be quite a common issue, especially after the first big wave of ex-Twitter-users hit the Fediverse. Nora Tindall has an excellent post explaining how to increase the throughput between Postgres and Sidekiq, so that it can handle far greater volumes. It involves tuning Postgres with PgTune, creating separate Sidekiq instances for each queue, and providing each instance with as many connections as Postgres can spare.

For my single-user server, I only ran PgTune and configured the single Sidekiq queue to use 25 connections. This has proved to be more than enough in my experience, even when a user with tens of thousands of followers shares my posts.

Speaking of Nóva, she and her fellow Hachyderm admins have experienced plenty of scaling problems as their server gained tens of thousands of users over the course of a few weeks. They've been very open about it, posting and streaming through it all. One of their other admins, Hazel Weakly, wrote up Scaling Mastodon: The Compendium, where they gathered scaling tips from all over the place into one long post. If I ever run into further issues with my instance, that will be one of the first places I turn to.

Handling All That Media

My server receives around 25,000 posts a day from its relays and (to a much lesser extent) from the people I follow. A lot of those posts have attached media, and the storage requirements add up quick.

Currently, there is 50GB of media attachments stored for my server. This is despite quite aggressive retention settings - 1 day for cached media, 7 days for cached content and 14 days for user archives.

If storage space is at a premium - and on most cheap VPS's, it very much is - you shouldn't expect to store files locally without issues. I would use cloud storage right from the beginning - not least because uploading local files to the cloud later on is a pain in the butt.

I ended up going with Backblaze B2 for storing my media. While I've heard of high latency with B2 [3], it hasn't been unbearable for me. I followed a Techbits.io guide to using Backblaze B2 with Mastodon that worked flawlessly for me.

I also followed their guide for serving B2 with Cloudflare (yes, I know, Cloudflare is problematic and I'm looking to move away from it). The combination means that all of this data costs me about a dollar per month to host & serve.

If I need to move, I plan to take a look at Jortage, a "communal cloud" designed specifically for hosting deduplicated Fediverse media. There are also S3-compatible self hosting options, like Garage and MinIO.

Don't Sleep on the Oracle Cloud Free Tier

That dollar per month is the most expensive part of my operation, [4] because the actual server is free: an ARM-powered virtual private server, offered as part of Oracle Cloud's Free Tier. 24GB memory & 100GB hard disk space, all for no charge. Marc-André Moreau has a great writeup about it on Cohost.

One catch is that not everything out there has an ARM build. This hasn't been a problem for any of the services mentioned above, but the Sidekiq metrics exporter that I use to monitor my Sidekiq queues with Grafana does not have an ARM Docker package. I ended up checking out the Git repository and building the Docker image on the Oracle server, and it's run without any issues.

Monitoring With Grafana

This is probably a luxury feature for a small instance, but you can set up a Grafana dashboard to monitor your system in real time, as I have.

A screenshot of my Mastodon server's dashboard
A screenshot of my Mastodon server's dashboard

The IPng Networks series on Mastodon monitoring is a must-read to getting monitoring working, along with tips on hosting a Mastodon instance in general. I also make use of the Sidekiq Prometheus exporter that I mentioned earlier for the all-important queue metrics.

Take a Look at Mastodon Forks

Most instances you'll encounter are running on vanilla Mastodon, and it's what most people are familiar with. Some aren't running on Mastodon at all - those are outside the scope of this post - and others are running modified forks of the original Mastodon source.

Running a fork is not risk-free. With fewer users & less resources than the mainline project, you may be more exposed to security issues, or even risk being left behind if the project is abandoned. The trade-off is getting features that the Mastodon developers either haven't implemented, or refuse to implement.

From what I have seen, the two most popular forks are glitch-soc and Hometown. I'm using glitch, as do some medium-sized instances like eldritch.cafe. It includes nice bonus features like support for longer posts, Markdown support, [5] and posts that are only visible on the same instance.

Defederate Early, Often and Proactively

There are a number of instances out there full of awful people, sharing horrible content with everyone they can - content that may even be illegal in your country. You don't want that anywhere near you if you can help it.

The good news is that Mastodon makes it easy to block other instances from communicating with your server, and it also supports "limiting" instances in cases where a full block isn't warranted. The bad news is that it's on you to put in those blocks.

The Fediverse community has informally settled on using the #Fediblock hashtag to inform other admins & users of bad actors that should be considered for defederation. Naturally, this can be a magnet for drama and interference by those same bad actors, but at time of writing it's the best way to spread the word about these troublemakers.

If you're running a one-user instance, you've got another problem: #Fediblock will be nearly empty! Even with relays, you'll probably only see a fraction of the posts carrying the hashtag. Furthermore, you'll be starting fresh, and people don't really talk about instances that have long since been blocked by the community - leaving you potentially unaware of them.

When starting out, I copied the blocklist from my previous server, and imported all 400 or so. [6] I didn't copy them blindly: especially for the larger instances, I reviewed whether they should be blocked or not, and made my own determination.

Next, I used my RSS reader to subscribe to mastodon.social's #Fediblock hashtag - every hashtag can be subscribed to via RSS, without an account - to get all the #Fediblock updates that the world's largest Mastodon instance is aware of.

I actually recommend using an RSS reader for this regardless of where your account lives. I only check the feed every week or so, and an RSS reader will let you mark items as read or unread, helping you keep track of what you've processed. Further, a good RSS reader will let you search the feed, so you can look up important context when necessary.

Don't Forget About Liability

There's an important topic that I'm not really qualified to talk about, nor do I hear much about it from other admins: liability. I'm not a lawyer or even mildly interested in this stuff, but we shouldn't ignore the topic.

Everyone thinking of running an instance (in the US, particularly) should read Denise's guide to potential liability pitfalls for people running a Mastodon instance. It's not legal advice, but it is a very comprehensive look at all the various legal trouble you could get into by hosting content on the web.

Backing Up & Migrating Postgres

My instance was originally on my home media server. I first attempted to move it to Oracle a month or so ago, but it failed when the database didn't restore correctly.

I didn't have any luck with the instructions on the Mastodon documentation site, but I did eventually figure it out. I practiced these actions outside of the migration before doing them for real, to make sure it would work.

The below directions assume you use a Docker container for Postgres, always named postgres, and your database name is mastodon:

  1. Shut down Mastodon, so that database writes stop

    • Don't shut down Postgres! You need that
  2. Dump the Mastodon database to a gzip archive

    docker exec postgres pg_dump -U postgres --clean --no-password mastodon | gzip > postgres-backup-mastodon.sql.tar.gz

  3. Dump the global objects (roles and tablespaces) of the Postgres server

    docker exec postgres pg_dumpall -U postgres --clean --no-password --globals-only > postgres-backup-globals.sql

  4. Copy both files to the target server

  5. Restore the globals to the new Postgres server

    cat postgres-backup-mastodon.sql | docker exec -i postgres psql -U postgres mastodon

  6. Restore the Mastodon database to the new Postgres server

    zcat postgres-backup-mastodon.sql.tar.gz | docker exec -i postgres psql -U postgres mastodon

I think that covers just about everything I want to talk about regarding hosting a Mastodon instance! There are other posts out there I want to share, however, that I didn't find a spot for above:

Update - February 9, 2023

If you've read this far, then thank you! I hope your adventure in running Mastodon is as rewarding as mine has been so far.

Again, I'm just a hobbyist, and I'm neither a sysadmin nor a Mastodon developer. I may have some of these details wrong. If I do, feel free to reach out and let me know!


  1. If you're looking for suggestions, Aurynn Shaw is very good at spotting trouble in the Fediverse. Kris Nóva and Esther are worth following as well. ↩︎

  2. As a bonus: when a major event happens, and a surge of new users buries the servers, it's fun to watch their descent into madness. 🤣 ↩︎

  3. I looked for the post I read that had more details about this, but I don't think it exists anymore. ↩︎

  4. ...ok, the domain name is the most expensive part. I've had that for a while, though. ↩︎

  5. Be careful if you actually use that. Most of your followers are using regular Mastodon, which doesn't really support formatted HTML to that extent. Links work as expected, but everything else mainly renders as plain text. I'd avoid lists in particular. ↩︎

  6. There may be automated import tools, but I never found them. I simply created 400 INSERT statements and injected them directly to the domain_blocks table in Postgres, like an animal. The management page was out of sync at first (Redis cache?), but it worked without a hitch. ↩︎

From Tumblr girl to engineer

https://mashable.com/article/tumblr-girl-learn-code
Permalink

I can absolutely relate to this. I started this blog in Tumblr about 15 years ago, getting praise for the fancy themes I put together. That blog served as a sandbox to hone my CSS skills in, trying trickier stuff as I became more comfortable. You can trace a line directly from those days to today, where I'm writing Vue components and building home-grown UI frameworks. The time I spent fooling around with that little Tumblr blog is still paying dividends to this day!

Now figuring out that I'm a Tumblr girl, that was its own adventure.

State of CSS 2022

https://web.dev/state-of-css-2022/
Permalink

I am very much looking forward to some of these new features rolling out in the next year or two! It will be fun to learn how to use them.

Imagine just starting with CSS today, though! 25 years of development has brought loads and loads of new features and quirks. It's come a long way from when I was in grade 6, saving something I made in FrontPage onto a floppy disk.

...hey, I was 10. I didn't know any better!

The First Day of the Rest of My Life

I have something to announce, and it's not that I've been hacked! The name is different, but I'm the same person as before... sort of. Let me tell you what's happened!

I don't know what it's like for most people when they reach adulthood, but I for one felt confused & empty well past that point. I didn't like who I saw in the mirror; life had little meaning. At first I thought I was missing things in life, and I was. Life became much better with friends, a good income, a loving partner & family. I love them all and I don't take any of it for granted. By that point, however, it was very clear that I wasn't missing things in life. I was living the wrong life.

Through a lot of soul-searching and - eventually - therapy, I realized the basic problem: I'm a girl! My life was empty because it wasn't the life I was supposed to have, and the person in the mirror wasn't me.

Transitioning is not an easy thing to do. It's a lot of work, I won't have all the privelege that I'm used to, and people don't always react kindly. Nonetheless, I have to do it. I was living a lie, pretending to be someone I'm not. If you know me, you know that I hate lying - well I'm done lying to myself, and I'm done lying to you.

Since this realization, the changes have been drastic. I love the woman I see in the mirror, and I'm excited to make the most of my 30's! I didn't even know what excitement meant before. I've got my life back, and I'm not just going through the motions anymore. I know I'll be a better parent, partner to my wife, daughter, co-worker, and friend to all. I wouldn't trade this life for anything.

So, yeah. I'm Courtney! I love the new me, and I hope you like me too.

❤️

Spruce CSS Framework

https://sprucecss.com/
Permalink

This appears to be a really nice little CSS framework, but what I love about it is how similar it is to the frameworks I've written code for recently. For instance, even the partial one I put together for this blog has color & spacing methods in Sass that translate to CSS variables - and Spruce has the same concepts, just with different names.

One reason I can be wary of adopting third-party CSS frameworks is that they just work differently than I'm used to - I have to change how I think, and not necessarily for the better. That doesn't seem to be the case with Spruce, and I love that.

Sabotage: Code added to popular NPM package wiped files in Russia and Belarus

https://arstechnica.com/information-technology/2022/03/sabotage-code-added-to-popular-npm-package-wiped-files-in-russia-and-belarus/
Permalink

Third-party dependencies can be very cool, but they're also a great way to introduce unexpected vulnerabilities to your code. For whatever reason, NPM packages seem particularly susceptible to these problems.

Russia's aggression is catastrophic and unjustified, and the apparent intent of the code was to add a little more pressure to Russia & Belarus to withdraw from Ukraine, by disrupting systems in those countries using the library. Unfortunately, this was a rotten way to do it, because the consequences are unpredictable and often unintended, and could affect good people doing good work. Especially if this unverified allegation is true.

A UX designer walks into a Tesla Bar

https://jenson.org/tesla/
Permalink

Tesla isn't the only company sacrificing usability for aesthetic or strategic purposes - looking at you, Apple - but it's particularly awful when it gets in the way of the driver. When Apple does it, it's only annoying someone holding a phone. When Tesla does it, it's distracting someone in control of 4,000 pounds of steel moving at 75 miles per hour.

Even something as simple as a blog has to consider compromises like this. For instance, take the permanent links for link posts such as this. Some blogs omit them entirely, or make them small icons to the left or right of the title, but this blog has a button with "Permalink" on it for each link post. It doesn't look very cool, but it's easily understood.

You would think that auto companies would have a more usability-conscious attitude than a silly tech blog run as a hobby, but alas.

SPAs were a mistake

https://gomakethings.com/spas-were-a-mistake/
Permalink

I'm not sure I agree that SPA's are a mistake - specifically, I believe loading new pages isn't a nice experience regardless of how fast they are, and that sharing client-side data across multiple views is more of a pain if it's not all on the same page.

However, as with other newer technologies, we shouldn't treat it as an undisputed best practice to adopt SPA's. They have pros and cons, and it's worth thinking through how SPA's and MPA's would affect your product before committing to either approach. In particular, an MPA page's Javascript bundle could include only what that page requires, as opposed to the code for every page on the site, which could make a huge difference to performance.

Say Hello to selectmenu, a Fully Style-able select Element

https://css-tricks.com/the-selectmenu-element/
Permalink

While it's months out at the very least - more likely, years - I'm excited for this. Being able to manipulate the style & functionality of the native selectbox is long, long overdue. A big reason why we need third-party components in our design systems is because the browser's native UI elements haven't changed much since the 90's.

I don't expect this to fully replace the need for those components, particularly where the options need to be fetched from the server, but I could easily see those components being able to extend this element, rather than reinventing it as dropdown components tend to do today.