How I recovered from Mastodon/Fediverse (actually Akkoma) database corruption

I moved domains again, this time from sour.coffee to momandpop.network. I also run Fediverse servers.

Why so quickly? In the typical AuDHD fashion, I didn’t like sour.coffee too much.

I run a VPS host for a living, and have two VPSes running not-Tor exit relays: one running my legacy neelc.org domain (running Rocky Linux 9) and one running the newer domains (running Rocky Linux 10).

I decided to “migrate” to momandpop.network by setting up Akkoma on the “legacy” VPS via Docker. Initially, it went smoothly. Well, until I wanted to migrate to the non-legacy VPS.

What did I encounter?

db-1      | chmod: /var/run/postgresql: Operation not permitted
db-1      | 
db-1      | PostgreSQL Database directory appears to contain a database; Skipping initialization
db-1      | 
db-1      | 2026-04-05 22:03:26.747 UTC [1] LOG:  starting PostgreSQL 14.22 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
db-1      | 2026-04-05 22:03:26.747 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db-1      | 2026-04-05 22:03:26.747 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db-1      | 2026-04-05 22:03:26.748 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1      | 2026-04-05 22:03:26.750 UTC [14] LOG:  database system was shut down at 2026-04-05 20:33:08 UTC
db-1      | 2026-04-05 22:03:26.750 UTC [14] LOG:  invalid magic number 0000 in log segment 000000010000000000000001, offset 0
db-1      | 2026-04-05 22:03:26.750 UTC [14] LOG:  invalid primary checkpoint record
db-1      | 2026-04-05 22:03:26.750 UTC [14] PANIC:  could not locate a valid checkpoint record
db-1      | 2026-04-05 22:03:26.914 UTC [1] LOG:  startup process (PID 14) was terminated by signal 6: Aborted
db-1      | 2026-04-05 22:03:26.914 UTC [1] LOG:  aborting startup due to startup process failure
db-1      | 2026-04-05 22:03:26.921 UTC [1] LOG:  database system is shut down

In short, database corruption.

While typically you could run pg_resorewal. But when I tried that, it left Postgres in a non-operable state.

I’m just grateful that

• The signing keys weren’t lost
• If you have signing keys, your “followers” will re-sync
• Nothing of significance was posted to Fedi from my “new server”

The second point is how some Mastodon administrators migrated to GoToSocial. I in fact initially used GTS before migrating to Akkoma.

I had a feeling this would eventually happen, but at least it didn’t happen on an “established” server. I’m not a Postgres fan myself, I actually prefer plain-old MariaDB/MySQL.

And if you want to follow me on Fedi/Mastodon, I’m @noc@momandpop.network.