Do not ever push to production without testing things first. I went and moved us to the beta branch because this commit caught my eye and I wanted us to have the emoji picker fixed: https://github.com/LemmyNet/lemmy-ui/commit/ae4c37ed4450b194719859413983d6ec651e9609

The beta branch was on dockerhub so I thought that it was at least minimally tested. I was sorely mistaken. Lemmy (the backend) itself would load, but lemmy-ui (the actual website that renders everything) would keep crashing when attempting to load. I couldn’t roll back because the database was migrated to this beta version and it couldn’t migrate back to the old version when attempting to launch said old version.

I had no choice but to restore from backup. We’ve lost a whole day’s worth of posts (anything after 3AM CST.) I’m really really sorry… blobcatsadpleading

I was just so excited to be able to unveil this, I didn’t take my time with actually testing it.

  • Burger@burggit.moeOPM
    link
    fedilink
    arrow-up
    9
    ·
    1 year ago

    Also, to add: Thank fuck for Borg. It’s one of the few backup solutions that hasn’t corrupted itself just from doing incremental backups.