Site downtime (2024-08-25)

[namida]

Lemmings Forums (as well as the NeoLemmix website) were down from about 01:30 UTC today (EDIT: looking at Discord, it may have been slightly earlier than this - first mention of it was at 01:20). The NeoLemmix website was restored at 03:10, and Lemmings Forums at 04:00, making this probably the longest downtime during the time I've hosted the site.

The downtime was caused by system updates failing, which in turn caused Apache (the web server) to refuse to load after a reboot. This meant that none of the websites hosted on the server were accessible. Fortunately the back-end remained accessible, so I was able to perform full backups of the sites in their current state - therefore, outside of cases like "posts that weren't yet posted before the site went down", no data should have been lost.

This issue could have taken a significant amount of time to investigate, diagnose and repair, so instead - I decided to create a new server and migrate the sites onto it. This is also why the sites didn't come back online at the same time - the NeoLemmix website has much less data (the backup was half the size), as well as being less reliant on specific versions of PHP. (This is another matter - the site needs to be updated to SMF 2.0.19 at least, so that it's PHP 8.0 compatible, and from there to SMF 2.1.X if possible. I'll worry about this once I've got the current version fully working again, and likely more a "tomorrow / next week" task.)

At the time of this post, core functionality is restored but some specific things - namely, emails sent from the site (registration confirmations and PM notifications) are likely not functional at the moment UPDATE: confirmed they're working, and I'm not sure whether other admins will still be able to access the back-end stuff - please get in touch if not. (Note that the IP address has changed; all those with access should know how to find it from the domain name.)

As I work on these matters, it's possible that the site may have brief outages, if any reboots are necessary. No further downtime is expected.

[Silken Healer]

Are you sure it was just a malfunction and not someone hacking Lemmings Forums? When I tried to acsess it I got a Firefox warning. Then, it had an image advertising a different website called fart.tax. Maybe whoever owns fart.tax hacked Lemmings Forums to get more users to their website.

[namida]

Are you sure it was just a malfunction and not someone hacking Lemmings Forums? When I tried to acsess it I got a Firefox warning. Then, it had an image advertising a different website called fart.tax. Maybe whoever owns fart.tax hacked Lemmings Forums to get more users to their website.

I responded in the other topic, but copying it here as well:

It wasn't hacked. That's a joke URL I bought a while back and have a placeholder on (which is hosted on the same server). Before LemmingsForums.net was properly imported, it was redirecting to another site on the new server I set up - and it happened to pick that one. (It was basically 50/50 whether it would pick that, or a crypto-themed Flappy Bird clone.)

[Silken Healer]

(Responding to your post to my topic as you couldn't merge them and locked them)

Ah, phew. Because there was a security warning I was really worried that it was hacked by a fart-based group of hackers or something

[namida]

Okay, I can confirm that site emails are working correctly - SMF has its own built-in SMTP handling (and thus, settings) so the settings carried over with the site backup.

[namida]

No further downtime is expected now. Aside from the possibility of issues with other admins accessing back-end stuff (which I'll need to wait to hear from them on), both sites should now be business as usual.

Except for the slight improvement of that they should also now be accessible over IPv6 too.

Need to look at the forums themself at some point, updating them to newer SMF. Aside from that, the only remaining check is making sure the automated monthly backups still run correctly - I have run the script manually and can confirm it runs fine, just need to confirm it automatically triggers too.

EDIT: Upgrade to SMF 2.0.19 is done. Despite SMF's website claiming SMF 2.0.19 only works with PHP versions up to 8.0, it seems to be working just fine on PHP 8.3. SMF 2.1.X will be a bigger project though, for another time.

[namida]

In this case, the downtime started after a manual reboot as part of trying to troubleshoot the update issue. However, it crossed my mind that I should really have some kind of automatic monitoring that notifies me if the site goes down - so I set that up too. I've now got it so that every 5 minutes, both lemmingsforums.net and neolemmix.com are checked, and if either of them are down, I receive both an email and a notification on my phone.

Of course, this doesn't mean I'll always be able to see, let alone deal with, the notification right away, but it will mean I get alerted quicker than just "next time I check the Discord or try to view the site".

[namida]

I can now confirm the automatic backups are working fine with the new server.