Site downtime 2024-10-11

[namida]

I won't have time to look into what happened here until about 12 hours from now, but I am aware of it. I rebooted the server and that seems to have sorted it for now.

[Simon]

For the record: The forum outage began before 16:00 UTC, i.e., at least 7 hours earlier than this post).

-- Simon

[namida]

I received a down notification email at 14:34 (but was asleep at the time) and back up at 19:23. These may be up to 5 minutes later than the actual times of going down / coming back up, as the service I use for this only checks every 5 minutes.

[namida]

Haven't been able to figure out much about what happened here. It does appear to be MySQL-related, so replacing that with MariaDB (or vice-versa, not sure which one is currently there) is one option if it continues to have issues.

[namida]

Same issue occurred about an hour ago. It seems the MySQL service is being terminated due to out-of-memory errors. I'll see if there's some settings tweaking I can do to avoid that in the future. All else failing, I might set up a script that checks periodically if it's running, and restarts it if not.

It seems messing with the memory allocations for it just causes it to refuse to start, so the auto-recover script might be the way to go here, at least for now...

[namida]

This issue occurred again this morning; I caught it within a few minutes this time.

I have now set up an automatic task that runs every 5 minutes and checks if the MySQL service is running, and if not, it starts it up again. Until I can figure out how to actually fix this issue, this will at least mean the site should automatically recover within 5 minutes without any need for me or another admin to intervene, when this happens. (Some further short downtimes that have occurred this morning are due to testing and debugging this.)