Server Move - Discussion and Status

[geoo]

Why are we thinking about a server move?
We haven't managed to contact the server admin, Adam, in a very long time. He seems to be paying the server bill, but there's uncertainty for how long that would last, and the forum could go down at any point in time, and data might be lost. Without server access we also can't update the forum software, which right now is outdated, buggy and probably prone to exploits for spambots like the recent invasion. On a new server with someone active being server admin we have complete control over everything.

This is what has happened so far:

• With Mr. K's help, we now have access to the admin control panel, but nobody of us has server access
• Temporarily, to fight the spam bots, user registration was turned off for a day, which I changed to new users requiring approval by an administrator. It seems like there haven't been new members trying to register within the past two days, so I'm setting in back to normal now and see what happens.
• We can't get hold of an up-to-date version of the database (that'd contain all user information, posts, everything we'd need to just move the forum over without issues)
• Adam (and maybe Liam, whoever he is) is the only one able to get us the full database and the domain www.lemmingsforums.com, we have tried to contact him, but without success so far

This is what we have access to right now:

• Threads and posts from the Lemmingswelt forum (2004-2007 or so)
• Complete database from the isamedia forum (2007-2009 or so)
• Database of this forum from January 09th, 2014. All newer data is missing
• namida can get the complete set of theads/posts/attachments of this forum at any point in time (I believe)

My suggestion is to allow some more time to contact Adam, discuss and not rush things, while still keeping posting in this forum and leaving namida's forum as the emergency option should the forum suddenly go down. (I don't think it's very likely that the forum will go down within the next month.) We should prepare everything for a swift move, and once preparations are finished set a date (and announce it a week in advance) for when the move shall take place. That day namida would grab all the data off the forum and import it over on the new forum, and then this forum would be closed (or at least become read-only to ensure new content will be posted on the new server).

The reason I suggest to keep posting here for now is that moving to the new forum would encourage reposting topics which would have to be merged later, which becomes messier the more time passes and thus puts us under time pressure.

If we can somehow get hold of Adam, this would make things a lot easier as we could just import the whole database and get a full copy of the forum, and in addition the domain www.lemmingsforums.com. So I encourage everyone to help us in our effort to contact Adam!
Should this not work, I'd suggest purging the few posts from the new forum and importing the members from the January 2014 database (this means we'd also have most PMs, and almost all posts would automatically already be linked to the users), while taking all the posts that namida will grab from here over. Furthermore, we could add the posts from the isamedia and lemmingswelt forums and try to link them to the user accounts (something that I feel very strongly about). If you forgot your password from January 2014, or if you registered later than January 2014 you could identify yourself and get access to your account/get it linked to the posts that you made. Maybe we could also get the PMs from the isamedia forum working, I personally find having an archive of my PMs important, but I don't know what other people think about that.

The plan I outlined is just my opinion.

Feel free to discuss!

[ccexplore]

Sounds like a good plan.  The server move is definitely overdue, but there's no need to rush things as long as we're beginning the preparations now.  Indeed I'd even suggest possibly a dry run of the move first to ensure everything works as planned with no surprises.

I'm actually surprised that the most recent database backup we had access to is from within this year, I would've thought it was a bit older.  Remind me how were we able to get the Jan 2014 database without involving Adam or otherwise actually getting direct access to the database?

Ideally we should carry over as many things as we are able to.  The priority would be posts after 2014, attachments, PMs, and finally the old stuff from isamedia/lemmingswelt.

I'm sure it's all implied but, when the move completes and the old forum closed or read-only, make sure to update home page of old forum with links to the new forum, if we aren't able to get access to the domain to simply redirect the current URL to the new forum.  Actually, I suppose that info could be put up even as soon as the move starts.

If we aren't able to migrate everything over, the old forum should be made read-only but not closed (at least for a short time), so access to the stuff that wasn't migrated is still possible.  In this scenario, the stuff that was successfully migrated could potentially be deleted by admin, so it stops showing up in search engine results.  If everything got migrated then obviously closing the forum altogether would be the cleanest option, as long as it's still possible to display something to direct the user to the new forum (or otherwise be able to redirect the current URL).

=============

While not the highest priority:  if we go down the route where we are porting recent threads and posts outside of a full database import, see if you can, in the process of porting, fix up URL bbcode tags and similar in posts that are linking to content in the existing forum, like http://www.lemmingsforums.com/index.php?topic=170.msg11755#msg11755" class="bbc_link" target="_blank">this link here as an example.  I know I do that from time to time.  Failing that, being able to compile a list of all such posts could also work and then you could leave it up to individual users to edit and fix the posts themselves after the migration, as they see fit.

[Prob Lem]

Remind me how were we able to get the Jan 2014 database without involving Adam or otherwise actually getting direct access to the database?

Something at the back of my mind is saying that that's when the previous server move (the one that broke notifications of PMs/subscriptions, and lost some attachments) was, so I'm guessing it might be something to do with that. http://www.lemmingsforums.com/Smileys/lemmings/winktounge.gif" alt=";P" title="Wink-Tongue" class="smiley" />

[namida]

Alright then. Since people seem to be happy with this option, I'll also discourage posting (but encourage registration and knowing the link just in case) on the new forum.

If anyone does manage to get contact with Adam, what we need is just the database plus the contents of the "attachments" folder on the server. (Yes, I do at this point have a backup of all current attachments, but if we can get the folder as-is that'll be easier to import, whereas the form I have them in would require parsing the pages to get the correct filenames and other metadata for them.)

Meanwhile, I'm going to experiment with the older forum backups to see if I can get them going well on the new server.


As far as contacting Adam or Liam goes, I dont' have any ideas for Liam so we'll ignore that for now. As for Adam, I notice his youtube channel has uploads as recent as one week ago, so I'd say it's still active; I left a message on his channel (I noticed Akseli had already done this too). I also identified that the phone number in the domain WHOIS lookup appeared to be a cellphone so I've sent an SMS message to it.

[mobius]

I'm pretty sure someone said he's on facebook-- might be a good idea to try contacting him there. I do not use face book regularly so I don't know how to do that.

Btw, do I read that you are able/willing to upload posts from previous forums like Lemmingswelt onto this new forum? If so, why wasn't this done before? [geoo or you(s) weren't in charge is the answer I'm guessing]

[namida]

I'm pretty sure someone said he's on facebook-- might be a good idea to try contacting him there. I do not use face book regularly so I don't know how to do that.

Btw, do I read that you are able/willing to upload posts from previous forums like Lemmingswelt onto this new forum? If so, why wasn't this done before? [geoo or you(s) weren't in charge is the answer I'm guessing]

I have Facebook, so I'll have a go at contacting him on there.
EDIT: Unless someone has a more up-to-date email address for him, no luck (maybe geoo can find one from the user database here?). However, I have managed to locate Liam's Facebook - no idea how useful that'll be; he hasn't posted anything publically since mid-late 2013, but it's possible more recent posts just weren't set to public; I won't be able to see that unless/until he accepts a friend request.

As for the old forums, one issue is a difference in format. But yes, the main issue is that doing so will require server access, which on this forum, only Adam (and maybe Liam) have, and they were either unable or unwilling to do so (or maybe just never thought of it). But yes, geoo has a backup of the old forums, which he's passed on to me, and I should be able to do something to get them up on the new board - I'm going to start looking into it very soon.

In case this one disappears without warning, I'm going to update my HTML backup of it on a daily basis where possible. Worst case scenario; we can at least get the posts and attachments back, which I believe is the most important thing.

[namida]

No response from either Adam or Liam yet. http://www.lemmingsforums.com/Smileys/lemmings/sad.gif" alt=":(" title="Sad" class="smiley" />

Anyway, in terms of importing the old posts, I've done a bit of looking into exactly how to do it, and I should be able to write SQL files (or more accurately, write an app to generate SQL files) to batch-add old posts. Shortly I'm going to try just adding a small hand-written snippet to make sure it works* (this'll be added to a mod/admin-only board on the new forum so most of you won't see it); once that works, I'll try to parse the LemmingsWelt database, which is ironically the one that's in a more convenient format for me to write the app for. If all goes well, I'll add IsaMedia next, and once both of those are in, I can look at doing something with *this* board, since it doesn't look like we're going to be getting a copy of the database...

One issue geoo raised (and this may be very important to some people) is private messages. Unless we get the database, there's simply no way I can import the private messages from this site, for obvious reasons. So, if we don't get the database then when we do move, make sure you save a backup copy of any important private messages somewhere.

EDIT: * This was 100% successful on the first attempt. There is one minor issue - even if the posts (going by timestamp) is older, the newly-added post is pushed to the top of the topics list. Therefore, until further notice, I've disabled posting on the new board. I'll re-enable it when either:
a) I find a way around this
b) These forums go down for more than 24 hours, unless it's an intentional action as part of the move
c) All posts have been imported

And just to confirm 100%: Yes, we will have the LemmingsWelt and IsaMedia posts imported too, not just the ones from *this* forum. http://www.lemmingsforums.com/Smileys/lemmings/laugh.gif" alt=":D" title="Laugh" class="smiley" /> However, it looks like we'll only have the attachments from *this* forum; the other two will be posts only.

[Mr. K]

I'm actually surprised that the most recent database backup we had access to is from within this year, I would've thought it was a bit older.  Remind me how were we able to get the Jan 2014 database without involving Adam or otherwise actually getting direct access to the database?

Admin panel allows a database export.  I have been taking a backup whenever I log in, but it seems this broke around the Jan 9th backup.  Every export I have after that is broken-- it just cuts off halfway through an early admin panel table and everything of use to us is completely missing.  I'm thinking there may have been a host-side change that is restricting how big a file it can generate so it gets truncated before we receive it.

Also, I did a search on my backups and it doesn't look like I have a backup of any of the old attachments.  I'm trying a few different queries to see if I might have named it weirdly.  It's hard to find anything at all by hand when you have about 10TB of files. :/

[Luis]

What about all the pictures that are in the posts? Will those be carried over too? I had posted a lot of pictures in this forum and it would be a shame for all of that to be gone and have to post them again. http://www.lemmingsforums.com/Smileys/lemmings/sad.gif" alt=":(" title="Sad" class="smiley" />

[Prob Lem]

What about all the pictures that are in the posts? Will those be carried over too? I had posted a lot of pictures in this forum and it would be a shame for all of that to be gone and have to post them again. http://www.lemmingsforums.com/Smileys/lemmings/sad.gif" alt=":(" title="Sad" class="smiley" />

How did you post them? As attachments here, or using an outside image-hosting service?

[Luis]

Outside image hosting like photobucket.

[Prob Lem]

No worries there - that won't be affected at all, as the BBCode pointing to the images remains the same between most different forum software and versions thereof. http://www.lemmingsforums.com/Smileys/lemmings/thumbsup.gif" alt=":thumbsup:" title="Thumbs Up" class="smiley" />

[namida]

Any attachment that is currently still attached to the post on *this* forum will be fine, too. It's only the older boards (and the disappeared ones from this board) that I can't recover. This still applies even if we cant get a copy of the database.

EDIT: I mentioned yesterday that I successfully added a post directly via the database, using hand-written SQL queries. I've now successfully done the same with automatically-generated ones based on simply supplying a board ID, a subject, a message, a username and a post time to a self-written piece of code. This means that now, all that needs to be done is writing the side that parses the existing forum files in some format and supplies the data to this code snippet. LemmingsWelt will be first, both due to it being the oldest and due to it being in an easier format to work with.

[namida]

Posts from LemmingsWelt have been successfully loaded onto the new forum!

EDIT: After posting this, I noticed a small error; namely that the timestamps on the messages are wrong. Time to look into that...

Next up; IsaMedia! Due to the format this one is in, which isn't particularly well-suited to anything I've made so far, the easiest way might actually be to first write an app to convert it to a format similar to that which LemmingsWelt is in. However, I'm taking a short break for now. http://www.lemmingsforums.com/Smileys/lemmings/tongue.gif" alt=":P" title="Tongue" class="smiley" /> (As in a few hours or so, not a few days. http://www.lemmingsforums.com/Smileys/lemmings/tongue.gif" alt=":P" title="Tongue" class="smiley" /> )


EDIT: Got the timestamps fixed. I'm now trying to come up with a piece of SQL code that'll fix the message ordering with minimal side effects. That way, I can allow posting on the new forum without having to worry about disappearing messages (not as in disappearing completely; more accurate would be to say "dropping to the bottom of the list even if the added messages are older").

[Simon]

Great work, this is much more saved content than expected.

Do we want the ancient posts from Lemmingswelt and Isamedia in the same postable boards as the 2009-2014 posts? The alternative is to make read-only boards for the ancient posts at the bottom of the list. I.e., do we want to bloat the boards or the board list? :->

-- Simon