Server Move - Discussion and Status

[namida]

It'd be the PM recipients who'd need to run that, not me. :p

[geoo]

Unless you wrote a script that changes every user's password using the AdminCP, logs into their account and then gets all the PMs (and leaves everyone with a password that they don't know). http://www.lemmingsforums.com/Smileys/lemmings/tongue.gif" alt=":P" title="Tongue" class="smiley" />

[namida]

Not even going to consider doing that, sorry. However, if PMs can be viewed from the admin panel, *that* would be an option maybe... don't think they can, though.

Anyway, I'm going to attempt to import all existing user registrations (EDIT: not including those who had zero posts at this time) from the old backup. I'll let you know soon whether or not it was successful.

[namida]

All user accounts have been imported. Note that after importing a second copy of my old account, I wasn't able to log in (it said the wrong password) so you may need to reset your passwords via the lost password feature. If anyone doesn't have access to their old email address anymore, let me (or any other admins on the new board, once some are added - not sure if moderators are able to help with this) know and I can update your email address, or if all else fails, set you a temporary password which you can use to log in and change it to one of your choosing.

Check http://lemforums.neolemmix.com/old/ImportUsers.txt" class="bbc_link" target="_blank">this list to see if your username has been imported. If it hasn't, it's because you either registered after the January database backup, or you had zero posts at this time; if this is the case, you'll have to register a new account. If you had already registered an account on the new one, your old account would've been skipped during the import if the username and/or the email address were the same. This only affects a small handful of users, but if you don't see your name on the list it may also be because you already registered an account on the new forum. I didn't compare IP addresses or other profile information to check for duplicates, so if you do have a duplicate account, please let me know so I can merge them or delete one.

Most profile information was also imported along with the accounts themself. However, some things (such as mod/admin status, post count) weren't.

I'll look into importing the old private messages soon. http://www.lemmingsforums.com/Smileys/lemmings/smiley.gif" alt=":)" title="Smiley" class="smiley" /> This way, you'll only need to back up (if we can't find a way around this) ones received since the backup.

EDIT: Working on it now! http://www.lemmingsforums.com/Smileys/lemmings/smiley.gif" alt=":)" title="Smiley" class="smiley" />

[namida]

All avaialable PMs from the January backup have been imported (provided the receiving account was imported too (or already registered) - doesn't matter whether or not the sending account was).

Please do check that none are missing just in case. And there likely won't be any way to transfer newer ones, so make sure you back those up somehow.

EDIT: Does anyone think the IsaMedia PMs should be restored too? I could probably restore these too, although more will probably be missing due to username/email changes (as I'm using these to match accounts from the backups to those on the new forum - only one has to match, not both).

At this stage, pretty much all that's left that's possible (aside from the IsaMedia PMs if anyone thinks it's worth bothering - it shouldn't be too much effort, as long as they're in the backup I have - haven't checked if they are) is the posts (which unlike LW and IsaMedia, will include attachments http://www.lemmingsforums.com/Smileys/lemmings/laugh.gif" alt=":D" title="Laugh" class="smiley" /> ) from here. I don't want to do them while people are still posting here though, for the reasons I've mentioned before. So... what to do from here?


(By the way, I probably won't be doing much on Christmas Day itself. But the days before and after should be perfectly fine.)

[Simon]

[08:22] <SimonN> namida42: incoming messages up to early 2014 are all there, thanks. Outgoing messages are empty still
[08:22] * PhilM has quit (Read error: Connection reset by peer)
[08:24] <namida42> i didnt attempt to do anything on the sent side, didnt really cross my mind
[08:24] <namida42> i can probably patch it up though
[08:25] <SimonN> hnn
[08:27] <namida42> remind me a bit later on, kinda had enough of it for now xD
[08:27] <SimonN> okay, will make a post to the forum

-- Simon

[namida]

[08:22] <SimonN> namida42: incoming messages up to early 2014 are all there, thanks. Outgoing messages are empty still
[08:22] * PhilM has quit (Read error: Connection reset by peer)
[08:24] <namida42> i didnt attempt to do anything on the sent side, didnt really cross my mind
[08:24] <namida42> i can probably patch it up though
[08:25] <SimonN> hnn http://www.lemmingsforums.com/Smileys/lemmings/smiley.gif" alt=":)" title="Smiley" class="smiley" />
[08:27] <namida42> remind me a bit later on, kinda had enough of it for now xD
[08:27] <SimonN> okay, will make a post to the forum

-- Simon

Simon already knows this due to being a guinea pig for checking if it works http://www.lemmingsforums.com/Smileys/lemmings/tongue.gif" alt=":P" title="Tongue" class="smiley" /> but for everyone who isn't currently on #lix; this is now done.

[namida]

PMs from the IsaMedia board *should* have been imported. I say "should" because I myself didn't have any from this time so can't verify for sure that it worked correctly; if anyone who *did* have some could let me know if they worked, that'd be great.

Note that they'll only have been imported if:
1. You have an account on the new forum. In other words, either a) you registered manually, or b) you had an account on here, with at least 1 post, at the time of the January backup.
2. Your username and/or email address is the same on the new forum (it will have been imported from what it is on *this* forum (or was at the time of the January backup, to be more precise) if your acccount was imported as described above) as it was on the IsaMedia forum.
3. In the case of sent messages, most likely they'll only have been imported if the recipient also met the above criteria.

Additionally, an order fix has been applied to the PMs similarly to how it was with the regular posts.

[namida]

Latest update; we now have the Lemmings-y emoticons on the new board. http://www.lemmingsforums.com/Smileys/lemmings/smiley.gif" alt=":)" title="Smiley" class="smiley" /> That's all from me for today, will continue tomorrow.

At this point, I'm suggesting we set a move-over date, and once this has passed we disable posting (but not viewing - I need to be able to grab a content dump as up-to-date as possible of course xD) on this board. Between the closing date here and the posts being transferred (it may take a few hours - ideally I'll have everything ready and working by then so the only thing left to do is actually run the import), you can always reply to existing topics here on the new board by creating a new topic there, and I can merge them once the posts from here are transferred over. My suggestion would be new year's day - that way most of the users will be too busy to need to post much here anyway - but of course this is something we all need to decide on rather than one of us saying "this is the date, and that's final", so please give your input. http://www.lemmingsforums.com/Smileys/lemmings/smiley.gif" alt=":)" title="Smiley" class="smiley" />

Once the posts are transferred, we can start on linking them to actual accounts (instead of as guest posts).

Also - when this happens, we'll need to update the News thing at the top of the board to reflect this. I believe geoo (or Mr K) should be able to do this from the admin panel, unless that's broken at some point.

[Prob Lem]

If you had already registered an account on the new one, your old account would've been skipped during the import if the username and/or the email address were the same. This only affects a small handful of users, but if you don't see your name on the list it may also be because you already registered an account on the new forum.

I'm one of the ones affected by this, as I'd already registered on the new forum quite early on in case this one fell over. I assume this won't present a problem for linking my posts from here to my account there, once the posts are moved over?

Latest update; we now have the Lemmings-y emoticons on the new board. http://www.lemmingsforums.com/Smileys/lemmings/smiley.gif" alt=":)" title="Smiley" class="smiley" /> That's all from me for today, will continue tomorrow.

Yay! I love the emoticons from here, and I was missing them over on the new forum. http://www.lemmingsforums.com/Smileys/lemmings/thumbsup.gif" alt=":thumbsup:" title="Thumbs Up" class="smiley" />

My suggestion would be new year's day - that way most of the users will be too busy to need to post much here anyway - but of course this is something we all need to decide on rather than one of us saying "this is the date, and that's final", so please give your input. http://www.lemmingsforums.com/Smileys/lemmings/smiley.gif" alt=":)" title="Smiley" class="smiley" />

New Year's Day is fine by me - new year, new start, right?

I guess we'll have to make our forumy New Year wishes either a little early, or a little late, eh? http://www.lemmingsforums.com/Smileys/lemmings/laugh.gif" alt=":D" title="Laugh" class="smiley" />

[namida]

If you had already registered an account on the new one, your old account would've been skipped during the import if the username and/or the email address were the same. This only affects a small handful of users, but if you don't see your name on the list it may also be because you already registered an account on the new forum.

I'm one of the ones affected by this, as I'd already registered on the new forum quite early on in case this one fell over. I assume this won't present a problem for linking my posts from here to my account there, once the posts are moved over?

Nope, it wont' present any problem at all. The only disadvantage you may have is that you'll have to manually re-add any profile info you want to carry over (such as signatures, contact info in your profile, etc).


So, my task over the next few days is to get together an app that can parse this boards. I'm already mostly done with the planning stage, but still need to do the actual coding. Of course, I won't actually add the posts until the deadline, but needless to say I'll be doing a few test runs (basically, generating the data for importing, just without actually doing the final import to bring it live) beforehand - one reason for choosing new years' is this should give me plenty of time to do it.

[Prob Lem]

Nope, it wont' present any problem at all. The only disadvantage you may have is that you'll have to manually re-add any profile info you want to carry over (such as signatures, contact info in your profile, etc).

That's awesome, thanks. http://www.lemmingsforums.com/Smileys/lemmings/thumbsup.gif" alt=":thumbsup:" title="Thumbs Up" class="smiley" />

The profile info thing isn't a problem for me at all - I was going to rejig my signature a little bit once the move was done, anyway! http://www.lemmingsforums.com/Smileys/lemmings/laugh.gif" alt=":D" title="Laugh" class="smiley" />

EDIT: Just had a browse on over to the new forum, and noticed that I'm unable to post or reply (I just wanted to try out the emotes, haha). I assume posting is disabled over there for now, even though there's a news note about merging duplicated posts?

[namida]

It was disabled, but it wasn't meant to be. http://www.lemmingsforums.com/Smileys/lemmings/tongue.gif" alt=":P" title="Tongue" class="smiley" /> Fixed now.

The reason for this is that before importing any old forum data (or even applying minor fixes usually) I back up the database, so that if it goes wrong (which it often does on a first attempt), I can simply jump back to the previous state rather than having to manually undo the damage. So, to avoid loss of posts, I also disable posting before the backup is taken. In this case, I must've forgotten to re-enable it.

[ccexplore]

It'd be the PM recipients who'd need to run that, not me. :p

I did some testing and have confirmed that the filters feature in WinHTTrack is sufficient to control its navigation in a logged in session, making the tool viable for grabbing PMs.  Which is a very good thing as it is more than just logout--there are other kinds of links like "delete" that could potentially do other unwanted stuff.

I'll admit that the steps are not easy for computer novices unfortunately, partly due to having to deal with cookies in order for WinHTTrack to make use of your active login session.  If you don't have too many PMs that need transfer (keeping in mind that everything before Jan 2014 are already in the database backup), you may find it easier to just use your web browser to save the few pages of PMs that need transferring, instead of getting the tool to do it for you.

For reference, here's a rough outline of how I've tested (only on a test account so far, since I want to be absolutely sure it doesn't accidently navigate a link that would perform a destructive action, one of the potentialdangers of crawling through a logged-in forum session):

1. create a directory (folder) to serve as working directory for WinHTTrack for crawling your PMs.  Then create another directory under it with a simple name like "myPMs".
2. on web browser, logged in to desired user account with "stay logged in" checked
3. grab/export the cookie file associated with lemmingsforum, which contains the data necessary to allow WinHTTrack to use your active logged-in session as it crawls (obviously needed in order to browse your PMs).  You may need a tool besides your own browser to properly get the cookie file, as WinHTTrack requires the cookie file to be in a specific format, which may not be the one used by your browser for storing the cookies.  You may also need to slightly modify the contents of the exported cookie file to make WinHTTrack happy.  For example, the tool I used seems to leave an extra "." before "http://www.lemmingsforum.com/" class="bbc_link" target="_blank">www.lemmingsforum.com" inside the cookie file, and I had to remove that extra period to get things to work.
4. Put the exported cookie file into your directories from step 1 (one copy in the working directory, and another copy in the "myPMs" sub-directory or whatever you named that), make sure the filename is exactly cookies.txt
5. now start WinHTTrack and start a new project.  For "Project name", set it to exactly the name of your sub-directory (eg. myPMs) from step 1.  Set the "Base path" to the working directory from step 1.  Hit Next.
6. Set "Action" to either "Download web sites" (if first time) or "Update existing download".  For the "URL", use "http://www.lemmingsforums.com/index.php?action=pm" class="bbc_link" target="_blank">http://www.lemmingsforums.com/index.php?action=pm" (no quotation marks of course).  Then click "Set options".  An "WinHTTrack" options window pops up.
7. Here's the important part:  go to "Scan Rules" tab, and add the following to the end of the line you see in the text box:

-*?topic=* -*?board=* -*?action=* +*?action=pm* -*sa=* -*togglebar*

A quick explanation of what this does:  the "topic" exclusion blocks crawling of forum posts, "board" exclusion blocks crawling of forum boards, and "action" exclusion blocks lots of actions you don't want the crawler to crawl and trigger unintentionally, like the link in the "logout" button for example (or worse, the "delete" buttons for your PMs!).  The "action=pm" inclusion then allows the crawler to crawl just the URLs that are specific to your PMs (both the inbox and sent).  The "sa" exclusion blocks site actions, which are associated with other unwanted actions like "prune messages" or changing your settings.  The "togglebar" exclusion blocks the "<<" thing you see near the top-right of the PM pages, which apparently changes your user preference of the way the PM page is laid out.

8. Now go to "Limits" tab and set "Maximum mirroring depth" to a reasonable number.  My suggestion is to look at how many pages of PMs you have in your inbox and in your sent box, take the maximum of the two, and then add like 3 to it and use that number.  I actually don't know if you really need to set a max here, but it does ensure that the crawler won't get into a runaway situation where it is not able to accurate detect things it already crawled and therefore never stops crawling.  Higher number simply means you are more likely to get pages saved by the crawler that are effectively duplicates.

9. Now click OK to commit the settings.  Then "Next" through the rest and let the tool do its thing.  It may take some time before it's done so be patient.

10. Once it's done, click the " view mirrored website" button and see what you get.  If all goes well, the page that you first see would be a mirrored copy of your PM inbox page.  If instead you get the page that asks for username and password, then the cookies you set up from steps 1-4 are apparently not getting used.  Manually browse through a few links to verify that your entire inbox and sent box get mirrored (by seeing that the URL displayed in the browser is referring to a local file on your computer, rather than the usual HTTP web address).  (Note: you may notice that the user avatar pictures are missing in the mirrored pages.  This is a side effect of the "action" exclusion rule, since the URLs to the avatar pictures do contain that "action" keyword in their URLs.  While it can fixed fairly easily by refining the scan rules, it really doesn't matter since the point is just to capture the contents of your PMs.)

Anyway, with that being a success, I'll take a look at capturing the PMs for my own account and for the QuizMaster account, and see if namida can do anything with the resulting files.

[namida]

Okay, if anyone can get their PMs via that method, I should be able to import them to the new forum.

Keep in mind that, while of course I'm not going to intentionally read through them, if any part is causing errors at any stage of the import process, I will need to take a quick glance at the part that's causing the error


By the way, ccexplore: I've been meaning to ask if you'd be interested in an admin position, with server access, given your technical skill level? I wouldn't be expecting you to do much, it's just so that in the event server access (or just a regular board admin) is needed, we have multiple people rather than ending up in a situation like we have here where two admins can't be contacted and the third doesn't have server access.