Welcome to the new Lemmings Forums... yet again.

Started by namida, December 15, 2014, 09:48:18 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

namida

My projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)
Non-Lemmings: Commander Keen: Galaxy Reimagined (a Commander Keen fangame)

ccexplore

I realize this is probably a problem unique to me :XD:, but I'm not a fan of seeing my own avatar in the header section of the forums (ie. near where the forum logo is located).  Any chance of not doing that (just like in the old forums) or failing that, have a per-user option to turn off displaying it there?

namida

Working on the PMs at the moment; once I'm done with that I'll look into it.
My projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)
Non-Lemmings: Commander Keen: Galaxy Reimagined (a Commander Keen fangame)

ccexplore

I'm reviewing some topics on the challenge board that were imported from the old lemmingswelt forum, and found that some of the posts' contents seem to be partially missing or otherwise not properly/fully imported.  For example, compare the following:

http://www.lemmingsforums.net/index.php?topic=263 (what was imported)

http://www.camanis.net/lemmings/lemmingswelt/forums/?topic=1117597280 (archive of what they should be)

Obviously the first post of the topic is affected but it's probably not the only one.  I found another one on page 2, and there are probably more but I haven't looked further yet.

namida

Ouch. o_O At this point, I'm not sure how practical it would be to restore them automatically (that's assuming they're not simply broken in the database backup I have); it might be possible by comparing exact timestamps, usernames and subject times to identify posts. It might come down to that it's simpler to do it manually, though that'd be quite a time-consuming task, especially given that there's no real way to automatically identify affected posts (unlike with, say, the missing attachments from the LF.com import, where the app detected which posts had missing ones and added a fixed note to each one, which could then be searched for to quickly locate affected posts - this is how Prob Lem was able to restore them all so quickly).

(the LemmingsWelt backups don't have any post ID numbers IIRC, and even in cases that do have them (eg. the IsaMedia backup), they won't be the same between the backup and this forum's database)

This is something I'll have to look into eventually but it probably won't be today.
My projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)
Non-Lemmings: Commander Keen: Galaxy Reimagined (a Commander Keen fangame)

namida

Quote from: ccexplore on January 04, 2015, 04:56:50 PM
I realize this is probably a problem unique to me :XD:, but I'm not a fan of seeing my own avatar in the header section of the forums (ie. near where the forum logo is located).  Any chance of not doing that (just like in the old forums) or failing that, have a per-user option to turn off displaying it there?

I've fixed this now. :)
My projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)
Non-Lemmings: Commander Keen: Galaxy Reimagined (a Commander Keen fangame)


ccexplore

Quote from: ccexplore on January 04, 2015, 05:12:30 PM
I'm reviewing some topics on the challenge board that were imported from the old lemmingswelt forum, and found that some of the posts' contents seem to be partially missing or otherwise not properly/fully imported.  For example, compare the following:

http://www.lemmingsforums.net/index.php?topic=263 (what was imported)

http://www.camanis.net/lemmings/lemmingswelt/forums/?topic=1117597280 (archive of what they should be)

Obviously the first post of the topic is affected but it's probably not the only one.  I found another one on page 2, and there are probably more but I haven't looked further yet.

I took a closer look and I think the problems are happening exactly in posts that made use of characters outside the ASCII range, with the affected posts getting cut off exactly at the first occurrence of such a character.  For example, in the thread linked to above, the first affected post started with "I've", where the apostrophe is not the normal one you get from the keyboard, but the fancier one that I think is actually classified as "right single quotation mark".  And as one can see, after the import we lost everything past the first "I".  The second affected post (reply #10) quoted a part of LemSteven's original post and choked at the – character after import, which again is not the keyboard hyphen but the fancier "En Dash" character.

So there might be a relatively simple way to identify and fix the affected posts after all?  At least I don't think we have to manually compare to figure out which posts are affected, if my theory's correct.

As you can see plainly in this post, the non-ASCII characters in question do work correctly on this forum when I copy and paste them into this post here, so it's not clear yet why they aren't working with the import.  One possibility is that the old forum used different character encodings for those characters than the new one, though I would've thought that would just cause the characters to be displayed incorrectly rather than cutting off the remainder of the post.

namida

#23
I'll look into that when I get home. (about an hour from now)
Thanks for identifying the likely cause. :)

EDIT: Okay, I looked at the source files from the backup, as well as the SQL file. Notepad++ assumes the SQL file is SHIFT-JIS encoding (a Japanese character set) which may be the source of the problem. I'm going to look at the database itself on here to see exactly how these posts imported, to see if
a) an automatic script could possibly fix them
b) if not, then if an automatic script could possibly at least identify them

In the latter case, depending on how many there are, I'll see whether it'll be less effort to write some script to re-import the broken ones, or to simply manually fix them up.

EDIT: Okay, so the cut-off happened while importing the posts through PHPMyAdmin - the full content is there (albeit with slight corruption on those characters) in the SQL file; it's gone in the copies of the posts on the database for this forum.

Thus, the easiest way to identify them will be to modify the code used to generate the LemmingsWelt SQL import code.

EDIT: Done this, and am currently running it. Basically; the code is set to print out (to a text file) a list of posts that have non-ASCII standard characters.


EDIT: Done this too. Although they are a minority of the LemmingsWelt posts (and since the test was simply "is there any character that isn't standard ASCII", it might have some false positives), there is a significant number of them, so I modified the app again to create a SQL file that'll add these posts to a temporary database table, with the characters causing the corruption converted to the HTML special character equivalent (eg, character 0x9F would become F;, just to give an example). If this goes well and imports successfully, I'll write a PHP script that attempts to automatically fix the broken posts based on this temporary table.
My projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)
Non-Lemmings: Commander Keen: Galaxy Reimagined (a Commander Keen fangame)

namida

Okay, should be fixed. It doesn't have the exact characters - I really have no idea why exactly this is - but honestly, I don't think it's a huge deal. The posts are definitely readable now.
My projects
2D Lemmings: NeoLemmix (engine) | Lemmings Plus Series (level packs) | Doomsday Lemmings (level pack)
3D Lemmings: Loap (engine) | L3DEdit (level / graphics editor) | L3DUtils (replay / etc utility) | Lemmings Plus 3D (level pack)
Non-Lemmings: Commander Keen: Galaxy Reimagined (a Commander Keen fangame)

grams88

Hi Everyone

I've noticed a new emoticon,  :lix: I don't think I remember seeing that one before maybe it was different in the past.

Proxima

 :lix: is new. She represents the characters from the game Lix.

Clam

Is it possible to stop the whole page jumping around when the news ticker changes?

IchoTolot

Quote from: Clam on January 15, 2015, 07:03:55 AM
Is it possible to stop the whole page jumping around when the news ticker changes?

Yes that would be great! I thought I was the only one with that issue :P

Nepster

There is a button in the upper right corner of the "News"-box, that shrinks the whole News section. This takes care of the page-jumping issue, if you don't mind missing updates in the news ticker.