Extracting levels from console versions

[WNivek]

You missed a pixel in the graphics for the Sunsoft special.

Hmm, you're right. I wonder how I missed that. Guess I wasn't as careful as I'd thought...
Anyway, thanks for the heads-up. I have corrected the error, and reuploaded the file.

[ccexplore]

Since the differences above are spotted by eye, I probably should at some point also make a program to do a full pixel-by-pixel comparison to make sure I didn't miss anything.  I also need to modify my program so it can create suitable levels for me to compare the object graphics.

It turns out I did miss a few other, very minor differences.  With the comparison done by program, I'm now confident that I've identified all differences in terrain pieces.  The result (old and new differences) can be seen in this image:

http://www.geocities.com/guestlevels/lemmings/genterrdiff.png

The left column are the Genesis graphics and the right column are the PC counterparts.  All the newly found differences are only a pixel or two.  This makes them quite hard to fix in a general manner, but on the other hand it also means not fixing them probably won't affect a level much if at all.

Next step is for me to do the same thing for object graphics.  Although with objects, the exact graphics doesn't matter much since what really counts in the gameplay is only the object's trigger area.

[ccexplore]

Next step is for me to do the same thing for object graphics.  Although with objects, the exact graphics doesn't matter much since what really counts in the gameplay is only the object's trigger area.

I have made comparisons to both the objects' graphics and trigger areas.  For the most part they're the same or nearly the same between the PC and Genesis versions, but there are some significant differences:

1) the Genesis version appears to have corrected the errors in the trigger areas for one-way walls and water objects (in the PC version the trigger areas for those objects are short by 4 pixels vertically, not extending all the way to the bottom of the object as one would expect)

2) In graphic set #4 (aka "Blue"/"Crystal"), the slicer trap and the laser traps are positioned quite differently from the PC version.  Their graphics appear to be shifted 4 pixels to the left, and for the slicer trap, 6 pixels down as well.  Their trigger areas in Genesis version are both shifted 4 pixels left and 4 pixels down.  This is one major reason why many of the laser traps aren't working in the current extracted levels when played on the PC.

===========================

Unfortunately, it's not possible to shift an object's position in a level by 4 pixels; the x-position is always aligned to multiple of 8.  So I can't really fix the graphic positioning issue for problem #2, but I may be able to deal with the trigger areas.

There are two approaches to handling the trigger area differences:

1) Create a modified set of GroundXo.dat files that uses the Genesis version's trigger area specifications.
2) Use the PC version's GroundXo.dat files, but adjust the objects' positioning in the level to compensate for the difference in trigger area positioning.

My preference is of course approach #1; however, I can see that it may be confusing and slightly inconvenient for people to maintain a separate set of GroundXo.dat files just for playing the Genesis levels, plus replacing the GroundXo.dat files may be trickier in other versions like Amiga or Mac.  So I'm likely going to provide two versions of each affected level, one using the modified GroundXo.dat files, and the other using the unmodified PC GroundXo.dat files but with modified positioning of objects in the affected levels.

------------------

One other thing to note is that based on the comparison of trigger area positoning, some traps (eg. the rope trap in Fun 26) really do appear to be not working in the Genesis level even when played on the console.  For now I will probably leave such issues unfixed since they are present in the actual console game as well.  If those nonfunctional traps are causing unacceptable major backroutes that totally ruin a level, then we can consider fixing them later at the stage when we start fixing other things like clipping and screen starting position.

I'm planning to modify my extraction program, and will hopefully be able to upload an update to the extracted levels by sometime tonight or tomorrow afternoon.

[WNivek]

Unfortunately, it's not possible to shift an object's position in a level by 4 pixels; the x-position is always aligned to multiple of 8.

I know it's not as widely compatible as a level format, but Lemmini's ini levels aren't limited in that manner. If you decide to also extract levels to ini, that one detail won't be an issue.
(Note that I'm not trying to suggest abandoning lvl. I know it's a much more widely accepted format, and should be considered the priority. However, it might be worth considering to additionally provide ini directly, in order to sidestep the 4-pixel issue for some users)

Out of curiosity, which version are you working from? I seem to recall seeing that there were three versions of the Gen/MD Lemmings - PAL, NTSC (rev00), and NTSC (rev01).
I don't know just what differences might exist (beside the obvious) but, if you're not already, it might be a good idea to compare between them and see if anything you're dealing with here is altered from one to another.

[ccexplore]

Out of curiosity, which version are you working from? I seem to recall seeing that there were three versions of the Gen/MD Lemmings - PAL, NTSC (rev00), and NTSC (rev01).

I don't know just what differences might exist (beside the obvious) but, if you're not already, it might be a good idea to compare between them and see if anything you're dealing with here is altered from one to another.

I'm not sure how to tell which version is mine.  I don't want to search the web since I don't know if they'll always indicate which version they are offering, so if you want me to go through all three versions you better either e-mail me the ROMs or point me to some web pages where I can find each version.

[Mindless]

PAL, NTSC (REV00), NTSC (REV01)

Hit "Telecharger" -- you may have to wait a bit if you download multiple ROMs.

[ccexplore]

Thanks Mindless.

It looks like the ROM I had is the "NTSC (REV00)" version.

I'll investigate the other versions later, but only after I'm done with the REV00 version level extraction to my satisfaction.

[Craig3410]

Nice work on extracting them.

Now if only I could compile the levels into a playable level pak; clicking "load level" and picking the right level 150 times gets annoying....  :smiley:

[ccexplore]

Progress update:  I finally updated my extraction program to both apply some fixes automatically, and warn me about the rest.

So I'm now at the process of reviewing "the rest" and see which need fixing.  I've gone through Fun and Tricky so far.  As mentioned before, when I'm done I'll release two versions of the extracted levels, one using the original PC-DOS groundXo.dat files, and the other using a modified version of the groundXo.dat files to better match the Genesis version.  The main difference is that the one using the modified groundXo.dat would have less changes in the levels themselves.

Since there are always subtle differences in the game mechanics itself (especially since the levels may be played on a multitude of ports and clones that support the LVL format), I've decided it's pointless to try to get the levels to be 100% identical to Genesis, so unless a difference has a significant effect on gameplay it won't be fixed.  For example, I mentioned that the laser traps are positioned 4 pixels more left in the Genesis version than the PC version, but in all instances where the traps are used, this doesn't seem to make any difference to the gameplay, so I would consider the difference not worth fixing (ignoring the fact that it can't be easily fixed anyway for PC Lemmings and many other ports).  Ditto for those terrain pieces that differ by one or two pixels in the Genesis version (unless that pixel really makes a crucial difference in a level, which so far haven't been the case).

[ccexplore]

I've now reviewed up to Mayhem, so only 60 more levels to go <pant, pant...>

Hopefully I'll be done with this by end of this week.

[edit:  finished reviewing Present.  Progress has slowed down because the two Genesis-unique ratings have more levels that need reviewing.  I also discovered that some levels need further adjustments due to differences between how Genesis Lemmings and DOS Lemmings calculate fall distances for non-walkers.  With great luck I might finish by Sunday but who knows......]

[ccexplore]

I'm still not done reviewing the levels, but I have finished making the alternate GroundXo.dat files for the Genesis levels.

http://it.travisbsd.org/lemmings/lemmingswelt/index.php?cmd=get&file=/GenesisGroundX.zip

There are two sets of GroundXo files in that zip file.  One set ends in "G" (eg. Ground0G.dat), and the other set ends in "H".  In both sets, I modified the trigger areas of various objects to better match the values in Genesis Lemmings.  The G set also has the palettes (ie. color scheme) modified so that it looks more like the Genesis graphics; the H set uses the original palettes of PC-DOS Lemmings.

How to use:  if you use Lemmix to play these levels, you can create a new style to make use of these files.  First, make sure to exit all running instances of Lemmix.  Then make a backup copy of the LemmixStyles.ini file in case you mess up the following steps and Lemmix stops working.  Now open up LemmixStyles.ini in a text editor.  Add the following to the beginning of the file:

[style_10]
StyleClass=TDosOhNoStyle
CommonPath=C:\Lemmix\
MainDataFile=main.dat
OddTableFile=
StyleDescription=Sega Genesis/Megadrive
StyleName=Genesis

(change the value for CommonPath from "C:\Lemmix\" to the correct path on your computer, pointing to the directory where Lemmix resides)

Then add the following at the end of the file:

[Genesis_0]
GraphicSetClass=TBaseDosGraphicSet
BrickColor=4291854368
GraphicExtFile=
GraphicFile=VGAGR0.DAT
GraphicSetArchive=
GraphicSetId=0
GraphicSetIdExt=0
GraphicSetName=Dirt
MetaInfoFile=GROUND0G.DAT

[Genesis_1]
GraphicSetClass=TBaseDosGraphicSet
BrickColor=4278214800
GraphicExtFile=
GraphicFile=VGAGR1.DAT
GraphicSetArchive=
GraphicSetId=1
GraphicSetIdExt=0
GraphicSetName=Fire
MetaInfoFile=GROUND1G.DAT

[Genesis_2]
GraphicSetClass=TBaseDosGraphicSet
BrickColor=4290818288
GraphicExtFile=
GraphicFile=VGAGR2.DAT
GraphicSetArchive=
GraphicSetId=2
GraphicSetIdExt=0
GraphicSetName=Marble
MetaInfoFile=GROUND2G.DAT

[Genesis_3]
GraphicSetClass=TBaseDosGraphicSet
BrickColor=4293980256
GraphicExtFile=
GraphicFile=VGAGR3.DAT
GraphicSetArchive=
GraphicSetId=3
GraphicSetIdExt=0
GraphicSetName=Pillar
MetaInfoFile=GROUND3G.DAT

[Genesis_4]
GraphicSetClass=TBaseDosGraphicSet
BrickColor=4293980400
GraphicExtFile=
GraphicFile=VGAGR4.DAT
GraphicSetArchive=
GraphicSetId=4
GraphicSetIdExt=0
GraphicSetName=Crystal
MetaInfoFile=GROUND4G.DAT

Notice how in each section the "MetaInfoFile" points to the new files I provided.  If you want to use the H set instead of the G set, simply change it to GROUNDxH wherever you see GROUNDxG above.

Save your changes to LemmixStyles.ini.  Then, unzip/copy the GroundXG/H files to the directory where Lemmix resides.  Also copy the VGAGRx.DAT files from DOS Lemmings or CustLemm into the Lemmix directory if you haven't already done so.  Now, to check that it's working, run Lemmix, and open one of the extracted Genesis LVL level file.  Lemmix would then make you pick a style to be used.  If you change LemmixStyles.ini as above, the drop-down list of styles should now have an entry named "Sega Genesis/Megadrive" at the end of the list.  Selecting that style and click OK, and the level should open using my Genesis style.

======================

If you use CustLemm or one of the original PC-DOS Lemmings game to play, the best thing to do is to make a copy of the entire folder containing CustLemm/Lemmings, to be used exclusively for playing the Genesis levels.  Then in that folder, copy the GroundXG (or GroundXH if you prefer) files over the original GroundXo files.  This means deleting the old GroundXo files, copy the new GroundXG/H files into the folder, and then renaming them so they end in "o" like the original files.

--------------------------------------

I don't know if other ports/clones of Lemmings besides the ones mentioned above use GroundXo.dat files, so these files probably won't be of use for Amiga Lemmings, WinLemm or Lemmini.

[ccexplore]

All right, I finally finished with my first pass with the extracted Genesis levels:

http://www.geocities.com/guestlevels/lemmings/genlvls_6-12-07.zip

This is a first pass because I haven't dealt with certain mainly aesthetic issues, such as needing to erase terrain that would be clipped out of view in Genesis Lemmings.  The main goal of the first pass is to make sure the levels work more or less as they would when played in Genesis, in terms of solving.

The zip file contains a bunch of folders, and you'd find that some levels exist in more than one folder.  This is because for some levels, I need to make multiple versions of a level.  The two factors that lead to this are:

1) If you choose to use the original DOS Lemming styles (groundXo.dat files) instead of the Genesis Lemmings styles (groundxG.dat or groundxH.dat) I provided earlier, some objects' trigger areas will be setup differently, and in some cases this may cause a level to not work correctly without changes.

2) If you choose to use CustLemm or one of the programs from the DOS Lemmings series to play the levels, these programs have the limitation of no more than 80 lemmings in a level.  Some Genesis levels have up to 100 lemmings.

Thus to account for these two factors, alternate versions of the affected levels were made to mitigate these issues.  The readme.txt in the zip file explains how to work out the correct set of files to use based on how you plan to play the levels.  If anyone has any questions regarding this please feel free to ask me.

Note:  if you're going to play the levels using CustLemm, make sure to use CustLem2.exe or nocdlem2.exe.  Those versions of CustLemm have the same maximum safe falling distances as DOS Lemmings, ONML, and Genesis Lemmings, unlike CustLemm or nocdlem.exe.  To illustrate the problem with CustLemm, note for example that you don't have to do anything to solve the "We All Fall Down" levels in CustLemm because of its higher safe falling distance!  Quite a few of the Genesis levels are sensitive to the correct falling distance, so just don't use CustLemm, use CustLem 2 or nocdlem2.  Similarly if you use Lemmix, you should not use the "Customize Lemmings" style to play the levels; use "DOS Original Lemmings" or my "Sega Genesis/Megadrive" styles instead.

==============================

Some gameplay-affecting issues I haven't resolved yet:

1) Present 6 and Sunsoft 26 are both levels where the tight time limit is a crucial aspect of the level, and both have 100 lemmings.  Thus the version of the levels that have the number of lemmings reduced to 80 may be easier than the original.  To mitigate this I would need to analyze the level in greater detail, and then change the terrain to compensate for the lower number of lemmings (eg. a lengthen the path to the exit or something).

2) I have left out all 5 special graphics levels.  Despite the fact that we got screenshots of those levels from deveria.com, I later found out that the screenshots are very close, but not quite as accurate as I'd like--in some cases the dimension of the screenshot isn't even correct (things like 510x159, when the expected dimension should be 512x160)!  So for the sake of making sure I'm using 100% accurate material, I've decided to hold off on those levels until I figure out how to extract the special graphics bitmaps directly from Genesis ROM.

3) Tricky 30 is still solvable in DOS Lemmings / Lemmix, but slightly less straightforward than in Genesis Lemmings.  This is due to subtle differences in how the two games keep track of falling distance, with DOS Lemmings being less forgiving in certain cases.  In the case of Tricky 30, it's simply not possible to mitigate the difference by modifying the terrain.  The solution in DOS Lemmings will require careful adjustment of the release rate unlike Genesis Lemmings.

------------

It should also be noted that in going through the levels, I specifically only looked through the levels that have interactive objects or terrain that I had identified to differ in DOS and Genesis Lemmings, and levels that were flagged for certain specific things (eg. more than 80 lemmings).  As I alluded to above with Tricky 30, some levels which have identical objects setup and terrain in DOS Lemmings and Genesis, may nevertheless be affected by a few subtle game mechanics differences.  Since I didn't actually playtest every single level, it's possible I might miss catching these issues on a few levels.

[ccexplore]

2) I have left out all 5 special graphics levels.  Despite the fact that we got screenshots of those levels from deveria.com, I later found out that the screenshots are very close, but not quite as accurate as I'd like--in some cases the dimension of the screenshot isn't even correct (things like 510x159, when the expected dimension should be 512x160)!  So for the sake of making sure I'm using 100% accurate material, I've decided to hold off on those levels until I figure out how to extract the special graphics bitmaps directly from Genesis ROM.

I have now extracted the special graphics bitmaps from the ROM itself.  Although it turns out I was somewhat wrong about the dimensions:  it is indeed stored in the ROM as 512x160, but it looks like the game doesn't actually let you scroll far enough right to see the 512th column of pixels, so in effect it's really only 511x160.

Download here:

http://www.geocities.com/guestlevels/lemmings/genspeclvls.zip

Again, some levels have more than 80 lemmings, so for them I also made alternate versions that have the number of lemmings reduced to 80.

Besides the LVLs, the zip file also contains the VGASPECx.DAT files that DOS-based Lemmings games (eg. DOS Lemmings, Lemmix, CustLem2) use to get the special graphics.  I have set up the LVLs such that they ask for VGASPEC 4-8, to avoid clashing with the existing VGASPEC 0-3 from DOS Lemmings.

Finally, for the other programs out there that support special graphics levels differently, I include PNG bitmaps of the special graphics as well.  They are sized 960x160 instead of 512x160, due to a side effect of my using the bitmaps as an intermediate step for creating the VGASPECx.DAT files (namely, VGASPECx.DAT files expect the bitmap size to be 960x160), so for your use, you might want to crop it back down to 511x160.  (The original 512x160 is centered horizontally in the 960x160 area, so the x range is really 224 to 735 inclusive.)

Speaking of cropping, because DOS Lemmings has a much larger playfield than Genesis Lemmings, you will notice that many of the special graphics levels will look abruptly cut off on the left/right when viewed in the DOS Lemmings playfield.  It's unfortunate but to fix it properly would take time and can get tricky (no pun intended) in some cases.

As it's clear that interest in this whole Genesis levels extraction thing has more or less waned, and I've basically completed what was asked, I'm going to stop working further on this for now until popular demands otherwise.  Enjoy!

[Fleech]

Well my interest certainly hasn't waned, ccexplore.

As I said before I'm happy to help out with adding the final touches to the levels (screen start, sorting out the clipping issues etc) if you want. I really appreciate the work you've put into this, so if you don't mind having other people editing the levels then I'd be glad to.

Let us know.

EDIT: I've just had an idea for when the levels are finished. EricLang created an empty game template for the Remake project. With his permission we should be able to insert these levels into it for a proper remake of the Mega Drive/Genesis version, shouldn't we? (obviously replacing the existing groundxx.dat files etc with the new ones).

[ccexplore]

As I said before I'm happy to help out with adding the final touches to the levels (screen start, sorting out the clipping issues etc) if you want. I really appreciate the work you've put into this, so if you don't mind having other people editing the levels then I'd be glad to.

Go ahead!

EDIT: I've just had an idea for when the levels are finished. EricLang created an empty game template for the Remake project. With his permission we should be able to insert these levels into it for a proper remake of the Mega Drive/Genesis version, shouldn't we? (obviously replacing the existing groundxx.dat files etc with the new ones).

I'm not entirely clear what you have in mind when you said a remake of MegaDrive/Genesis.  The Remake project that has been talked about, if I recall, has more to do with compiling a whole new set of levels from mostly the CustLemm levels people have created over the years.  Maybe I'm thinking of something else, I haven't followed that thread for quite a while.

Anyway, it sounds like the template thing can be of use to recreate a version of MegaDrive/Genesis Lemmings on the PC.  However, I think to most accurately experience Genesis Lemmings, it is best if Lemmix can be extended to handle the subtle differences in game mechanics for example (and maybe even major differences like the smaller 512x160 playfield, though I'm of two minds on that).  I already talked to EricLang about the possibility for Lemmix/LemmixPlayer to support more versions (eg. Mac), and I have done most of the reverse engineering work already for Genesis Lemmings, so ultimately this will be up to him.