Lemmings Forums

(EDIT: This topic was originally for an extraction tool to convert to Lemmini/jLevelBuilder formats; hence the first few posts are about that. For the download of LemSet, click here.)


Here's a little tool I wrote that'll extract the graphics from DOS Lemmings into Lemmini-friendly format. This might be useful for those who want to create precisely-adjusted levels for DosLemm in jLevelEditor, or those who simply prefer the low-res graphics.

It should be noted that the trigger area images it creates ([style]om_#.gif), while they are accurate to how DOS Lemmings performs and to the images extracted (try copying/pasting them over the image), Lemmini does not seem to like them. You'll probably want to use these ones for displaying trigger areas in jLevelEditor if you're creating levels for DOS, but if your goal is to use the graphics in Lemmini itself or create levels to be played in Lemmini, use Lemmini's existing versions of these files. I included a copy of them (in the "default" folder) just in case you don't have them.

This program will create the INI files. However, as I didn't work out the conversion table for trap sounds, the trap sounds it uses are all hardcoded; so if there are any gaps in the graphic set files you provide (or any diversion from the standard ordering), some traps may end up having the wrong sound. If someone would like to make a list for me of what sound effect ID in DOS = what sound effect ID in Lemmini, I'll be more than happy to work that into the code of the program, but I can't be bothered working it out myself.

And yes - this program can convert graphic set 9 (Xmas Lemmings) to Lemmini format! As far as the trigger area problem bug mentioned above goes, I created a custom trigger area graphic for the set 9 exit that SHOULD work; it's in the Default folder along with all the actual default ones. (The exit is the ONLY object in Set 9 that has a trigger area that does anything; and as we know, Lemmini completely ignores trigger areas with no effect.) You're on your own for converting the Xmas Lemmings levels, though... =P

Also - while the GROUNDxO.DAT files can (and should) be used as-is, the VGAGRx.DAT files must first be extracted using ccexplore's ldecomp or Mindless's Lemmings Tools. VGAGRx.DAT will extract into two sections; the first should be saved as VGAGRx.DAT.0 and the second as VGAGRx.DAT.1. Mindless's tools will automatically suggest these filenames.

Source code can be found here.

Just to keep updated, I have successfully written code to extract DAT files. The existing program would be a pain in the ass to add this into (believe me, it is VERY sloppily coded), but I'm working on a new program which is more tidily coded and thus easier to add features to. My plan is for it to support not just extracting graphics to Lemmini, but more or less being a full-featured manager for Lemmings data files (basically, capable of editing as much as possible within the DAT files, apart from levels (since there's already three level editors out there) - I'll probably include basic functionality for levels but not full-blown editing, stuff along the lines of changing skills/title/etc). So you'll (finally) be able to create your own graphic sets. =D

I'm currently working on the code to compress them again. Initially I plan to just use faux-compression (ie: repeatedly using "store exact bytes"), but the algorithm is quite simple so at some point I'll probably develop a proper compression algorithm.

Just a warning: If you're going to start creating now (I wouldn't reccomend it, I'm notorious for not finishing what I start), the most likely format I'd use would be 24-color bitmaps. Not sure how I'll handle transparency, most likely using death magenta (it's worth noting that Lemmings only uses 18-bit color values, so you can use 255,0,255 for transparent and, say, 254,0,254 if you want the actual color - both would come out the same after converting to 18-bit anyway, but they can be detected seperately *before* the conversion). No idea what I'm going to do about objects yet. Of course, if you have any suggestions for what could work well, fire away. Keep in mind that I'm quite averse to using pre-existing libraries to load files, I prefer to write my own code so I know exactly how it works; this means that I'm probably not going to want to work with complex formats and would rather stick to simple ones like BMP.

So, at the moment, it's uncompressing all 10 default VGAGR files in a little under 11 seconds. Not bad, considering how inefficient my code tends to be. xD

Working on the compression algorithm now. Initially aiming for a "just store the damn thing in a way that will decompress" (using 00's or 111's, for thsoe who know the format), but before I go on to write other code I want to write an actual proper compression algorithm.

At this point, it seems to generate the bitstream perfectly, but there's a glitch somewhere in the transferring it to bytes...

EDIT: Bug located, and after close examining, it's actually nowhere near as severe as I thought! The encoding is fine, the transferring to bytes is fine - the problem is it just cuts off what it writes to file a byte or two early. This should not be hard to fix.

... Oh god. The problem was that I forgot to include the header when calculating the compressed data size. Stupid mistake really, but hey, easily fixed once I worked it out.


EDIT: Success! I wrote a little snippet to test my various routines - basically, the code instructed it to "decompress all VGAGR files, recompress the extracted data into new files, then decompress those new files". (The last part simply because, if I know my decompression works - which it does, I can tell by comparing it to the output from ccexplore's or Mindless's tool - the easiest way to test if my compression works is to attempt to decompress it again.) The output from the first decompression is identical to that from the second, both of which are also identical to the output from ccexplore's/Mindless's tools. =D AWESOME. Next job, compression algorithm to replace the faux-compression currently used. I already have a very good idea of how I'm going to write it.


EDIT AGAIN: Success! It now compresses the files. My algorithm usually results in files about 200 or so bytes larger than the original files - but one major exception is VGAGR5.DAT, where it beats the original by nearly 2KB. Is that file known for being inefficiently compressed or anything?

As posted over in the Lemmings Discussion forum, I've had success to some degree! Now, a lot of this is hardcoded, it required a bit of manual hex editing, and currently my compression algorithm is not very much liked by LemEdit or CustLemm (however, Mindless's decompression tool has no problem with it - didn't test with Lemmix yet) - so I had to output uncompressed data then manually compress it with Mindless's tool. So, this is mostly a proof-of-concept at this stage, but I'm working on an actual tool that can easily be used.

Nonetheless, I present to you the first ever DOS-compatible custom graphic set for Lemmings. =D

Works as-is with LemEdit and any DOS version of lemmings (and SHOULD work as-is with Lemmix too); if you want to use jLevelBuilder to mess around with it, use the extractor tool in the first post.

EDIT: For those who can't (or don't want to) download it and run it themself, but want to see it in action instead of just a screenshot, here: http://www.youtube.com/watch?v=dQ8U-6AgnKE

Good job! 

Thanks!
Next goal is to ditch all the hardcoded data and make it actually read from a file (or take user-entered values). I'll probably work on fixing up the compression algorithm after that; it's not as major IMO since we already have a program that can take care of that.

Some interesting finds so far:

1) ccexplore's documentation mentions that the default styles, terrain pieces only ever use the upper 8 pallete entries. I can confirm that they are NOT restricted to this - they most certianly can use the lower 8 too. Likewise, the mask can be completely seperate from the bitmap data, even though the default styles all use one of the bitmap planes as a mask.

2) Those who are familiar with Lemmings' mechanics probably expected this - but there are two types of trigger areas that aren't documented, specifically, "turn lemmings left" and "turn lemmings right". These are almost certianly the same effects used by the game when you place a blocker; but you can attach those effects to objects too! (Note: I haven't yet tested how well Lemmix handles this, only tested in the actual DOS game. At any rate though, it wouldn't be a major modification to Lemmix to add support if it's not already there.) You can also make an object place a steel area; this could be useful if you want larger steel areas than the level format inherently handles (or a larger quantity of them), though it's the first two that I'm expecting might see some really creative usage.

3) ccexplore's documentation mentions that the second bit of animation_flags seems to do something. From what I can tell, it simply tells if the object animates at all (kind of redundant when you can just work it out from the frame data, but that's the only thing I can come up with).



I'm wondering - is there any known information about the CGA version? And if not, could anyone provide me with a copy of the CGA version so I can experiment? Wouldn't mind adding support for that too. (My program already handles EGA, not that that's a huge task.)

Very nice!

After you describe it, it sounds logical to have special objects with directional funneling capability, but I've still did not expect them.

-- Simon

2) Those who are familiar with Lemmings' mechanics probably expected this - but there are two types of trigger areas that aren't documented, specifically, "turn lemmings left" and "turn lemmings right". These are almost certianly the same effects used by the game when you place a blocker; but you can attach those effects to objects too!

Yep, that's exactly how blockers work.  I hope Lemmix does it that way too, though there's no way to tell for sure without testing, and given that it already handles certain aspects of trigger areas incorrectly (eg. 0-valued triggers not canceling steel areas), it's a toss up as to what the actual result is.

In fact in addtion to "turn left" and "turn right" there is also a (admittedly far less useful) third value, 10 I think, which is used to represent the "neutral" central column in the blocker's field.  It has no effect on lemmings besides that its presence would prevent blocker assignments nearby, because the game checks for and disallows blocker assignments that would overlap with existing blocker fields (any "turn left", "turn right" as well as "neutral").

I'm wondering - is there any known information about the CGA version? And if not, could anyone provide me with a copy of the CGA version so I can experiment? Wouldn't mind adding support for that too. (My program already handles EGA, not that that's a huge task.)

The "alternate version" download from abandonia.com should have all versions of EXEs: vgalemmi.exe, cgalemmi.exe and tgalemmi.exe. 

IIRC the CGA hardware only supports one color graphics mode which is 320x200 with basically 4 fixed set of colors (ie. 2 bit per pixel).  The layout of pixels in the graphics files (after decompression) will most probably mirror the video memory layout, so with a little Internet research and some experimentation you should be able to work it out.

Awesome. Well, first thing is to get this completely working for DOS. First version will support the full features of graphic sets, but will require the use of an external tool to compress the output (it won't require an external tool to decompress, only to compress - but no GIF support yet). It'll generate the sections for you. After that, I'll probably work on fixing up the compression algorithm - are you aware of any oddities that might be causing Lemmings / LemEdit to reject them, yet your tool, Mindless's tool and my own (all of which also decode the official files perfectly) can decompress them no sweat? Upper limits on the offsets or something that are lower than the file format allows for?

In terms of coding, I've just coded loading and saving INI files. There's actually only 10 values per object that need to be defined (and 6 of these are to do with trigger areas, while a couple of others are set to fixed values for windows, so you generally don't need 10 for each object), so it should be a quite simple format. That aside, the INI file can also be used to define the pallete - alternatively, you can use a 16x1 pallete.bmp file with one pixel for each color. Any color that isn't in the pallete becomes transparent. (It converts to 18-bit before matching so if, say, you have 252,252,252 in your pallete and use 255,255,255 in your image, it will count that as a match.)

(I'm aware scanning the images and building a pallete based on what's used would be easiest, but that's a feature I can add later. For now, I just want to get out something that works.)

are you aware of any oddities that might be causing Lemmings / LemEdit to reject them, yet your tool, Mindless's tool and my own (all of which also decode the official files perfectly) can decompress them no sweat? Upper limits on the offsets or something that are lower than the file format allows for?

Mostly "no".  I vaguely recall having observed cases of graphics issues in DOS Lemmings (but nothing that causes outright rejection) if the compression on VGAGrX isn't good enough, but I don't have any concrete details to share around this.  Decompressed size shouldn't matter from what I recall seeing in the code.  I suggest not worrying too much about it for now, just get something that works and see if anyone reports any problems.

Well, LemEdit gives an error message, while the games just black screen...
Anyway, I've noticed a way in which my algorithm could be MAJORLY improved, so I'll probably work on both of those things at the same time. But for now... yeah. Focus is on getting it to output non-hardcoded-value-based GROUNDxO.dat files. Which isn't a hard task, it's just a task I haven't completed yet. xD

Well, LemEdit gives an error message, while the games just black screen...

What happens if you try to use the assets in Lemmix (either level editor or player)?  That may provide some insights as well.  It doesn't sound to me like the issue has anything to do with compression or decompression, especially if the data decompresses identically across multiple tools.  I strongly suspect you have bad data in your groundXo file.  If the problems were with the graphics files (ie. vgagrX.dat), I would expect at worst corrupted graphics being displayed but shouldn't result in error messages or black screens.

Not possible. As I said (or at least thought, maybe I forgot to write it), if I take the output VGAGR file from my app, decompress it with Mindless's tools, then recompress it again with the same, it works perfectly - hence the download.

I tried with the Lemmix player. No problems, it loads perfectly.

The only thing I can think of that may help is that LemEdit gives an error screen with some information. I can't work out exactly what the information's referring to, but maybe you can?

Lix displays tileset and level properly. The L1 file reading code was written according to the same documentation as yours.

Johannes, the Droidlings author, made some L1 tilesets himself and published them (link to that topic). Those tilesets were broken in Lix, but ran fine in Lemmix. They have never been tested in Custlemm or Lemmings 1 itself.

Therefore, even though Lix loads your set nicely, it's not a seal of guarantee for matching the L1 file type definitions. There's even a slight chance for the descriptions to have subtle errors or to leave out specifications of irregular forbidden cases. But most likely, Lix and Lemedit misinterpret, next likely is that Lemmix misinterprets, only then that the description has errors. Custlemm is a cracked L1 executable and should be 100 % accurate.

I can't do much with the Lemedit error, it looks like debugging info for people with its source.

-- Simon

Not possible. As I said (or at least thought, maybe I forgot to write it), if I take the output VGAGR file from my app, decompress it with Mindless's tools, then recompress it again with the same, it works perfectly - hence the download.

Sorry, I misread your posts and didn't realize you are trying to get your program to handle compression itself.  If you haven't done so already, please attach the copy of file that is compressed by your program and not working with LemEdit etc. and I'll take a look when I have time.

[edit: from reading my documentation, I can only think of one thing so far that could lead to a case where the data decompresses perfectly in Lemmix but leads to an error in LemEdit/Lemmings.  Suppose you calculated the number of bytes in compressed data incorrectly so that it's actually larger than the exact correct value.  This means you effectively have bogus "extra bytes" in your compressed data.  In my documentation I say "you determine when to stop decompressing by keeping track of how many bytes you have decompressed so far".  This leads to the possibility that Lemmix would stop once enough bytes are decompressed, without trying to consume any extra bytes in your compressed data, and thus you get perfectly correct decompressed data.  However, if LemEdit actually goes on based on the number of bytes of compressed data, then it wouldn't stop as soon as enough bytes are decompressed, and would try to go on processing the bogus extra bytes as well, thereby leading to problems.

Just a wild guess for now.  Once I have an actual file hopefully I can understand better.]

Here's the result of remaking the 10 official sets from the decompressed data. The program uses its own output (from the decompression algorithm) as source data - however, I am 110% sure there's no problems with the decompression algorithm, only the compression one. I've included the source code as well, in case that's any help in tracking down the error. (It's QB64, which is basically just an extended and cross-platform version of QBasic.)


The code's been worked on a bit since then; at this point, the official games (I've tried both original and CustLemm for this) don't refuse to load, but they do take a few seconds then display really glitchy graphics. LemEdit still crashes; everything else still works fine.

It's coded with QB64 GL Version and I'm compiling it under Windows, not that I'd think those factors would hugely matter. The code is somewhat untidy (though very neat by my standards xD), so just let me know if there's anything you don't understand what it's meant to do. Also - let me know when you've downloaded it so I can remove the attachment, I'd rather not have the source too widely distributed until it's working properly at least.

EDIT: Attachment removed.

Downloaded.  I probably won't get around to looking at anything until much later today, but you can remove the attachment anytime now.

Just need to do some testing and maybe iron out a few bugs, and the first release should be ready. =)

At this stage, it doesn't compress, for the reasons mentioned above. It'll output the GROUNDxO.DAT file, as well as VGAGRx.DAT.0 and VGAGRx.DAT.1 - the ground file is ready-to-go, the VGAGR files should be run through Mindless's DAT compression tool.

Okay, here you go! Enjoy!
It's not the easiest thing to use at this stage, but the important part is: it works, at least.

I attempted to explain how to use it in the included text file, but since I'm terrible at explaining things, as well as that some people can understand examples better than explanations, I also included a sample style with the correct file names and a correctly set up INI file to compile into a graphic set. It's the same tree style that I posted a couple of days ago to demonstrate the tool working - but now compiled entirely from the image files and an INI file, instead of half the information being generated from hardcoded values. =D

If you need any help with how to work it, just ask. =) Runs on Windows. If anyone's wanting to run it on Linux, try WINE first, if that doesn't work, I'll see if I can compile a native Linux version. (There is a QB64 version for Linux, and from what I've heard code ports VERY well from one platform to another in QB64, so...)

Remember - you need to use Mindless's tools to compress the two sections into one VGAGR file at this stage, the app doesn't have a working compression routine yet. To save you having to search, I included a copy of the tools in this archive. (Of course, if Mindless would prefer I didn't redistribute them, I'm happy to remove it and provide a link instead - I just didn't want to risk ever having a case where people have copies of the app but Mindless's tools are no longer possible to find, thus making the app useless.)

It'll probably be a while before I post any further updates, because:
1) I should be studying for exams, not working on Lemmings projects, at the moment.
2) It works, even if it's not the friendliest app.



EDIT: For creating levels with your custom styles, you can either:
- Use LemEdit. Once the VGAGR is compressed, you can use the custom style in LemEdit directly.
- Use Lemmix. Once the VGAGR is compressed, you can use the custom style in Lemmix directly.
- Use jLevelBuilder. After compiling the set, use the extraction tool in the first post. (You do not need to compress the VGAGRx.DAT then decompress it again - just use the files LemSet generates directly with the extraction tool!) I'm not sure exactly how you configure jLevelBuilder to put the right graphic set number in the output file, though. (You could, of course, just replace one of the default sets.)
- Possibly Lix? I haven't tried it myself so I don't know what it does or doesn't support or what formats it expects; unfortunately, Lix doesn't seem to like my PC.

In general - if the editor you want to use can use native formats of DOS Lemmings, it should be able to handle the custom styles no problem. If it can handle Lemmini styles, you can use the extractor on the first page after compiling your style. If it can't handle either of those (Clones comes to mind), you're on your own sorry. =P

EDIT: Download link removed, check this post for an updated version that fixes a bug with traps.

- I'm not sure exactly how you configure jLevelBuilder to put the right graphic set number in the output file, though. (You could, of course, just replace one of the default sets.)

At the moment, jLevelBuilder deals with official Lemmings sets only.
I'll add a fix for supporting extra sets.   Done, and saving ".ini" files works nicely.
To add new tilesets, simply create a new folder in jLevelBuilder's Lemmings dir, and put your graphics in this folder.
This update will be included in the upcoming next release.



Please note that the ".lvl" file-fomat won't allow the use of extra sets ; only levels saved as ".ini" (Lemmini) can support new tilesets.
In case of saving levels as ".lvl" files, -and as you said-, people can do the trick by replacing one of the default sets.

Well then, it looks like the best option for jLevelBuilder is to overwrite the default sets...

LJLPM, I'd reconsider not supporting the extra sets in LVL files - ESPECIALLY with this tool being available now. I'm not saying go out of your way to make it convenient, but perhaps, if a user attempts to save a non-default graphic set level as a LVL file, require them to manually enter which graphic set number the level uses, and some kind of similar option when loading (of course, telling whether it uses a default or a custom one is hard, if not impossible...)


Question... suppose I had a copy of jLevelBuilder in two seperate folders, could I put the default graphics in one and my custom ones on the other (with the custom ones renamed to the names of the official ones, of course), and each one would work with its respective graphics?

Well then, it looks like the best option for jLevelBuilder is to overwrite the default sets...

LJLPM, I'd reconsider not supporting the extra sets in LVL files - ESPECIALLY with this tool being available now. I'm not saying go out of your way to make it convenient, but perhaps, if a user attempts to save a non-default graphic set level as a LVL file, require them to manually enter which graphic set number the level uses, and some kind of similar option when loading (of course, telling whether it uses a default or a custom one is hard, if not impossible...)

It's not a choice of mine. It's simply the way .lvl files work.

from http://www.camanis.net/lemmings/files/docs/lemmings_lvl_file_format.txt
BYTES 0x001A to 0x001B
Normal Graphic Set: 0x0000 is dirt, 0x0001 is fire, 0x0002 is squasher,
          0x0003 is pillar,0x0004 is crystal, 0x0005 is brick,
          0x0006 is rock, 0x0007 is snow and 0x0008 is bubble.

Because of limited-size storage issues, ".lvl" files handle tileset indexes only (wheareas Lemmini's ".ini" files save tileset names, allowing the use of extra sets).
Assuming I let the user decide which tileset index to use for saving her/his extra set (ie: #3, for the extra-set "Jungle"), then, when the level will be loaded by someone else in Lemmix/Lemmini/... it won't be possible to know it has to load the "Jungle" extra set instead of the expected classic "Pillar" set.
It's not a matter of using jLevelBuilder or not...

Question... suppose I had a copy of jLevelBuilder in two seperate folders, could I put the default graphics in one and my custom ones on the other (with the custom ones renamed to the names of the official ones, of course), and each one would work with its respective graphics?

Yes, sure, you can have different folders with different tilesets!

Yeah, I realise you're restricted to what you can save.
What I mean is, if you're using a custom tileset, and attempt to save a LVL file, the program should ask "What graphic set number do you want to put?" and it'll decide from that what it puts into that byte.
Trust me, I definitely know the level format. "TalveSturges" was one of my millions of past aliases.

(Also - did you add support for set 9 (Xmas)?)

The problem isn't the saving, the problem is the loading when you'll share your level.
"Xmas" set isn't supported atm.

As for loading it, simply allow people to change the set. The person working on it will know what style it's meant to have.

I'd imagine it would be simple to add Xmas in. The program posted in the first post can extract the graphics from the original DOS file (it even generates the INI file).

It is unlikey I'll add support for extra-sets when saving/loading .lvl files (specially since there's already an easy way to handle this by using Lemmini's .ini file-format).
About the Xmas set, I've successfully used your tool + Mindless' to import this set as an "extra-set" in the editor. I'll see later if I take time to handle this new set for .lvl files...

The original DOS game can't load Lemmini-format levels though. There's no point creating a DOS style if you plan to use it with Lemmini.

Here's a video of the one-way fields in action:

http://www.youtube.com/watch?v=Xs1duiR9aU4

By the way: Lemmix handles them perfectly. =D

Sorry, I misread your posts and didn't realize you are trying to get your program to handle compression itself.  If you haven't done so already, please attach the copy of file that is compressed by your program and not working with LemEdit etc. and I'll take a look when I have time.

  Jesus, didn't expect to spend an entire night figuring out what the heck's going on.  But I finally got it.

I'm actually completely astounded that no one has run into this problem before.  Either the people who wrote working compression programs have figured this out before but never told me about it, or we've just been very lucky all this time (possibly because of decreased usage of LemEdit and CustLemm?)

The root cause is that Lemmings is not actually using separate memory buffers for the compressed and decompressed data, but instead is using the same buffer for both.  That is, one block of memory is allocated large enough for the decompresse data, then the compressed data is stored there, and then the decompression effectively transforms those bytes of compressed data into bytes of decompressed data stored in that same place.  It works because:
  1) data is both read and written in "back to front" order during decompression (in fact, now the ordering makes sense in light of using the same memory buffer)
  2) decompressed data is larger (more bytes) than compressed data

So basically as long as you are writing out the decompressed bytes further ahead in memory than you are reading in bytes of compressed data, then the decompression will work correctly even though at the end of it, all the bytes of compressed data will be overwritten with bytes of decompressed data.  Because you are either writing to locations not used to hold any compressed data, or to locations containing compressed data that you've already processed earlier and no longer need anymore.

But the compression algorithm has to take this into account somehow and ensure that during decompression, you never get to a point where the writing out of decompressed data can overtake where you are reading compressed data, as that would corrupt the compressed data that haven't been processed yet, and hence prevent decompression from continuing correctly.

As for loading it, simply allow people to change the set.

Sorry, but if I chose to disable the "tileset" list after having loaded a level, it's because of security against potential bugs (because tilesets don't share the same number of items!) 
So the users who want to use the editor + custom tilesets for making DOS Lemmings's levels (.lvl files), will simply have to use already-existing tileset names for their custom tileset (ie: "special").

The YouTube video is very cool, by the way!  (it's nice to see fresh art for Lemmings)

Fair enough. I can see why it would be a hassle to code for that.

ccexplore: Awesome, thanks. At some point I'll work on recoding to avert that issue. Hm... would I be right in thinking then that the first code (in the stored order - the last code in processing order) would *have* to be a reference one in order for it to work properly, then? Of course, for VGAGR files you could simply add some junk data that's never actually referred to by any objects, but for other files (especially levels) that could be problematic (however, it seems that so far, everything's managed to deal with it).

Also - a thought on the unused bytes in the DAT files, since they're big-endian, isn't it possible that the data sizes are just 4-byte values?

An interesting discovery about terrain. You know how I mentioned that colors that don't set the highest bit will display fine, as will those with a seperate mask? It does seem, however, that if the high bit isn't set for the color, that pixel will be regarded as non-solid by the DOS game (Lemmix does not exhibit this behaviour.)

EDIT: Also - I worked out what the second bit in animation_flags seems to be. If it's set, the object will animate at the start of the level, without being triggered. (If you set both bits on a trap, for example, you'll notice it animates once immediately when the level starts before sitting dormant and waiting for a lemming.) Lemmix crashes with this setup, so you'll have to try it in CustLemm or the original game.

Here's an updated version of LemSet that'll set the flags properly.



EDIT: Removed, see here for an update which now also compresses the files instead of relying on an external tool.

Hm... would I be right in thinking then that the first code (in the stored order - the last code in processing order) would *have* to be a reference one in order for it to work properly, then?

I think the short direct encoding (ie. 00 nnn bytes ...) can actually also work there.  The game's decompression algorithm would've already read the byte that contains the "00 nnn" before it got overwritten, so even though the memory location containing "00 nnn" is already overwritten before processing the final 00nnn encoding, the byte containing those bits was already read earlier and so the algorithm can still pull the correct bits out from the read value (as opposed to having to read it from the memory location where it got overwritten).  Whereas the long direct encoding (ie. 111 nnnnnnnn bytes) will not work as a final processed code, because the "111 nnnnnnnn" cannot fit within one byte--the game would've already read the byte containing "111" but not yet "nnnnnnnn" when both become overwritten from processing the second-to-last code, and thus you lost the nnnnnnnn.

To be fair, it does seem like it could be theoretically possible to create LVL data that a non-aware compression algorithm would use the long direct encoding there.  (Indeed, even several consecutive short direct encodings there would become problematic.)  I will have to do some more testing and investigation to better understand this.

Of course, for VGAGR files you could simply add some junk data that's never actually referred to by any objects, but for other files (especially levels) that could be problematic (however, it seems that so far, everything's managed to deal with it).

The junk data approach is probably the simplest way to handle this for VGAGR files.  I don't believe a general method exists--in theory you could be given basically uncompressible data, such that there'd be no way to encode the data in the compressed format without taking up more bytes than the original data.  Whenever that's the case it'd be impossible for the algorithm to work correctly using the same memory buffer for both.

Also - a thought on the unused bytes in the DAT files, since they're big-endian, isn't it possible that the data sizes are just 4-byte values?

Well yes, the format may have been designed as such, but the DOS game definitely only uses the lower 2 bytes and ignore the upper 2.  It is probably due to the fact that real-mode DOS programs like Lemmings are 16-bit--conceivably the same file format could've been used across other platforms like Amiga, where 32-bit arithmetic might have been natively supported by the CPU and those ports could handle the data sizes as 32-bit values.

One more thing:  for VGAGRx data specifically, it looks like the maximum size of decoded section data that the game can correctly handle is 65535-16 = 65519, rather than 65535 as one might expect.

Here's an update which now does the compression as part of the style building process. You no longer need to use an external tool. =D

EDIT: No need to post a new reply really. Another update; this one improves INI support and changes slightly how the pallete matching works (previously, if your custom pallete included a color that was also in the default pallete, it'd use the default pallete entry instead of the custom pallete one, which would result in the DOS games treating the pixel as non-solid even though it would display correctly). As for the INI - previously it had to adhere to a strict format (no blank lines, line feed at the end of every line including the final one, no comments); it should now be able to load INIs much more effectively. One final addition - you can now specify a number to use in the output filename (ground0o.dat and vgagr0.dat for example), previously it was hardcoded to use "groundXo.dat" and "vgagrX.dat". If no number is specified, it will still use those names as a default.


EDIT: I'm not sure if I've mentioned this anywhere, so next to the download seems the best place to put it.
When desgining terrain graphics, it should be noted that the first 8 colors (the default ones) will NOT register as solid in DOS games - they will display, but they will not act solid. Only the custom 8 colors will function fully. This does not apply to Lemmix; the entire pallete will function as solid there.

Source code can be found here.

To be fair, it does seem like it could be theoretically possible to create LVL data that a non-aware compression algorithm would use the long direct encoding there.  (Indeed, even several consecutive short direct encodings there would become problematic.)  I will have to do some more testing and investigation to better understand this.

Here's an example.  The attached level will choke up on LemEdit (when loaded from a compressed source like a levelpak DAT file) and Lemmings.  In fact, it is possible to manually create this level from scratch in LemEdit, and save it into a levelpak with LemEdit, and still if you try to reload the level from the levelpak in LemEdit it will throw an error.

The level is created so that the initial portion of level data (which contains basically all the level stats like time limit, skillset, etc.) all uses unique values which also never appear anywhere in the rest of level data.  This forces the use of "direct encodings" as the last few encodings in processing order.  For simplicity I purposely kept the level itself is fairly empty (the only element is 1 entrance trapdoor) to help ensure that there is no way to encode the initial portion of level data by referencing bytes from those parts of the data.

Having understood this problem, I believe it is also possible to hack Lemmings to use separate buffers at least for handling levels, to avoid this problem.

I've noted that BMPs saved with some programs (Paint.NET seems to often offend here) will automatically save as 256-color if possible, which LemSet is not compatible with. If you have this problem, open the image and re-save it in MS Paint, making sure to choose 24-bit.

I haven't upgraded this or anything in quite a while, and same goes for the underlying Lemmings Functions libraries (I only ever really completed them to enough of an extent to make LemSet, to be honest).

Now - I will warn you, my coding is. MESSY. AS. HELL.
But if you want to brave it for whatever reason, I'm now releasing the source code for LemSet.
It's under my usual rules for anything I release open-source (or open-resource, as the case more often is). As long as credit is given, do whatever the hell you like with it.
Use QB64 to compile. I've only tried to compile it under Windows, but there are QB64 compilers for Mac, Linux and Android too, if anyone wants to have a go at porting it to those systems.


EDIT: Uploaded the source for dlgfxex too. It's also QB64. It doesn't use the Lemmings Functions library from LemSet or any other external files; everything is contained in the one source file. (It's also ten times messier than LemSet's code...)

It's good for software to have version numbers

I only tend to use them on major projects. Stuff like Lemmings Plus, or my RPG Maker game... these kind of tools when made by me rarely tend to see more than a first release and then one or two bugfix updates. xD

Here's an update! This adds support for making graphic sets with 32-color palletes. Of course, they will only work on engines that also support it, though with careful pallete selection it is possible to create a style that uses extra colors but also works on 16-color engines. It also allows pixel-perfect trigger areas.

I've left the previous version up, just in case anyone wants to be 100% sure of compatibility with 16-color engines (also, because its output is slightly smaller, in case that ever matters).

32-color palletes and pixel-perfect trigger areas require a compatible engine, of course. They will be supported by my updated Lemmix. I'll also release a CustLemmix update to support the 32-color palletes, but since the whole point of CustLemmix is retaining traditional mechanics, the trigger area update won't be applied. (The way I see it, purely-graphical enhancements do not break the traditional mechanics.) EDIT: Updated CustLemmix has been uploaded, check the CustLemmix topic.

Here's another new tool, based mostly on the same code; a tool for editing MAIN.DAT.

http://www.mediafire.com/download/xac3o8fnqo72t3a/MainEditor.zip

Also, use this link for LemSet, rather than the attached files, as that one is slightly outdated:

http://www.mediafire.com/download/sa7nj936o0obfb9/lemset.zip