LemSet - Custom graphic set compiler for DOS Lemmings (Sourcecode released)

[ccexplore]

Not possible. As I said (or at least thought, maybe I forgot to write it), if I take the output VGAGR file from my app, decompress it with Mindless's tools, then recompress it again with the same, it works perfectly - hence the download.

Sorry, I misread your posts and didn't realize you are trying to get your program to handle compression itself.  If you haven't done so already, please attach the copy of file that is compressed by your program and not working with LemEdit etc. and I'll take a look when I have time.

[edit: from reading my documentation, I can only think of one thing so far that could lead to a case where the data decompresses perfectly in Lemmix but leads to an error in LemEdit/Lemmings.  Suppose you calculated the number of bytes in compressed data incorrectly so that it's actually larger than the exact correct value.  This means you effectively have bogus "extra bytes" in your compressed data.  In my documentation I say "you determine when to stop decompressing by keeping track of how many bytes you have decompressed so far".  This leads to the possibility that Lemmix would stop once enough bytes are decompressed, without trying to consume any extra bytes in your compressed data, and thus you get perfectly correct decompressed data.  However, if LemEdit actually goes on based on the number of bytes of compressed data, then it wouldn't stop as soon as enough bytes are decompressed, and would try to go on processing the bogus extra bytes as well, thereby leading to problems.

Just a wild guess for now.  Once I have an actual file hopefully I can understand better.]

[namida]

Here's the result of remaking the 10 official sets from the decompressed data. The program uses its own output (from the decompression algorithm) as source data - however, I am 110% sure there's no problems with the decompression algorithm, only the compression one. I've included the source code as well, in case that's any help in tracking down the error. (It's QB64, which is basically just an extended and cross-platform version of QBasic.)


The code's been worked on a bit since then; at this point, the official games (I've tried both original and CustLemm for this) don't refuse to load, but they do take a few seconds then display really glitchy graphics. LemEdit still crashes; everything else still works fine.

It's coded with QB64 GL Version and I'm compiling it under Windows, not that I'd think those factors would hugely matter. The code is somewhat untidy (though very neat by my standards xD), so just let me know if there's anything you don't understand what it's meant to do. Also - let me know when you've downloaded it so I can remove the attachment, I'd rather not have the source too widely distributed until it's working properly at least.

EDIT: Attachment removed.

[ccexplore]

Downloaded.  I probably won't get around to looking at anything until much later today, but you can remove the attachment anytime now.

[namida]

Just need to do some testing and maybe iron out a few bugs, and the first release should be ready. =)

At this stage, it doesn't compress, for the reasons mentioned above. It'll output the GROUNDxO.DAT file, as well as VGAGRx.DAT.0 and VGAGRx.DAT.1 - the ground file is ready-to-go, the VGAGR files should be run through Mindless's DAT compression tool.

[namida]

Okay, here you go! Enjoy!
It's not the easiest thing to use at this stage, but the important part is: it works, at least.

I attempted to explain how to use it in the included text file, but since I'm terrible at explaining things, as well as that some people can understand examples better than explanations, I also included a sample style with the correct file names and a correctly set up INI file to compile into a graphic set. It's the same tree style that I posted a couple of days ago to demonstrate the tool working - but now compiled entirely from the image files and an INI file, instead of half the information being generated from hardcoded values. =D

If you need any help with how to work it, just ask. =) Runs on Windows. If anyone's wanting to run it on Linux, try WINE first, if that doesn't work, I'll see if I can compile a native Linux version. (There is a QB64 version for Linux, and from what I've heard code ports VERY well from one platform to another in QB64, so...)

Remember - you need to use Mindless's tools to compress the two sections into one VGAGR file at this stage, the app doesn't have a working compression routine yet. To save you having to search, I included a copy of the tools in this archive. (Of course, if Mindless would prefer I didn't redistribute them, I'm happy to remove it and provide a link instead - I just didn't want to risk ever having a case where people have copies of the app but Mindless's tools are no longer possible to find, thus making the app useless.)

It'll probably be a while before I post any further updates, because:
1) I should be studying for exams, not working on Lemmings projects, at the moment.
2) It works, even if it's not the friendliest app.



EDIT: For creating levels with your custom styles, you can either:
- Use LemEdit. Once the VGAGR is compressed, you can use the custom style in LemEdit directly.
- Use Lemmix. Once the VGAGR is compressed, you can use the custom style in Lemmix directly.
- Use jLevelBuilder. After compiling the set, use the extraction tool in the first post. (You do not need to compress the VGAGRx.DAT then decompress it again - just use the files LemSet generates directly with the extraction tool!) I'm not sure exactly how you configure jLevelBuilder to put the right graphic set number in the output file, though. (You could, of course, just replace one of the default sets.)
- Possibly Lix? I haven't tried it myself so I don't know what it does or doesn't support or what formats it expects; unfortunately, Lix doesn't seem to like my PC.

In general - if the editor you want to use can use native formats of DOS Lemmings, it should be able to handle the custom styles no problem. If it can handle Lemmini styles, you can use the extractor on the first page after compiling your style. If it can't handle either of those (Clones comes to mind), you're on your own sorry. =P

EDIT: Download link removed, check http://www.lemmingsforums.com/index.php?topic=860.msg17741#msg17741" class="bbc_link" target="_blank">this post for an updated version that fixes a bug with traps.

[LJLPM]

- I'm not sure exactly how you configure jLevelBuilder to put the right graphic set number in the output file, though. (You could, of course, just replace one of the default sets.)

At the moment, jLevelBuilder deals with official Lemmings sets only.
I'll add a fix for supporting extra sets.   Done, and saving ".ini" files works nicely.
To add new tilesets, simply create a new folder in jLevelBuilder's Lemmings dir, and put your graphics in this folder.
This update will be included in the upcoming next release.

http://imageshack.us/photo/my-images/266/pb78.png/" class="bbc_link" target="_blank">http://img266.imageshack.us/img266/494/pb78.png" alt="" class="bbc_img" />

Please note that the ".lvl" file-fomat won't allow the use of extra sets ; only levels saved as ".ini" (Lemmini) can support new tilesets.
In case of saving levels as ".lvl" files, -and as you said-, people can do the trick by replacing one of the default sets.

[namida]

Well then, it looks like the best option for jLevelBuilder is to overwrite the default sets...

LJLPM, I'd reconsider not supporting the extra sets in LVL files - ESPECIALLY with this tool being available now. I'm not saying go out of your way to make it convenient, but perhaps, if a user attempts to save a non-default graphic set level as a LVL file, require them to manually enter which graphic set number the level uses, and some kind of similar option when loading (of course, telling whether it uses a default or a custom one is hard, if not impossible...)


Question... suppose I had a copy of jLevelBuilder in two seperate folders, could I put the default graphics in one and my custom ones on the other (with the custom ones renamed to the names of the official ones, of course), and each one would work with its respective graphics?

[LJLPM]

Well then, it looks like the best option for jLevelBuilder is to overwrite the default sets...

LJLPM, I'd reconsider not supporting the extra sets in LVL files - ESPECIALLY with this tool being available now. I'm not saying go out of your way to make it convenient, but perhaps, if a user attempts to save a non-default graphic set level as a LVL file, require them to manually enter which graphic set number the level uses, and some kind of similar option when loading (of course, telling whether it uses a default or a custom one is hard, if not impossible...)

It's not a choice of mine. It's simply the way .lvl files work.

from http://www.camanis.net/lemmings/files/docs/lemmings_lvl_file_format.txt" class="bbc_link" target="_blank">http://www.camanis.net/lemmings/files/docs/lemmings_lvl_file_format.txt
BYTES 0x001A to 0x001B
Normal Graphic Set: 0x0000 is dirt, 0x0001 is fire, 0x0002 is squasher,
          0x0003 is pillar,0x0004 is crystal, 0x0005 is brick,
          0x0006 is rock, 0x0007 is snow and 0x0008 is bubble.

Because of limited-size storage issues, ".lvl" files handle tileset indexes only (wheareas Lemmini's ".ini" files save tileset names, allowing the use of extra sets).
Assuming I let the user decide which tileset index to use for saving her/his extra set (ie: #3, for the extra-set "Jungle"), then, when the level will be loaded by someone else in Lemmix/Lemmini/... it won't be possible to know it has to load the "Jungle" extra set instead of the expected classic "Pillar" set.
It's not a matter of using jLevelBuilder or not...

Question... suppose I had a copy of jLevelBuilder in two seperate folders, could I put the default graphics in one and my custom ones on the other (with the custom ones renamed to the names of the official ones, of course), and each one would work with its respective graphics?

Yes, sure, you can have different folders with different tilesets!

[namida]

Yeah, I realise you're restricted to what you can save.
What I mean is, if you're using a custom tileset, and attempt to save a LVL file, the program should ask "What graphic set number do you want to put?" and it'll decide from that what it puts into that byte.
Trust me, I definitely know the level format. "TalveSturges" was one of my millions of past aliases.

(Also - did you add support for set 9 (Xmas)?)

[LJLPM]

The problem isn't the saving, the problem is the loading when you'll share your level.
"Xmas" set isn't supported atm.

[namida]

As for loading it, simply allow people to change the set. The person working on it will know what style it's meant to have.

I'd imagine it would be simple to add Xmas in. The program posted in the first post can extract the graphics from the original DOS file (it even generates the INI file).

[LJLPM]

It is unlikey I'll add support for extra-sets when saving/loading .lvl files (specially since there's already an easy way to handle this by using Lemmini's .ini file-format).
About the Xmas set, I've successfully used your tool + Mindless' to import this set as an "extra-set" in the editor. I'll see later if I take time to handle this new set for .lvl files...

[namida]

The original DOS game can't load Lemmini-format levels though. There's no point creating a DOS style if you plan to use it with Lemmini.

[namida]

Here's a video of the one-way fields in action:

http://www.youtube.com/watch?v=Xs1duiR9aU4" class="bbc_link" target="_blank">http://www.youtube.com/watch?v=Xs1duiR9aU4

By the way: Lemmix handles them perfectly. =D

[ccexplore]

Sorry, I misread your posts and didn't realize you are trying to get your program to handle compression itself.  If you haven't done so already, please attach the copy of file that is compressed by your program and not working with LemEdit etc. and I'll take a look when I have time.

 http://www.lemmingsforums.com/Smileys/lemmings/XD.gif" alt=":XD:" title="XD" class="smiley" /> Jesus, didn't expect to spend an entire night figuring out what the heck's going on.  But I finally got it.

I'm actually completely astounded http://www.lemmingsforums.com/Smileys/lemmings/shocked.gif" alt=":o" title="Shocked" class="smiley" /> that no one has run into this problem before.  Either the people who wrote working compression programs have figured this out before but never told me about it, or we've just been very lucky all this time (possibly because of decreased usage of LemEdit and CustLemm?)

The root cause is that Lemmings is not actually using separate memory buffers for the compressed and decompressed data, but instead is using the same buffer for both.  That is, one block of memory is allocated large enough for the decompresse data, then the compressed data is stored there, and then the decompression effectively transforms those bytes of compressed data into bytes of decompressed data stored in that same place.  It works because:
  1) data is both read and written in "back to front" order during decompression (in fact, now the ordering makes sense in light of using the same memory buffer)
  2) decompressed data is larger (more bytes) than compressed data

So basically as long as you are writing out the decompressed bytes further ahead in memory than you are reading in bytes of compressed data, then the decompression will work correctly even though at the end of it, all the bytes of compressed data will be overwritten with bytes of decompressed data.  Because you are either writing to locations not used to hold any compressed data, or to locations containing compressed data that you've already processed earlier and no longer need anymore.

But the compression algorithm has to take this into account somehow and ensure that during decompression, you never get to a point where the writing out of decompressed data can overtake where you are reading compressed data, as that would corrupt the compressed data that haven't been processed yet, and hence prevent decompression from continuing correctly.