Graphic set editor (V1.01 Update released)

[namida]

Current version: 1.01

Download
Source Code (Delphi 7)

You can create a VGASPEC file with this too. Just create a graphic set with a single terrain piece (that being the level graphic) and no objects, and save it with a filename that starts with "x_". If you add a second terrain piece, any solid pixel in it will become steel (if the level its used in has autosteel enabled). Likewise, a third piece if present, any solid pixel will act the same as a regular terrain piece with the "One Way" option selected (it still pays attention to the One Way Invert flag as usual).



So, while I'd take it partly the lack of graphic set creation is simply due to lack of artistic talent (or at least, the perception of a lack of it) and/or willingness, I figure that making a better graphic set tool than LemSet is probably a worthwhile endeavour.

So my question is - how exactly should it look and function? Should it be integrated into NeoLemmix Editor, or a seperate program? (My preference is the latter, personally.)

I've been working on something, though it's still early days yet - at the moment this is practically just a mock-up (albeit designed in the Delphi IDE, so I can easily make it functional); the only thing that actually does anything so far is the "Black background" checkbox, which does more or less exactly what it says on the tin.

Any comments, suggestions, etc are welcome.

More likely than not, this tool will be made fully open-source as soon as it's complete, so if anyone wants to add support for other engines, they'll be able to do so.


For reference;
The File menu just contains your standard "New", "Load", "Save", "Exit" options.
The Tools menu will have, among other stuff, the "Import Cheapo style" option.



Up to date list of target import/outport formats - struck out ones have already been implemented (though a version with support implemented may not nessecerially have been released yet).

Target Import Formats
First version
> LemSet graphic set source "code"
> GSTool-specific INI / PNG source "code" (derivative of LemSet format)
> Cheapo Style Extractor output <removed in V0.99>
> DOS / Lemmix graphic sets
> NeoLemmix graphic sets
Eventual
> New NeoLemmix graphic set format
> VGASPEC files
> Lemmini graphic sets
> SuperLemmini graphic sets
> Cheapo graphic sets (directly from STY file)
May consider if enough interest
> WinLemm graphic sets
> LJLPM's Lemmings clone (subject to LJLPM being okay with this)

Target Export Formats
First Release
> Old-Format NeoLemmix Graphic Sets <removed in V1.01>
> LemSet graphic set source "code" <removed in V1.01>
> GSTool-specific INI / PNG source "code"
Eventual
> New NeoLemmix graphic set format
> SuperLemmini graphic sets
May consider if enough interest
> LJLPM's Lemmings clone (subject to LJLPM being okay with this)

Other Features
First Release
> Viewing all properties of pieces
> Editing all properties of pieces
> Adding and deleting pieces
> Viewing graphic set palette (still used for brick/minimap color and explosion particles even in RGB-based sets)
> Editing graphic set palette
> Viewing all frames of objects
> Adding and deleting frames from objects
> Importing individual frames from PNG files
> Exporting individual frames to PNG files
> Displaying trigger area on object image
> Reordering pieces
Eventual
> Importing individual frames from BMP / GIF files
> Exporting individual frames to BMP files
> Importing entire object graphics as a strip of frames in PNG format
> Importing entire object graphics as a strip of frames in BMP / GIF formats
> Exporting entire object graphics as a strip of frames in PNG format
> Exporting entire object graphics as a strip of frames in BMP format
> Support for storing trap/teleporter/etc sound effects in a graphic set
> Support for properly managing the sound effects stored in a graphic set
> More user-friendly and flexible managing of custom sound effects in a graphic set
May consider if enough interest
> feel free to leave suggestions!

Why no BMP in first release? Because I'd have to add a way to select transparent colors, as while BMP-with-transparency does exist, it's extremely obscure. PNG on the other hand supports it as standard. Also, because supporting two formats is trickier than supporting one, so my main goal is to just have it working first.

Why no export to GIF? Because GIF is limited to 256 colors, and it's very plausible some graphic sets may use more than this, especially with 24-bit color ranges. Why support importing it then? Because Lemmini graphic sets use it and I plan to support importing them, and it's very little extra work to add importing other images from it. Besides, even though it's never used (and as such probably won't be supported for it in the first release), technically LemSet does support GIF files too.

[GigaLem]

nice someday i can get to work on SonicLems when the time comes

[namida]

A somewhat related question - how do you feel about replacing the graphic set format with a new one, like was done a while ago with the level format?

Primary advantages would be support for full 24-bit color instead of just 18-bit, better futureproofing for new features (there's basically *no* free space left in the existing format), and the ability to have a single file instead of two files for a graphic set. Also, better support for choosing a preview frame - currently, any object where the graphic is outside the first 64KB of the object section of the VGAGR file, can only have the first frame as the preview frame.
EDIT: Also, no longer limited to 32 objects and 128 terrain pieces per graphic set. Due to the level format (though this could also be changed if nessecary), the limit would be 256 of each; if the level format were indeed to be changed to accomodate more pieces, the only limitation for each would be how many pieces you could fit the graphics for into 4GB, so realistic filesizes would become an issue long before any file format limitation does. Another additional advantage is pieces would no longer be limited to 248x248 in size, though I doubt larger pieces would see very frequent use. Nor would their width have to be multiples of 8, though again, this is a minor detail.

Primary disadvantage would be even less compatibility with existing software - current format, if loaded even in software without NeoLemmix support, will usually at least give something vaguely resembling the intended graphic, provided that there's less than 64KB (uncompressed size) each of terrain and object data - especially so for 4-bit or 5-bit graphic sets; the former should graphically turn out perfect (though functionality may be off), and the latter is still usually pretty close. Of course, the NeoLemmix Editor would be updated to support it. Beyond this, I can't really think of any disadvantage to it. (It goes without saying that of course, the format would be documented on the NeoLemmix website.)

Backwards compatibility for existing format graphic sets would most likely be retained, although all existing graphic sets would be included (in the new format) with future updates. Thus the primary motivation for keeping existing format support would be on the offchance that someone actually makes a custom graphic set for traditional Lemmix (or for some reason really wants to use the non-remastered versions of the LPII graphic sets, I guess).


The new format would most likely be a three-section compressed DAT file, with one section for metadata (this would be roughly the same as current GROUNDxO.DAT files), and one section each for terrain and object graphics. Although strictly speaking the latter two don't have to be in two seperate sections; they could very plausibly be combined into one. The actual format of the graphics would likely be some kind of compressed non-planar bitmap format (in addition to the DAT compression layer), something fairly simple to implement but also suited towards the kind of graphics typical in a Lemmings graphic set. (I've trialled one such scheme - though I haven't experimented much with tweaking it yet - and filesizes of the resulting graphics files are similar to those of existing VGAGR files, despite the extra 6 bits of color depth.)

[namida]

Progress: So far, it loads the metadata and terrain images (haven't yet added objects) of graphic sets (in existing DOS / Lemmix / NeoLemmix) formats fine for 4-bit and 5-bit graphic sets. It doesn't work with the 19-bit (RGB-based) ones properly yet; while it gets the correct shapes, the colors are way off. (EDIT: Fixed. Turns out it was the kind of stupid oversight I'm infamous for; in this case, I was bitwise shifting the values left by far more than they should be.) The code to do this was recreated from scratch rather than re-using NeoLemmix (or its editor's) code as the code in those is quite a mess by this point, as well as that they load to an internal "format" that's somewhat different from the one I'm using for this app.

Also, interestingly, when I dump these images to BMP files (since I haven't given it any way to display them in-app yet), it seems to be saving an alpha channel! I never knew BMP format supported that; even Paint.NET can't save BMP with an alpha channel (yet can load it).



I've had to (at least for now) remove the "black background" checkbox. While this can be made into an option, it'll be a fair bit of work; it's something I can look into once the core features are working.

In more positive news, it's now capable of fully loading a graphic set from GROUNDxO / VGAGRx (or G_xxxx / V_xxxx; they're identical anyway except for filenaming scheme) files and displaying the data in-editor. It doesn't yet filter out options that aren't relevant to a selected object type; eg: key frame could be chosen for an exit. Such "smart" features are coming later. (Also, the "Show trigger areas" isn't functional yet.)

(If anyone's wondering about the "Resolution" option, no, I'm not planning to add higher resolutions to NeoLemmix. Rather, it's so that if, for example, you import a Lemmini graphic set, it'll know it has to halve the size of everything.)

EDIT: Okay, some smart detection of which features are relevant or not has now been implemented. For example, it won't enable the secondary trigger area options on anything other than Two-Way Teleporters or Single-Object Teleporters (since nothing else uses them). It also won't enable the selection of triggered / constant on anything other than exits, as all other types of objects always expect a specific selection for this; exits are the only one where both options are supported.

Next, I'll probably work on importing other types of graphic sets - planned other formats to support for import are extracted Cheapo styles, LemSet source files, and possibly Lemmini / SuperLemmini styles. I plan to have all of these (except maybe Lemmini / SuperLemmini) supported in the first release.

[namida]

Starting work on the Cheapo import now. This should be a LOT simpler than it was when adding to LemSet, firstly because most (heck, probably all) the bugs in the Style Extractor have been ironed out by now, and secondly because there are XML parser libraries available for Delphi, so unlike with LemSet (which was written in QB64) I don't have to write an XML parser; I just have to write code that makes use of the parsed XML data, which is the much easier side.

At first it'll just support importing from extracted styles (via Essman's style extractor), but if it's not too difficult to do, I very well may look into directly importing from STY files.


EDIT: Terrain is now loading perfectly fine; haven't coded it to load objects yet.

EDIT: Cheapo graphic sets are now fully loadable. (Extracted ones that is, not directly from the STY file. Direct STY file importing is something that I probably will wait for a later version to include.)

EDIT: Started work on importing LemSet (uncompiled) format now. Convenience aside, an advantage of supporting importing LemSet source files directly rather than compiling them in LemSet then importing the DAT files, is that if the images are in 24-bit color, it'll import them as such, whereas LemSet downgrades to 18-bit (due to current file format limitations) while compiling. Also, it'll be able to export to this format in both fully LemSet-compatible setups, and an "extended" setup that supports new properties LemSet didn't, thus those who prefer to work with INI and image files can edit the graphic sets that way then just use the tool to rebuild them.

EDIT: Terrains are correctly loaded at least from RGB-based LemSet sources (whether in 18-bit mode or 24-bit mode); still need to try 4-bit and 5-bit ones.

EDIT: Seems they're fine. Just for reference, testing so far has been based on the Dirt set (both DOS and NeoLemmix versions), the Tree set from LPII (traditional version), the Circuit set from LPIII, the Galaxy set from Cheapo, and the Horror set from Doomsday Lemmings. The choices were based on the dirt set being a "default" as well as being one that exists in both untouched and NeoLemmix-ised versions, the Tree and Circuit sets just being convenient and somewhat varied options from their respective games (with one being 4-bit and one being 5-bit), the Galaxy set having a large variety of object types as well as both rectangular and circular trigger areas, and the Horror set being pretty much the only (non-converted-from-Cheapo) RGB-based graphic set so far.

EDIT: LemSet importing is now 100% supported! Next feature I'm going to work on is saving to LemSet format (both a strictly LemSet-compatible format, and a slightly extended one which will save some extra properties that this tool can handle but LemSet can't - the latter is higher priority, but the differences should be so small that a few simple IF branches are all that's needed to differentiate them).

EDIT: A slight change on a previous decision. Due to the relative ease of doing so, most likely, saving in SuperLemmini format will also be supported (though not nessecerially in the first release) for any graphic set that doesn't make use of NeoLemmix-exclusive object types. I may also consider Lix if there's enough demand for it and not too many extra features need to be implemented to support it. I don't intend to support output for any older engines.

EDIT: LemSet-like format exporting successful. I tested an import and an export of a graphic set (importing it also from LemSet file), and the only differences in the INI file were the addition of a few lines specific to this tool (such as one to mark that the images already contain transparency information and don't need to be patched according to a defined transparent color), and that a very small number of values that were absent (and thus LemSet would use default values) are instead explicitly stated, though with values that match the default ones LemSet would've used.

EDIT: Tested again with the "LemSet Compatibility Mode" (which basically generates 100% LemSet-compatible output), the images (apart from the palette image) are identical to the input ones (apart from being in PNG format rather than BMP), the INI file is identical apart from some values being explicitly specified instead of implicitly using defaults. No additional tool-specific lines. So at this point, the tool can display properties (and the first frame) of pieces, as well as import graphic sets from either compiled GROUNDxO/VGAGRx files, extracted Cheapo styles, or LemSet source files; and export graphic sets to LemSet source files in either fully LemSet-compatible mode or an "extended" mode that contains extra information beyond what LemSet can handle (but within what this tool can). The one missing feature is that it doesn't yet preserve any palette entries other than the one used for the builder bricks / minimap; the other ones are still used in one place in NeoLemmix, namely the explosion particles from bombers / stoners.
Nothing is actually editable yet, so at this point it's just a convertor / viewer, but at this pace it won't be long before it can truly be called an editor.

[Essman]

At first it'll just support importing from extracted styles (via Essman's style extractor), but if it's not too difficult to do, I very well may look into directly importing from STY files.

If you need the details on how to load STY binary files, the code to do it is in the style extractor:
https://github.com/spadapet/cheapstyle/blob/master/CheapStyle/Style.cs

[namida]

So, I've writtten the code to export GROUNDxO / VGAGRx (NeoLemmix-format; they're not DOS compatible. They should be compatibile with the recent updated versions of traditional Lemmix if you manually stick within its limitations such as trigger areas divisible by 4, but this is not an officially supported feature) files. And tested it.

The results?

Compiled from the same LemSet source as a graphic set that normally turns out to be about 80KB, and takes LemSet about 20 - 30 seconds to process...
It creates one only 59KB, at a speed that's virtually instantaneous. The DAT compression/decompression algorithm (which accounts for about 70% of LemSet's processing time, and LemSet's DAT code is also well-established to be very inefficient filesize-wise) is directly copied from the NeoLemmix Editor, but the graphics format conversion is completely new code.

With an EXE that, even despite having a GUI, and several features that LemSet doesn't have, is only about a third of the size. Guess I should probably also make a Delphi equivalent of LemMain once this tool's completed to a decent extent, since LemMain suffers from speed and output filesize issues far more than LemSet does...



For my use, LemSet is now completely redundant, but until it actually supports editing (rather than just importing and exporting specific formats, even if LemSet source "code" is one such format meaning you can already create graphic sets with it), I don't feel it's ready for release. Still, it's coming along very nicely, is the main point I'm making here.

By the way, what do you guys think about this icon? (obviously, attached image is magnified)



All properties of existing terrain and objects can now be changed, although their graphics can't yet (except by exporting an INI/PNG version of the graphic set, modifying it there, and reimporting). Also found one minor bug in the Ground/VGAGR exporting, and one major bug in the INI/PNG exporting, which are now both fixed.

[namida]

So, I decided to make the object/terrain display a bit tidier.

Now, it centers it in its display area (rather than snapping to the top-left corner of it), and also automatically resizes it. It won't give oddly-proportioned resizes that distort the image or even make it unclear; resizes are always proportional. When downsizing (extremely rare) it always does so by factors of 2 (eg, half size, or quarter size, etc), while when upsizing it always does so by whole number multiples of the original size (so it could go to double, or triple, or quadruple the original size, but never, say, 2.5x original size).

Trigger area display is now also semi-working; semi in that I still need to make it blend with the original image rather than blatantly overwrite it.

At this point, everything you see here is functional, except the "Add" / "Delete" buttons under the terrain/object lists, the "Use Current" buttons next to preview/key frame inputs, and the 6 buttons directly underneath the preview image.

(The loaded graphic set in the screenshot is the Circuit style from LPIII, if anyone was wondering.)

Another kinda neat feature I added: When saving (or loading) Ground/VGAGR pair files, obviously, this means two files. While it still gives you full control over what the input/output file names are, if it can recognize the first filename as either "groundXo.dat" or "g_xxxx.dat", the second save dialog will have "vgagrX.dat" or "v_xxxx.dat" as the default name.



EDIT: Hey, look in that second screenshot. Look what option not only mysteriously reappeared on the panel, but now works too!

Also, now, everything works except for the add/delete buttons (for both frames and pieces) and the Import/Export graphics buttons (ie: you *can* navigate through frames now, and use the "Current Frame" buttons for preview / keyframes).

I've also changed the displayed (and input) frame numbering to be 1-based instead of 0-based. This may slightly annoy the more tech-junkie types (including myself), but it's a minor thing and will probably make a lot more sense to the average user.

[namida]

I decided to make the pieces list a bit more descriptive...

Remaining features before a first release are just adding/removing frames from objects, and reordering pieces.

[namida]

Okay, well, it's at a state where I feel like I can release something now. Please be advised that (as you can tell) this is a very new tool.

Do not try and do stuff you're not meant to do (such as open invalid files), unless you're intentionally doing it to test for issues and don't have anything important open at the time.
Keep regular backups of anything you're working on with it. The best format to use for the backup is the Full INI/PNG format (Export -> INI / PNG -> Full).
Do not try to open any graphic set format that's not yet supported. Those that *are* supported are LemSet source files (and this tool's own extended version of such), DOS / Lemmix / NeoLemmix graphic sets, and extracted Cheapo graphic sets. Lemmini graphic sets are NOT supported yet,  though they almost certianly will be in a near-future update.
If you encounter any issues, especially if you know exactly how to reproduce them, report them to me. Unless there's a very good reason not to, please report them in this topic, not via PM / IRC / etc - it's easier to keep track of them this way.
Expect issues. This is the first time I've seriously attempted to write a GUI application from scratch*, and as such, there's bound to be problems somewhere that I haven't noticed.
When using the "Import" and "Export" buttons to load/save graphics for terrain pieces or object animation frames, ONLY PNG IS SUPPORTED so far. Support for BMP (and probably GIF) will come later.
Most importantly of all, DO NOT ATTEMPT TO EAT IT.

https://www.dropbox.com/s/lbd48x520cqu08h/GSTool_0.7_Beta.zip

Source code (Delphi 7) available on request; I'll publicly publish source code (rather than the on-request) system once more of the core features (such as other formats) are added, and I've tidied up some of the messier parts. ...or more likely, just added the features, and not tidied up the messiness.

* Unless you count NeoLemEdit, but that wasn't using true GUI components as such, and ran on an underlying system intended for games, not applications. And if you've checked the source code, you'd know it's one of the kludgiest things in history. xD

[Wafflem]

That's a really great tool! It is indeed far more user-friendly than LemSet (though I haven't even tried LemSet at all).

This is not for now, but in the future, is it okay if I use this tool to add objects to the LP styles (e.g. the slowfreeze from the Crystal style into the Sky style)?

[namida]

Of course! The Lemmings Plus styles (and the Horror style) are there on a "do whatever you like with them" basis. This is generally true of any content I make, apart from levels, which I prefer not to be re-used in any way while they're still new (but am fine with it once they're a bit older).

That being said, if you're distributing the style files with NeoCustLemmix packs (rather than Flexi-based packs), you might want to use unique names/numbers for them rather than replacing the existing ones, as a matter of convenience for anyone playing it. You can either give them an unused number, or use a naming scheme similar to:

ground15o.dat > g_dynalem_sky.dat
vgagr15.dat > v_dynalem_sky.dat

If doing so, note that the name of the style (the part between the g_ / v_ and the .dat extension) cannot be longer than 16 characters, and cannot start or end with a space (but it can contain spaces in the middle). You should also avoid any character not in the ASCII character set - if you're not sure what that means, just stick to numbers, letters and common punctuation(eg: spaces, _, -, etc), and you'll be fine. You should also avoid naming them the same as any current graphic set's internal name (in most cases their internal name is the same as their listed name in the NeoLemmix Editor; the one exception is that the Christmas style's internal name is "xmas").

Both the player and the editor can recognize these kind of filenames.



For this task of moving objects between graphic sets, while definitely already possible, it's not very convenient to do so at the moment. Needless to say, it'll be made far more convenient in future updates. Also, there's no need to worry about palette issues, as this tool always saves graphic sets as RGB-based ones (it can most certianly load non-RGB ones, of course).

[namida]

As far as the possible new format goes - I've yet to write a loading routine, which also means as of yet it's a tad hard to tell whether the save routine saved the data accurately or completely botched it up; but in terms of filesize, it's showing a lot of promise. I've tested two graphic sets - the Horror set, and one other (which I can't reveal too much about at the moment but I will say it has a VERY large variety of colors in it), both of which are 18-bit graphic sets in existing format, and I have 24-bit source image files for. In the case of the Horror set, results for both the old and new format (both compiled with this tool, rather than going off the existing LemSet-made version) are roughly the same filesize; while for the other graphic set, it's actually ~15KB smaller despite the extra 6 pixels of image depth. And, the time taken (on my PC) from the point at which the "Save" button (on the file selector popup) is hit, to the time at which saving is complete, is about 1.1 seconds for the Horror set, and 0.7 second for the other one.

Indeed, this new format removes the limitation of piece widths having to be divisible by 8. It also theoretically allows an unlimited number of pieces (provided the graphics for them don't exceed 4GB in size after encoding but prior to DAT compression, which realistically, it's unlikely that limit will be hit by any reasonable graphic set), though in practice this would currently be limited to 256 terrains + 256 objects per graphic set due to the level format.

Now, here's hoping it's actually saving the accurate images and not a load of garbage... time to write a loading routine and find out!

EDIT: Although it works, it's not actually giving a significant filesize improvement over just storing a plain ol' ARGB bitmap image, so I'm going to change it to just use such, but with a shortcut of not storing the RGB values if the A value is zero. On the other hand, the metadata format is very nice.


(This indeed technically means that the graphic set format uses 32-bit images, although currently there are no plans for NeoLemmix to support partial transparency; the format's capacity for it is mainly just future-proofing.)

[namida]

Alright so, the tool has fully implemented the format, the NeoLemmix player can now load the format, and as such, and we now have 24-bit color graphic sets at last! Or 25 bit if you count the alpha channel. (The format actually stores a full 8-bit alpha channel, but NeoLemmix doesn't support this yet; all non-zero alpha values are treated as 255.) Still left to do is support loading the new format when embedded in the EXE; currently it can only be loaded from an external file, but this should be very easy to do.

Here's a screenshot showing it. And yes... you are right if you think you've never seen this graphic set before.

[Wafflem]

Sweet! (pun intended )

I'm assuming this will be part of a new small pack you're creating, similar to Doomsday Lemmings?