2.00 Tree with text/image files > custom formats (Rant!)

[Simon]

I have made several attempts to push this. There is at least one thread exactly on this issue, but people haven't joined that discussion much back then.

But this is super important. More people than back then have been using the old setup, and know the problems. I want to drive this point home once and for all.

Tree of raw images and accessible control files = GOOD.
Custom formats for tiles or graphic sets = BAD.

I don't care if the control files have the same keywords as Lix files. I don't care whether one animation goes in a collective image, or in many images. I do care that every single file is a standard, worldwide-known format. I do care that data of different types (header values, images) be not merged in a container. They must be separate files that the host program will interpret together.

This has a ton of advantages:

People can use the tools they like on the files. Everybody is adept at different tools, but they all get the job done well. Even if you provide the best tools you can design for blob formats, you lose a million other tools. You forsake hard-earned experience. You lose scriptable tooling and shell interfaces. Well yeah, some Windows users don't care, and would happily work on tasks in a crazily unscalable way. :> Everybody else will immediately rage.

People don't want to learn a new tool; they want to get ideas from brain to usable file ASAP. And the most usable file is a simple image. It's the job of the host program to chew that into whatever bytes the game likes. It's not the designer's job to tear their perfectly well-represented image into whatever this year's NL version happens to like.

And no, it's not enough to have them make their images first, then later force creators to mass-convert content with a special tool. Omit the need for that conversion.

If the control files are getting complex, you can still offer your own tool to work on the data. People don't have to use it, but they can fall back to it.

Even the most stupid text file is admissible to search and replace, grep, sed, etc.; binary files aren't.

People can collaborate, and send each other good, descriptive patches. This is super important. If one tile needs work, that exact file can be fished from the tree, edited, and sent to the set maintainer.

The current NL culture requires you to be the very maintainer in the first place, before you can even think of making a patch! I would have sent Ichotolot some fixed one-way arrows for the castle tileset, had it been possible. Icho would have merged the patch. Icho doesn't care about the arrow's movement direction to make the patch himself!

(This phenomenon is prevailing in open source. Send in a patch, it gets merged, everybody is happy instantly. But put it on the bugtracker instead, and you have to hope that someone else cares enough to fix that particular thing, in a reasonable time, and in a reasonable way.)

The often-referenced "Neolemmix user" is not our lovely auntie who doesn't use a computer much, but is strangely eager to play trans-ONML-difficulty user packs. The typical user is reasonably computer-literate, knows many non-Lemmings games, and is often a level-designer themself. They have used other programs! They know better than to delete random files from a tree, and then complain that "it doesn't work".

Graphic-sets are archaic and damaging. I have to make a precise definition here. A graphic-set, of which usually only one is selected per level, is a subclass of all possible tiles, and only tiles from the chosen set may be used in a level.

Graphic-sets in NL are fundamentally different from what counts as a graphic set in Lix. A Lix tileset is a certain subdirectory in the tree. It is separated by its path from other sets. The game doesn't care if all tiles come from somewhere particular in the tree. It's perfect, because it matches the user's mental model of a tileset, and doesn't impose any restrictions during level building.

Graphic sets have corresponded to the users' ideas of L1, because all users were players. No user was a level designer in L1 when it came out in 1991!

In a time where people are both designers and players, a blob format per graphic set completely obscures what everything is. A graphic set, by this very name already, claims to be a set of graphics. Therefore, it should be a set of graphics, readily accessible on the disk. It should not be some untouchable blob that magically pops out tiles when the host program eats it.

Likewise, a gadget is a special tile: a graphic plus some meta-information. Put the graphic in a graphics file, readily accessible. Put the header info in a sepearate file, readily accessible as text, or as an image mask.

And obviously, graphic tiles should be referenced by their path name, or maybe by hash value. Never with a number that is dependent on a set. Don't even form the idea that for tileset mix-and-match, you could number sets, and number graphics in each set. Distributed content development is incompatible with numbering, plain and simple.

With this representation of single tiles, which is compellingly the most adequate representation of a single tile, it's apparent that a restriction per level to subclasses of tiles is arbirtrary and ill-motivated.

People have undergone needless chores for several days to work around the limitations in current NL. There's the stupidly named Epic tileset, it's simply a union of all 9 or 10 tilesets from L1/ONML tilesets. Giga has put the 500-something tiles of namida's various tileset into one union set. All by hand. No automation possible.

Ichotolot has meticulously put together L2 graphic-sets for NL from loose images. He has made good use of some programs (eidcut, eidrecol) that I wrote years ago to batch-process tilesets represented as images, and that made it tremendously easier for him. These programs were originally written to make Lix tilesets -- but because Lix only takes popular formats, the tools were immediately useful for his non-Lix purpose.

Despite existence of the Epic tileset, or similar union sets, they're rarely used. Being able to mix-and-match has not lead to only mixed-and-matched levels, not at all. Good taste doesn't go out of style.

Don't be working hard to do stupid things! Leverage the capabilities of a well-designed host program!

For releasing a level pack, it's not hard to program a checker that goes through all levels, and makes sure all necessary tiles from the tree are released with the pack. This is of the same difficulty as going through all levels, and checking what currently-existing graphic-sets are used.

People can use version control on their files. This ties in with the collaboration point; people can make the tool visualize what has been changed and how. Formatting patches becomes straightforward. But it is a sane practise for any project, programming, level design, graphics design, etc. (Chains are git is easy and all the cool kids are doing it.)

In particular, if you don't want to put your tiles under version control, you don't have to. All you have is a tree of images, which you can handle with whatever tools you please. That's the point.

Worldwide-known standard formats also compress better than whatever either of us could cook up. In case libpng is too hard to use directly, there are other libs for handling PNG.

To quote the Kernel coding style:

[Typedefs should only be used for] (a) totally opaque objects (where the typedef is actively used to _hide_ what the object is). Example: "pte_t" etc. opaque objects that you can only access using the proper accessor functions [= Neolemmix-specific tools, here]. NOTE! Opaqueness and "accessor functions" are not good in themselves. The reason we have them for things like pte_t etc. is that there really is absolutely _zero_ portably accessible information there.

Images and control files are super portable, there is no reason to make access hard. Don't hide what they are. Any format that is not a plain image, or a plain text file, makes access hard.

Maybe a classic cbloom quote, for good old ranting's sake?

Developers very regularly get some crazy ideas in their head about why they absolutely must do something in a crazy over-complicated way, and they cling to some silly reason why they can't just do it the simple obvious way. It takes a 3rd party coming in to rip up the code and take out the stupidity. (This used to be sort of my specialty).

-- Simon

[ccexplore]

Hmm, maybe a case study is more compelling than listing arguments that are thoughtful but mostly only resonate with people already familiar with programming mindset (ie. points like "write your own tool", "collaborate, and send each other good, descriptive patches", "not hard to program a checker", "People can use version control on their files", etc.), which may help explain the previous lack of discussion.

My experience with Chip's Challenge and one of its clones, the Wonderland game series by Midnight Synergy, have shown that two games with essentially the same concept and similar mechanics, can wind up with vastly different audiences with vastly different preferences:  the CC crowd is extremely competitive and obsessed with solving levels in the least number of steps, and do tend to prefer difficult levels.  The Wonderland crowd is for the most part very much the polar opposite, tending to prefer prettier but easier levels, or taking the game to non-puzzle directions (creating more of a richer storyline in the level with small puzzles to advance the story), and have almost zero interest in competitive scoring.  I think in this case it may have a lot to do with the marketing strategy Midnight Synergy used.  It is also perhaps the case that when you start off with a game that has no built-in level editing support requiring skilled people to reverse-engineer and program their way into one (like Lemmings here and CC there, and unlike the Wonderland case), it is bound to be biased towards the more "hard-core" fans that are more likely to be competitive and proficient at programming skillsets.

And yet, the Wonderland experience do seem to prove that loose files for custom content are not too much an obstacle or deterrent for most users (or at least the fans), even literal grandmas (ones who I suspect from their forum postings, are far more proficient at baking a pie than, say, using git, but most certainly not as helpless as Simon's hypothetical "lovely auntie" even if they probably aren't into a Wonderland's analog of "trans-ONML-difficulty user packs" either ).  The game's support for custom 3D models and 3D textures are handled as loose files pretty much exactly the way Simon described, and as far as I can tell it has no discernible negative impact on people's ability to make use of custom content both in terms of creating levels with them as well as playing levels using them.  It helps that there is an active, friendly and helpful community with varying computer proficiency to seek help with, so any issues related to missing files and so forth tend to get resolved quickly and satisfactorily on the forums, by the more proficient or more experienced users.  In worst case, a reply in the forums with step-by-step screenshots of what to do on the computer, is usually successful.

Now to be fair, with Wonderland, IIRC the custom stuff are basically all optional--missing/misplaced custom content would affect how the level looks and given the level design tendencies there, probably would detract from the overall experience of playing the level, but strictly in terms of solving the level, the level still loads and solves the same way even with missing content.  This is of course completely opposite to our games here, where the level is most likely all but unplayable if some graphics it depends on are missing.  That said, ultimately even if missing files can be fatal to a level, I don't think it is as much a problem as one may imagine--in the Wonderland community, people have taken to simply compiling the set of all downloaded custom content and periodically upload those as single zip files, which for the most part seems to work reasonable well as a recovery mechanism (but again, not a perfect test since missing custom stuff is mostly non-fatal to start with).

[addendum/caveats: IIRC, custom levelpaks never existed in that game franchise, except maybe for the final game (by which point I'm no longer active in the community so I don't know if they ever released an editor that supports creating and distributing a full set of linked custom levels with an overworld hub level, like the game itself is structured by that point).  Otherwise, custom levels have always been single levels in Wonderland.  So there is no data to drawn from the case study for informing on "levelpaks as loose files versus single file for the pack".  The levels themselves are in proprietary binary format, and AFAIK no one there has ever rant about the level data being binary instead of text.  Also, IIRC, a fair number of core game elements like exits, switches and gates can't actually be customized at all [other than modifying the graphics files used by the game itself, which then affects every level so it's not really customization in the way you want), so custom content are more typically decorative in nature.]

[namida]

The folder / file tree format is going to be supported. As I have said, it will never become the primary format, because the primary user involvement with graphic sets is using them, not making or editing them; and a single file is more convenient to this structure. For those who want to use it - either for WIPs, or if it'll make them feel better, because they'd rather store even the default ones / other addon ones in this format and pretend a binary format doesn't exist - a folder structure will be there.

I would say that given that a folder structure is supported, any further discussion on this falls into the territory of "it's not enough that my preference is supported, I insist that everyone else uses it too"... keep in mind that no matter how de-linked from a numbering system any given folder (or binary) structure may be, it'll still have to at some point implement either some numbering system, or some compatibility layer with a numbering system (or a very complicated conversion process for older content), if we want to be able to support other formats beyond the native one.

[geoo]

What if...your native binary format is a zipped version of the folder format? *gasp*
(If you are afraid of lusers accidentally unpacking them, change the extension from zip to something else, like nxz). Best of both worlds.

EDIT: Forgot to say, I support Simon in pretty much all his points from the first post.

[Nepster]

Some thoughts from a user somewhere in the middle of your lovely aunt and an expert programmer.

1) File tree for terrain pieces
I support this addition as it works well for Lix and SuperLemmini. However I wasn't really happy with Lemmini in that respect: Sometimes a missing terrain piece would crash Lemmini and one had to reload the terrain pieces from WinLemm, which made keeping backups necessary. In total I have now ca. 5 different(?) style folders as backups, which is a huge mess. So:
1a) More options for users to mess around means that better error handling is required. And if an error occurs, the error message should be descriptive, e.g. detailing which terrain pieces are missing.
1b) Someone (i.e. namida) should collect all custom terrain pieces and provide a style folder containing all of them. I do not wish to have several style folders for different NeoLemmix level packs (or even single levels). And when merging style folders I do not want to check whether identically named pieces are actually identical. If the latest version of NeoLemmix automatically comes with the full style folder, then even better.

2) File format for terrain pieces
@namida: What file format do you plan to use for single terrain pieces added via the file tree? For me the biggest advantage of commonly used formats is the easy way to view them by browsing the style folder. Creating new pieces or changing them is rare enough that I would be willing to use special tools for it.

3) Graphic sets
The Lix community level pack shows that different styles are rarely combined in one level. Mostly "foreign" terrain pieces are used as decoration, for erasing terrain or because the style misses e.g. steel pieces. Thus I see not really a reason to allow mixed graphic sets if that means huge efforts to implement. On the other hand I see at the moment no reason to disallow mixing separately stored terran pieces from the file tree.
geoo's count showed that my previous believe was very wrong and different styles are frequently used in Lix.

4) Conversion problem from native to folder structure
I can guess where the problem is, but probably one can add a replace-tool in the editor? This would allow to replace all terrain pieces of type X with type Y (even better if there is an easy way to do multiple such replacements at the same time). This would at least create the possibility to manually transform a level from using the native terrain pieces to pieces from the style tree.

5) Music files
I would greatly appreciate the possibility to select the music by browsing through some music folder. Even "Tim6P" is more descriptive than track "17" (not to mention properly named music files). Moreover this allows for more than ca. 250 tracks and has hardly any chance that different users have different tracks with the same name (compared to having different tracks with the same track number).

6) Level pack .exe
I prefer not to have a separate style folder, music folder, etc. for each big level pack. At the moment I see two possibilities:
6a) Compress only the raw level data, not the core game mechanics, styles, music, ... Then one would start NeoCustLemm and load from there all pack specific data.
6b) Store everything including the style and music folder in every level pack .exe.
In my opinion option 6a) has several advantages:
- After fixing some bug/glitch/... one only has to update the NeoCustLemm file, not every single level pack file. Users like me are lazy and still use V1.28n for LPI for example.
- Smaller level pack files (though the file sizes are small enough that this shouldn't be a problem nowadays).
- Continuing support for level packs possible, even if the designer isn't active any more.
But of course I have no idea how difficult it would be to code 6a).

Edit: Corrected misapprehension in argument 3).

[Simon]

and a single file is more convenient to this structure.

No. Absolutely no.

You brought some arguments in the past, which I have criticized here, and you haven't replied to any criticism there yet.

A badly accessible format is bad for the player who never designs content. There will be fewer creative works made by other people, and that is bad for the player.

Please show why you deem a custom format superior.

I would say that given that a folder structure is supported, any further discussion on this falls into the territory of "it's not enough that my preference is supported, I insist that everyone else uses it too"

You're a systems designer.

Your responsibility is to nudge people into doing the proper thing. You want to foster culture around your system, not prevent it. I have shown why the tree is the proper thing for that.

Supporting it and pushing it are entirely different.

implement either some numbering system, or some compatibility layer with a numbering system (or a very complicated conversion process for older content), if we want to be able to support other formats beyond the native one.

If your old format clashes with decentralization, or depends on numbering, then I acknowledge we should be careful here, yes. You have a ton of cruft to pull into the new system, there are many interesting level packs.

That doesn't invalidate any arguments about why the tree is superior, or why naming is good and numbering is bad.

-- Simon

[Proxima]

The folder / file tree format is going to be supported. As I have said, it will never become the primary format, because the primary user involvement with graphic sets is using them, not making or editing them

I think that's putting the cart before the horse. If editing graphics sets were easier/more accessible, then people would do it more. For instance, during the development of the Lix community set there were several times when one of us wanted to add a single piece or small number of pieces to help realise a level idea: mobius' thin platforms, Nortaneous' diagonal wooden slats, my 3-tile steel blocks, geoo's late additions to the sandstone set. With Lix's structure, this was very easy: all we had to do was make the piece and put it in a folder.

[namida]

1b) Someone (i.e. namida) should collect all custom terrain pieces and provide a style folder containing all of them. I do not wish to have several style folders for different NeoLemmix level packs (or even single levels). And when merging style folders I do not want to check whether identically named pieces are actually identical. If the latest version of NeoLemmix automatically comes with the full style folder, then even better.

In regards to this, I was thinking that some kind of system for naming graphic sets should be established. Of course it'd still rely on everyone following it (most likely with strong nudges, although not outright enforcement, from NeoLemmix itself). This would remain equally relevant regardless of whether a folder structure or single-file format is used.

4) Conversion problem from native to folder structure
I can guess where the problem is, but probably one can add a replace-tool in the editor? This would allow to replace all terrain pieces of type X with type Y (even better if there is an easy way to do multiple such replacements at the same time). This would at least create the possibility to manually transform a level from using the native terrain pieces to pieces from the style tree.

I've already done some thinking on how to handle this, since some degree of conversion when importing old levels needs to happen anyway, due to the difference in how certain elements (eg. one-way walls) would now be handled. I have ideas on how to handle it in either a traditional-ish-graphic-set setup or a folder-structure setup; I'll let you guess which one it's easier to handle it in. (Hint: Not the one that everyone thinks is superior.)

5) Music files
I would greatly appreciate the possibility to select the music by browsing through some music folder. Even "Tim6P" is more descriptive than track "17" (not to mention properly named music files). Moreover this allows for more than ca. 250 tracks and has hardly any chance that different users have different tracks with the same name (compared to having different tracks with the same track number).

This is a planned feature. (In fact, technically NX1 itself already supports such a thing, but support for it in the relevant tools was never implemented. That's probably something I can look at fixing with quick updates to the editor and Flexi Toolkit at some point...)

6) Level pack .exe
I prefer not to have a separate style folder, music folder, etc. for each big level pack. At the moment I see two possibilities:
6a) Compress only the raw level data, not the core game mechanics, styles, music, ... Then one would start NeoCustLemm and load from there all pack specific data.

This was one of the earliest discussions regarding NX2, and it was quickly decided on that yes - NX2 will have a single EXE which loads game data. The editor will be seperate, but it'll be an all-in-one editor suite, not a collection of several seperate tools.

[geoo]

4) Conversion problem from native to folder structure
I can guess where the problem is, but probably one can add a replace-tool in the editor? This would allow to replace all terrain pieces of type X with type Y (even better if there is an easy way to do multiple such replacements at the same time). This would at least create the possibility to manually transform a level from using the native terrain pieces to pieces from the style tree.

I've already done some thinking on how to handle this, since some degree of conversion when importing old levels needs to happen anyway, due to the difference in how certain elements (eg. one-way walls) would now be handled. I have ideas on how to handle it in either a traditional-ish-graphic-set setup or a folder-structure setup; I'll let you guess which one it's easier to handle it in. (Hint: Not the one that everyone thinks is superior.)

Well, how about we start a fruitful discussion about how this can be done sensibly? Maybe you don't like a file tree because you don't have the same implementation details in mind that we do? I assume the reason why you think your lovely binary blob is so nice is because internally, you still want to assign a number to each terrain piece?
So let me make a suggestion for how to implement legacy support (the alternative is to just provide a batch level converter and drop legacy support which would certainly make your code cleaner):

Assumption (correct me if I'm wrong): Currently, each terrain piece belongs to a graphics set. Each NL1 graphics set has a number associated to it, and within a graphics set each tile has a number associated with it, same for objects.

Solution A: name each terrain piece e.g. 23.png, encoding the internal number, and for the existing tile sets just export all the images of pngs with the associated file names. For the already existing tile sets, NL just needs to remember which graphics set ID corresponds to which graphics set (folder) name. I'd advise against keeping this convention for newly created tile sets as it doesn't allow you to organize your tiles all that well, but it's feasible if you want to stick to numbering tiles.

Solution B: because each terrain piece/object is a pair (graphics_set, piece_id), NL has a dictionary that maps each pair to a path to the corresponding terrain piece. This feels a bit cleaner as if you wanted to, you could now also give the original terrain pieces descriptive names as you can for all new terrain pieces, but I don't think there's much demand for the original terrain pieces having names.

In the new level format, each terrain piece/object is referred to by the path to the location of the .png/.bmp/whatever. Maybe just a base name without the file extension, as for objects you'll also have associated files with a name derived from the base name, e.g. object.png, object_mask.png, object_meta.txt.

Of course, if you want to even keep numbers for tile sets, then you could also name the containing folder simply by the number, but that has the issue that two people making a new tile set in parallel might end up choosing the same number for it.