[DISC][PLAYER] Styles cleanup - Duplicates / near-duplicates

[namida]

We recently received a style submission. I will call this style "Style A". This style was created in old-formats by one author, and converted and submitted by a different author. This style in question was almost identical to "Style B", from the same author who created Style A (but Style B was submitted, a long time ago, by the author themself). I didn't compare the exact physics, but at a quick glance, it differs only slightly in terms of shading. On top of this, both of these are in turn just a derivative of Style C (which was among some of the first NL custom styles), by a different author altogether.

I feel that this is an excessive level of duplication. Style C vs Style B, there is some merit - while they're intended to be (though aren't always) physics-compatible, there's a significant difference in visual. As in, one is almost DOS-style, and the other is more full-color, modern style. However, A vs B is literally just a slight difference in brightness. As such, Style A was rejected from the download - it would have been utterly nonsensical IMO to accept Style A at the same time as we're removing orig_dirt_md.

My concern however, is that when I brought this to the attention of the user who converted and submitted Style A, they brought to my attention some other styles that are partial or wholly duplicates with recolorings, such as GigaLem's Xmas variants of ONML styles, or Mobius's "tancastle" style. To be fair - these are generally a lot more distinguished from the source styles than Style A and Style B were from each other; these two were so close that orig_dirt and orig_dirt_md were more different than these two were. But I also wonder if we really need even that - we're already dealing with a very over-bloated styles download, can we trim some of the fat? We currently have about 42MB of styles (to be fair, this also includes sound effects; but on the flipside, this is zipped size), consisting of a total of 166 styles.

Ideally, eventually NL will get back its download-on-demand features so new users will only have to download a few core styles (the official ones, probably including L2 / L3, and maybe some of the most commonly-used custom ones like the Lemmings Plus ones, the Lemmini Castle set, etc) and those that are rarely / never used will just sit on a server doing nothing instead of bloating the download / user folders. However, this is likely still quite some time away, and I think we need to do something in the meantime. I suspect that when Nepster made the call to start including all styles with NL's download, it was not envisioned that the number of styles available would explode like this - there were maybe 40 or so styles back then. (Indeed, if I'm being perfectly honest, the potential for this was a reason I was averse to moving to these kind of simple formats. Having somewhat of an effort-based barrier to entry helped keep the quantity down and the quality up; though it's undeniable that the flipside is true too, that there have been some excellent styles created that might not have happened under the more-difficult old-formats way of doing things.)

I have implemented a feature for the next update (V12.7.0-RC6 or V12.7.0-stable) that can be used to specify replacement pieces / styles for any removed ones - this can work either on the level of entire styles, or just single pieces (I've made use of the single-piece replacement feature to deal with a duplicate piece in one of my styles, for example).

Perhaps a good starting point is figuring out which of the current styles are actually used by anything. For example - GigaLem's Xmas variants of OhNo styles, I'm fairly sure the only use those saw was in Holiday GigaLems, which I don't believe GigaLem has any intention of bringing over to new-formats, so do we really need to keep the styles? Unfortunately though, unlike Lix, NL doesn't have a single central repository of levels, so every pack would need to be tracked down and checked to determine this. (It would also not be very practical to maintain a central repository, unless it's based on user-submission, in which case it likely won't catch every existing level.)

The alternative option may be to go back to how we used to do things - NL comes only with a few core styles, and users must download any extras on their own. Content authors would need to point out what styles are used in their packs, and ideally, where to get them. Downside of this, is that if someone's download link breaks, the style is lost, and all content that relied on it is broken. Also, we now have a whole bunch of content that uses a mix of "core" and "non-core" styles, and creators of this would need to go back and figure out what's included vs what needs to be downloaded.

[IchoTolot]

The alternative option may be to go back to how we used to do things - NL comes only with a few core styles, and users must download any extras on their own. Content authors would need to point out what styles are used in their packs, and ideally, where to get them. Downside of this, is that if someone's download link breaks, the style is lost, and all content that relied on it is broken. Also, we now have a whole bunch of content that uses a mix of "core" and "non-core" styles, and creators of this would need to go back and figure out what's included vs what needs to be downloaded.

I am 100% against this. I would rather download 1000 styles than tracking everything down manually and deal with tons of broken links and new topics about missing styles errors like in the old days (Seriously there was one every week or so).

I see that we need a bit more quality control though to not let the number explode unnessesarily. Maybe for any future submissions exclude slight recolors or near duplicates to avoid any further excessive duplication.

For the older near duplicates: Exactly list them, examine the usage and then decide over a possible exclusion.

[namida]

I see that we need a bit more quality control though to not let the number explode unnessesarily. Maybe for any future submissions exclude slight recolors or near duplicates to avoid any further excessive duplication.

One possible issue here: We can't stop people from posting their custom styles on the forums, we can only exclude them from the NL download, or at most, make NL specifically recognize those styles' names and use the original even if the duplicate is present. (Okay, technically, as forum admins, in theory we could delete posts of such styles and even ban people who repeatedly post them, but this feels ridiculously heavy-handed in practice - and this is coming from someone who was perfectly okay with preventing new posts in the pre-V10 levels releases forum to discourage new pre-V10 content.) Thus, would this "quality control" just end up leading to a de-facto version of the old ways, just with a much larger amount of "core" content?

[IchoTolot]

I see that we need a bit more quality control though to not let the number explode unnessesarily. Maybe for any future submissions exclude slight recolors or near duplicates to avoid any further excessive duplication.

One possible issue here: We can't stop people from posting their custom styles on the forums, we can only exclude them from the NL download, or at most, make NL specifically recognize those styles' names and use the original even if the duplicate is present. (Okay, technically, as forum admins, in theory we could delete posts of such styles and even ban people who repeatedly post them, but this feels ridiculously heavy-handed in practice - and this is coming from someone who was perfectly okay with preventing new posts in the pre-V10 levels releases forum to discourage new pre-V10 content.) Thus, would this "quality control" just end up leading to a de-facto version of the old ways, just with a much larger amount of "core" content?

I am totally against any censorship. If people want to post duplicate styles let them, it's their right. It's only their problem to maintain then and the standard library won't support those styles.

Only the size of the standard library download is the issue here and needs to be prevented from explode any further. The user can hord up as many tilesets as they like, just that standard download is the issue.

[namida]

I am totally against any censorship. If people want to post duplicate styles let them, it's their right.

Yes, I agree - my point about being able to delete the posts is saying "This is physically possible, but it would be ridiculous to actually do it".

[Strato Incendus]

I appreciate namida's attempt to keep things anonymous here - me, personally, I am fine with everyone knowing that the user who sent the tileset in question to Nessy was me. (I'm not giving away whose tileset we're talking about here, though I think there's enough information on the forums already to figure out for those who are interested.)

I think we already agreed a while ago not to create any new versions of tilesets which are merely recolourings, IIRC? This case was special, because the tileset I converted is actually an interim version, i.e. an older, different looking version of a tileset which, in its final state, has become part of the official styles download by now. I had used this style as it was being developed, and wanted the levels created during that time period to still look the same way they were originally created.

Of my own accord, I never would have gone out of my way to actively recolour any existing tilesets.

So I think this is more comparable to providing backwards-compatibility for old content, rather than encouraging the creation of new content?

Of course, with graphic sets, this is a more difficult case than with whole levels, because any graphic set that can be used to play old levels can also be used to create new levels with. While with different versions of NeoLemmix, creators have to jump through some hoops to create content for outdated versions, any style that's available in the style folder, even if it's just intended for playing, can be just as easily accessed in the level editor as any other, "established" tileset.

Thus, I acknowledge these two things aren't strictly comparable. But in general, I'd be on the side of "let's allow the existing recolourings we have or have had in the past, but let's not create any completely new ones." I mean, there's nothing stopping anyone from taking e.g. the original pillar tileset and re-painting it in pink camouflage, but really, what's the point?

Removing existing recolourings, meanwhile, like the festive Rock and Brick tileset, the golden Crystal tileset (does that still exist?), or mobius's tanCastle, would end up breaking levels using them, and thus be equivalent to a cull. I already had levels break simply due to just single pieces being removed from tilesets by the original creator of that tileset.



This is also something we should consider creating some sort of "code of conduct" for: If you create a graphic set and release it to the public, starting from that point, anyone can create levels with it. (If it's a preliminary version, one can explicitly advise people not to create levels with it yet, just like when new skills are getting introduced which aren't finalised yet.) Adding further pieces to a tileset is no problem, then - it used to be, in Old Formats, in which new pieces could only be added at the end of the list, otherwise they would have messed up any levels using this tileset, because the order was relevant. Now that levels only refer to each piece by name, adding stuff is a lot easier.

But removing pieces from existing graphic sets, especially when it comes to terrain, can break a bunch of levels without the level designers noticing it. The style gets updated with the next download, and suddenly certain levels don't work anymore.

So the question is really: "Who 'owns' a graphic set?" Just the original creator? If we think in terms of intellectual property, the graphic-set designer owns the tileset, but anyone who designed levels with that tileset owns the level. Therefore, a graphic-set designer removing pieces from an existing tileset violates level designers' intellectual property.

Unless we want to think of graphic sets more like "licensing platforms", such as Steam, or online trading card games - where the players (in this case, this includes level designers) don't own anything, but only get the permission to use the content, but this can be revoked at any time, potentially.

Of course, we can easily fall down a rabbit hole here if we apply the same logic to level packs - does a player "intellectually own" their solution? And thus, when the pack creator fixes a backroute and destroys the solution, is this also a violation of intellectual property? Players put a lot of time and brain power into trying to solve levels, that's a fact.

[namida]

So I think this is more comparable to providing backwards-compatibility for old content, rather than encouraging the creation of new content?

If there are physics differences between the two styles (which is even more likely given the status as an "early version" of the style that's later been treated as an entirely separate style), then having both of them exist is a very strong potential source of confusion and this takes priority over what content it will or won't break. If there are no physics differences, then nothing will break as long as NeoLemmix correctly redirects from one style to the other when loading levels.

If there are some pieces that exist in one version but not the other, they should be merged into the primary version.

[Proxima]

I'm glad this is being discussed. As some of you remember, I was very upset when the cull of dirt_md was announced, and it wasn't the cull itself that was hurtful, it was the lack of any consultation -- one minute everything's fine, the next, some of the content I'm working on is scrapped and I never got to have any say in the matter.

So maybe we can start over and have a proper discussion this time.

The real problem here is the bloat of the styles folder, not the existence of recolours -- there are over 160 styles and only a handful of them are recolours. Also, most individual users don't need anywhere near all the styles -- if I trimmed my folder to the styles used by the packs I am playing, it would be well under half that size.

So, what's to be done? Namida's suggestion of download-on-demand seems to solve everything, if it works more or less like I imagine (downloading a pack automatically comes with a download of any styles you need but don't have installed). The only problem is, as namida says, it may not be coming for some time.

As an interim measure, I would suggest that packs include a text file that lists all styles they use, and namida creates (I imagine this wouldn't be too hard) a tool for generating such files. Then the end user can easily determine which styles they personally don't need, and delete them.

If we agree that download-on-demand will happen eventually, then what is the harm in allowing recolours to exist? As with all less-used styles, they will only bloat the end user's style file if that particular user chooses to opt in.

This has all felt like a huge slap in the face to me, and I can't help asking: is it really that important to you to have 160 styles in your styles folder rather than 166? Will that really solve any problems?

[Dullstar]

I would like to see download-on-demand return. There's a lot of styles currently, of varying quality.

On the topic of recolors: I know we have recoloring for lemming sprites. Out of curiosity, how complicated would recolor support for entire styles be? It could be a way to allow recolor styles to exist without causing excessive bloat, although a way to distinguish them from their parent sets would probably be best. I envision a system where you would select the parent style you want and then would have the option to choose a recolor from there. Low-effort recolors could be a potential problem created by this, though, so there would probably have to be some somewhat stringent standards for what's enough to be accepted into the automatic downloads (though I'd say any from the official games I think should be allowed to be made).

I actually don't think the filesize is a huge problem, at least once they've been downloaded. What I think is the bigger problem is that the list of styles available in the editor is bloated and hard to browse. I'd suggest two possibilities for dealing with this (that are not mutually exclusive): 1) allow the user to create a "favorites" list of styles so they can quickly find the ones they use the most, or 2) allow the user to hide unwanted styles.

[IchoTolot]

On the topic of recolors: I know we have recoloring for lemming sprites. Out of curiosity, how complicated would recolor support for entire styles be? It could be a way to allow recolor styles to exist without causing excessive bloat, although a way to distinguish them from their parent sets would probably be best. I envision a system where you would select the parent style you want and then would have the option to choose a recolor from there. Low-effort recolors could be a potential problem created by this, though, so there would probably have to be some somewhat stringent standards for what's enough to be accepted into the automatic downloads (though I'd say any from the official games I think should be allowed to be made).

I think Dullstar is onto something here and it could be the solution for Proxima's/Strato's problems while also reducing the file/folder clutter! 

Possible solution: Don't treat recolors as a new standalone style in a new folder, but have an option inside the editor to choose an existing variant/recoloring of the parent style. Variants are stored in a textfile inside the parent style's folder.

It's just the question how the recoloring is acieved. We want to avoid having to duplicate all piece files inside the folder. Rather than that a textfile could be made where a color variant is specified. An example for the specifications would be: Change the following colors to these colors now.
It is be quite a bit of work to define all the nessesary color swaps in this textfile as there are probably many nessesary ones, but it's way better filenumber/folder wise than to duplicate everything.
This way the variants are also 100% identical physics wise and only provide a recoloring.

Example for the dirt style would be: You create a dirt level and then choose the genesis/etc variant from a selection field that comes up in the editor when recolor variants exist. All variants and how they are generated are stored in a textfile inside the style's folder.
This way things like the golden crystal style could be added without having another nearly identical style folder.

[Strato Incendus]

Indeed this is something I had thought of, too - I'll have to check again, though, I may actually have combined e.g. gigalem_treemd and gigalem_treemdbright on the same level. In such cases, a blanket approach of recolouring all pieces, either to just gigalem_treemd or just the bright version, wouldn't suffice to replicate the physics in the same way as standard tileset mixing does.

More generally speaking, it would be impossible to mix several different recoloured tilesets on the same level. This in turn would make them much less versatile than standard standalone tilesets, and feel like a step back to version 1.43, at least as far as these specific styles are concerned.

Or am I misunderstanding something here?

[IchoTolot]

Or am I misunderstanding something here?

Yes.

The option to choose the style variant would have to be coupled with the terrain/object choosing bottom bar and not with the main theme.

Let's say, you choose a main style, then select (for example) orig_dirt in the terrain/object bar below and opon choosing that another field appears where you can choose the variant which is read from the mentioned textfile by the editor. The images on the selection bar change on the fly. So you can choose tiles from orig_dirt and/or (both is then possible) the genesis variant.

So mixing between terrain/objects would be indeed possible after my example suggestion based on Dullstar's idea.

[Strato Incendus]

Thanks for the explanation!

In that case, this does sound like a great idea indeed! I'm fully in support of this approach!

[Flopsy]

I am in support of what has been mentioned above about having a dropdown menu for different styles within styles.

However I would like to call attention to this topic

https://www.lemmingsforums.net/index.php?topic=4486.msg77811#msg77811

[namida]

I am in support of what has been mentioned above about having a dropdown menu for different styles within styles.

However I would like to call attention to this topic

https://www.lemmingsforums.net/index.php?topic=4486.msg77811#msg77811

I went into more detail about this and proposed something that might in general help with such cases in that topic, but I'll quickly summarize here: This topic is really about styles that are, as a whole, duplicates or near-duplicates of other styles. Single pieces that do properly fit in multiple styles aren't such a big deal.