[SUG][NeoLemmix Level Packs] Difficulty Ratings Scale

[WillLem]

Suggested Difficulty Scale for Custom NeoLemmix Levels/Packs

After discussion, the majority of us seem to agree, more or less, that the following scale is appropriate:

(I've placed the various ranks of Lemmings Redux onto this suggested scale).

Beginner - brand new players - Gentle
Easy - begins to stretch the player with basic puzzles - Quirky
Medium - introduces more complex puzzles and level design concepts - Zany / Manic
Hard - expects the player to have a solid grasp of the game, puzzles become particularly challenging - Lunatic
Advanced - requires more in-depth knowledge of a wide range of solving techniques, strategies and tricks
Extreme - challenges even the most accomplished solvers



(Previous OP):

I'd like to propose the following sliding-scale difficulty rating for creators to assign to their packs, which may help to give a more specific idea of what a player can expect based on their ability. It may also help players to identify their ability level, and choose packs accordingly.

If adopted, creators would be encouraged to apply the ratings as part of any level pack upload in the following manner:

Pack Name [Difficulty 0-3] - this would indicate that the pack features levels which range in difficulty from beginner through to advanced, and therefore can be enjoyed by more casual players as well as more experienced players who wish to take a break from more challenging packs! It may also serve as an ability-boosting pack for less experienced players.

Pack Name [Difficulty 3-5] - this would indicate a more difficult pack that is best attempted by advanced players who are looking for further challenges.

Pack Name [Difficulty 4-6] - this would indicate a pack that is best attempted only by very experienced players who have established themselves as master solvers.

Pack Name [Difficulty 0-6] - this would indicate that the pack spans the full range of difficulty ratings.

Here are my proposed difficulty ratings explained:

0) Beginner - this rating indicates that the pack includes tutorial levels for the absolute beginner who has no prior experience of playing the game, and needs to be introduced to the skills, objects and their basic functions, and general gameplay.

Examples of levels in this category: Just Dig!, Only Floaters Can Survive This, You Need Bashers This Time.
Examples of techniques: Basic application of a few skills.
Examples of objects: The levels may feature fire, water and traps, but these will be very easy to avoid and may not even form part of the main route.

1) Novice - this indicates levels which expect that a basic tutorial will have been completed, and the player is able to navigate their way through reasonably easy levels with plenty of available skills and various possible solutions.

Examples of levels in this category: Smile If You Love Lemmings, Keep Your Hair On Mr. Lemming, Don't Let Your Eyes Deceive You.
Examples of techniques: Basic application of a number of different skills.
Examples of objects: The levels may feature fire, water and traps, but these will be very easy to avoid and may not even form part of the main route. The levels may also gently introduce objects such as teleporters and pickup skills.

2) Intermediate - packs with this rating will be generally easygoing packs that players of all ability levels can enjoy; the levels will be easy to solve and fun to play through, serving the purpose of keeping a player's basic skills well exercised. There may be the occasional challenge, but these will be reasonably solvable in the first attempt.

Examples of levels in this category: We Are Now At LEMCON ONE, Konbanwa Lemming San, Lemmingology.
Examples of techniques: Considered application of a number of different skills.
Examples of objects: The levels may feature fire, water and traps that require some thought to avoid. The levels may make use of teleporters and pickup skills to enhance the level, but not necessarily make it more difficult.

3) Advanced - this rating indicates that some of the levels will be more challenging and require above-average ability and a less casual approach to the game; solutions will be less obvious, and will include more advanced tricks, techniques and strategies more familiar with experienced players.

Examples of levels in this category: Turn Around Young Lemmings!, I Have A Cunning Plan, Compression Method 1.
Examples of techniques: Advanced application of a number of different skills, introduction of lateral-thinking puzzles and strategies.
Examples of objects: The levels may feature fire, water and traps that require considered strategy to avoid. The levels may make use of teleporters and pickup skills to enhance the level and increase the difficulty of the puzzle to some degree.

4) Expert - this rating suggest that the levels are of a very high standard of difficulty, requiring much more precise use of multiple skills and combinations of advanced techniques which would not necessarily be obvious to a beginner or a casual player. It will generally take several attempts to complete a level, and the levels will usually have a very limited number of possible solutions.

Examples of levels in this category: The Great Lemming Caper, Save Me, Lemming Tomato Ketchup Facility.
Examples of techniques: Precise application of a number of different skills, increasingly complex puzzles and strategies.
Examples of objects: The levels may feature fire, water and traps that require specific solutions within the puzzle to avoid. The levels may make use of teleporters and pickup skills to make a puzzle more complicated.

5) Master - packs in this rating will include levels of extreme difficulty, requiring established knowledge of most if not all advanced tricks and solution methods; levels will have very complex and precise solutions, and only the very minimum number of skills and time required for the intended solution. The levels will also expect perfect timing, knowing when as well as how to apply a particular skill.

Examples of levels in this category: It's Hero Time!, No Added Colours Or Lemmings, The Race Against Cliches.
Examples of techniques: Very precise application of a number of different skills, complex puzzles and strategies comprising very specific solutions.
Examples of objects: The levels may feature fire, water and traps that require specific solutions within the puzzle to avoid. The levels may make use of teleporters and pickup skills to make a puzzle more complicated.

6) Legend - these levels are only for players of the very highest solving ability. They will normally feature several combinations of extremely complex and painstakingly precise techniques, as well as pixel-perfect timing. These levels will often introduce brand new techniques not seen before, and use them in such a way to increase the difficulty of the puzzle considerably.

Examples of levels in this category: Anything from the final ranks of Lemmings Plus Alpha or Lemmings United.
Examples of techniques: Precise and unusual application of a number of different skills, extremely complex puzzles and strategies.
Examples of objects: The levels may feature fire, water and traps that require specific solutions within the puzzle to avoid. The levels may make use of teleporters and pickup skills to make a puzzle more complicated.

[namida]

Difficulty is subjective. What one person finds hard, another might not. On top of that, it's generally very hard for creators to accurately judge their own packs, especially relative to other people's packs (as opposed to their own earlier / later packs).

While I encourage creators to try and indicate the difficulty of their pack, I don't think a "standardized" system is going to work out all that well - and we certianly won't be enforcing one.

A more realistic approach, though this would be a "distant future" thing if it happens at all, would be a content database packs can be downloaded from, that would also accept user ratings re: difficulty. While still not super-reliable, players could then at least use these to get a more useful indication.

[Proxima]

I fully support the idea of some sort of database or overview page that makes it easier for users to find content, and including difficulty ratings so that it's easier for users to find content appropriate to their skill level, so they can get more enjoyment out of the game and to help those who want to to gradually work their way up the ladder until they are ready to tackle the really tough stuff.

I think it would be great if we (as a community) could put our heads together, come up with a standardised system we can agree on, and apply it to existing packs (with each pack having a rating for its start and end points, since most packs have some progression from easier to harder within the pack). We might disagree on exactly where a certain pack belongs, but hopefully our ratings would average out to sensible decisions.

The current system of leaving it to the pack author is better than nothing, but has three obvious problems: some pack authors don't bother at all; pack authors are not the best at judging their own difficulty; and the lack of any discussion means that different pack authors mean different things by, e.g. "Easy" (see for example Lemmings Destination, which is rated as "Easy - Very Hard" by its author -- the first rank would be between 5 and 6 on WillLem's scale).

Since I've thought about this so much, maybe I'll have a go at writing up my own proposal for a difficulty scale to use as a starting point. I don't think we should start off from WillLem's, because it doesn't match well to the range of usermade content -- we don't need the first six gradations to cover the range from "Just Dig" to "It's Hero Time" when the vast majority of usermade content is harder than anything in the original games.

[WillLem]

I agree that any such system ought to be compiled by the community, by gathering data and feedback from various users from a range of skill levels.

My proposal is mainly a template: I'm sure that the range of difficulty is wider than I've suggested and I think that really anything harder than the original games would be considered "Master" or "Legend" by my standards, but there are probably subcategories within and even beyond those that I haven't encountered yet.

I'd like to be part of the discussion for whatever system gets implemented, simply because I'm relatively new to NeoLemmix and I'm probably about average ability when it comes to solving; I think my perspective could be very useful.

I also think that Namida's idea of a content database is definitely a good proposal for implementation and could go hand in hand with this idea (downloading everything needed for a level pack from one place); collating all of the packs together and grouping them by difficulty would be a great development.

[WillLem]

Proxima has suggested that ranks, rather than packs, should have universally-agreed difficulty ratings, and I think this is a great idea. Of course, they could still be titled whatever the user wishes, but they could be assigned a number or letter based on their difficulty.

So, you could have:

Fun (1)
Tricky (2)
Taxing (3)
Mayhem (4)

and:

Flib (2)
Blah (3)
Jibble (4)
Wumpus (5)

or even:

Snib (4)
Weff (5)
Blopsy (6)

We'd just have to agree (as a community) on what exactly 1, 2, 3, 4, 5 and 6 mean. Here's what I suggest:

1 - Anything that's a tutorial level, any-way-you-want-it, far more skills than needed, basically Fun/Tame-style
2 - Levels that require a bit more thought, but still have multiple possible solutions and plenty of available skills
3 - Levels with fewer possible solutions and requiring specific tricks/methods, but perhaps with some leniency in the skillset and/or relatively clear indications of what's meant to be done
4 - Levels requiring one very specific and relatively obscure solution, and only providing the necessary skills for that solution
5 - Levels requiring one very specific solution involving difficult-to-spot or even brand new tricks and methods not widely known to the community, and only providing the necessary skills for that solution
6 - Giving a level a "6" would indicate an extreme degree of complexity, with the expectation that not many players will be able to solve the level

Of course, a "5" level essentially becomes a "4" level once a trick or method becomes common knowledge, but this has always been true of Lemmings games, and indeed many puzzle games. To some extent, the same could happen for "6" levels as well, although there are certain levels that stand the test of time in this regard, and are always considered extremely difficult.

The easiest way to assign a number to a rank is to give the most difficult level in the rank a number rating, and then simply apply it to the entire rank. Of course, difficulty within ranks can fluctuate, but it gives a clear indicator for the general standard of the rank (and, by extension, the pack), which is all this is meant to be.

---

N.B. The rating system should always indicate the designer's intention, because it's impossible to take into account how a group of players will approach solving the level. For example, a designer could assign their level a "5", but there may be several possible alternative solutions or backroutes. The level is still, however, a "5" due to what the intended solution is. If a player is savvy enough to craft out a different solution, it doesn't make the level itself any less difficult.

[mantha16]

i dont rank my packs in order of difficulty so if it became a required part of building packs i wont build anymore and like namida says its hard to judge sometimes so yes I agree some indication of difficulty is really useful for players I am 100% against enforcing a difficulty rating system or making people order their packs easy to hard cos sometimes its nice to play things all over the place having an easy level following a difficult level can be a relief.

[Strato Incendus]

1 - Anything that's a tutorial level, any-way-you-want-it, far more skills than needed, basically Fun/Tame-style
2 - Levels that require a bit more thought, but still have multiple possible solutions and plenty of available skills
3 - Levels with fewer possible solutions and requiring specific tricks/methods, but perhaps with some leniency in the skillset and/or relatively clear indications of what's meant to be done
4 - Levels requiring one very specific and relatively obscure solution, and only providing the necessary skills for that solution
5 - Levels requiring one very specific solution involving difficult-to-spot or even brand new tricks and methods not widely known to the community, and only providing the necessary skills for that solution
6 - Giving a level a "6" would indicate an extreme degree of complexity, with the expectation that not many players will be able to solve the level

Interesting categorization! I think all my New-Formats packs (Lemmings World Tour is both Old and New Formats) will be in the 3-5 range.
Lemmicks might be considered in that range as well.
Pit Lems might be 2-4 or maybe 2-5, though I can think of very few trick-based levels in Pit Lems that are so obscure that they would warrant a five. Pit Lems is the only one of my packs with a regularly high number of any-way-you-want levels, because back then I thought this was what the community wanted, and what they meant by "puzzle".

Meanwhile, I've used the crash of my old laptop as an incentive to do a new and clean download of the entire Lemmings Plus Series for New Formats, and started playing Lemmings Plus I, this time with the intention to do it continuously. I've breezed through in one session well into the third rank, until the large number of consecutive any-way-you-want levels started boring me to death. I know that eventually, once the first real puzzle pops up in that pack, it will hit me out of nowhere and most likely stump me. I just hope that with all the experience (and knowledge of obscure tricks) I've picked up in the meantime, I will be able to make further progress this time than during my last attempt.



But yeah, any-way-you-want levels, or those that leave you with many spare skills at the end - levels which you label as difficulty 1 or 2 - I just hate those. A pointless waste of the player's time. If you can do basically anything, it feels like nothing you do matters. Any decision you make on a level can be the correct one. And if you make a false decision, you have plenty of skills left to salvage the attempt without even having to rewind.
Usually, not even the requirement of having to save every lemming puts much of constraint on the "level's indifference". (This was kind of what I was poking fun at in Lemmings World Tour, with my "Great Level of Indifference" - an X-of-everything level that also had a save requirement of 0 - until somebody set the minimum save requirement in New Formats to 1, apparently .)
Even worse, when a bunch of levels can be solved in the same way (e.g. with Basher-cancel-staircases), it also gets repetitive. Combine that with the large number of remaining spare skills and feels like I'm backrouting every level.

I assume Lemmings Plus I was namida's very first pack, so it's by no means a fair comparison - I'd have to compare it to Paralems, which was my very first pack, and Lemmings Plus I is definitely much higher in quality than that. At that point, I guess it was also something revolutionary in and of itself to simply get any type of new custom Lemmings levels in the classic tilesets with the classic skills. But in direct comparison to the other packs we have available for New Formats now, this one so far doesn't seem to have aged well. I expect it to get really challenging on the top ranks, of course... it just takes ages to get there.



Maybe this means that a pack shouldn't span the entire difficulty spectrum from 1 to 6? Because then no beginner will be able to finish it, and any advanced player will be bored for a very long time by the early ranks until the pack finally gets up to their skill level?

Or maybe the difficulty curve of my own packs is just too flat - starting out fairly challenging compared to other packs which start with any-way-you-want levels, but then basically remaining at that level of difficulty, maybe only increasing level complexity and "obscurity" in the course of the ranks...

The pack with the best difficulty curve in my opinion is still SubLems, which starts very easy and becomes fairly hard at the end... but I wouldn't say it goes all the way up to 6. Otherwise, I for one wouldn't have been able to complete it.

[Proxima]

i dont rank my packs in order of difficulty so if it became a required part of building packs i wont build anymore and like namida says its hard to judge sometimes so yes I agree some indication of difficulty is really useful for players I am 100% against enforcing a difficulty rating system or making people order their packs easy to hard cos sometimes its nice to play things all over the place having an easy level following a difficult level can be a relief.

Just to make it clear, we are not suggesting that it should be required to order packs from easy to hard. Just that many packs cover a wide range of difficulty, so that it's hard and maybe less meaningful to assign a rating to the pack as a whole, and more meaningful to assign ratings to the individual ranks. For a pack like Sammings where each rank has similar difficulty, the ranks would get much the same rating and that's perfectly okay.

You make a valid point that difficulty can also vary a lot from level to level, but if we tried to assign difficulty ratings to individual levels, assigning the ratings would be a lot more work (so fewer people would bother) and there would also be a lot more disagreement -- I remember when DireKrow and I were playing LPI/Psycho, we had a lot of disagreement with each other and namida over which levels we found hard. I have a hunch that if we looked at ranks as a whole, our evaluations would be a lot more similar. For instance, pretty much everyone who's played LPI will probably agree that Psycho is (overall) harder than Danger; there might be disagreement over how much harder.

[Proxima]

We'd just have to agree (as a community) on what exactly 1, 2, 3, 4, 5 and 6 mean. Here's what I suggest:

1 - Anything that's a tutorial level, any-way-you-want-it, far more skills than needed, basically Fun/Tame-style
2 - Levels that require a bit more thought, but still have multiple possible solutions and plenty of available skills
3 - Levels with fewer possible solutions and requiring specific tricks/methods, but perhaps with some leniency in the skillset and/or relatively clear indications of what's meant to be done
4 - Levels requiring one very specific and relatively obscure solution, and only providing the necessary skills for that solution
5 - Levels requiring one very specific solution involving difficult-to-spot or even brand new tricks and methods not widely known to the community, and only providing the necessary skills for that solution
6 - Giving a level a "6" would indicate an extreme degree of complexity, with the expectation that not many players will be able to solve the level

I think this is on the right lines, provided of course that the categories are not interpreted rigidly. Levels can have one specific solution and be very easy: "Just dig", "All or Nothing", "Rules to fall". Conversely, levels can be open-ended, with more skills than necessary or even allowing more lemmings to die than are necessary for the solution, and still be very hard: "Survival of the Craftiest", "Path of Wickedness", "Labyrinth of Despair" (the Lix level, not the NL level it's based on). In general, difficulty is a combination of length of solution; obscurity of tricks involved; temptingness of red herrings ("entropy"); how much leeway you have in terms of skills, save requirement and time; and difficulty of execution. Higher difficulty in any one of these factors could push a level up a category even if the others are not present.

Maybe this means that a pack shouldn't span the entire difficulty spectrum from 1 to 6?

I mean... who said it should? There are packs such as the Lix community set, which does span the entire difficulty range (in fact, I would say its six ranks correspond fairly closely to WillLem's 1-6) because it aims to be a "flagship" pack putting together the entire community's best efforts. Finishing the pack is expected to be a substantial commitment. Revenge of the Lemmings is in a similar boat.

But other than those community packs, I can't think of any pack that literally spans the entire range. Namida's LPI is probably 1-5, while his harder LP packs might be 2-6 or even 3-6.

[WillLem]

i dont rank my packs in order of difficulty so if it became a required part of building packs i wont build anymore and like namida says its hard to judge sometimes so yes I agree some indication of difficulty is really useful for players I am 100% against enforcing a difficulty rating system or making people order their packs easy to hard cos sometimes its nice to play things all over the place having an easy level following a difficult level can be a relief.

I wouldn't suggest that any such system be enforced, or even obligatory. I'd simply suggest that it be there as a universal indicator of the pack's difficulty, should the author want that to be taken into account by players, and particularly - for the purposes of compiling a leaderboard.

Choosing not to give your pack a difficulty rating is as valid a creative decision as any, and certainly shouldn't preclude anyone from playing your pack - it can simply be a "surprise", which I'm sure many players would enjoy from time to time

However, those who pride themselves on creating particularly challenging levels, or those who simply want to attract players by giving an indication of how difficult (or not) the pack is expected to be, can do so in a more standardised way that the majority of the community agree on and understand. It would only take a few creators to begin using the system and it would either catch on or not; I wouldn't expect everyone to use it, in any case.

Lemmicks might be considered in that range as well.

For sure, I'd probably even put Lemmicks into "4 and up" territory - most of those levels require a specific solution that you can either figure out or not, I don't remember there being any levels that the player can just riff with. As we've previously discussed, I think a rank or two of 2 - 3 rated levels would not have gone amiss!

But yeah, any-way-you-want levels, or those that leave you with many spare skills at the end - levels which you label as difficulty 1 or 2... A pointless waste of the player's time. If you can do basically anything, it feels like nothing you do matters. Any decision you make on a level can be the correct one. And if you make a false decision, you have plenty of skills left to salvage the attempt without even having to rewind.

I totally disagree. The best way to learn anything is to get creative with it. These kind of levels allow the player to do that; they can explore the game at their own pace, in their own way, and this can be its own reward. Of course, this is obviously best for beginners and novices, but even experienced players can stumble upon new tricks, new ways of doing things, ways to optimise skill usage, by being given the freedom to explore in a less restricted setting. Hardcore puzzles are great, and offer a lot of satisfaction when you complete them, for sure, but easier levels have their place. They also help to give a player momentum and, as Mantha said, a bit of "relief" in between the more difficult ones.

Even worse, when a bunch of levels can be solved in the same way (e.g. with Basher-cancel-staircases), it also gets repetitive

To be fair, I'd consider Basher-cancelling a comparitively advanced trick that I absolutely wouldn't expect beginners to know about, and certainly wouldn't feature in a "1-2" rated level, or even a "3". Also, repetitiveness is possible at any level of difficulty.

if we tried to assign difficulty ratings to individual levels, assigning the ratings would be a lot more work (so fewer people would bother)

Absolutely. I think the simplest way to assign a rating would be to rate the most difficult level in the rank. That way, the rank is (in theory) never more difficult than its given rating, although it can of course be less so.

Levels can have one specific solution and be very easy: "Just dig"

That's why I added "and relatively obscure" to the definition of a "4" level

That said, I agree that the definitions should be treated as loosely as possible, and it is very much a spectrum, as you've discussed.

In general, difficulty is a combination of length of solution; obscurity of tricks involved; temptingness of red herrings ("entropy"); how much leeway you have in terms of skills, save requirement and time; and difficulty of execution.

Absolutely. I'd expect that the definitions could be further quantified by these sorts of things, to a greater or lesser extent. Also, if an author has created a level which would otherwise be straightforward if not for, say, a tight time limit or use of red herrings, then I would expect that they would know to rate it a bit higher in terms of difficulty even though it might otherwise "fit" the definition of a lower rating.

As always, the definitions I've suggested are merely a starting point on which to build, or perhaps from which to create a comparitive starting point based on whatever factors arise from discussion.

[Strato Incendus]

They also help to give a player momentum and, as Mantha said, a bit of "relief" in between the more difficult ones.

Oh, believe me, I'm the first one to advocate for breather levels in between difficult ones! Some other people criticise that as a "fluctuating difficulty curve"; I think it helps maintaining player motivation and allows the cognitive resources to regenerate.

However, such a "breather-level" pattern is usually not how these any-way-you-want levels are employed. They're usually back to back, clogging up the lower ranks. Meaning, and experienced player could basically take the chance and just skip rank 1, possibly also rank 2. But of course, we don't do that, because we don't want to be "philistines" .

Both issues boil down to the same thing, though:
You simply can't starve people of their dopamine hits for too many levels in a row. If you do, they'll quit. Both too many easy levels AND too many difficult levels in a row lead to this same result.
Where your personal level of tolerance in this regard lies, that of course will differ from player to player.

[Proxima]

Meaning, an experienced player could basically take the chance and just skip rank 1, possibly also rank 2. But of course, we don't do that, because we don't want to be "philistines" .

Who's "we"? I've often seen experienced players, such as Icho, jump in at the top rank of multi-rank packs, and no-one ever criticises them for it.

Both issues boil down to the same thing, though:
You simply can't starve people of their dopamine hits for too many levels in a row. If you do, they'll quit. Both too many easy levels AND too many difficult levels in a row lead to this same result.
Where your personal level of tolerance in this regard lies, that of course will differ from player to player.

Sure, but it seems like you complain a lot about levels that are too easy for you, when you are simply not the intended audience, just as you are not the intended audience for levels, ranks or packs that are too hard for you. There is nothing wrong with authors producing easy content, just as there's nothing wrong with authors producing difficult content. What really matters is having some kind of system to help players find content at the level they want to play at.

[mantha16]

dont get me wrong i like the idea of a rating system its just i rate all mine as easy and icho said some of my levels arent as easy as i think so id have no idea how to rate them lol

[Dullstar]

I often think it's hard to rate your own levels' difficulties. You know the solution, after all, and it might be more (or less) well-hidden than you thought. My tendency seems to be to underestimate difficulty.

[mantha16]

maybe testers could offer suggested ratings?