[SUG][NeoLemmix Level Packs] Difficulty Ratings Scale

[IchoTolot]

Just to be clear:

My main problem in the pack topics was mostly that the scale here was already seen as 100% forum wide accepted and other users who were not part of the discussion were already told "Hey this is what we use!" despite no great forum wide agreement!

What do people think about the idea of representing the various stages of difficult as numbers (or letters)?

So, the above scale could be represented thus:

1 - Beginner
2 - Easy
3 - Medium
4 - Hard
5 - Advanced
6 - Extreme

That way, packs could be labelled [Difficulty: 1-4] or [Difficulty: 5-6]

I'm not sure how I feel about this idea. I think I'm suggesting it on the grounds that it would save space in the topic title whilst indicating the range of difficulty presented by the pack, but words are always more helpful, particularly if someone isn't familiar with the suggested scale.

This was the last post before the advertisement campaign through new pack topics began!

I highly veto against making it already marked as widely accepted at this point!



But back to the problem at hand:

1.) If we come to a scale widely accepted: I think it's best to use moderator powers to roll out the scale widely and base the rating on author+user experience. Packs already rated for a long time should receive the same/similar rating if no big objections in some cases are made.

2.) My problem with the currently suggested scale: The advanced case. What is advanced? It is more unclear than you think! It could come just after medium and before hard for example (from here on you need more "advanced" stuff).
We need a term that is crytal clear here! To adjust the scale another rank is needed. We could add one between "hard" and "extreme", but alse between "medium" and "hard" could work. We just need a term that is crystal clear in its meaning so that every user can see right away where it stands in the scale without seeing the whole scale before. "Advanced" does not achieve this.

3.) Just using plain number like the suggested 1-4 for example does not have much information value. New players don't know the scope of the scale or if 4 still means easy. And no, they won't likely read the topic for the scale even if it's pinned! That's why we need clear terminology!

[IchoTolot]

Now another post about level difficulty and why you cant simply put a graph on it.

From my experience the following holds true: For a good evaluation on a packs difficulty you won't get around having veteran users play the pack!

Simply saying if we have many skills and most/all of them have to be uses = hard level does not reflect the nature of levels.

Of course there are quite a few cases where this indeed fits, but you still would have too many exceptions.

I have seen many cases like that, where the majority of skill assignments (even if they were many) are crystal clear. Like if you have a gap with no terrain around it a builder has to go there and you have a skill that is 100% locked in place. Like that levels can get easy very fast if a lot of the skills are effectively locked in place.

Lemmings is PSPACE-complete (https://arxiv.org/abs/1202.6581) and it is very hard to judge its difficulty on easy countable variables. Let's just say you can programm chess computers, but a computer solving all kinds of lemming levels is a far distant dream if not even impossible.

A better approach would be using more vague but also more fitting terms like "entropy".

As I would double post my exaplanations, here I will simply link an older topic where we already discussed the matter: https://www.lemmingsforums.net/index.php?topic=3341.0

My first post in there describes the terminology.

We also had another topic discussion the option to reduce said entropy: https://www.lemmingsforums.net/index.php?topic=4023.0


As said term is hard to grasp the best difficulty evaluation is to have some experienced players go through the packs and judge their difficulty. Also, more experienced pack authors who played and solved many levels are more likely to judge the pack's difficulty closer to the experienced difficulty by the user from my experience.

[WillLem]

This post isn't as long as it looks, there are a lot of quotes in here! Still, grab a cuppa first if you're up for reading it

That said, I'm not really sure what we can do about it ... Theorycrafting based on "number of assignments" or anything similar doesn't really capture the essence of difficulty.

Sure, but that's why we're having this discussion, to try to "capture the essence of difficulty" and then make it into something workable.

Maybe we should encourage more of the people who play packs, rather than the authors, to indicate where they believe a pack's difficulty would fall on a scale?

Now you're talking. That's a brilliant idea! The only problem I can see with it is that it necessarily relies on the availability of players in such a position, meaning that there could be some length of time between a pack's release and its difficulty rating being assigned. A minor problem, maybe, but a problem nonetheless.

I still think that encouraging designers to get a handle on the difficulty of their own levels is important, not just for the sake of rating the pack, but for their own benefit as content creators; it can only help to improve and refine their output, as Strato has demonstrated.

Imagine there is a custom level pack named "A".  A's first rank has [medium] difficulty overall and final rank has [hard] difficulty overall. But the easiest levels of A have [easy] difficulty and the hardest levels of A have [very hard] difficulty. Then between [Medium-Hard] and [Easy-Very Hard], which one is more appropriate indication?

I always think that a pack should indicate its hardest difficulty; I'd go with "Medium-Very Hard" for the example you've given. Reason: it shows that the player can expect even the first rank to get fairly tough, and the pack as a whole to go into "Very Hard" territory by the end. It just seems more useful to the player.

I'm more than willing to share my opinion on whether the pack's difficulty is indicated properly / which pack is harder between pack A and pack B or something like that once we establish difficulty scale everyone can agree with.

Great stuff! I'm sure your opinion will be very highly valued

I highly veto against making it already marked as widely accepted at this point!

Fine, but you could have said something much sooner (i.e. when this conversation was last being had). I took the lack of replies following mine as a sign that there were no further disagreements or discussions to be had.

I think it's best to use moderator powers to roll out the scale widely and base the rating on author+user experience.

Awesome! This would be very helpful

Packs already rated for a long time should receive the same/similar rating if no big objections in some cases are made.

Why? Longevity has no value when there has been no prior, explicitly agreed understanding. In fact, the problem has come about for this very reason, so any previous scale should automatically be dismissed and re-assessed based on new understanding.

2.) My problem with the currently suggested scale: The advanced case
...
We could add one between "hard" and "extreme", but alse between "medium" and "hard" could work. We just need a term that is crystal clear in its meaning so that every user can see right away where it stands in the scale without seeing the whole scale before. "Advanced" does not achieve this.

Fair enough. I think that adding something between "Medium" and "Hard" might be better in this instance, since the tendency is for creators to underestimate difficulty; increasing terminology towards the lower end of the scale is therefore more useful.

I also agree that terminology (as opposed to characters) is far more useful; I'd be happy to scrap any notion of numbered/lettered ratings.

From my experience the following holds true: For a good evaluation on a packs difficulty you won't get around having veteran users play the pack!

I'm not entirely sure what you mean here to be honest...

Lemmings is PSPACE-complete (https://arxiv.org/abs/1202.6581)

My brain turned into melted marshmallow whilst trying to read and understand this

Let's just say you can programm chess computers, but a computer solving all kinds of lemming levels is a far distant dream if not even impossible.

Ah, OK. That makes sense. Thanks

A better approach would be using more vague but also more fitting terms like "entropy".

I see what you mean here. It basically adds finesse to my proposal that the "skills:intended uses" ratio is a good indicator of difficulty, since it accounts for exactly how those skills are intended to be used, and how well concealed those uses are in the level's design.

I still think that any level which only provides the skills required to solve it should almost never be considered "Easy." Beginner/tutorial levels are an obvious exception, as are levels for which the skills have obvious "use this skill here" points (e.g. a level with a swimming pool, a gap and a pillar which provides 1 Swimmer, 1 Builder and 1 Basher, or similar setup).

Even "Medium" difficulty should really only start to introduce the kind of puzzles which require a very particular solution to be used. Exactly how difficult that solution is to find would then move the level higher up the scale.

Your point about "Trickery" is also compelling: levels which rely on tricks to gain a "Hard" rating are almost always no longer "Hard" when a player knows the trick, meaning that the rating is entirely based on the player's level of experience (which, I would suggest, should not necessarily be the case).

As you've said: how well the trick is concealed, or the manner in which it's used in the level contribute far more to the level's difficulty than simply the trick's presence in the level. Such levels are, therefore, even harder to rate in terms of difficulty.

the best difficulty evaluation is to have some experienced players go through the packs and judge their difficulty. Also, more experienced pack authors who played and solved many levels are more likely to judge the pack's difficulty closer to the experienced difficulty by the user from my experience.

Agreed, again. This is a very good idea, and if players are happy to do this, then all we need to do is agree on a scale and begin to implement it as closely as possible.

It's worth remembering, though, that experienced/high-ability solvers are more likely to underestimate difficulty; expanding the scale to incorporate more degrees at the lower end of the scale ought to combat this, otherwise a lot of levels will end up getting lumped together as "Easy", which is exactly the problem we have now.

[IchoTolot]

What about replacing "Advanced" with "Harder"

"Harder" is clearly more difficult than hard and I would argue clearly lower than "Extreme".

[WillLem]

What about replacing "Advanced" with "Harder"

"Harder" is clearly more difficult than hard and I would argue clearly lower than "Extreme".

Maybe if we decide which levels of difficulty need a definition first, then the words will come naturally.