[SUG] Update contest format to include/encourage more players/designers

[WillLem]

Here's a more extreme example to illustrate what I'm proposing:

Designer A's Level A gets 100 votes
Designer A's Level B gets 98 votes
Designer B's Level A gets 99 votes
Designer C's Level A gets 97 votes
Designer D's Level A gets 1 vote
Designer E's Level A gets 1 vote
...
Designer Z's Level A gets 1 vote

Should Designer C be awarded 3rd place, or Designer A?

[IchoTolot]

Here's a more extreme example to illustrate what I'm proposing:

Designer A's Level A gets 100 votes
Designer A's Level B gets 98 votes
Designer B's Level A gets 99 votes
Designer C's Level A gets 97 votes
Designer D's Level A gets 1 vote
Designer E's Level A gets 1 vote
...
Designer Z's Level A gets 1 vote

Should Designer C be awarded 3rd place, or Designer A?

I would still say Designer A.

Again in my book the contest is purely about levels, not designers and I would not call it a problem just a different focus.

unbalanced in favour of only the most prolific and popular designers

This I would 100% call not true!

The best/most popular levels win. The author is second fiddle here.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

But that gave me another idea:

How about we only post the levels without giving the authors name!

The authors will only be named after the final vote! This would remove next to all influence of the authors.

[Proxima]

In DROD contests, we do as WillLem is suggesting -- if the same designer places twice in the top 3 then their lower entry isn't counted and the 4th place moves up to 3rd.

This does make me wonder. Instinctively, I was against doing this for Lemmings contests, but why is that, if I'm okay with it for DROD?

I guess the main reason is that I don't see what it would achieve. There are no prizes for 2nd and 3rd place anyway. So I don't see how changing the way results are presented would do anything to encourage more designers to take part. It's always the case that as a new entrant into contests, you have a low chance of winning against the Icho/Armani juggernaut; but not no chance, and the best way to win a contest is to have patience and keep trying. Not everyone has the patience and time to do this.

[namida]

be unbalanced in favour of only the most prolific and popular designers, as it demonstrably has been for quite some time.

How is it unbalanced in their favor though? They win because they consistently submit the best levels. There's no special rule that gives IchoTolot or Armani an advantage over other levels; they're just really good at making levels.

Honestly, this is almost starting to feel like a "let's give other people participation trophies" and/or "let's list more than the top 3" angle in practice (as in practice, all we'd be doing is listing 4th place but calling it 3rd). Indeed, perhaps it's better to make no change to how places are awarded, but to adopt the suggestion of keeping the entries anonymous until results are announced (assuming a practical and reasonable way to handle updates can be found). Maybe that should be trialed for one or two contests before deciding on whether to make it permanent?

To be clear, the only idea here that I'm strongly opposed to (and even this, it's up to Icho and kaywhyn to make the final call, not me), is the proposal of determining multiple of the top places via a single voting round (rather than an elimination format). Mass-voting is okay in the earlier rounds but I really think (given that STV-style voting isn't feasible) elimination is the only right way to handle the closing rounds.

[WillLem]

Again in my book the contest is purely about levels, not designers and I would not call it a problem just a different focus.

Those who often win are unlikely to see a problem with the current system

unbalanced in favour of only the most prolific and popular designers

This I would 100% call not true!

The best/most popular levels win. The author is second fiddle here.

It obviously isn't "100% not true". The most prolific and popular designers tend to win the contests; the data proves that. It's probably at least 30% true, otherwise why would this topic need to exist? I have very little stock in the level design contests themselves; my goal is to help improve Forum interaction.

How about we only post the levels without giving the authors name!

The authors will only be named after the final vote! This would remove next to all influence of the authors.

Very good idea, I'd be up for at least trying this for a contest or two to see how it goes.

However, I'd still want to see 3 names in the final results.

I guess the main reason is that I don't see what it would achieve. There are no prizes for 2nd and 3rd place anyway. So I don't see how changing the way results are presented would do anything to encourage more designers to take part.

The reward is being placed in the top 3, and having your name displayed there. A bronze medal is still a medal, and many people would benefit from having one of their levels recognised alongside the usual winners.

It's always the case that as a new entrant into contests, you have a low chance of winning against the Icho/Armani juggernaut; but not no chance, and the best way to win a contest is to have patience and keep trying. Not everyone has the patience and time to do this.

Right, which is why we need to re-balance the way the results are gathered. Those with multiple levels in the contest shouldn't have an automatic advantage over those with only one level in the contest. Any system that allows designers to enter more than one level optionally and then rewards the same designer twice is therefore unfair, or at least unbalanced. Ideally, all designers would enter the same number of levels; in that system, rewarding the same designer more than once would at least be more fair, but still problematic IMHO.

How is it unbalanced in their favor though? They win because they consistently submit the best levels. There's no special rule that gives IchoTolot or Armani an advantage over other levels; they're just really good at making levels.

Their reputation as good level designers perhaps affords their levels a certain amount of bias when it comes to voting. There's no way to prove this really, but I would be interested to see what would happen if we did hold a contest where the level author wasn't disclosed until the end of the contest.

Honestly, this is almost starting to feel like a "let's give other people participation trophies" and/or "let's list more than the top 3" angle in practice (as in practice, all we'd be doing is listing 4th place but calling it 3rd).

I'm disappointed that this has come up. I've been trying to avoid that particular way of viewing what I'm suggesting here because it cheapens what I'm actually trying to achieve. My goal is simple: I want more people to feel encouraged to participate in the contests, thus (hopefully) increasing Forum interaction and preventing stagnation of the contest format. One way of doing this is to award the top 3 designers, rather than the top 3 levels.

There may be other ways.

Indeed, perhaps it's better to make no change to how places are awarded, but to adopt the suggestion of keeping the entries anonymous until results are announced (assuming a practical and reasonable way to handle updates can be found).

This would certainly help to remove any possible bias during the voting stages and I'm all for it, but I'd personally still want to see 3 designers being awarded first, second and third place rather than 3 levels.

If we keep awarding levels rather than designers, we can end up with a situation where a single designer takes all 3 of the top spots. Even if it rarely (if ever) happens in practice, it's still a serious flaw in the contest format IMO.

The only way to fix it would be to allow one level per designer per contest. But, people don't want that either. So, the only way to keep the end results fair and balanced is to reward the best 3 designers regardless of level placement. We don't need to call it "first", "second" and "third" place if that's bothersome; instead we can award Gold, Silver and Bronze, with a simple rule that the same designer can't be awarded more than one medal. It isn't the same as arbitrarily bumping 4th place up to third, it's literally looking at which 3 designers' levels appear at the top of the voting list, ignoring the number of levels themselves, and then making the awards.

Maybe this is one to at least try and see how it plays out in practice. It's not really asking a lot, and those who participated in the contests might enjoy the recognition they might otherwise not have had.

Mass-voting is okay in the earlier rounds but I really think (given that STV-style voting isn't feasible) elimination is the only right way to handle the closing rounds.

Why? I'm not sure I understand the difference it makes. If anything, it just means that the same people have to keep voting for the same levels. Or, if the level they voted for doesn't make it into the final round, they then only have the choice to either vote for a level they didn't think should win, or not vote.

Ideally, there should be one voting round. The basis of your vote should be "I think this level should win the contest". Otherwise, why vote at all?

I feel less strongly about this than I do about rewarding 3 designers rather than 3 levels, though. The voting system itself is probably a different conversation for a different topic. All I will say here is that multiple rounds are a bit of a bore, and - importantly - sporadic Forum users are likely to miss entire rounds, which is another glaring flaw in the contest format.

[namida]

If we keep awarding levels rather than designers, we can end up with a situation where a single designer takes all 3 of the top spots. Even if it rarely (if ever) happens in practice, it's still a serious flaw in the contest format IMO.

And as long as it's three different levels, I don't really see the issue. Why does a level become less worthy of winning just on the basis that an entirely seperate level from the same author was also really good?

Why? I'm not sure I understand the difference it makes. If anything, it just means that the same people have to keep voting for the same levels. Or, if the level they voted for doesn't make it into the final round, they then only have the choice to either vote for a level they didn't think should win, or not vote.

Ideally, there should be one voting round. The basis of your vote should be "I think this level should win the contest". Otherwise, why vote at all?

I already explained this in reply #11.

[WillLem]

And as long as it's three different levels, I don't really see the issue. Why does a level become less worthy of winning just on the basis that an entirely seperate level from the same author was also really good?

The issue is that we're only looking at levels by 1 designer. If you don't think that's a problem, then we simply disagree and there isn't really anywhere that the conversation can go from there.

I already explained this in reply #11.

I disagree with what you've said in that explanation. People should vote for whichever level they think should win out of all the available levels. Whichever level gets the most votes places top in the list, and so on.

If we allow multiple votes per user, that should solve the problem outlined in your reply. Here's how:

There are 100 users. Level A divides people; 50 love it, 50 hate it. It gets 50 votes. Level B is more universally liked, and gets 70 votes. Assuming that Level B was the second choice of the 50 people that loved Level A, 20 of the people that hated Level A helped nudge Level B into the winning position.

[WillLem]

OK, maybe the contests simply aren't for me. I'm done attempting to provide anything constructive to this debate, or to the contest format.

My suggestions aren't getting any support, and the valid points I've raised regarding the contest's problematic structure and voting system are being completely ignored. For those reasons, this isn't something I want to pursue any further.

[IchoTolot]

I noticed that this was simply locked, but I will make a final statement anyway.

First, I don't see a sign that any statements were ignored here, just discussed rather than automatically changes being implemented based on these.

Also, both methods of either awarding the designer or the level are valid contest methods. Neither of those are problematic in themself, but just have a different focus. Both work and you can make contests with both.

So why did we adapt the level focus?

- The reason I see why people choose to participate in a level design contest is not simply to get a medal/placing in the top 3. It is to create levels with interesting rulesets and play other levels with these rulesets. Winning is secondary. The joy and also excuse to make and play new levels takes the forefront.

- A hint that this holds indeed true is that in the past only 1 or 2 levels could be submittet and people actively wanted to be able to submit for all 3 rules just to A) create more levels and B) play more levels. Again, this shows the interest lies more in the levels not the designers.

As a result, the levels should be awarded in the case for this contest which I also see reflected in the poll.

Those who often win are unlikely to see a problem with the current system

This I highly find unfair to just accuse me of not wanting to change things because I win contests.

A) Over the time (even before you joined the forum) first namida and then myself consistently adapted the contest design to address widespread public desires. Adopting the 3 rule system, reworked the voting to reduce tiebreaker rounds, adding the possibility to update your entries, maling a compilation pack were all such adaptations as an example.

B) I really like making contest levels - so of course I create levels for all 3 rules. Yes, this increases the chance that one of my levels wins, but the core of the reason why I am doing this is not the victory, but because I like to create levels after rulesets - sometimes I even make entries with suboptimal chances to get far because I like a certain gimmik. And with consistently high effort being placed in creating these levels results come around.
I can exclude myself from voting even, but I highly doubt that this would bring any difference in the number of contest entries!

C)

"The most prolific and popular designers tend to win the contests; the data proves that."

I think you are interpreting the data wrong and that is why you see the contest design as problematic. "popular" I highly doubt that this has anything to do with it. People are not just voting like "ah that is Icho's level that one gets my vote" - I still have voting flops!. Again, Armani and myself entered all contests with 3 entries and gathered lots of experience in adapting to rules and make quality levels in terms off appearance, difficulty and creative solutions. Of course this will most likely show results as more time passes and it is quite hard to beat.

I really really try to make it to up to as many people as I can, but over the years I ever so often hit my limitations. I learned to accept that I can't satisfy everyone and that not everyone will approve the decisions that I made. But in the end decisions need to be made.

As a result from this (apart from ensuring a clear level top 3 every time):

I want to make a testrun with the anonymous playing/voting phase. If it goes poorly we can always scrap it again for the next contest or even change it back on the fly.

[Simon]

this isn't something I want to pursue any further.

Please don't lock it nonetheless. There will be more useful discussion.

-- Simon

[WillLem]

Please don't lock it nonetheless. There will be more useful discussion.

Unlocked so discussion can continue, but I have nothing more to add other that what's already been said.

Icho, apologies if the comments felt directed at you personally, that wasn't my intention. I hope that the topic might prove to be helpful at some point.

[Simon]

Thanks!

You count the number of designers who have shown (= placed 1st, 2nd, or 3rd) exactly once in the past n contests, and you value a high count for this. Let's assume Nessy did 50 % worse than he really did (2 showings), i.e., assume he showed only once. This will improve your metric by 33 % or by 66 %, depending on who takes the now-open slot. Is that desirable?

Apologies, but I'm not sure I understand the question. Could you rephrase?

My point was that you didn't pick the best metric (evaluation function) to turn your raw data into a snappy conclusion.

Your metric was: We count the number of different designers show exactly once across the first 3 places of the past ~20 contests. We want to increase this count.

My counterexample was: Nessy showed twice, and he's not so prolific. According to your metric, it would be better if Nessy showed only once instead of twice. That will bump your count (of designers who showed exactly once) from 3 to 4, i.e., we improve the metric by 33 %. This would suggest that we should somehow hamper designers from showing a second time in a later contest.

I thought: A better metric for your point is total number of different designers across the first 3 places of the past ~20 contests.

unbalanced in favour of only the most prolific and popular designers

This I would 100% call not true!
The best/most popular levels win. The author is second fiddle here.

I'd only give it 95 % to be untrue. :-) The voting system has indeed a slight tendency to favor the same designer within the same contest.

This is because the initial voting buckets aren't random. You make three buckets, one per rule, and vote within each bucket. Later, survivors enter one big bucket for the semifinals. Assume designer X's level from bucket 1 survives. This doesn't affect buckets 2 and 3, but it makes all other levels from bucket 1 less likely to enter the semifinals. Now, because every designer can only enter once per rule, you have a correlation: Each of X's other levels (which cannot be in bucket 1) are individually more likely to enter semifinals than designer Y's levels individually (because Y can have one of his levels in bucket 1).

I don't think this slight tendency is a problem. If you still want to eliminate this slight tendency, don't make a bucket per rule. Make random buckets across all rules. Or change the voting system altogether.

-- Simon

[namida]

I like the bucket rule because it also gives an idea of the best levels in each category. This could be done with (maybe post-contest so it doesn't delay the overall results) an extra poll or two, though, if buckets were mixed.

[Dullstar]

It's already been brought up but I would like to explicitly mention support for the idea of changing the rule design from 1st place makes all 3 rules to each of the top 3 makes 1 rule; if someone is trying to earn that reward, hitting top 3 should be more attainable.

[namida]

To take this concept even further: How important are the mixed rounds, especially given that some rules inherently produce stronger levels than others?

Perhaps a way forward is to get rid of the three rules system, in favor of more frequent contests with a single rule. These in turn could be run in (near-)parallel - essentially, not much different from the current setup, except that each rule is treated as a seperate contest and we just find the winner for each rule, without comparing the rules to each other. This in turn works nicely with "each winner picks one rule", and gets rid of any debate over whether winners should be levels or authors (as we'd be back to one entry per author under such a setup, aside from LOTY which is almost universally agreed should be purely about levels). Voting in turn could be reduced to two or three rounds - all levels for a rule, (maybe) top 3, and finally top 2. Yes, for a set of three rules this would be six or nine polls, but each rule can be run in parallel rather than having to wait for the last one to finish voting, if desired.

And this still allows for creators to get the frequent opportunities with a variety of rules, that was a motivation for the "2 of 3 rules" and eventually "you can enter all 3" changes that were popular.

The one (very minor) concern that comes to mind is, we tend to like having old rules repeated once in a while. But are people with an opportunity to pick just one rule, going to want to sacrifice that on repeating an old one? (Or conversely, will we run into an issue of too many repeats?). Maybe this isn't a problem; if we go this route, I suggest at first just let whatever happens happens and see how it works out, but if we end up feeling that repeats are too rare or too common, one approach could be along the lines of "Rule 1 must always be original; if you win on this, you must come up with an original rule. Rule 2 must always be a repeat, if you win on this, you must pick a past rule to repeat. Rule 3 is free choice and you can go either way."