Should we change the contest voting system?

[Proxima]

Back in 2019, there was an overhaul to contest voting, because the old system often led to a huge number of voting rounds, which is a lot of work for the organiser and puts a heavy burden on voters if they want to commit to taking part in the whole thing, which can lead to excluding those who don't.

The current voting system, while definitely better, is still flawed. We now have only one vote for each rule; that's great. But if the numbers fall out unluckily, we can end up with a situation like that in Contest #29, where a mixed group of six levels needs five or even six voting rounds to establish a winner.

So what's the solution? Well, first we need to decide what the goals of the voting system are. At the moment, they seem to be:
* To establish an overall winner -- obviously the most important goal.
* To establish a winner for each rule, since this carries a certain amount of prestige in itself, especially for rules like "Make a Tame level" where entries have no chance of winning overall.
* To establish overall 2nd and 3rd place -- is this still felt to be important? There are no prizes for these places, but do those who regularly take part in contests care about them?
* To separate out the top three and then rank them, so that the overall winner is not decided until the last voting round.

If all of these are still felt to be necessary, my suggestion would be:

* As in the current system, each rule has one vote only, no tiebreakers.
* The first round of the mixed group aims to establish a top 3. If this isn't possible because of ties, a tiebreaker is held.

For example, with the votes from contest #29 (6 for A, 2 for B, 3 each for C, D, E, F), A would be into the final, B would be out, and a tiebreaker would be held to find the top two of C, D, E, F.

Ideally, this system limits the mixed group to three rounds plus one possible tiebreaker. Unfortunately, because of the requirement that an exact number of entries qualify for the top three, it's still possible for the tiebreaker to need a tiebreaker (e.g. C, D, E, F get 7, 4, 4, 3 votes respectively). I would suggest that in that case, we could just be okay with not having a third place (or having a tied third place), so that the mixed group cannot have more than four votes.

[Simon]

For me, the main function of the pools is to generate a few cool-down weeks between playing the entries for contest n and seeing the rules for contest n + 1. I enjoy the time off. Still, inefficient polling nags me naturally, i.e., you've presented a nice design problem.

I don't even know how Icho prunes the list between rounds. It looks like he drops the bottom ~third between rounds, and errs on the side of keeping levels instead of dropping levels. I wouldn't mind dropping levels faster than that.

The more rounds you run, the more voters will see other people's votes of earlier rounds. This may change opinion about levels. This can be objectively good (we sleep over it, we look at the entry a second time), objectively bad (we blindly follow others, we cast an uninformed vote in hope of doing the least damage), or neutral (feelings fluctuate and we merely take several samples). Example: When Rhizome won Contest 27, it didn't collect the most votes in early rounds, but it did in the later rounds. I hope was for the first reason.

The first round of the mixed group aims to establish a top 3.

Yes, if you want to crown a single winner and award a prize, it's good to run the final top 3 or top 2. I believe I don't mind what happens before this.

-- Simon

[WillLem]

My 2 pennies' worth on this.

As someone who has never won a level design or LOTY contest, sometimes it's good to see how far you can make it through the contest. So, more rounds can mean that those who don't win at least get to make it through a few rounds and feel some satisfaction/achievement from that.

With that said - yes, lots of polling/voting can become annoying, and the overall goal here seems to be to reduce this.

A good system might be to leave all levels up for voting for the entire contest; this might incentivise more interaction from Forum members who might otherwise not bother to vote, since the goal would become to push your preferred level as high up the list as possible. It would also provide a more meaningful representation of what percentage of voters actually prefer the level which ends up winning - i.e. it prevents users having to choose from a small pool of levels in the final round which they might not have voted for in the first round.

We could also then reward the top 3 levels, for example, rather than only the eventual winner. There is already a choice of 3 prizes of the level design contest, why not give these out to the top 3 winning levels? Obviously, the winner gets first choice of prize, and so on.

This might also incentivise more people to take part, knowing that getting into the top 3 is a worthwhile pursuit.



I would also strongly advocate for a "one level entry per author" and "one winner per rule" system.

For LOTY, nominations can still be many levels per person, but the nominated author should then get to choose which of their nominated levels they want to put forward, and that's their contest entry.

For level design contests, authors must choose which contest rule they want to take part in carefully, and make an entry for that rule and that rule only. Then, if we go with a "one winner per rule" system, that means that 3 different authors will get a win. I'd then hand the choice of 1st, 2nd and 3rd place over to the level design contest author - since they proposed the rules, perhaps they should get the final say in which level best meets the parameters of their proposed rule. This removes voting entirely from this part of the contest.

I believe this would encourage more people to take part in these contests.

[Silken Healer]

There's only 2 prizes since the news ticker was removed

[WillLem]

There's only 2 prizes since the news ticker was removed

Maybe the prize could be choosing a rule in the next contest, then?

That way, the 3 rules are shared between the 3 winners, 1 for each previous rule.

[Dullstar]

I don't think the forum software supports it out of the box, if at all, but I wonder if there's a way we could implement ranked choice.

[Proxima]

Thanks to kaywhyn and Icho for reminding me of this topic The discussion was left unfinished, and I believe it's time to come back to it.

Two things have happened since. WillLem started a separate discussion about contest voting, and while it didn't achieve his objective, it did consolidate a couple of things. The community definitely likes the "each author gets one level per rule" system and likes establishing the winner of each rule and the overall top 3, so it's important for any reform of the voting system to keep these.

And Contest #30 showed that reform is needed, because worst cases are not just theoretical; they keep happening. A vote - tiebreaker - vote - tiebreaker - vote cycle, combined with the group votes, resulted in eight votes for a single contest. Not as bad as things used to be, but still bad.

This is too many. It's unnecessary work for the organiser and voters, and it leads to people losing interest, and more chance that people are drifting in and out of the votes and they don't represent the opinions of any one group. So I've come up with two proposals for reforms:

1. A Simple System

* As currently, each rule has one voting round, no tiebreakers. Drop the "survival rate" maths and have the top two of each rule go through: after all, if the second-best Rule 1 is also the second-best overall, it doesn't deserve to lose out just because there were only three Rule 1 entries.

* The mixed group is also one voting round. The highest-place from this round wins, and the second and third-highest come second and third. The only time there is a tiebreaker is if multiple levels tie for first; if there's a tie for second then we just announce the results as a tied second place.

Response to Namida's Objection
(in spoiler tags because of length)

Spoiler

In Will's topic, namida objected to a proposal that was basically the same as the above, on the grounds that it might be inaccurate because it only takes account of people's first preferences.

For example, suppose there are six levels in the mixed group, A, B, C, D, E and F, and they get 10, 9, 8, 3, 2, 1 first-place votes respectively.

By the above system, A wins. But what if all the D, E and F voters had C as their second preference and hated A? By elimination voting, this would give C six extra votes and put it first.

Response to this:

Firstly, Arrow's theorem says that there is no possible completely fair way to derive group preferences from individual preferences if there are more than two alternatives. Any possible system has advantages and disadvantages. You might think elimination voting is fairer because it takes second preferences into account -- but doesn't it seem right that second preferences should count as less important than first preferences? But how can you decide on a "right" value for how much less?

Keeping with the same example: suppose all the B voters have A as their second preference and hate C. By elimination voting, this doesn't count for anything because B is still in the running, and so the second preferences from the D-E-F voters still push C ahead of A. Why is it fair that some people's second preferences are counted and others' aren't?

Ultimately, the choice between voting systems has to come down to a number of considerations including practicality, and that's why I strongly argue for the above system even though it could be argued that it doesn't produce the best result in some artificial edge cases. If A wowed the voters enough that 10 people placed it first, and no other level got more than that, that's a pretty compelling argument that it does deserve first place.

2. A Compromise System

In case people still really dislike the above proposal, I also thought of this compromise:

* As above, one vote for each rule with two levels going through, no tiebreakers.

* The mixed group has two rounds: one to get the levels down to a top three, and one to rank those three.

* Tiebreakers are replaced with run-offs. If there is a tied third place, hold a run-off between the levels that tied for third, and the winner doesn't re-enter the voting; it just gets third place and the final is between just first and second.

This system still has the possibility of worst cases that need tiebreakers (e.g. a four-way tie for first), but these should become much rarer.

[namida]

One other possibility that comes to mind is, during the non-mixed phase, hold all three rules' votes simultaneously. This wouldn't cut down on the number of votes, but would get them done much quicker (and can also potentially be combined with other ideas).

[IchoTolot]

Personally I like the "2. A Compromise System" proposal.

Comments:

- One hard objection is one point: "Drop the "survival rate" maths and have the top two of each rule go through:" - This can get highly unfair in some cases! Let's say one rule is pretty popular and another is not. One gets 10 levels and one 4. This would mean that in one case half of the levels survive and in the other only a fifth.
The "maths" take me like one minute and is no real extra work and just lessens possible rule bias. Different survival rates also don't cause extra voting rounds.

- "The mixed group is also one voting round." Main reason I am not siding with "A Simple System". I really don't like voting on large groups that straight up determine final positions. It often results in major ties and less precise outcomes.
Also, there is a chance that the mixed group is split into 2 when larger ties happen in the voting of the rules.

But back to "2. A Compromise System":

Again, I would still object the "Drop the "survival rate" maths and have the top two of each rule go through:" part.

- "The mixed group has two rounds: one to get the levels down to a top three, and one to rank those three." I can side with the suggestion that the placement of the top 3 is ideally just one round. The first part we already try to do.

- "Tiebreakers are replaced with run-offs. If there is a tied third place, hold a run-off between the levels that tied for third, and the winner doesn't re-enter the voting; it just gets third place and the final is between just first and second." This is the major suggestion for me here, that can effectively reduce the voting time! If there is support for this I can adapt it.

As you said there can still be tiebreakers needed in worst case scenarios.
The most common run-off scenario I would see in "get the levels down to a top three" when multiple levels tie for the 3rd place. In extreme cases where 3+ levels tie for 3rd then in the run-off additional run-offs could even be nessesary.

Anyway I will see what comments pop up further down the line and then make a poll.

[WillLem]

+1 for simple system.

The compromise system still has too many voting rounds. For instance, why do we need a separate round to rank the top 3? The mixed round can possibly determine this on its own.

* To establish a winner for each rule, since this carries a certain amount of prestige in itself, especially for rules like "Make a Tame level" where entries have no chance of winning overall.

This, IMO, is the biggest problem with the contest in general. If people already know which levels are likely to win, why even have a contest?

[Proxima]

This, IMO, is the biggest problem with the contest in general. If people already know which levels* are likely to win, why even have a contest?

"Make a Tame level" and similar rules are rare. It's much more the case that sometimes, we pretty much know that one rule can't win and the real contest is between the other two, rather than knowing which rule will win -- and certainly not which level!

[IchoTolot]

One thing that came to my mind about the "Simple System":

What should happen when we got a clear first and 2nd place, but a tie (or major tie) for the third?

It feels weird to have a vote after the winner is set in stone, but to have like 3+ people being 3rd is not optimal as well in my opinion.
This can also be problematic for major ties for the 2nd place.
I think to have at least somewhat clear finals we need that one little step to the top 3. A single mixed round has too many edge cases here.

In both suggestions implementing run-offs instead of tie-breakers should have the most value.

Discussion has just started please do not count the following as a final statement and just as a first suggestion!

My first proposal would be :

1.) One vote for each rule, no tiebreakers. Still adapt the survival rate for group size.

2.) One mixed round to determine the top 3. Ties will be resolved with a run-off round where the winner here simply lands on the place in question, but does not survive further.

3.) One final round to determine the (remaining) top 3 placements.


So for 3 rules the minimum would be 5 rounds of voting with 1 possible run-off round.

The other simple system would reduce the number from 5 to 4 for comparison, but leaves the top 3 edge cases stated earlier.

[WillLem]

"Make a Tame level" and similar rules are rare. It's much more the case that sometimes, we pretty much know that one rule can't win and the real contest is between the other two

This definitely seems problematic IMO. Is there any way to address this so that contests are more balanced and don't automatically favour one rule over another?

What should happen when we got a clear first and 2nd place, but a tie (or major tie) for the third?

Is it out of the question to do away with tiebreakers and/or run-offs and just award joint 3rd place to multiple contestants? Less voting, more people get recognition for their work and for attracting votes in the first instance.

Still adapt the survival rate for group size.

Strongly against this. If the contest provides 3 rules and 1 rule is less popular in terms of entrants, that doesn't automatically mean that those entries are lower quality than the entries in the more popular rules. If those levels have attracted votes, they should progress through the contest.

Suggestion, then: the mixed round could consist of the top 6 levels overall by number of votes, as opposed to just taking the top 2 (or any other number based on survival rate) from each rule. This eliminates the need for survival rate, and empowers the users' votes (which, IMO, should definitely be a goal in all this).

3.) One final round to determine the (remaining) top 3 placements.

Why? The previous round may have given the top 3 placements. If there are no ties, why another arbitrary round?

[IchoTolot]

"Make a Tame level" and similar rules are rare. It's much more the case that sometimes, we pretty much know that one rule can't win and the real contest is between the other two

This definitely seems problematic IMO. Is there any way to address this so that contests are more balanced and don't automatically favour one rule over another?

I think this could only be solved with: Create you own special contest with suited rules, so that specifically non puzzle focused levels are encouraged.

You just can't boost the popularity of certain level types if you do not specifically enforce that type via a rule for all.

Strongly against this. If the contest provides 3 rules and 1 rule is less popular in terms of entrants, that doesn't automatically mean that those entries are lower quality than the entries in the more popular rules. If those levels have attracted votes, they should progress through the contest.

That is not at all what currently happens! On the contrary: The adapted survival rate strictly ensures that all rules are treated fairly!

Without it the less popular rules get an unfair advantage over the popular ones even!

Suggestion, then: the mixed round could consist of the top 6 levels overall by number of votes, as opposed to just taking the top 2 (or any other number based on survival rate) from each rule. This eliminates the need for survival rate, and empowers the users' votes (which, IMO, should definitely be a goal in all this).

This has the problem if in one rule only 6 people vote and in another 10, the round in which some people missed the voting has a disadvantage.

The previous round may have given the top 3 placements. If there are no ties, why another arbitrary round?

I would not call that round arbitrary. The vote would be different as you can't vote for the elimiated levels anymore and you get a clearer and more direct comparison between the top contenders for the first place.
My vote definitely changes if my favorite level got voted out.

Is it out of the question to do away with tiebreakers and/or run-offs and just award joint 3rd place to multiple contestants? Less voting, more people get recognition for their work and for attracting votes in the first instance.

I really don't like having a lot of people on the same placement - we had 4 winners at one point and it felt really underwhelming und lackluster and I would even argue it lowers recognition on top of cluttering the ranking. Reference: https://www.lemmingsforums.net/index.php?topic=3057.0
At some point simply no-one wins anymore as winning loses its meaning as it is less of a challenge.
It just arises in question in a direct vote-off which entry would fair better.
To be honest, I would have my entry rather have a fair face off and lose a place and I doubt that this would change much in terms of recognition.
I highly doubt it that people are more likely to submit entries when they know ties can happen more regularly!

[WillLem]

I think this could only be solved with: Create you own special contest with suited rules, so that specifically non puzzle focused levels are encouraged.

Not a bad idea, I've also considered this.

Maybe in the future, if I get time and if SuperLemmix gains a big enough audience, I might hold a SuperLemmix-only contest with the focus on the other level types. This is a huge IF, though. My time for Lemmings stuff is getting more and more limited these days.

That is not at all what currently happens! On the contrary: The adapted survival rate strictly ensures that all rules are treated fairly!

Ah OK. Maybe I misunderstood exactly how it works...?

This has the problem if in one rule only 6 people vote and in another 10, the round in which some people missed the voting has a disadvantage.

Fair point.

My vote definitely changes if my favorite level got voted out.

This is kind of my point, though: if your favourite level got voted out, why should you now be voting for a level you didn't think should win? Votes now mean less in this round, surely?

To be honest, I would have my entry rather have a fair face off and lose a place and I doubt that this would change much in terms of recognition.
I highly doubt it that people are more likely to submit entries when they know ties can happen more regularly!

Yes, fair point. But voting still doesn't seem the best way to break the tie.

Try it for a few contests, maybe? List the ties as joint placewinners instead of breaking them with more voting, and see what people think: do we want more votes and less people in the list, or are the joint winners happy to be listed and do away with extra voting? I think this is one to try out and see how it goes in practice.