Mass Verifier shows fail despite solving replay

[Forestidia86]

Sorry for bringing up another thing but that just occured to me:

I solved Rubix Sinister level "Holly Jingle?" (v.0.9.3) and the replay was automatically saved in solved/rubix/Sinister.
When I mass verify it then it shows fail whereas when I look at it it succeeds.

I've attached the replay.

verifier.txt shows only 5300 Phyus but it lasted 5457 Phyus.

(FAIL),replays/solved/rubix/Sinister/hollyjingle-Forestidia-2017-11-26-031453.txt,levels/single/rubix/Sinister/hollyjingle.txt,Forestidia,97,99,5,5300

[Simon]

The automatic replay verifier stops 5 minutes after the final player action, and declares the level won or lost based on how many lix made it into the exit. Reason: It cannot distinguish long waits that eventually win from infinite loops that don't win.

Either the checker should become more intelligent, and restart the counter-until-mercy-kill whenever a lix exits. But I'd rather see Rubix give fewer initial lix or a faster spawn interval.

Sorry for bringing up another thing

I'm happy every time you post issues. It's high-quality feedback and shows what's in demand.

-- Simon

[Forestidia86]

Yeah, I suspected that something like that could be the issue.
Hm, wouldn't it be another option to just increase the time until it stops?

[Simon]

Some replays leave stray lixes on the game field, don't nuke, and don't have a path for the lixes to exit or die. These replays will never terminate. I have to define an arbitrary cutoff, or the replay checker would get stuck on such replays.

-- Simon

[Forestidia86]

Some replays leave stray lixes on the game field, don't nuke, and don't have a path for the lixes to exit or die. These replays will never terminate. I have to define an arbitrary cutoff, or the replay checker would get stuck on such replays.

Yeah, that was actually clear to me; I just meant something like increasing the time to 10 min instead of 5 min, to just give a wider frame of grace. But of course, it's your decision; it was just a suggestion and I know that this case is prone to slippery slope.

[Simon]

I consider 5 minutes borderline already. As a healthy side benefit, the checker gets our attention to these levels with long waits. I feel like Holly Jingle can be cut to 10 or 15 lix without allowing extra solutions.

If we were playing this level, after our final assignment, we would press turbo-fast-forward and watch for 5*60*15 Phyus / (9*60 Phyus per second at turbo-fast-forward) = 8.3 seconds doing nothing. If the computer cannot handle turbo-fast-forward at 60 fps, it's more than 10 seconds. I'd prefer to cut initial lix in this case.

-- Simon

[Forestidia86]

I have thought a lot about it how to find a solution that keeps the integrity of the verifier but allows nevertheless for an abort after 5 minutes.

Couldn't you give an indication that it has aborted due to expiration of the 5 minute period, e.g. that it says then ABORT instead of FAIL or something like that.

An instrument like the verifier is built up on trust in its integrity, so false results are detrimental for it I think.

[namida]

NeoLemmix declares such replays to be "Undetermined" rather than "Failed". Perhaps Lix should do the same?

[Forestidia86]

How is your position to replays that save the required amount but go on infinitely? Are they ok, since they are in some sense proof of solvability? But in another sense they are broken; though they could be fixed fast and easily.
(I think Proxima's desynched replay to Clam's Humane "Shelf Life" is such a one btw.)

[Simon]

I've forgotten about this thread. Made issue #271: Verifier should report mercy kill.

The mercy kill happens rarely enough that you don't suspect it immediately, and differs glaringly from the interactive mode. I remember my own confusion when, last year, some of Rubix's replays got mercy-killed. I mistakenly reported broken levels to him. Definitely mark in the output differently from failed.

Undecided on whether mercy-killed winning replays should be marked in yet another way. My hunch is to treat them as normal wins. It's courteous to nuke at end of replay, but the nuke should be as little a part of the win-loss-deciding physics as possible.

-- Simon

[Forestidia86]

mercy-killed winning replays

Only out of curiosity:
Are replays that win before the 5-min period mercy killed at all? From my tests I had the impression that as soon as the last required lix is deemed saved the verifier breaks up and gives OK. (It never showed more than the required amount of lix as saved and skill assignments afterwards were not counted etc.)

[Simon]

Hah, correct, the verifier terminates immediately after winning. I forgot that I aborted winning replays immediately. The reason is that most replays in the database win, therefore it's worthwhile to optimize this case.

-- Simon

[Forestidia86]

Hah, correct, the verifier terminates immediately after winning. I forgot that I aborted winning replays immediately. The reason is that most replays in the database win, therefore it's worthwhile to optimize this case.

That makes absolutely sense considering the purpose of the verifier. But the whole information about lix saved and skills, phyus used can get then misleading.

[Simon]

verifier terminates immediately after winning

lix saved and skills, phyus used can get then misleading.

Correct, and I don't have a good answer to this.

Design history: I store user's level results of levels won. I call these results trophies, and a trophy contains lix saved, skills used, and phyus (physics updates, happens 15 times per second at normal speed) spent. Originally, I've wanted to rank solutions by ordering these lexicographically. Every time a lix scored, I would update the potential trophy, thus ignoring any skills assigned or phyus spent after the final scoring. The Lix level browser even displays lix saved and skills used, but I'm not sure whether I still discard skills after the final scoring, as originally planned.

The replay verifier was designed such that it could run in noninteractive mode on a server, testing replays sent to the server, and buliding highscore tables from that.

It wasn't designed as a pack maintenance tool, but I'm super happy that it has become such a tool instead. While preparing for a Lix release, I run it several times over all the included levels. I'm proud of its performance, and this performance can change how I work. It's really nice to try small changes to the code, and test whether all the 600+ levels still stay solvable.

The exact trophy fields aren't important on a won replay, but they still carry useful information on a failed replay. Not sure how to adapt. On one hand, it's nice to display identical information in the singleplayer browser and on the replay verifier; on the other hand, the replay verifier should be blazingly fast.

-- Simon

[Forestidia86]

verifier terminates immediately after winning

lix saved and skills, phyus used can get then misleading.

Correct, and I don't have a good answer to this.

Maybe just give the information in the verifier.txt that skills, lix saved etc. are not counted after the final required scoring and that the phyus told in the file are bound to lix saved (of the required amount). (I noticed that if you save no lix then phyus just show 0 and got the impression that phyus are only counted by saved lix (of the required amount), which fits to what you have explained.)