Stop maps from crashing (OutOfMemoryError)

[Forestidia86]

I've tried to force it in singleplayer (map: "What could possibly go wrong? (3p)") and got the OutOfMemoryError.
But even with debug build it only gives a very short message. (Probably not different from release build?) I've just attached it for completeness. (Old laptop, Win 7, 4 GB RAM, ca. 1,7 GB for graphics (shared memory, only 64MB dedicated on graphics card))
The taskmanager said that I've used approx 3 out of 4 GB available for me. (Lix ate up about 1,8 GB at this point. That is really much for this kind of game though not generally. But my laptop uses shared memory, so that graphics memory is to a big part handled via RAM as well.)

I will try to run against Allegro debug libraries but can't promise anything

[Simon]

Thanks for forcing this. It's uncanny how Lix uses 1.8 GB on Windows.

After the a couple crashes in yesterday's mulitplayer session, I openend top (command-line program) and watched memory throughout the remaining 2 hours of the session. Lix used 2.x % of my 16 GB, never any more than 3 % = 480 MB.

That's still more than I thought, but not exceedingly much. Will look into it.

D is garbage-collected and I force collections every couple seconds. I could force them even more aggressively, and migrate more of the fundamental data to reference counting.

-- Simon

[Forestidia86]

I've tried it with Allegro debug libraries but the crash seems to come from Lix not Allegro, at least I got the Lix crash message and not the MVC++-RTL message. The allegro.log gets huge very fast (in about 30 min about 800 MB(!)).

[Simon]

Very good, then I call reasonable libraries and am merely greedy myself.

-- Simon

[Forestidia86]

I've created an allegro.log for just opening a big map, letting Lix spawn, doing some actions and changing to a big map and wait a bit there. There was no crash, but it shows the inner workings of the handling Allegro does. The file is about 9 MB big, shall I provide it?

[Simon]

Yep, please attach. It's likely of no use to debug out-of-RAM, but I'm curious.

-- Simon

[Forestidia86]

Attached the log for the short session. It has more than 70000 lines though many of them have similiar content (dealing with bitmaps).

[Forestidia86]

It's probably a stupid question but the error message mentions:
@src\core\exception.d

I don't find such in my Lix-src folder, not even a core-subfolder. I'm a bit confused.

[Simon]

If it's not in Lix, then it can be in the D runtime or in Phobos, the D standard library. You can search for this (truncated) path within the D installation directory.

In this case, it's D/dmd2/src/druntime/src/core/exception.d.

With out-of-RAM, there is no single statement that always crashes. Even if we had a stack trace and would find a fat allocation that crashes with out-of-RAM, there is no obvious bug in that allocation. The real problem is that I waste memory throughout the program, and that allocation happens to be the first that was too much.

-- Simon

[Forestidia86]

If I have a big map open the RAM rises for some time until a certain value. If it has reached a certain high point and I change to another huge map, crash seems more likely (RAM rises beyond this point in some cases then I think).
So it really looks like a certain point of RAM-usage is stepped over, although I've still RAM left. (Nevertheless the RAM-usage is scarily high for my old laptop.)
Though it's still not consistent and I can't see a real pattern in the workings. I have the impression that it's different how low the RAM-usages sinks when closing the map.

[Simon]

Thanks, that's a good baseline. I agree how that RAM usage is scary.

I've spent the better part of today in the code, working on the replay/physics caching. Now, I call the garbage collector more often and return memory to the OS every time old states could be dropped from the cache.

Please pull from unstable, and build debug-replay (commit 3153eb07) and test:

• Does Lix eat as much RAM as you tested earlier today?
• Is this fixed? #294 Manual savestate, sporadic replay desync
• Is this fixed? #295 Savestate shouldn't preserve future when (don't keep replay after ◀▮)

-- Simon

[Forestidia86]

• Is this fixed? #294 Manual savestate, sporadic replay desync
• Is this fixed? #295 Savestate shouldn't preserve future when (don't keep replay after ◀▮)

It doesn't seem to desync anymore with the method I found.
To preserving future: The case mentioned in the issue shouldn't work this way anyways since rewinding before an action cancels if you don't keep replay (, unless you've meant A,C instead of A,B in the end?). But you've mentioned another case in the thread, where you load instead of framestep back. Well loading cuts now all out what doesn't belong to the savestate (even if you have keep replay on and it's after the savestate), so with that the future replay is discarded as well.

• Does Lix eat as much RAM as you tested earlier today?

Unfortunately RAM still rises high especially in the course of changing maps from time to time. Still got that error eventually.
I looked at the ressource monitor to have a more precise number for Lix' consumption and the highest with crash was ca. 1,4 GB. It really brought the RAM to the edge for my laptop but it was always more allocated for the game then used up as far as I could see. Although the computer had to care constantly for adjusting and increasing the allocation if necessary.

[Simon]

Re 2 fixed replay bugs: Cool, thanks for confirming! Yeah, you're correct that #294's example is invalid. And you're correct that the second example (in other LF thread) is valid and fixed by the fix of #294.

Re RAM: That's a sad result, but certainly very valuable feedback. I can work on more manual memory management from the bottom upwards, but that will take lots of work, and makes the code more complicated.

Ideally, before shaving this yak, I'd get my hands on a Windows machine, see the RAM ballooning myself, and toying around with some debugging. Made issue on github: #296 RAM balloons to over 1.5 GB, Windows

-- Simon

[Forestidia86]

I've tested with the newer laptop that has 8GB RAM.
I compared the behavior of 0.9.8 with your fix in the unstable.
And it looks like the fix has at least improved the situation:

With fix: I had at most 1,1 GB RAM used with changing maps a couple of times. I didn't seem to have this insane rising generally and seemed to be more stable.

0.9.8: Even on the first map rising to 1,5 GB RAM without doing anything, just waiting. Then nuke and change to "Infinitus (8p)" resulted in immediate crash. I had plenty of RAM still free, it seemed to have been more allocated to the game then used up by the game => nevertheless crash.

[Forestidia86]

Attached video that shows Ressource Monitor, tab RAM, to show memory consumption of Lix 0.9.8 in huge multiplayer maps up to crash.