Just done some more experimentation on this.
The complete desktop freeze appears to be something that my machine just does when the OS's disk IO hits 100% - which is what happens when something starts using a lot of swap. ESPECIALLY when the dufus at the keyboard keeps alt-tabbing to check memory usage!
Connecting over SSH, I can continuously monitor memory usage without issue during a swap-fest.

However, I do think VT should manage its memory usage more aggressively. Here's what I saw while testing logging in to the NW corner of Identify's server today:

1. Start VT, join server.
2. RAM usage just tops 8GB, but I get into the game
3. Single-digit FPS, but enough to get in the car and slowly start moving away
4. Game starts loading more lots
5. RAM usage increases even more, and hits the swap hard
6. Game freezes, and shortly afterward my connection to the server is lost
7. RAM usage sits at about 9GB
8. Hit "reconnect"
9. RAM usage starts increasing again when it gets to the "rendering" progress bar
10. At about 10GB usage, I lose connection to the server again
11. Game is unresponsive, RAM usage keeps increasing
12. Shortly after reaching 13.4GB used, the game cursor becomes responsive again, and I can quit.

Shouldn't the client have freed up memory after disconnecting from the server the first time? I've noticed that it doesn't appear to, even from voluntary disconnections?

It think it might also be a good feature to automatically scale down the view/render distance, if the game sees that it's starting to approach using the machine's total available RAM.

Seems to be a major memory leak.. probably should fix that because 30 minutes of play and it starts lagging terribly..perhaps a unity engine issue

When I'm in the Lexxy Office zone my memory usage goes to 8gb. When I suicide and go back to spawn it drops down to 500mb.

I'll continue investigating

Right I have done the following tests on both my windows and linux machines:
1. Go to the LexxyFox Economic and Lag District and get heaps of stuff put into both system and graphics memory
2. Deathwarp back to spawn.

On windows memory usage drops down to 500mb.
On linux memory usage (as reported by top) remains the same, but performance returns to normal.

WHAT CAN BE DONE NOW:
- You can manually set the "unrender distance", its below draw distance. Chunks outside of that range will be destroyed.
- Avoid the Lexxy Lag District

ANALYSIS:
I think what may be happening is Linux correctly frees the memory but the os does not remove the allocs from the program.

EDIT: This is the "Linux Ate My Ram" problem. The OS is allocating memory to the process, but is not deallocating until its needed by something else. In short its "used but available"

Also I think the "problem" is that since 1.0.14 vram usage has been doubled (as there are both LoD and HighRes versions of each chunk). LoD only has minor geometry reduction. Both LoD and HighRes are created in the same pass.

Possible ways forward
- Create LoD and HighRes at the same time, but only upload LoD to GPU and keep HighRes in system ram. Same memory consumption as now, but hopefully the OS will be smart enough to throw the HighRes into Swap.
- Only generate High Res versions of chunks when nearby (tradeoff - 2x cpu usage as chunks have to be built twice)
- Force only LoD usage in high stress situations. LoD chunks really don't look that bad, you'll just see a drop in texture quality.
- Geometry reduction for LoD chunks
- Dynamic draw distance

Originally posted by SnapperTheTwig:

ANALYSIS:
I think what may be happening is Linux correctly frees the memory but the os does not remove the allocs from the program.

EDIT: This is the "Linux Ate My Ram" problem. The OS is allocating memory to the process, but is not deallocating until its needed by something else. In short its "used but available"

I thought it might be something along those lines, but if that were the case, shouldn't logging into the server the second time have the OS try to re-use already allocated memory, rather than immediately start allocating *even more* RAM to the VT process?

A further update to this problem:

I've created a test map that consists of a core of nothing but office towers in 4 bases regions. The rest of the map is blank, so I can test loading lots of buildings then leaving.

EDIT: I've also upgraded the memory management of the game. If the game can't keep up to 60fps, it'll reduce draw distance. If the game is using more than 70% of VRAM it won't create LoD chunks and will actively unload terrain not in your draw distance (from unrender distance onwards). If the game is over 100% of VRAM it will become super aggressive in unloading terrain.
Also if you're logging into a server and are at the "rendering chunks" screen and 70% VRAM is reached it'll take you straight into the game and start doing this behaviour.


Calling malloc_trim(0) every now and again frees up resident memory usage, which means that the kernal is not reclaiming freed memory (as we've just discovered). However virtual memory usage blows up. I'm going to research/try different memory allocators.

If this doesn't work then I'll have to write my own (which probably won't make 1.0.15)

Update update: I've set 1.0.15 live in the Experimental branch.
(Select Betas->Experimental)

This should greatly reduce your VRAM usage.

You, sir, are a legend!

I built up one corner of my single-player build map with tall office towers filling a 4-base turf area. After an initial FPS hit on arrival (as the view distance scaled down), I was tearing around the streets at a pretty constant 25-30fps. With version .14 I was getting a pretty steady 7-8 FPS just stood still in the middle.
RAM usage never seemed to exceed 6GB (as expected), whereas with the previous version, at the same location in the centre of the district, I was almost touching 100% memory usage just from loading in.


But since life is never all kittens and rainbows, here's some things I noticed during my testing:

1) The view distance seems to constantly "bounce" up and down (somewhat jerkily) when hovering around the 60FPS mark. It's especially noticeable if you're not moving. The game still plays smoothly, but the visual effect is somewhat jarring.

2) When in the high-density area, the 3D previews when selecting a prefab lot to build didn't appear - all I got was the red facing arrow, rotating around nothing.

3) I got a CTD when driving into the zone at high speed. There was no dump file in the logs dir, and I have been unable to reproduce. IIRC, the same thing happened the first time I found Lexxy's lag district, so probably nothing specifically related to these changes.

4) Some parts of the lots I built weren't being loaded/updated - a few buildings were cut in half, with the missing half still rendered as grassland. But I tried driving in to one, and it was certainly as solid as an office tower. Getting those lots to un-load and coming back seemed to fix that issue.

1.0.15b is live in experimental branch

Should smooth out 1, fix 2, fix 3.

4 is most likely due to the chunk unrendering algorithm deciding that "yeah! we've free'ed enough vram! Lets stop unloading" and therefore leaving a rendered chunk next to an unrendered one outside of your draw distance.

1 is indeed a lot smoother. I still see it bounce if I sit right at the boundary of my lag district, but it's a lot less pronounced now.

2 I can confirm fixed.

3 I was unable to replicate the crash when it *wasn't* fixed, so I'll take your word for it!

I think 4 only happens/ed when building new lots, and perhaps only when they were outside my FOV. I gave it a go again today, but wasn't able to replicate the issue with .15b.


Noice.