Hi, while searching for something else here I found this thread which seems to be related to your topic:
https://steamhost.cn/steamcommunity_com/workshop/discussions/18446744073709551615/3195865512461370328/?appid=1066780

This is extremely helpful. Thank you.

Turns out that both engine and gui threads must return the same state, otherwise the game crashes with an assert:

Originally posted by peteraklnz:

Turns out that both engine and gui threads must return the same state, otherwise the game crashes with an assert:

Why am I not surprised... Tbh I couldn't get a warm fuzzy from the other thread. Although it sounded like the OP found a solution, the later posts seemed to indicate it didn't work because of the semi-randomness of when / how often things execute.

The whole furball is still a complete mystery to me. I still struggle with figuring out the syntax of basic commands from the api ref doc maze (see other thread)...

Simple View (may be a bit off)

Simulation / Script Thread:
Savegame Load ->

Start of Simulation Frame *here* ->
GameState for Game -> Apply all Commands -> Run Simulationen -> LuaFn GameScripts Load -> LuaFN GameScripts update -> LuaFN GameScripts Save -> copy of GameState to GUI -> Next Simulation Frame

GUI Thread:
GameState for GUI -> LuaFn GameScripts Load (Data from Thread Script)-> LuaFN GameScripts guiUpdate
sendEvents (to CommandList)
get Events from UI Elements > guiHandleEvent

Building Preview, the State you see renderer, mouse input and so on.


Town Developer Thread:
-> Sends Commands to upgrade roads and build builidngs.


LuaFN Save will be used when saving a game, the data will end up in filename.sav.lua

Note: The serialize function is quite slow with a lot data to filename.sav.lua

It is a good idea to use different files and / or naming conventions for worker and GUI threads. Try testing for api.gui == nil: if true, you are not in the GUI thread.

All of that started because I wanted to limit memory consumption of my mod. Since I really only need most of the state in the "engine" thread I wanted to limit what the "gui" thread had access to. Ultimately that turned out not to be necessary - the ever increasing memory consumption was caused by really poor default garbage collection by the game. Forcing manual periodic garbage collection fixed the problem.

How frequently do you run the manual garbage collection?

in the "engine" thread - roughly once every 90 seconds. In the UI thread, if the mod is not displaying any components - also every 90 seconds, but any of the windows are visible - I don't run it at all, as it tends to slow the menu down significantly.

Cool thanks!

This week I entered the world of game_scripts finally... It's been veryyyy slow going 😵 The API docs are reallyyyy weak so I ended up putting print commands on everything I could find to learn the structure of various tables and the sequencing of the various engine and gui functions. It helped me figure out when each of the functions runs for the first time and how often so I could visualize the flow. You probably did similar judging from earlier posts above. I'm currently using the guiInit for my one-time setup stuff.

But there's still so much guessing on the meaning of parameters in the various resource tables that I can't find in the docs. It's trial and error line by lineeee... Painful but finally making some progress...

It's definitely a lot of trail and error. What I found frustrating that particularly at the begging each error ends in a crash which means I have to re-launch the game over and over again.

One of the things I wish someone told me about is how the 'state' actually works with those two threads (ui and engine) and the fact that UI thread runs in ticks (every 200ms), whilst the 'engine' thread is real-time. So you have to think through how the state will be kept and synchronised.

The other thing is that lua doesn't handle well arrays that are indexed by sparse numbers. And basically any id in the game is a number. If you want to use train id or station id you always have to convert it to a string before using as an index, otherwise lua tries to pre-allocate memory for the "missing" indexes which explodes memory consumption and slows things to a halt very quickly.

Originally posted by peteraklnz:

It's definitely a lot of trail and error. What I found frustrating that particularly at the begging each error ends in a crash which means I have to re-launch the game over and over again.

One of the things I wish someone told me about is how the 'state' actually works with those two threads (ui and engine) and the fact that UI thread runs in ticks (every 200ms), whilst the 'engine' thread is real-time. So you have to think through how the state will be kept and synchronised.

The other thing is that lua doesn't handle well arrays that are indexed by sparse numbers. And basically any id in the game is a number. If you want to use train id or station id you always have to convert it to a string before using as an index, otherwise lua tries to pre-allocate memory for the "missing" indexes which explodes memory consumption and slows things to a halt very quickly.

Yup I def have lots of crashes/reloads... Just the other day I wish I had recorded how many. Oh wait... yup 112 stdouts in my crashdumps folder in the past 3 months lol 😅 (ofc a few sessions were not crashes).

The API docs indicate the engine state (simulation) update is 5 Hz. It doesn't specify the gui update rate, it mainly focusses on round-tip response between them, which is necessarily slower:
https://transportfever2.com/wiki/api-testing/topics/states.md.html

Hmm... 🤔 I'm puzzled by that memory allocation theory because that wasn't my (albeit primitive) understanding of Lua. Quite the opposite, I was reading this week that one of Lua's main brags is that it doesn't allocate memory to non-existent elements in a "sparse" table, they simply don't exist. It has no concept of an outer boundary size / volume / shape, it only contains the elements added to it:
https://www.lua.org/pil/11.2.html
(plus another mention of it a couple pages later)

So whether the "gap" between table element indices is 1 or 1 million or random text, memory usage is the same.

But if you actually fill all those "positions" (with zeroes), that's a whole different story...

The memory issue might be an artefact of the C++ <-> Lua API, but it's definitely something I have noticed. The other issue with memory is that the particular version of Lua doesn't allow for memory pre-allocation, so populating large data structures can get noticeably slow.

Originally posted by peteraklnz:

The memory issue might be an artefact of the C++ <-> Lua API, but it's definitely something I have noticed.

Oo that's a good point!

Where/how are you monitoring mem usage? Can you identify your own mem consumption independent of everything else going on in the other processes?

...Stumbled over this while looking for something else:
api.util.getLuaUsedMemory()