Getting consistent crash to desktop with RunAndGun on v0.17 and v0.18 running on Ubuntu. Nothing is being logged at the time of crash

Originally posted by Gerbal:

Getting consistent crash to desktop with RunAndGun on v0.17 and v0.18 running on Ubuntu. Nothing is being logged at the time of crash

I'm glad you reported this. This is a known issue on mac, but I don't have a Mac so I cannot test on it. I do have a Ubuntu machine (which I normally don't use for Rimworld), so I hope I can finally pinpoint the issue now. As a temporary fix you can try to place RunAndGun as high as possible in the load order, as that helped for someone with the same issue

I'm also seeing the issue on Fedora. If I move it to the top of the mod list (right after hugslib it no longer crashes as you state. I can test moving it around in the list or get you additional info if you want.

Originally posted by bhamjtr:

I'm also seeing the issue on Fedora. If I move it to the top of the mod list (right after hugslib it no longer crashes as you state. I can test moving it around in the list or get you additional info if you want.

Thanks for reporting this. You could try that and if you find any additional info I that would be awesome, but I have to warn you: It's a really weird issue, so it may be really hard to find additional info. Let me show you what I tried on Ubuntu, and you'll see:

1. Tested save with +/- 25 mods, RunAndGun in bottom of load order: crash
2. Tested save with only Hugslib, two Giddy-up mods, and RunAndGun in bottom of load order: crash
3. Tested save with only Hugslib, two Giddy-up mods, and RunAndGun as high as possible in the load order: no crash.

After this I ran Rimworld with the debug flag on to capture any errors, and now the weird stuff:

4. Tested save with only Hugslib, two Giddy-up mods, and RunAndGun in bottom of load order again: no crash - even after trying multiple times, with tens of raids spawned.
5. Tested save with +/- 25 mods, RunAndGun in bottom of load order: no crash - only tested once.

After the last test I ran out of time and had to go somewhere. I'll do some more testing today, but I hope you can see it's a pretty strange problem. I have some hypotheses I'm going to test:

- Running rimworld directly from the executable with the debug flag somehow prevents the crash.
- Test 2 was an unlucky false positive, and test 5 an unlucky false negative (very unlikely but possible)
- Mod options somehow have something to do with it, and by changing the load order, the options are cleared, and the crashes stop. (is still weird though, why would they get faulty in the first place?)

If anyone has additional ideas feel free to step in :).

After more testing I managed to get the crash again with RunAndGun in the bottom of the load order, and captured the error. I'm afraid there's really not much I can do about it: it seems to be a problem in the base game, but it may be triggered by my mod. In the following topic the same error is discussed:

https://ludeon.com/forums/index.php?topic=37241.15

So the only thing I can do about it, is advising anyone using Linux or OSX machines to put RunAndGun as high as possible in the load order.

Thread 1 "RimWorldLinux.x" received signal SIGILL, Illegal instruction.
(gdb) mono_backtrace 13
#0 0x405951f3 in RunAndGun.Settings:.cctor () + 0xa3 (0x40595150 0x405951f4) [0x7f9364a2dd48 - Unity Root Domain]
#1 0x7f930c013ff5 in (wrapper dynamic-method) Verse.AI.MentalState:PostStart_Patch1 (object,string) + 0x15 (0x7f930c013fe0 0x7f930c013ffa) [0x7f9364a2dd48 - Unity Root Domain]
#2 0x00007f9252be0758 in ?? ()
#3 0x40595220 in Verse.AI.MentalState:PostStart (string) + 0x0 (0x40595220 0x40595229) [0x7f9364a2dd48 - Unity Root Domain]
#4 0x00007ffd9f1a0100 in ?? ()
#5 0x4166f8a4 in Verse.AI.MentalStateHandler:TryStartMentalState (Verse.MentalStateDef,string,bool,bool,Verse.Pawn) + 0x4b4 (0x4166f3f0 0x4166fbaf) [0x7f9364a2dd48 - Unity Root Domain]
#6 0x00007f92b47f3ee0 in ?? ()
#7 0x416cd87e in Verse.Game:GetComponent<object> () + 0xae (0x416cd7d0 0x416cd90c) [0x7f9364a2dd48 - Unity Root Domain]
...

Is that what you were talking about? cf. http://www.mono-project.com/docs/debug+profile/debug/#debugging-with-gdb for how to use gdb to introspect mono applications.

@lashtear. Thanks a lot for this, that debug info is really helpful. I tried debugging it again today, using the new information you uncovered, but without success. I did find a workaround though (which I discuss in the second paragraph). The problem is that the class that throws the exception (Verse.AI.MentalState:PostStart_Patch1 ) really is the most boring class there is, and all it does is accessing the settings provided by Hugslib. I ran a couple of tests this morning, and the first 4-5 times I got consistent crashes. But after these initial crashes, I suddently coudn't reproduce the crashing anymore. Instead, I started getting exceptions that included SIGILLs again, but now they were throwns by the game core. Really strange, since I didn't change any code.

Now the strangest thing yet: I tried running rimworld through gdb, and suddently no crashes or exceptions occured at all. So apperently when running Rimworld through gdb, the instructions that are considered illegal, are suddently not considered illegal any more.

By the way, I didn't completely understand how you introspected rimworld, even after reading that tutorial. I'm not very familliar with both mono and gdb. I did include everything needed in the .gdbinit file, but it's not really clear for me how to proceed after that. so maybe you can just provide the commands you ran?

I can confirm running it from inside gdb seems to mask the problem; I was able to analyze via gdb by attaching after the process started; e.g. on debian/ubuntu machines with pidof, something like:

p=$(pidof RimWorldLinux.x86_64); gdb /proc/$p/exe $p

As mono likes to send a lot of signals around, you'll need to do something like:

(gdb) handle SIGXCPU SIG33 SIG35 SIG36 SIGPWR SIGSEGV nostop noprint pass
(gdb) cont

The cont lets you continue execution, in this case until something bad happens. For me, I found that the problem seemed to hit when routing enemies (flee pawn state) -- either at home or in an encounter, while ridable animals were around-- but that's still pretty anecdotal.

Mono spawns a lot of threads, and gdb is capable of examining all of them (try: thread apply all backtrace), but most are related to sound handling (FMOD or pa_* for pulse audio) or event timing stuff. At least one thread is related to graphcs (SDL_* for simple directmedia layer)-- and we can ignore all of those for this.

The mono_backtrace I used in the previous sample is a small gdb script as described by that page I linked. Something like:

define mono_backtrace
select-frame 0
set $i = 0
while ($i < $arg0)
set $foo = (char*) mono_pmip ($pc)
if ($foo)
printf "#%d %p in%s\n", $i, $pc, $foo
else
frame
end
up-silently
set $i = $i + 1
end
end

where mono_pmip is a C function *within mono* that helps examine mono structures. Note that after various types of crash, continuing to invoke program functions like this can continue to mutate and further destabilize the environment. That said, there are additional ones for examining values and the like, but not well-integrated with gdb at this point.

I'm currently poking around at the gdb python api to make a better set of mono-debugging tools... and seeing if I can get a simple-to-reproduce example of the problem.

I'm not very C# savvy, being more involved with things a great deal lower-level, but I'm happy to try to answer questions if I can.

Hm, was able to duplicate inside a run-from-gdb run, got:

#0 0x00000000404cf233 in ?? ()
#1 0x7fffbc035f75 in (wrapper dynamic-method) Verse.AI.MentalState:PostStart_Patch1 (object,string) + 0x15 (0x7fffbc035f60 0x7fffbc035f7a) [0x7fffefba4d48 - Unity Root Domain]
#2 0x00007ffeeecdca00 in ?? ()
#3 0x404cf220 in Verse.AI.MentalState:PostStart (string) + 0x0 (0x404cf220 0x404cf229) [0x7fffefba4d48 - Unity Root Domain]
#4 0x00007fffffffa860 in ?? ()
#5 0x40811454 in Verse.AI.MentalStateHandler:TryStartMentalState (Verse.MentalStateDef,string,bool,bool,Verse.Pawn) + 0x4b4 (0x40810fa0 0x4081175f) [0x7fffefba4d48 - Unity Root Domain]
#6 0x00007fff3f726ee0 in ?? ()
#7 0x4063745e in Verse.Game:GetComponent<object> () + 0xae (0x406373b0 0x406374ec) [0x7fffefba4d48 - Unity Root Domain]
#8 0x00007fff3c402708 in ?? ()
#9 0x00007fff4cd1f3c0 in ?? ()
#10 0x00000000026dbcb1 in ?? ()
#11 0x00007ffff7fcba60 in ?? ()
#12 0x00007fff42383200 in ?? ()
#13 0x40810eb1 in RimWorld.LordToil_PanicFlee:Init () + 0xb1 (0x40810e00 0x40810ee2) [0x7fffefba4d48 - Unity Root Domain]

On the forum I found:

https://ludeon.com/forums/index.php?topic=37269.0 which may have further insight into harmony-caused mono vm crashes...

I just released an update that fixes the crashing issues! Thanks for the elaborate instructions and your debugging effort, lashtear. Without your help I probably wouldn't have been able to solve the issue. The problem was caused by a harmony patch indeed. Even if I created a stub method and used that for the patch, the crash still occured. I ended up looking for another class to inject the patch in, and that solved the issue. The original method I patched was a virtual method implemented by a subclass. This was also the case for Kiame, so I guess Harmony isn't able to patch such methods on Unix systems properly.

Moreover, thanks to your instruction I now know how to introspect mono applications, and this will be very useful if I encounter similar issues in the future. Thanks again :)

woot :D

Okay serious problem dude, for some reason when I use this mod all of my items go to the same level, in other words my chunks are on the same level as my items so when I go to highlight medicine it also includes a bunch of sandstone chunks

I highly doubt this is caused by RunAndGun as the code targets literally nothing related to items like chunks. How are you so sure it is caused by it? Did you check the error log?

No, I probably should have but when I was loading my mods on I found this issue and then pinpointed this issue by constantly loading and reloading my mods and run and gun was the one which it happened with. Highly possible however that one of my other mods just clashes with the RunAndGun mod which is why this is happening (on a side note doesn't effect steel slag for some reason).

It could be a mod conflict indeed. If you have this issue again or if you can easily reproduce it, please report the error if one is thrown, as the error will most likely show exactly what code causes the issue, and in many cases what mod conflicts.