Crash during timewarp at space center

[ts826848]

Steps to reproduce:

1. Start game, load "dev" save
2. Click "Warp to complete" for launchpad refurbishing
3. Crash around 4 hours remaining for launchpad refurbishing

Save: https://www.dropbox.com/sh/wh6qo46lbns39wn/AABZwWk_yUmXdgiy0qrWnjpQa?dl=0
Journal: https://www.dropbox.com/s/lxiyuhyn9gztzpq/JOURNAL.20171122-074220?dl=0

Required mods (as far as I can remember):

• RO
• RP-0 dev
• KCT
• Procedural parts
• Principia
• SXT
• RealFuels
• BluedogDB

Trimmed down part from macOS diagnostic report:

Process:               KSP [98865]
Path:                  /Users/USER/*/KSP.app/Contents/MacOS/KSP
Identifier:            unity.Squad.Kerbal Space Program
Version:               Unity Player version 5.4.0p4 (5.4.0p4)
Code Type:             X86-64 (Native)
Parent Process:        ??? [1]
Responsible:           KSP [98865]
User ID:               501

Date/Time:             2017-11-22 01:03:09.020 +0000
OS Version:            Mac OS X 10.13.1 (17B48)

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGILL)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Illegal instruction: 4
Termination Reason:    Namespace SIGNAL, Code 0x4
Terminating Process:   KSP [98865]

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x00007fff64f2ae7e __psynch_cvwait + 10
1   libsystem_pthread.dylib       	0x00007fff65066662 _pthread_cond_wait + 732
2   libc++.1.dylib                	0x00007fff62e19cb0 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libc++.1.dylib                	0x00007fff62e1a728 std::__1::__assoc_sub_state::__sub_wait(std::__1::unique_lock<std::__1::mutex>&) + 46
4   libc++.1.dylib                	0x00007fff62e1a790 std::__1::__assoc_sub_state::wait() + 46
5   principia.so                  	0x000000021510ac64 principia__FutureWait + 116
6   ???                           	0x000000012fc53495 0 + 5096420501
7   ???                           	0x0000000111a163cb 0 + 4590756811
8   ???                           	0x0000000111b898af 0 + 4592277679
9   libmono.0.dylib               	0x0000000101d25012 mono_jit_runtime_invoke + 1766
10  libmono.0.dylib               	0x0000000101e4f42a mono_runtime_invoke + 117
11  unity.Squad.Kerbal Space Program	0x0000000100bc2f3a ScriptingInvocationNoArgs::Invoke(MonoException**) + 42
12  unity.Squad.Kerbal Space Program	0x0000000100bc2f09 ScriptingInvocationNoArgs::Invoke() + 25
13  unity.Squad.Kerbal Space Program	0x0000000100baf3fa MonoBehaviour::CallUpdateMethod(int) + 234
14  unity.Squad.Kerbal Space Program	0x000000010096f304 void BaseBehaviourManager::CommonUpdate<FixedBehaviourManager>() + 404
15  unity.Squad.Kerbal Space Program	0x0000000100b51fbe PlayerLoop(bool, bool, IHookEvent*) + 1262
16  unity.Squad.Kerbal Space Program	0x000000010115e6a1 -[PlayerAppDelegate UpdatePlayer] + 321
17  com.apple.Foundation          	0x00007fff3f898d99 __NSFireTimer + 83
18  com.apple.CoreFoundation      	0x00007fff3d760464 __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 20
19  com.apple.CoreFoundation      	0x00007fff3d7600e4 __CFRunLoopDoTimer + 1108
20  com.apple.CoreFoundation      	0x00007fff3d75fbda __CFRunLoopDoTimers + 346
21  com.apple.CoreFoundation      	0x00007fff3d7573ab __CFRunLoopRun + 2427
22  com.apple.CoreFoundation      	0x00007fff3d756797 CFRunLoopRunSpecific + 487
23  com.apple.HIToolbox           	0x00007fff3ca6b866 RunCurrentEventLoopInMode + 286
24  com.apple.HIToolbox           	0x00007fff3ca6b5d6 ReceiveNextEventCommon + 613
25  com.apple.HIToolbox           	0x00007fff3ca6b354 _BlockUntilNextEventMatchingListInModeWithFilter + 64
26  com.apple.AppKit              	0x00007fff3ad68a23 _DPSNextEvent + 2085
27  com.apple.AppKit              	0x00007fff3b4fde6c -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 3044
28  com.apple.AppKit              	0x00007fff3ad5d831 -[NSApplication run] + 764
29  com.apple.AppKit              	0x00007fff3ad2c9d2 NSApplicationMain + 804
30  unity.Squad.Kerbal Space Program	0x000000010115cfde PlayerMain(int, char const**) + 638
31  unity.Squad.Kerbal Space Program	0x0000000100002034 start + 52

Unfortunately, nothing is recorded in Player.log, nor stderr.log.

BetterTimeWarp is present, but I didn't have to interact with it for the crash. It sets the RSS timewarp rates back to stock rates for whatever reason, though, so idk if that has an impact

[MicroDoser]

I also am getting this running Windows 7 on a PC.

Only at KSC, when in the flight screen I can warp without a crash. Crash happens after approx 20 days of warp.

Mods from the post above which I also have are
RO
RP-0 dev
KCT
Procedural parts
Principia
RealFuels

[pleroy]

@KSPjohnFX: Could you please give us your logs? We'll need them to debug and figure out if this is the same issue. See the FAQ for details.

[pleroy]

@aw1621107: I have been able to reproduce the crash. Thanks for the journal, it made my life much easier.

The immediate cause is that we have a stack overflow when trying to downsample the history. This needs to be fixed by writing the downsampling code iteratively instead of recursively, but that's not the crux of the problem.

The crux of the problem is that we have a trajectory where the positions and the velocities are not consistent, so the Hermite interpolation yields large errors. I have no idea how this could happen, so more analysis will be needed.

[ts826848]

I sat in the tracking center and tried increasing the timewarp rate in various ways. Unfortunately, I get 2 to 4 frames per second in the tracking center, so the changes were quite rapid.

• Increase timewarp slowly using the . key, waiting a second or two between key presses: vessel fell towards Earth and was destroyed, as expected
• Press "Warp to Completion" button, instantly increasing timewarp to max: vessel suddenly accelerates to 1000-2000 km/s, moving away from Earth until KCT time-to-completion reaches around 4 hours, at which point the game crashes.
• Increase timewarp rapidly using the . key by mashing it: vessel starts fall towards Earth, then suddenly accelerates to a 1000-2000 km/s. Behavior then matches that of "Warp to completion" button. Only difference is there are a few frames at the beginning where the vessel can be seen to fall towards Earth.

Max timewarp rate is 100'000x, because BetterTimeWarp likes to reset RSS timewarp rates to that of stock for some reason.

[MicroDoser]

Here is my journal from a crashed game. I do not have the save that was regularly crashing as I have found the game pretty stable when warping in the flight screen. I can recreate a save or any particular logs you ask for though.

EDIT : Just to let you know, I do not have bettertimewarp installed.

[pleroy]

We now believe that the problem comes from the Adams-Moulton integrator used to compute the velocities. The positions are integrated using a Quinlan integrator, and then, from the accelerations, the velocities are computed using an Adams-Moulton integrator. Most of the time this works fine, but when the vessel gets close to a singularity both integrators misbehave (e.g., they don't conserve energy) which is expected. However, they misbehave in different ways, so the velocities and the positions end up inconsistent, even once the vessel has moved far away from the singularity: the velocity is mostly unrelated to the change in position. Trying to fit Hermite polynomials to the positions and velocities result in "squiggly" polynomials and downsampling fails. We got "lucky" that the downsampling algorithm was stupidly written recursively, otherwise we would probably never had noticed this.

[pleroy]

@KSPjohnFX: your journal seems to exhibit the same bug as @aw1621107's.

[MicroDoser]

Glad to have provided helpful data. Hope the solution is relatively easy for you. Very handy that Principia makes it's own journals.

[pleroy]

Unfortunately, as we dig deeper into the issue, properly fixing it seems to be horrendously complicated. We are going to give it precedence, because we understand it's a real pain for users, but it's likely to take several weeks.

[MicroDoser]

Ouch. Maybe advise users in the meantime to only warp when in flight to stop more reports flooding the forum thread?

Thank you very very much for the work you are about to put in for free on a mod for a game I play. I only wish I knew more about the language KSP mods are coded in so I could help.

[pleroy]

So another problem is that, when the vessel goes close to a singularity, the integrator is no longer symplectic and doesn't conserve energy, typically resulting in ridiculous accelerations. Furthermore, the integrator, being multistep, starts having wild oscillations that result in wild variation of the velocity. The proper fix is to detect that we are moving too close to a singularity (because we are inside a celestial) and delete the vessel. I have started some work towards this (#1640). (You'd hope that KSP would do this detection, but no, if you warp fast enough through a celestial everything is cool.)