By default slurm sends SIGINT to the process 30s before SIGKILL.

This turns into KeyboardInterrupt in python, which we trap in dream.core.Dream.sample. We then do some adjustments to state in fitters.Dream.solve before returning from fitters.Fitdriver.fit to cli.main and calling bumps.cli.save_best.

We need to minimize the amount of work done between the KeyboardInterrupt and saving the problem state so that we can make sure it is complete within 30 s.

Since KeyboardInterrupt may happen during dream.state.State._generation or dream.state.State._update there is a chance that state may be inconsistent. Glancing at the code this shouldn't matter much (e.g., the parameter values may be updated but the logp will still have values from n generations ago, so at worst this will be considered as outliers in the statistics) but this needs to be checked carefully. We may need to mark the state as dirty before the update and clean afterward to minimize the window in which they may be inconsistent. We need to check that numpy array slice assignment is well behaved if SIGINT arrives in the middle.

We should check the other optimizers for KeyboardInterrupt behaviour.