In Step the user can edit the scene while simulation is running but this can't be done in the middle of timestep. So I need to use mutex to serialize access to the scene: the mutex is locked by calculating thread while performing timestep and by GUI thread while altering objects. The problem arises when calculation of one timestep becomes too long (for example if user sets too low tolerance): if GUI tries to lock the mutex it becomes unresponsive till the end of the timestep and the user can't abort the calculation !
The obvious solution is to abort current timestep as soon as the user starts altering the scene and restart it later. But this can slow down the simulation and make it not smooth. My solution is to first try locking with timeout and abort the timestep only if locking fails:
// Try to lock but do not wait mode than one frame
With such approach simulation will be slowed down only if it is already slow (i.e. when simulation of one frame simulation takes longer then 1/FPS). In the other case simulation will proceed smoothly and user actions will be delayed only slightly.