Hello,
I am using the thermo command to output data to the log file every N=1000 timesteps. I am also writing out restart files every N=200000 timesteps.
When the simulation is killed because it exceeds wall time on our cluster, the last timestep recorded in the log file (69822000) is less than the timestep of the last restart file (restart.72400000).
I realize that the frequency is different, but how can it be that much less? Why doesn’t the log file contain the very last timestep since it records every 1000 steps?
Thanks,
Adam
Hello,
I am using the thermo command to output data to the log file every N=1000
timesteps. I am also writing out restart files every N=200000 timesteps.
When the simulation is killed because it exceeds wall time on our cluster,
the last timestep recorded in the log file (69822000) is less than the
timestep of the last restart file (restart.72400000).
I realize that the frequency is different, but how can it be that much less?
Why doesn't the log file contain the very last timestep since it records
every 1000 steps?
because of unix file i/o semantics and because your calculation was
terminated with great prejudice.
file i/o on unix/linux machines is by default block buffered, i.e. the
file will only be written to when a certain amount of data has
accumulated (typically 4096 or 8192 bytes) for as long as it is held
open. this massively improves the performance, since adding to a file
and committing those changes at every write command can slow down the
calculation. you don't see it in restart files, because the closing of
the file enforces a full commitment of all pending write requests.
please check out the "thermo_modify flush yes" command as a means to
tell the unix/linux OS to make a better effort to commit pending
writes to disk more frequently.
in general however, it is strongly recommended to tune your
calculation so that it doesn't get forcibly terminated at all, as the
OS may kill the calculation during some operations that may be
difficult to recover from.
axel.