Most textbooks do not deal in-depth with small systems, because:
- Small system simulations do not (directly) give valid results for bulk systems
- Small system results are easily obtained through foundational probability or direct simulation.
This is one of those results.
First consider a 1D stepper that randomly walks either backwards or forwards. Half the time it will take its next step towards the origin rather than away, so during that step its square displacement will be reduced.
Now consider two such steppers, uncorrelated. A quarter of the time, both steppers will simultaneously step towards their respective origins. Generalising, n steppers will all step towards the origin 2^{-n} of the time.
Now consider a 3D stepper. In a simple isotropic space its x, y and z motions are uncorrelated. So you might as well consider it as 3 uncorrelated 1D steppers instead of 1 3D stepper. By simple logic, n uncorrelated steppers in 3D space will all be stepping back towards the origin all at once 2^{-3n} of the time.
Of course this is irrelevant for almost all MD simulations. If you simulate 10 argon atoms in 3D, all of them will step towards the origin in x, y and z simultaneously about 2^{-30} of the time. As a quick hack, 2^{10} \approx 10^{3}, so this is about a billionth of the time – or, if you simulate in 1 fs timesteps, you will have to simulate the system for about 1 microsecond to see one femtosecond-timestep in which all argon atoms are simultaneously moving towards the origin instead of at least one moving away.
But if you are simulating a single vacancy then MSD can most certainly decrease over time for statistically detectable portions of your trajectory.