Just a short reply for now.
If you are having numerical instability,

make sure you use the “linear” keyword (not “spline”)

use a short, low resolution table
dihedral_style table linear 90
I think the main problem is that your dihedral angle table has too many entries.
Your dihedral angle forces look noisy, (at this angular resolution).
Do you really have enough data collected from your allatom simulations to justify using such a narrow 0.05 degree difference between angles in the angle and dihedral tables? Having narrow bins will reduce the amount of data in each bin and this will increase the noise in your coarse grained force field
(I assume the log of the data in each bin is proportional to the energy at that angle in your course grained force field. Taking the derivative with respect to angle to get the forces this will increase the noise even further. That’s why the force curves looks so much uglier than the energy curves.)
For now, try using something much larger instead, like 4 degrees between entries in the table instead of 0.05 (so, ~90 entries the table). Your dihedral energies and forces will be much smoother and you are less likely to numeric explosions.
If that works, you can gradually try increasing the resolution of your dihedral table to 120 (3 degrees), or 180 (2 degrees) or higher. And once your simulation is stable you can try using “spline” instead of “linear”)
But 7200 table entries (0.05 degrees) is too many (for your force field at least).
Incidentally, thanks for including the dihedral energy and force curves. That helped me understand what might be causing your problem.
Andrew
P.S.
I’m glad people are using dihedral_style table for boltzmann inversion.