Hi!
Have you visualized your system with triclinic box? What I would do is to use ‘replicate’ command and see if pbc is set correctly. It might be that your setup in triclinic case is not really correct and when periodic boundary conditions are applied, there are some atom overlaps. That’s my guess, at least.
Best wishes,
Manana