I'm simulating a big spherical colloid with a diameter of 10nm in a box including water and a polymeric(PMMA) wall and the size of the simulation box is 15nm X 15nm X 16nm.
The system without the colloid, using only cpus, has its best performance while running on 800 procs (approximately 450 particles per procs on average) and it runs with a rate of almost 9 ns/day (please see attached log file pmma_water.log for details).
However, after creating the colloid in the system and deleting the particles that is now occupied by the colloid, I'll have a system with a large colloidal particle (therefore a large void) but less overall particles in the system. Since I have different size particles and therefore wide range of cutoffs, I'm using multi neighbor list. Keeping the same number (approximately 450) of particles per procs, I'm using 660 procs in this case. However, this simulation runs more than 4 times slower than it was before adding the colloid which is understandable since now the colloid has created a large void in the system and many of the processors are left with no particles and are running idle! (please see attached log file pmma_water_colloid.log for details).
Next, I tried using LAMMPS capability of balancing the simulation to evenly distribute the load over the processors. I used "fix balance" to balance the load but I was unable to get any better results (please see attached log file pmma_water_colloid_balance.log for details).
MPI timing breakdown shows that in the case of system without the colloid most time consuming operation are Pair and Kspace with 32 and 50, respectively. However, after adding the colloid, Comm and Neigh become the bottleneck and stays the same even after applying "fix balance". Also after applying "fix balance" and writing out the partitioning pattern consecutively using "dump subbox" doesn't seem to work as I was expecting the partition in which the colloid is located in be large as no other particles can be around it within 5nm (its radius) from its center (I have attached domain partitions images in partitions_evolution.tar.gz where I'm showing the colloid (blue sphere) with a small diameter so partitions can be visible. )
Should I expect better performance for this system using "fix balance" or it needs other treatments?
I would appreciate any comment on how I can improve the simulation while I have this big particle (large void) in the system.
pmma_water_colloid_balance.log (8.72 KB)
pmma_water_colloid.log (7.91 KB)
pmma_water.log (6.14 KB)
partitions_evolution.tar.gz (678 KB)