I am continuing to run tests on the Fireworks framework as a part of the CASCADE project. Many of my tests are at large scales and I have been trying to evaluate the bottlenecks on the system.
One bottleneck I have noticed is when multiple workers attempt to access the same, single mongoDB. Seemingly, the workers, in competing to access the centralized DB, are left waiting to access the resources (e.g. Fireworks).
That said, I have considered testing MongoDB’s sharding capabilities as a way to horizontally scale the database, reducing the load on any single DB instance. I was curious if you had any familiarity regarding sharding on the MongoDB and how this would integrate with Fireworks. I will be working from scratch to set up a test DB on the NERSC systems, but if you have any thoughts or experience on the subject, I would greatly appreciate any feedback.