Lets’ say I have a solid system made with N atoms in a box with a volume of V as shown below. In this case, the number density of my system is simply \frac{N}{Volume\; of \; occupied\;3D\; space}.

but what if I have a system with N atoms in the same box, but N_{1} atoms are in the solid phase and N_{2} atoms are in the gas phase \left( N_{1} + N_{2} = N \right).

How should I calculate the number density in this case? Do I consider the whole box volume V?

The density of the system in both cases is relative to the volume of the box, as the box defines your system. If you want the density of the material in whichever phase, you’ll need to sample several smaller volumes that are entirely contained within that phase, count the number of particles in it, divide by the sample volume, and average that over all samples.

I have a similar problem. I am thinking of first using something like a DBSCAN to find the size of cluster, outermost atoms, etc. Then density can be calculated within that volume. I think a similar approach can work for your case.