Fair enough. Now I see why you are confused. A "hexagonal" unit cell
is not a hexagon.
Take a look at this web link:
https://en.wikipedia.org/wiki/Trigonal_crystal_system
According to popular convention at least, a "hexagonal" unit cell is a
specific kind of triclinic (rhombohedral) unit cell with 60 and 120
degree angles in the XY plane. You can use this kind of
diamond-shaped unit cell to build a periodic system with hexagonal
symmetry. For this reason, nobody bothers to write software which
supports boundary conditions which are actually shaped like hexagons,
but I did not get the impression you are asking for this feature. It
sounded like you just want to find a way to run your simulations of
graphene/graphite. Just try using rectangular boundaries, and make
your box large enough that it encloses an integer number of
diamond-shaped unit cells. Here's that picture from the example I
mentioned earlier again:
http://moltemplate.org/images/nanotube+water/graphene_unit_cell.jpg
As this discussion is no longer specific to LAMMPS, perhaps we
should continue in private.
Cheers
Andrew
(Incidentally, I recall some simulation software programs support
"truncated octahedron" shaped periodic boundaries, but they do it for
a very specific reason, and I always thought it was not worth the
considerable labor.)