I've had discussions related to rotated lattice issues many times, so I
feel obliged to chime in on this thread....
This is a difficult thing to understand, but LAMMPS is in fact correct (it is simply difficult, at least in my experince, to wrap one's head around the concept of a crystal structure when the coordinate vectors no longer line up with our usual x, y, and z axes). The "repeating unit" that is ultimately used can be obtained by rotating the lattice vectors according to the "orient" key words and then creating a rectangular prism that circumscribes the rotated unit cell. The three "orient" vectors you specify, when normalized, give the columns of a rotation matrix. The inverse of this matrix (since it's an orthogonal matrix, that's also its transpose) represents a rotation the other way (i.e., from the lattice vector back to the original).
Let R be the matrix that rotates from lattice coordinates back to the
originals. Then the bounding box is therefore the minimum x/y/z and
maximum x/y/z components of all the vectors R . v, where v is [hkl] (h,
k,l in {0,1}). In your case, the bounding box in the y-direction goes
from -2/sqrt(6) to +2/sqrt(6), which are the second components of
R . [1,1,0] and R . [0,0,1], respectively. Note that this assumes
a cubic lattice; in non-cubics, it's R . P . v, where P is the matrix of
cell vectors (a, b, and c; also called a1, a2, and a3).
As for how they line up at each boundary, that's slightly harder. I
think you're guaranteed, for this lattice, to get lined-up coordinates
for repeats in each direction that are multiples of 3, 6, and 2 for x,
y, and z, resp., but I have not tested that. You might, for specific
lattices, be able to use fractions of those numbers and still get a
"perfect" lattice, but I find trial-and-error to be the fastest method
in many instances.
Karl D. Hammond
University of Tennessee, Knoxville