I want to calculate the stacking fault energy for a hexagonal system, for the pyramidal plane (10-11) with direction 1/3[11-23]. In order to maintain periodicity in this direction I need to define as new x-axis the direction 1/3[11-23].
If x' is 1/3[11-23], y' is on the plane with 60 degrees angle from x' and z is normal to the plane, so z forms 90 degrees with x and y.

I have tried to use the lattice orient to re-orient my system but there is a difficulty since the 60 degrees angle always includes a sqrt(3), that does not let me define properly the new axis, since it is a real number. This sqrt(3) exists even if I define my system in ortho-normal space or hexagonal space. Moreover, the a1 and a2 vectors are equal but a3 vector is larger by a real number (1.624 the c/a ratio in hexagonal systems).
I have tried to rotate the atoms (displace_atoms command) of the system along various axis, so they are align to the direction I need. I think because my system is not periodic along the z axis, this command does not rotate my atoms as I want.
I think the best command to use is rotate a region, which will align the box vectors to lattice vectors. This command includes a time-variable v_theta that rotates the atoms only after a run (correct me if I am wrong). I am not sure what this run should be, since for the calculation of SFE I do not do any MD, just MStatics.

Does anyone have a suggestion or experience on this problem?

A lot of the stuff you discuss is not relevant to your problem, which
at its core is:

"""
Create atoms in a box with the following periodic unit cell vectors:
"If x' is 1/3[11-23], y' is on the plane with 60 degrees angle from x'
and z is normal to the plane, so z forms 90 degrees with x and y."
"""

Without knowing the actual atom positions, I can't be sure, but you
can very likely do this entirely in LAMMPS with a complicated
'lattice custom' command followed by simple create_box and
create_atoms commands. You probably do not want to use the orient
keyword with the lattice command. Why? In LAMMPS, if you rotate the
lattice, the periodic cell is not rotated, as explained here:

But, as I said, you probably don't need to rotate the lattice. Just
set the 'a' vector in the lattice command equal to [a, 0, 0], the b
vector at 60 degrees to that, and the c vector at 90 degrees to those,
and follow that with basis keywords for each atom. If it is too hard
to work out the basis atom coordinates by hand, you can always use
some other software or a small script to generate the coordinates.

Aidan

p.s. I agree that it would be much nicer if the periodic cell lined up
with the a, b, c, axes specified in the lattice command, but LAMMPS is
not set up to do that.

I was aware that if you rotate the lattice, the periodic cell is not rotated but I thought it could help me line up the x axis with the a1 lattice vector. Still, in this way I would lose the periodicity in the other directions.

Thank you for the hint on how to create my lattice. I was trying to avoid this method since in this specific direction I need to find the positions of the basis atoms (I guess not that difficult) and the surface between x and y (for the SFE), while the a and c vectors have an angle of 63 degrees (not 90 as I mistakenly said earlier).