I have two layers of atoms. I want to use Tersoff potential to model the atomic interactions per layer and l-j potential for
interlayer interactions. I have read the manual and it seems that a simple hybrid command can not do that. How can I achieve what I want?
You can do this with pair hybrid. The trick is to assign type 1 to atoms in layer 1, and type 2 to atoms in layer 2 (assuming all atoms are the same element, but the method works for the more general case too). You will then define two Tersoff pair styles, one for type 1, and one for type 2, and then an LJ style for the 1-2 interaction. It is important to recognize that the physics of this model are, well, unphysical, so you should take all results with a grain of salt, unless you or someone else has spent some time thinking carefully about this.