I want to simulate bilayer or multilay system, but I am not sure how to define the potential.
For example, for a bilayer system, I have A and B two types of atoms and 4 atom types in the models, type 1 and 2 for atoms in the firt layer, and type 3 4 for atoms in the other, and I wish to use a Tersoff potential to compute interactions within each layer, and use LJ between layers. There is no interation between A-A or B-B in each layer.
Then how should I define the potential? Is the following right?

pair_style hybrid lj/cut tersoff
pair_coeff 1 2 tersoff AB.tersoff A B NULL NULL
pair_coeff 3 4 tersoff AB.tersoff NULL NULL A B
pair_coeff 12 34 lj/cut 1.0 1.5

or

pair_style hybrid lj/cut tersoff
pair_coeff * * tersoff AB.tersoff A B A B
pair_coeff 12 34 lj/cut 1.0 1.5

I want to simulate bilayer or multilay system, but I am not sure how to
define the potential.
For example, for a bilayer system, I have A and B two types of atoms and 4
atom types in the models, type 1 and 2 for atoms in the firt layer, and
type 3 4 for atoms in the other, and I wish to use a Tersoff potential to
compute interactions within each layer, and use LJ between layers. There is
no interation between A-A or B-B in each layer.
Then how should I define the potential? Is the following right?

your description is a bit confusing, so i don't know exactly what to
answer except the following.

the best way to determine whether your input is working as expected is
to construct a minimal test input that is so simple that you can
validate the results yourself with pencil and a piece of paper. for
simplicity, you can also turn the LJ part off simply by setting the
epsilon parameter to 0.0.

pair_style hybrid lj/cut tersoff
pair_coeff 1 2 tersoff AB.tersoff A B NULL NULL
pair_coeff 3 4 tersoff AB.tersoff NULL NULL A B
pair_coeff 1*2 3*4 lj/cut 1.0 1.5

this *cannot* work. the documentation explains why.

or

pair_style hybrid lj/cut tersoff
pair_coeff * * tersoff AB.tersoff A B A B
pair_coeff 1*2 3*4 lj/cut 1.0 1.5

this doesn't match with what you are saying you want to do. so likely
neither is correct.
also, the LJ parameters are highly suspicious. the look like
parameters for reduced unit calculations, while tersoff requires metal
units.