Why? Is LAMMPS giving you an error message or warning message about your image flags? What is that error message? Have you looked for that error message here:
http://lammps.sandia.gov/doc/Section_errors.html
You control the boundaries of the box either by
- editing your DATA file (assuming you are using a LAMMPS DATA file to store the initial coordinates of the atoms in your system). Near the top of the data file, there is a “Boundary” section. You control the size and shape of the boundary there. See:
http://lammps.sandia.gov/doc/read_data.html
(In moltemplate, this is called the “Data Boundary” section.)
- using the “change_box” input script command. For details, see:
http://lammps.sandia.gov/doc/change_box.html
Why are you worried about how LAMMPS calculates the neighborlist? LAMMPS can simulate systems with rectangular or triclinic/hexagonal boundary conditions, and LAMMPS should calculate the neighborlist correctly regardless. In my previous post I suggested it might be more convenient to use rectangular boundaries instead of hexagonal boundaries, but both are possible in LAMMPS.
It seems to me that you should look at some of the LAMMPS examples. I found the “peptide” exam to be particularly helpful.
There is a list of turning here:
http://lammps.sandia.gov/tutorials.html
and some nice videos here:
https://youtube.com/watch?v=GXA2PyqKYdY
https://m.youtube.com/watch?v=ZMiveX6n-1w
If you are using moltemplate, the introduction/tutorial chapter (chapter 4) provides a list of helpful LAMMPS and VMD commands you will need.
Cheers
Andrew