The parameter "costheta0" in LAMMPS' new Tersoff
potential, usually just called "h" in the literature,
is understandably limited to being between -1 and 1,
or else an "Illegal Tersoff parameter" error results.
Unfortunately, I found a couple journal articles that
treat "h" as a parameter with an arbitrary range of
values, not as a cosine of anything:
P.A. Ashu, et al./Journal of Crystal Growth 150(1995)
176-179
R. Smith/Nucl. Inst. Methods B 67(1992) 335
A more recent article, M.A. Migliorato et al./Physical
Review B 65(2002) 115316, used Tersoff parameters for
indium from the article by P.A. Ashu, et al.
That said, it was easy enough to change the LAMMPS
code so that a value of "h" with a magnitude greater
than one wasn't rejected.