The cvff improper style documentation states that the Wilson out-of-plane angle is used. It defines this as follows: if the atoms "are ordered I,J,K,L then the Wilson angle is between the plane of I,J,K and the plane of J,K,L."
In contrast, definitions that I have seen elsewhere say that the Wilson angle is instead between the line K-L and the plane I,J,K. This is clearly not the same as the angle between the two planes, although they would of course be similar when the out-of-plane displacement of L is small. This definition is given in a book by Wilson (angle theta on page 59, figure 4-4):
http://books.google.com/books?hl=en&lr=&id=CPkvsDrPiv0C&oi=fnd&pg=PA1&dq=wilson+decius+cross+1955&ots=vjpHB7SwU3&sig=x02sCl1FPPDTtmswPmo9jGIeq7I#v=onepage&q=angle&f=false
I have seen a number of modern references with this definition too, e.g. page 5 figure 2 in:
http://scitation.aip.org/getpdf/servlet/GetPDFServlet?filetype=pdf&id=JCPSA6000133000003034114000001&idtype=cvips&doi=10.1063/1.3429610&prog=normal
or search for Wilson here:
http://www.charmm.org/documentation/c32b2/mmff.html
I believe that the angle between the planes should be called the improper dihedral angle instead. Please can the documentation on the cvff improper be updated to make this clear?
There is also some discussion of this in relation to cvff here (the conclusion appears to be that cvff should actually use the Wilson angle rather than the angle between planes):
http://www.ccl.net/chemistry/resources/messages/1996/09/19.008-dir/index.html
Tom Coles