Here are some references on the subject:
-Numerical Recipes (any edition, see the chapter on Fourier Transform
-J. Chem. Phys. 1993, 98, 4984-4990
-J. Chem. Phys. 2003, 119, 11792-11805
The "power spectrum" is defined in Num. Rec. as the square of the
magnitude of the Fourier transform.
The two journal articles define the density of states as the same
integral that you gave, but multiplied by some constants (1/kT terms).
Even in that case I think the square of the magnitude must be used,
otherwise you could see negative values in the density of states.
In either case, I believe that only the real parts of the Fourier
transform should be used. The bigger issue in my opinion is that using
canned FFT routines result in N/2 data points in the power spectrum,
when the original VACF contains N data points. The only suggestion
offered in Num. Rec. is to augment your VACF with zeroes until you have
2N data points.
We're working on code to calculate VACFs from LAMMPS dump files, but we
still need to validate it. Let me know if you want to compare codes.