I would like to do some phonon calculation on V2O5 compound. I selected the compound with least Hull value. But I am getting imaginary modes in my calculations. Is there any other thumb rule to select the compounds apart from the Hull value ?
It seems you are referring to mp25620, the most stable compound against decomposition among the V2O5 polymorphs we know about. We do not currently have phonon data available for comparison. We don’t know how your calculation was constructed, and debugging it is out of scope for us. Thus, it makes most sense here to simply clarify what we mean by stability / e_above_hull and how we obtain it.

We use thermodynamic phase diagram to estimate the compound stability. Detailed documentation can be found here. The method has been benchmarked among many compounds to ensure its reliability and accuracy.

Matched ICSD entries can be used as an indicator to justify the stability. Most of the time the data from the ICSD are observed in experiments, i.e. the compound can be synthesized/obtained experimentally. There are at least 12 ICSD ids associated with this very phase (mp25620), so this phase has been repeatedly reported.
It is true that the stability can be gauged from the phonon spectra; however, MP does not have this feature presently due to its complexity and high computational cost. Phonon calculations often involve some sort of perturbation and Brillouin zone warping; given other considerations such as strain states, onsite Coulomb interaction correction, etc., the phonon spectra calculation can sometimes be tricky.
Existing literature supports there being no imaginary part of the phonon in their calculations for this compound, e.g.,
http://journals.aps.org/prb/pdf/10.1103/PhysRevB.89.045109 and
http://www.sciencedirect.com/science/article/pii/003810989290095Q.
[Thanks to M. Liu for assistance with this answer]
The dynamic stability of the compound has to be checked by computing the phonon dispersion. I selected the compound mp25620 and optimized the geometry with the input files you provided (INCAR, KPOINTS and POSCAR). After ensuring the proper force convergence (~10^4 eV, best in this case !), I computed the phonon dispersion using DFPt (IBRION 8). This exercise has been done with various supercells of sizes 1x1x1, 2x2x1,2x2x2,3x2x2 etc. in all cases I am getting a an imaginary mode along YGamma direction. The paper you attached doesn’t show full phonon dispersion.
SYSTEM = interface
ALGO = Fast
EDIFF = 1e8
ENCUT = 520
IBRION = 8
ICHARG = 0
ISMEAR = 5
ISPIN = 2
LDAU = True
LDAUJ = 0 0
LDAUL = 2 0
LDAUPRINT = 1
LDAUTYPE = 2
LDAUU = 3.25 0
LMAXMIX = 4
LORBIT = 11
LREAL = FALSE
LWAVE = False
MAGMOM = 45 100.6
NELM = 100
NSW = 1
PREC = Accurate
SIGMA = 0.05
ADDGROD = TRUE