The study of cool atmospheres is currently a fruitful field of interaction between theorists developing better atmosphere models and observers discovering new cool objects and analysing their emission spectrum. Save for a few close binaries with known parallax and a dynamical mass determination, comparison of spectra to models is the only method available to observers to derive the fundamental physical parameters of a substellar object, such as mass, temperature, age or metallicity. Most of our knowledge of key substellar parameters thus rests upon theoretical models reliability. This issue is particularly acute for the newly imaged exoplanets, whose parameter range of youth and extreme low-mass has not been probed before, but we show that model reliability remains problematic even for field L and T dwarfs, whose observed spectra have guided the model development over the past fifteen years. We present several models-to-observations comparisons, which remind that even the last-generation atmosphere models keep having trouble quantitatively reproducing overall spectral parameters such as absorption band strength measured through spectral indices and broad band absolute magnitudes. These tests confirm that the models cannot yet be used at face value to read off physical parameters, and need empirical calibration on well constrained benchmarks. While the models are sufficiently predictive to produce useful differential measurements over limited parameter ranges, using them without empirical rescaling and anchoring generally leads to significantly inaccurate results.