We present the work developed within the science team of the Very Large Telescope Interferometer Spectro-Imager (VSI) during the Phase A studies. VSI aims at delivering ~ 1 milliarcsecond resolution data cubes in the near-infrared, with several spectral resolutions up to 12 000, by combining up to 8 VLTI telescopes. In the design of an instrument, the science case plays a central role by supporting the instrument construction decision, defining the top-level requirements and balancing design options. The overall science philosophy of VSI was that of a general user instrument serving a broad community. The science team addressed themes which included several areas of astrophysics and illustrated specific modes of operation of the instrument: a) YSO disks and winds; b) Multiplicity of young stars; c) Exoplanets; d) Debris disks; e) Stellar surface imaging; f) The environments of evolved stars; g) AGN tori; h) AGN's Broad Line Region; i) Supermassive black-holes; and j) Microlensing. The main conclusions can be summarized as follows: a) The accessible targets and related science are extremely sensitive to the instrument limiting magnitude; the instrument should be optimized for sensitivity and have its own fringe tracker. b) Most of the science cases are readily achievable with on-axis fringe tracking, off-axis fringe tracking enabling extra science. c) In most targets (YSOs, evolved stars and AGNs), the interpretation and analysis of circumstellar/nuclear dust morphology requires direct access to the gas via spectral resolved studies of emission lines, requiring at least a spectral resolution of 2 500. d) To routinely deliver images at the required sensitivity, the number of telescopes in determinant, with 6 telescopes being favored. e) The factorial increase in the number of closure phases and visibilities, gained in a single observation, makes massive surveys of parameters and related science for the first time possible. f) High dynamic range imaging and very high dynamic range differential closure phase are possible allowing the study of debris disks and characterization of pegasides. g) Spectro-imaging in the near-infrared is highly complementary to ALMA, adaptive optics and interferometric imaging in the thermal infrared.