Context. It is well known that stars with giant planets are, on average, more metal-rich than stars without giant planets, whereas stars with detected low-mass planets do not require to be metal-rich. Aims: With the aim of studying the weak boundary that separates giant planets and brown dwarfs (BDs) and their formation mechanism, we analyze the spectra of a sample of stars with already confirmed BD companions both by radial velocity and astrometry. Methods: We employ standard and automatic tools to perform an equivalent width (EW) based analysis and to derive chemical abundances from the CORALIE spectra of stars with BD companions. Results: We compare these abundances with those of stars without detected planets and with low-mass and giant-mass planets. We find that stars with BDs do not have metallicities and chemical abundances similar to those of giant-planet hosts, but they resemble the composition of stars with low-mass planets. The distribution of mean abundances of α-elements and iron peak elements of stars with BDs exhibit a peak at about solar abundance, whereas for stars with low-mass and high-mass planets the [Xα/H] and [XFe/H] peak abundances remain at ~-0.1 dex and ~+0.15 dex, respectively. We display these element abundances for stars with low-mass and high-mass planets, and BDs versus the minimum mass, mC sin i, of the most massive substellar companion in each system, and we find a maximum in α-element as well as Fe-peak abundances at mC sin i ~ 1.35 ± 0.20 Jupiter masses. Conclusions: We discuss the implications of these results in the context of the formation scenario of BDs compared with that of giant planets. Appendices are available in electronic form at http://www.aanda.org