Context. Solar chromospheric fibrils, as observed in the core of strong chromospheric spectral lines, extend from photospheric field concentrations suggesting that they trace magnetic field lines. These images have been historically used as proxies of magnetic fields for many purposes. Aims: Use statistical analysis to test whether the association between fibrils and magnetic field lines is justified. Methods: We use a Bayesian hierarchical model to analyze several tens of thousands of pixels in spectro-polarimetric chromospheric images of penumbrae and chromospheric fibrils. We compare the alignment between the field azimuth inferred from the linear polarization signals through the transverse Zeeman effect and the direction of the fibrils in the image. Results: We conclude that, in the analyzed fields of view, fibrils are often well aligned with the magnetic field azimuth. Despite this alignment, the analysis also shows that there is a non-negligible dispersion. In penumbral filaments, we find a dispersion with a standard deviation of 16°, while this dispersion goes up to 34° in less magnetized regions.