In this work we analyse the physical properties of a sample of 56 spectroscopically selected star-forming (SF) Lyα emitting galaxies at 2.0 ≲ z ≲ 3.5 using both a spectral energy distribution (SED) fitting procedure from rest-frame UV to mid-IR and direct 160 μm FIR observations taken with the Photodetector Array Camera and Spectrometer (PACS) instrument onboard the Herschel Space Observatory. We define LAEs as those Lyα-emitting galaxies whose rest-frame Lyα equivalent widths (Lyα EWrest - frame) are above 20 Å, the typical threshold in narrow-band searches. Lyα-emitting galaxies with Lyα EWrest - frame < 20 Å are called non-LAEs. As a result of an individual SED fit for each object, we find that the studied sample of LAEs contains galaxies with ages mostly below 100 Myr and a wide variety of dust attenuations, SFRs, and stellar masses. The heterogeneity of the physical properties is also seen in the morphology, ranging from bulge-like galaxies to highly clumpy systems. In this way, we find that LAEs at 2.0 ≲ z ≲ 3.5 are very diverse, and do not have a bimodal nature, as suggested in previous studies. Furthermore, the main difference between LAEs and non-LAEs is their dust attenuation, because LAEs are not as dusty as non-LAEs. On the FIR side, four galaxies of the sample (two LAEs and two non-LAEs) have PACS detections. Their total IR luminosities place all of them in the ULIRG regime, and all are dusty objects, with A1200 ≳ 4 mag. This is an indication from direct FIR measurements that dust and Lyα emission are not mutually exclusive. This population of red and dusty LAEs is not seen at z ~ 0.3, which suggests an evolution with redshift of the IR nature of galaxies selected via their Lyα emission. Herschel is an ESA space observatory with science instruments provided by European-led Principal Investigator consortia and with important participation from NASA.Appendix A is available in electronic form at http://www.aanda.org