The quest for a B-mode imprint from primordial gravity waves on the polarization of the cosmic microwave background (CMB) requires the characterization of foreground polarization from Galactic dust. We present a statistical study of the filamentary structure of the 353 GHz Planck Stokes maps at high Galactic latitude, relevant to the study of dust emission as a polarized foreground to the CMB. We filter the intensity and polarization maps to isolate filaments in the range of angular scales where the power asymmetry between E-modes and B-modes is observed. Using the Smoothed Hessian Major Axis Filament Finder (SMAFF), we identify 259 filaments at high Galactic latitude, with lengths larger or equal to 2° (corresponding to 3.5 pc in length for a typical distance of 100 pc). Thesefilaments show a preferred orientation parallel to the magnetic field projected onto the plane of the sky, derived from their polarization angles. We present mean maps of the filaments in Stokes I, Q, U, E, and B, computed by stacking individual images rotated to align the orientations of the filaments. Combining the stacked images and the histogram of relative orientations, we estimate the mean polarization fraction of the filaments to be 11%. Furthermore, we show that the correlation between the filaments and the magnetic field orientations may account for the E and B asymmetry and the CℓTE/CℓEE ratio, reported in the power spectra analysis of the Planck353 GHz polarization maps. Future models of the dust foreground for CMB polarization studies will need to take into account the observed correlation between the dust polarization and the structure of interstellar matter.