Quantitative thermodynamical, dynamical and magnetic properties of the solar and stellar plasmas are obtained by interpreting their emergent non-polarized and polarized spectrum. This inference requires the selection of a set of spectral lines that are particularly sensitive to the physical conditions in the plasma and a suitable parametric model of the solar/stellar atmosphere. Nonlinear inversion codes are then used to fit the model to the observations. However, the presence of systematic effects, like nearby or blended spectral lines, telluric absorption, or incorrect correction of the continuum, among others, can strongly affect the results. We present an extension to current inversion codes that can deal with these effects in a transparent way. The resulting algorithm is very simple and can be applied to any existing inversion code with the addition of a few lines of code as an extra step in each iteration.