The deepest ground-based optical surveys (in terms of surface brightness mu~31 mag/arcsec^2) show that we are close to the dreaded source confusion limit. The typical distance between sources in these images is a few arcseconds, on the order of the natural seeing in these images. When this source confusion limit is reached, low surface brightness science is severely compromised. In this contribution we present our analysis of the background source density in the ultra-deep LIGHTS survey obtained with the LBT telescope. We will show what the field-to-field variation is, the difficulties in estimating realistic backgrounds, and how these problems will affect new deep ground-based surveys such as LSST. We will also show the prediction of the limiting surface brightness we can expect to reach (with natural seeing) in the near future, and how the breakthrough in this field will have to come in the next few years from adaptive optics imaging over large areas and/or the use of deep space surveys.