This work presents the first comprehensive study of structure formation at the peak epoch of cosmic star formation over 1.4 ≤ z ≤ 3.6 in the Cosmic Evolution Survey (COSMOS) field, including the most massive high-redshift galaxy proto-clusters at that era. We apply the extended COSMIC BIRTH algorithm to account for a multitracer and multisurvey Bayesian analysis at Lagrangian initial cosmic times. Combining the data of five different spectroscopic redshift surveys (zCOSMOS-deep, VUDS, MOSDEF, ZFIRE, and FMOS-COSMOS), we show that the corresponding unbiased primordial density fields can be inferred, if a proper survey completeness computation from the parent photometric catalogues, and a precise treatment of the non-linear and non-local evolution on the light-cone is taken into account, including (i) gravitational matter displacements, (ii) peculiar velocities, and (iii) galaxy bias. The reconstructions reveal a holistic view on the known proto-clusters in the COSMOS field and the growth of the cosmic web towards lower redshifts. The inferred distant dark matter density fields concurrently with other probes like tomographic reconstructions of the intergalactic medium will explore the interplay of gas and dark matter and are ideally suited to study structure formation at high redshifts in the light of upcoming deep surveys.