The yields of evaporation residues, fusion-fission, and quasifission fragments in the Ca48+Sm144,154 and O16+W186 reactions are analyzed in the framework of the combined theoretical method based on the dinuclear system concept and advanced statistical model. The measured yields of evaporation residues for the Ca48+Sm154 reaction can be well reproduced. The measured yields of fission fragments are decomposed into contributions coming from fusion-fission, quasifission, and fast-fission. The decrease in the measured yield of quasifission fragments in Ca48+Sm154 at the large collision energies and the lack of quasifission fragments in the Ca48+Sm144 reaction are explained by the overlap in mass angle distributions of the quasifission and fusion-fission fragments. The investigation of the optimal conditions for the synthesis of the new element Z=120 (A=302) show that the Cr54+Cm248 reaction is preferable in comparison with the Fe58+Pu244 and Ni64+U238 reactions because the excitation function of the evaporation residues of the former reaction is some orders of magnitude larger than that for the last two reactions.