Bars are strong drivers of secular evolution in disk galaxies. Bars themselves can evolve secularly through angular momentum transport, producing different boxy/peanut and X-shaped bulges. Our Milky Way is an example of a barred galaxy with a boxy bulge. We present a self-consistent N-body simulation of a barred galaxy which matches remarkably well the structure of the inner Milky Way deduced from star counts. In particular, features taken as signatures of a second ``long bar" can be explained by the interaction between the bar and the spiral arms of the galaxy. Furthermore the structural change in the bulge inside l=4° measured recently from VVV data can be explained by the high-density near-axisymmetric part of the inner boxy bulge. We also compare this model with kinematic data from recent spectroscopic surveys and estimate the pattern speed of the bar of the Milky Way.