After the formation of the sun the Protoplanetary disk formed the planets of the Solar System out of a disk-shaped mass of dust and gas left over from the formation of the Sun.
The early formation of the solar system- from the Protoplanetary disk
The inner Solar System, the region of the Solar System inside 4 AU, was too warm for molecules like water and methane to condense, so the planetesmals that formed there could only form from compounds with high melting points, such as metals like iron, nickel, aluminum, and rocky silicates. These rocky bodies would become the terrestrial planets (Mercury, Venus, Earth, and Mars).
Formation of the Planets
These compounds are quite rare in the universe, comprising only 0.6% of the mass of the nebula, so the terrestrial planets could not grow very large. The terrestrial embryos grew to about 0.05 Earth masses and ceased accumulating matter about 100,000 years after the formation of the Sun; subsequent collisions and mergers between other planetesimals also formed by Protoplanetary disk allowed terrestrial planets to grow to their
Planets forming from and among the planetesimals in the inner solar system.
However, one question is why did Mars came out so small compared with Earth ? Earth's Equatorial radius is 6,378.1 km and Mars' Equatorial radius is 3,396.2 km or 0.533 Earths.
A study proposes that Jupiter had migrated inward about 1.5AU when Saturn formed, which means it moved into the area that would become known as the asteroid belt.
The early Asteroid Belt
The early Asteroid Belt would contain dry asteroids and water-rich objects similar to comets. This means that Jupiter must have accumulated mass there at the expense of Mars then later Jupiter migrated back to its present position. Jupiter thus would have consumed much of the material that would have created a bigger Mars.
During the Solar System's formation, Mars was created out of the protoplanetary disk that orbited the Sun as the result of a random run-away accretion. Mars has many distinctive chemical features caused by its position in the Solar System. Elements with comparatively low boiling points such as chlorine, phosphorus and sulfur that are much more common on Mars than Earth; these elements were probably removed from areas closer to the Sun.
At about 4600 million years, the Proto-Earth formed at the inner hotter edge of the habitable zone of the Solar System. At this stage the luminosity of the sun was very high yet the size of the sun was small. Liquid water may have existed on the surface of the Proto-earth, probably due to the greenhouse warming of high levels of methane and carbon dioxide present in the atmosphere making it similar to a cross between modern day Venus and Mars. After another 3750 million years during the Archean Eon, the Precambrian Super eon and Cryptic era start as the Earth–Moon system forms, possibly as a result of a glancing collision between proto–Earth and the hypothetical proto-planet Theia.
The Formation of the Moon