Low-level exposure to environmental heavy metals during pregnancy is common, and the effects of such exposure on the growth and development of the fetus may continue after birth. In this paper, we reviewed the epidemiological studies on heavy metal exposure during pregnancy and children's physical development at home and abroad in recent years, focusing on the associations between four common heavy metals, namely, cadmium, mercury, lead, and arsenic, and children's physical development levels and growth trajectories. The results of existing studies showed that cadmium and mercury exposure during pregnancy was associated with early childhood obesity, lead exposure during pregnancy may lead to dual effects of childhood obesity and growth retardation, and arsenic exposure during pregnancy was associated with childhood growth retardation. At the same time, cadmium and lead exposure during pregnancy had a more significant effect on boys, and there was a sex effect. Most studies of children's growth trajectory showed that exposure to cadmium, arsenic, and lead during pregnancy was often associated with lower childhood body mass index (BMI)/weight-for-age Z-score (WAZ) and height-for-age Z-score (HAZ) growth trajectories, even into adolescence, where the first trimester and the third trimester may be critical windows for cadmium and lead exposure, respectively. The results of mercury exposure during pregnancy associated with children's growth trajectories were controversial due to differences in methods for evaluating mercury exposure. In addition, there may be certain combined effects of mixed exposure to heavy metals during pregnancy on the physical growth and development of children. There is still a need to further elucidate the effects of single metal and multiple heavy metal interactions on children's physical development by combining population-based epidemiological studies with multiple sample sources and time points, and to strengthen basic research to broaden the new understanding of involved mechanisms.