In recent years, the quality of male sperm has shown a downward trend year by year. The male infertility rate in China continues to increase. It has been found that the impacts of environmental factors on male semen quality are mainly negative. Inorganic metallic elements as environmental contaminants have become a class of chemicals that cannot be ignored, and their health impacts on human reproductive systems have been received widespread attention and research. They certainly play a significant role in impairing male reproductive ability and are relative to the lower and lower semen quality. This review focused on the relationship between exposure to environmental metallic elements and semen quality of humans and animals, as well as summarized specific results from epidemiological studies, animal trials, or molecular experiments to provide a theoretical basis for protecting male reproductive capacity.