Men whose mothers consumed less than a glass of wine a day while expecting had sperm counts much lower than those exposed to little or no alcohol while in the womb.
The results could explain why sperm quality has declined in recent decades, Dr Cecilia Ramlau-Hansen and colleagues told a meeting of the European Society of Human Reproduction and Embryology in Rome.
They studied 347 sons of 11,980 women recruited to a Danish health survey between 1984 and 1987 and found that if the mothers had drunk 4.5 or more drinks a week while pregnant, the sperm concentration of their sons 20 years later was a third lower.
A drink was measured as 12g of alcohol, the equivalent of about half a pint of beer, a small glass of wine or a glass of spirits.
Dr Ramlau-Hansen, of Aarhus University Hospital, Denmark, said: "Our study shows there is an association between drinking a moderate amount of alcohol, about four to five drinks a week, during pregnancy, and lower sperm concentrations in sons."