Men who ate higher amounts of cheese, butter, milk and yoghurt were between 7% to 76% more likely to develop the disease, a study by US researchers found.
Experts at the Mayo clinic in Chicago reviewed 47 studies published since 2006, with more than a million participants, to better understand the risks of prostate cancer associated with plant and animal-based foods.
Two meta-analyses and seven cohort studies on dairy consumption found the men had an increased risk of between 7% and 76% of developing prostate cancer.
But other studies did not find a link.
Researchers detected no link between an increased risk of prostate cancer and eating red and white meat, processed meat and fish.
But they found a decreased risk of developing the disease in vegetarians and vegans.
John Shin, MD, a Mayo Clinic oncologist and lead author on the study, said: "Our review highlighted a cause for concern with high consumption of dairy products.
"The findings also support a growing body of evidence on the potential benefits of plant-based diets."
Dairy products are rich in calcium and the researchers suggest the mineral may play an important role in the link between higher consumption and increased risk.
But they caution that more research is needed to determine the nature of the associations found.
The study is published in the Journal of the American Osteopathic Association.
Tom Sanders, Professor emeritus of Nutrition and Dietetics, King's College London, said there were "weaknesses" in the methods used in the study.
He said: "Vegans have about a 35% lower risk of prostate cancer than meat-eaters but this may reflect the fact they are usually much lighter than meat-eaters rather than the absence of dairy foods from their diets.
"It is notable that the incidence of prostate cancer is much higher in black men of African ethnic origin, who generally consume few dairy products, compared with white men.
"The prevalence of prostate cancer has increased markedly in south-east and east Asia, where few dairy products are consumed, which would indicate that lifestyle factors, other than dairy food consumption, are responsible for the global epidemic prostate cancer."