The incident was one of the worst involving United States or coalition troops since the start of the nearly 14-year war.
The US-led international coalition said the service members were killed in an apparent blue-on-blue incident, which an Afghan official said took place in southern Zabul province.
It is thought to be one of the most serious cases involving coalition friendly fire during the war.
“The casualties occurred during a security operation when their unit came into contact with enemy forces. Tragically, there is the possibility that fratricide may have been involved. The incident is under investigation. Our thoughts are with the families of those killed during this difficult time,” the coalition said in an announcement.
One of the worst incidents of friendly fire came in April 2002 when four Canadian soldiers, all in their 20s, were killed when an American F-16 jet fighter dropped a 500lb bomb on a group near a night-firing exercise in southern Kandahar.
The so-called Tarnak Farm incident saw American pilots Harry Schmidt and William Umbach initially charged with manslaughter though the latter’s charges were later dropped and the former’s reduced to dereliction of duty.
A senior police official in southern Zabul said the coalition soldiers may have been killed when they called for close air support after coming under fire.
Provincial police chief General Ghulam Sakhi Rooghlawanay said there was a joint operation by Afghan and Nato troops in the area’s Arghandab district early on Monday.
He added: “After the operation was over, on the way back, the joint forces came under the attack of insurgents, then foreign forces called for an air support. Unfortunately five Nato soldiers and one Afghan army officer were killed mistakenly by Nato air strike.”
The coalition would not comment on his version of events and Nato headquarters in Brussels also declined to comment.
The Taleban claimed responsibility for the attack against the joint force.
The insurgents have intensified attacks on Afghan and foreign forces ahead of the country’s presidential election run-off on Saturday.
The Kabul government is concerned there could be more violence over the vote, although the first round in April passed relatively peacefully.
A Taleban spokesman claimed a “huge number” of Nato soldiers were killed or wounded in Arghandab on Monday. The Taleban are known to exaggerate their claims.
A Nato statement said a service member died on Monday as a result of a non-battle injury in eastern Afghanistan.
The deaths bring to 36 the number of Nato soldiers killed this year in Afghanistan, with eight service members killed in June. Casualties have been falling in the US-led military coalition as its forces pull back to allow the Afghan army and police to fight the Taleban insurgency.