The Scot was an early leader during the World Championship event at the London Stadium but failed to control the race for the duration.
Faith Kipyegon of Kenya won gold with 4:02:59, a time bettered by nine athletes in the field this year alone. Jenny Simpson of the US claimed second.
“I gave it everything I could,” Muir told the BBC. “But in the last 50m I got tied up.”
Muir, who was an early favourite for the event, looked a certainty for a medal with 200m remaining but lost out on a podium place in agonising circumstances.
“Muir wound it up and she just ran out of legs in those final 20 seconds,” athletics expert Allison Curbishley told the BBC. “The master tactician of Jenny Simpson of the USA again gets on the podium.
“Two Brits in the top six is phenomenal but Laura Muir in that fourth position, that is heartbreak, absolute heartbreak. The world record holder finished 12th. That’s where we are at with this event.”
In January, Muir broke the British indoor record over 5,000m at the Emirates Arena in Glasgow.
Muir’s time of 14 minutes 49.12 seconds beat fellow Scot Liz McColgan’s 25-year-old record by 14 seconds.