After Michael Tidser had cancelled out Tom Beadling’s opener for the Fife visitors it looked as if the points would be shared until Oliver, pictured, pounced in stoppage time.
“We nicked it today, there is no doubt about that, and I’m not going to shy away from that,” Duffy admitted.
“To be honest, if it ended up a draw I wouldn’t have been too disappointed because I didn’t think we played that well.”
Morton started well with a Gary Harkins free-kick forcing a smart early save from Robinson.
However, the opener came at the other end of the park when Pars new-boy Beadling, on-loan from Sunderland, took advantage of poor defending to rifle home.
The Cappielow side replied before the break when Oliver nodded to Tidser who drilled a shot into the net from the edge of the area.
Dunfermline were denied almost immediately when Nicky Clark dispatched a tremendous overhead kick into the top corner but it was controversially ruled offside.
Morton applied pressure on the restart and had two efforts cleared off the line in quick succession, Robinson clawing Luca Gasparotto’s header away before a wayward header from Pars skipper Callum Morris forced another last-ditch clearance.
Robinson came to Dunfermline’s rescue again when he held an Oliver shot which lacked the necessary power or accuracy to trouble the in-form keeper.
Ton looked like they would grab a winner four minutes from time when new signing Frank Ross, on-loan from Aberdeen, jinked past his defender before curling an effort goalward only for Robinson to produce another excellent stop.
But Robinson was finally beaten again at the death when Bob McHugh outpaced Morris to a loose ball, rounded the stranded keeper and unselfishly slid the ball to strike-partner Oliver who finished easily.
Allan Johnston, the Dunfermline manager, said: “I thought we were the only team that was going to win the game, but too many times this year – especially against Morton – we’ve been caught with a sucker punch.
“You look at Nicky Clark’s goal in the first half, he’s three yards onside, I’ve seen it on the video and that changes everything. It’s a really poor decision.
“I thought we were the better team so it’s a wee bit frustrating again but we’ve got to pick ourselves up.”