Mediate, beaten by Tiger Woods in a play-off for the 2008 US Open at Torrey Pines, completed a wire-to-wire win at Benton Harbor in Michigan to claim his first over-50s major.
But the 53-year-old was pushed all the way by Montgomerie as the big Scot produced a gutsy second defence of the Alfred S Bourne Trophy.
Chasing a fourth major victory in the senior ranks, Montgomerie covered the first 10 holes in the final round in four-under.
His two-putt birdie at the tenth moved him within a shot of Mediate as the back nine turned into a two-horse race.
Montgomerie shaved the hole with a birdie attempt at the 13th, where Mediate’s par putt just caught the edge of the hole.
The defending champion then dropped his only shot of the day at the par-4 14th following a poor first putt before the 15th proved another pivotal hole.
Montgomerie let his putter drop as he covered his eyes in disbelief as a 15-foot eagle putt burnt the edge of the hole without toppling in.
And he immediately had salt rubbed into his wound as Mediate, having been forced to lay up after missing the fairway from the tee, then rolled in a 10-foot birdie putt to stay two ahead.
If that wasn’t enough to convince Montgomerie, bidding to emulate Hale Irwin by winning this event three years in a row, that it wasn’t to be his day this time around, he certainly knew two holes later.
Mediate pushed his tee shot into sand at the par-3 17th but holed out for a birdie, giving him a three-shot cushion playing the last.
Both players finished with pars, Mediate closing with a 66, one better than his playing partner.
Mediate finished on 19-under 265, three better than Montgomerie, who, in turn, ended up three ahead of third-placed Bernhard Langer (67) and American Brandt Jobe (68).
“I did nothing wrong,” said Montgomerie. “Went out and shot 67. All credit to Rocco. He shot 66. His up-and-down game today was brilliant. The first five holes he had five putts and to get momentum against that is very difficult. He’s got to give you something and he never gave me anything.”
Mediate paid tribute to the runner-up, saying: “Monty is relentless. But I made everything - I just kept making and making and somehow kept it together.”
On holing out of the bunker at the penultimate hole, he added: “It was ridiculous but I did a lot of stuff this week that you aren’t supposed to do.”