The four-time Commonwealth champions - who have won every match they have ever played at the Games - surged ahead with three tries in a dominant first-half display.
But, when the Glasgow 2014 hosts regrouped at the break, the Kiwis resorted to some cynical tactics to ensure they retained their proud unbeaten record.
At one point, they had just five men on the field after Gillies Kaka and Tim Mikkelson were sin-binned for blatant fouls which robbed the Scots of a scoring chance.
Gregor, whose side must now beat Barbados and Canada on Saturday night to progress from Pool A, said: “We have played New Zealand loads of times on the World Series over the years, but that is the closest we have taken them for a while.
“You could tell they were worried about us. They requested independent touch judges before the game, which they wouldn’t have done in the past.
“The crowd helped us as well, but it’s a frustrating result.”
At the home of Scottish football giants Rangers, 45,000 fans helped generate an atmosphere more akin to an Old Firm derby than a rugby contest.
New Zealand did their best to hush the Scotland support as Ben Lam found space down the right to jog in the opening score.
Even with three full 15-a-side internationals in their starting line-up - including British and Irish Lion Stuart Hogg - the Scots could not live with the world champions, who streaked further ahead with Sherwin Stowers brace.
But, spurred on by the home crowd, Scotland came out for the second seven-minute period all pumped up.
First Lee Jones darted in to put his side on the scoreboard after Mark Bennett did well to make space for the Glasgow wing after drawing in his opposite number.
Then, with the pressure mounting, Kaka and Mikkelson resorted to underhand tactics to stem the blue tide washing over the Kiwi defence.
Both men were sin-binned within a matter of seconds of each other, allowing Jones to make the most of the Scots’ two-man advantage and cross over again with a minute left.
The Scots knew they had to win possession back straight from the restart if they were to have any hopes of ending the All Blacks’ 25-game winning streak.
But the reigning champions showed their class as they clung onto the ball before seeing out the final stages without any further damage.
Double try-scorer Jones said: “It was close, but we let them get away in the first half. A 17-point lead is always going to be tough to bring back in. But we kept hold of the ball in the second half and gave ourselves a chance.
“They cynically had to break down a couple of our attacks which could have led to tries.
“But we showed we have got enough to get out the group. That’s our first game and we will only get better. Our target is to medal and if we can get through the group we are well capable of doing that.”
Earlier on Saturday, Wales followed up resounding results from the Kiwis - who thumped Canada 39-0 in the opening match of the day - and Australia (62-7 winners over Sri Lanka) to smash Malaysia 52-0.
Adam Thomas had the Dragons ahead after just 80 seconds, and James Davis’ second-half hat-trick rounded off a comprehensive victory.
In Pool B, England got their campaign under way with an emphatic 57-0 victory of their own. Daniel Norton, Philip Burgess and James Rodwell all touched down for doubles in the eight-try win over Sri Lanka.