However, assistant manager George Mathieson is not getting carried away.
He said: “It is great to have a lead from the away leg but this tie is far from over.”
Headers from Harry Milne and Scott Ross gave Cove a first-half lead before Jamie Longworth grabbed a vital goal for Kilby in stoppage time.
Cove were without manager John Sheran, pictured, who is still recovering at home after suffering a heart attack last week.
But they got off to a dream start at K-Park, taking the lead after just three minutes.
Daniel Park’s corner to the back post picked out Milne and he rose highest to power home a header.
Paul Woods almost equalised for the hosts when he jinked his way through the Cove defence before rifling a strike towards the bottom corner, but Stuart McKenzie produced a fine stop.
In 25 minutes Ryan Strachan got on the end of Park’s driven corner and Kilby keeper Willie Muir had to be at his best to turn it away.
The visitors doubled their lead after 35 minutes when Kilby again struggled to deal with a corner and Ross headed the ball beyond Muir from close range.
Craig Howie made a vital block in 70 minutes to deny Cove a third when he got back to clear Jamie Masson’s strike that had Muir beaten all ends up.
Substitute Longworth came close to pulling one back in 76 minutes, but McKenzie was equal to his 20-yard drive.
And in stoppage time Longworth opened up space for a shot on the edge of the box and drilled in a crucial goal for the hosts, with their manager Stuart Malcolm saying: “Jamie’s goal has made a very difficult job a bit easier.”