It is understood the intrusion happened earlier in the week, before adverse weather conditions forced the show to pre-record an episode on Friday as Storm Arwen battered the UK with winds as high as 98mph.
A statement from the programme said: "A security breach was identified and an intruder was removed from the location with immediate effect.
"The safety of both our celebrities and our production crew remains our primary concern."
It has been an eventful first week at Gwrych Castle in North Wales, where the programme is being filmed for a second year because of the coronavirus pandemic.
On Thursday Richard Madeley announced he was leaving the show after he was taken to hospital as a precaution when he fell ill in the early hours of the morning.
The TV star later said he felt "fit as a fiddle" and was "gutted" that Covid restrictions meant he was not able to return to the show.
Friday's episode of the ITV series was the first to be pre-recorded due to the weather conditions.
Co-hosts Ant McPartlin and Declan Donnelly recorded their links early in the evening as part of "precautionary measures" put in place by ITV following severe storm warnings.
The duo posted a video from outside the castle ahead of the show, with the wind whipping and roaring in the background.
"You've probably heard we've had to record the show a bit earlier tonight because of the winds here at Storm Arwen," said Donnelly.
"It's bad, isn't it, it's really windy. You can hear it, it's rattling the tent, it might blow this away, so we've got to get it done and get out of here," said McPartlin.
"It's due to get worse later so everyone is afraid that the satellite is going to go down and all of that, so it's a bit risky to do a live show, so we're going to have to do it recorded tonight, which is the first time we've ever done that," added Donnelly.
He said: "It's very, very windy but even worse than that it was lentils for lunch, so Ant is a bit windy as well."