The Dumfries-born singer-songwriter raced on to the stage – with a pineapple on his head to mock their hairstyles – while the Irish brothers John and Edward Grimes performed a version of Queen's Under Pressure.
It was initially thought that the stage intruder was a member of the public. However, Harris was later identified as the culprit.
He had been due to appear on the sister show The Xtra Factor.
An X Factor spokesman said the musician, who collaborated with rap star Dizzee Rascal on No 1 single Dance With Me in 2008, was removed from the studio by security guards.
He said: "Apparently Calvin is a huge John and Edward fan, but it's unacceptable to storm the stage while an act is performing," the spokesman said.
The judges commended the Grimes brothers for keeping their cool in spite of Harris's antics.
Meanwhile, X-Factor judge Simon Cowell praised the twins, saying their latest performance was the best yet. He claimed, however, that he still could not judge them "in the real world".
He added: "There's no point me getting angry or serious about this; if the public like you, they like you."