Speaking to Talk Radio, Fox said that he was not expecting people to "kick off" over his comments, adding that those who criticised him for racism are "everything they accuse you of".
He said: "The wokeists, I think, are fundamentally a racist bunch. They see colour everywhere.
"Identity politics is extremely racist as well."
He added that he often gets it "in the neck" for being a "posh boy actor" who is part of a famous acting dynasty, which includes his father James.
Fox said that he is fine with people "taking a pop" at him, but added: "I just lose interest when someone says I want to unload both barrels of a shotgun in your face."