Mr Oliver has come under regular attack by online commentators since he came out to support the Union in 2014, with criticisms intensifying after he described the prospect of a second independence referendum as a “cancerous presence” and Alex Salmond as a “round wrecking ball of a man".
Mr Oliver, in an interview with the Mail on Sunday, said he felt he has become a “lightning rod” for nationalists and described the vitriol as ironic given he had held “very middle of the road” views all his life.
He added: "I feel as if I’m in an entrenched position in a war of attrition.
"Since 2014, I’ve been a lighting rod for the nationalists because I spoke up in favour of the United Kingdom.
"It’s become an article of faith now that if you’re truly Scottish you would vote for independence and therefore if you’re in favour of the continuation of the United Kingdom then you are not Scottish.
"That’s just become a binary position that’s being pushed by a section of the population. But I grew up British and Scotish. I never gave it a second thought.
Most recently, Mr Oliver was condemned on social media after describing his “mortifcation and heartbreak” that Prime Minister Boris Johnson and his family appeared to have cut short their holiday in Applecross, Wester Ross, because they ‘felt unsafe’ after pictures of their holiday home appeared in a newspaper.
He said: “With Boris Johnson, no one except him and his own people know why he left Scotland early to return to London.
"But there was certainly a perception that he’d been scared or chased out of Scotland and I thought ‘I don’t want even that suggestion about this country’.”
Mr Oliver is due to stand down as President of the National Trust for Scotland later this month after three years in the post.
He said that during is tenure he felt he could not speak freely on issues that mattered to him given the potential damage to the reputation of the organistion.
Mr Oliver said: “I couldn't say anything as me.”
Mr Oliver, who lives in Stirling with his wife Trudi and their three children, said some of the comments made about him had been “poisonous” – with many of them about his appearance.
Mr Oliver added: “I get endless comments about my hair or about my face or the way I walk or the things I do on TV and a lot of it is just vindictive, personal attack. I just won’t give into it.
"The longer it goes on and the more vicious it gets, the more obliged I feel I have to stand up to it.
"When I feel like I am being confronted by bullying, I don’t feel I can credibly say to my kids they have to stand up to bullies unless they see my doing it too.”