In fact, the dovecot is a historic monument dating back to the 16th century, and – despite appearances – it's not really in his garden at all. It is discreetly fenced off with its own entrance and under the care of Historic Scotland.
But as he relaxes in the modern, open-plan home created in what used to be the Dovecot tapestry studio in Corstorphine, Mr Barrett – who will step down as Lib Dem MP for Edinburgh West at the general election – admits the expenses scandal has left "a real cloud" over Westminster.
And the 55-year-old – a former youth football star, who in his previous career ran a firm behind a string of TV ads and a Bafta award-winning documentary – has little sympathy for those who abused the system.
"There has been a lot of incompetence, a lot of neglect about the details of what people should have done, but there have been some people who milked the system to the maximum.
"And at the end of the day, I think those guilty of major offences should be locked up."
He also feels the expenses scandal will leave a serious long-term legacy, starting with the election.
"More MPs will be standing down this time than at any other election since the war.
"I think this will be a really nasty election with lots of candidates challenging the incumbents over the minutiae of their expenses.
"About 300 MPs are going to be brand new. We will have a hugely inexperienced parliament dealing with the worst economic crisis in living memory. That's the long-term legacy. We have a difficult time ahead."
But he does feel irritated that all MPs have been tarred with the same brush.
"Even those folk who have not really been affected like myself, it does stick in the throat that I'm constantly reading that living in a one-bedroomed flat in London for three or four days a week is somehow a perk."
And he says the system that leaves MPs to arrange everything for themselves and claim it back is a recipe for trouble.
"When I was a councillor, I had staff, IT, a desk and an office to work in. When I became an MP I was told, 'Go out, find an office, rent it from the landlord, get hold of your IT, pay your bills and claim the money back.
"Until that's swept away, until MPs are given the tools to do the job, it will be a major distraction and we'll never restore public trust."
It was in the summer that Mr Barrett, first elected in 2001, took everyone by surprise by announcing he was going to quit Westminster despite having the second safest Lib Dem seat in the UK.
He said he wanted a change – and nearly six months on he is convinced he has made the right decision.
After long service as a community councillor, councillor and MP, he says it is time for a change in the life-work balance.
"I feel in campaigning terms I've been on the go for 25 years.
"I feel young and fit enough to go and do something new. If I stayed on for another term I'm going to be approaching 60 and that's going to be too late to do something new.
"The other thing is I want to have a bit more time, now I'm a granddad."
He has two granddaughters – Maria, aged five, and Hope, two.
"Maria has the same problems as David Cameron's son Ivan had – cerebral palsy and epilepsy. She is unlikely ever to be able to walk and I'd rather spend the weekend taking the kids to the park."
At the moment he has an average of three or four constituency events on a Saturday and much of Sunday is spent dealing with e-mails and correspondence.
"On Sundays I usually see the kids for a couple of hours and it's not enough.
"Maria has started school and she was in the nativity. She was the star – literally, she had to hold the wand with the star. Her conversation is starting to develop, she's got a sense of humour and she's great company."
Mr Barrett is currently considering with his artist wife Carol what he should do next. He has nothing lined up, but he is adamant he is not retiring.
"The thought of sitting back and putting my feet up is appealing for a day."
He briefly considered standing for the Scottish Parliament or European Parliament or trying to get back onto the council, but now rules that out.
If life had evolved differently, Mr Barrett might have found himself on the football field instead of running in elections.
At Carrick Knowe Primary School, he played for the school team against nearby Broomhouse Primary, whose players included Graeme Souness.
And later he had a trial for Partick Thistle, but was rejected.
Before he became an MP, Mr Barrett ran his own communications business, producing corporate videos for big companies like KiwkFit, Standard Life and Royal Bank, and making TV commercials and documentaries, including a Scottish Bafta-winning film about Monktonhall colliery.
And he tells how filming a fashion show at Duns Castle in the Borders ended with him dancing on the catwalk with Ivana Trump.
"I was there doing a corporate film for a fashion designer. There was a bit of glitz and glamour, Ivana Trump was over.
"At the end of the night, when I was tidying away, the catwalk turned into a disco and Ivana Trump asked me if I would like to dance on the catwalk."
He also made videos at no cost for causes like the Sick Kids hospital and a documentary about children dying from cancer in Ukraine following the Chernobyl nuclear disaster.
So he might return to that kind of work. "Maybe at a time of recession, creating jobs in Edinburgh would be no bad thing."
He would also like to maintain his interest in overseas aid, having served for six years on the Commons international development select committee. And his role as disability spokesman for the Lib Dems could lead to further involvement on that front too.
He successfully campaigned to get a better deal for blind people on mobility allowance and was pursuing the issue of compensation for thalidomide sufferers when the Government announced this week another 20m and an apology.
"My mother was offered thalidomide when she had morning sickness with my younger brother. The GP said 'great new drug – have this' and she turned it down. My younger brother would have been a thalidomide victim."
He says he believes there is still a long way to go in changing society's attitudes to disability.
"We've moved forward on gender issues, race issues and religion, but I think we're way behind on disability issues.
"I have asked employers, if an employee became disabled how far would they go to help them if they broke their legs, if they had vision problems if they had mobility problems or hearing problems. Most employers would go that extra mile. What I've been trying to get is that degree of consideration transferred to recruitment because a lot of people, who have a range of disabilities that doesn't stop them doing the job, don't get as far as the interview."
Mr Barrett will be taking time to consider his next move. But he says: "I've always been self-motivated. I'm really quite relishing the challenge of the years ahead. I feel almost rejuvenated."
But he quickly adds he is still 100 per cent committed to the Lib Dems and, come the election, he'll be out there doing his share of leafleting.