What Gordon Brown did next

ON Wednesday, Gordon Brown MP arrived at the headquarters of the Purvis Group, a construction company based at the Cartmore industrial estate in Lochgelly. For the past three days the character of the former prime minister had been assaulted by the comments of colleagues, as recorded in Peter Mandelson's memoirs, The Third Man.

He was, according to Tony Blair, "Mad, bad and dangerous," a "mafiosi" who subjected him to "a naked, undisguised threat". As Prime Minister, Brown was a disaster whose election campaign slogan should be, according to Alistair Darling, Douglas Alexander and Lord Mandelson: "Futile", "Finished" and "F**ked".

Yet, as he strode into the boardroom, accompanied by two assistants, as well as a pair of armed protection officers, he was greeted by one more low blow. The chairs and tables were bedecked in the black-and-white strips of Dunfermline Athletic, whom the Purvis Group sponsor. Brown, an ardent Raith Rovers fan, frowned then broke into a beaming smile. As Bob Garmory, chairman of the Fife Chamber of Commerce, joked: "We didn't want him to feel too at home."

Over a lunch of tuna, cheese and ham sandwiches, Bob Purvis, who set up the company 30 years ago with his wife, and now employs 400 staff, explained in stark terms the problems facing construction companies in the current economic climate. If anyone in the room pondered Gordon Brown's role in the creation of their woes, it was not expressed, instead they appreciated, as one said, "any hand on the pump or hand taking an oar is welcome".

Yet the stark contrast in Mr Brown's current appointments was apparent, as Mr Garmory said: "We were joking afterwards it can't be easy for a man who has been in meetings with President Obama about releasing 300 billion to save the banking world to come back to Fife and try to figure out if it's the right thing to do to support local companies in getting Fife council contracts. It's a bit of from the sublime to the ridiculous, but, in fairness to the man, he sat and took copious notes and challenged his advisers."

To David Cameron, Brown is the invisible man, skulking up in Scotland and avoiding the Houses of Parliament to which he has been re-elected. When during a recent debate on the Finance Bill, Harriet Harman encouraged him to praise Brown for his handling of the financial crisis the new Prime Minister replied: "I'd be delighted to, if he could be bothered to turn up to this House." The political blogger Guido Fawkes runs a regular "Where's Gordon?" feature and a running meter which records his days in attendance, 2, votes, 0, and days paid, 61. For the perception is that Brown had sunk into a black depression and was nursing an almighty huff, yet the reality is more complicated.

On Monday evening, he and his wife, Sarah, threw a party in the Mappin Pavilion at London Zoo for staff and helpers from the Labour Party headquarters, while the previous day they had hosted a party for the schoolfriends of their eldest son, John, who will leave London behind and begin the new term at the local school in North Queensferry.

On both occasions, Brown appeared jovial and relaxed, but this conceals the bitter disappointment, extreme exhaustion and anxiety about his political history he has experienced during the past two months.

According to several people who know Brown well, he went through a "black period" immediately after leaving office. He had been desperate to secure a coalition agreement with the Liberal Democrats and despite Nick Clegg's statement that he could not remain as leader, felt he could remain in place for up to a year in order to secure the nation's economic recovery. He was "stung" and "very upset" by the Sun's front page headline which branded him: "The Squatter in 10 Downing Street" and which failed to recognise his constitutional duty to remain in place until a coalition deal was complete.

Immediately after leaving Downing Street, Mr Brown remained in London for a few days, while Sarah drove north and broke down shortly after crossing the Border. When Mr Brown returned to the family home in North Queensferry, his natural instinct was to sit and brood. However, Sarah insisted on keeping him perpetually moving, ushering him into the garden to play football with his boys when they returned home at weekends.

At night, though, when the boys were in bed, he probably couldn't help but brood on what might have been. A historian himself, Brown is painfully aware that he will be remembered as a prime minister who was not carried over the doorstep of Number 10 by the electorate but who instead punched through the wall and pushed his neighbour out. The fact that his "neighbour" Blair phoned him twice at home to commiserate and assure him that Clegg had been unfair in his attitude was little comfort. As he told one friend in recent weeks: "I just can't communicate".

When Menzies Campbell failed to challenge Charles Kennedy for the leadership of the Liberal Democrats, he later said that he thought about his error for ten minutes every single day. For Brown a similar length of time is devoted to his failure to called an election in 2007 when the Conservatives were in disarray, the global recession as yet unknown, and a Labour victory all but guaranteed.

"He thinks about the decision a lot," said a friend. "He wouldn't be human if he didn't. In many ways, he is an uncertain person, uncertain about his abilities and that was the problem when he became prime minister. He too often doubted his decisions and also tried to micromanage everthing, which you just can't do."

Another friend of the family said: "It was pretty traumatic. He does analyse things and he does think could I have done things better. The last time I saw him he was exhausted – he was absolutely exhausted and that has got to be recognised."

The first task Brown set himself was to write to friends, family, even those with whom he had fallen out, to thank them for assistance on his long political journey. The card was the celebrated photograph of the family leaving Number 10 with their two sons and, while a personal message was written by hand at the top, the central message was printed and read:

"As we leave Downing Street and return to Fife, where Gordon will continue his work as MP for Kirkcaldy and Cowdenbeath we wanted to thank you for your kind wishes and words of support. We have been incredibly moved to hear from so many people from all parts of the country and all walks of life who appreciate the work that Gordon has done to deliver fairness in Britain and in the wider world. We will, of course, continue to do all that we can to serve our country and those shared values and to cherish the wonderful family life we have built away from the public's spotlight.

"Gordon, Sarah, John and Fraser"

One friend commented that it was apparent that Gordon had later begun to relax when a personal note arrived written, legibly in ink, instead of the frenzied, barely readable notes in heavy black felt tip pen he favoured in the past. The World Cup has also been a welcome distraction, with Brown watching as many games as possible and, at the slightest interruption such as a phone call, immediately recording them to watch later. As he shouted at friends: "Don't tell me the result!"

While Sarah prefers to work on her forthcoming autobiography on her laptop on the kitchen table, her husband retreats to his study on the ground floor which overlooks the front lawn and has a black-and-white photograph of Jim Baxter, who played left-half for Raith Rovers, Rangers and Scotland.

Although he has been snubbed by the electorate, and Madame Tussauds, for that matter, who have declined to make a wax mannequin of him on the grounds that he was not elected, rendering him the only PM in the past 100 years to be so ignored, he is intent on securing his place in history as the man who saved the global economy.

Two days each week he is joined in the study by Kirsty McNeill, his former speechwriter, with whom he is writing a book on the cause and solutions to the economic crisis. Once this has been completed, he is set to embark on his memoirs and has already begun gathering clippings of his long career for research. They are unlikely to be international best-sellers.

Yesterday a publishing insider said: "Gordon Brown has schemed since day one to become prime minister. Then, when he got the job, it was a disaster. He was patently unsuitable for the job, but the one thing he did do was promise no boom or bust and then he presided over the biggest economic catastrophe since the Second World War. Nobody is interested in a piece of self justification from Gordon Brown."

Asked about the value – Tony Blair got 4 million – he said: "Prime ministerial memoirs do sell and if he did a gossipy book that was full of background then he might get a good serialisation deal of 300,000 - 400,000 maybe a 500,000 all-in. I don't think many publishers want to give him the platform."

IF the imagery of Psalm 23, a favourite of his father, the Rev Brown, was a physical place, then for his son, Gordon, it would be North Queensferry in the "kingdom of Fife'. "Fresh and green are the pastures, where he gives me repose/ Near restful waters he leads me, to revive my drooping spirit."

At his home, a cream-coloured former manse, he enjoys sweeping views over the Firth of Forth and a large lawn that backs on to green fields. To say he has returned to his roots is an understatement. The Brown family's history has been traced back 300 years in the area and spreads out to almost every village and town in his constituency. "We can trace back our family working on the land at Inchgall Mill, near Lochore, in the 1700s, and I have a family tree of ancestors who were born, lived and are now buried in Ballingry, Lochgelly and Dysart as well as in Kingskettle where much loved and missed family members are at rest," Mr Brown explains in a "Welcome from Gordon" on his website.

Staff in his constituency office are anxious to explain that Brown is not "re-connecting" with his constituents, as this indicates that there has been a separation, but that he is instead devoting time that he simply never had before to tackling issues. The first goal was to visit as many schools as possible before the summer break and prior to visiting, headteachers were told that he was anxious to meet as many pupils as is possible. As one colleague explained: "We don't want kids going home saying, 'Gordon Brown was at our school today'. We wanted them going home saying, 'I met Gordon Brown.'" (Even when he failed to meet everyone, and a mother complained to his wife, Sarah responded on Twitter: "Sorry GB missed meeting your daughter when he visited her school yesterday, but he enjoyed seeing the school.")

So he's played keepie-up at the sports day for primary schools at Dalgety Bay; congratulated staff at Burntisland Primary School on their first-class report from Her Majesty's Inspectorate of Education; been serenaded with songs by pupils at his old primary school in Kirkcaldy West; been piped by the Lochgelly School Pipe Band; discussed globalisation, the collapse of banks and the fortunes of Cowdenbeath Football Club with 200 pupils at Beath High School; and met with the student presidents at Adam Smith College, one place where he still remains chancellor.

While prime ministerial visits to schools by Brown were restricted to a maximum of 30 minutes, he's now been able to spent three times as long with, at times, the odd uncomfortable moment. One well-read pupil at St Andrew's High school said that he had read Brown's 1989 book, Where There's Greed: Margaret Thatcher and the Betrayal of Britain's Future. "You lambasted Margaret Thatcher," said the pupil, "then, when you had her visit you in Downing Street, you said she was a wonderful woman. How do you square that up?"

For each weekday visit the uniform remains the same, dark blue suit, white shirt and neatly knotted tie. However, weekend events allow him to, if not let his hair down, then at least lose the tie and loosen off the top button as when he visited the fun day held a few Saturdays past in Kirkcaldy's West End, where he wandered among the stalls with Sarah and the boys. Then, there is the scone or two, to which he is partial when visiting the cafe at the Harbourmaster's House in Dysart while celebrating the rise of tourism in Kirkcaldy.

The question of whether Mr Brown has genuinely exchanged the high-profile international summit for the tedium of the constituency surgery is difficult to pin down. The image of him sitting in a draughty church hall listening to constituents' complaints is difficult to imagine, yet, for security reasons, his presence isn't advertised in advance but his office do insist he has been meeting constituents by appointment.

"Gordon," Sarah Brown has recently been telling friends, "still has one big job in him." The question is: What will it be? And the answer is as yet unknown to the couple themselves. While Sarah continues to support and assist the couple's two charities, PiggyBankKids, which helps create opportunities for children and young people in the UK, and the Jennifer Brown Research Fund, named after the daughter they lost, which seeks solutions to difficulties in pregnancy and to save newborns, Gordon remains committed to assisting Africa and the Third World.

He will not, as Tony Blair has, accept one of an ever-increasing list of well-paid directorships designed to funnel six-figure sums into the family bank account. You only have to read Sarah's Twitter feed, in which she extols the benefits of the Oyster Card to nip around London by public transport to appreciate that the Browns do not aspire to the lifestyle of the Blairs, with their vast country estate and ever-expanding London townhouse. In terms of property, the couple own their family home in North Queensferry, while the small London flat Gordon owned is now said to be in Sarah's name.

The couple, according to one friend, may in the future consider a move to Edinburgh, possibly to Merchiston, close to their friend JK Rowling. Special Branch, who provide round-the-clock security to the Browns, are said to be concerned at the difficulty of providing adequate protection to their current home, which backs onto fields and is visible from the main road. While he was prime minister, this was achieved by a heavy police presence and an agreement that Lothian and Borders Police and Fife Constabulary maintain an armed response team in the close vicinity, yet since his resignation this has been scaled down. Another friend insisted this was not correct.

So what job, if any, does he covet? While recent invitations to lecture in America may be welcome as Brown has a deep love of the East Coast and has holidayed regularly on Cape Cod, he would accept the role of head of the International Monetary Fund. Head of the World Bank is a role reserved for an American, the leader of the IMF usually goes to a senior European politician, currently Dominique Strauss-Kahn, who is expected to step down next year in order to challenge Sarkozy for the presidency of France. There is, however, a small snag. It is for the prime minister to nominate Britain's candidate for the IMF and it is hard to imagine David Cameron, who blames Brown entirely for Britain's economic woes, to insist, with a straight face, that he is now the best man for the job. Instead, his Lib Dem foe, Vince Cable, could be foisted on the IMF in a bid to get him out of the coalition's hair.

To friends and family, there is relief that Brown now has the time to devote to the job mentioned on the steps of Number 10 as he bid the public adieu. As dusk fell on Downing Street he said that as he left "the second most important job I could ever hold, I cherish even more the first – as a husband and father". While the health of both their sons is the couple's primary concern, anxiety over Fraser, who has cystic fibrosis, has, in the past, been acute, but friends say Gordon is keen in conversation to emphasis his son's fitness and well-being and they are comforted by how medical advances continue to extend the life expectancy of sufferers. Redundancy, at least from running the country, has its advantages, as Sarah explained to her followers on 30th May: "GB their dad took them to Morrisons in Cowdenbeath to get new World Cup sticker albums/Lots of 'Oooh, I've got two Ferdinands & Gerrards, can I swop for a Beckham?"

Gordon Brown is expected to maintain his low profile until September when the Labour Party elects a new leader, preferably his favoured candidate, Ed Balls, who has remained loyal to his boss, and, when questioned about Mandelson's accusations about Brown's behaviour by Andrew Neil, staunchly defended them by answering: "We". The next step for Gordon Brown will be interesting, for, as his friends and supporters believe, his is too big a brain to restrict its focus to a single Scottish constituency. Or, as six-year-old John announced, as he and his brother stood on the desk inside Downing Street in the last three ticking minutes of his father's Premiership: "Daddy, you know everything."

• Additional reporting by Graeme Breen