To begin with it was a love affair.
It has taken just five years for the relationship to fall apart. The bitter row between Martin Sixsmith and Jo Moore has dominated the news for the last week. But this esoteric controversy is merely the symbol of a much broader conflict between traditional civil servants and the New Labour machine, with its elite army of spin-doctors and special advisers. The two sides cannot understand each other. What started as a marriage made in heaven has turned into a bitter, acrimonious and destructive divorce. New Labour and the civil service simply no longer speak the same language.
The modern British civil service dates back to the famous Gladstonian reforms of the 1870s. As Prime Minister Gladstone instilled the values which have driven officials ever since: integrity, honesty, duty, loyalty. Above all, Gladstone instilled one great and over-riding principle. He determined that civil servants should give their loyalty to the state - technically to the crown - and not, in the ultimate event, to the government of the day.
Until Gladstone came to power it had been very different. The situation resembled the arrangement that still exists in the United States today. The arrival of a new prime minister meant a change in the whole machinery of government, right down in some cases to the local postmaster. One of the great attractions of being prime minister was the ability to reward one’s followers and make them rich. If you read the correspondence of any prime minister before Gladstone you find that an inordinate amount of space is devoted to finding remunerative employment for the relatives and dependants of his followers.
Certainly offices of state were regarded as especially profitable. Paymaster-General, for example (the job occupied by Geoffrey Robinson until he was forced to resign three years ago) was the foundation of several immense 18th-century fortunes. It was this louche and lax state of affairs that shocked the austere Protestant sensibility of William Ewart Gladstone. He determined to put a stop to it.
And he was very successful. The Gladstonian settlement lasted for more than a century. The principle of an independent civil service was accepted by all the main political parties. In practice civil servants were happy to execute the wishes of governments of whatever colour. The civil service administered the nationalisation, punitive taxation and welfare state of the post-war Attlee government. It also administered Margaret Thatcher’s privatisation and tax cuts.
Occasionally attempts were made to interfere with this neutrality. Lloyd George was one prime minister who deeply resented the restriction placed upon him by independent officials and attempted with some success to set up an alternative government of his own. It was not a coincidence that Lloyd George’s government - for all its other merits, and he did lead Britain to victory in the First World War - was by far and away the most corrupt government of the 20th century. Margaret Thatcher was another offender. A strong-minded premier, she consolidated power in Downing Street and preferred to take advice only from her favourites.
But the real systematic abuse only began with the arrival of New Labour in 1997. The ideologists behind that famous election victory simply did not accept the Gladstonian distinction between party and state. They did not believe in the idea that the civil service was an independent machine. They believe that it should be partial, answerable only to New Labour. There was a clue to this in Labour’s 1997 General Election statements, with Tony Blair’s breathtaking rhetoric: "We are the political arm of the British people."
All the arguments since, the vicious infighting over Jo Moore, the hounding of Martin Sixsmith and many other less well known episodes, come down to just this. Labour does not believe that the civil service should be independent. That is why since 1997 it has set out to create what practically amounts to two parallel governments. There is the traditional civil service. And there are the special advisers, with their allies ready to do their bidding. The special advisers are for the most part youngish men and women who formerly worked for the Millbank party machine. Though paid for by the state, their loyalty is purely to New Labour. According to strict civil service rules, these special advisers are not allowed to tell civil servants what to do. In practice many of them do. Stephen Byers’s special adviser Jo Moore was famous for it. In some departments this army of advisers and spin-doctors is known as "Hitler Youth".
In Downing Street the traditional civil service was shoved out of the way. The office of Principal Private Secretary to the Prime Minister - a crucial constitutional job - was to all intents and purposes abolished. It was replaced by the post of Chief of Staff, which went to Jonathan Powell, a party hack. In order to give Powell extra muscle, Tony Blair quietly changed the law of the land. Special "orders in council" were issued enabling Powell and the Downing Street press secretary Alastair Campbell to have formal executive power over mainstream civil servants.
It should be said at once that New Labour had the best possible motives in making these changes. It simply believed that the civil service was not focused enough to make the radical change it believed Britain desperately needed. Yet one of the curious results of Labour’s casting aside of the Gladstonian settlement was a return to the cronyism and corruption of the pre-Gladstonian era.
The Ecclestone scandal - when Tony Blair was ready to change government policy on tobacco advertising in return for a 1 million gift from party donor Bernie Ecclestone - could never have happened under the traditional regime. Jonathan Powell, as chief of staff, allowed Ecclestone in to see the Prime Minister without civil servants present, a highly irregular procedure.
The recent Mittal Affair is another case in point. The traditional British civil service would surely never have permitted Tony Blair to send his letter to the Romanian prime minister, with its mendacious claim that Mittal’s steel business was a "British" company. It would have been far more scrupulous.
One of the most contemptible and disturbing aspects of the last five years is the way that senior British civil servants have allowed New Labour to have its way. Labour could never have got its way without the endorsement of the men at the very top - first the Cabinet Secretary Robin Butler and now Sir Richard Wilson, who retires in the summer. These men allowed New Labour to trample all over the traditional civil service and to change its very nature. They have opted for a quiet life rather than stand up for the inheritance from their Victorian forebears.
It is the civil servants further down the scale who have suffered from the bullying and favouritism of the New Labour spin doctors. Some were simply driven out of the civil service. Jill Rutter, press secretary at the Treasury in 1997, is one example. None of her seniors lifted a finger to save her. Within 18 months of Labour’s victory in 1997 all but two departmental heads of information were out of a job.
The civil war in Stephen Byers’s transport department is being fought out in public. But there are far more battles going on behind the scenes. If Tony Blair has any sense he will back off. No government can rule without the co-operation of the civil service.
Peter Oborne is the political editor of the Spectator.