Lesley Riddoch: Savour the moment when usual suspects don't come out on top

The election result drew such a mixed reaction in Scotland, that we all – with apologies to Dingwall – needed Dundee United's exuberant Scottish Cup victory celebration to provide some sort of collective, cathartic release. With supporters clad in the brightest, most attention-grabbing colour in the spectrum, it was maybe no surprise the Tangerines played consciously to the crowd. Small club, small town – no wonder players and spectators seemed almost umbilically linked.

But there was one important thing that made those wild moments of post-match frenzy special for supporters of other teams and of none. Missing from the faces of those jubilant players was the look that normally dominates the joyless, predictable world of pre-determined victory: the smug look of entitlement.

Clearly, none of those radiant young men believed success would come their way by right. Virtual newcomers to Hampden success, they had to battle doubt every inch of the way. Jumping like crazy children, mimicking the dance moves of their orange-emblazoned fans, the air-punching players looked as they felt. Deliriously happy. And that's a different face of success from the usual, cold, hard, controlled demeanour.

This is what happens when the usual suspects – for once – do not prevail. Of course, pessimists are always present, patrolling the perimeter of public debate to ensure no mass breakout from the prison of conventional thinking can occur. Goal scorer Goodwillie will obviously be headhunted, captain Andy Webster is already on his way to Ibrox, manager Peter Houston will be poached.

Lessons are being beaten into us with every negative, carping comment. Don't overreach yourself. Don't think you can change the "natural order". Don't presume. Don't challenge the big boys. Normal service will resume as soon as possible. More's the pity – because sometimes a radical change is better than a rest.

The same is true in politics. Former Labour energy minister Brian Wilson wrote in The Scotsman last week about the mess he inherited from the Tories in 1998 as Scottish education minister. Within weeks, Labour had swept away the whole straining apparatus of vouchers, assisted places and opting out – a departure from state tradition that only two Scottish schools had chosen to try. The Tories were completely and contemptuously out of step with Scotland by the time they left office. But their stubborn attachment to unpopular policies also characterised Labour in its dog days. Already, the new coalition has scrapped the shameful practice of holding children at Dungavel and the pointless ID card scheme. I'll bet Labour voters will be cheering like everyone else.

I'll grant you these crimes against common sense and human dignity are minor compared to the wholesale destruction of Scotland's manufacturing base. But the underlying problem is similar. Like the Tories in 1997, Labour in 2010 radiated a complacent sense of entitlement and had developed an unreflective attachment to all their old policies, whether successful, stalled or failed.

Gordon Brown had a profound belief in his right to govern, an inability to admit mistakes and a visceral mistrust of outsiders – voters included.

All these are very human characteristics, just like putting on weight as you get older and nagging errant offspring for doing what you did as a child.

Long years at the top numb sensitivity. Inspiration becomes harder to find. Fear of losing control guarantees loss of control. All these patterns affect ageing governments as surely as they affect ageing individuals.

The answer isn't an age-related cull – it's recognition that self-destruction is inevitable when new ideas, new faces and different perspectives can only thrive outside a political party, family or marriage. The answer is to build in structural diversity as the only protection against institutional conformity. But Britain won't.

The tendency of all institutions in our ageing, anxious society is to atrophy into joyless conservatism and knee-jerk hostility towards diversity because that makes control impossible.

Witness the reaction to calls for quotas to deliver more women MPs. Even the normally well-adjusted Channel 4 broadcaster Jon Snow answered a challenge about all-male TV leaders' debates with the patronising remark: "Well, I suppose we could all have cross-dressed." Titter ye not.

Conservatives accept that the unregulated market has failed to deliver genuine growth or economic stability. And yet an unregulated market for people is allowed to decide our candidates for parliament and local government. So strong is the subconscious fear of tackling our society's class, gender and age divides, Britain prefers to leave large sections of the population outside the tent. The women who battle through are visually diverse, but often politically conformist.

Quotas work. But there's no point searching British history for evidence because we've rarely used mechanisms with any great enthusiasm. There's also no point listing the personal demerits of female MPs generated by Labour's women-only shortlists until someone starts judging the vast herd of retiring, lacklustre men.

A society that works gives everyone a reason to get up in the morning. It encourages trust and provides basic optimism that government will apply reason and energy to problems rather than dogma. Such societies embed diversity not entitlement into every level of governance. The Scandinavians use quotas to encourage women into public life, but Britain fears diversity too much to seriously consider change.

So does Labour. The leadership race is already looking like another wasted opportunity for it to renew and challenge itself. What's needed are candidates with vision but without a belief in Labour's divine right to govern. Andy Burnham hasn't even joined the race, but my money's on him.