Monty Python's Eric Idle on his new musical 'Spamalot'

• Eric Idle

He is in Edinburgh to promote the musical Spamalot and has with him several coconuts and tins of Spam. Our photographer sees his chance and asks the comedian to pose with the processed meat. Idle juggles it, pretends it is a camera and, finally, balances it on his head. "I went to university, you know," he says in mock humiliation as the tin tumbles to the floor.

Idle, of course, has suffered greater indignities in the name of his art. In Monty Python, he was the one who would ask a complete stranger, "Is your wife a bit of a goer?" In Rutland Weekend Television, he sat naked to sing the closedown theme. And in Nuns On The Run with Robbie Coltrane, he spent most of the movie dressed in a habit. He has even endured that most gruelling of experiences, being directed by fellow Python Terry Gilliam in The Adventures Of Baron Munchausen.

"He can only flourish in chaos," he says with awed affection for the maker of Brazil and Twelve Monkeys. "I'm exactly the opposite. The more that can be controlled and thought out, the better chance I think you have to pull it off. But he likes to improvise – and improvising on a film set is very expensive – but that's his way. He's a chaotic, mad, creative genius."

Of all the old team – Gilliam, John Cleese, Michael Palin, Terry Jones and the late Graham Chapman – it is Idle who has done most to keep the Python name alive. But although it is tempting to see him as a man trading on past glories, with Idle that theory doesn't stack up.

For a start, he has sustained a post-Python career that is enviable in its own right. In the mid-70s, there was Rutland Weekend Television, a spoof station supposedly broadcasting from the country's smallest county ("It had no studio audience so I was never sure what was funny and when people say I should do the best of, I don't know what the best of is"). Neil Innes was a regular and it was with him that Idle created The Rutles: All You Need is Cash, a brilliant pastiche of the Beatles, presented documentary-style with contributions from Mick Jagger, Ron Wood and George Harrison himself.

"It was one of the most fun shows I ever did," says Idle, who staged a Rutles concert in LA in 2008 during which the spoof Get Up And Go morphed into the genuine Get Back. "The difference between parody and the real thing is that the audience go crazy when the real thing happens."

Later work has included appearances in The Simpsons, and it is Idle's everyman tones you hear on the theme tune of One Foot In The Grave. "I like the new," he says. "I like, 'What are we thinking of this week?' 'What's the wrong direction to go?' The art is to go over there, to look for things that are dangerous or silly or that make everybody say, 'You're out of your mind.'"

Despite this attitude, Idle came to recognise that whatever work he did, people would always frame it in terms of Monty Python. "You can't escape," says the man who presented The Life Of Brian as an oratorio in the Royal Albert Hall last year. "Whenever you try to branch off into some area new, they say, 'Monty Python'. So I just found a clever way of using it in a positive way, taking some of the ideas, which are so fabulous, and having, for example, dancing Knights of Ni. It pleases me. It doesn't do the original any harm and it becomes its own thing."

The tricky thing for Idle was maintaining not just the celebrated Python silliness, but also its subversiveness. It is easy to forget that when Monty Python first aired in 1969, it met with as much bewilderment as laughter. It was a minority programme on BBC2, broadcast at a time when many people didn't even have BBC2, and it took years before it was welcomed into the mainstream. The storm of religious protest kicked off by Life Of Brian in 1979 was notable for its scale and intensity, but it was not the first time the Pythons had been out of favour.

"The series started in 1969 and it was very much, 'What the heck are you doing?' – they hated it," he says. "In fact, it was awful when we suddenly became cuddly old comedians. What happened to the offence? The Establishment opens up to admit you. But I think The Meaning Of Life, since it was made in 1983, still manages to look pretty offensive – all the vomit."

This legacy meant that when Idle had the idea of turning Monty Python And The Holy Grail into a musical – Spamalot – it was with the intention of creating something new. What persuaded the other Pythons to give him the go-ahead was The Song That Goes Like This, a number written with John Du Prez that, in time-honoured Python fashion, sets out to puncture pomposity.

"That song was one of the earliest things we wrote and it was completely improvised," he says. "Du Prez was playing the piano and I said, 'What's this?' He said, 'I dunno,' so I started, 'Once in every show, there comes a song like this, where is the song that goes like this…' We hit the tape recorder and improvised the whole song. It had never happened before and it was just great. The Pythons just loved that song: that's why they said yes."

There were also practical considerations that meant, even if he had wanted to, he could never have transferred the film to the stage without considerable effort. "It is not easy to do: 17 drafts, five years' work… it's not evidently a musical," he says. "Just because it's a comedy doesn't mean the serious rules of drama don't apply. If you're watching a play, you're not going to sit there for 90 minutes if you're not invested in the story."

Contrary to what you may think, such hard work is typical of the Pythons themselves. Celebrating chaos, absurdity and silliness takes discipline. "You can't write comedy if you're smoking dope," says Idle. "You can't find the typewriter. Python would work ten till five, business hours. We were very serious because we were writers. That's what we did first of all. We wrote for Frost, we were BBC writers, so we got good at editing our own material, putting it into different shapes and that's when it started being funny." And, for Idle, it's been funny ever since.

Spamalot is at the Playhouse, Edinburgh, 18-23 October

Messiah

• This article was first published in the Scotland on Sunday on October 10, 2010