Bayley, 57, doesn't suffer fools gladly. Taste is everything to him. He hates instant coffee, the Turner Prize, and Sweden ("After two days it turns me into a violently anti-social person"). He is a self-described know-all.
This is the man for whom the term "design guru" was invented. Back in the 1980s, Terence Conran appointed him chief executive of the Design Museum. In 1989, he was made Chevalier de l'Ordre des Arts, France's top artistic honour. Later he became the creative director of the Dome but resigned before it opened – accusing the government minister in charge of the project, Peter Mandelson, of being a visionless dictator.
Today he consults for global brands including Absolut Vodka, TBWA and Foster Associates. He loves to "interpret" objects. Sitting in his chic back garden behind the Oval, sporting a goatee and a suntan, pouring me an expert expresso, he resembles a resting Italian director. The striped tailor-made suit is exquisite (he even wears a suit to the beach).
He's obsessed by the correct design of teapots. His last book was on cars, for God's sake. How could the female body – messy, primal, willful – match up to his fastidious aesthetic standards?
But actually Woman As Design: Before, Behind, Between, Above, Below is a bit of a revelation. Instead of the laddish in-jokes you'd expect of a Jeremy Clarkson ("Just look at the chassis on that"), it's actually a great big generous tome.
Billed as a mix of design, cultural history, fashion, erotica and observation (the subhead Before, Behind, Between, Above, Below is taken from a John Donne poem), it's full of sumptuous images of women from Aphrodite to the present day. And not just super-thin ones. Beth Ditto is in there, along with the wonderful, portly Venus of Willendorf (from around 23,000 BC).
In periods when we are impoverished, there is a vogue for voluptuous women, he says. "It wasn't my intention but I think (my book has] vaguely hit the zeitgeist. Everywhere there are articles on how curves are back."
The "conservative feminist" lobby may still lynch him, of course, for objectifying women. There are chapters on breasts and bottoms. He asks quite openly: if woman had been designed, what was the brief? But the language is never silly or squeamish. "I've been married a long time and have two children, so I'm a realist," he tells me sternly.
He looks at how evolutionary biology and male desire have tried to shape woman. "Bras are for men," he says provocatively. "They lift, separate, there's double the pleasure for the male gaze. Put it this way: if you're a woman and you need to contain the movement of your breasts, there are far better ways to do it."
The approach can be scattergun. But actually, I love the way he reminds us that the female body is genius in 3D form: "It's the most potent symbol there is." No wonder art and architecture and even the hourglass Coke bottle have copied it.
At the end of the book he even posits the idea that with new fertility techniques "it is entirely possible that we could do without males". He doesn't sound too worried.
"Despite my affected, fey demeanour, I've always been fascinated by women and enjoy their company," he says, showing me a fabulous 1970s image of women demonstrating against consumer capitalism.
Because Woman As Design is a love letter. Most specifically, it's to his wife Flo, a graphic designer and illustrator. She was actually the inspiration for the whole book.
"We were talking over dinner and she started ranting about the imbecility of most product designers and how incompetent they were – I think we were probably looking at our hand-held Dyson vacuum at the time," he recalls. "And she started talking about the area between a woman's legs, saying that just construed as a piece of design: 'I don't know any designer who could do anything so complicated and subtle and functional and beautiful.' It was such a fabulous moment – I just went from there."
They've been married 28 years, with two children, Bruno, 23, a journalist, and Coco, 22, who is taking an MA in medical anthropology, and he credits Flo with everything he's achieved. Until he met her, he had no taste or insight.
"She's intuitive, I'm acquired. I'm a tart, she's elegantly reticent. So it's a nice balance," he beams.
He married Flo two weeks after they met at Conran's office in 1981, where she worked in the graphic design department; he was 28 and "squatting" in the basement (dreaming up Conran's Boilerhouse project at the V&A, which went on to became the Design Museum).
Born in Cardiff in 1951 (an indulged, "rather fat" only child) and educated at Manchester University and Liverpool School of Architecture, Bayley was a show-off art historian who knew little about real life. Flo, though, was sexy and cool.
Although he always cared passionately about the way things look (his father worked in the aircraft industry; aged 11 he was given an Olivetti typewriter), his lifestyle aesthetic comes from Flo. She taught him about proper coffee and good oil and Poilne bread and growing your own basil.
Bayley is an elitist but he is a democrat for luxury. "My inspiration has always been that of the old Italian Communist Party – the best salami for everyone. Kasmin, the art dealer, once said disparagingly of Terence (Conran]: 'His problem is he wants the whole world to have a better salad bowl.' I always thought that was rather a fine idea. But alas it never happened, because the world didn't share Terence's taste."
Conran, his mentor, comes up a lot. Yes they had spats – and often don't speak for months – but there's huge affection there. At the height of the 1980s they nearly Habitat- ed the world together. "The Design Museum was about brainwashing," he admits rather naughtily.
I ask him about the strong women he admires. Coco Chanel, obviously: "The person who designed modern woman! Remember, that elegant Chanel suit also has a pocket for cigarettes." He loves Catherine Deneuve, and Carla Sarkozy for the sheer cheek of saying that no woman should wear make-up after 28.
And he tells a hilarious anecdote about being invited to lunch – and then being blanked – by Anna Wintour when she was editor of British Vogue.
"Emma Soames was the go-between and I promise you, may the good Lord strike me down if there's a word of exaggeration, we went to Ciconne's and Anna ordered half a tomato. She had her dark glasses on and she did not say a word. In these situations, my hysteria mounts, and I say far, far too much. I still don't know what it was about. Clearly, I failed the test."
Recently he and Germaine Greer joined forces to oppose a notion that Britain no longer cares about beauty. Bayley reminded opponents that Botticelli's Venus was modelled by a "common Florentine hooker, and that at Topshop in Oxford Street today you'll find a huge, inspired and energetic audience in pursuit of beauty". As for Greer, she was "completely whacko but brilliant".
He has great stories about the Dome (someone really should make a British comedy about it). To consult on the contents of Richard Rogers's building, he offered them some of the biggest brains in the world – Susan Sontag, Umberto Eco. "They said: 'Can't have foreigners!'" he groans.
He jokes that if he hadn't resigned, he'd be in the Lords now. In the end he was vindicated by the tat they filled the Dome with. But he still regrets that Mandy – a fellow aesthete – didn't become a soulmate. What does he think of him now he's back in the political frontline?
"I think he's brilliant – wholly unprincipled but very, very smart."
Bayley's next book is on ugliness "which is really another book about beauty". Then, after the female body, he's tackling men. He promises it won't be too willy-orientated. "I want to look at the non-biological aspects of male identity – tribal behaviour, manners, roleplay, clothes. Why do men play golf or love driving fast?" he wonders aloud.
I ask if having a 22-year-old daughter has made his book on women more liberal?
"Coco hasn't looked at a page," he roars. "We get on very well but she has absolutely no interest. It's like that line – no man is a hero to his valet: no man is a hero to his daughter. And I'm really proud of her for that."
Woman As Design is published by Conran Octopus, priced 50.