"I've often wondered why that stayed in my memory because I don't have great recall of my boyhood," he says. "I guess it was because it had this enormous ambivalence, it was clearly wrong, and yet there was a horrible sense of a frolic about it."
So the incident lodged in the mind of the boy who would become a priest and eventually a bishop. It would come back to him vividly more than 60 years later when he found himself writing a book about the human condition, considering our capacity for casual evil.
It's eight years since Holloway, now 74, retired as Primus of the Episcopal Church in Scotland, renouncing his faith in organised religion while his faith in God slipped quietly away through the back door. In 2004, when his book Looking In The Distance was published, he said he expected it to be his last on questions of faith.
Yet here we are, in the tranquil Edinburgh flat he shares with his American wife, Jean, discussing Between The Monster and the Saint: Reflections on the Human Condition, which has just been published by Canongate.
"It's incorrigible," he says thoughtfully. "I think it's chronic, this preoccupation, I don't think there's any way off it for people like me. There are people who just can't accept life and get on with it. They get on with life, but are always also asking the big questions."
It doesn't matter that the questions are probably unanswerable, or that we've been asking them for thousands of years. It's the chance to wrestle with them that engages Holloway, his sharp eyes sparkling at each fresh idea. A philosophical fencing match is somehow good for the soul.
And so we return to the question of evil, one of the great thorns in the flesh of Christianity. If God is creator, and wholly good, where did it come from? If He is all-powerful, why does He allow it to exist? Holloway has long since renounced the conventional idea of a God and a Devil, but he talks a lot about "force", a concept he draws from the writings of French philosopher Simone Weil to describe the relentless onward motion of the universe which drives the baser part of us.
The day before we meet, he was in a "traffic incident" in which he challenged a gang of youths who were vandalising the bollards in one of the city's contraflows. "They were very menacing and abusive, and I was very angry. I was getting revved up, they were getting revved up, it could have become violent. I revisited that bit in me that wishes I had a baseball bat in the back of the car. It touched bits in me that I didn't respond to, but I could have. We carry all this stuff in us."
Yet we have the seeds of hope in us too, he says, in the ability we have to stand back and see "force" at work in us. "In our consciousness, we have the space to challenge it. I think we're capable of as much good as evil. Part of the argument of the book is for us to be so self-aware and self-reflective that we give this reflective pity a chance against the force that also dominates us."
Another memory: it is 1968, the summer of the race riots, the student uprising in Paris, and angry protests against the war in Vietnam. Robert Kennedy has just been shot. Holloway is 34, taking the train across the United States. "The (Vietnam] war was terrible, a monstrous mistake, like the one we're in the middle of at the moment, but even the opposition to it seemed to be drawing out the worst in people. I felt it was us at our worst.
"I remember sitting on the train feeling bemused, puzzled, sorrowing, and I picked up Koestler's Darkness at Noon, and the epigraph was Dostoevsky's great quote: "Man, man, one cannot live quite without pity". It soothed me. I suppose my book is a plea for pity."
The other word he uses almost as much as "pity" is "kindness". It's a mild-mannered manifesto for the human race. But the Holloway of today is a more mellow man than the one of ten years ago, bristling with anger and disillusionment in the wake of the 1998 Lambeth Conference, when Anglican bishops voted to condemn homosexuality. It was the final nail in the coffin of his faith.
"Something did wither in me. I think I probably got beaten down over the years by cruelty, by the side of religion that is cruel. Increasingly I started encountering the cruelty in my own faith neighbourhood.
"I was angry, but it's eased up, and it may be that part of the mellowing is a mellowness towards people who say these horrible cruel things. They still have to be resisted, but we're all victims of forces that we don't understand. If I'm pleading for pity, I somehow need to find pity in my heart for people like that."
He is also less vehement in his unbelief. While he says that he can no longer believe in a "divine clockmaker outside (the universe] who set it going", he describes himself as "agnostic", or in a state of "expectant emptiness". "I'm not going to make any confident predictions. I'm still a religious person. I still get much out of going to church, I just don't do it in a doctrinal way. Hypocrisy used to be going to church and not practising it. But a lot of people think I'm a hypocrite for saying that I don't believe it and still going to church.
"I choose to be identified with generous-hearted, compassionate Christianity because I think that it has beauty, because I love community, because I like to be challenged by the best of it.
"But reality is so mysterious, isn't it? I don't think there's any neatness here. There is a human need to tidy things up, and the neaten-uppers are the ones who tend to turn into monsters, who want to herd us all into great regiments that march in step. No, I'm all for eccentric untidiness. And kindness."
Looking back on his early life, he feels he was more attracted to the ceremony and drama of the church than the theology. A working-class boy from Alexandria, the rector of the Episcopal church was the first interesting, educated man he'd met. He left home at 14 for a church school and preparation for the priesthood.
Holloway's obvious abilities took him up the church hierarchy swiftly, although he says he always felt uneasy about power. Perhaps that's one reason why, by the time he was Primus of the Episcopal Church in Scotland, he was a "loose canon", courting controversy by supporting women priests and gay marriages, publicly voicing his doubts.
His 1999 book Godless Morality, published while he was still in office, argued for the need for a moral framework in society outside of religion. It was widely praised in the secular world and widely criticised by Christians of all denominations. To borrow from a song lyric, he was in the spotlight, losing his religion.
By the time he retired in 2000, the mitre had become something of a burden. Free of it, he became a vigorous writer and broadcaster, and a capable chairman of the Scottish Arts Council. He isn't a church leader any more, yet people follow him, a congregation of doubters, the religiously dispossessed who find themselves on the fringes of the church, but can't stop asking the big questions.
It's not easy to be a doubter these days, walled in by the certainty of fundamentalist believers on one side and of Richard Dawkins-style neo-atheists on the other. Holloway became a rallying figure – articulate, engaging, vastly well-read, who dared to describe religion as "good literature not bad science". It has been said that, if he wanted to, Holloway could start a church of unbelievers. He will not.
"I think probably the biggest, the loveliest thing that's happened to me in the last 10 years is that I no longer feel I have to convince anyone about anything, I'm not proselytising for my version of the truth. That's actually been quite liberating."
But this too is part of the appeal. The more he doesn't try to convince us, the more convinced we are.
• Between the Monster and the Saint: Reflections on the Human Condition is published by Canongate, priced 14.99. Richard Holloway will be at the Book Festival on Sunday 10 August at 8pm.