Now a new battle is brewing at the 600-year-old castle, on the banks of the River Doon, after a dotcom millionaire bought the property in a multi-million pound deal from the family.
Kate Armstrong, one of the co-founders of the Confused.com website, has launched a court attempt to evict an elderly tenant who has occupied a small cottage and three-acre plot on the land for 13 years.
There are also claims that the wealthy online insurance guru has removed a dairy farmer from a field he had rented since 1953 and evicted a group of fishermen who have rented a beat in the grounds for generations.
One tenant has now accused Armstrong and her husband Malcolm of instigating a "new Highland clearance" on the estate they bought for an estimated 2.5 million.
Retired civil servant William Connell lives in the Kennells Cottage, where he first raised ducks to sell their eggs to Chinese restaurants and now keeps a small flock of sheep. The whitewashed cottage, although in need of repairs on the inside, is in an idyllic setting and Connell hoped he might see out his days there.
However, he was served with an eviction notice in November by the Armstrongs, who are believed to have purchased the castle and grounds earlier last year.
Connell insists, however, that he will not leave and his court battle against the eviction was heard before a sheriff for the first time last week.
Speaking after the first hearing, Connell said: "I have loved living at the Kennells. I am a registered keeper of sheep, and I have a small number of animals grazing the land there.
"There's a small pond on the site where I used to rear ducks for their eggs, which I sold to the Chinese restaurants in the area.
"I really did think this was my home for life. I couldn't see any reason why I would leave. It is a bit of a hovel inside, but it is an absolutely beautiful setting."
However, he said his idyll was shattered when the head of the Clan Kennedy, the Marchioness of Ailsa, died in 2007 at the age of 91 and Cassillis was sold to the Armstrongs.
He said: "There is a local dairy farmer whose family had rented a field at Cassillis for 53 years. They were told to go. A consortium of fishermen had rented a beat on the River Doon from the estate for many years. At first they were told they couldn't use the bothy in the grounds, and then they were ejected.
"Now Mrs Armstrong has come for me, serving me with an eviction notice last November. It is like something from the Highland Clearances, with tenants seen as nothing more than an obstacle to be removed.
"But this is not the 18th century. I intend to fight this all the way and will not be budging from my home."
He added: "My case will be based on the fact that the tenancy I have is actually an agricultural tenancy, due to my ducks and sheep, and is therefore not a normal short assured tenancy that can just be got rid off. It is my hope that the law will come down on my side, rather than the side of this rich person who is trying to create her own little kingdom here."
Cassillis had been the property of the turbulent Kennedy family since the 13th century. Their history, wrote the castle historian Nigel Tranter, "was one long catalogue of violence, savagery and sudden death".
After the death of her husband in 1994, the Marchioness of Ailsa, head of the Kennedy clan, lived on at the castle until her own death.
By this time her siblings were well established in homes nearby and none chose to live in the castle, so it was put on the open market for offers over 2.5 million.
Before buying the castle, Armstrong shared in a 21m windfall handed out to senior employees of the Admiral Insurance Group when they sold part of their stake in the firm.
Armstrong, 47, an Australian mother of three with an IT background, helped found confused.com, which is owned by Admiral, and is thought to have become extremely wealthy on the back of the website's success.
Admiral, in which Armstrong is a shareholder and non-executive director, also owns insurance brands such as Elephant.co.uk, Diamond and Bell Direct.
Despite repeated calls to the Armstrong home and requests made through their lawyers, Kilmarnock-based McIntosh and Wylie, nobody would answer questions on the eviction procedures at Cassillis.
Their lawyer, Campbell Fullarton, said: "I have been instructed that they have no desire to speak to the press."
Piece of history for the price of used car
WHAT must be the cheapest castle in Scotland is set to change hands for the price of a second-hand car.
The Castle of St John, an imposing medieval tower house in Stranraer, once used as the headquarters for government troops during the Killing Times of the 1680s, is on the verge of being sold – for just 6,000.
The castle, which dates back to 1510, is owned by the Earl of Stair, whose family has presided over it for 370 years. The crossbench member of the House of Lords is in talks to sell it to Dumfries and Galloway Council, which plans to turn it into a town museum. A deal is imminent, with an official bid tendered last week.
The building has served various purposes down the years, including being the town gaol between 1815 and 1907. In the 17th century it was a garrison for John Graham of Claverhouse and his government troops during the suppression of local Covenanters.
While John Dalrymple, the 14th Earl of Stair, has allowed public access to the castle since it was reopened as a visitor centre in 1990, the exterior needs considerable re-pointing and interior refurbishments are required.
Historic Scotland approved a grant of more than 54,000 to carry out the repairs in 2008, but the project is in limbo until the council takes over ownership.
The local authority and its predecessors have held the castle on a tack – an historic long lease – for 999 years since 1763, but Dumfries and Galloway Council last week decided to make an official bid for the title deeds.
Lord Stair said it would not reach as big a sum as some might think on the open market because of its cramped location in the town centre. "With a location in the town centre, the price is less, and it is reduced further if you are selling it with a 700-year lease to the local council."