Kinloch Castle on the Isle of Rum lies in a largely derelict state with at least £7m required to bring it back into good use.
The property has been in the care of Scottish Natural Heritage, which owns most of the island, since 1992.
Kinloch Castle Friends Association (KCFA) have worked for almost 20 years to have the castle conserved.
It made a formal bid for an asset transfer in May with hopes to tackle urgent building works and then open the castle as a bed and breakfast, with accommodation for 50 people. A bar and bistro were also planned.
KCFA said it was "extremely disappointed" that is bid to secure the future of the castle, boost jobs and support the local community had failed.
A spokesman said: "If this castle was on the mainland there would no question about its continuing survival.
"We have identified potential alternative sources of funding, and we will be pursuing further discussions with SNH and the Scottish Government.
"Our exploration of other funding mechanisms within the financial community has provided independent corroboration that our business plan is robust.”
KCFA said visitor numbers to the SNH nature reserve had halved because of a lack of accommodation on the island - which had a population of 21 at the latest official count - while neighbouring Eigg had seen an upturn in visitors.
Rum has also recently lost the services of a ranger, the organisation added.
The hostel run at Kinloch Castle was closed in 2013 with the building now suffering from outbreaks of wet and dry rot, and water ingress.
The association said it plans would support 50 jobs, both at the castle and in businesses that benefited from it being re-opened.
The statement added: "While Rum is now experiencing inward investment and regeneration, with the establishment of a fish farm off the island and the construction of new housing stock, both for the largely transient fish farm workers, and also for local residents.
"However additional employment opportunities are required to encourage more people to settle on the island, reversing the decline in the resident population, and ensure the viability of the island’s primary school."
Kinloch Castle, a Category-A listed building, was a former shooting lodge built in 1897 as the private residence of wealthy Lancastrian industrialist Sir George Bullough
It is considered a significant piece of architectural and social history, which reflects the new popularity of Scotland as a destination and second home of the rich during the Victorian era.
The castle has been on the Buildings at Risk Register since 2004.
Nick Halfhide, SNH’s Director of Sustainable Growth, said: "Having carefully considered all the evidence presented, we concluded that the merits of the request fell short of the necessary requirements for a successful asset transfer application.
“KCFA carries out a range of important work and we fully understand that they will be disappointed by this decision. We hope that it does not discourage them from engaging with SNH and partner organisations which are looking to secure the best interests for the castle.”
KCFA is made up of members from the Rum community as well as those from further afield who have become interested in the preservation of the castle.