The Darien scheme: The colonial dream that altered course of Scotland's history

• The fleet sets sail

The Darien chest, used to store money and papers associated with the Company of Scotland, top, and above, the fleet sets sail

In 1695, in a historic decision, the Company of Scotland trading to Africa and the Indies was established by an Act of the nation's parliament.

The new corporation was granted unusually wide powers. For 30 years it was to have the exclusive privilege of trade between Scotland and America and a perpetual monopoly of commerce with Africa and Asia. In America, Africa and Asia the directors were empowered to establish colonies, held of the Crown of Scotland, in uninhabited places or in areas where there was no previous European settlement. The Company's ships were to enjoy complete freedom from all customs duties for a period of 21 years.

In February 1696 the subscription opened with 400,000 sterling of capital on offer. A mere six months later that figure was reached. It was a remarkable achievement, especially when compared to similar ventures of the period. Moreover, it represented an extraordinary investment for a poor country.

The subscribed capital was nearly two and half times the estimated value of Scotland's annual exports — and that in a year of harvest failure when the newly established Bank of Scotland (founded 1695) was also seeking hefty capitalisation.

The Company of Scotland had caught the national mood. It was no longer simply a business speculation but virtually a patriotic crusade.

Nearly 1,500 Scots pledged money. The two major burghs of Edinburgh and Glasgow were pre-eminent, together subscribing roughly a third of the total capital. The great and the good were also well represented - indeed, it was the support of the rich, famous and titled which perhaps above all else helped to give credibility to the enterprise. On the morning the books were opened, a throng of subscribers was headed by members of the Scottish aristocracy.

Of most interest, however, was the geographical spread of subscription and its varied social composition: clergy, tradesmen, farmers, merchants, lawyers, physicians, mariners, soldiers, landowners, professors, as well as labourers, servants and students, all made contributions. In addition, though the suspicion cannot be proved definitively, there was probably also significant 'invisible' subscription from expatriate Scots and other interests both in London and Europe.

In the meantime, plans for the venture had taken a dramatic turn at home. One of the Company's directors, William Paterson, had persuaded his colleagues not to bother with African and Indian trade but instead to concentrate their energies in establishing a Scottish colony and free port at Darien, on the Isthmus of Panama in central America, to attract the commerce of both hemispheres.

Paterson was a figure of apparent substance. Born in Dumfriesshire in 1658, he had travelled widely, especially in the Caribbean, and had good knowledge of the prospects there. His credibility was also enormously enhanced by his key role in the promotion of the Bank of England in 1694.

Paterson had conceived the grandiose plan for Darien much earlier and had apparently discussed his proposals both in Amsterdam and in some of the German principalities in the 1680s.

The foundation of the Company of Scotland now gave him his great opportunity. As he later put it with passionate eloquence: "The time and expense of navigation to China, Japan, the spice Islands, and the far greatest part of the East Indies will be lessened more than half, and the consumption of European commodities will soon be more than doubled. Trade will increase trade, and money will beget money, and the trading world shall need no more to want work for their hands, but rather want hands for their work. Thus, this door of the seas and the key of the universe, with anything of a sort of reasonable management, will of course enable the proprietors to give laws to both oceans and to become arbitrators of the commercial world, without being liable to the fatigues, expenses and dangers or contracting the guilt and blood of Alexander and Caesar."

It was a dazzling vista. Despite the enormous risks and multiple dangers, here indeed was a panacea for all Scotland's economic ills and, at the same time, a means of releasing the country from debilitating dependency on the unsympathetic English.

The warnings of the sober and the cautious went unheeded. They rightly argued that the Spaniards would inevitably retaliate against this threat to the very heart of their American empire at a location which, though unoccupied, still lay between the two major bullion-exporting ports of Carthagena and Portobello. The hostility of England was also certain.

The succession to the throne of Spain hung in the balance. The king, William III, had a crucial interest in maintaining good relations with the Spanish court as his major ally in the war against Louis XIV. Scots interlopers making mischief in strategic territory claimed by Spain was hardly likely to promote good relations with a vital ally. Indeed, when word of the Darien scheme leaked out, William angrily denounced the project's promoters as 'raging madmen'. By his command, all English colonies in the Americas were to deny succour and support to the Scottish expedition. Even the Pope voiced his opposition at the damage to the Catholic faith that might be done in the region by these upstart Calvinist Scots.

Yet, in spite of such hostility, Paterson's bold plan and charismatic personality convinced his fellow directors. A key determining factor was the widespread belief that Spain was a paper tiger whose great days of imperial and military glory were in the past. The Scots, because of their successful venturing to the West Indies, were familiar with some of the recent stories about its failing powers. Henry Morgan, the legendary buccaneer, had marched across the Isthmus with just over 1,000 men and destroyed a much larger Spanish force that attempted to bar his path to Panama. Eight years after the sack of that city, Portobello was taken by a few hundred buccaneers.

Two expeditions set sail from Scotland. The first left in July 1698 with 1,200 settlers and arrived the following October off the mainland of Panama at 'Caledonia Bay'. The settlement was named New Edinburgh, and Fort St Andrew was erected nearby to defend it`. After a mere seven months, however, the site was abandoned, a decision forced by a shortage of provisions and a terrible fever epidemic. Over 300 of the colonists, including Paterson's wife, had perished, and their bodies were buried in large pits within the settlement.

Only one of the four ships made it back to Scotland. All suffered further heavy loss of life. Before news of the disaster reached home, a second expedition had set out in June 1699. It arrived in Caledonia Bay to find a deserted and ruined settlement.

As one commentator put it: 'The site marked out for the proud capital which was to have been the Tyre, the Venice, the Amsterdam of the 18th century was overgrown with jungle and inhabited only by the sloth and the baboon'.

Against all the odds, however, the new settlers decided to stay. Spirits rose when a Scottish force, under Captain Alexander Campbell of Fonab, routed a Spanish detachment at Toubacanti. Victory in this skirmish merely postponed the inevitable and further enraged the enemy. A week later a Spanish flotilla arrived to take revenge. Hopelessly outnumbered and outgunned, the Scots surrendered after a fortnight and left Darien for good in April 1700. The extraordinary attempt to establish an international emporium without the costs of territorial empire had ended in abysmal failure, with catastrophic loss of blood and treasure. Of the 13 Company ships that had ventured across the Atlantic, only three returned to the Clyde.

Bitter recriminations quickly followed. Paterson, the visionary, was attacked as a 'Pedlar, Tub-Preacher, and … Whimsical Projector' who had 'bewich'd' the Scottish nation with his 'Golden Dreams'. But Perfidious Albion became the main target of the devastated and humiliated Scots. A proclamation by William, denying succour from the English colonies to the survivors of the abandoned settlement, which condemned many on the fleeing vessels to death by malnutrition, was long remembered as an act of treachery. In April 1705, five years to the month since the settlers had left Caledonia Bay for the last time, an English ship rumoured to have sunk one of the Darien vessels put into Leith. Its captain, Thomas Green, and his crew were arrested on charges of murder and piracy. The trial and subsequent execution of the unfortunate Green and two of his officers, amid the joyous celebrations of the Edinburgh mob, further poisoned Anglo-Scottish relations. This was a bitter time of national soul-searching. Pious Presbyterians saw the Darien fiasco, followed in February 1700 by a major fire in Edinburgh, as a terrible manifestation of divine displeasure. The idea that Scotland was a chosen nation, bound by Covenant to the deity, remained a widespread belief in the country and God's wrath had to be assuaged by repentance. The General Assembly of the Church of Scotland called for national days of prayer and fasting.

The significance of Darien has long been debated by historians. Traditionally the project has usually been dismissed as a fantasy or a 'tragic farce'. More recent opinion has been kinder. The Company of Scotland symbolised, as could no other initiative, the determination of the Scots to seek their economic future in the New World. This ambition was not realised successfully at the time, but was certainly achieved beyond the nation's wildest dreams across the Atlantic in the new political context created after 1707.

Moreover, if the idea was so crazy, it becomes difficult to explain the vigorous opposition to it of several European nations. The English Board of Trade in 1697, for instance, concluded in a report that it would be 'no very difficult matter for any European Prince or State to make some secure settlement on the Isthmus of Panama'. Action therefore had to be taken urgently to pre-empt the Scots. If they succeeded, it was feared the population of the English colonies in the Caribbean would drain away to Darien.

The whole issue was deemed a matter 'of the utmost importance to the trade of England'. Moreover, not all was lost. The Darien investors were eventually handsomely compensated under the terms of the 1707 Act of Union by the so-called 'Equivalent', valued at nearly 400,000 sterling, together with the additional sweetener of 5 per cent interest a year over a period of nine years.

The mass subscription to the Company of Scotland also helped to spread experience of debt and credit among a much broader section of Scottish society. It has recently been argued that this helped to initiate 'a financial revolution', which helped the development of Scottish banking and Edinburgh's pole position in the nation's management of monetary affairs in the 18th century.

But Darien's pivotal significance, in the final analysis, was political rather than economic. The disaster effectively served notice on the Union of the Crowns. It proved conclusively that when the vital interests of Scotland and England were in conflict, the monarch would always opt to support the position of the more powerful kingdom. That was a recipe for continuing tension and crisis, and the governance of the United Kingdom clearly needed radical revision if this outcome was to be avoided in future. At the same time, the Darien fiasco brought home brutally to thinking Scots that their nation was not only in crisis but was rapidly running out of credible alternatives. It simply could not go it alone in the colonial sphere, where massive military and naval resources were now vital for achieving success. All this did not lead inevitably to the Union of the Parliaments.

There were still passionate debates to take place on the varied options of federalism, independence (even links with other nations like the Dutch) and full incorporating union. What Darien did, however, was to place the future of the Anglo-Scottish constitutional relationship on the political agenda even more emphatically than ever before. That was its most significant historical legacy.

• Tom Devine is the Sir William Fraser Professor of Scottish History and Palaeography and Director of the Scottish Centre for Diaspora Studies in the University of Edinburgh. This article appears in the programme for the National Theatre of Scotland's production of Caledonia.