Walter Scott Prize for historical fiction: The new time-travellers

• Walter Scott realised the value of historical fiction as a vehicle for critiquing his own society. Seven worthy heirs to that legacy have been shortlisted for the prize that bears his name. Picture: Evening News

JUST before one o'clock this afternoon, in the library at Abbotsford, the Duke of Buccleuch will hand over a 25,000 cheque to one of seven shortlisted authors and the Walter Scott Prize for historical fiction will be born.

The seven authors – Hilary Mantel, Adam Foulds, Robert Harris, Sarah Dunant, Iain Pears, and Adam Thorpe – is an excellent one, proof positive that historical fiction is on the rise. The question is, why should this be so?

Historical fiction works by presenting something familiar but simultaneously distant from our lives. Its world must have heft and authenticity – it must feel right – but at the same time, the reader knows that the novel is a representation of something that is lost, that cannot be reconstructed but only guessed at. This dissonance, it seems to me, lies at the heart of historical fiction and makes it one of the most interesting genres around.

Yet the historical novel can help also us to think about how we live now. Scott recognised this, and wrote about the past in order to comment upon his contemporary society. Yet it is not just a matter of political critique; put simply, the historical novel allows us to comprehend our own position in history, and in doing so, enables both a perspective and an important humility. We understand ourselves as part of historical process. We can laugh at Edward Waverley's imbecility and lack of understanding of his surroundings, but we must too recognise that we are as innocent as him of what is actually happening around us, of how we are being manipulated, and the consequences of our actions.

So why is historical fiction so popular? One reason is simply education, as readers gain pleasure from discovering anew familiar periods or exploring new territory. The historical novel is a protean, evolving form, and caters to readers keen on military matters as well as those who prefer stirring adventure or horror story. Alessandro Manzoni claimed that the historical novelist puts "flesh back on the skeleton that is history", and readers do enjoy the ways in which writers can bring the past to life. There is also an element of escapism, as fictions of the past allow readers to imagine other worlds and other times – and other identities.

Historical novels have often been sidelined or derided for not being serious enough, or taking liberties with facts; "bodice-rippers with a bibliography". History should have gravitas, and novels are seen as a corruption of the past, as something inauthentic or untrue, as a mode that encourages a sense of the past as frippery and merely full of romance and intrigue. Good historical writers, such as Catherine Cookson, Georgette Heyer and Jean Plaidy, became critical shorthand for sensationalism, romance and escapism as contrasted with the gravity of History. Historical fiction became the preserve of the popular novelist and those who were good at it – Bernard Cornwell, Philippa Gregory – were ignored or patronised despite their massive popularity and at times compelling narratives.

This is in many ways a genre-related problem. No-one criticised Salman Rushdie's investigation of history in Midnight's Children, or thought to attack Angela Carter's meditations upon the past in Nights at the Circus, or seriously undermined Toni Morrison's Beloved. Critics rarely disparage the work of Phillip Roth or Don DeLillo, both intently serious authors who engage with the past. Similarly, using fiction to explore history was clearly a serious and important thing to do in the century after Scott wrote, something that is borne out by the works of his many imitators, among whom are some of the finest writers ever to publish: Tolstoy, Flaubert, Woolf, Eliot, Dickens, Pushkin, Dumas.

The past decade has seen historical fiction regain critical traction and credence. The most visible sign of this is the sudden acceptance of the genre by mainstream literary writers. Key novelists – Robert Harris, Sarah Dunant, Maria McGann, Hilary Mantel, Iain Pears, and Sarah Waters – have established the genre as committed, investigative and thoughtful, but they have also emphasised how entertaining it can be. Other writers have contributed to this movement – Kazuo Ishiguro, Rose Tremain, Ian McEwan, for instance, all wrote important novels that brought the genre to critical attention. The works being written now testify to a new appreciation of the possibilities of historical fiction. History itself has fragmented and shifted – there are now multiple histories available to write about: of women, of class, of race, of sexuality.

Given this revival in the historical novel, it is apt that Walter Scott's name should again be associated with the best in fiction about the past. Scott's huge influence, and his popularity, is often forgotten, and one only need to read Waverley or Ivanhoe to recognise his mastery and significance as a writer. Today's shortlisted authors are worthy heirs to his legacy.

Jerome de Groot is the author of The Historical Novel (Routledge, 2009) and Consuming History (Routledge, 2008).