The Shake It Off Singer - who is dating Dumfries DJ Calvin Harris - was said by sources in the US to be in talks to buy the Tower of Lethendy property, around 15 miles north of Perth.
The 16th century estate - which is on the market for around £4.6 million and is described by agents as “the ultimate Scottish residence” - boasts its own 18-hole golf course, a heated swimming pool and tennis courts.
But, writing on her official Twitter page, the star said: “’Cause baby I could build a castle out of all the bricks they threw at me’ -a line from New Romantics.
“But I’m not actually buying a castle.”
The B-listed Scottish baronial manor house dates back to 1570 and is at the heart of a 40 acre estate.
According to US gossip sites, Swift, 25, wants to own a property in Edinburgh to be nearer Harris’s family, who are based in Dumfries.
The £4.6 million price tag may be far beyond the reach of the average house buyer - but just last year Swift spent almost $20 million on a 10,000 square foot penthouse apartment in the Tribeca area of New York.
Swift and Harris got together in February and went public in April.
The couple are reported to have earned £93 million between them in the past year thanks to their hugely successful music careers and massive advertising deals.
A source told US OK! Magazine: “It’s like a fairytale castle - she fell in love with it the minute she saw it.
“Taylor is all about romance. She’s looking forward to snuggling up by a roaring fire with Calvin and embracing Scottish traditions.”
Tower of Lethendy was built using local red sandstone in the Scottish baronial style and features eight bedrooms as well as a variety of turrets, towers, oriel windows and crow-stepped gables.
The estate is presently owned by a Canadian family and has been used as an upmarket wedding venue and a venue for group holidays.
Twenty years ago the interior of the property was given a major - and luxurious - overhaul.
Also included in the sale are two gate cottages, two staff cottages, an estate workshop, a greenhouse and a swimming pool cabana.
The 39-acre grounds are home to a croquet lawn and woodland garden, through which a small burn flows to a sunken pond.
A walled garden is home to apple, pear and plum trees as well as redcurrant and blackcurrant bushes.
Selling agents Savills describe the property as “one of the premier residential estates in Scotland”.
In their prospectus, they add: “The reception rooms and bedroom suites are eminently suitable for hosting lavish house parties yet the house is equally suited to being a comfortable and homely family abode.
“The grounds and gardens surrounding the castle are exceptional.”
A spokeswoman for Savills declined to comment on the speculation.