This story almost certainly relates to the invasion of Poland (1655-60) by the Swedes and their allies during the reign of the Swedish king Charles X, but it has been considerably embellished in that the Swedes took artifacts and decorative parts of castles rather than a whole castle.
The known sinkings from which artefacts have been recovered occurred in the Vistula River, Warsaw. It is believed that much of what was taken came from the Royal Castle and the Kazimierz Palace, both in Warsaw. Some of the items which did make it back to Sweden can now be found decorating Swedish historical sites.
In addition to sculptures and collections of paintings and books, the Swedes took,
architectural details, such as ornamental loggias, splendid
staircases, window framings and paved floors...
Source: Hubert Kowalski, 'Archaeological Examinations of the Bottom of the Vistula River Near the Citadel in Warsaw in 2009-2012' (2012)
Most goods were
loaded on boats and transported along the Vistula to the Baltic Sea
and then to Sweden.
Sinkings were not uncommon on the Vistula with adverse weather and changes in the water level sometimes causing boats to run aground or hit stone obstacles. Of particular interest to the archaeological project started in 2009 was
One case of a vessel going aground was recorded by several documents
from this period. In a letter to King Jan Kazimierz from 1 September
1656 the city’s Burgomaster Juliusz Ginther mentioned not only the
reason why the vessel was stopped, but also provided information on
the approximate location of the lost and abandoned transport. It
encompassed pieces of architecture, columns, marble ornaments and
cannons robbed from residences in Warsaw.
Note that it wasn't just the Swedes who did the plundering. Concerning Kazimierz Palace,
...a large collection of ancient sculptures which...[were]...plundered
during the Deluge by Frederick William, Elector of Brandenburg, and
taken to Berlin;2 while the garden sculptures were taken to
So, a lot of loot but not a whole castle or palace to be rebuilt in Sweden (or anywhere else). Based on the documented sinking, past rumours and an apparent 1906 find, a Polish academic started looking for these sunken ships in 2009. Then,
In late summer 2011, when the water level was at waist level, Dr
Hubert Kowalski tripped in the water and pulled out a piece of nicely
worked marble with a beautifully engraved symbol.
... That same year they were able to pull out some 98 pieces of
architecture: among them huge 700 kilogram lintels, arcs, and
parapets. In 2012 the particularly heavy elements were raised by
helicopters. That year also brought a beautiful fountain with
mannerist masks of savage men. In 2015, when the river hit a record
low, there were more finds – the most striking was beautifully
sculpted cannons and a lot of cannonballs.
All in all, over the last five years the team was able to pull some 20
tons of Baroque architecture out of the water, most of it, as Dr
Kowalski suggests, from the Villa Regia (Pałac Kazimierzowski), one of
the many luxurious palaces of Warsaw plundered and looted by the
Swedes in 1656.
Drought conditions in 2012 brought more artefacts to light, as seen in the image below:
"The artefacts in the Vistula riverbed revealed by low water levels are pictured with a power plant in the background in Warsaw September 18, 2012. A huge cargo of elaborate marble stonework that sank to the bottom of Poland's Vistula river four centuries ago has re-appeared after a drought and record-low water levels revealed the masonry lying in the mud on the river bed." Image and text source: Reuters, 'Drought in Poland reveals 400-year-old sunken treasures'
The recovered items have provided much useful details for a part of the Kazimierz Palace previously known only from a short description. The recovered items will also be used in an attempt at reconstructing the decorations in other parts of the palace.
Much of what did survive the journey to Sweden is still there:
The great majority of the looted works of art, elements of
architecture, and other riches made it safely to Sweden and have been
there ever since. These works of art have entered Swedish royal
collections and state museums. In the meantime, elements of
architecture from Poland have become part of the Swedish cultural
landscape, like the two bronze lions from Warsaw's Royal Castle which
today stand near the Royal Residence in Stockholm.