I don't believe that these are merely popular stories. If you at least partially trust the content in the Draft History of Qing:
The following translations given are my own; caution: may be heavily paraphrased.
Draft History of Qing, Vol. 445
Wú Zhàotài, courtesy name Xīngjiē, born in Máchéng. With benevolence and kindness, Wú Zhàotài maintained an amiable personality, and advised and encouraged moral conduct. Graduated as an imperial scholar on the second reigning year of Guāngxù (approx. 1876 CE), served for 10 years, then rose to the rank of Imperial Censor. At a later time, the national defence forces became lax, and the navy in particular was in stagnation, yet the Imperial Court shifted their expenditure to construct the Summer Palace. Zhàotài submitted a report to the Imperial Court, vigorously contending: "The surroundings of the capital (Beijing) are in disaster, people are without homes, and the roads are filled with the dead. This is the time for the Court to cut its lavish meals and entertainment; these are not the days for construction work [on palaces]! [I] beg [the Court] to cease building the Summer Palace, to restore calm and provide hope to the people, and to continue the honorouable and frugal legacy of our forefathers (dynastic founders)." The Empress Dowager was furious, and stripped him of his imperial titles. After returning to his hometown, he took up the position of a lecturer and chairman of the Lóngquán and Jīngxīn academies. Passed away on the second reigning year of Xuāntǒng (approx. 1910 CE).
Draft History of Qing, Vol. 479
Wáng Rénkān, courtesy name Kězhuāng, hailed from Mǐn County of Fújiàn Province. Grandson of Imperial Secretary Qìngyún. Graduated as the top-scoring imperial scholar on the third reigning year of Guāngxù (Approx 1877 CE), serving as an imperial historian. Governed and directed the Shānxī provincial academy, took the position of Vice Imperial Examiner of the Guìzhōu, Jiāngnán, and Guǎngdōng provinces, and held a position at the Imperial Library. At a later time, the Russian Empire sought (the territory of) Ili, and sent the imperial official Chónghòu to negotiate a treaty. [Due to the unfavourable result of the treaty], Rénkān and other fellow imperial historians including Cáo Hóngxūn impeached him. After disaster struck the Gate of Supreme Harmony, [Rénkān] and Hóngxūn thoroughly considered the contemporary political situation, and advised the Imperial Court to halt construction of the Summer palace, saying: "The funds in the imperial coffers are clearly not to be used for this construction work. How can we wrong the people like that, having the funds leak out of the treasury, and wasting the people's blood and sweat? We (your ministers) can report to the Imperial Court about 'funds not yet used [for the construction of the Summer Palace]', but how will the Court explain themselves with these 'funds not yet used' to all under heaven?" Such frankness are these words.
Even though Draft History of Qing doesn't appear to be incredibly reliable, any attempt to tell an alternative narrative should give an explanation as to why this particular content given in Draft History is problematic first. It is, after all, the continuation of the Orthodox Histories of China.