I don't believe that these are merely popular stories. If you at least partially trust the content in the Draft History of Qing:
The following translations given are my own; caution: may be heavily paraphrased.
吳兆泰,字星階,籍麻城。與仁守友善,互相厲以道義。光緒二年進士,閱十年,以編修考授御史。時國防廢弛,海軍尤不振,朝廷乃移其費修頤和園。兆泰上疏力爭,略謂:「畿輔奇災,嗷鴻遍野,僵僕載塗,此正朝廷減膳徹樂之時,非土木興作之日。乞罷園工,以慰民望,以光繼列祖列宗儉德。」太后怒,罷其官。 歸里後,歷主龍泉、經心書院講席,充學務公所議長。宣統二年,卒。
Draft History of Qing, Vol. 445
Wú Zhàotài, courtesy name Xīngjiē, born in Máchéng. With benevolence and kindness, Wú Zhàotài maintained an amiable personality, and advised and encouraged moral conduct. Graduated as an imperial scholar on the second reigning year of Guāngxù (approx. 1876 CE), served for 10 years, then rose to the rank of Imperial Censor. At a later time, the national defence forces became lax, and the navy in particular was in stagnation, yet the Imperial Court shifted their expenditure to construct the Summer Palace. Zhàotài submitted a report to the Imperial Court, vigorously contending: "The surroundings of the capital (Beijing) are in disaster, people are without homes, and the roads are filled with the dead. This is the time for the Court to cut its lavish meals and entertainment; these are not the days for construction work [on palaces]! [I] beg [the Court] to cease building the Summer Palace, to restore calm and provide hope to the people, and to continue the honorouable and frugal legacy of our forefathers (dynastic founders)." The Empress Dowager was furious, and stripped him of his imperial titles. After returning to his hometown, he took up the position of a lecturer and chairman of the Lóngquán and Jīngxīn academies. Passed away on the second reigning year of Xuāntǒng (approx. 1910 CE).
王仁堪,字可莊,福建閩縣人,尚書慶雲之孫。光緒三年一甲一名進士,授修撰。督山西學政,歷典貴州、江南、廣東鄉試,入直上書房。時俄羅斯索伊犁,使臣崇厚擅定條約,仁堪與修撰曹鴻勛等合疏劾之。太和門災,復與鴻勛應詔陳言,極論時政。其請罷頤和園工程,謂:「工費指明不動正款,夫出之筦庫,何非小民膏血?計臣可執未動正款之說以告朝廷,朝廷何能執未動正款之說以謝天下?」言尤切直。
Draft History of Qing, Vol. 479
Wáng Rénkān, courtesy name Kězhuāng, hailed from Mǐn County of Fújiàn Province. Grandson of Imperial Secretary Qìngyún. Graduated as the top-scoring imperial scholar on the third reigning year of Guāngxù (Approx 1877 CE), serving as an imperial historian. Governed and directed the Shānxī provincial academy, took the position of Vice Imperial Examiner of the Guìzhōu, Jiāngnán, and Guǎngdōng provinces, and held a position at the Imperial Library. At a later time, the Russian Empire sought (the territory of) Ili, and sent the imperial official Chónghòu to negotiate a treaty. [Due to the unfavourable result of the treaty], Rénkān and other fellow imperial historians including Cáo Hóngxūn impeached him. After disaster struck the Gate of Supreme Harmony, [Rénkān] and Hóngxūn thoroughly considered the contemporary political situation, and advised the Imperial Court to halt construction of the Summer palace, saying: "The funds in the imperial coffers are clearly not to be used for this construction work. How can we wrong the people like that, having the funds leak out of the treasury, and wasting the people's blood and sweat? We (your ministers) can report to the Imperial Court about 'funds not yet used [for the construction of the Summer Palace]', but how will the Court explain themselves with these 'funds not yet used' to all under heaven?" Such frankness are these words.
Even though Draft History of Qing doesn't appear to be incredibly reliable, any attempt to tell an alternative narrative should give an explanation as to why this particular content given in Draft History is problematic first. It is, after all, the continuation of the Orthodox Histories of China.