The oldest mention I see of the White House being at "1600 Pennsylvania" is in The Advance 7 June 1906 which says:
The President is not referred to in the rooms around Congress nowadays as "the President" or as "the man in the White House." Instead they say "the man up at 1600" and the explanation of it is that the White House is 1600 Pennsylvania avenue.
Older references say the White House is at Pennsylvania and 16th, rather than using the number "1600".
For example, the 1888 Washington, what to See, and how to See it
A Sightseer's Guide says:
Pa. Ave & 16th st.
Roose's companion and guide to Washington and vicinity (1877) says:
The President's House is about a mile and a half from the Capitol, in a direction west by north. It stands between 15th and 17th sts. n. w., on a high plateau, where Pennsylvania and New York avenues intersect, and fronting it is Lafayette Square. It is built of a yellowish freestone painted white, and is popularly known as The White House.
An 1884 map considers the White House to be on G street.
The 1863 Diary by Adam Gurowski also says:
White House in G street