I can' speak to the individual (see update at bottom of answer) you are referring to, but the practice of banks paying interest on deposits predates his birth date of 1869. An example illustrating this from a publication by the state of Indiana in 1836:
Many banking institutions in other states, where such restrictions do
not exist, allow a small interest on deposits on which business
transactions are based. Such a course would call in large sums now
inactive and induce many to save and deposit a portion of their
earnings. For our canal fund deposited in New York a bank of that city
is now paying four percent, but while this unusual prohibition in our
charter exists our own banks cannot do for us that which foreign banks
are doing.
So you can see, though the practice is far from universal, it was available from banks in New York city by 1836.
There is mention of interest on deposits in some entries on Roman banking practices, but I haven't found any sources there:
The Roman empire at some time formalized the administrative aspect of
banking and instituted greater regulation of financial institutions
and financial practices. Charging interest on loans and paying
interest on deposits became more highly developed and competitive.
(emphasis mine)
-----update--------
The individual you were looking for is Simon Swig
Simon Swig (1862 in Pren, Lithuania - July 30, 1939 in Taunton,
Massachusetts) was an American banker, politician and philanthropist.
and he died in Taunton Mass., not New York.