Sure he could be removed legally, as the example of Khrushchev shows. Officially the only source of power in USSR was the Communist Party as defined in article 6 of the constitution from 1977 which was able with a simple majority to remove everybody it wanted. Theoretically even Stalin could be removed, there is even an article about such hypothetical case, but theory is not always the same as practise.
The translation of article 6:
The leading and guiding force of Soviet society, the core of its
political system, state and public organizations is the Communist
Party of the Soviet Union. The CPSU exists for the people and serves
Armed with Marxist-Leninist teachings, the Communist Party determines
the general perspective of the development of society, the line of the
internal and foreign policy of the USSR, directs the great creative
activity of the Soviet people, gives a systematic scientifically
grounded character to its struggle for the victory of communism.
All party organizations operate within the framework of the Constitution of the USSR.
Also in constitutions befor 1977 there were similar articles, like article 8 from constitution 1924.
The supreme authority of the Union of Soviet Socialist Republics is
the Council of Soviets, and in the period between the Council meetings
— the Central Executive Committee of the USSR, consisting of the Union
Council and the Council of Nationalities.
And in constitution 1936 article 30.
The supreme body of state power of the USSR is the Supreme Soviet of