There was no really such pre 1800. As Samuel Russel points out, the modern concept of nations was born only during and after the French revolution. Hungary and Transylvania from the late medieval times were part of the Habsburg empire, but they were not united until 1867. The official languages of Hungary were the following:
1000-1784: Latin
1784-1836: German
1836-1844: German and Hungarian
1844-1849: Hungarian
1849-1867: German
1867-1918: Hungarian
This shows that there could not be a Magyarization process because even Hungarian was oppressed before 1836.
Hungary had the same sort of autonomy within the empire as the other Habsburg principalities had (Transylvania, Galicia, etc.), but in principle it was an absolute monarchy.
Most of the noble families were Hungarian speaking, even though there were several areas of Hungary were the majority was non-Hungarian (like Upper Hungary which became later Slovakia or Transcarpatia which is now in Ukraine). There were also a few noble families of non-Hungarian origin, but they also learnt Hungarian to some extent. This means in particular that within the nobility the Hungarians were over-represented, but not in other areas of the society.
Moreover, in the 1830-40s one Hungarian course become compulsory in schools, and in the 1848 revolution the Hungarian leaders wanted to introduce Hungarian as the only language of the education (this did not happened in the end since the independence war was lost against the Habsburgs).
Magyarization usually refers to the oppressive politics during the dualist times in 1867-1918, when the Habsburg Empire was reorganized into a union of two member states, Austria and Hungary. This was also when Transylvania was united with Hungary.
While the constitution then gave individual rights to the people of the minorities, it did not give community rights to them, that is, an equal status to their language (except for Croatian in Hungary, and to some extent Czech in Austria). Hence, in the Austrian part there was Germanization while in the Hungarian part there was some Magyarization.
Even during these times the ratio of minorities increased at some places, although there was also some places where it decreased. There were also some notable exceptions, for example bank notes were printed in six languages, several state documents (like birth and marriage certificates) were available in minority languages as well, and church service and religious education were absolutely legal (and equally funded by the state) in minority languages.
Croatia had an extensive autonomy within Hungary, and Croatian was the only official language there. The oppression became stronger in 1907 though, from when the minority and religious schools were forced to teach mostly in Hungarian (by the Lex Apponyi) until the end of the First World War (while there was still a little allowed time for minority classes).