I think the other answer misses some crucial distinctions.
Culture
The culture of the Seljuks was non-Greco-Roman; the same applies to their language. A good summary of who they were comes from Gibbon (Chapter 57):
Since the first conquests of the caliphs, the establishment of the Turks in Anatolia or Asia Minor was the most deplorable loss which the church and empire had sustained. By the propagation of the Moslem faith, Soliman deserved the name of Gazi, a holy champion; and his new kingdoms, of the Romans, or of Roum, was added to the tables of Oriental geography. It is described as extending from the Euphrates to Constantinople, from the Black Sea to the confines of Syria; pregnant with mines of silver and iron, of alum and copper, fruitful in corn and wine, and productive of cattle and excellent horses. (52) The wealth of Lydia, the arts of the Greeks, the splendour of the Augustan age, existed only in books and ruins, which were equally obscure in the eyes of the Scythian conquerors. Yet, in the present decay, Anatolia still contains some wealthy and populous cities; and, under the Byzantine empire, they were far more flourishing in numbers, size, and opulence. By the choice of the sultan, Nice, the metropolis of Bithynia, was preferred for his palace and fortress: the seat of the Seljukian dynasty of Roum was planted one hundred miles from Constantinople; and the divinity of Christ was denied and derided in the same temple in which it had been pronounced by the first general synod of the Catholics. The unity of God, and the mission of Mahomet, were preached in the moschs; the Arabian learning was taught in the schools; the Cadhis judged according to the law of the Koran; the Turkish manners and language prevailed in the cities; and Turkman camps were scattered over the plains and mountains of Anatolia. On the hard conditions of tribute and servitude, the Greek Christians might enjoy the exercise of their religion; but their most holy churches were profaned; their priests and bishops were insulted; (53) they were compelled to suffer the triumph of the Pagans, and the apostasy of their brethren; many thousand children were marked by the knife of circumcision; and many thousand captives were devoted to the service or the pleasures of their masters. (54)
Main Answer
But, in answer to the question:
Was it simply because geographically the area was covered by Eastern Roman Empire? Was that because only ruling class was Seljuk and later Turkish and populace was still "Roman"? Or did the Seljuks and Turks consider themselves to be cultural heirs to Rome?
Answer: The Seljuks claimed what Rome (or Constantinople) had ruled with that title. Their leader was the 'Sultan of Rum' and thereby eligible to rule over everything that the (Eastern) Roman Emperor had ruled. This means that we are not looking at a cultural link, but a political one (which became more important when the Ottomans resurrected it later on).
Muscovite Connection
This is similar to how Muscovy claimed the mantle of highest Orthodox rulership after 1453 by appropriating the concept of the 'Third Rome' along with the localized version of titles of the (Eastern) Roman Empire (specifically, 'caesar' for 'czar'):
At the end of the 15th century, the emergence of the idea that Moscow is a truly a new Rome can be found; the whole idea of Moscow as third Rome could be traced as early as 1492, when Metropolitan of Moscow Zosimus expressed it. Metropolitan Zosima, in a foreword to his work of 1492 Presentation of the Paschalion, quite clearly expressed it, calling Ivan III "the new Tsar Constantine of the new city of Constantine — Moscow." This idea is best known in the presentation of the monk Philotheus of the early 16th century:
So know, pious king, that all the Christian kingdoms came to an end and came together in a single kingdom of yours, two Romes have fallen, the third stands, and there will be no fourth. No one shall replace your Christian Tsardom according to the great Theologian [cf. Revelation 17:10].
Seljuk & Ottoman Claims on Rum
This is also well documented for the 14th century resurrection of the Ottoman claim, e.g., in the case of Bayezid I:
Despite this, the first Ottoman sovereign considered unanimously by historians as a Sultan is Bayezid I, since he was the first to seek and obtain approval to use it from a Caliph (the Abbasid Caliph in Cairo). This happened in 1394-1395. His title Sultan al-Rum ('Sultan of Rum', i.e. the Byzantine lands) reflects the widespread perception that Bayezid was going to conquer Constantinople (an expectation only broken by the sudden arrival of Timur on to the stage). It is also noteworthy that contemporary Western chronicles refer to Bayezid as imperator, i.e. 'emperor'.
From 'Istanbul and the Civilization of the Ottoman Empire' with a bit more historical elaboration:
The most significant feature of the title Sultan of Rum is its connotation of a defined territorial sovereignty. The Great Seljuks had been Sultans of Islam--excercising the one and indivisible worldly power in the universal Muslim state. The Seljuks of Rum--and the Ottomans who revived their claims and titles--were Sultans of Rum, that is, of a definite country and people. The land of Rum was Anatolia, and for a time the Turks even called themselves Rum, after the country they inhabited. The extension of the Ottoman state into Europe reinforced this claim; the lands of Rum--of the Byzantine Empire, or rather, of Greek Orthodox Christendom--embraced territories in Europe as well as in Asia, and it was natural for the new masters of an important part of the estate to seek to acquire the whole of it.
Further, by Moffat (although the link is suspicious) but this is how Google Books showed the quotation to me:
When the Seljuk Turks overran central Turkey in the 11th century, they called themselves the 'Seljuks of Rum' because they had conquered Roman territory.
This link, however, speaks similarly:
One group was the Seljuks who controlled Syria and Iran, and another were those in the eastern lands of Iran and Central Asia. This latter group became known as the "Great Seljuks" and their capital was located at Isfahan. Yet another subdivision came to be known as the "Seljuks of Rum". The term “Rum” comes from the Arabic word for the Roman Empire. The Seljuks called the lands of their sultanate “Rum” because it was established on territory traditionally known as Roman, meaning Byzantine, by Muslim armies.
I think this article, 'Tekfur, fasiliyus and kayser: Disdain, Negligence and Appropriation of Byzantine Imperial Titulature in the Ottoman World', could be quite good as well but it is paywalled for me.
Final note: It should also be noted that the titles of the rulers were a more important distinction in the past than the 'name' of the country the ruled. There was (typically) no general 'constitution' or 'contract' to set out the name of the land; there would have been the ruler with a title, and then that ruler's people would style themselves after him (along with whatever ethnic and cultural ties they had).
