The official religion in the Balkans was Calcedonian Christianity during the later Roman Empire. Then, about AD 600, the region was invaded and settled by pagan Slavic tribes.
Naturally both the eastern church headquartered at Constantinople and the western church headquartered at Rome occasionally sent out various missionaries to try to convert the Slavic tribes in the Balkans. The eastern Roman of "Byzantine" Empire also sought to reconquer the Balkans.
Nomadic Bulgars settled in the Balkans and mingled with the Slavic tribes. The Bulgars converted to eastern and Orthodox Christianity and conquered most of the Balkans and converted most of the Slavs to the eastern type of Christianity. The First Bulgarian Empire became a mighty rival of the eastern Roman or "Byzantine" Empire.
The tribes who lived in the most northwestern part of the Balkans adopted western, Roman Catholic, Christianity and the Latin Alphabet and became known as Croats. The similar tribes who lived in the south central parts of the Balkans adopted eastern, Orthodox, Christianity and the Cyrillic alphabet based on the Greek alphabet, and became known as Serbs. The spoken language of Serbs and Croats is called Serbo-Croatian. Note that the Slavs living in Bosnia, between Croatia and Serbia, would probably also speak Serbo-Croatian, and modern Bosnian is a form of Serbo-Croatian.
[04-20-2019. Because closely related tribes came from the north and settled the neighboring regions that became Croatia, Bosnia, and Serbia, Croats, Bosnians, and Serbians have always been culturally similar, and it was easy for members of one of those groups who settled in a region occupied by another similar group to become culturally assimilated over generations and centuries]
Eventually the eastern Roman or "Byzantine" emperor Basil II "The Bulgarslayer" completed the reconquest of the Bulgarian empire and all of the Balkans in 1018. The "Byzantine" empire naturally continued to make Orthodox Christianity the official religion in all the Balkans.
The final official spit between the Roman Catholic and eastern Orthodox Churches came in 1054 after centuries of increasing differences.
The Kingdoms of Croatia and Hungary were joined in a personal union after about 1100, giving Hungary an interest in the Balkans.
Stefan Nemanja of Serbia revolted against the "Byzantine" Empire after 1180, and in 1185 the Bulgars revolted against the "Byzantine" Empire and founded the Second Bulgarian Empire. In 1204 the 4th Crusade captured Constantinople and the "Byzantine" Empire dissolved into many smaller states.
For decades greater powers such as Epirus/Thessalonika, The Latin Empire, Nicaea, Serbia, and Bulgaria competed for control of the Balkans and smaller states were conquered or vassalized by the larger ones. And the more powerful kings of Hungary and Croatia often extended their power and influence into the Balkans. Many Balkan regions were sometimes vassals of Hungary.
Note that Bosnia was in an angle between Croatia and Hungary, which had the same king during the Middle Ages, and so was made a vassal of a powerful Hungarian ruler first and most often during the Middle Ages. And Bosnia was right at the border of Serbia and quick to ally with or become a vassal of the more powerful Serbian rulers.
So some Bosnian Christians joined the national Bosnian church, but others became Roman Catholics like the Hungarians and Croatians, while others became Eastern Orthodox like the Serbians. And over the centuries for various reasons many Croats and Serbs settled in Bosnia, where the native language would have been Serbo-Croatian and thus easy for them to understand.
The above information is basically Balkan history 101, a very brief and elementary history of the Balkans in general and Bosnia in particular.
The nature of the Bosnian church is little known today.
Roman Catholicism and Eastern Orthodoxy initially predominated in different parts of what is today Bosnia and Herzegovina; the followers of the former formed majority in the west, the north and in the center of Bosnia, while those of the latter were a majority in most of Zachlumia (present-day Herzegovina) and along Bosnia's eastern border (from Drina to Vrbas river)
A separate Bosnian church was established in 1252 when the Bosnians refused to be put under the authority of a Hungarian bishop. Many Bosnians remained members of the Roman Catholic or Eastern Orthodox churches.
This separate Bosnian church existed for about 200 years, although it is uncertain whether it ended before, during, or after the Turkish conquest in 1463. It probably gradually faded away as its members became more and more Roman Catholic or Eastern Orthodox to fit in with local communities.
The Bosnians were often accused of being heretics, like the Bogomils of the Balkans and their Cathar offshoot in southern France. The bloody and genocidal Albigensian Crusade of 1209 to 1229 against the Cathars showed how dangerous it was to be accused of Bogomilism.
In fact popes preached several crusades against Bosnia, one canceled in 1203, and two in 1221 and 1225 that never got off the ground. Another crusade against Bosnia was called in 1234, resulting the Bosnian Crusade of 1235-1241, a Hungarian invasion of Bosnia that only ended when the Mongols invaded Hungary in 1241.
No wonder many Bosnians were willing to break with the Roman Catholic Church in 1252.