I do not know of any hard evidence that any of these people groups returned to Babylon. There is an article[1] that I found that interprets a vision of Zechariah as relating to the Samaritans returning to Babylon. Specifically it proposes that Zechariah's vision of the Ephoh refers to the Samaritan people and their removal from Judah back to Babylon. The originator of this interpretation does a wonderful job explaining why this view makes the most sense when compared to existing ones. However, at the end of the paper, the author must admit that even though their thesis seems sound "we know of no movement of Samaritan's back to Babylon".
With that said, there are many reasons to believe that very few, if any, of the people that were relocated to Samaria from these 4 cities returned home after the Assyrian empire ended. Not only does this group remain in existence today, there is good reason to believe they remained in Samaria at least through the 6th and 5th centuries like asked.
First, a significant amount of time had passed between when these people were transplanted to Samaria and when they would first have had an opportunity to return home in the 6th or 5th centuries. The northern kingdom fell to the Assyrians in approximately 721 BC and remained in power of both territories until the end of the 7th century, between 629 BC and 609 BC. It is unlikely that a significant group of people would return home after over 100 years of living in a land unless they were forced to do so for some reason or another.
Furthermore, the book of 2 Kings records these original people groups to have been residents of Samaria even to the date that the book was written. While the book of 2 Kings records the event through about 586 BC, is was not believed to have been written until close to the end of the Neo-Babylonian empire around 550 BC. Therefore, when the author states in 2 Kings 17:34 that these 4 people groups "To this day they persist in their former practices", is referring to the time the book was written and not the earlier time periods. This is significantly into the time period of the Neo-Babylonian empire which started in 626 BC and ends in 509 BC.
In addition, even during the time of the Persian empire, we find biblical references to these same people groups still residing in Samaria. Ezra 4:2 quotes these Samaritans as stating they "have been sacrificing to him ever since the days of Esarhaddon king of Assyria who brought us[them] here". This would imply that a significant portion of Samaritans remained in the land near Judah and have not left. Even though this statement was made at the time Zerubabel, the same people group writes a letter many years later to Artaxerxes who reigned over Persia from the middle of the 5th century to the 420BC. In this letter, they again identify themselves as "nations whom the great and noble Osnapper took captive and settled in the cities of Samaria". In which they specifically mention Babylon as being one of the cities they came from in Chapter 4, verse 9.
Lastly, they have a religious connection to the God of Israel and do not desire to return home. Even though it seems that the nations who were moved to Samaria brought their idols and gods with them, they also worshiped the God of Israel. Since much of the disagreement between the Samaritan and the returning Jews was due to location of and involvement of rebuilding the Temple, it would seem they held the land as important to their religious views.
In short, there is no known evidence supporting a migration of people form Samaria to Babylon in the 6th and 5th centuries. Evidence and explanations for the contrary view are much more abundant. Therefore at this time, it is much more likely that there was no signification number of people moving from Samaria to Babylon during that time-frame.
[1]: Assis, E. (2010). Zechariah's Vision of the Ephah (Zech. 5:5-11). Vetus Testamentum, 60(1), 15-32. Retrieved August 9, 2021, from http://www.jstor.org/stable/20700044
