Australia was a lot less hospitable area than Polynesians, Indonesians, and other islanders were used to.
Although technically in the tropics, the fact that people lived on islands meant that the sea was a moderating influence on the climate, and "temperate" weather crops such as breadfuirt and sweet potatoes could grow on them. Also, the sea provided a ready source of fish.
Not so Australia, which being continental, had a less temperate climate. Jungles were thicker, crops scarcer, and food harder to find. As mentioned in other answers above, the "Australians" had more potent weapons, probably because of the need to hunt mammals, rather than fish.
It's possible that Pacific Islanders found themselves in Australia from time to time. Most probably did not survive. Perhaps a few survivors were absorbed into the local populations.
What did NOT happen was people visiting Australia, leaving, and then saying, "Let's go back there with the friends and family.