How did Bligh navigate during his open boat voyage after losing The Bounty?
Did any of the men have previous experience in these areas? What did he have at his disposal to aid in navigation?
The book The Eventful History of the Mutiny and Piratical Seizure of H.M.S Bounty by Sir John Barrow includes a chapter on the remarkable voyage of Bligh and his 18 companions in their 23-foot boat, drawn from Bligh's description of the voyage. Unfortunately, Barrow concentrates on the hardships of the men and includes few details on how navigation was accomplished.
Bligh's own, more lengthy account, was given in A Narrative of the Mutiny on Board His Majesty's Ship Bounty which was published in 1790. This includes details of the mutiny itself and the subsequent voyage of Bligh and his companions.
As far as the navigation tools available, Bligh states that as he and his companions were placed in the boat, the boatswain and seamen were allowed to collect items to take on the voyage.
Mr.Samuel [ship's clerk] got 150lbs of bread, with a small quantity of rum and wine. He also got a quadrant and compass into the boat; but was forbidden, on pain of death, to touch either map, ephemeris, book of astronomical observations, sextant, time-keeper, or any of my surveys or drawings.
A Narrative of the Mutiny on Board His Majesty's Ship Bounty, pg5
It's not entirely clear whether the "compass" referred to in the passage, was a magnetic compass or a divider caliper. However, given that they had no charts, the latter would seem to be of little use. Also Bligh's log has regular entries on wind direction and direction of travel so the presence of a magnetic compass seems more likely. The quadrant was used to observe latitude and while longitude was taken 'by account' (i.e. by dead reckoning).
[May 9th] At noon, I observed the latitude to be 15°47'S; course since yesterday N 75°W; distant 64 miles; longitude made, by account, 8°45'W.
A Narrative of the Mutiny on Board His Majesty's Ship Bounty, pg49
When the weather prevented an observation, then latitude was also taken by account.
[May 12th] At noon it was almost calm, no sun to be seen, and some of us shivering with cold. course since yesterday W by N; distance 89 miles; latitude, by account, 14°33'S; longitude made 13°9'W. The direction of my course is to pass to the northward of the New Hebrides.
A Narrative of the Mutiny on Board His Majesty's Ship Bounty, pg53
In addition to the navigation tools already mentioned, it's implied that Bligh used the stars to navigate since the account mentions a couple of occasions when the poor weather made it impossible.
The whole day and night of the 15th were still rainy; the latter was dark, not a star to be seen by which the steerage could be directed, and the sea was continually breaking over the boat. On the next day, the 16th, was issued for dinner an ounce of salt pork, in addition to their miserable allowance of one twenty-fifth part of a pound of bread. The night was again truly horrible, with storms of thunder, lightning, and rain; not a star visible, so that the steerage was quite uncertain.
The Eventful History of the Mutiny and Piratical Seizure of H. M. S. Bounty, pg104-105
Often, because of the weather, they simply went where the sea carried them.
The night was a dark and dismal one, the sea constantly breaking over us, and nothing but the wind and waves to direct our steerage.
The Eventful History of the Mutiny and Piratical Seizure of H. M. S. Bounty, pg105
As for navigating the barrier reef, it seems that they came upon this unexpectedly and handled the navigation of it by sight and sound.
Accordingly, at one in the morning of the 28th, the person at the helm heard the sound of breakers. It was the 'barrier reef' which runs along the eastern coast of New Holland, through which it now became the anxious object to discover a passage: Mr. Bligh says this was now become absolutely necessary, without a moment's loss of time. The idea of getting into smooth water and finding refreshments kept up the people's spirits. The sea broke furiously over the reef in every part; within, the water was so smooth and calm, that every man already anticipated the heartfelt satisfaction he was about to receive, as soon as he should have passed the barrier. At length a break in the reef was discovered, a quarter of a mile in width; and through this the boat rapidly passed with a strong stream running to the westward, and came immediately into smooth water, and all the past hardships seemed at once to be forgotten.
The Eventful History of the Mutiny and Piratical Seizure of H. M. S. Bounty, pg110-111
Bligh knew from his earlier travels with Cook roughly where he was at that point and, therefore, knew the rough heading for Timor. He estimated correctly the boat's position 11 days later and was able to estimate the entire distance covered in the voyage -
On the 11th Lieutenant Bligh announced to his wretched companions, that he had no doubt they had now passed the meridian of the eastern part of Timor, a piece of intelligence that diffused universal joy and satisfaction. Accordingly, at three in the morning of the following day, Timor was discovered at the distance only of two leagues from the shore...It appeared scarcely credible to ourselves that, in an open boat, and so poorly provided, we should have ben able to reach the coast of Timor in forty-one days after leaving Tofoa, having in that time, by our log, a distance of three thousand six hundred and eighteen nautical miles; and that, not withstanding our extreme distress, no one should have perished in the voyage.
The Eventful History of the Mutiny and Piratical Seizure of H. M. S. Bounty, pg118
As Mark has noted above, Bligh was denied charts when he was set adrift by Fletcher Christian and the other mutineers. He was, however allowed to take a copy of John Hamilton Moore’s Practical Navigator in addition to the sextant that Mark mentioned.
Bligh knew his initial position and his (approximate) destination. With the sextant and Hamilton-Moore's Practical Navigator, he was able to make latitude observations and so navigate by dead reckoning. He also had a disciplined crew on his boat and he had luck on his side.
Bligh's sextant, and other items from that remarkable voyage are held in the collections of the National Maritime Museum in London.
Not a full answer, but one of the key requirements when sailing by dead-reckoning is to calculate the leeway accurately, so that the course sailed can be adjusted properly from the course steered. (For those without sailing experience, leeway is the drift downwind due to drag that reduces the velocity made good for the desired course.) This video illustrates a means of measuring leeway as the angle between a line extended dead astern from the boat with one extended along the dead water in the boat's wake.
Numerous other references on techniques for coastal navigation (usually done by dead reckoning and periodically adjusted by sight references to known landmarks) and celestial (or offshore) navigation are available on the web. Bligh and his companions would have used a combination of these techniques, along with sounding weights for determining the nature of the bottom under the boat, during the voyage to East Timor. As noted in @Steve Bird's answer, the one notable landmark encountered by which Bligh could correct the accumulated dead-reckoning errors was the Great Barrier Reef, 11 days our from their East Timor destination.