This wikipedia article on Botany Bay suggests the following reason:
Governor Arthur Phillip sailed the Armed Tender Supply into the bay on 18 January 1788. Two days later the remaining ships of the First Fleet had arrived to found the planned penal colony. Finding that the sandy infertile soil of the site in fact rendered it most unsuitable for settlement, Phillip decided instead to move to the excellent natural harbour of Port Jackson to the north.
I have always been intrigued by the decision to move as Port Jackson is not so far away from Botany Bay.
To this day there are not many buildings on the north side of Sydney Harbour, most of the settlement to my understanding was built near Farm Cove which is hardly a trek from Botany Bay.
So was the poor soil the real reason for moving to Sydney Harbour or where there other more important reasons?