If the Act of Settlement 1701 did not pass, who then would have been next of kin after the death of Anne of Great Britain?
Anne's half-brother, James the Old Pretender would be her next of kin. He was the closest living relative; son to the deposed King James II & VII and sibling to Queen Mary II. However, it seems what you really want to know is who would succeed Anne to the throne of England. In that case, it would still be the heirs of Sophia the Electress of Hanover.
England was simply not willing to accept the return of a Catholic monarch. The Act of Settlement forestalled a potential succession crisis by fixing the royal succession on the line of Electress Sophia. But even without it all Catholics were already excluded from the royal succession. Over a decade earlier, the Bill of Rights of 1689 provided that:
And whereas it hath been found by experience that it is inconsistent with the safety and welfare of this Protestant kingdom to be governed by a popish prince ... enacted, that all and every person and persons that is, are or shall be reconciled to or shall hold communion with the see or Church of Rome, or shall profess the popish religion, or shall marry a papist, shall be excluded and be for ever incapable to inherit, possess or enjoy the crown and government of this realm and Ireland and the dominions.
-1 Will. & Mar. sess. 2 c. 2
Thus excluding James II & VII, and his successors, from the thrones of Great Britain. Even this was merely a reflection of the existing political environment and mood of the nation. It created a pro forma exclusion of the Jacobites, but their chances were very minimal either way.