You have asked two questions here. Part of the answer would appear to be contained in the pdf copy of documents relating to the seizure of American vessels from the New York Public Library site on the Abolition of the Slave Trade (titled 'Doc No 34') that you linked in your question (the first link, currently labelled simply '1').
For the question in your title:
How were false US flags identified on slave ships in the transatlantic slave trade?
The answer is stated simply in the extract of the communication between Mr Stevenson and Mr Webster, dated May 14th, 1841, which appears on pp 11-12 of that document:
"His lordship [Lord Palmerston] then expressed an opinion that the right existed of ascertaining in some way or another, the character of the vessel; and that by her papers, and not the colors or flag which might be displayed."
- (emphasis appears in the original)
If the ship's papers did not match the flag being displayed, then the vessel was flying a false flag.
The US reluctance to permit such checks is explained in the same document:
"I at once assured him that under no circumstances would the Government of the United States consent to the exercise of the right on the part of any foreign nation, to interrupt, board, or search their vessels on the high seas. That to admit the right of a foreign naval officer to decide upon the genuineness of the papers of American vessels by boarding them, or bringing their captains on board of British cruisers, was in effect allowing the right of search, and therefore utterly indefensible."
It seems that Lord Palmerston disagreed, and had replied that, in the view of Her Majesty's Government:
"... it could not be regarded as a right of search".
So, the US position was that by boarding their vessels to determine whether the ship's papers matched the flag she was flying, the Royal Navy was violating their sovereignty. The British position was that they were merely confirming the nationality by checking the vessel's papers.
That remained the US position until 1862.
Which brings us to your second question:
How long were these ship captains able to fly false US flags before the US permitted the Royal Navy to search their ships?
The answer is until a treaty was agreed between the US and the UK in 1862, fully two decades after the correspondence discussed above.
Permission was granted to the Royal Navy by the United States (and to the the US Navy by the United Kingdom) to visit and search vessels suspected of being involved in the illegal slave trade under the terms of the Treaty between the United States and Great Britain for the Suppression of the Slave Trade, also known as the Lyons–Seward Treaty.
Article 1 of the Lyons–Seward Treaty stated that:
The two high contracting parties mutually consent that those ships of their respective navies which shall be provided with special instructions for that purpose, as hereinafter mentioned, may visit such merchant vessels of the two nations as may, upon reasonable grounds, be suspected of being engaged in the African slave trade, or of having been fitted out for that purpose; or of having, during the voyage on which they are met by the said cruisers, been engaged in the African slave trade, contrary to the provisions of this treaty; and that such cruisers may detain, and send or carry away, such vessels, in order that they may be brought to trial in the manner hereinafter agreed upon.
From that date, Royal Navy ships had the right to stop and search ships flying the flag of the United States which were suspected of involvement in the illegal slave trade. US Navy ships had similar rights for vessels carrying the UK flag, if they were suspected of involvement in the illegal slave trade..