At that time the trade of the United States was primarily with Britain or the Spanish main. The English trade was conducted in pounds sterling and the Spanish trade in the Spanish milled dollar. Ultimately the dollar was selected as the U.S. standard of value.
The merchants themselves would have dealt primarily in bank notes, but the banks eventually would settle imbalances in specie, usually in gold.
In other words, what happens is this:
(1) Liverpool cotton broker contracts to buy cotton from an American broker in New Orleans. The price will be quoted in pounds sterling on the Liverpool exchange.
(2) The American broker will buy the cotton in Spanish dollars using a bill of credit drawn on an American bank. In other words the planter will get pieces of paper from the broker (bank notes). These notes are denominated in dollars.
(3) The broker will now ship the cotton to Liverpool.
(4) If the cotton arrives the Liverpool broker will sell it on the Liverpool exchange and receive pounds sterling drawn on an English bank. I.e., he will receive paper denominated in pounds sterling. At the same time he will send a draft to a correspondent bank in New York remitting to the New Orleans broker the agreed sum in pounds.
(5) The New York bank will perform the exchange, taking the pounds and producing in return a sight draft in dollars. This sight draft is then sent to the New Orleans broker. He now has a credit in dollars drawn on a New York bank. He can use that money in New York, or have it forwarded to his bank in New Orleans.
(6) Meanwhile, the New York bank has credit in pounds sterling and a debit in dollars. The bank accumulates all these debits and credits and consolidates them, so eventually they will have a single credit (or debit) in pounds sterling against a London bank (which will assume the debit from the Liverpool bank).
(7) Ultimately this imbalance is resolved by the transhipment of gold. So, if the London bank owes the New York bank, say 5 million pounds sterling, then this amount will be converted to gold and the gold will be sent by packet ship to New York. And vice versa, if New York owes London, then it sends the gold. These settlements generally happened monthly.
In this description I have simplified things because I have neglected freight and exchange discounts, which make the transaction much more complicated, but this is the essential skeleton of what happens.
So, to answer your question, the merchants themselves deal in paper (drafts of various kinds) and eventually any residual imbalances are settled in gold by a few very large banks which consolidate the debits and credits of all of the traders and merchants.
To come full circle, let's talk about what happens next even though it has nothing to do with international trade. Now, the New Orleans broker, lets call him "David Chambliss" has a sight draft (a "check") drawn on the United Bank of New York. So, this document says:
The United Bank of New York agrees to pay David Chambliss of New Orleans $5,000 on demand. Signed, S.Derby, Treasurer of the United Bank of New York.
Chambliss has this document since the New York correspondent mailed it to him. Chambliss can use this draft to buy something in New York, but lets say he wants to have the money locally so he can buy more cotton. What he will do is take the draft to his bank in New Orleans and write on it:
Pay to the Bank of New Orleans, [signed] David Chambliss
When the Bank of New Orleans gets this, it will credit Chambliss $5,000 and he can then withdraw silver dollars from the bank to buy cotton. Meanwhile, one of two things will happen depending on whether the United Bank of New York is a correspondent bank of the Bank of New Orleans, meaning that it has an account there. If so, then the Bank of New Orleans will send the draft to the United Bank of New York which will credit the account of that bank with the $5,000. If not, then the Bank of New Orleans will send the draft to whichever bank is its New York correspondent, say its the Mercantile and Trade Bank of New York. In this case the New Orleans Bank will receive the credit from Mercantile and Trade, and Mercantile and Trade will settle up with United.