As Semaphore points out, the UK is paying interest on consolidated annuities. His links are better, but here is a story from the New York Times. (Note: Semaphore corrected my quibble about UK vs GB; he was right, I've removed my text).
France continues to pay on a debt from 1738.
But the oldest active debt is Dutch:
The very oldest active bonds, however, were issued in 1624 by the Hoogheemraadschap Lekdijk Bovendams, a Dutch water board responsible for upkeep of local dykes, and verified to be still paying annuities in 2003. KickAssFacts
(Aside: I'll be surprised if anyone finds a debt that is much older than that; I lack the scholarship to make a coherent argument, but the nature of debt, credit and money change around that period.)
How are these debts paid?? Like all government debts since the time of D'israeli, they are secured by the faith and credit of the government and paid through government revenues. Government has a variety of ways to raise money including taxes, fees, inflation, seignorage, etc. Note that in direct contrast to OP's assumptions, these are first world debts paid by first world debtors, and as far as I know, the creditors are also first world entities.
Interest on government debts is always paid to the holders of the debt, and since debt can be bought and sold, there isn't really an answer to the "who" question. The historic cause of the debt is the same as the cause of all debt; Britain borrowed money. In this case Britain borrowed money to pay off the South Seas bubble.
That includes borrowing that may have been used to compensate slave owners when slavery was abolished, to relieve the famine in 19th-century Ireland and to bail out the infamous South Sea Company, which caused the bubble in 1720.
Rather comical to ask whether the oldest debt is to the world bank (which was created in 1944).
Edit: I woke up on the wrong side of the bed this morning and my inner curmudgeon is running rampant. I hope that OP will forgive my acerbic tone. I can either delete this answer or edit it after I've had some sleep.