Pope Liberius was exiled to Thrace for a while in the mid 4th century.
During the early middle ages it was not unheard of for the Byzantine emperor to send people to arrest (or murder) the Pope in Rome when he started acting too independently for the emperor's taste. A prime example is Pope Martin I, who was arrested on the orders of Emperor Constans II and banished to the Crimea. The dispute was over an arcane detail of nature of the trinity, but the root issue was really about who had the ultimate authority over the church, Pope or Emperor.
As time went by, this got harder and harder for the Byzantines to do, and eventually the Popes started looking toward western authorities (starting with the Catholic Franks) for temporal protection rather than the Byzantines.
During much of the late middle ages, the papacy became pretty much a political plaything for north Italian rulers. It also controlled a fair bit of territory in central Italy, and thus spent a fair bit of time doing stuff that today we would consider rather inappropriate duties for the Vicar of Christ, like fielding armies and waging war on his Italian neighbors. Some of the popes during this period were decidedly less-than-holy men, even by the standards of the day.
My personal favorite from this period was (Anti)Pope John XXIII. He was put on trial, and as Gibbon put it, "The more scandalous charges were suppressed; the vicar of Christ was accused only of piracy, rape, sodomy, murder and incest." Eventually he was imprisoned in Germany, released, and (incredibly), made a Cardinal.
Still, typically what it took to get arrested was a dispute with a king, such as happened between Pope Boniface VIII and Phillip IV of France. The pope excommunicated Phillip and a local Italian family head, so they had the Pope arrested and tried to get him to abdicate. He refused, so he was just beaten up badly and left to die. There seems to be some dispute over his actual cause of death, but it occurred barely a month after his release.