I wonder if Ali gave allegiance to (or otherwise accepted the reigns of) these three caliphs? Did he do this freely or under duress? There wasn't any major conflict started by Ali during these reigns, and "the establishment" eventually elected Ali as the fourth caliph after Uthman, so it seems that he got along fine?
The accepted story is that he did (according to Al-Tabari, for example). Each time a Caliph was chosen, all top early Muslim commanders gathered during the crowning ceremony and swore allegiance to the new leader.
That said, there are a number of dissenting scholars who consider that the official Muslim accounts are unreliable to the point that the first three Caliphs might not be actual historical figures. See for example this book.