I was hoping I'd have more of a chance to write something thorough, including sources, but I'll have to make this short. I'm basing this answer primarily on my understanding of Japan both pre- and post-Restoration on the works of Ryotaro Shiba, four of which have been translated and actively consider this period ('Clouds above the Hill', 'Ryoma!', 'Drunk as a Lord', and 'The Last Shogun'). I've previously expanded on Shiba's historical accuracy here.
The hierarchy in Japan was top–down; the shogun was the head of the shogunate, and everything was linked to him during the Edo period. This also meant that the "inner daimyo" (those who fought for the Tokugawa at Sekigahara) were favoured above the "outer daimyo" (those who fought against the Tokugawa at Sekigahara)—or, at least, the "inner daimyo" had the more central territories for themselves. I've described some of the methods that the shogun used to keep the daimyo in line economically here along with a brief description of the setup: one centre for foreign politics, and three hundred local centres of power.
This overall structure lead to stagnation in most han by the 19th century though some reformist lords did exist (e.g., Kanso). Also, while during the Sengoku Jidai many daimyo had engaged in fighting, the situation was very different by the middle of the 19th century almost none of them did so. The Tokugawa's attempt to turn the daimyo and their upper ranking samurai into effective administrators had worked quite well:
This was something extraordinary for the times. During the Boshin War, many domains had joined the imperial forces, but there was not a single instance of the lord himself leading the domain troops. Now here was the former lord of Sasayama saying he would join the military himself.
—Shiba, 'Clouds above the Hill, Vol. 1'
At the same time, while the Tokugawa had fixed societal classes early on in the 17th century, the old truth of merchants making money and other classes, especially the samurai, losing it was as true in Japan as elsewhere. This gave a basis for societal upheaval as well as monetary supply from the non-established classes.
If one investigates where the sonnō jōi movement started, this is in the "outer domains'" lower classes. These people (including lower samurai ranks, as many of the domains had various samurai rankings: nearly all of the top Restoration leaders came from the lower, foot samurai, classes) had the most to gain from societal differences, because not only were they comparatively worse off than the upper samurai, but also their domain was seen as "worse off" compared to other domains.
Yet, in many ways, until the concept of "the Japanese nation-state" was derived during the years leading up to the Restoration (and, indeed, after), many of the people rebelling against the shogun were hoping to flip the situation: the daimyo had no incentives to raise the Emperor up above themselves as a 'new shogun' (which eventually happened, even if tradition limited his power). After the Restoration, all of the military and civil administration was under the Emperor's power (in theory)
Regarding the situation of the loyalty of former samurai to their han after the Restoration, the below quote is illustrative (especially as it describes a time approx. fifteen years after the fall of the Shogunate: old loyalties ran deep even if there was no longer a formal connection between the lord and his former subjects):
This shows in what high regard the former lords of the old domains were still held at this time. After the Restoration, civil officials and military officers worked directly under the emperor. They were “the emperor’s officers” in theory. But the civil and military officials who were former samurai were in a delicate situation. As a matter of etiquette, they still maintained the manner of loyal retainers toward their former lords’ families.
—Shiba, 'Clouds above the Hill, Vol. 1'
When one wants to understand the position of the Emperor (or, the Imperial Court) vs the Shogun in the popular mind, looking into the development of shinto and kokugaku is probably the best way to go. Kokugaku is an exceedingly interesting topic, though I'm not the greatest expert on it myself.
However, one of the most obvious sources of power in the land was the Emperor: he could bestow court ranks and also authorize daimyo to carry out special missions. This is well evidenced in the early 1860's where control over Kyoto (and the Imperial Court) was the key—whoever controlled the city could issue edicts against the opposition, essentially outlawing them.
