Disunity at the Federal level is not a good answer because every Indian State encountered by the British had a much larger population and resource base than could be fielded by the Company.
In any case, even had India been united, so long as it remained a land based military power, for logistical reasons, it could never secure its own littoral and would have been bound to concede essentially what the Manchus did after the Opium wars.
The Japanese, it is true, first united their country and then westernized it but, unlike the Indians, they had actually improved on Dutch technology when first exposed to it. They also had a great maritime tradition. Though ultimately marginalised by Hideyoshi's successors, this small 'modern sector' could rapidly expand in response to the shock of the U.S Navy's demand for the opening of Japanese ports. Nothing similar could be said of India.
The real problem faced by the Indian Polity lay in the intense competition for the throne and the perquisites of power within each State. It was a case of brother against brother, Uncle against nephew and, even if the dynasty was cohesive, there was the ever present danger that the head of the Army or the Administration would launch a coup.
Why was this the case? One reason is that the principle of primogeniture (i.e. the eldest son takes everything) had not taken root. There were two reasons for this
1) Unlike Christian Europe, India had polygamy. In the short term, this meant Princes could make multiple marriage alliances and thus a potential rival might accept the Prince's suzerainty in the hope that his own grandson, from the daughter he has married off to the Prince, will gain the throne. However, this meant that the sons of the Prince saw each other as rivals and were supported by their maternal Uncles in scheming against, and even killing, each other.
The Chinese got round this problem by enforcing Confucian morality which stressed one's duty to one's elders. Here a younger brother's rebellion against an elder could be condemned as filial impiety and evoke a feeling of horror.
To some extent, Hinduism achieved a similar result by holding up Lord Rama as 'purushottam'- the ideal man. However, Turko-Afghan dynasties had no similar tradition. Indeed, a conflict between the sons of the Emperor could result in the best soldier/intriguer coming to the top- e.g. Emperor Aurangazeb. However, his successors paid a heavy price partly because of their own incompetence and decadence but also because loyalty to the dynasty was wholly divorced from loyalty to the present incumbent of the peacock throne.
2) The other reason the principle of primogeniture, which would have granted indefeasible legitimacy to the eldest son and greatly reduced factionalism within the state, did not take root has to do with the traditional pattern of land-holding. Looking at this 'bottom up', we find that Indians preferred co-parcenary inheritance such that all the descendants of a sept had a notionally equal interest in land. It was in the interest of the 'big man' of the sept to leave the question of who was entitled to what ill-defined and to direct the energies of 'have-nots' within the sept to prey upon their supposed 'tenants' or the territories of rival septs. Naturally, this made for disunity at the local level.
Looking at things 'top down', we find that the Prince could not put an end to local anarchy because his revenues derived from feudal land-grants. Originally the members of a sept whose coparcenary title in land had been recognized by the Prince in exchange for the military service of a given number of troops, would have shown loyalty & esprit de corps and performed well militarily. However, longer term, as the Prince required more and more money (for example to pay for the import of horses, or artillery and foreign mercenaries to operate the canons) rather than war-bands, the feudal land grant lost its military character and became an instance of 'tax farming'- i.e. entrepreneurs would bid for the right to collect taxes from a territory in return for a fixed payment of tribute.
In practice this meant the rise of a new type of 'Feudal Lord' from the trading castes as well as enterprising members of the clerisy (generally members of either the 'Writer' caste (Kayastha) or secularized (niyogi)Brahmins.
This meant that the Feudal principle of Legitimacy itself, let alone primogeniture, went by the board. Instead, tenants took military service during the campaigning season for a money wage which they could use to pay their own taxes. But, what incentive did they have to fight rather than run away if their side appeared to be losing? Similarly, what great attachment could the tax-farmer feel for the Prince he was obliged to pay?
In those European Kingdoms which made a successful transition from Feudalism to Modern Nation States, land-holding was first fully delinked from feudal obligation and found a new legitimacy in the legal forms underpinning the commercial practices of 'Civil Society'- i.e the Third Estate.
The King, in order to finance his military establishment, had to do a deal with a Parliament which represented those who remitted the taxes he could levy. Prussia, in this respect, was exceptional, but then the Prussian period of dominance ended in disaster for Germany.
Nothing of the sort happened in India. True, the Merchant's guild might have a countervailing power over the King but there was no mechanism such that the two parties could agree a mutually advantageous agenda. One reason for this was that the Merchant guilds- though claiming a hoary antiquity and reinforced by draconian caste based rules- could easily split. Brother might separate from brother and decide to back a rival Prince.
Timur Kuran, the Turkish economist, has pointed to Islamic inheritance law- which tends to terminate a business enterprise on the death of its founder so as to divide its assets among his heirs- as the reason for Islam falling behind the Christian West. Hindu inheritance law could militate against this- Jinnah, the founder of Pakistan, quoted this as the reason Muslims couldn't compete on a level playing field with Hindus and thus must have their own country- but,in practice, this was seldom the case because Political uncertainty was so severe as to make long term considerations irrelevant.
To reiterate- Uncertainty and the concomitant path-dependence of the economyin the face of random shocks- had a lot to do with lack of clarity as to who legitimately owned what. Lack of primogeniture at the top together with coparcenary land title at the bottom were important systemic contributors to this Uncertainty.
These factors, by themselves, without any outside intervention, would have been enough to ruin most of India's export industries unless the Indians were able to secure safe sea routes for their merchandise and thus had an alternative market overseas when and if domestic conditions became chaotic.
As a case in point, South India had been famous for its 'wootz' steel swords for a thousand years. This industry did not collapse because foreigners had a superior product but because of excessive uncertainty and random shocks in the domestic market. Essentially, the investment of agricultural surpluses in cottage industries would tend to break down under conditions of political chaos unless one could both import food and export manufactured items by a safe route. Britain, once it gained Naval Mastery- which only happened because its Parliament was successful in forcing the King to do what was advantageous to the propertied class- did indeed experience an Industrial Revolution such that its Textiles and Steel and so on dominated the World market for almost a hundred years. India's political uncertainties as well as its rulers neglect of the one safe avenue of trade- the Sea- meant that its skilled workers declined or disappeared while their peers in England went from strength to strength. We have all heard the name 'Wilkinson Sword' or 'Wedgwood Pottery'. The Wootz sword has disappeared from history though, no doubt, it once boasted a Nock or a Wilkinson. The same thing can be said about Dacca Muslin or Madras Chintz and so on.
This brings me to the real reason the East India Company was able to take over India.
The propertied class in Britian was able to get their King and his Cabinet to do things which made them safer and richer. The working classes of Britain showed great patriotism and esprit de corps when press-ganged into the Navy or made so drunk by the recruiting sergeant as to accept the 'King's schilling'. The result was that the productivity and force projection capability of the British (and by mimetic effects, their European rivals or imitators) greatly increased in a manner which ultimately made them irresistible by any save an equally industrialized and technologically advanced force.
However, at the core of this forced productivity improvement in the military sector was the Civilian Control of the bourgeoisie demanding an ever bigger 'bang for the buck' in the shape of access to yet bigger markets and larger distributions of prize money. In return, Britain, uniquely, came out of its wars with sounder money and a declining per capita military burden. By contrast, the Maratha merchants were crushed by the Peshwa's taxes and increasingly defected to the territories of John Company.
But this trend only reinforced the solvency and credit-worthiness of the admittedly corrupt and often incompetent E.I.C. (Though, in justice it must be said, that some Collectors appointed by the Company showed entrepreneurial genius of a high order- for example in the opening up of Sylhet towards Meghalaya.)
Indeed, as Mahatma Gandhi pointed out, it was the Indian merchants of Surat who were the first to realize that a British 'hoondi' (Bill of Exchange) was gilt edged. Soon Indian Princes and Captains got the message. When the Brits said 'we'll give you a pension of Rs. x per annum' that pension would in fact be paid for all eternity.
One final point, the Brits, unlike the French, initially discouraged Missionaries and respected all Religions. They took the trouble to administer Justice according to local practices and in conformity with precepts of the relevant Religious authorities.Thus,the true paradox of British rule in India was that, at the ground level, it seemed more 'timelessly Indian' than the chaos that preceded it. Indeed, during the course of the Nineteenth Century, the Anglophile 'comprador' (i.e. a guy who gets rich working with the new overlord)shed his obnoxious Westernized veneer and, more often than not, invented a fake pedigree for himself as an upholder of traditional values.
Of course, there are many other explanations for the success of the E.I.C, all more or less paranoid or motivated by some obviously silly political ideology. Some are backed by impressive empirical research. However, they all cash out as instances of 'kakathaliya'- coincidence not causation.