The thing is that at the time in question, France was actually quite diverse (and yet sufficiently unified on a political level to become a rather successful democratic nation-state rather than crumble like Austria-Hungary). And as you correctly surmised, Alsace-Moselle (the German “Elsaß-Lothringen”) loomed large in French minds but not quite for the reasons you suspected.
The problem is not so much that France had no time to focus on Switzerland or Belgium but rather that it could not define itself based on some narrow ethnolinguistic features, precisely because Alsace was obviously strongly influenced by German culture. In France, it was (and to some extent still is) commonplace to contrast the “German” (ethnolinguistic) vision of what a nation is with the French vision exemplified in particular by Ernest Renan, who developed a whole theory of the nation designed to support claims that peripheral regions (and especially Alsace-Moselle) really are French.
France was, therefore, first, the area under the influence of the French kings and later a nation that sees itself as being based on the (actual or theoretical) consent of its citizens. At times (e.g. during the revolutionary wars or, in a twisted way, during colonization) there was even some temptation to consider that France could grow to encompass large parts of Europe or the world to bring progress to everyone (colonization was actually very ugly but, in France, its predatory nature was not explicitly acknowledged in the way it could be elsewhere; even the unification of mainland France itself was not always a smooth process).
The French language was therefore not a given, a way to sort people into French and non-French, but something coming from the top as a way to realize this nation. At the end of the 19th century, there are millions of French citizens (to say nothing of the colonies) who do not speak French or, at the very least, do not speak it as their first language (in Brittany, the Basque country, etc.).
It's interesting to note that during one or two centuries, the French language was extremely prestigious. It was the language of many European courts and thinkers actually debated whether it was intrinsically more suited to sophisticated thought than other languages. A bit later, in 1794, Henri Grégoire famously advocated the “eradication” of local languages and patois (the dialects many French people actually spoke) to spread the French language, linked in his thinking with progress and democracy. The common thread and relevance of all this is that the French language is presented not so much as something distinctly French as it is something France gifted to the world (but that everyone might want to speak).
In this context, it makes sense to claim territory that (in some way) belonged to France in the past rather than focus on languages and unifying French speakers is not a priority (or could even backfire badly).