The Cabinet Historique et topographique militaire was created by a decree the 28th August 1794. The decree goes in detail about the work and the organization to the point of naming who does what. A second decree (16/06/1795) has also elements of organization. The decrees don't mention office hours.
The work done by the bureau in support of the armies was important but not time critical in a way that would need special hours.
Even though the idea of office hours is somewhat anachronistic, the hours mentioned by Andrews would be odd. Nevertheless, there's no source for these hours.
I would think then the Bureau had not odd hours (the other would be that its hours wouldn't be odd at the time, but as far as I know, they would be) and there's no source supporting such thesis. Also, the idea that a 8 hours day would leave a lot a free time is somewhat strange. It's not a big day but not a small one either.
If someone can provide a source I'll happilly revise my opinion.
As for Andrews:
I couldn't consult Napoleon the Great. But the text it's the same in his Napoleon: A Life [the same book with UK title]. Andrews has only two references for the passage that don't justify the points of fact (even less the reasoning). A primary source, i.e. a quote from a letter that Napoleon wrote to his brother. A secondary source, p. 128 of a book from historian Howard Brown.
Andrews use of Napoleon's letter is strange: "three days later [20/08/1795] he was crowing to Joseph: ‘I am at this moment attached to the Topographical Department of the Committee of Public Safety for the direction of armies.'"
The full letter is available in translation. It starts "I am attached for the present to the topograpical board of the Comittee of Public Safety for the direction of the armies; I replace Carnot."
It then goes on for a full page on completely unrelated matters. I don't see any crowing. Self-agrandizement (to replace Carnot?)? well it's Napoléon...
As to the analysis of the Bureau functions it seems to be based on a page of Brown. I couldn't check this book but I would be surprised that it would bring something new to this answer (a part giving some credibility to Andrews ideas about the Bureau). The assertion about the hours goes without reference and its not clear that its source is Brown (there's another unrelated reference before the assertion).
For the decrees:
Bonnal de Ganges, Edmond. Les Représentants du peuple en mission près les armées, 1791-1797, d’après le dépôt de la guerre, les séances de la Convention, les archives nationales, par Bonnal de Ganges,... Tome IV. Les représentants et les armées dans la politique. Paris: A. Savaète, 1899. pp. 443-451
Available from archive.org
Translation of Napoleon's letter [I've checked with the french, the English is a bit antiquated but ok]:
Napoleon. The Confidential Correspondence of Napoleon Bonaparte with His Brother Joseph. Vol I. New York: D. Appleton and Company, 1856. pp.271-2
Available from archive.org
Brown, Howard G. War, revolution, and the bureaucratic state: politics and army administration in France, 1791-1799. Oxford historical monographs. Oxford : New York: Clarendon Press ; Oxford University Press, 1995.
Roberts, Andrew. Napoleon: A Life. Penguin Publishing Group, 2014.
Roberts, Andrew. Napoleon the Great. London ; New York: Allen Lane, an imprint of Penguin Books, 2014.