This site claims the letter to be dated 30 March 1796 (Apparently 10 Germinal) and sent from Nice. This is a mere three weeks after their whirlwind romance and marriage on March 9 that year, shortly after Napoleon's appointment to command the bankrupt Army of Italy.
From context the "visitors" might be parents of Napoleon's step children by Josephine - Eugene and Hortense - and this may in fact be what Josephine intended to imply. The intimation is clearly made that Josephine was not yet "made up" for the day, despite the mid-morning hour, so such visitors would not alarm her new husband much. However:
Joséphine, left behind in Paris, in 1796 began an affair with a handsome Hussar lieutenant, Hippolyte Charles. Rumors of the affair reached Napoleon; he was infuriated, and his love for her changed entirely.
I could not confirm timing but it is not inconceivable that Josephine had already met
Charles and begun, or begun contemplating, an affair with the dashing young cavalryman, given this comment from Napoleon's letter:
And then there are the four days from the 23rd to the 26th; what were you doing, since you were not writing to your husband? Ah, my love, that ‘Vous’ and those four days make me long for my former indifference. Woe to the person responsible!
It is well known that the best lies, and the easiest to maintain, are the smallest lies. Eugene and Hortense were already 15 and 12, so how could an early morning visitor be permanently hidden from Napoleon and his family. I suspect that the "visitors" referred to were Charles (and possibly companions), but that Josephine phrased the reference to imply that they were visitors for the children rather than for herself.