- Casablanca is further from Tunis than London is from Berlin. Would you have regarded London in January 1943 as safe enough?
- The Vichy French had switched sides easily enough a two months earlier, so there was no realistic sense of domestic trouble in Casablanca, or French NW Africa.
- The premise that that German attack at the Kasserine Pass might have thrown the Allies out of NW Africa falls under the realm of alternative history; wasn't happening no-how, no-way. It caught completely green American recruits, under trained and under poor leadership, by surprise, and succeeded to a much greater extent than it truly had any right to.
- Churchill regularly proved his bravery during the Blitz, wandering around London at all hours of the day or night; he might easily have engaged in a caper such as you suggest. My belief is that Stalin, and Roosevelt at least through the 1944 election, regarded themselves as indispensable, and would thus not have put themselves in personal jeopardy.
With the battle for Stalingrad still underway, one can easily imagine Stalin's potential difficulties in getting to Casablanca. Any further East might similarly be problematic for Churchill and Roosevelt at that point in time.
Also, the troops needed no rallying by January 1943. The writing is already on the wall with Paulus surrounded in Stalingrad, and Rommel/Kesselring for all intents and purposes already thrown out of North Africa. Midway has already broken the Japanese navy. (Only Hirohito's war council could lose the war in an afternoon, as they did by rolling the dice so emphatically with that battle.)
London to Berlin: 580 miles ( 932 km)
Casablanca to Tunis: 1027 miles (1653 km)
Alexandria to Tunis: 1194 miles (1922 km)
While the Henkel 111 had the range (1800 km) to fly a mission to Casablanca from Tunis, it's cruising speed of 310 km would have meant surviving 5 hours of attack by Allied fighters to arrive over the target. As a level bomber it would have had to flatten an area probably of a square mile or so in order to have a realistic chance of eliminating the assembled leaders, even assuming they failed to relocate within the city. That latter means getting one or two bombers through is most likely insufficient to achieve the goal, most of the wing would have to get through for a good probability of success.
Update - re safety of Stalin flying from Moscow to Casablanca and back
In Hinge of Fate, Churchill records his itinerary from Gibraltar to Moscow in August 1942 as:
Gibraltar -> Cairo (over the Spanish and Vichy territories for about the first 4 hours in daylight and with an armed escort, undisturbed though nerve-wracking!) -> Tehran -> Kuibyshev -> Moscow. However the planned stop-over at Kuibyshev was skipped (to WST's regret at missing an expected feast) as weather was favourable and time urgent.
FDR writes (ibid p 668) to Churchill:
My route [to Casablanca] would be either from here to Trinidad and thence to Dakar and thence north, or from here to Natal (Brazil) and across to Liberia or Freetown, and north from there.