For very practical reasons, these pure leisure activities did not take place close to enemy shores, or in climates that were just too uninviting. Close to the "Golden West" of American targets airplanes would be just much too quick to approach and the highest latitudes just too cold and often just too windy to be of any joy.
But the crews did try to enjoy every single opportunity at fresh air, everywhere. "On deck" on a boat means also on the top of the conning tower! Given the cramped conditions, bad smells and comparative darkness in the boats' interiors, this was almost necessary.
As the question is framed, it appears to be about being on the hull of the boat, outside the conning tower. This was done to make repairs, to practice with the secondary weapon, to dispose of waste. Just to name activities still involving duty.
But pure leisurely spare time was restricted to the relative safety of the air gap when look outs could relax somewhat since ships were relatively slow and planes unlikely. A surprising detail might be found in that at night the crew's movement was more restricted than during the day. Clear skies and long range visibility of enemy crafts was the deciding factor.
A map of the air gap shows that the vast majority of the Atlantic was for a long time considered quite safe for U-boats:

The Mid-Atlantic gap was an area outside the cover by land-based aircraft, those limits are shown with black arcs (map shows the gap in 1941). Bluish dots show destroyed ships of the Allies
One patrol of a particular boat was found completely documented with accompanying pictures:
Taken during the summer of 1942 by an onboard war correspondent, the photographs show a U-boat in action within the Atlantic and Caribbean, as the German submarine service teetered on the brink of what was, with hindsight, the unstoppable downward slide into defeat. However, at the stage of the war at which they were taken, U-boats could still spend time surfaced without fear of Allied air attack within the mid-Atlantic and were raking a harvest of considerable numbers of Allied merchant ships. (p10.)
As evidenced from the above quote leisure activities, fresh air, and keeping fit and stretched was essential and often possible in the first half of the war in the Atlantic.
Diving from the bow of U564, Gabler (top left, in the foreground) is dressed in attire known in German naval terminology as a 'peeled banana'. Even Suhren (above and left) took a dip in the warm Atlantic – one of the rare occasions when he divested himself of his red scarf. The Kriegsmarine also developed salt-water soap (below left), used by Teddy and his crew as Webendorfer sprays them with water piped from the engine room. Opinion remains divided over the merits of this soap: Suhren thought it wonderful; others were less than enthusiastic about the waxy residue
that it sometimes left on the skin. (p102–103.)
[…] the cramped, humid interior of a submarine often played havoc with man's physical well-being. A ritual observed by both Suhren and Gabler while within the 'Atlantic Gap' was the morning walk (above). Several lengths of the stern deck every day helped keep them fit, particularly Suhren, who suffered back pain from long stationary periods on the bridge. Moreover, the momentary ability to find solitude allowed the two men to talk without being overheard. (p164.)
On 1 September 1942, all available crew were drawn up on the stern deck for the surprise announcement of Teddy's award of the Swords to his Knight's Cross and Oak Leaves, as well as his promotion to Korvettenkapitän. (p 173.)
Pictures and quotes from: Lawrence Paterson: "U-Boat War Patrol: The Hidden Photographic Diary of U-564", Chatham: London, 2004.
Up until 1943, things were going well. The fate of this boat, German submarine U-564 shows that crash dive was not always hampered by crew on deck:
An Armstrong Whitworth Whitley sighted the two U-boats in the Bay of Biscay the following day and shadowed them. U-564 was unable to dive due to the damage already sustained. By 16.45 hours the Whitley was running low on fuel and attacked U-564. The two U-boats damaged their attacker with anti-aircraft fire but the aircraft's depth charges fatally damaged U-564 and she sank at 17:30 hours. The damaged Whitley was forced to ditch, where a French trawler rescued the crew. There were 18 survivors from U-564 including the commander. U-185 picked them up and transferred them to the German destroyer Z24 two hours later.