They lived in shanty towns and makeshift shelters until the government built apartment blocks. In many cases families lived in shacks or even tents for years. When I was a young boy I was friends with two girls whose mother had come from Germany and had been a little girl during the war. Once they showed me a handful of photographs her mother had saved. One of the photographs was of her home, which was a corrugated tin shack in a railroad yard. The dimensions of the shack were about 8 feet by 8 feet by 6 feet tall. This was where she and her mother lived. Her father was killed in the war and her grandfather had been a surgeon, but also died by that time.
Many Germans and other refugees lived in concentration camps run by the allies which were called "displaced persons" camps or "DP camps". In some cases the DP camps were Nazi concentration camps that had simply been taken over and reused by the allies. For example, the notorious concentration camp at Dachau was used as a DP camp after the war. By 1946 there were over 2,000 camps in Germany housing refugees, all run by allied soldiers.
There were so many homeless people that those who could be housed in concentration camps were the lucky ones. Many, like my friend's mother, lived in shacks that they hid away wherever they could. Millions of the less fortunate who were captured in allied dragnets were at first kept in open fields called "Rheinwiesenlager". These were large grass fields of thousands of acres surrounded with berms and guarded by soldiers with machine guns. After a few months the allies opened these camps and released millions of completely destitute people to shift however they could, or be funnelled to DP camps.
People took shelter in basements, old military bunkers, pillboxes, and even just caves made out of rubble. The only regular food were bread rations given out by the US Army. In 1946, the ration was 1275 calories, and to get this you had to sign up, be interrogated and risk being arrested for "war crimes" or be put into a "de-nazification" prison.
There are very few photographs from occupied Germany because it was illegal for Germans to own cameras and similar regulations were enforced to prevent allied soldiers owning cameras or from taking pictures. It was also illegal to take aerial photographs of occupied Germany.
This situation only persisted until 1948. In 1949, with the founding of the Republic of Germany the new government began large, subsidized housing programs, which included both apartment blocks and single-family homes. Even with these programs Germany has a relatively low rate of home ownership today and most Germans rent their dwellings. Two photographs below show the progression. The first, circa 1949, shows Quonset huts in Hamburg. Notice the large rectangular bomb shelter in the center that survived the bombing. The second photo from the 1950s shows the same view after government-subsized housing had been constructed.
Hamburg before and after government subsized housing program. Notice the gigantic square bomb shelter has remained throughout.