In the Wikipedia article on Japanese architecture, it says in the Heian era, and I quote, "Heavy materials like stone, mortar and clay were abandoned as building elements, with simple wooden walls, floors and partitions becoming prevalent." As someone mentioned, it doesn't give a clear reason for the discontinuation of the use of such materials.
What did Japanese architecture look like with those "heavy materials?" I tried searching things like Jomon and Yayoi architecture, but the buildings look like they're made of wood and straw, and Asuka and Nara architecture looks less obviously wooden, but in the study.com article I will link to, quote, "Most buildings of (the Nara) period were made out of wood, usually painted in red, black and golden colors."
Edit: I came across a series of history PDFs by the Japan Lime Association. They're in Japanese, but they could be used as a partial answer to the question of non-wooden architecture in general.