The region to east of Bengal is Myanmar. In between them, the Himalayan foothills taper off rather abruptly into the Bay of Bengal. They still form quite a formidable obstacle. Southeast Asia itself is an isolated place with rugged terrain. I'm not an expert, but it looks like pre-modern civilizations there were located along waterways, and depended heavily on them for mobility.
Myanmar primarily consists of the Irrawady Valley, a large river valley emptying into the Bay of Bengal. It was the first place in Southeast Asia to be significantly connected to the outside world because it was located along the trade route from East India to China. Burmese people migrated here around 200 B.C., until being conquered by another Burmese people around 800 A.D. Both of these people were located on the Tibetan Plateau before migrating into the region from the north.
The first people, the Pyu, began the earliest known civilization in the Irrawady Valley, c. 200 B.C.-800 A.D. As I already mentioned, it was based along the trade route from East India to China. The Pyu were described by 8th century Chinese authors as being monastic and unwarlike. Based on this description, it can be assumed that the region was still isolated and idealic in the 8th century.
The Khmer Empire (802-1452) was the high point of pre-modern Civilization in Southeast Asia. Cambodian Legend suggests that a dynasty of Hindu Kambojas were the founders the Khmer Empire. Kamboja is also the name of a Scythian people in Northwest India who received mention in the Mahabharata. The Sdok-Kok-Thom inscription, written in both Sanskrit and Khmer, describes the founding of Khmer by the Kambojas, and the lives of the first twelve Khmer Kings. The modern name of Cambodia is derived from the Sanskrit Kapuchea, which means the "land of the Kambojas".
There is a saying that "where goods can go, so can armies". The inverse of it is probably your answer. Central Asia (including Northern India) and Southeast Asia are only connected by maritime routes. An invasion of Southeast Asia from India would have had to have been by sea. South India, on the other hand, was adjacent. It was also a lucrative region to invade, being the global entrepot of luxury goods.
Khmer Empire, Formation and Growth