Typically it refers to a river, and "lower" is down-river (closer to the coast), while "upper" is up-river (further inland). This is because the land at the mouth of a river is lower (in altitude) than the land near its source. Basic physics here.
This goes for most instances where you see an upper/lower distinction. For your examples, Egypt is based on position on the Nile (which flows northward into the Mediterranean) and the other two both used position on the Danube (which flows eastward into the Black sea) as their directional indicator.
This principle sometimes get applied in unexpected places. For example High German (or "upper German") refers to the dialects spoken in the interior of the European continent, while Low German is the dialects spoken nearer the coast. It is not (as many assume) any commentary on the perceived quality of the dialects themselves*.

* - unlike the terms high church and low church, and many other English high/low couplets relating to social things, which are totally a commentary on their perceived quality.