Latitude can be calculated from observations of stellar objects (typically using something like an astrolabe) and a bit of math. The Greeks could do this as early as 150BC, but only on dry land. The Mariner's Astrolabe wasn't invented until around 1300 CE.
Nobody had a good way to determinte longitude in realtime aboard a ship before the invention of the marine chronometer in the early 1700's. The closest anyone came was the Chinese, who managed to work out the longitude of various places on the Indian trade routes in 1421 by placing observers on said places to observe various lunar and stellar positions simultaniously. This information may have made their maps better, but wasn't particularly useful to a navigator out of sight of land.
Before that, the typical technique used was dead reckoning, which was incredibly inaccurate. Basically, the navigator would chuck a hunk of wood out the back of the ship, try to estimate their speed based on their relative speed to the jetsam, and try to calculate their distance from the last time they did that based on that speed. Obviously this doesn't take currents into account at all, and any errors are likely to accumulate every time you do it.
What was typically done in the Medeteranian in ancient times was that navigators just kept themselves in sight of land. Even then, bad things could happen. For instance, the Odyssey is essentially a story of an ancient Greek who got blown off course sailing home from nearby Anatolia, and spent 10 years trying to find his way home.
