In Jewish custom, all calendar questions were decided by the court (Synedrion).
It was a duty of everyone who spotted the new Moon to report to this court as soon as possible.
But of course, the new moon is not always visible, the sky can be covered by clouds for example, so the Synedrion decided when
to start the new month, based on all available evidence, including the knowledge of the average length of the month which they calculated based on previous observations. I take this information from the book of
Shlomo Sternberg, Celestial mechanics, vol. I, Benjamin Inc., NY 1969,
and he refers to
Maimonides Code, Ch. XVIII, sections 5-9, "Laws of the sanctification of the new moon". Sternberg refers to a Hebrew edition, and I am not sure whether an
English translation exists.
EDIT. Much more detail can be found in the article by N. I. Idelson, History of the calendar, (Russian, 1925, reprinted in his book Essays in the history of celestial mechanics, Moscow, Nauka, 1975, also in Russian). I cite (my translation):
It was a duty of Synedrion in Jerusalem to determine the
beginning of the month. It formed a special committee for this purpose. This committee
had to perform its own observation and hear the evidence of vitnesses, there had to be at least two such vitnesses, and their testimonies had to agree. Moreover,
it was a duty of every Jew, even on Saturday, to go to Jerusalem to give a testimony. After the decision was made by Synedrion, it was recorded, and
the chief judge sanctified the new month with a prayer. Then they began to inform the population: for this a system of fire signals was used, they were lit on the hilltops. Later they started to send messengers from Jerusalem to other cities.
In the case of doubt, the principal feasts (which were all tied to the lunar calendar) were celebrated for two days.
Actually there is an English translation of Maimonides:
https://books.google.com/books/about/Sanctification_of_the_new_moon.html?id=jq8rAAAAIAAJ
See also this paper which discusses the technical details:
http://u.cs.biu.ac.il/~belenka/mimo.pdf
EDIT2. There is no evidence that Babylonians (or Jews, or anyone else before the Greeks) tried to answer the question "how the Moon REALLY moves?" They only addressed VISIBLE phenomena. And the conjunction of Sun with Moon is never visible, except at the time of a solar eclipse. Geometric models of celestial motion are Greek invention. Early attempts do construct such space models did not survive to our time, almost all we know about Greek astronomy comes from
Ptolemy. Theoretical constructions of Babylonians also did not survive. All we have are numerical tables of visible phenomena from which we try to reconstruct the method of composing such tables. The average length of the Lunar month
was known to the Babylonians with high precision, and this number (29 days 12 hours 44 1/3 minutes) was incorporated into Jewish calendar in 499 bc. But sometimes the lunar month is 29 days and sometimes 30, and the ancients were very far from being able to explain the pattern.
