There were no publishers, no royalties, and no copyright.
All these things were invented after the spread of the printer press.
If you are a scientist/philosopher, you would write your book yourself, or hire a scribe if you are rich enough. Then you will send it to a friend, and/or read to your students. Almost all books in mathematics and astronomy begin with something like "Dear Aristodem, I promised to explain you..."
as private letters. Your friend is probably also a philosopher, he has some students. He will tell them about the book and some of them will want to copy it. If the demand is large and the students are willing to pay to save their time, they will hire a scribe. As the rumor will spread more people will ask for copies. In any case you will never be payed for writing a book. Or you address your book to a ruler, and send to him. He may ask someone to read him aloud. If he likes it, he will order some copies. This custom: dedicating books to rich and powerful sponsors persisted till the early modern age, by the way. And they rewarded the author. This was what happened with new books.
Unlike the works in science and mathematics, works of poetry, tragedy and history were performed in public. This made some authors very famous, and people were looking for their works to copy.
There were also libraries. Some very rich people collected books. They would search for the manuscripts, borrow them and have their own staff of copiers. The most famous library was maintained by the first Ptolemy's, Greek rulers of Egypt. But several other large libraries are known. There is a famous story that they asked the Government of Athens to give them the originals of Aeschylus, Sophocles and Euripides plays, which were probably preserved in some government storage. Athenians demanded an enormous deposit. Ptolemy payed the deposit, and did not return the manuscripts. (Probably sending copies back to Athens). They also enforced a law that all ships arriving to Alexandria must declare all books on board. The books were copied and returned. As I understand the ruler payed only for copying, to the person who copied.
All such information suggests that there was no market in the modern sense. If you wanted to have a book, you will find someone who owns it, and either copy it yourself or pay to someone to copy it.
By the way, I suppose that theater performances were free. They were sponsored either by the city or by rich people. The city also arranged competitions of the
theater plays, and winners were awarded prizes. I suppose this was their only income from the plays.
This describes the situation in ancient Greece and early Hellenistic times.
In Greece proper (before Alexander's conquests) the main kinds of literature
(besides science, history and philosophy) were theater plays and verses. Verses were simply copied by people who liked them.
In the late Hellenistic times, it is possible that some commercial writing begins.
We have the first novels from that time, and it is not clear whether the authors could make any profit of them. This is also the time when commercial art workshops appear, making paintings and statues for sale.
EDIT. The thing like a "professional writer", a person who lives on honorariums and royalties becomes regular only in the later 19th century. But of course, information on the ancient authors that we have is very incomplete. In most cases nothing is known about the person except his surviving books themselves.