Ptolemy Egypt was ultimately Hellenistic in origin and fell into the wars of succession after Alexander the great died. Ptolemy III was the height of their reign, after that the Ptolemies faced several issues...infighting and succession being the largest. Their challenges:
1) War with Seleucid. There were several conflicts (6 in total https://en.wikipedia.org/wiki/Syrian_Wars), including a pretty full capitulation of Egypt and a puppet monarchy
2) Egypt was not united. Under Ptolemy, Egypt lost direct rule over 'upper Egypt' (Nile upstream, south in direction). The battle of Raphia included the mass rearming of the Egyptian population and that would lead to revolt. https://en.wikipedia.org/wiki/Hugronaphor was the first and https://en.wikipedia.org/wiki/Ankhmakis Ankmakis was the final ruler there as the Ptolemy's were eventually able to crush the rebellion.
3) Inbreeding. One of the Egyptian traditions the Ptolemy reign saw was a return to inbreeding (King Tut's rule saw heavy inbreeding) and Ptolemy II picked this up again. He earned the nickname "Philadelphus" to describe his marriage to his sister Arsinoe II. This trend continued with Ptolemy IV and his marriage to hi sister Arsinoe III (This was done in part to keep the Macedonian bloodline separate from the native Egyptian population, though it's can be speculated that it was also done to show Egyptians that the Ptolemy's were true Egyptian rulers).
Edit
I can note that Ptolemy III was married to Berenice of Cyrene (think modern Libya), which was his cousin (not sister, somewhat better?). As much as we can point to Ptolemy III as the height of Hellenistic Egypt, it should be noted that Berenice was a very strong ruler herself, potentially involved in battles and a renowned equestrian (apparently participating in and winning in Olympic games). She was originally engaged to Demetrius the Fair, but had him murdered after she found him with her mother. She was strong enough that after her husband Ptolemy III passed away, her own child king Ptolemy IV had her killed so she could not claim the throne herself.
Ptolemy V (or Ptolemy Epiphanes as he was called) was the son of a brother-sister wedding. He was married to Cleopatra I of Syria (named Syrian, but traced her origins through Seleucid, ultimately Greek), a wedding arranged with the Seleucid empire that held peace until her death. The Cleopatra name would be passed to many of her female descendants. From this point forward, no new genetic material would enter the Ptolemy bloodline (Ptolemy VI to X could trace their lineage back to these 2 and nobody else).
/Edit
4) A messy mix of politics and intrigue. Ptolemy VI was defeated in the wars with the Seleucids and a puppet monarchy was installed in Egypt under Ptolemy VI. Alexandria's population rejected this and adopted Ptolemy VIII as their leader (Ptolemy VI's younger brother was Ptolemy VIII, though known as Physcon (the fat) as he wasn't considered part of the Ptolemy reign yet) . Rome intervened (threats only) and the Seleucid withdrew, leaving a triumvirate of Ptolemy VI, Physcon, and Cleopatra the II as rulers (All three siblings, Cleopatra being married to her brother, Ptolemy VI, at this time).
Ptolemy VI tried two separate assassination attempts on his younger brother Physcon, both of which failed but ended in Physcons capture. Instead of killing his brother, he arranged Physcons marriage to his daughter, Cleopatra Thea (uncle marrying niece here). Questionable if that marriage ever happened as Ptolemy VI died on campaign soon after. Cleopatra II had her son proclaimed Ptolemy VII, however Physcon returned and married Cleopatra II (his sister and brothers ex-wife). Physcon then had his wife's son Ptolemy VII (Physcon's nephew) killed and assumed the title Ptolemy VIII. It's a bit confusing here, Ptolemy VII was Ptolemy VI's son, and Ptolemy VIII was Ptolemy's VII's uncle and Ptolemy VI's brother. To further confuse this, Ptolemy VIII would marry Ptolemy VI and Cleopatra II's daughter, Cleopatra III (To clarify, Ptolemy VIII married his sister Cleopatra II after his brothers/husband of Cleopatra II's death. When Cleopatra II's daughter Cleopatra III from Ptolemy VI was of age, Ptolemy VIII married her (his niece/step child) while still married to her mother. THe intrigue here was Ptolemy VIII never wanted to marry Cleopatra II, but did so to solidify his claim all the while intending to marry Cleopatra III once she became of age). I hope I got all that right as it's a silly web that gets a bit worse...Ptolemy VIII's children with Cleopatra III were able to say their dad was grandma's brother and their cousins were also their half-siblings.
Ptolemy VIII (Physcon) would extract some revenge upon Alexandria and her intellectual elite, killing many and expelling more, and permanently altering Alexandria's status in the world. Around 10 years later, this culminated in a riot in Alexandria that saw the Royal palace burn and Ptolemy VIII, cleopatra III, and their children fled. Cleopatra II turned to her son Ptolemy Memphites as the new Ptolemy ruler (I'm a bit unsure on lineage here, but I believe Ptolemy Memphites was the son of Cleopatra II and Physcon/Ptolemy VIII), however Ptolemy VIII/Physcon got his hands on his son Ptolemy Memphites and sent his dismembered body back to his sister/wife Cleopatra II. There would be a short civil war between Alexandria (supporting Cleopatra II after Physcons attack on the intellectuals of Alexandria) vs the rest of Egypt that viewed Physcon as the proper ruler. Cleopatra II would eventually flee Egypt, leaving the rule to her brother Ptolemy VIII and her daughter Cleopatra III. When Ptolemy VIII died, he left the rule to either of his sons (didn't care, Cleopatra III was to choose)...she chose her youngest but the people of Alexandria would want her oldest to become Ptolemy IX. Ptolemy IX would marry his sister Cleopatra IV, but Cleopatra III would push her daughter Cleopatra IV out of the picture and replace her with her other daughter Cleopatra Selene. Cleopatra III (probably the epitome of the absurd political intrigue) would then expel her son Ptolemy IX and replace him with her younger son she had supported earlier as Ptolemy X (Ptolemy IX would return twice though with the crown passing from Ptolemy IX to X a few times, all civil war). Ptolemy X would eventually have his mother (Cleopatra III) killed, putting an end to her involvement in Egypt. Ptolemy X would eventually be chased out of Egypt by the people of Alexandria.
If I have it right...Ptolemy IX's child had 1 grandma/grandpa, and the great grandpa / great great grandpa was the same person. I'm not 100% sure on the effects of inbreeding, but the Ptolemy family tree was a pretty straight line by this point.
** Just a side note, the Ptolemy's never referred to themselves as Ptolemy VI or Ptolemy VIII. It was Ptolemy Philometor and Ptolemy Physcon. The numbering of Ptolemies was a much later invention by historians.
So with all that said:
Did all of Egypt, or just the Ptolemies decline?
Both. Wars of succession take it's toll on a people, and once a people are no longer united, an empire and it's people decline. Much of Ptolemy VI rule and his successors can be defined as civil war between Alexandria and the rest of Egypt and that took a heavy toll on Egypt. The slaughter of Alexandria's intellectual elite was a huge hit and dropped Alexandria from what is arguably the most enlightened city of it's time.
