I see two different questions: why are there so few promotions at the top, and why is the attrition rate so much higher at the top?
To the first, I would blame a top-heavy force, i.e. there is a very high ratio of flag officers to troops, particularly at the very top. Promotions to general staff tend to come with appointment to commands, which are political, and few and far between to begin with. "Star creep" or "rank inflation" mean that a large number of generals and admirals are occupying the top of the heap, leaving little room for advancement. After all, why would you leave? It's an achievement and responsibility which has been compared to that of a Fortune 500 senior executive, and though your compensation will be at least an order of magnitude smaller, it's not a bad lifestyle. Gates' 2010 report is not the first time the brass creep has been raised, nor is the U.S. the only country where it is a concern, but it is a significant factor, particularly since the end of the Cold War as troop numbers and the number of commands have been shrinking.
To the second I attribute a variety of factors, all of which ultimately point to opportunity cost. As the chances for a promotion shrink, the burdens of military life become progressively more onerous compared to opportunities in the civilian world. Consider that
- The requirements for promotion become more onerous at higher ranks.
- The pyramid gets narrower at the top. Officers passed over for promotion once are unlikely to get a second recommendation; those passed over twice are subject to discharge or retirement.
- Perhaps as importantly as anything else, O-3 comes relatively soon after one's service commitment ends, and one earns a military pension during the long wait at O-4, but beyond that the incentives are few.
There are Army regulations governing when promotions are awarded to whom. For example, there is a minimum Time In Grade (TIG) of several years, depending on current rank. There are educational guidelines as well; for example, to be promoted to captain, a first lieutenant must have earned a bachelor's degree or its equivalent, and to be promoted to major, he or she must have advanced education. Not every soldier will meet such requirements; a mediocre physical fitness report can get you passed over.
Battlefield promotions of the sort seen in films are exceptionally rare. There is a service-wide promotions board which reviews all officers recommended for a promotion. But while promotions at lower ranks are almost perfunctory, candidates at higher ranks receive much greater scrutiny. The regulation even states that majors or lieutenant colonels passed over twice for promotion are retained essentially at the pleasure of the Army, and would otherwise be directed to retirement or discharge. Moreover, the board is working for the good of the Army, not necessarily your career, and your timing could be off. Only 36% of eligible lieutenant colonels were promoted to colonel in 2011, whereas 91% were five years previously.
Timing is influential in another way. To become an officer in the U.S. armed forces, one is either graduating from the academy or an ROTC program or is completing an officer training program probably following some time as an enlistee. This will entail at least a 3-5 year active duty service obligation, and a concurrent 8-year total military service obligation, which can be even longer in effect in certain occupations (e.g. for example, naval aviators' commitment clock starts running only after they complete their flight training).
So if the median O-1 (Army second lieutenant, Navy ensign) is 22 or 23, a large percentage of them will make it to O-2 (Army first lieutenant, Navy lieutenant junior grade) if only because they are required to remain in the military for at least as long as it takes to make that grade, and the requirements for that grade are relatively low. From there it isn't much time to O-3 (Army captain, Navy lieutenant, etc.). But by age 30, many will have tired of military life— the bureaucracy, the moving, the danger of getting blown up— and many will leave, particularly those with families.
Those who remain after that have a strong incentive to serve out their 20 years. Once an officer (or enlisted Non-Commissioned Officers (NCOs)) makes it to the 20 year mark, they are eligible to retire. Under the old system, a military pension was based on 50% of the base pay averaged over the "high three" years. Many US Army officers and NCOs decide at the 20 year mark to retire because the compensation and lifestyle of a private sector job begins to look very appealing. There is also a saying in the military that after 20 years you are working for 1/2 pay, because one could at that point retire, collect 1/2 their base pay, and then go on to also earn a very comfortable salary in the civilian world. A newer retirement system referred to as a "blended retirement" began in 2017. Many consider it less generous than the old 50% pension system. You have two decades to figure out how to apply the contacts and management skills you have acquired in a new career as a civilian.