Short Answer: Cricket was surprisingly popular in the United States through the entire 19th century. However, baseball was backed and promoted by dynamic marketers like A.G. Spalding. Baseball came to be associated with all-American manly athleticism, while cricket came to be associated with snobbish aristocrats with English pretensions.
Sociologists Jason Kaufman and Orlando Patterson have an article that answers this exact question, so I'll let them explain.
Baseball and cricket were both very popular in the 19th century
Cricket was popular in the US until well after the Civil War. The world's first official international cricket match took place between American and Canadian
teams in 1844.
While the increasing popularity of baseball
did present a formidable challenge to American cricket, the two games
existed comfortably side-by-side throughout the 1850s and 60s. It was
not uncommon, in fact, for cricket and baseball teams to challenge one
another to matches in their rival’s sport.
The wealthy began to value Cricket for its English pedigree
Though cricket was originally popularized in the United States by working-class immigrants from the British Isles, it later became a sport practiced by only a select few Americans . . . The most distinctive feature of the history of cricket in both the United States and Canada is its elevation to a pastime for elites only.
Savvy marketers promoted baseball
However much Americans enjoyed playing cricket, the sport never developed the kind of infrastructure that leads to a mass fan base: frequent matches with large crowds and intense rivalries. Baseball, however
. . . was later blessed by a cadre of
brilliant entrepreneurs determined to make it the “nation’s pastime.”
One such person was A. G. Spalding, star player, manager, league
organizer, and sports manufacturer. To call Spalding an impresario or
a marketing genius would be a bit of an understatement. He engaged in
every part of the game, from promoting star players and intercity
rivalries to squelching nascent efforts at labor organization among
players.
Savvy marketers disparaged cricket as effeminate
Spalding was pretty scathing in his depiction of cricketers. He wrote in 1911:
I have declared that Cricket is a genteel game. It
is. Our British Cricketer, having finished his day’s labor at noon,
may don his negligee shirt, his white trousers, his gorgeous hosiery
and his canvas shoes, and sally forth to the field of sport, with his
sweetheart on one arm and his Cricket bat under the other, knowing
that he may engage in his national pastime without soiling his linen
or neglecting his lady
A baseball player, on the other hand, is manly:
When he dons his Base Ball
suit, he says good-bye to society, doffs his gentility, and
becomes—just a Ball Player! He knows that his business now is to play
ball, and that first of all he is expected to attend to business. . .
And just in case Spalding was too subtle above, he repeats one more time that America:England::Manly:Effeminate.
Cricket is a gentle pastime. Base Ball is War! Cricket is an Athletic
Sociable, played and applauded in a conventional, decorous and
English manner. Base Ball is an Athletic Turmoil, played and applauded
in an unconventional, enthusiastic and American manner.
The end result? "By the eve of the First World War very few were still alive who could recall the days when cricket had a chance to become America’s national pastime.”