In LINGVA LATINA A Companion to Familia Romana Second Edition by Jeanne Marie Neumann, an unofficial companion to the Lingva Latina course, she states that:
When a master manumitted … a slave, that slave became a freedman, or lībertus/līberta. Although no longer part of the familia,… (emphases mine)
I suspect this is wrong, and indeed seem to remember reading that the Familia Caesaris specifically included his freedmen, whom freeborn citizens both despised and feared.
Which is correct?
Short Answer
How true Jeanne Marie Neumann's statement is depends on, firstly, the definition of familia used and, secondly, the circumstances of individual freedmen and freedwomen. As one classicist has noted,
the Roman family is an ambiguous concept and defies easy definition.
Thus, without some qualification, Neumann is over-generalizing; there were clearly cases where freed slaves remained part of the familia (especially those who physically remained within the household) but also cases where they had a less familiar relationship and were clearly no longer part of even a broadly defined familia. Also, as @Mark Olson points out in a comment, we're covering a very long time period and the meaning of familia did not remain constant over time. What follows below primarily covers the period the late republic to (at least) the time of Ulpian (died 220s AD).
Details
One problem with her statement lies in the multiple possible definitions of the Roman term familia. On the one hand, you quite rightly mention Caesar familiaris (there's a whole book on this by P. R. C. Weaver, Familia Caesaris. A Social Study of the Emperor's Freedmen and Slaves.). On the other hand, the most common definition of familia encompassed those in the household; this would certainly include family members and slaves under the authority of the paterfamilias, but it could also include freedmen and freedwomen depending on (1) the conditions of their manumission and (2) whether or not they continued to work as a part of the household after being freed.
On the definition of familia,
‘Household’ is the usual English translation of Latin familia, a term to which the jurist Ulpian (Dig. 50. 16. 195. 1–5), understanding its application to both property and persons, assigned several meanings: the physical household; the persons comprising a household (e.g. patron and freedman); a body of persons united by a common legal tie such as all kin subject to a living *paterfamilias, or a body more loosely connected such as all agnatically related kin; a body of slaves, or slaves and sons; and all blood descendants of an original family founder.
(my highlighting)
These varying definitions are further complicated by the relationship between the former slave owner and the freed slave and, if we extend your quoted text, we can see that Neumann does not take account of the conditions of manumission nor freedman/woman's life after manumission:
Although no longer part of the familia, the libertus [freedman] now belongs among his former master's dependents or clients (clientēs) and still has obligations to his former master.
However, as the classicist Robert Knapp observes,
The ongoing relationship of a freedman to his patron ranged widely. There might be none at all (if the patron was dead, or if payment for freedom had severed all important ties), or a very close one if the freedman remained physically in the patron’s household.
Source: Robert Knapp, 'Invisible Romans' (2013)
Classicist K. R. Bradley makes a similar observation in Discovering the Roman family : studies in Roman social history (1991), observing that keeping freed slaves within the familia was particularly common among wealthy Romans.
Citing Roman sources helps but they do not necessarily take account of varying circumstances. For example, the Roman jurist Ulpian (died 223 or 228 AD) stated:
The freedman was in no sense independent of his previous owner. His obligations were both social and economic: it is to be noted that obligations of social respect (obsequium) assimilate the freedman’s relationship to his patron to that of a son to his father.
The figure of father and patron ought always to be respected and sacred in the eyes of a freedman or a son.
Source: H. Dessau, 'Inscriptiones Latinae Selectae', cited in Thomas Wiedemann, 'Greek and Roman Slavery'
In practice, some freedmen had obligations similar to or greater than that of the paterfamilias' son, others had much greater freedom. Thus, it is hard to generalize (as Neumann seems to have done). Knapp adds that,
Circumstances varied depending on whether the freedman remained in the patron’s household or set up in his own home and establishment. In the household, the freed slave would receive food and lodging, but would lack the freedom of action inherent in living on his own.
Evidence for freedmen as part of a familia which included the paterfamilias' children and slaves comes from inscriptions:
The position of freedmen as members of their master’s household (familia) just like his children and his slaves, is illustrated by large numbers of inscriptions giving them the right to be buried in the family tomb…
Cited in Wiedemann
One such inscription reads:
Sabbio, slave of our Emperor, manager of the Claudian Aqueduct, made this for himself and for his wife Fabia Verecunda, with whom he lived most faithfully for twenty-four years, and for his freedmen and freedwomen, and for their under-slaves (vicarii), and for all their descendants.
Although this practice of sharing tombs with freedmen and slaves became less common (for practical and social reasons) among the wealthy with a large familia during the late republic, it continued among lower class familia. Thus, the practice was by no means universal.
Keeping freedmen and freedwomen as a part of the familia could benefit both the paterfamilias and the former slave. In addition to the prestige of a large familia, there were practical benefits. Note this example of the Emperor Augustus' wife Livia:
Livia's household staff provided many services that were available not simply to the owner and her immediate family but to the slaves (and freedmen and freedwomen) who made up the familia as well: the cooks, caterers and bakers, fullers, wool-weighers, clothesmenders, weavers and shoe-makers, nurses, pedagogues, midwives and doctors – these were all functionaries whose labour contributed to the material well-being of the familia as a whole.
Source: Keith Bradley, 'Slavery and Society at Rome'
The familia could also include musicians, for example those of Piso and Mark Anthony, who could be either slaves or freedmen. Another way for a freedwoman (or even a freedman) to remain part of the familia was through marriage; frowned upon by some of the elite it may have been, but it did occur.
It was common practice for a freedman to adopt his former master's nomen and praenomen, one of the more famous examples being Marcus Tullius Tiro, freedman of Marcus Tullius Cicero and part of Cicero's familia.
