13

Cirillo and Taleb [1] used 72 epidemics in the last 2,500 years with more than 1,000 fatalities to show that the distribution of fatalities is fat-tailed. When confronted that their study, being based on 72 most-noticed epidemics, would suffer from selection bias by Ioannidis et al [2], Taleb et al [3] defend by saying that Cirillo and Taleb's "dataset likely contains most (if not all) of them." However, Ioannidis et al in [2] say that "Thousands of outbreaks might have exceeded 1,000 fatalities."

If we believe Taleb, the total number of epidemics in the last 2,500 years with a death count of >1,000 should be in the lower hundreds. So if anyone is able to help me find sources which can test this hypothesis (by showing if they are actually much more than say 200), we can know who is right.

[1] https://doi.org/10.1038/s41567-020-0921-x

[2] https://forecasters.org/blog/2020/06/14/forecasting-for-covid-19-has-failed/

[3] https://forecasters.org/wp-content/uploads/Talebetal_25062020.pdf

11
  • 7
    I would have to dig properly into Cirillo's sources to see how their definition works, but as a counterpoint, McNeil's 'Plagues and Peoples' (1976) has a list of epidemics recorded by Chinese sources in an appendix; at a quick glance it looks to have 200+ entries. (It is a very simple list, though - 1608, epidemic in Yunnan, etc, only occasional mention of death rates) Commented Oct 5, 2022 at 16:49
  • 2
    @Andrew thanks for introducing me to McNeil's work. I just had a brief look at the list provided in the appendix and even though most of them don't have a death count provided, those which have are >1,000. Regarding Cirillo's sources, look at Table 1. He has mostly used the Wikipedia page on the list of epidemics. Commented Oct 5, 2022 at 17:31
  • 1
    Logically you'd think Taleb et al probably have the right of it, as epidemics feed off of large crowds of humans and the historical human population of the Earth is also quite ... er ... "fat tailed". But of course testing our logic with data is what science is all about.
    – T.E.D.
    Commented Oct 5, 2022 at 17:57
  • 6
    Of course, the 'default position' should be again Ioannidis'? But how is this 'defined`? How many epidemics went on in the Americas after Europeans set sail? Is this just one big measles, one big smallpox epidemic? I'd say 'no, quite a few'? This alone stains Taleb's assumptions to the extreme? How are deaths attributed & counted? A 'bad flu season' alone seems to have accounted for more than 72 _epi_demics with +1000 (real!) fatalities, but SARIs almost never went into such hyperfocus (Russian Flu, Spanish Flu-like exceptions), as those usually 'cull' the infirm & old? Commented Oct 5, 2022 at 21:23
  • 1
    @ItalianPhilosophers4Monica One may of course just look up "fat tailed", but for all those readers here that do not know this terminus technicus from statistics, it would indeed increase the attractiveness of this post if you'd include a (very short) description/explanation of this concept? (Perhaps at least a link to en.wikipedia.org/wiki/Heavy-tailed_distribution or sub pages?) Commented Oct 7, 2022 at 7:12

2 Answers 2

25

The key quotes in full, I think, are that Ioannidis et al critique it with:

Examining 72 major epidemics recorded through history, they demonstrate a fat-tailed mortality impact. However, they analyze only the 72 most noticed outbreaks, a sample with astounding selection bias. The most famous outbreaks in human history are preferentially selected from the extreme tail of the distribution of all outbreaks. Tens of millions of outbreaks with a couple deaths must have happened throughout time. Probably hundreds of thousands might have claimed dozens of fatalities. Thousands of outbreaks might have exceeded 1,000 fatalities. Most eluded the historical record. The four garden variety coronaviruses may be causing such outbreaks every year [15,16]. One of them, OC43 seems to have been introduced in humans as recently as 1890, probably causing a “bad influenza year” with over a million deaths [17].

Taleb et al defend this with:

Finally, when the authors in [15] state that "Tens of millions of outbreaks with a couple deaths must have happened throughout time," to support their selection bias claim against 1, they seem to ignore the fact that the analysis deals with pandemics and not with a single sternutation. The class of events under considerations in 1 is precisely defined as "pandemics with fatalities in excess of 1K," and their dataset likely contains most (if not all) of them. Worrying about many false alarms in the tail of the distribution of pandemic fatalities is thus misplaced.

Unfortunately, neither of these is amazingly helpful. Ioannidis's citations do not explicitly say "here's some estimates for how many people these mundane illnesses are killing". On the other hand, Taleb's rebuttal seems to just say "we're not worrying about all the tiny outbreaks, just the big ones", and does not address the key issue of whether that list of 72 is comprehensive.

Instinctively, I have some doubts. Cirillo's list is (as noted in comments) heavily sourced from Wikipedia's list of epidemics; I make it 55/72 (56 with one dual-cited). Ten more are cited to a page on "ListFist", which itself seems to be citing back to the same Wikipedia list. and then there are a couple of sources used once or twice. The WP list as of March 2020 is interesting - comparing it to the one in the article, it looks like they have mostly just skipped ones with no quoted death toll. For example, in the "15th century and earlier" section, WP includes an unknown death rate plague in 412 BC, and in 746 AD, neither of which are shown here (though they do include the 664 British plague, which WP had a blank death toll for).

So, some immediate problems appear. Firstly, the Wikipedia article does not claim to be comprehensive (later versions do misleadingly say "a list of the largest", but this is not in the 2020 version), but they seem to have assumed it was.

Secondly, it is trivial to follow the sources in Wikipedia and find a death toll well in excess of 1000 for some of those "blank" ones. For example, the cited source for the 1636 English plague gives a reasonably precise figure of 10,400 in its introductory paragraph, and mentions other outbreaks in 1603 and 1625 - 1603 but not 1625 is mentioned in Wikipedia, neither in Cirillo. It turns out 1625 killed 26,350 people per a footnote in that article; 1603 "over thirty thousand". We've just extended the list by 4%, just with one country in a 33-year period.

Thirdly, just looking at Cirillo's list, the distribution is striking. Up to the mid-sixth-century AD, they are all in the Mediterranean world; 562, 627, 688 are in Mesopotamia; 664 in Britain; 735 is the first in Asia proper. Then nothing until the Black Death in the fourteenth century. It seems vanishingly unlikely that, to pick some of the obvious points:

  • a) China or India - which had a population in the tens of millions - did not have any substantial epidemic outbreaks for most of its history;
  • b) There was only one major epidemic outbreak pre-1 AD;
  • c) There were none in the early Mesopotamian civilisations or in ancient Egypt;
  • d) There were none in Europe or the Mediterranean for most of the medieval period;

On that first point, in a comment above I mentioned McNeil's Plagues and Peoples (1976). It has an appendix of epidemics recorded in Chinese sources; at a guess, there are around 200. These include what appear to be very major epidemics (killing a substantial proportion of the population in some provinces) in 312, 322, 413, 468, 682, 707, 762, 806... and those are just the ones with relative death tolls noted. It would be challenging to calculate numbers for an analysis like this one, granted, but it seems hard to just dismiss them from consideration because no estimate is available.


Update: skimming the papers citing the original Cirillo paper, we find Intensity and frequency of extreme novel epidemics (PNAS, 2021), which offers a dataset that they have compiled.

In summary, the 1600 to 1945 dataset includes 182 epidemics with known occurrence, duration, and number of deaths, 108 known to have caused less than 10,000 deaths, and 105 for which only occurrence and duration are recorded, for a total of 395 epidemics.

So that gives us almost 200 identified in a 350 year period with over 10,000 deaths, plus an indeterminate number of deaths unknown - well in excess of the original figure

11
  • 1
    (See my Qcomment for: 'OC43='RussianFlu' at that time…; ID still contested of course ;) The almost funny thing is: Taleb's end might be still 'correct'' (though: he's most probably not). It's a pity that they don't address at all this rather gaping hole in their methods… 'Validity' is reduced quite a bit automatically by such standing flaws Commented Oct 5, 2022 at 22:49
  • The early Mesopotamian civilizations and (most of) ancient Egypt would fall outside the stipulated 2,500 year time frame. But otherwise, +1, very nice fisking. Commented Oct 6, 2022 at 7:13
  • 1
    @TheReal_Skywalker I read that as being "182 with number and above 10,000", given the totals add up that way. But it's a bit odd: their linked dataset seems to give deaths in 000s and has 161 as 0-10, plus 129 over 10 and 105 unknown ("-999"). Recounting from that list and using our original threshold, I get 171 with a listed count of 1000 deaths or above in 1600-1945, and 216 in their extended dataset (1500-2020) Commented Oct 6, 2022 at 11:44
  • 1
    @LаngLаngС ah, that should be 412 BC - corrected. The original paper did indeed include the Antonine plague! Commented Oct 6, 2022 at 13:52
  • 1
    Thx. Am glad not having edited this minor BC2AD! Note that 412 is most often speculated to have been influenza (Potter:_Influenza_ seems even dead sure(https://doi.org/10.1046%2Fj.1365-2672.2001.01492.x) !). Meaning: another of the usual SARIs! Which was probably only noted down because that time young kids suffered the most? Thus says my thumb calculator: for flu alone there must have been much more epidemics than NT assumes? Commented Oct 6, 2022 at 14:27
6

While my question has been answered well by @Andrew, I'd like to mention here what Corral (2021) had to say since he shares similar concerns about Black Death being the only event in the dataset from 750 to 1450 AD and also provides additional data with respect to Spain which is not included in the dataset (while @Andrew provided for China):

In fact, an important limitation when studying the distribution of fatalities in epidemics comes from the available data, not only because of the small sample size (N = 72 in the data of Ref. [6]), but also from the incompleteness of the data (with a bias in favor of very large events that resampling [6] cannot correct) and from the lack of homogeneity in time (with just one event, the Black Death, between 750 and 1450, and 11 events since 2008 in the data of Ref. [6]).

In any case, one can dig a little in the existing records and find many more historical events. As an example, Villalba [43] reported several epidemics in Spain with more than 10 000 fatalities that are not considered in the data of Ref. [6] (these missing Spanish epidemics took place in 1283, 1394, 1490, 1564, 1589, 1637, 1726, 1741, 1784, and 1800). Certainly, there is nothing special about Spain, and other countries can contribute more or less in the same way with more “hidden” epidemics. Nevertheless, the compilation of a reliable record for historical epidemics is something that should not be done by probabilists, statisticians, or physicists and needs to be carefully undertaken by true epidemiologists and historians. We urge here for the necessity of such an important endeavor.

4
  • 1
    This is helpful to see, thanks! Looks like we were not the only ones to be concerned about the data used. Commented Oct 6, 2022 at 12:49
  • 1
    Well (criticizing Coral here)...there is something special about Spain in that it was in the middle ages the most developed corner of Western Europe, and in the Renaissance the richest.
    – T.E.D.
    Commented Oct 7, 2022 at 13:28
  • @T.E.D. What's that? "Spain" or Alandalus? (Does even Corral's src Villalba differentiate that? For that period our mental maps for geography and politics esp in that area are often quite distorted. If this is meant to only count outbreaks in 'Christian ruled Spain', then the rest of 'Muslim ruled Spain' might have a few epidemics in store on top of that!) Also note that only 2 'missing data points' are medieval. Commented Oct 7, 2022 at 15:02
  • @LаngLаngС - Geographic term of course. "Spain" as a nation didn't exist until the very late 15th century (and that's arguable).
    – T.E.D.
    Commented Oct 7, 2022 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.