The Iraq Math War
Commentary: Why the CDC and the Pentagon sought to discredit the first scientific tally of Iraq's civilian death toll.
|
|
in early 1999, Les Roberts traveled to Bukavu, a city of more than 200,000 in the Democratic Republic of the Congo (drc). The country's brutal civil war was in full swing, and a nearby region, Katana, had been largely cut off from the outside world for nearly a year. Roberts, a former Centers for Disease Control (cdc) epidemiologist who'd taken over as director of health policy for the International Rescue Committee, wanted to see how the locals were faring.
Every morning for weeks, Roberts and his team rode into the jungle. After finding a spot they'd selected randomly on a map, they approached the people living in the area and asked them about recent deaths in their households. When Roberts finally crunched the numbers, he determined that the mortality rate in Katana was two and a half times the peacetime rate. The next year, using a similar approach, he concluded that the war's overall death toll in eastern drc at the time wasn't 50,000, as widely reported, but a staggering 1.7 million.
Roberts' results helped boost the reputation of conflict epidemiology, a fledgling discipline that applies the tools of public health research to the surprisingly difficult question of how many civilians die in war zones. Historically, soldiers and journalists have been the main sources of real-time casualty estimates, leaving the truth somewhere between propaganda and a best guess. Researchers are still revising death tolls for wars that ended decades ago; estimates of civilian deaths in Vietnam even now range from 500,000 to 2 million or more. The methods Roberts helped pioneer aimed to end some of that uncertainty.
Advocates and policymakers quickly discovered the power of scientifically valid mortality studies to spur leaders into action. After Roberts presented his Congo data on Capitol Hill in 2001, US aid to the country jumped tenfold. A cdc survey of mortality rates in Kosovo was used as evidence in Slobodan Milosevic's war crimes trials at The Hague. Multiple such surveys in Darfur contributed to former Secretary of State Colin Powell's decision to condemn the Sudanese government for facilitating genocide. And Roberts' 2001 estimate of deaths during Sierra Leone's civil war has been widely accepted.
But when Roberts took on the challenge of tracking civilian casualties in Iraq, he was quickly reminded that, as with the use of sampling in the US census, statistical methodology can become highly politicized. He found his Iraq work misunderstood, misrepresented, even written off as propaganda. Lifting the fog of war, Roberts discovered, isn't a question of finding the most accurate number, but one people are willing to accept.
in september 2004, Roberts, then at Johns Hopkins University, arrived in Baghdad to supervise a nationwide mortality survey in collaboration with Gilbert Burnham, codirector of the university's Center for Refugee and Disaster Response. At first Roberts accompanied the Iraqi physicians who were conducting the interviews. But after police detained two of them, Roberts and the doctors decided he should stay behind. "They all realized that being with an American was something radioactive," he says.
So while Roberts sat in his hotel room, his research teams fanned out across the country, following the approach he had used in the Congo five years earlier. The technique, known as a two-stage cluster survey, works much like an opinion poll in which interviews with a random sampling of people are extrapolated to reflect the views of an entire population. Roberts' teams surveyed 990 households located near 33 randomly selected spots, more than the minimum number of "clusters" epidemiologists consider necessary to get an accurate picture of what's going on in a country.
Upon returning home, he and Burnham analyzed the data. They were floored: In the 18 months after the American invasion, the numbers suggested, roughly 100,000 Iraqis had died as a result of the war, 60 percent of them violently. That dwarfed the figure from the widely cited website Iraq Body Count, which had tallied no more than 19,061 deaths by scouring press reports and official documents. The Iraqi government's numbers were also much lower. The researchers sent their study to the prestigious medical journal The Lancet, which published it in October 2004.
The unexpectedly large death toll elicited skepticism, and questions about the methodology. The study had a wide "confidence interval" of 8,000 to 194,000. "This isn't an estimate. It's a dart board," scoffed Slate military writer Fred Kaplan.
But leading epidemiologists and statisticians insist the study is valid. A confidence interval is structured like a bell curve, with the numbers in the bulging middle far more likely to be accurate than those at the tapering ends. It was a larger interval than Roberts and Burnham had hoped for—a consequence of their sample size and the uneven distribution of violence in Iraq. That didn't render their estimate meaningless, however, just easy to dismiss. "I expected to be criticized," says Roberts, who has since joined the public health faculty at Columbia University. "I was more struck by the lack of press coverage."
He didn't help matters by telling reporters he'd opposed the invasion, leading the AP to suggest that the study's timing was politically motivated. Critics, meanwhile, have questioned Roberts' decision, in the year following the Lancet article, to launch a short-lived congressional run in upstate New York as a pro-science, anti-war Democrat. Roberts resents the notion that scientists should stay out of politics. "Everyone who writes about public health problems wants them solved," he says. "No one who writes about measles is neutral."
Roberts' politics don't bother John Tirman, who runs the Center for International Studies at mit. "I thought it explained as few other things had the origins of the insurgency," he says of the study. "In a country like Iraq where there are very strong kinship networks, where if someone is attacked and killed it obligates a very large number of men to defend the community, this large scale of violence suggested that there were a large number of Iraqis that were essentially being drawn into the insurgency by the way the invasion and occupation was conducted."
Tirman (who—full disclosure—served as a board member for Mother Jones' parent foundation during the 1990s) helped secure funding through mit for a second, larger survey. In the spring and summer of 2006, the team's researchers canvassed the country yet again, visiting more than 1,800 households clustered around 47 sites. As of that July, Roberts and Burnham would later estimate, the war had claimed about 655,000 Iraqi lives, suggesting that about 1 in 7 Iraqi families had lost someone because of the ongoing violence. As in the first study, there was a wide confidence interval—plus or minus about 275,000 deaths. But even the low end of the range suggested a death toll far beyond anything previously reported.
That October, after the new findings appeared in The Lancet, the critics pounced, again honing in on what they called fuzzy math. In a Wall Street Journal op-ed Steven E. Moore, a pollster and former adviser to Coalition Provisional Authority chief Paul Bremer, declared, "I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points."
"That's wrong," says Jennifer Leaning, a professor at Harvard University's School of Public Health. "You can sample very large populations with 33 clusters." Other epidemiologists I contacted agreed.
But there were some valid critiques: By their own admission, Roberts and Burnham had to rely on outdated population estimates to set up the Iraq study, which overlooked war-induced migration as a result. Michael Spagat, an economist at the University of London, argued that the way the interviewers chose their starting points—they'd abandoned the handheld gps devices used in the first study, deeming them a security risk—would have led them to homes near main thoroughfares, where ied explosions would be more common. And because the lead authors weren't on hand to monitor the second survey, Spagat and others have suggested that the interviewers simply lied. (That the new results agreed with the team's earlier findings for the same period suggests that the doctors did their job properly.)
In any case, such problems are common in war zones, according to nearly a dozen leading survey statisticians and epidemiologists I spoke with. "Iraq is not an ideal condition in which to conduct a survey, so to expect them to do the same things that you would do in a survey in the United States is really not reasonable," says David Marker, a senior statistician with the research corporation Westat. Even if the outdated population data led the researchers to a 20 percent overestimate, Marker explains, the revised death toll would still be at least a couple hundred thousand. "These methodological concerns don't change the basic message."
The White House struck back with its own basic message: The study was bunk. Never mind that Roberts and Burnham had used methods similar to those employed for the Kosovo survey and others approvingly cited by the Bush administration. With the notable exception of This American Life producer Alex Blumberg, most reporters dutifully slapped Roberts' research with the "controversial" label. And when asked about the study directly, President Bush declared that it had been "pretty well discredited."
"By whom? By him and his political staff?" snaps Bradley Woodruff, who retired last year from his job as a senior cdc epidemiologist. Woodruff has conducted mortality surveys himself, and considers Roberts' research solid. But when cbs's 60 Minutes sought to interview Woodruff about the Lancet study in 2007, the cdc wouldn't allow it. And when Rep. Dennis Kucinich invited Woodruff to Washington to discuss the study, his bosses nixed that, too. "I never had this kind of censorship under previous administrations," he says.
more than two years later, the Iraq study remains mired in controversy. But other recent findings suggest that Roberts and Burnham were on the right track. In the summer of 2006, the World Health Organization conducted a large family health survey along with Iraq's Ministry of Health, interviewing about five times as many people as Roberts and Burnham had, and in a more distributed fashion. In August, Mohamed Ali, a who statistician, reported his preliminary results to colleagues at a Denver statistics conference: Nearly 397,000 Iraqis had died because of the war as of July 2006.
That number falls at the low end of Roberts and Burnham's confidence interval, which ranges from roughly 393,000 to 943,000. But while epidemiologists and statisticians are still pondering questions raised by differences between the two surveys, there's no longer much doubt among them that Iraq's civilian casualties number in the hundreds of thousands.
This grim statistic continues to elude most Americans. According to a February 2007 AP poll, Americans' median estimate of the number of Iraqis killed since the invasion was just 9,890. And while the Pentagon has presented limited estimates of civilian casualties, it has yet to release any numbers for the total toll since the invasion.
Roberts had set out to provide a legitimate number that might be used to inform public policy. For now, at least, that policy has been to keep the truth buried in academic journals—and beneath the sands of Iraq.
Correction appended: An earlier version of this story inaccurately stated that the Iraq Body Count had tallied no more than 23,000 deaths. We had estimated this figure using their published data, and did not obtain the more precise figure of 19,061 until after press time.
Robin Mejia has written for Science and Wired.
Photo: Andrew Hetherington

The Lancet study may be sound, but this statement is completely false.
A confidence interval does not correspond to any distribution. Saying it's bell shaped is entirely wrong.
A 95% confidence interval states that if you were to repeat this poll many times (same sampling method, same number of samples), and if your assumptions about the distribution of the population are correct (gaussian, poisson, whatever), then 95% of the times the confidence interval you calculate will contain the "true" result (number of dead Iraqis in this example).
That's ALL it tells you. Saying that the true value is more likely to be at the center of the confidence interval than at the edges is patently wrong.
And not a word about the thousands who wereand to this day continue to be deliberately targetted and killed by Islamo-fasicts. Go figure!
I suppose this is what you mean by 'not a word about...'
I seem to remember us (U.S.) trying a guy for 'crimes against humanity' for 269 people who were killed over 30 years ago. We convicted him. And then we hanged him.
So even if the lowest possible number (19061) is used, what does that say about us (U.S.)?
I personally believe the 600,000 number for the Iraqi death toll, as I also believe that the US death toll and number of wounded is far lower than we are being told as well. And we have zero number for the contractors - who outnumber our mulitary forces and are in just as much danger (and no, they don't all work for Blackwater!)
Could fifty plus years of American misguided foreign-policies in the ME have anything to do with it?
The US CIA removed a democratically elected president of Iran in 1953 and replaced him with the Shah. Resulting in the theocratic government it has today. Ronald Reagan "Negotiated" with Iran in the Treason fomenting Iran-Contra weapons for cocaine scandle.
Ronald Reagan referred to Saddam Hussein and Osma Bin Laden as "Freedom Fighter's", the GOP "Greed & Oppression party" have re-labeled "Terrorist's".
If the US had followed through with the Carter energy Policy, then it wouldn't be so beholden to ME oil today.
Saudi Arabia has been exporting Wahhabism form of Islamic religion where a martyred male will meet 72 virgins after his death. If you grew up in squalor that is partly responsible by US policies, I think you would find this to be a rewarding end too.
Just like these delusional "Christian's" who will rise to a mythic Heaven to live eternally.
Or a General Jesus is out to kick some Islamic booty in this farce War Of Terror based off of lies and deceit, like Caribou Barbie stated to departing ANG soldiers recently.
Before engaging mouth, think!
The US has been the leading exporter of state-sponsored terrorism for its whole existence all the while talking about democracy and human rights. Well, the facts don't fit the myth.
The US has always been about profit and free market capitalism right from the beginning and the people have been lied to the whole time. I'm reading John Perkins' "The Secret History of the American Empire" and have read his "Confessions of an Economic Hit Man". It is sickening what our "elected" officials (the ones the corporatocracy chooses for us anyway) have wrought on this earth for the sake of profit for the big corporations who own them. The US has been behind the deaths of millions of people all over the world from Indonesia, East Timor, all of Latin America and Africa to Iraq. And the US "news" media never tells the people. Talk about the "Evil Empire"!
Yes it is difficult to determine exactly how many people have died so far due to illegal and immoral wars based upon lies. How many people left instead of died due to the destruction of their infrastructure. Technically if someone died because they had a heart attack that they would have survived because last week their hospital was reduced to rubble is a war casualty. If someone dies from dissentary because their sewage treatment plant no longer works (due to military action or lack of supplies/maintenance due to war) they are a war casualty. And the list goes on. The US military doesn't count them. But we are no less responsible for their deaths than if they were killed by our weapons. They died because a we started a war that should never have been.
Freethinker says it well. If you think the US is just defending herself you mental. We've been committing atrocities for well over a century. We do not hear about it in our news. We do not read about it in our history books. But it's been and being done. Other peoples know too well. They are the families of friends of these victims. It's not a matter of 'hating America' as some simplistic thinkers believe. I love my country with all of my heart. So I have to stand against those who harm my nation. Starting illegal wars based upon lies harms my country. Immoral economic policies that impoverish and devastate other nations harms my country. Each foul act has it's repercussions. Over 90% of the world is against the Bush administration. Doesn't that hurt America? Read Howards Zinn's A People's History of the United States and get an idea of real US history. As FreeThinker said, read John Perkin's books and get an idea of the real US foriegn economic policy. The US is not alone in these crimes against humaity. But as Americans we should hold ourselves to the highest standards. And if you believe for a second that we should act in accordance with the teachings of Jesus you can't for a second oppose change in the US.
Second, I want to tell all my American friends (and less friends) that Islamo-Fascism is something that exists only in some US brains and US media. In the real world, there's nothing like that. There is a guy called Bin Laden who is a CIA creation back in Afghanistan when they needed to fight the Russian Bear.
Third, whatever the number of deaths in Iraq, Iraq invasion will remain forever in the human book history as a shame and a dishonor in American History: impossible to justify (except by unacceptable propaganda).
Kindest regards
An Arab an a Muslim, as well as the father of a family and a honest (as much as possible) earth citizen.
How about the only country to ever use a nuclear weapon, which everyone knows will not distinguish between soldier and civilian when dropped on a city center. 140,000 killed in Hiroshima and 80,000 killed in Nagasaki - civilian death estimates.
How about the fire bombing runs over Dresden (24,000-40,000 civilians) that didn't target military bases, but just cities.
Would the US lie to get into a war? Can you say Gulf of Tonkin?
Rather than "redundant", I think you mean oxymoronic, like "conservative think tank" or "republican values."
Nasty reading habit you have there - I said we haven't targetted civilians since FDR( you know - WW2 which is where your examples come from) - and we haven't. Only your friends the Islamo-Fascists target civilians, but that doesn't bother you, does it? - after all they hate America also.
Oshtkd- No it is not redundant to say leftist automaticlly means anti-American. Just look at the posts of your fellow "blame America first" fellow travelers on this post.
To the best of my knowledge, that's correct, but I'd like to go a little further and point out that each time you draw a new sample from a population and construct a CI from that sample, it will have its own upper and lower limits. The second CI might have upper and lower limits that are similar to those of the first one (or perhaps even identical) -- but they might be radically different, too. Whenever you construct a 95% CI, by definition there is a 95% chance that it is one of the CIs that might capture the true value of the characteristic of interest (referred to as the Greek letter mu). And the same will hold true for every other CI constructed by the same method -- there will be a 95% chance that it captures mu, but its upper and lower values probably will be different (reflecting the difference in the sample). But that's not the same thing as looking at a given interval and saying there is a 95% chance that the true value lies between its upper and lower limits, as so often is claimed. The best you can say about a given interval is that there's a 95% chance that it is one of the intervals that captures mu, and that its upper and lower limits happen to be [x] and [y]. It is the distance between the limits (wide or narrow) that is of more interest than their precise values.
As explained by Devore & Peck in their textbook Statistics: The Exploration and Analysis of Data, "It is tempting to say that there is a 95% chance that mu is between [x] and [y]. Do not yield to this temptation! The 95% refers to the percentage of ALL possible samples resulting in an interval that includes mu.... if we take sample after sample from the population and use each one separately to compute a 95% confidence interval, in the long run roughly 95% of these intervals will capture mu. [A given interval] either includes mu or it does not... We cannot make a chance (probability) statement concerning this particular interval. The confidence level 95% refers to the method used to construct the interval rather than to any particular interval."
In some cases, mu might be near the center of the interval and in some cases it might be near the edge -- and in some cases (about 5%, if you're working with 95% CIs) it won't be within the interval at all. We have no way of knowing where the true value really lies, short of measuring the entire population. Which is why we turn to statistics in the first place.
The authors quoted above created a figure that nicely illustrates these concepts. It consists of a vertical line representing mu and 100 horizontal lines representing 95% CIs computed from 100 different samples. 93 of the horizontal lines intersect with the vertical line, some near their center and some near the edge. The other 7 don't intersect with it at all. (They could have shown 95 intersecting intervals and 5 non-intersecting intervals. Their depiction of 93 and 7 reinforces the idea that we're dealing with probability here.)
What about Agent Orange? What about cluster bombs? What about land mines? What about the Palestine Hotel?
You've got a very conveniently selective memory.
It's not a matter of "blaming American first" as you like to spout. It's really about making sure America is the greatest nation on Earth! It's about making certain that our minorities and our marginalized - the tired, weak, huddled, and poor - are taken care of. It's about making sure that America doesn't damage it's reputation because of a poor or overly aggressive military strategy. It's about making sure that our military strategy is really benefiting America and not just creating new enemies as in the past. It's about fighting for the best of America - the fact that we've always led on issues like democracy, slavery, women's suffrage, civil rights, gay rights, protecting the rights of the vulnerable.
As MLK says, and I paraphrase: The arc of history bends toward justice.
That is the true history of America. And that is what we are fighting for.
People who are unable/unwilling to criticize their country become tools of the state.
Ignorant, knee jerk "criticism" of one's country is not admirable - it only uplifts our enemies whom those of your ilk aid and abet with your "blame America first" ideology.
Fred T., since you're ignorant, er, confident in your positions, why don't you use your full name??
Fred T. Please reread Clunker's post. Then go read MLK's biography.
I would not "like" to do anything to you - I would like to do something for you. I would like for you to acknowledge the error of your ways - if you put aside the knee-jerk anti-Americanism that is so characteristic of the American left you might find enlightenment. It is not too late to become a patriot.
For a short time boots on the ground real journalists were reporting IRAQI deaths based on observation and interviewing medical personnel .
When they began differing with US and British reports the same reporters were shunned. In fact some were recalled from IRAQ and today they are arm chair journalists.
Locate John Burns and all the CNN tapes.
Any number of dead less than +100,000 according to my notes, is understated .
What Americans should be more concerned about is and was: the total control of information being reported to US citizens as "fact".
Here we are in 2008 and still cannot validate the human loss.
@Fred T:
It's "patriots" like you that have gotten us into this mess in the first place. I have absolutely no desire to be a "patriot" in your sense of the word.
The Lancet "data" have been refuted in detail by competent analysts who had no ideological ax to grind.
And don't worry - no one would accuse you of being a patriot of any "kinfd".