Over the Transom 3/23/18

The very first edition of “The Joy of Cooking” was self-published by the St. Louis hostess and housewife Irma Rombauer in the first years of the Great Depression. A relatively modest volume, it collected some four hundred and fifty recipes gathered from family and friends, garlanded throughout with chatty headnotes and digressions regarding the finer points of entertaining, nutrition, menu planning, and provisioning. Since that original edition, the book has become one of the best-selling cookbooks of all time. It also has undergone eight significant revisions: Rombauer’s list of recipes exploded into the thousands; entire chapters were added (frozen desserts) and dropped (wartime rationing). (“The” was dropped from the title in the mid-sixties.) The 1997 edition was a particular departure, replete with contributions from superstar chefs and celebrity food writers. “Joy” purists considered it something of a heresy (the Times memorably called it “the New Coke of cookbooks”), and were relieved when the 2006 edition returned to classic form.

Short of that hiccup, “Joy” has been subject to very little criticism in its eighty-seven-year life. Smart, bossy, funny, a little bit cornball, the book has been a staple in countless American kitchens, a go-to gift for newlyweds and recent grads, its adherents spreading the gospel to their own children. (When my parents’ ragged copy of the 1964 edition succumbed to water damage a few years ago, my mother delivered the news as if a relative had died.) About the worst that’s been said of the book is that it’s more useful as a general-reference volume than as a recipe go-to, which—given the cooking world’s overabundance of recipes and its shortage of genuinely useful reference books—is actually sort of a compliment.

So it came as a shock, in 2009, when the prestigious scholarly journal Annals of Internal Medicine published a study under the pointed headline “The Joy of Cooking Too Much.” The study’s lead author, Brian Wansink, who runs Cornell University’s Food and Brand Lab, had made his reputation with a series of splashy studies on eating behavior—in 2005, for instance, his famous “Bottomless Bowls” study concluded that people will eat soup indefinitely if their supply is constantly replenished. For “The Joy of Cooking Too Much,” Wansink and his frequent collaborator, the New Mexico State University professor Collin R. Payne, had examined the cookbook’s recipes in multiple “Joy” editions, beginning with the 1936 version, and determined that their calorie counts had increased over time by an average of forty-four per cent. “Classic recipes need to be downsized to counteract growing waistlines,” they concluded. In an interview with the L.A. Times, Wansink said that he’d decided to analyze “Joy” because he was looking for culprits in the obesity epidemic beyond fast food and other unhealthy restaurant cooking. “That raised the thought in my mind: Is that really the source of things? . . . What has happened in what we’ve been doing in our own homes over the years?”

John Becker, the great-grandson of Irma Rombauer, lives with his wife, Megan Scott, in Portland, Oregon, and they are the current keepers of the “Joy” legacy. When the results of Wansink’s research were released, they and their publishers were blindsided. With the help of Rombauer’s biographer, they posted a response on the “Joy” Web site criticizing some of Wansink’s methods and calling attention to his sample size—out of the approximately forty-five hundred recipes that appear in later editions, he’d chosen eighteen, a mere 0.004 per cent of the book’s content. But they stopped short of rejecting Wansink’s conclusions outright. “Joy” had always been an idiosyncratic operation, written and rewritten, over the years, by strong personalities who held forceful and often conflicting opinions. (Becker’s grandmother, Marion Rombauer Becker, and father, Ethan Becker, were each eventually added as co-authors.) “We assumed that he was probably correct, and that the recipes probably had increased in calories per serving,” Scott told me recently by phone. “If we had wanted to impugn the reputation of a sitting Cornell department head, I think we would’ve found a really tough row to hoe.”

But the study turned up again and again over the years, becoming part of the conventional wisdom on obesity—a “stand-in,” as Becker puts it, for the “Sad American Diet.” A cartoon that was commissioned by Cornell’s Food and Brand Lab and published with the original study depicts a beefy newer edition of the book haranguing an older edition, jeering at its brother, “I have 44% more calories per serving than you do!” Wansink’s tiny sample set, especially, gnawed at the couple. In his study report, Wansink explained the size as a methodological necessity, writing that “since the first edition in 1936, only 18 recipes have been continuously published in each subsequent edition.” But, in researching the cookbook’s ninth edition (scheduled for 2019), Becker and Scott had created an encyclopedic catalogue of thousands of legacy “Joy” recipes, and they counted several hundred recipes that had remained comparable from one edition to the next. When, in 2015, Wansink’s cartoon landed in Becker’s in-box yet again, he decided to conduct his own research. Becker started his analysis cautiously, hoping to find a few counterexamples in “Joy of Cooking” with which to push back against Wansink’s findings. Instead, he told me, “I was, like, ‘Oh, my God, there’s a lot more.’ I mean, the numbers are turning up in our favor, and they’re definitely not determining what Wansink’s got.”

Then, last month, the BuzzFeed reporter Stephanie Lee published a sweeping exposé of Wansink’s research. Academic standards call for researchers to articulate a hypothesis ahead of time, and then to conduct an experiment that produces data that will either prove or disprove the hypothesis. Lee’s article—which was based on interviews with Cornell Food and Brand Lab employees, and also private e-mails from within the lab, which were obtained through a public-records request—showed that Wansink regularly urged his staff to work the other way around: to manipulate sets of data in order to find patterns (a practice known as “p-hacking”) and then reverse-engineer hypotheses based on those conclusions. “Think of all the different ways you can cut the data,” he wrote to a researcher, in an e-mail from 2013; for other studies, he pressed his staff to “squeeze some blood out of this rock.” One of Wansink’s lab assistants told Lee, in regard to data from a weight-loss study she had been assigned to analyze, “He was trying to make the paper say something that wasn’t true.”

Lee’s report wasn’t the first time that doubt had been cast on Wansink’s work: in 2016, he published a blog post (which he later deleted) revealing that he had encouraged graduate students to do this sort of data fishing; the post resulted in a flurry of critical coverage toward his methods. But Lee’s was the most comprehensive and damning account. “Year after year,” she concluded, “Wansink and his collaborators at the Cornell Food and Brand Lab have turned shoddy data into headline-friendly eating lessons that they could feed to the masses.” Two days after Lee’s story was published, John Becker posted on the official “Joy of Cooking” Twitter account, “We have the dubious honor of being a victim of @BrianWansink and Collin R. Payne’s early work.”

Around the same time, Becker sent his own vast archive of material related to Wansink’s study—including a Microsoft Excel spreadsheet tracking the calorie count of hundreds of “Joy” recipes over time—to several academics, including to James Heathers, a behavioral scientist at Northeastern University. Heathers is one of a platoon of swashbuckling statisticians who devote time outside of their regular work to re-analyzing too-good-to-be-true studies published by media-friendly researchers—and loudly calling public attention to any inaccuracies they find. Heathers’s own work—particularly his development of a modelling tool called S.P.R.I.T.E., which allows likely data sets to be reconstructed from published results—has led directly to the amendment or retraction of a dozen academic papers in the past few years, including several authored by Wansink.

Heathers told me that the problems he’s found in Wansink’s studies are generally within the numbers themselves: faulty arithmetic, sloppy recording, subsets of data that disappear at times and then “magically reappear” later, and conclusions that reverse-engineer improbable samples. (Working backward from the results of a study about vegetable-eating habits in children, Heathers determined that Wansink’s conclusion was only valid if one child had devoured sixty carrots at once. Wansink published a lengthy correction, clarifying that the experiment was conducted with “matchstick carrots.” He later retracted the study altogether.) The methodological flaws Heathers found in “The Joy of Cooking Too Much” are of a different sort: because the recipes in question are fixed information, the actual data—ingredients, quantities, nutritional information—aren’t subject to manipulation. Instead, Heathers found issues with the study itself. “The problem is not that it was added up wrong,” he said of the data. “It’s that there’s no real way to add it up right.”

The recipes were compared on the basis of serving size, for instance, but ten of the eighteen recipes that were studied do not specify what counts as a serving. (“Joy” ’s chocolate-cake recipe yields simply “1 cake.”) The small sample size was especially problematic, Heathers explained, because the calorie changes in the eighteen recipes that were studied varied drastically, from a hundred and thirty-four per cent increase in the goulash to a thirty per cent decrease in the rice pudding. “That’s not a reliable pattern!” Heathers said. Wansink also insisted on only comparing recipes that bore identical names in different “Joy” editions, regardless of the accompanying recipes, which sometimes led him to compare two entirely different dishes. He liked to point to gumbo as one of the most egregious calorie gainers, but the recipe from 1936, a clear soup of chicken and sliced vegetables simmered in water, has almost nothing in common with the sausage-studded, roux-thickened chicken variety featured in the 2006 book. “It’s like comparing a Chateaubriand to a whole roast steer,” Heathers said, “and saying they’re both roast beef.”

When I reached Wansink this week by phone, in his office at Cornell, he told me that he stands by the analysis in “The Joy of Cooking Too Much.” “This is a really nice methodology that’s set out,” he said of the study. But he acknowledged that his team had faced challenges, saying, “You’ve got to be very careful that the recipes are comparable, and that the sample frame is one that can be acceptable to a journal, and be seen as fair.” When I suggested that Becker and Heathers found fault with his study on those very grounds, he said that the published data was an abbreviated version of a paper that was “really, really, really quite long,” and that ultimately had, at the request of the journal’s editors, several key elements removed, though he couldn’t recall what those elements were. (He had no comment on the findings of Lee’s BuzzFeed report.)

Studying what and how people eat is a messy science, in large part because it’s extremely difficult to control human behavior: in 2016, the British epidemiologist Ben Goldacre, discussing obstacles to his ideal experiment,noted, of its hypothetical subjects, “I would have to imprison them all, because there’s no way I would be able to force 500 people to eat fruits and vegetables for a life.” Even factors we assume to be absolute can fluctuate; the calorie content of a particular ingredient can change depending on the preparation method, and even on how well it’s chewed. The result is an academic literature full of often contradictory advice—Eating animal fats causes massive weight gain, avoid it! Eating animal fats is the only way to lose weight and keep it off, add it to your morning coffee!—that can amplify consumer anxiety toward how and what to eat.

One point that remains consistent across virtually every nutrition and health recommendation is that eating home-cooked meals prepared with fresh ingredients correlates with better health. This is something that Irma Rombauer seemed to understand instinctively. In an edition of “Joy of Cooking” from the early sixties, she and Marion Becker advised that “well-grown minimally processed foods are usually our best sources for complete nourishment.” With uncanny foresight, on the book’s very first page, they also issued readers a warning: “The sensational press releases which follow the discovery of fascinating fresh bits and pieces about human nutrition confuse the layman,” they wrote. “And the oversimplified and frequently ill-founded dicta of food faddists can lure us into downright harm.”