Why We Sleep: a tale of institutional failure

Published 2020-03-24.

Introduction

This is the story of how UC Berkeley dropped the ball when they were made aware that one of their professors possibly committed research misconduct. It is also the story of how publishers Press and Penguin failed to even acknowledge the ball's existence.

Falling in love with sleep

In the summer of 2019, a friend recommended I read Matthew Walker's book Why We Sleep. I did, and I loved it. I was thoroughly convinced that I needed to be more diligent about getting enough quality sleep. So, I bought a smartwatch in order to gather data on my sleeping patterns. I also bought five additional copies of the book, and handed them out to friends and family. So far, so good.

Caught red-handed

A few months later, I came across Alexey Guzey's article Matthew Walker's "Why We Sleep" Is Riddled with Scientific and Factual Errors, and I was blown away. It's well worth a read, but if you decide to skip it, you should know that Guzey thoroughly fact-checked a single chapter. He found a lot of problems. The most obvious case of misconduct, in my opinion, was omitting part of a chart from a quoted paper. This is the original chart from the paper Walker cites:

In Why We Sleep, it somehow became this:

The omission of the 5 hour bar made it seem like there is better evidence for Walker's claim than there actually is. I strongly believe it falls under UC Berkeley's definition of falsification:

Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.

Note that, while I believe this to be a clear case of research misconduct, I am not making any claims about what consequences Walker or anyone else should face. That's a different question.

Blowing some whistles

Even though I take pride in my ability to adjust my opinions in the face of new evidence, it can really hurt, and it sure did this time. Now what should I believe? And what fragments of false belief were left behind in my mind? On top of that, I had to notify the people I gave the book to. When there are that many errors in a single chapter, who knows how many there might be in the rest of the book? It was clear to me that the book could not be trusted.

I reached out to Penguin, UC Berkeley and the Norwegian publisher Press (which published the Norwegian translation).

The worst reaction was from Press — they didn't respond at all. Penguin said that the editorial team would take a look, but I never heard back from them. UC Berkeley responded promptly but concluded that they would not conduct a formal investigation.

Below, I'm publishing my email exchange with UC Berkeley so that others can see it rather than it just gathering dust in my email archive. In a way, it's unfair to pillory UC Berkeley, considering that they actually have a process for dealing with research misconduct and at least responded to my complaint. Compare that to the publishers, who neither have a process nor responded. Nevertheless, I expect more from a renowned research institution like UC Berkeley, so I still believe I'm right to publish our correspondence.

Institutional failure

I think we should demand a lot from institutions and very little from individuals. Humans can be lazy, selfish, biased and greedy. They can also be also hard-working, altruistic, fair and generous. But the dark side of humans isn't going away any time soon. Institutions, however, can do a lot better than individuals. (The scientific community is an example of such an institution — despite individual scientists' many imperfections, science keeps progressing.)

Therefore, I view this farce as a series of institutional failures rather than personal ones. And my hope is that we can improve these institutions by criticizing them when they fail. That is what I'm trying to do with this article.

Email exchange with UC Berkeley

Because this failure is UC Berkeley's more than any individual's, I have edited the exchange slightly by removing the names of their representatives. In order to keep it as short as possible, I've also taken out greetings like "Hi" and "Kind regards". However, in order to be transparent, I've uploaded the entire exchange here.

So, see the emails below and make up your own mind. You can comment on Andrew Gelman's blog, Twitter, Hacker News, r/slatestarcodex or email me directly.

So, without further ado, the email exchange is below. It took place in the first three months of 2020.

Me:

Matthew Walker, a professor at UC Berkeley, wrote Why We Sleep (2017). It turns out that it's riddled with scientific and factual errors.

Also see “Why we sleep” data manipulation: A smoking gun? by Andrew Gelman, director of the Applied Statistics Center at Columbia University.

Is this something you can look into? Or should I reach out to someone else?

UC Berkeley:

It is indeed one of my duties at UC Berkeley to look into allegations of research misconduct. I will consult with colleagues in the field to get their assessment of the concerns raised about the data citation and analysis in Walker's book.

Me:

Great, thanks a lot :)

Me (a few weeks later):

I'm curious about whether anything has happened in this case.

Also, you might be interested in knowing that Guzey recently appeared on the Smart People Podcast, where they discussed his critique. (The first part is a more general interview, so you might want to skip ahead to the 13:45 mark.)

UC Berkeley:

We did follow up on the allegations of research misconduct against Matthew Walker by following our usual protocol, asking a qualified researcher from a relevant discipline—someone without ties to Professor Walker—to determine whether the claims warrant further investigation and inquiry. This individual found that Professor Walker had included a link to the allegations on his website, that he has addressed the issues raised, and that he has announced his intention to correct any mistakes in the next edition of his book. Based on the facts and these findings we found no cause for further investigation. We believe, based on the evidence, that while there were some minor errors in the book, which Walker intends to correct, there was no research misconduct per our definition (here).

Me:

Thanks for getting back to me. I have a couple of follow-up questions:

Based on the facts and these findings we found no cause for further investigation. We believe, based on the evidence, that while there were some minor errors in the book, which Walker intends to correct, there was no research misconduct per our definition (here).

I have to say, I find it very surprising that you don't think there's any cause for further investigation. After all, I.A.b states:

Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record (emphasis mine).

As documented by Guzey, Walker clearly omitted data by removing a bar from a chart when reproducing it. The omission makes it seem like the research clearly supports his hypothesis when, in fact, it doesn't. How did you conclude that this does not fit your definition of research misconduct at least well enough to warrant further investigation?

This individual found that Professor Walker had included a link to the allegations on his website, that he has addressed the issues raised, and that he has announced his intention to correct any mistakes in the next edition of his book.

Where, exactly, does he do these things? If I'm not mistaken, https://www.sleepdiplomat.com/ is Walker's website, and https://www.sleepdiplomat.com/author is the section about Why We Sleep. But I haven't found any link to the allegations there.

UC Berkeley:

Thank you for your interest in this matter, which we have pursued in accordance with our policy. In conversation with Walker and with the professor who conducted the inquiry, the conclusion was that the bar omitted from the graph on the book did not alter the findings in an appreciable way, and more importantly, that the bar was not omitted in order to alter the research results. The other errors Guzey cites are similarly minor. Walker transparently refers to this errors in his blog here: https://sleepdiplomat.wordpress.com/2019/12/19/why-we-sleep-responses-to-questions-from-readers/#6_or_fewer, and announces his intention to repair them in the next edition of the book. It seems that there is a difference of opinion as to the significance of the errors and the omission of the bar from the graph, and difference of opinion is explicitly addressed in our policy here.

Note: The link in the email was broken, so I fixed it. Originally, the link target was "http://d. Research misconduct does not include honest error or differences of opinion. ' (§ 93.103, 42 CFR Part 93)."

Me:

Thanks for the clarification.

Walker transparently refers to this errors in his blog here: https://sleepdiplomat.wordpress.com/2019/12/19/why-we-sleep-responses-to-questions-from-readers/#6_or_fewer, and announces his intention to repair them in the next edition of the book.

The article you're linking to

• does not mention the chart with the missing data;
• is not on any of Walker's official websites;
• does not mention Walker by name; and
• is, as far as I can tell, not linked to by Walker anywhere, e.g. his websites or Twitter feed.

Note: As of January 2021, the article does, in fact, mention the chart. I don’t know if I made a mistake at the time or if a mention of the chart was added later.

I don't understand how this is acceptable, not to mention transparent. It seems pretty clear to me that Walker isn't properly owning up to this.

It seems that there is a difference of opinion as to the significance of the errors and the omission of the bar from the graph, and difference of opinion is explicitly addressed in our policy here.

It's clear that Walker omitted "data or results such that the research is not accurately represented in the research record." I don't see how the sentence about difference of opinion is relevant in this case.

Anyway, I feel like we're getting a bit lost in the weeds here. The reason I'm unsatisfied with your response is that I think research integrity in general is very important for society, and in this specific case I personally spent quite a lot of time and resources on Walker's book. I bought a pile of them which I gave away to friends and family, and I got gear to track and improve my sleep.

Guzey uncovered many errors by thoroughly reading a single chapter of a book Walker even cites in his own papers. And Gelman says that we've entered research misconduct territory. Now, I don't know what to believe. I worry that science dies by a thousand such cuts.

Meanwhile,

• the book is still being sold and read;
• it's likely that a lot of readers are unaware of the problems;
• it's not at all obvious how readers are supposed to become aware of the problems;
• the quality of all the other chapters is anyone's guess;
• there's no timeline for an updated edition that I'm aware of; and
• UC Berkeley is tarnishing its reputation by not getting to the bottom of it.

Therefore, I'm asking you to please reconsider initiating a formal investigation.

UC Berkeley:

We have completed the inquiry into Mr. Guzey's allegations to our satisfaction. I don't wish to go into the weeds either in order to contest any of your statements, especially given the current crisis situation we are all experiencing, so I will need to conclude our conversation here. I hope that you are staying in good health.

Me:

Ok. Thanks, you too :)

Me (a few days later):

As I've slept on it, I don't feel good about putting this issue to rest. So I'm thinking about publishing a write-up (including our full email exchange for transparency).

I can't immediately see any problems with making our conversation public, but I wanted to run it by you in case there's anything I've missed. Do you have any thoughts on the matter?

UC Berkeley:

My understanding is that my university email communication is a matter of public record, and I try always to communicate with that standard in mind. There is no prohibition that I know of against publication of my emails with you, in other words. I would just ask that you contextualize my communications in a transparent and accurate way. I do not mean to enter into a debate on the issue of misconduct, but I have followed up on the allegations against Walker according to our policy, as I would with any allegations of misconduct, which we take seriously.

Thank you for asking.

Me:

My understanding is that my university email communication is a matter of public record, and I try always to communicate with that standard in mind. There is no prohibition that I know of against publication of my emails with you, in other words.

Great, thanks for clarifying that.

I would just ask that you contextualize my communications in a transparent and accurate way. I do not mean to enter into a debate on the issue of misconduct, but I have followed up on the allegations against Walker according to our policy, as I would with any allegations of misconduct, which we take seriously.

Yeah, I'll try to be as accurate as I can and be clear about what is fact and what is opinion. Also, I think we should expect a lot from institutions and very little from individuals. Therefore, I view this as a series of institutional failures, not personal ones. So I'll do my best not to be personal.

Me:

I've written a draft and attached it to this email. I wanted to run it by you in case I've made any errors or been unfair in any way. So if you have any comments, please let me know.

Conclusion

I would love to do a root cause analysis to figure out what went wrong where, but I unfortunately don't have the necessary access or resources. And, given the incentives involved, it might not be possible. So I'm left with speculation, which I'll refrain from here. That said, I think it's pretty clear that UC Berkeley needs to step up their game. For the publishers to do anything at all would be a great start.

Further reading

While this post is about a specific story on reporting research misconduct, it's also about whistleblowing more generally. My actions probably have no negative impact on neither my career nor my mental health, but others are not so lucky. To read more about the ugly side of whistleblowing, check out Retraction Watch's interview with whistleblowing researcher Kathy Ahern about her 2018 paper Institutional Betrayal and Gaslighting: Why Whistle-Blowers Are So Traumatized. The Financial Times also has a story on it, but that article is behind their paywall.

Acknowledgements

Thanks to Alexey Guzey, Andrew Gelman and Anna Hoiseth for reading drafts of this. Thanks to Kathy Ahern for telling me about her research on whistleblowing.