How to Read Between the Health & Wellness (Head)lines

Dr. Cooper

Welcome to the latest episode of the Catalyst Health, Wellness, and Performance Coaching podcast. I’m your host, Dr. Bradford Cooper of the Catalyst Coaching Institute. And if you’re not surprised by how much you enjoy this week’s interview with Dr. Jeremy Horpedahl, I’ll be stunned. Here’s why, this is obviously a podcast devoted to health, wellness, and performance. Dr. Horpedahl is a professor of economics and a scholar. Wait, what? Yeah. Yeah. You heard me right. And seriously, you’re going to find this one really, really interesting. I’ve been following Dr. Horpedahl for over a year on Twitter. His post are consistently among my favorites. He effectively combines humor with frankly good old fashioned reality check at a time when unfortunate majority of health and wellness information is well basically personal opinion or twisted stats. I knew if we could get Dr. Horpedahl all to join us, it would be an engaging eyeopener for all of us, and it absolutely was.

Dr. Cooper

For those of you who are coaches that are listening in, we are incredibly excited about the details of our 2020 coaching retreat and symposium, which were just finalized. If you haven’t heard the latest, you’ve got to check it out at CatalystCoachingInstitute.com, just go to the retreat tab. We’ve got it all in there. You probably already know it’s the only retreat devoted exclusively to health, wellness and life coaches, but that’s not all that makes it unique here in 2020. In the midst of the COVID-19 concerns, we didn’t simply flip our agenda on to zoom and call it a retreat like most are doing. Instead, we started completely fresh. We completely revamped it, making it into an event that while obviously eliminating all the safety and travel concerns that we all have right now resulted in a two day event, that might just be the catalyst for your life and your career going forward. Take a look at the website, if you’re interested, drop us an email Results@CatalystCoachingInstitute.com. I won’t go into details right now with the broader audience, but please reach out if you have questions. Now, let’s get an eyeopening reality check with dr. Jeremy Horpedahl on the latest episode of the Catalyst Health, Wellness, and Performance Coaching podcast. Dr. Horpedahl welcome to the Catalyst Health, Wellness and Performance Coaching podcast.

Dr. Horpedahl

Thanks a lot, Brad. I appreciate you asking me on.

Dr. Cooper

It’s going to be fun. And I think folks might look at the title and go, wait, what stats, what does that have to do with anything? But seriously not joking one bit, your tweets, they are some of my favorites of anyone I follow on Twitter and I follow quite a few folks. They always produce this.

Dr. Horpedahl

Well thank you.

Dr. Cooper

Absolutely. Well, thank you. I mean, it really brings to light some things that I think most people, they just don’t think about. I always get this sense when I read your stuff. I go, Oh yeah, of course. Yeah. That makes perfect sense. But, but it doesn’t at first, like people see the headline, they go, Oh yeah, that must be true. And you come back on the backside and you go, well, let’s kind of dig into that a little bit more. So what got you started doing that kind of stuff? Have you always enjoyed kind of looking under the hood and pointing out what might be obvious when you do the research, but isn’t obvious to the person that’s focused on the headlines all the time.

Dr. Horpedahl

Yeah, I think I’ve always kind of enjoyed that as an, as an economist by training. You know, part of what we do is in many cases kind of enlighten things that didn’t seem obvious, right? So economists focus a lot on things that are unintended consequences or secondary effects things. So I think we always do like, you know, saying, what really does this mean? And economists also work a lot with data. So part of our training is just getting comfortable with that. Before I got my PhD, I, I worked for, the labor department in a different state in South Dakota. And so I got used to, you know, seeing where does this data come from? How is it created? What goes into it? And, you know, that helps you better appreciate it, but also to know the limitations of the data.

Dr. Cooper

Well, let’s get your Twitter handle out of the way. Because people have got to follow. We’ve got to get everybody following you on here because this is just fun, the way you approach things. And there’s always a little bit of humor involved. So what’s your Twitter handle for everybody?

Dr. Horpedahl

JMHorp. So that’s for J M, Jeremy Mark. So it’s M like Mark and then H O R P is just the first four letters of my last name.

Dr. Cooper

Perfect. Perfect. All right. So I want to come back to some of the ones you found most humorous or most disturbing in a few minutes, but can you just kind of warm up the audience a little bit, give the listeners a flavor of maybe one you’ve responded to recently that might wet their appetite if they’re not already following you?

Dr. Horpedahl

Sure. So, you know, one thing that comes up a lot in, surprisingly a lot, not just on Twitter, but in public debates about education is just how much are we funding public education, especially K through 12. And how much does that change over time? And a lot of people seem to have the impression that we’re spending less on education by some measure. So this is what I’ve, actually on Twitter posted many times just kind of showing the very basic data to show no, we actually are spending on K through 12 education a lot more than we were in the past. And even a lot more than, or a little more than we were in the recent past. I think people kind of have this impression and there’s certainly particular school districts or States where it’s true, you know, in one year there was a budget cut, but then I think just for little anecdotes like that, you know, people get the impression that, in some way, defunded, public education, such that, you know, we’re now spending less. But when you look at the data and so this, often my tweet is, as you know, I just post some data, right? No commentary, just like here is the best available data we have. Right? So currently we spend almost 15 and a half thousand dollars per student on K through 12 education, public education in the U S and then if you go back, you know, in 1990, it was about 11 and a half thousand. If you go back to 1970, it was about six and a half thousand. So you know, that number has gone up over time. Now just, you know, presenting that data doesn’t mean everything’s fine. You know, maybe we should be spending more. Maybe we should actually look at not the total number, because some States spend a lot less, right. So it’s just kind of a first peek at the data, but it’s amazing even in prominent newspapers, how often they’ll get that wrong and say for decades, we’ve been decreasing education spending and that’s, that’s just not true.

Dr. Horpedahl

And then whatever point they’re making in the article, to me, I just get distracted with that. Like if you lead off with a statistic, that’s clearly wrong, I’m not gonna be able to trust anything else you’re saying. So, and again, there’s lots of things we could be doing better with public education, but, including that a lot of that money doesn’t actually go to teachers or instruction. Right? That’s another important thing to know. And you can get all this data from the department of education, right? This is not some data that I cooked up or it’s from some biased source. It’s from the us department of education, it’s publicly available on their website. But people just, they just don’t check that, which sometimes baffles me, like, how could you not check that?

Dr. Cooper

Well, as the writer in a, in a major newspaper or magazine. That’s exactly right. And, and the cool thing is you’re not making a value judgment. You’re not coming out and saying, yes, you’re wrong. We need to spend less on education. You’re just saying, well, wait, wait, your lead sentence was, we’re doing this. The reality is we’re actually not doing that. Now, could we do more? Yeah. Maybe, that’s not this conversation. This conversation is, is your data accurate?

Dr. Horpedahl

Right. For me as an economist, we are very data driven people. And, you know, if you start out with a number that’s wrong, it’s just like, you’ve lost me. Even if you make lots of great points throughout the article, it’s just, I’m going to turn it off, or I’m gonna go to Twitter and correct you.

Dr. Cooper

I love it. See folks, that’s why you gotta be following this guy. All right. So one of the things you mentioned in that was the adjustment for inflation. I think it sounds to me, or in following you, it’s kind of come to my forefront that that’s a common mistake. People are comparing this to that, and they’re forgetting that, that adjustment over 20 years, 30 years, 40 years, whatever it is. Can you walk us through that process just a little bit and why that’s such a critical factor when you’re analyzing data over time.

Dr. Horpedahl

Yeah. And I think this is, I think this brings us to another point I’d like to emphasize is that, you know, sometimes people know a little bit about it, they’re a little sophisticated, but then they, they kind of go too far with that. So, you know, the numbers, I just rattled off to you. Those were adjusted for inflation. A lot of people see those numbers and they say, Oh yeah, but you got to adjust them for inflation. I say, no, no, those already are. So I try to always make it clear, you know, when I post things that, yes, it’s adjusted for inflation, but just know that, I mean, an economist, unless there’s good reason not to adjust it for inflation, like sometimes there is, the economist is always going to give you inflation adjusted numbers. Like, you know, we came up with that idea of adjusting for inflation. So we’re gonna try to give it to you in a way, and there are questions about, you know, what’s the right way to adjust for inflation. Those are all good questions, but, you know, yes, you need to adjust for inflation, right? Because the purchasing power of a dollar is different today than it was in 1960. But if someone looks at it and they say, okay, so for 1960, we’re spending $4,000 per student in today’s money. You know, someone says, Oh, you gotta adjust that for inflation. But, I mean, for if we’re really spending $4,000 in 1960 per student, I mean, that would be like an astronomical amount, right?

Dr. Cooper

Yes, and a house costs $17,000 back then.

Dr. Horpedahl

Yeah, so some people on this data have had a little bit of sophistication, but then they don’t actually know what they’re talking about.

Dr. Cooper

And that’s the reason we wanted to get you on here. Because when it comes to health and wellness and even performance things, there’s such a focus on headlines, fads, individual case studies, that kind of stuff. And so people that are coaching or people that are coming alongside other people and trying to help them, they’re falling into these traps. And that’s why I wanted to bring you on and talk about just more generically, we won’t even get into that many specific health and wellness things. Although I think we can touch on a few just what people can look for and what they might miss. So let’s jump into that. What are some of the most common mistakes people make when they’re reading or using statistics in their writing?

Dr. Horpedahl

Yeah. And I think we will maybe, a little bit get to talk about some, some things that are specific to health. But yeah just in general, again, as an economist, here’s a number that people talk about a lot in our field, which is the unemployment rate. And one common mistake is just not knowing what goes into that number, right? So you see this, the Bureau of labor statistics reports this every month and the media reports when this number comes out, the unemployment rate, and especially in what’s going on in the past few months with our economic shutdown. And I think it’s even more important to know what does this number mean? It’s a useful number, but it has limitations. I think just knowing what goes into that. So just in case your listeners don’t know, the unemployment rate is the percent of people that are not working, but what you’re dividing by is either people that are working or people that are looking for work. So people that aren’t looking for work are not included in this number. So if, for whatever reason, either, you know, you’re in college and you’re not looking for work, you’re retired, you’ve just given up looking for work. And none of that’s included in this number. So when you see that number, it’s important to know that. But then again, there’s the, there’s a subset of people that have some sophistication, they say, Oh, it doesn’t include people who have quit looking for work. Therefore it’s a garbage measurement, just ignore it. You know, that’s also not true. That’s going, that’s going too far. Right? So there are other measurements which try to take account of the people who have stopped looking for work for various reasons. But I think the most common error, I see in a lot of data, is just not knowing what the data you’re looking at means. And then making judgements, recommendations without even knowing what goes into this number.

Dr. Cooper

So, if someone is reading an article and it could be anything, reading about a new diet or they’re reading about a new exercise program or how this weight program creates the greatest abs ever, or whatever it might be. When they’re reading something that includes statistics that are quoted, what would be some red flags, what would be some things as they’re looking at, they go, Oh, wait a minute. I don’t know that I can trust this article because X, are there certain questions that we as readers should be looking for regardless of the topic?

Dr. Horpedahl

Yeah, absolutely. There are, although it’s, the red flags are different all the time. So one thing which I think is a big red flag is, is if there’s no clear source provided. Now on social media, it’s very common to not include a source. Although that still frustrates me because, you know, it’s just a link. It’s actually very easy to do, especially in a news article, usually, you know, news articles don’t have a hyperlink to a, you know, a link in them. But if they only even say where has this data come from, to me, that’s a big red flag either, either because they don’t want to tell you where it came from, or just because it indicates to me that the person writing it, hasn’t actually done their research, right. They heard this third hand from someone else. So the person writing it, if they’re not accurately describing the source, then I have to go myself and say, where’d this come from, I got to Google around and figure out what’s their source. To me, that’s all an indication that they’re probably going beyond what they know. Now for, you know, for science, journalism, and science reporting, they’re often not going to be an expert in every area of science they’re writing about, but, but still, if they’re not giving me a source, that’s a red flag. Another thing that I do, and this is going to vary by person, but I try to use things that kind of bother me to look into things. So for example, I’m a very optimistic person about the world. I know that it’s a, it’s a hard time to be an optimist right now with what’s going on in the world. But you know, when I see something that I think seems overly pessimistic for me, that’s a, that’s a motivation to dive into the data and to look and actually see where it is.

Dr. Horpedahl

So, especially if there’s something where it’s, there’s no source and it’s really pessimistic, I say, I just got to look at this. You know, now sometimes it turns out it’s correct. And I try to be honest sometimes on Twitter, I’ll say, Oh, I thought this was wrong, but actually it was true. You know, I looked into it and it was true, but I think people can, sometimes we say, we want to try to not, not be biased. But I think you could use your own biases to say, I’m going to investigate this. And then hopefully they’re also, and there are, I know, you know, people in the world that are very pessimistic and hopefully they’re also checking things that seem too optimistic, right? So this is kind of an ideal, you know, public, public model of science. But hopefully if we have people with different biases, looking at investigating things, we can get, you know, some sort of truth coming out of it and better understand what’s going on in the world. Maybe that’s a little too optimistic of me, but, you know, those are kinds of things that I think get me motivated to look at things. You know, optimism versus pessimism isn’t the only kind of biases obviously. But, but you know, what’s interesting to you or does something challenge a core belief that you have about, you know, how the world works or how science works. You know, that’s OK to investigate that. Now sometimes, you know, things that challenge your core beliefs are actually true. Some of our core beliefs might be wrong. So it’s, it’s actually, I think it’s still healthy to look into those things. Even if ultimately you, you find out you were wrong and the source was right.

Dr. Cooper

But, but you learn in the process. So I’m going to repeat back for everybody. Just emphasize this. Is there a source cited? Folks when you read an article, you read a tweet, you read a post somewhere and it’s just clearly somebody’s opinion, just immediately put up that big red flag and go, wait a minute. I don’t, I can’t trust this. And it doesn’t matter how many people posted it. It doesn’t matter how many friends liked it. If there’s not a source cited, you’ve got to take a step back. We’ve got to take a step back and say, hang on a second. Is this legit? Or is it just a bunch of people shouting from the rooftop louder than everybody else? So good. Really good. All right. So what about when graphs or other visuals are used? That seems to be a common, I hate to use the word trick, but it’s kind of a trick that people will throw out this graph. And so you glance at it cause everybody’s in a hurry and you go, Oh my gosh, this must be true. Look at that graph. Look at that visual. Any tips on what to look for when it comes to those things?

Dr. Horpedahl

Yeah. I’ll first say that, you know, graphically displaying information is a real art and it can help to, you know, I mean the picture being worth a thousand words, that’s a cliche for a reason. I mean, it can be a very nice way to convey useful information. I mean, some of my favorite economics articles, you can often pick out the key chart or tables from the article and say, this table tells you everything you need to know about the article. So I think that, you know, graphs and visuals are very useful, but there’s also a lot of ways you can be manipulated by those. So I’ll just go, I’ll give you a couple things to look for. And then maybe we can talk about some of the humorous ones, we’ll see with some of these examples. So, you know, anytime you have a graph right there, there are two axis. So there’s, we usually call it X, which runs across the bottom. Right. And usually that’ll display something like time, right? So this will be, you know, 2001, 2002, 2003, if you’re displaying some data over time. And then there’s the Y axis, which is the vertical axis, which is usually where you’re going to be displaying your data. So the, each of those axes can be manipulated in different ways. And usually it’s by only showing you a part of it in some way. So the Y axis can be manipulated in a couple of ways, but one of the big ones is to not include a zero on the Y axis, right, it starts somewhere. So let’s say the graph starts at 50% and they’re trying to show, you know, people’s, you know, favorites, how many, what percent of people like these different flavors of ice cream, right. And let’s say, you know, they start the graph at 50%, let’s say 50% like Rocky road and 60% like mint chocolate chip ice cream. Right. If you start the graph at 50%, I’m trying to describe this, right. The graph is 50% and one’s at 50 and one’s at 60. It might look like, depending on how you do the axis, it might look like twice as many people like the mint chocolate chip as like the Rocky road, when really there’s only a 10 percentage point difference, right? So you want to look at that axis and unless there’s a really good reason for not including the zero, you want to have that zero. You want that line. You want that, especially like on a bar charts, you really want to see that zero. So you can see what’s the difference between these two numbers.

Dr. Cooper

Well, and we probably see that a lot with weight studies and I shouldn’t even say studies, but weight claims, you know, this one causes this much loss, put it in a graph. And the fact that one group lost five pounds, the other group lost six pounds, but we started at the 4.8 mark. It makes it look like group two that did X, Y, Z special diet that it’s the magic thing nowadays actually lost all this extra weight. And the reality is it was one and a half pounds out of, you know, a 200 hundred pound person. So I think that’s the type of thing that you’re talking about there.

Dr. Horpedahl

Yeah, absolutely. So, yeah, just, you know, be very careful of that. And maybe, you know, in this study, you know, a one and a half pound difference, maybe that is meaningful in some sense, but you can have the graph display in a certain way that it looks like a huge difference when it’s actually just a tiny difference.

Dr. Cooper

Right. That’s a great point. So it still might be meaningful, but it’s not the magic formula. It’s not this end all be all that the graph is unfortunately showing. Yeah. Good.

Dr. Horpedahl

So another way in which a graph might be manipulated is on the X axis. So that’s where it’s usually showing time. But depending how you arrange it, it might show your data, but, and this is what people usually refer to as cherry picking. So you see this a lot with trends over time. If someone picks a five year period of time and say, Oh, look at this, right. If we go back to the education, spending might be that one, right? Look at this, this number has been, it’s declined, 20% things are really going down. Well, if you’ve picked a particular five year slice of time, unless there’s a really good reason for picking that, right? Like maybe there was some big policy change that happened in there. You know, I always want to see the full data, like give me the full data. I can, I can then visually I can zoom in on those five years, but show me the full data. A lot of times, if you look at a short period time, like, let’s say a graph only shows you half the year. Like what about the other half of the year? What about the other six months what’s going on with this data? It may have just been a blip. I mean, data goes up and down all the time. There’s kind of randomness to all the data in the world that we observe. If you’re trying to, unless there’s something really specific they’re trying to make, you know, especially if they’ve cut this off at some point in the past, like the data stopped in 2010, but what stopped in 2010, there’s gotta be more recent data than that, what’s been going on with it recently.

Dr. Horpedahl

So I want to see as much data as possible. And then if we need to zoom in, we can do that. But that’s the way the X axis is manipulated. So that here’s a couple of other ways that are different from axis manipulation. One of them is focusing on the share that say each group gets of something versus the total amount. Right? So, this is, especially when we look at things over time, looking at the share that each group has, may not always be the best way to look at something. Sometimes it might be, but especially if the total amount of stuff. So here’s an example that I’ve seen come up a lot lately is looking at the amount of wealth that each generation has and how that’s changed over time. If you look at the, say at, you know, their age right now, how much wealth gen X has versus how much well the baby boomers had at the same age, right?

Dr. Horpedahl

So, you know, at the age of, you know, gen X is what they’re around 38, 39 up to, you know, their late fifties right now. If you look at boomers, when they are the same age, baby boomers had a lot bigger share of the wealth at the time. But if you look at how much wealth each person actually has today, gen X is about the same as baby boomers. They have about the same amount, but they have a much smaller share. So then we think, well, why is that? Well, there are more people alive today and there’s more wealth out there in the world. So even though gen X has the same amount of wealth as their, you know, their parents might have had, they have a smaller share of the total just cause there’s more wealth on their total. So sometimes looking at the share does make sense. And in most cases, you know, if someone is showing you, you know, everything adds up to a hundred percent, a graph like that over time, it might be trying to manipulate you. So I would at least want to see the data, not as just shares, I’d want to see the absolute amounts. I want to see it both ways. If that, if that makes sense or if you want me to elaborate at all.

Dr. Cooper

No I think that’s good. It, just reminds us to continually ask those questions, continue to say, well, wait a minute. What is this really? Is this truly the best way to show it? And how have others shown it that have looked at this in a non, I don’t know, prejudicial way, if you will. What about some humorous or maybe disturbing inaccurate use of data, graphs, statistics, that you’ve seen recently that might kind of stick in folks’ brains as they, as they leave this?

Dr. Horpedahl

Yeah. Well, I’ll describe, I think a couple of examples that are kind of, I think, classic ones that tie into some extent to these things I’ve just told you, and maybe possibly we can get these posted somewhere for people to see, but I’ll try to, I’ll try to describe them, describe the graph. One of my favorite ones with the, the truncated axis is something which, you know, isn’t like a political thing. It’s about, it’s something that someone posted on a news article about KFC had a new menu item out called the crispy chicken twister. And they were comparing the calories in this versus the calories in some other food products, like a piece of pizza or a taco, and the chicken one was kind of in the middle, right? So it was about 650 calories from this KFC crispy chicken twister. But then when you look at the way the graph has presented, it looks like the pizza almost has zero calories. And that the chicken has 10 times as many calories. And it looks like a taco has twice as many calories as the chicken twister, but really what they’ve done is they’ve narrowed the range of the axis here from 590 calories to 720 calories. So even though, you know, all these things are within you know, plus or minus 50 calories of each other. And so I don’t know exactly what some point someone’s trying to make this graph, but when you look at it, it’s just very deceptive. And I don’t know what we even learn from looking at that. It was also just very poorly done because it didn’t, it listed like the other brands like taco bell and burger King. It didn’t even say what the item was. It was, it was, it was just this kind of very poorly put together graph. But you know if you’re trying to make dietary choices based on that, you know, knowing the calories and the fat is important, but you know, presented this way, I just, I just always see this and say, this is just a clear example of someone, it’s not even like KFC was manipulating. Just a business journalist put this graph together to try to say, here’s this new product. Here’s how it compares to these other things. And it just, it’s just not useful the way it’s.

Dr. Cooper

Yeah. That’s the word that came to me is there’s, there’s no value in it except to maybe sell newspapers because it grabs your attention. And folks remember, that’s oftentimes why people put graphs and articles together is to sell newspapers or to get re shares and that kind of stuff. So, yeah. Great example.

Dr. Horpedahl

Just a table of numbers would have been more useful, right? Like this one’s 590, this one’s 650 and this one is 720. That actually would have been more useful, a nice font and maybe some colors, but sometimes a table’s actually better.

Dr. Cooper

Yeah, absolutely. Absolutely. Good. I love that one. Any other similar to that?

Dr. Horpedahl

Yeah. Here’s another one again, I’m trying to be very apolitical here. It’s about food again, but this is a, this is in the category that we sometimes call infographics. So infographics are where, and not to pick on any one publication, but USA today used to always on their front page, have an infographic. Here’s some information you should know, presented in graphical form. And sometimes they just try to get too creative with these. So here’s a favorite of mine. This isn’t from USA today. This will not ruin anyone’s reputation here, but, this one was showing, they were trying to show in a given year, I think this was in 2002. From 2002, they were trying to show the amount of sales, each fast food chain had, right? So they got McDonald’s and burger King and pizza hut, taco bell and Starbucks, but they did it using their corporate logos instead of just a bar chart or a table.

Dr. Horpedahl

So, so each logo was supposed to represent their sales. So in this, just to pick one example, burger King in this year had about three times as much in sales as Starbucks. So what they did in the graphic is they made the burger King logo three times as tall as the Starbucks logo. But of course, a corporate logo is a two dimensional object, not a one dimensional object. So if you make it three times as tall, it’s also three times as wide meaning the, the logo is actually nine times the size. So you look at the burger King logo, it looks like it’s nine times as big as the Starbucks logo. Even though the sales are only three times as much. Now they did also write the numbers on there for us. So that was nice. But the visual impression, and then McDonald’s, McDonald’s looks like it’s, you know, nine times as big as burger King when it’s only about three times as much, again. And also for some reason, I guess this was just the time in 2002, they stuck the map of Afghanistan in there as well. So here’s the GDP of Afghanistan. Of course Afghanistan is not circular at all. Right. It’s kind of, it’s wider than it is tall. So, you know, then they stuck that in there and said, you know, Afghanistan’s in between burger King and McDonald’s, and just kind of, they threw that in to. What are you even trying to do here? I know what you’re trying to do. None of this is really, there’s a much simpler way to present this, that would convey the same information. It would be useful. Yeah. It would be useful. And you maybe wouldn’t run into copyright issues too by using their logos.

Dr. Cooper

All right. Perfect. All right. Just two more. Any suggestions, so listeners are generally in the health and wellness arena or performance arena, and they’re constantly being subjected to extreme claims. It is such a big thing in health and wellness. It’s it drives me insane, but they’re seeing things that are based oftentimes on hand picked statistics. Can you help tune up our x-ray vision in addition to things that you’ve already shared with us?

Dr. Horpedahl

Yeah. I think the biggest thing I would say to look for in this general area, health and wellness, is to look at what type of study it is. So if you’re, you know, anytime you see a result, especially if you’re skeptical of it, look at what type of study is. And there’s going to be two main types of studies you’re going to see. One is called an observational study, and one is called an experimental study. So I’ll try to just briefly define these and then tell you which one’s better and why, which is the experimental study. So you want whenever possible to have an experimental study. An observational study, I would say these are usually, they’re not worth anything in most cases, but they’re very widely used. Cause they’re, they’re very easy to do. So an observational data, or observational study collects data. That’s just asking people kind of what they do and we’re already assuming, your people are telling the truth and they’re not, you know they don’t have bad memories.

Dr. Horpedahl

But just, you know, let’s say we’re interested in the, you know, the effects of vitamin C, right? So you ask people, you know, how much vitamin C are you consuming? And, and then, or you might ask me how much, how many oranges have you eaten and so on and so forth. And then you ask them about various things about their health, right. And, you know, did you get the cold this year? Right. So on and so forth. Right. And so you try to then ask them not just about those two things, right. You’re interested in, you know, does vitamin C prevent the cold. Right. But you ask them about, you know, okay, how much time do you exercise? You know, where do you live, so we know what the climate’s like. You know, what’s your income and they try to control for all those factors. They’ll say, so the study, you know, will say, the headline will say, you know, vitamin C you know, lowers risk of cold, you know, scientists collected data. It might be a big number. Cause it’s very easy to get observational data. We have data from a half a million people and we track them over a decade and we controlled for all relevant factors, such as income and geography. And then we found that yes, there is a relationship between taking more vitamin C, you’ll get less colds. These studies I say are generally worthless because there’s usually some unobservable things that they didn’t collect data on, or maybe not even possible to collect data on. Right. Like, you know, in our current environment, you know, if you wear a mask, right. Might be a very good question to know about whether you have the cold. Right. But even things that are unobservable, like how often do you shake hands with people and you know, do you cover, do people that are around you cover their mouth when they cough, right? Like we don’t, you don’t know all those things. There might be other things just about you, right. Even though they ask you about how many minutes a week do you exercise? Like, there’s still a lot of things about your health, which we don’t know and are probably more important for whether you get sick rather than just how much vitamin C you’re taking. So observational studies are studies like that, where you’re just kind of asking people, did you do these things and does something correlate with, with another thing? Right? So when we hear this expression, you know, correlation does not imply causation, we’re usually talking about observational studies.

Dr. Horpedahl

The type of study that’s better. And it’s kind of in some sense, the gold standard, is what we call experimental studies. So experimental studies are studies where you’re going to have a treatment and control group. So you’re going to have, you’re interested in the effect of something, you know, whether it’s vitamin C or a prescription drug or a particular exercise routine, and you say, okay, we’re going to find a bunch of people. And it doesn’t actually, here’s the thing, it doesn’t need to be half a million people. You could have a really good study with just a hundred people if you do it right. So you can, if you have a hundred people divided by two groups of 50 people a week and you do, there’s a lot of tests you have to do to make sure, you know, are the two groups similar? Right. But then you say, okay, one group, we’re going to give you this drug, one group we’re going to not give it to you. Although they usually give them a placebo drug. So they don’t, so you don’t know that you’re not getting the drug. Right. So we’ll give one group of the people, this drug, one group of the people won’t get it. And then we’re going to see, we’re interested in some outcome, right? Like blood pressure or whether they have a heart attack or do they lose weight. We’re just interested in outcome. Or maybe we don’t know what the effect is. So we’re going to measure lots of things. So again, you ask them, you’re still asking people about things, right? Have you been sick? You know, what’s your weight. So you’re still asking people. But the important thing here is you have two groups of people you’re studying. Now, you still need to control for all sorts of things. If you’re controlling for all the right things. And if you have created two groups of people, which are similar in most ways, you’ve randomly assigned people to each of those two groups, we can have some reasonable certainty that there actually is an effect from giving someone the drug or the vitamin or the exercise routine.

Dr. Horpedahl

There’s still a lot of questions about how big is the effect, is that statistically important? Is it important for, you know, the one and a half pound difference, right? The one and a half pound difference we talked about before that may be statistically different. Right? We could say that, yes it had an effect. But still you might think, okay, am I going to take a, a thousand dollars pill to lose one and a half pounds? You know, maybe, maybe not. Right. So we want to know how big is the effect, but having the experimental study is an important way to do it. Now, maybe you can’t always do this, right? So there are some things where either it’d be very hard to do it that way, or it might be unethical. So one common example is, you know, whether smoking causes cancer, it would probably be unethical to, you know, force half of your group to smoke for 40 years and prevent the other half from smoking, that would be very hard to do and probably unethical.

Dr. Horpedahl

So sometimes in cases like that, we might have to rely on observational studies and then, you know, tie that to other sciences that study, you know, how smoke affects the lungs, right? There are people who actually work on that, physically, right. Biologists and chemists that work on things like that. But in general, if something’s an observational study and you think if you look at it, you say, okay, this is clearly an observational study. It may not say that, but if you read enough studies, and don’t be intimidated by these things, you can read them.

Dr. Cooper

Exactly great point.

Dr. Horpedahl

Even if you don’t understand all the statistics, you can, you can pretty quickly look at something and see whether it’s observational or experimental. And if it’s observational look at it and say, is there a good reason for it to be? Or is it just because this was the easy way to do it, right? Could they have reasonably done this as an experimental study? And if they could have, and they didn’t, then that’s a real good reason to be skeptical of the result. So that’s, I think one kind of general area. Now in areas of health and wellness, I think you’re lucky because you can do these kinds of random trials in most cases. Sometimes they’re called randomized controlled trials or experimental trials in social sciences. Like I’m an economics, but actually it’s very hard to do that. You can’t randomly separate the population in the U S into two groups and give one of them one type of monetary policy and other, the other kinds. So we have to rely in social sciences on natural experiments to kind of approximate these experimental studies. You know, in the physical sciences, it’s actually in most cases, very easy to do this. Sometimes it costs money to do it. Often it costs a lot more than the observational study, but usually it can be done. So if it hasn’t been done, to me, that’s a big red flag. But then, you know, within experimental studies, I think the things that I said, like a lot of times, people will focus very much on this, is it statistically significant? And here’s one of the tricks of science, the bigger number of people who observed, the easier it is to get a statistically significant result. So if you have, you actually have an experimental study with a million people, like if you could actually get that many people, and it’s actually pretty easy to get a statistically significant result, but whether that’s meaningful in a sense that, okay, so this, this drug reduces the chance of heart attack, statistically significant, but how much does it reduce it? Well, it reduces it by half a percent and taking this drug cost $5 million. Well, okay. Probably, maybe it’s not worth it, even though it is statistically significant.

Dr. Cooper

Well and I think that’s a great point. Significant and meaningful are two completely, they sound like the same word. They’re not, they’re completely different.

Dr. Horpedahl

Yeah. Statistical significance just means that the, the result is not due to chance. Or we have a pretty, it’s never even a certainty. We have a pretty high probability that’s not due to chance.

Dr. Cooper

Right. We can’t even for sure say it wasn’t, but yeah, we have a pretty good probability.

Dr. Horpedahl

Exactly. A normal standard would be to say it’s 95% chance that it’s not likely, in other words, if we did 20 studies like this, one of them would be a fluke, 19 would be correct. Right. So, but just because it’s not a fluke doesn’t mean it’s actually something you should act on. Right. So we’d want to know the size of the effect.

Dr. Cooper

Real quick, come back to the correlation versus causation. That’s something you and I deal with all the time. And I’m sure most of our listeners have heard that phrase, but can you clarify that for folks? Cause I think that in itself is a powerful reminder as people are reading studies.

Dr. Horpedahl

Sure. So if we’re looking at any kind of study, right, and we see a correlation between two variables, meaning they kind of move together, you know, it might be age and income. So as people get older, their income tends to go up. You might say, okay, is it the fact that they’re getting older, that our employer is just willing to pay older people, more income? Probably not. Right. And that one, I think we know, as you get older, you have more job experience. You’ve acquired more skills. You have more connections in the labor market, right? Like all these sorts of things might cause your income to go up as your age goes up. But anytime you have two variables correlated, you don’t want to assume that one is causing the other. Because often the biggest problem is which is causing which if age income are correlated? Like, in that one we could say, okay, age happens on its own. So income clearly couldn’t cause that, but for most things, one could be causing the other. So we don’t know what’s causing what, and we don’t know if there’s some third factor that’s actually causing both of them. So that’s why, you know, experimental studies are kind of the best because they really get away from this problem of whether two things are correlated. Because if you are separating out one group and giving them this thing, then you, then you have a pretty high degree of certainty that that must be causing that. Giving them that pill like nothing else has changed about them, that must be causing that.

Dr. Horpedahl

Now once again, in the social sciences, this is very hard to deal with since you’re not usually dealing with experimental data, you’re dealing with in many cases, observational data. And so economists and other social scientists like political scientists and criminologists and sociologists have spent a lot of time trying to develop sophisticated tools to see if one thing is causing the other. But it’s still, it’s very hard to do. But if you stick with experimental type studies that are not in the social sciences that are in the physical sciences or in health, that’s usually a pretty good indication that one thing is causing the other thing that the correlation is one thing causing the other. So here’s another one where, you know, people say that phrase and some people will shoot down any study with this phrase, they’ll say, well, correlation, doesn’t prove causation, researchers tear out their hair. And they say, we worked really hard to try to make sure that this is actually one thing causing the other. And some pundit will just dismiss it and say oh studies. Yeah. Correlation doesn’t prove causation. I mean, if you think about it, you step back and think about what someone is saying when they say that is, they’re saying if they dismiss any study with that is you’re saying, you know, science doesn’t actually work and we can’t actually learn anything about how the world works, which is certainly not true over time. We’ve learned a lot more about how the world works, how the human body works and, and a lot of that is from experimental type studies. But yeah, so that’s, that’s an important phrase to know, but it’s another one where don’t take it too far. Don’t, don’t let a little knowledge make you be dismissive of everything. There, there are good ways to try to look and see if one thing is causing another thing, but many studies do make that error and it is useful to be able to know when are they making that error.

Dr. Cooper

Beautiful, beautiful, last question. Any, just kind of wide open, but any final words of wisdom for the listeners who are super committed, our focus on this podcast is evidence based practices. Let’s not chase the headlines. Let’s not follow the fads. Let’s go with evidence. So any last words of wisdom for the folks that are saying, that’s why I listen, this is my thing. I want to make sure that my practice or my coaching or my advice to friends or whoever it might be is not fad-based. Any finals kind of tips for them going away.

Dr. Horpedahl

Yeah, I think that for a person, especially someone who’s say got a college degree or a master’s degree, I think you can probably do better than you think at reading an academic study. When you, if you want to check a source the first time you do it, you might be so intimidated. You might pull this up and say, I don’t even know what they’re saying here. I can’t judge whether this, this source is valid. I think you actually can. And you get better at this over time. I think in your area, one of the biggest things you want to look for is, is it observational or experimental? And if it’s experimental, you want to look at how big is the effect, not just are they related, right. Not just vitamin C reduces cold, like, okay, how much does it, like what’s the effect size. So being able to, to read a paper, to skim through it and say, okay, is this observational? If it’s not, if it is, probably just throw it out, if it’s experimental, you know, what is the effect size? I think you can do this much better than you think you can, and you get better at it over time. And looking at a study, I want to emphasize is, we’re not always trying to poke holes at things and say, Oh, that’s wrong, I’ve debunked that. We want to figure out, you know, what are the good studies and what is actually good information to, to better our health and wellness, to better under our understanding of the economy, to better our understanding of, you know, all sorts of political things.

Dr. Horpedahl

We want to be able to have scientists do research. You know, the average person should be able to look at that and say, okay, I couldn’t do the study. I don’t know what everything means, but here’s a few key things to look for. And you know, also sometimes you try to look for a source, especially if you’re not an academic like me, you’re not at a university and you try to pull it up and says, Oh yeah, you can read this article. It’s $35, $45 to read one article that may be junk anyway, go to your library. Any library, doesn’t even have to be a university. You could go to your local public County library and you can usually either get it there, or they have a program called interlibrary loan, which is where they will go to other libraries that are in their network and they will get it to you. And usually if it’s not available to you immediately through your library, libraries pay for access to actual journals. And they don’t always have all of them, especially if it’s a small County library, but usually 24 to 48 hours. They’ll, they’ll email you the PDF. I mean, it’s a very, it’s an almost automated process where they’ll go and say, okay, it’s from this journal this year, what libraries have access to it? Requests will go out to a library that has access to it and I’ll send it back to you. And it’s all on board with copyright. You’re not stealing from anyone. There are repositories online where you can get stuff, the copyright issues on those are little, I’m a little skeptical of some of those places. So in 24 to 48 hours, if you have, you know, a public library or even university library, you can go to, I mean, you can, you can get these things pretty quickly. And I know things move at the speed of light these days with information, but you know, you can wait a day to get, to get an article. You don’t need it right now and you can, you can get just about anything. And I think if you read these things, you get better at knowing what to look for and knowing what’s, what’s a good study, what’s not. And for the good studies, you know, how important is the result?

Dr. Cooper

Good stuff. Dr. Horpedahl, really appreciate it. Thanks for jumping in with us on this stuff.

Dr. Horpedahl

Yeah, I really appreciate you asking me. So thanks, it was great talking with you, Brad,

Dr. Cooper

Keep up the good work. I’m going to keep following you on Twitter. We’re going to have a lot more people listen to you now. So let’s keep that stuff coming. I love it. Alright. Take care. Well, now you can see why we invited Dr. Horpedahl to join us and why he’s been published so widely and is adding to his Twitter followers every day. Thank you to you for tuning into the number one podcast for health and wellness coaching. And thanks for sharing it with others. Next week, we are really excited to welcome dr. Vincent Pedre the bestselling author of Happy Gut to the show. I just finished reading his book and we’re going to dig into how tuning into what’s happening within our gut can positively influence so much more than we ever realized in terms of our overall well being. If you’ve enjoyed the podcast, you might also be interested in the YouTube coaching channel. It’s literally youtube.com/coaching channel, where you’ll find a growing library. I think we’re up to around 60 videos now that compliment the podcast topics. Now we know how to see through the fake headlines and claims let’s go get better based on the evidence, not the fad and not those who shout loudest on social media. Let’s do this together. This is Dr. Bradford Cooper signing off, make it a great rest of your week. And I’ll speak with you soon on the next episode of the Catalyst Health, Wellness, and Performance Coaching