Mistaking data consistent with your hypothesis for data establishing your hypothesis is a surprisingly common mistake, even for highly trained, experienced scientists.  The subjective experience is common: you develop and carry around a theory on some topic and over the course of your day, you run into evidence (anecdotes, or other scientific findings) that would be predicted by your theory.  So you think, “hey, that fits too, my theory must be right.”

That’s fine for informal theory development, but when you want to really test your hypothesis, it doesn’t work.  After you identify data consistent with your theory, you need to figure out what theory that data actually rules out.  The problem is that it is often the case that competing theories make similar predictions, so data fitting your theory doesn’t prove the alternate theory wrong.  This is why the process of doing science generally requires collecting new data that is carefully designed to discriminate between alternate theories.  This requires both (a) figuring out the alternate theories and (b) designing a good experiment — and so it can be hard.

An interesting example that has been popping up again in popular discussion is the question of the genetic contribution to IQ.  This is an area where everybody thinks they have a strong theory and it’s remarkable how poorly almost everybody does at considering alternates.

For example, you think genes contribute strongly to IQ and you notice that apparently IQ-related success seems to run in families, so you think “Aha! My theory is supported.”  Maybe you’ll even correctly spot the ‘null hypothesis’ that “IQ doesn’t run in families” as being disconfirmed by these data.  But the real alternate is “environmental factors determine IQ more than genes” and since families largely share environments, your data don’t discriminate.

Note that this doesn’t mean we know which theory is correct.  Both theories are consistent with the observed data and no strong conclusions can be drawn.  I imagine this is very frustrating to non-scientists because you’d rather than a clear answer than a perpetual state of uncertainty.  Scientists have to live with uncertainty all the time and it can make it tricky to talk about your work — you want to make simple statements, but you don’t want to overstate your confidence.

The topic has come up because Charles Murray is once again in the news, happily going around talking about his theory that (a) genes are a major/primary determinant of IQ and (b) these genes vary substantially across race.  If you know the actual science being done around this area, you know that neither of those statements are established to the point of ruling out any plausible alternatives.  Even setting aside the question of “what is measured in an IQ test” we know for sure that genes have an effect, education has an effect, and there is increasing evidence of other non-education environmental effects (lead, stress, nutrition).  Nobody knows the relative importance of these effects — and really careful thinkers are also well aware the relative values change across samples within the population (e.g., nutrition effects don’t count for much across a sample that is all well-nourished across the lifespan).

But if somebody like Murray presents the messy data, knowing that a lot of racist listeners are going to simply hear confirmation bias, and then make no effort to argue for non-racist intervention policy to improve the environment (that is entirely supported by the data he presents), then the rest of us can confidently rule out the hypothesis that Murray isn’t a racist.

If you are really interested in the science in this area, there are much better people to be paying attention to:

https://www.vox.com/the-big-idea/2017/6/15/15797120/race-black-white-iq-response-critics