The original experiment, conducted in the 1960s, involved 29 participants. Of these 29, only 6 were able to correctly infer the rule on their first try. This means that they were able to learn enough information from repeated spoken sequences, and their feedback, to come to the correct conclusion without making any mistakes (i.e. they learned from their repeated trials of sequences that fit or break the rule). 10 out of the remaining 22 individuals (one had given up at this point) got it right on their second try, still leaving 13 individuals who had not yet determined the rule. It took a total of five attempts for everyone to either correctly determine the rule, or give up entirely.
This task is surprisingly difficult for people, even those who have studied logic. The reason that most of the subjects of this experiment, and most people who attempt the test since then, fail, is because of confirmation bias. When giving sequences, people generally think of sequences that confirm their thoughts on what the rule is, but they fail to give sequences that may falsify their idea, thus missing crucial data.
From a logical perspective, there are two things you need to do to properly confirm your hypothesis as to what the rule is:
- Give sequences that abide by your rule to confirm that they do fit the actual rule. These should be varied.
- Give sequences that contradict your rule to confirm that they do not fit the actual rule. These should also be varied.
The experiment, and subsequent experiments, show that people always follow the first rule, but they seldom follow the second rule.
This is what I was referring to earlier when I stated that this experiment exemplified a fundamental aspect of human nature. People often seek information to confirm their beliefs, especially when those beliefs are challenged, but they almost never seek information to disconfirm them. In fact, many are unwilling to even accept the possibility that their beliefs are wrong in the first place, an important pre-requisite to ensuring your beliefs actually reflect reality.
In my book Method Matters, I discuss how the author of Sherlock Holmes himself, Arthur Conan Doyle, developed an obsession with the existence of fairies, not in the Sherlock Holmes books, but in real life. Two teenage girls created fake photos of fairies using cardboard cutouts, and upon discovering this, Doyle spent more than a decade seeking out evidence to confirm their existence while dismissing numerous sources of evidence that proved otherwise.
Imagine how different this world would be if everyone were to always apply both rules as a natural course of action. Imagine how many fewer arguments you would have at the dinner table where “that one uncle” expresses certain dated opinions. Imagine how much our quality of life would improve if the millions of people who let their own misconceived opinions and egos get in the way of progress were to suddenly reform. This experiment hints at a fundamental human defect that takes years, or even a lifetime to overcome.
As a brief appendix, I’ll outline the technical details of the logic behind this experiment. Imagine two statements P and Q, and an implication P -> Q, which reads “if P is true, then Q must also be true.” In the context of this experiment, Q is your hypothesis about what the rule is, and P represents three number sequences you give to confirm or deny your rule, so P1 is the first sequence, P2 is the second, etc.
To properly find the rule, you need to give sequences that both confirm and deny your hypothesis in order to get the information you need based on the responses you get from those sequences:
- To confirm your hypothesis includes correct sequences based on the rule, give sequences corresponding to P->Q. In other words, give sequences you think fit the rule to ensure that they actually do.
- To confirm your hypothesis doesn’t include incorrect sequences, and to confirm your hypothesis is correct as is, use counter-examples corresponding to !Q->!P, where “!” means “not,” or the negation of P/Q. Also, use the for !P–>!Q.
In the second point above, !Q->!P is testing if an invalid sequence will not fit your hypothesis. It can be read as “if my hypothesis is incorrect, it will validate an incorrect sequence.” For example, if you think the rule is “even numbers,” then !P corresponds to a sequence like 1, 3, 5, and !Q corresponds to a negative response.
Note that the order of P and Q is reversed in the first part of the second point. This is called the contrapositive statement. Normally, we cannot directly compare !P->!Q because this could lead to a fallacy of denying the antecedent, where we assume that since a premise is false, the conclusion must also be false. In this specific 2-4-6 task, we actually can do this specifically because it is the case that !P->!Q, which means, if an invalid sequence gets a positive response, your rule is invalid. The reason we can do this here is because it is specifically a fact of this experiment that this is true; we did not derive this from P->Q.
The test of !Q->!P ensures that we don’t make a false rule that doesn’t include valid sequences, and the test of !P->!Q ensures that we can’t make an invalid sequence that our rule does include.
