The Fallacy of Market Prediction
Spoon boy: Do not try and bend the spoon. That’s impossible. Instead… only try to realize the truth. Neo: What truth?Spoon boy: There is no spoon.
Neo: There is no spoon?
Spoon boy: Then you’ll see, that it is not the spoon that bends, it is only yourself.
~ The Matrix
Traders and investors are faced with a seemingly intractable problem when participating in markets — they must place bets on future outcomes which are unknowable even as they’re cognitively wired to be risk averse when faced with uncertainty.
Put simply: unknowable future + human desire for simple answers = problem.
In this piece we’ll briefly discuss how most people square this peg and also the correct framework you should use instead.
The Brain Is Lazy
The brain is an amazing piece of hardware. It’s comprised of roughly 90 billion nerve cells which are linked together by trillions of synapses. It’s estimated that the human brain operates at approximately 1 exaFLOP. That’s equivalent to a billion billion calculations per second, multiples faster than what the largest supercomputer can accomplish over any meaningful period of time.
But all this processing power takes up a lot of energy — a bit over 20% of the body’s total usage — making it by far the most energy intensive organ.
Because of these large energy needs, the brain has evolved like the rest of the human body for maximum efficiency. After all, the primary driver of evolution is survivability, and energy sources (food) used to be scarce. So the human body evolved to balance a brain that can solve complex tasks — helping us move to the top of the food chain — with a need to efficiently use the body’s energy.
As a result, we got a brain that’s awesome, but also lazy.
It uses a host of cognitive tools (ie, heuristics, biases, anchoring, etc.) as a way jump to conclusions without expending too much energy consciously thinking through every single problem it’s faced with.
For the most part, this system works great. The brain is unmatched in its ability to subconsciously perform pattern recognition and come to an adequate conclusion with little information.
This process is described in Daniel Kahneman’s classic book Thinking Fast and Slow. Kahneman breaks down the cognitive functioning of the brain as follows:
System 1 operates automatically and quickly, with little or no effort and no sense of voluntary control.System 2 allocates attention to the effortful mental activities that demand it, including complex computations. The operations of System 2 are often associated with the subjective experience of agency, choice, and concentration.
The automatic operations of System 1 generate surprisingly complex patterns of ideas, but only the slower System 2 can construct thoughts in an orderly series of steps . I also describe circumstances in which System 2 takes over, overruling the freewheeling impulses and associations of System 1. You will be invited to think of the two systems as agents with their individual abilities, limitations, and functions.Systems 1 and 2 are both active whenever we are awake. System 1 runs automatically and System 2 is normally in a comfortable low-effort mode, in which only a fraction of its capacity is engaged. System 1 continuously generates suggestions for System 2: impressions, intuitions, intentions, and feelings. If endorsed by System 2, impressions and intuitions turn into beliefs, and impulses turn into voluntary actions. When all goes smoothly, which is most of the time, System 2 adopts the suggestions of System 1 with little or no modification. You generally believe your impressions and act on your desires, and that is fine— usually.When System 1 runs into difficulty, it calls on System 2 to support more detailed and specific processing that may solve the problem of the moment.
We’re not going to go into more detail here on the benefits and disadvantages of the System 1 and System 2 model, nor the host of tricks the mind uses to make this system function (here’s a cool Bias Codex). Those have been covered elsewhere and if you play in the markets, then you should just read the book. It’s worth your time (side note: Michael Lewis’ new book is a good primer on Kahneman).
All you need to know is that the brain evolved to seek out simple, linear answers when confronted with complex situations, especially where there’s an information deficit.
Basically, the brain is not well suited for markets.
Squaring the Peg: How Most People View Markets
For the many advantages that our brains offer, they can be extremely dangerous to our trading capital.
That’s due to the following two reasons:
Our brains operate on pattern recognition and therefore like simple, linear answers.
Information overload results in cognitive tunneling (System 1 thinking) and our minds instinctively follow the path of least resistance when confronted with complexity.
The problem with “reason 1” is that markets are not simple or linear. They are endlessly complex and dynamic.
Therefore “reason 2”, following the path of least resistance in coming to conclusions, is the opposite of what you should be doing.
These cognitive handicaps lead people to think that prediction in markets is possible. It’s not.
The prediction fallacy is based on the assumption that the future is fixed (linear) and all important variables are known (total information). This is, of course, patently false.
Regardless, many of us stillsubconsciously use this mode of thinking as our primary mental model.
Here’s a sketch of how most people think about markets.
Simple and linear.
Here is how slightly more “sophisticated” market participants think about markets.
This is better than Figure 1, but still wrong.
To think in probabilities is to give a statistical weight to different outcomes. But this statistical weight is useless — sometimes worse than useless — because a number of essential variables will always be unknown. On top of that, you can never know just how many of these unknown variables there are. Therefore the situation is not actually calculable, nor repeatable.
Thinking in probabilities leads to false confidence and suboptimal outcomes.
The market isn’t like Poker where assigning probabilities using known variables and experience makes sense.
In poker, things are quantifiable. You know how many cards are in the deck and you can calculate probabilities for your hand and the other hands at the table. You can then use these in conjunction with bet amounts to arrive at a statistically meaningful edge where you can surmise both pot odds and the expected value of your actions.
Of course, like in all things, there are some exceptions. Quant and algorithmic traders, for example, exploit inefficiencies, quantitative correlations, and other relationships to find statistically meaningful edges. But these are often very short-term and eventually always get competed out of the market. This is why quant funds need to continually evolve to find new edges.
Since most of you aren’t Jim Simons, this exception probably doesn’t apply to you. Most of us are left using our minds in the best way possible to arrive at better outcomes.
How You Should Think About Markets
“As we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns — the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.” ~ Donald Rumsfeld (emphasis mine)
Markets are full of unknown unknowns.
No matter how smart you are, or the amount of advanced tools you have at your disposal, or even how much exhaustive research you do, you’ll still be partially blind when putting on a trade.
And the dangerous thing is... you can never know exactly how blind you are.
We need to implement a better mental model for markets when dealing with these unknowns.
Possibilities not probabilities.
Probabilities imply linear and rigid known and unknown outcomes. They instill false confidence.
Possibilities imply the dynamic, fluid acceptance of unknown unknowns. They confer insecurity.
Thinking in possibilities instead of probabilities not only makes you a better risk manager by stripping you of your overconfidence, but it also primes your brain to avoid fixating on a single set of outcomes.
Fixating on a single set of outcomes wires us to look for confirming evidence while blinding us to opposing information. This is called confirmation bias. This bias tends to become more powerful the more time you sink into researching an idea (sunk cost bias).
Thinking in possibilities keeps our minds open to competing and opposing information. This is absolutely vital for any trader or investor who wants to have long-term success in markets.
It’s the whole “strong convictions, weakly held” idea. In fact, it’s an integral part of being mentally flexible, which is one of five key character traits of all the greatest traders.
It also keeps our conviction in check.
When filled with a strong sense of humility, knowing we don’t know all the key variables, we tend to operate in a manner more conducive to better outcomes. We manage risk better, size positions better, and are more apt to recognize when we’re wrong, allowing us to quickly become right. As Ray Dalio would say:
You can’t make money agreeing with the consensus view, which is already embedded in the price. Yet whenever you’re betting against the consensus, there’s a significant probability you’re going to be wrong, so you have to be humble.
Here is a sketch of a better way to think about markets.
A crucial part of thinking in possibilities is defining what inputs (price action, fundamental news, sentiment, etc.) would raise or lower your conviction for each possibility.
Basically, define what’ll convince you that you’re wrong and also what’ll increase your conviction that you’re right.
This is not a one-time process. It’s a continual study that you should be running in the back of your mind. Since the market is fluid and always changing, so should be your conviction levels on what’s possible and likely.
Sounds tough, right?
It is. Like I said at the beginning, our minds are not wired for markets. It takes conscious effort to think this way and we’ll always be limited by the biases we carry; many of which we’re unaware.
And it actually gets even tougher. Not only are there unknown unknowns in markets, but markets are also reflexive. This means that mythinking about the possibilities and yourthinking about the probabilities and Joe Schmoe’s thinking about simple predictions ALL affect the market. This in turn affects our thinking about the possibilities, probabilities, and… you get the picture.I’m not trying to throw you down the rabbit hole. This is just how markets function. Reflexivity was put forth by George Soros (though the philosophical origins of it actually stem from Karl Popper).
Here’s the actual way you should think about markets:
It’s a headache right?
And that’s the point. Thinking about markets is hard if you’re doing it right.
You should always maintain a certain amount of insecurity — the good kind of insecurity that makes you constantly question your assumptions, beliefs, and mental models.
Ed Seykota used to say, “There are old traders and there are bold traders, but there are very few old, bold traders.”
If you want to survive in this game, you need to recognize that you operate with a constant information deficit.
You can never know how large this deficit is. Therefore “prediction” is impossible. And assigning actual “probabilities” is useless.
Everything is only “possible” and everything is always unknowable.
This is why thinking in possibilities and increasing your cognitive flexibility is paramount to your long-term trading survival.
Malcolm Gladwell once retold the following story about George Soros (possibly the most cognitively flexible trader to have played the game).
An old trading partner of Taleb’s, a man named Jean-Manuel Rozan, once spent an entire afternoon arguing about the stock market with Soros. Soros was vehemently bearish, and he had an elaborate theory to explain why, which turned out to be entirely wrong. The stock market boomed. Two years later, Rozan ran into Soros at a tennis tournament. “Do you remember our conversation?” Rozan asked. “I recall it very well,” Soros replied. “I changed my mind, and made an absolute fortune.” He changed his mind!
