Surveys of financial sentiment often reveal a striking paradox. Everyday investors frequently report low expectations for future returns and express high anxiety about impending stock market crashes. At the same time, actual stock market prices continue to rise, completely decoupling from the pessimistic sentiment captured in the polls.
An investigation designed to shed light on this disconnect is detailed in a working paper published by the National Bureau of Economic Research. The research explores whether broad surveys misrepresent the market by treating all respondents equally, regardless of whether their money is actually at risk. The study finds that the gap between survey sentiment and market prices stems from pessimistic investors stepping away from the market entirely.
The Knowledge Gap
Standard financial models frequently assume that average survey responses reflect the beliefs of the typical investor actively buying and selling stocks. However, broad surveys sample a wide population that includes individuals who have liquidated their portfolios.
Researchers William N. Goetzmann of Yale University, Dasol Kim of the Office of Financial Research, and Robert J. Shiller of Yale University designed a study to test this dynamic. They categorized active traders who currently bear stock market risk and set prices as marginal investors.
In contrast, they defined individuals who exit the market as non-marginal investors. The researchers wanted to know how the departure of these pessimistic individuals alters the financial environment. They specifically sought to understand how this exit concentrates risk among the smaller pool of remaining investors.
Measuring the Belief Gap
The research team analyzed over two decades of individual responses from the Investor Confidence Survey, spanning from 1996 to 2025. They focused on specific questions asking respondents what they expected the market to return over the next six months. They also isolated a question asking respondents to estimate the exact probability of a catastrophic market crash in the near future.
To establish an objective market baseline for comparison, the researchers extracted expectations directly from S&P 500 option prices. Options are financial contracts that act as insurance against stock market movements. By analyzing the prices traders pay for this insurance, the researchers calculated the actual market consensus on expected returns and crash probabilities.
The researchers calculated a statistical spread for each data point. This spread represented the numerical distance between the subjective survey responses and the option-implied market baseline. They then cross-referenced this spread with other survey questions asking whether respondents were actively advising others to stay less invested in stocks.
Pessimism on the Sidelines
The analysis revealed that surveyed investors were consistently more pessimistic than the option-implied baseline. Survey respondents expected an average 1.4 percent return over a six-month horizon. In comparison, actual market prices implied a 4.2 percent expected return for the exact same timeframe.
The difference was even larger when estimating perceived crash risks. The average survey respondent estimated a 19 percent chance of a catastrophic crash over the next six months. The option market priced the likelihood of a catastrophic market decline at just 5.6 percent.
The researchers found that this gap was driven almost entirely by the specific investors who advised stepping away from the market. This non-participating group held beliefs far removed from market realities, logging expected returns that were consistently negative. Conversely, those who indicated they were remaining fully invested held expectations that closely matched the option-implied baseline.
Subjective crash beliefs specifically predicted whether someone would exit the market. Higher individual fears of a crash were robustly linked to a higher likelihood of an investor stepping out. This pattern held true even when those same investors expected high overall market returns, showing that crash fear alone drives market participation.
Concentrating Market Risk
The study also examined how this shifting investor base affects future stock market performance. When pessimistic individuals liquidate their holdings and leave the market, the overall equity risk does not simply disappear. Instead, the remaining active investors must absorb a much larger share of the total risk.
To take on this highly concentrated risk, the remaining investors require a higher premium to stay invested. The researchers note that this dynamic is linked to higher subsequent market returns over the following months. When the spread between survey pessimism and market prices widens, it negatively predicts future market returns, meaning higher aggregate pessimism precedes a rising market.
The researchers also evaluated the performance of individual stocks in relation to this crash-belief spread. They found that individual stocks that perform well when the aggregate crash-belief gap widens tend to earn lower average returns over time. The researchers interpret this as evidence that these specific stocks act as a hedge, providing a safe haven during periods when crash fears are elevated.
The research team also looked at how investor disagreement affects this pricing dynamic. They found that the pricing effects were much stronger among stocks that had high levels of disagreement among professional financial analysts. The authors argue that high disagreement facilitates a larger transfer of risk from pessimistic exiting investors to optimistic active investors.
Implications and Caveats
This composition channel provides a structural explanation for a known financial puzzle regarding survey sentiment. High survey pessimism often predicts higher future market returns because it signals that fearful investors have already sold their assets and retreated. The beliefs of these non-marginal investors no longer directly impact the price of stocks, but their absence reshapes the risk profile for everyone else.
The study relies on observational survey data and option prices to document these associations over a long time horizon. The authors note that the option-implied baseline relies on mathematical bounds derived from variance rather than precise point estimates. They also clarify that changes in market composition are observationally correlated with subsequent returns, rather than established through controlled experimental manipulation.
