Really Big Claims Based on Too Little Data Carl Richards is a certified financial planner in Park City, Utah. His new book, "The Behavior Gap," was published earlier this year. His sketches are archived here on the Bucks blog. Few things frustrate me more than people making claims based on really small amounts of data or no data at all. Last week a friend sent me a segment from CNBC that highlights this issue in real time. It all started with a conversation between Brian Sullivan, co-anchor of "Street Signs," and Michael Shinnick, the manager of the Wasatch Long/Short Fund. The segment begins with this exchange: Brian Sullivan: Mike, I guess I've sort of made the case for your industry, and active management. Michael Shinnick: ... The basic thought that we have is that this religion, if you will, of efficient markets and modern portfolio theory doesn't actually hold up in practice. If Mr. Shinnick is right that modern portfolio theory and efficient markets don't hold up in practice, then it would show up in the data we have on active managers. We should see the most skillful managers staying at the top of their class year after year. If it's as easy as being "long the stuff that is undervalued and short the stuff that's overvalued," as Mr. Shinnick says, then we should see it showing up in past performance data. For a look at the evidence, let's use Standard & Poor's most recent Persistence Scorecard. One of the goals of the scorecard is to answer that question of whether the past performance of active managers really matters. In this instance, the numbers do not support the case for active management. Very few funds manage to repeat top-half or top-quartile performance consistently. For the five years ending September 2011, only 9.72 percent of large-cap funds, 6.08 percent of mid-cap funds and 3.27 percent of small-cap funds maintained a top-half ranking over five consecutive 12-month periods. That means that only 9.72 percent of all the large-cap funds managed to be above average, every year, for five consecutive years. Think about that for a minute. Less than 10 percent of funds managed to be better than just average every year for five straight years. It's worth noting that you can avoid all the bother and just "buy" the average using a low-cost index fund. But let's pretend you don't know that. If you only heard this particular conversation on CNBC, your natural conclusion might be that you should sit down and try to figure out how to find a great manager. If that is the case, let's look at your options if you had done so in 2006. First, you'd start by looking at managers with great, five-year track records and limit it to managers in the top 25 percent. However, even though you are picking from the top of the list, consider what happens in the following five years. Only 12.23 percent of the large-cap funds in the top 25 percent in the five years ending in 2006 stayed in the top 25 percent over the next five years. In fact, only 25.9 percent of large-cap funds with a top-quartile ranking over the five years that ended in September 2006 finished the next five years above average. To be clear, this post isn't about Mr. Shinnick personally but about the claims that are made and the data (or lack thereof) that is used to support them. In this instance, I would argue that far from making the case for active management, the data actually shows how difficult it is to find a mutual fund that will do well over the five years going forward based on the previous five years. And while I know it is tempting to think you can find those mythical managers that do well over time, please accept the data that says the odds are stacked against you, and it will lead you to do dumb things with your money.