
Moneyball for Tech Startups: Kevin’s Remix
Several people have pointed me to Dan Frommer’s post on Moneyball for Tech Startups, noting that “Moneyball” is actually a pretty good summary of our approach to seed-stage investing at RSCM. Steve Bennet, one of our advisors and investors, went so far as to kindly make this point publicly on his blog.Regular readers already know that I’ve done a fair bit of Moneyball-type analysis using the available evidence for technology startups (see here, here, here, here, here, and here). But I thought I’d take this opportunity to make the analogy explicit.I’d like to start by pointing out two specific elements of Moneyball, one that relates directly to technology startups and one that relates only indirectly:
- Don’t trust your gut feel, directly related. There’s a quote in the movie where Beane says, “Your gut makes mistakes and makes them all the time.” This is as true of tech startups as it is of baseball prospects. In fact, there’s been a lot of research on gut feel (known in academic circles as “expert clinical judgement”). I gave a fairly detailed account of the research in this post, but here’s the summary. Expert judgement never beats a statistical model built on a substantial data set. It rarely even beats a simple checklist, and then only in cases where the expert sees thousands of examples and gets feedback on most of the outcomes. Even when it comes to evaluating people, gut feel just doesn’t work. Unstructured interviews are the worst predictor of job performance.
- Use a “player” rating algorithm, indirectly related. In Moneyball, Beane advocates basing personnel decisions on statistical analyses of player performance. Of course, the typical baseball player has hundreds to thousands of plate appearances, each recorded in minute detail. A typical tech startup founder has 0-3 plate appearances, recorded at only the highest level. Moreover, with startups, the top 10% of the startups account for about 80% of the all the returns. I’m not a baseball stats guy, but I highly doubt the top 10% of players account for 80% of the offense in the Major Leagues. So you’ve got much less data and much more variance with startups. Any “player” rating system will therefore be much worse.
Despite the difficulty of constructing a founder rating algorithm, we can follow the general prescription of trying to find bargains. Don’t invest in “pedigreed” founders, with startups in hot sectors, that have lots of “social proof”, located in the Bay Area. Everyone wants to invest in those companies. So, as we saw in Angel Gate, valuations in these deals go way up. Instead, invest in a wide range of founders, in a wide range of sectors, before their startups have much social proof, across the entire US. Undoubtedly, these startups have a lower chance of succeeding. But the difference is more than made up for by lower valuations. Therefore, achieving better returns is simply a matter of adequate diversification, as I’ve demonstrated before.Now, to balance out the disadvantage in rating “players”, startup investors have an advantage over baseball managers. The average return of pure seed stage angel deals is already plenty high, perhaps over 40% IRR in the US according to my calculation. You don’t need to beat the market. In fact, contrary to popular belief, you don’t even need to try and predict “homerun” startups. I’ve shown you’d still crush top quartile VC returns even if you don’t get anything approaching a homerun. Systematic base hits win the game.But how do you pick seed stage startups? Well, the good news from the research on gut feel is that experts are actually pretty good at identifying important variables and predicting whether they positively or negatively affect the outcome. They just suck at combining lots of variables into an overall judgement. So we went out and talked to angels and VCs. Then, based on the the most commonly cited desirable characteristics, we built a simple checklist model for how to value seed-stage startups.We’ve made the software that implements our model publicly available so anybody can try it out [Edit 3/16/2013: we took down the Web app in Jan 2013 because it wasn’t getting enough hits anymore to justify maintaining it. We continue to use the algorithm internally as a spreadsheet app]. We’ve calibrated it against a modest number of deals. I’ll be the first to admit that this model is currently fairly crude. But the great thing about an explicit model is that you can systematically measure results and refine it over time. The even better thing about an explicit model is you can automate it, so you can construct a big enough portfolio.That’s how we’re doing Moneyball for tech startups.

The VC "Homerun" Myth
In spreading the word about RSCM, I recently encountered a question that led to some interesting findings. A VC from a respected firm, known for its innovative approach, brought up the issue of "homeruns". In his experience, every successful fund had at least one monster exit. He was concerned that RSCM would never get into those deals and therefore, have trouble generating good returns.
My initial response was that we'll get into those deals before they are monsters. We don't need the reputation of a name firm because the guys we want to fund don't have any of the proof points name firms look for. They'll attract the big firms some time after they take our money. Of course, this answer is open to debate. Maybe there is some magical personal characteristics that allows the founders of Google, Facebook, and Groupon to get top-tier interest before having proof points.So I went and looked at the data to answer the question, "What if we don't get any homeruns at all?" The answer was surprising.I started with our formal backtest, which I produced using the general procedure described in a previous post. It used the criteria of no follow-on and stage <= 2, as well as eliminating any company in a non-technology sector or capital-intensive one such as manufacturing and biotechnology.
Now, the AIPP data does not provide the valuation of the company at exit. However, I figured that I could apply increasingly stringent criteria to weed out any homeruns:
- The payout to the investor was < $5M.
- The payout to the investor was < $2.5M
- The payout to the investor was < $2.5M AND the payout multiple was < 25X.
It's hard to imagine an investment in any big winner that wouldn't hit at least the third threshold. In fact, even scenarios (1) and (2) are actually pretty unfair to us because they exclude outcomes where we invest $100K for 20% of a startup, get diluted to 5-10%, and then the company has a modest $50M exit. That's actually our target investment! But I wanted to be as conservative as possible.The base case was 42% IRR and a 3.7x payout multiple. The results for the three scenarios are:
- 42% IRR, 2.7x multiple
- 36% IRR, 2.4x multiple
- 29% IRR, 2.1x multiple
Holy crap! Even if you exclude anything that could be remotely considered a homerun, you'd still get a 29% IRR!As you can see, the multiple goes down more quickly than the IRR. Large exits take longer than small exits so when you exclude the large exits, you get lower hold times, which helps maintain IRR. But that also means you could turn around and reinvest your profits earlier. So IRR is what you care about from an asset class perspective.For comparison, the top-quartile VC funds currently have 10-year returns of less than 10% IRR, according to Cambridge Associates. So investing in an index of non-homerun startups is better than investing in the funds that are the best at picking homeruns. (Of course, VC returns could pick up if you believe that the IPO and large acquisition market is going to finally make a comeback after 10 years.)
I've got to admit that the clarity of these results surprised even me. So in the words of Adam Savage and Jamie Hyneman, "I think we've got to call this myth BUSTED."(Excel files: basecase, scenario 1, scenario 2, scenario 3)

Simulating Angel Investment: Kevin's Remix
Jeff Miller has done a couple of nice posts on "A Simulation of Angel Investing" here and here. I think it's terrific that Jeff actually asked the question and tried to answer it with simulation. However, his answer of 20 is way too low because of two key oversimplifications. Using a more sophisticated methodology, I'll show that a better answer is 100 to 150.You may recall that Saving the World with Startups explained the "why" of RSCM. Our goal is to increase the number of technology startups. In some sense, this post describes the "how". Well, at least part of it. One of the biggest barriers to getting a company off the ground is finding working capital. Ergo, we need to figure out how to facilitate investments in startups. More precisely, we need to promote seed-stage investments because those are what help founders initially launch their companies.The ideal solution would be an investment vehicle that can turn huge chunks of money into digestible seed-stage bites with a return that induces plenty of investors to participate. But here are some slightly scary statistics. 50% of all seed-stage startups fail and returns come disproportionately from the top 10%. As all you poker players in the audience will note, you're making big bets with high variance. The natural question is, "How many bets should you place?"To answer this question, I've built several generations of seed-stage investing simulations for RSCM. My models are rather complicated because we wanted to evaluate a bunch of secondary questions such as whether it's better to do follow on investments, what happens if the balance between seed and Series A valuations changes, and what happens in cases where a startup does poorly initially but then takes off. Therefore, I actually had to model the startup lifecycle round by round and the mechanics became very complex. (If you're not a quant, you can stop reading now. Things are going to get real geeky real fast).However, a simplified single-round version of my model will illustrate the missing pieces of Jeff's model. The first is what diversification means. He focuses on the risk of total loss and the chances of not getting at least one "hit". In my opinion, the question you really want to ask is what the probability is that you'll under-perform the market by more than a given amount. For example, what's the probability that you'll under-perform by more than 25%? The logic here is that you invest in an asset class because of the overall return of that asset class, so you want to know the chances that you'll realize returns in that ballpark.The second key oversimplification is that Jeff uses a discrete probability distribution of returns. If you've read Taleb's The Black Swan, you know this is a mistake because at least some seed-stage outcomes probably follow a Pareto distribution. The key characteristic of this distribution is that regions of extreme outcomes are self similar. So not only do the top 10% of companies represent a disproportionate share of the returns, the top 10% of the top 10% represent a disproportionate share of those returns. And so on. And so on. 20 investments may be enough to get you a fair share of the top 10%, but not enough to get you a fair share of the top 1%.So here's my simplified model, which roughly follows Jeff's qualitative taxonomy:
- 50% failures: the company utterly fails. The investor gets 0 money returned.
- 20% break even: the company achieves some limited success and the money returned follows a lognormal distribution with a minimum of 0, a, mean of 1, and a standard deviation of 1. So an average outcome is 1.0x and 1 standard deviation above is 2.0x.
- 20% decent: the company achieves substantial success and the money returned follows a lognormal distribution with a minimum of 2, a mean of 4, and a standard deviation of 4. So the minimum outcome is 2x, the mean outcome is 4x, and 1 standard deviation above is 8x.
- 10% homeruns: the company achieves massive success and the money returned follows a Pareto distribution with a location of 10 and an index of 1.5. So the minimum outcome is 10x and the mean outcome is 30x.
Now, we can compute the expected value of an investment as .50*0 + .2*1 + .2*4 + .1 *30 = 4.0. The data I've seen puts the average hold time for successful angel investments at 6 years, so this would imply an IRR of about 26%. This is in line with the available research on angel returns (RSCM has a summary of this research here).I ran a simulation with these parameters using Oracle's Crystal Ball, producing an overall return distribution for a run of 100K trials. Here's the excess distribution plot (the probability that the money returned will exceed a given multiple), truncated at 50x for some semblance of readability:

The return across the entire simulation was 4.05x (very close to the analytically expected return of 4.0x). The maximum return was 8,361x (think Andy Bechtolsheim's $100K investment in Google which was eventually worth about $1B). The top 10% accounted for 77% of the total return. The top 1% accounted for 35%. The top .1% accounted for 17%. We can already see that a portfolio of 20 will be insufficient.The source file is AngelSimulation. Most of you probably don't have Crystal Ball so this will look like a pretty useless Excel file to you. However, I set up the run to output just the AngelSimulationData in an Excel file. Anyone can analyze this with standard charting tools or import the data for use by his own code.I've also got another AngelSimulationPortfolios with a macro that generates 10K random portfolios of a given size from the trial data. I've run it for portfolio sizes from 10 to 200 in intervals of 10. After sorting the portfolio returns at the specified size, the macro calculates the probability of hitting 75% of the market return by seeing what percentage of the portfolio returns are greater than 3.0. Here's a chart of those probabilities:

[Edited 5/14 in response to suggestion from AN]. As you can see, 20 investments isn't nearly enough if you're a fund investing other people's money. Worse than a coin flip that you'll hit 75% of the market return. In fact, in my simulated portfolio data, there's about a 7% chance that you'll lose money with a portfolio of 20 investments. Personally, I'd say you want a fund to be in the 100 to 150 investment range. But it's different for individual investors putting in their own money. I'd say you want to hit at least a 50% chance of realizing 75% of the market return, which would be 30 investments. Now, if you think you think you have some forecasting skill and less than 50% of your seed investments will fail and/or more than 10% will be homeruns, 20 may be plenty.Of course, if you accept the thesis that 100-150 is the right range for a fully diversified fund-like portfolio, you may now be asking yourself how making that many seed-stage investments is logistically possible. The challenge is actually worse than that. Due to vintage risk, you probably want to make 100-150 investments per year or at least every few years. But that's a story for another day...

You Can't Pick Winners at the Seed Stage
[EDITED 05/08/2009: see here] The majority of people I've talked to like the idea of revolutionizing angel funding. Among the skeptical minority, there are several common objections. Perhaps the weakest is that individual angels can pick winners at the seed stage.
Now, those who make this objection usually don't state it that bluntly. They might say that investors need technical expertise to evaluate the feasibility of a technology, or industry expertise to evaluate the likelihood of demand materializing, or business expertise to evaluate the evaluate the plausibility of the revenue model. But whatever the detailed form of the assertion, it is predicated upon angels possessing specialized knowledge that allows them to reliably predict the future success of seed-stage companies in which they invest.
It should be no surprise to readers that I find this assertion hard to defend. Given the difficulty in principle of predicting the future state of a complex system given its initial state, one should produce very strong evidence to make such a claim and I haven't seen any from proponents of angels' abilities. Moreover, the general evidence of human's ability to predict these sorts of outcomes makes it unlikely for a person to have a significant degree of forecasting skill in this area.
First, there are simply too many random variables. Remember, startups at this stage typically don't have a finished product, significant customers, or even a well-defined market. It's not a stable institution by any means. Unless a lot of things go right, it will fall apart. Consider just a few of the major hurdles a seed-stage startup must clear to succeed.
- The team has to be able to work together effectively under difficult conditions for a long period of time. No insurmountable personality conflicts. No major divergences in vision. No adverse life events.
- The fundamental idea has to work in the future technology ecology. No insurmountable technical barriers. No other startups with obviously superior approaches. No shifts in the landscape that undermine the infrastructure upon which it relies.
- The first wave of employees must execute the initial plan. They must have the technical skills to follow developments in the technical ecology. They must avoid destructive interpersonal conflicts. They must have the right contacts to reach potential early adopters.
- Demand must materialize. Early adopters in the near term must be willing to take a risk on an unproven solution. Broader customers in the mid-term must get enough benefit to overcome their tendency towards inaction. A repeatable sales model must emerge.
- Expansion must occur. The company must close future rounds of funding. The professional executive team must work together effectively. Operations must scale up reasonably smoothly.
As you can see, I listed three example of minor hurdles associated with each major hurdle. This fan out would expand to 5-10 if I made a serious attempt at exhaustive lists. Then there are at least a dozen or so events associated with each minor hurdle, e.g., identifying and closing an individual hire. Moreover, most micro events occur repeatedly. Compound all the instances together and you have an unstable system bombarded by thousands of random events.
Enter Nassim Taleb. In Chapter 11 of The Black Swan, he summarizes a famous calculation by mathematician Michael Berry: to predict the 56th impact among a set of billiard balls on a pool table, you need to take into account the the position of every single elementary particle in the universe. Now, the people in a startup have substantially more degrees of freedom than billiard balls on a pool table and, as my list above illustrates, they participate in vastly more than 56 interactions over the early life of a startup. I think it's clear that there is too much uncertainty to make reliable predictions based on knowledge of a seed-stage startup's current state.
"Wait!" you may be thinking, "Perhaps there are some higher level statistical patterns that angels can detect through experience." True. Of course, I've poured over the academic literature and haven't found any predictive models, let alone seen a real live angel use one to evaluate a seed stage startup. "Not so fast! " you say, "What if they are intuitively identifying the underlying patterns?" I suppose it's possible. But most angels don't make enough investments to get a representative sample (1 per year on average). Moreover, none of them that I know systematically track the startups they don't invest in to see if their decision making is biased towards false negatives. Even if there were a few angels who cleared the hundred mark and made a reasonable effort to keep track of successful companies they passed on, I'd still be leery.
You see, there's actually been a lot of research on just how bad human brains are at identifying and applying statistical patterns. Hastie and Dawes summarize the state of knowledge quite well in Sections 3.2-3.6 of Rational Choice in an Uncertain World. In over a hundred comparisons of human judgment to simple statistical models, humans have never won. Moreover, Dawes went one better. He actually generated random linear models that beat humans in all the subject areas he tried. No statistical mojo to determine optimal weights. Just fed in a priori reasonable predictor variables and a random guess at what their weights should be.
Without some sort of hard data amenable to objective analysis, subjective human judgment just isn't very good. And at the seed stage, there is no hard data. The evidence seems clear. You are better off making a simple list of pluses and minuses than relying on a "gut feel".
The final line of defense I commonly encounter from people who think personal evaluations are important in making seed investments goes something like, "Angels don't predict the success of the company, they evaluate the quality of the people. Good people will respond to uncertainty better and that's why the personal touch yields better results." Sorry, but again, the evidence is against it.
This statement is equivalent to saying that angels can tell how good a person will be at the job of being an entrepreneur. As it turns out, there is a mountain of evidence that unstructured interviews have little value in predicting job performance. See for example, "The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings" [EDITED 10/17/2011: New link to paper because old one was stale]. Once you have enough data to determine how smart someone is, performance on an unstructured interview explains very little additional variance in job performance. I would argue this finding is especially true for entrepreneurs where the job tasks aren't clearly defined. Moreover, given that there are so many other random factors involved in startup success than how good a job the founders do, I think it's hard to justify making interviews the limiting factor in how many investments you can make.
Why then are some people so insistent that personal evaluation is important? Could we be missing something? Always a possibility, but I think the explanation here is simply the illusion of control fallacy. People think they can control random events like coin flips and dice rolls. Lest you think this is merely a laboratory curiosity, check out the abstract from this Fenton-O'Creev, et al study of financial traders. The higher their illusion of control scores, the lower their returns.
I'm always open to new evidence that angels have forecasting skill. But given the overwhelming general evidence against the possibility, it better be specific and conclusive.