Share Podcast
Can Collective Intelligence Beat the Market? (with Numerai’s Richard Craib)
A new approach to building a trading strategy: crowd-sourcing financial predictions from data scientists.
- Subscribe:
- Apple Podcasts
- Google Podcasts
- Spotify
- RSS
Quantitative hedge funds (or “quant funds”), which rely on the work of employed mathematicians to develop complex trading strategies, are nothing new. But what if the mathematical work is outsourced to anyone, via a contest where the best predictions are rewarded with cryptocurrency?
Richard Craib, founder of Numerai, explains to Azeem Azhar why his $70 million fund uses collective intelligence to perform well, despite the turmoil in the markets.
They also discuss:
- The history and theory of quantitative investing.
- When collective minds can work better than brilliant individuals.
- How Richard found himself accidentally explaining how “quant funds” work to Howard Morgan, the co-founder of one of the most successful “quant funds” in the world.
Further resources:
AZEEM AZHAR: Hi there, I’m Azeem Azhar, and this is the Exponential View podcast. Now I want to start with a little story, and just please bear with me, it is connected to our discussion today. So back in the 1980s, a mathematician and a computer scientist teamed up to found an investment fund, which would become known as Renaissance Technologies. And for the next 40 years, that fund applied a lot of maths, sitting on top of growing amounts of computing power to, well, beat the market. The firm’s flagship, fund Medallion established in 1988 has returned in excess of 60 or 70 percent annually for more than 30 years. And I have been fascinated by Medallion and their quantitative approach to investing. And a few years ago, I was introduced to an entrepreneur who was building a new type of quantitative hedge fund. Now lots of people do this. There are lots of quantitative funds. I had invested in one such fund with some good fortune in the mid-2000s, but something about this story stood out. This entrepreneur was using a new platform for collective intelligence to build his trading algorithms. That was the first thing. Second was that he had been backed by Howard Morgan, one of the co-founders of Renaissance. And I thought there really had to be something interesting going on here. And it’s taken me a few years to get this conversation together, but I’m delighted to have Richard Craib, the founder of Numerai, with me today. Welcome to Exponential View.
RICHARD CRAIB: It’s good to be here.
AZEEM AZHAR: It’s a fascinating market, this idea of mathematical quantitative funds. The Renaissance story is a really amazing one. Is that the one that sets the hallmark for this approach?
RICHARD CRAIB: It definitely does, even though it’s a very competitive market, they seem to be twice as good as everyone else somehow and very consistently. So I remember in college reading about them for the first time and also just being very inspired by how unusual their return stream was.
AZEEM AZHAR: Unusual being — it rarely went down and it just went up. And often with funds, there’s a bit of volatility, they’re losing months, losing years, not so with them.
RICHARD CRAIB: No, exactly. In 2008s, fine. Now this year, they’re doing fine. Last year, fine. COVID, fine.
AZEEM AZHAR: You’ve set up a vehicle that also invests in the market using quantitative methods to Numerai. I’m really curious about that connection with Howard Morgan, who, of course, after he founded Renaissance, went on to be a really wonderfully supportive venture capitalist to many entrepreneurs. How did that connection come about?
RICHARD CRAIB: So there was a guy, Bill Trenchard, who was a partner at First Round [Capital] and he said, “come meet me.” And I had just started Numerai, and I had been living in the U.S. for three weeks, and I was explaining quantitative investing to him. And in the room was this other guy who was a bit older than him and me. And I started explaining it to him as well. And he said, I’m familiar with quantitative finance. I co-founded Renaissance.
AZEEM AZHAR: Wow.
RICHARD CRAIB: So I was like, “whoa, that’s crazy.” And it was a really nice thing for me because what I found pitching Silicon Valley VCs is they don’t have any grip on quant finance. They don’t need to, they’re investing in technology bubbles. So it was a strange and very lucky meeting.
AZEEM AZHAR: You impress Howard in some way in your story because he decides to back you out of the thousands of founders he sees in any given year. I guess I could ask him that myself, but what do you think it was about the story?
RICHARD CRAIB: Well, actually it’s funny. They really did like me and they did like Numerai, but they actually declined investing. And I was like, this sucks. And the reason they declined was they were like, “well, look, we invest in startups that are tech companies where you’re going to maybe IPO or there’s going to be an acquisition.” And Numerai was so weird and so differently structured that it wasn’t suitable for First Round’s portfolio. So I was very unhappy, but I decided, “you know what? I’m going to try to stay in touch with them.” And then a week later I wrote the first blog post introducing Numerai. And we got a lot of users in the first few weeks, and then I sent them that and then they said, “oh, okay, fine. We’ll invest personally.” Weirdly, later on Union Square Ventures end up leading our A round. And because they’re actually quite tight, they have similar LPs. They figured out how to actually roll their investment back into First Round, which I thought was pretty good of them.
AZEEM AZHAR: So let’s go through this idea of quantitative investing at the very, very high level. What is quantitative investing, as opposed to buying a mutual fund, or an ETF, or owning a few shares that you think are going to do well?
RICHARD CRAIB: There’s actually many, many flavors of it that are in some ways completely different, even though you’re trading business, you can be very different. So high frequency trading is maybe less about modeling and more about having perfect market access and market information as fast as possible to make trades that just arbitrage, whereas what we are doing is much longer term. In fact, we’re longer term than Renaissance, the Medallion fund, would be by a long way. We end up holding stocks for three to four months. And in this medium horizon frequency, quant hasn’t been applied as much. And certainly, I think we were one of the first doing machine learning on this time horizon. And so, because you’re buying something for a bit of a longer time, you have to get data where you know things about the company.
AZEEM AZHAR: I guess the point there being that with these high frequency funds, you’re getting these streams of data and you’re looking for opportunities that may only exist because it’s mispricing for a few hours, or a few days, and you come in and you get out, and you come in and you get out. And you don’t really need to worry about the fundamentals of a company — its balance sheet, or its inventory turns, or its days outstanding. You just look at those numbers quite often, just like price and volatility and the depth of orders that are waiting to clear. But you, if you’re holding for three to six months, you need to understand a little bit about a company’s prospects.
RICHARD CRAIB: Exactly. We are investing, but we’re doing it in a quantitative way — as in there’s no human involvement in making the trades. And the unique part about Numerai, of course, is all of the modeling and all the models are built by people around the world.
AZEEM AZHAR: Yeah. That is a unique part. That is a collective intelligence where, rather than hiring super smart physicists and mathematicians out of the top grad programs, you’re distributing the opportunity to data scientists anywhere, which I find a fascinating part to the story. Before we get there, though, tell us about how well the fund does. I’ve looked at your returns, they look pretty impressive. One wants to think of returns in a few different ways: the absolute level of return, the return relative to a benchmark that you can be compared against, but also the risk-adjusted return. Are you making money by taking undue risks, or are you making money and still being very sensible about the amount of exposure that you could take?
RICHARD CRAIB: I think that’s a very key point that many people don’t understand. When an investor invests in a quantitative market-neutral fund like Numerai’s, they’re giving us a mandate where they’re saying, “we already have a lot of tech stocks. We already have a lot of venture capital exposure. We already have a lot of private equity exposure. We don’t want you to take any risks that are identical to those risks because we want you to do well when those things are doing badly,” like right now. So it’s a very different thing than “here’s the money, try to make the most money you can make.” You’re not trying to do that. It sounds weird. We’re in a hedge fund, but we’re not trying to make the most money.
AZEEM AZHAR: Because your investors have got other exposure, right? And so you want to behave, you want to be zagging when the rest of their investments are zigging.
RICHARD CRAIB: And that’s why I think it’s actually quite confusing for young people looking into hedge funds, they’re like, this hedge fund that has 20 PhDs working there. Why did they only make eight percent return this year? That doesn’t sound right. I made 40 percent. And that’s usually because you took risk and they didn’t take any risk in a technical way, where they took no exposure to any industry. Every time they went longer tech stock, they also went shorter tech stock. And so that’s what we’re doing. We’re building these portfolios where if you look at them, you say, “man, this is a random collection of stocks. It’s almost like you’re not betting on anything at all,” but we are, we’re betting on our own alpha.
AZEEM AZHAR: And alpha is that return of above the market. Now in 2021, you returned about 12.5 percent where the benchmark of other funds are taking your kind of strategy, which is known as equity market neutral. That means that you don’t really care what happens to the NASDAQ or the S&P 500. It can go up. It can go down. That benchmark was running between seven and eight percent. So you were outperforming the benchmark. Now as an occasional fund investor myself, a couple of the metrics that I tend to look at are the Sharpe ratio or the Sortino ratio. The Sortino is, I suppose the way I think about it is, how much exposure to downside risk did the investment manager take in order to give me the upside? So how many times could we have lost our shirt? And that’s normally a number that is below one, you’re terrible. Above one, it’s acceptable. Above two, it’s pretty good. How do you stack up on something like a Sortino ratio?
RICHARD CRAIB: So yeah, we’re now running in about 2.3, 2.4.
AZEEM AZHAR: Oh, you’d pass my screen.
RICHARD CRAIB: So I passed your test.
AZEEM AZHAR: Yeah.
RICHARD CRAIB: And that’s very good. It’s also especially good for the time period that you’re measuring it over because remember, in this time period, what have you had? You’ve had March 2020, COVID crash, market down 30 percent, our funds down one and a half. You had meme stock rally. So you have a lot of these kinds of risk events that have happened and the broader high volatility in the market. And somehow our returns are like this straight line up. Over the whole period, our volatility has been less than 10 percent annualized, whereas the markets can run on 20, 25 [percent].
AZEEM AZHAR: That’s really the setup here. We’ve got this interesting fund and you’ve had some good returns and you’re demonstrating these good metrics that show that it is performing well without taking undue risk. So let’s get back to how you do this. We touched on this a little bit earlier — this idea of a collective intelligence, rather than hiring six or seven really, really bright people. This seems to be the heart of your special source. So how does the intelligence come about, and how do the decisions get taken?
RICHARD CRAIB: So every quant fund will say, “we have the best people and we have the best data,” but they all have the same pitch. So they’re not all right. And they’re actually duplicating each other’s work. So Numerai took a very different approach and says, “well, how can we in actual fact have the best people, and the best data?” And the way to get the best people is to make an open system — to make the first internet hedge fund, where anyone can download our data and submit models back to us. Now, important to note, this website isn’t for everyone. We are not trying to get a hundred million users. It’s not like a Snapchat or something. We’re trying to get a few thousand, but they’re going to be the best data scientists in the world. And also we get them to put skin in the game with staking. So there’s that fit part makes it like we are actually able to ensemble all of these models that all of these people are incentivized to continue to improve.
AZEEM AZHAR: Okay. So you make data available to data scientists around the world and they can build models. Now a model is effectively a thing that makes predictions on data. So data comes in and you’ve trained it, and then you see some new data, and it makes a prediction. It can perform well or less well. And that process reminds me quite a lot of this service that I used years ago called Kaggle. Kaggle was like a Reddit for data scientists, where data scientists could come and participate in competitions, which might be to do some machine vision competition or some natural language processing test. And I think Google acquired Kaggle because it was a great way of building their profile amongst data scientists. So is it a bit like that, but it essentially, you offer the data, people compete?
RICHARD CRAIB: Yeah. And I was a Kaggle user — at my first job I would play in these Kaggle competitions. And they’ve networked all of these data scientists. And so when Numerai started, it spread in the Kaggle community, and there are some of the best data scientists in the world there, and a lot of them are using Numerai now.
AZEEM AZHAR: So the thing that then I think becomes really important is how do we construct a mechanism that achieves two things. The first thing is that there is money at risk here. There’s people’s capital at risk because this is a fund. And we’ll talk about how you get your capital in a bit. So you want there to be a sense of authenticity, a sense of reliability, a sense of trustworthiness amongst the participants. I think the second thing, of course, is that participants are also giving up their time. And in Kaggle, you competed for prizes, but there needs to be some form of incentive for people, for them to want to stick around. And, of course, I guess the third thing is that you need to have a selection mechanism. I don’t want to do myself down, but I suspect that my model is not going to be the best of the models that you might see from your data scientists, were I to go on your platform. So how should we think about each of those things in turn?
RICHARD CRAIB: So when we started, we actually started paying people in Bitcoin, $400 Bitcoin.
AZEEM AZHAR: Wow. You were over paying relatively, right?
RICHARD CRAIB: And the reason was we had all these users around the world and it was the easiest way to pay, but it actually wasn’t working in the beginning because of the following attack. So you could make a thousand accounts, submit a thousand different models, and one of them is going to get lucky, and then you’ll get paid the Bitcoin. So, you just make 10,000 accounts. So you’re submitting models that you don’t really believe in because you’re trying to game the system kind of. So what we had to do was fix that incentive. And that’s why we created staking with NMR, our own token. So we made our own token. We gave it away for free. There was no ICO. It was just given away to our users and allowed them to stake it on their models to say, “I think this model will work. And here’s my money to prove that I believe it will work.” And what happens if your model performs badly is we can burn your stake.
AZEEM AZHAR: And burning means is — I lose it.
RICHARD CRAIB: Exactly.
AZEEM AZHAR: Right. And if my model works well, what happens? Is it like Spotify where, if my song gets played a lot, I get a tiny sliver of the income stream?
RICHARD CRAIB: So actually, yeah. So, the way we weight models internally, when we’re putting them all together, is by taking the stake-weighted average of all of them. So if you’re staking a lot, you’ll have a higher weight. And therefore, you can earn more money because you’re more responsible for the returns of the overall model. And so, yeah, we’ll pay you based on your model’s performance.
AZEEM AZHAR: So you’ve got a number of data scientists. They download this data and they say, “well, I think my model is going to do this.” And they stake a certain amount of NMR, which is the token. And what if someone’s really wrong? If someone puts in a really terrible model, does that have a permanent impact? What happens to them is they get wiped out, right? Their next turn, they’re not there because they lost all their NMR, but what happens to your own model performance?
RICHARD CRAIB: So if someone wanted to, they could make an extremely red model, but it would basically get burned off very quickly. And so we don’t really mind if there’s some models in there that aren’t yet at the perfect weight. Maybe this guy’s like, oh, this guy’s a bit overconfident. Well, it doesn’t matter, in the long run the meta model will get to the right weights because he’ll be burned off. So it’s always getting better for that reason.
AZEEM AZHAR: So the meta model is the model of models?
RICHARD CRAIB: Exactly.
AZEEM AZHAR: How do you compile that? You’ve got N models out there. How do you ensemble them into a single meta model?
RICHARD CRAIB: They’re actually not sending us models, they’re sending us predictions. So they’re sending us 5,000 predictions that are coming out of their models. And so we’ll have a CSV file with 5,000 rows. This is Google and this is Apple. And we give a ranking of which of their favorite stocks to go long, which are their favorites to go short. And all of this is obfuscated, but you have this CSV vector of predictions, and we take the stake-weighted average of that. So, that’s exactly how it works. And then we take that, put it in our portfolio optimizer to make trades.
AZEEM AZHAR: So I’m going to play that one back for people for whom adding 500 dimensional vectors and averaging across thousands of them is not something that just comes to hand. So the stake-weighted average is essentially, if I think that we should be buying a hundred Apple shares and you think we should be buying no Apple shares at all, and we both stake the same amount, the stake-weighted average would be by 50 Apple shares because that’s the average of our two recommendations.
RICHARD CRAIB: Yeah, pretty much. But none of the models are saying how many shares to buy, but effectively they are saying how much they like the stock. And if everybody likes Apple, that’s going to be one of the most likely things for us to want to buy.
AZEEM AZHAR: So there’s a little bit of central intelligence that Numerai is providing, when you look at the meta model of recommendations to turn that into something that is specific and actuated in the trading markets.
RICHARD CRAIB: It’s not that we have another model that we then, “oh, okay, we’ll choose these bonds and make these trades.” It’s that we just are taking these predictions and saying, “how can we make a risk neutral portfolio that represents those predictions?” So here’s what might happen is maybe there are a few more tech stocks that are liked by the Numerai community. Well, our investment mandate with our investors says we can’t be net long tech. So we have to find some shorts to balance them out. So that’s what’s happening, and that’s called portfolio optimization. And that optimizer is only using the meta model signal from the users.
AZEEM AZHAR: Why does this collective of minds work better than a group of employed and directed, brilliant people?
RICHARD CRAIB: The first answer is intuitive. If you have a model, it’s going to not work all the time. This is like a guarantee it’s like, there’s no free lunch. There’s no perfect model. So some models are going to fail sometimes. And you want to have other models in your system that are helping in the times when some others are failing. And that’s the magic of ensembling models is that you get to cancel out the errors of the one model with the other models. And then you end up with a much more accurate and higher Sharpe model in the sense that the volatility of your returns will be lower. That’s why it’s particularly useful in finance to use many, many models. And Numerai has the most.
AZEEM AZHAR: But is there an optimal number in bringing this collective intelligence together? Does it make a difference to continue to grow this pool of data scientists providing you with models?
RICHARD CRAIB: So interestingly, we give out an example model, which is an XGBoost model, if you’re familiar with data science that uses decision trees. And we give it out as an example model, and it’s actually quite good. It beats a lot of users sometimes, but we give it out for free. You can use this, this is our baseline. Try to improve on this, but it’s already quite good. And so the question is, okay, well, let’s say we take that internal model, which is our best model, and we back test it and see how well it would’ve done versus the meta model. And in time, especially in times of market stress like now, our internal model gets killed, but the meta model does fine. And so that’s the magic of ensembling so many models together. It’s like you start to become a bit more like the Medallion fund from Renaissance.
AZEEM AZHAR: Do you update your core baseline model with the things that you are learning on a regular basis through the meta model? So that baseline keeps rising, or is there a danger, then, that you’re starting to explore the same local space of possibilities?
RICHARD CRAIB: One thing we do, which is always give out new data sets, so we’ll give out new features. So there’s always new stuff to explore.
AZEEM AZHAR: So I think when you look at a company, there’ll be a handful of obvious features that you might have, like the current share price and the market capitalization, the number shares outstanding, earnings per share, and so on — those are all features. And over time you just want to provide more and more features like the amount of intangible assets on the balance sheet and so on and so forth because the more that the models can find optimal paths.
RICHARD CRAIB: There’s no question that machine learning is terrible with bad data. So the more data you give, and in our case, the more data on companies, the better you can do. And also the more unusual you could make a model. So when we had only a few features, 30 features, you’re right. A lot of the models that were submitted were almost the same because they’re trained on the same data. But now that we have 1,000 features, there’s a lot of room to be like, “you know what? I’m going to drop the first 500 features and just make my model on these features. And then I’m going to drop the earlier part of the data because I want my model to work on recent history.” You can do all these things, and if you believe they’re going to work and you stake on them, we’ll add that model to the meta model.
AZEEM AZHAR: How quickly would your meta model decay if it wasn’t being updated? You’re in this environment where anytime a trading strategy gets a bit of advantage, it’s noticed in the market. And so there’s a battle with other people who will see that advantage and try to nibble at it away. So what is the kind of longevity of the meta model?
RICHARD CRAIB: Ours is long because, in some ways, because our time horizon is long and the data set we already give out is 30 years long. Plus we’re trying to be market neutrals and neutral to so many things. So we’re not really trying to catch the next wave of something, or if you ever read something in the news, Numerai’s not going to be exposed to that because anything that’s known, like sector risk, country risk, factor risk, is something we’re neutral to. It’s not like a trading — they’re trying to make trading profits. Numerai is trying to make profits from investing. And that’s why this longer term piece is significant.
AZEEM AZHAR: So when a data scientist is playing in Numerai, they will build some reputation because they’ll be getting better and better and better. Do you take into account someone’s reputation, or is that reputation captured by the fact that someone with high reputation has got more of the NMR token, so it can stake higher?
RICHARD CRAIB: We used to have systems like, could we just use their reputation, but we decided to stop doing that and just trust the stake. So here’s the danger, you have a model, it’s working really well, and so you’re building this amazing reputation. And then you decide, “I’m not comfortable with my model anymore, I think it’s played out.” Then you unstake. And now we are still trusting that model. No, if you don’t trust it, we shouldn’t trust it. So that’s why the stake-weighted average works the best.
AZEEM AZHAR: Now I’ve been thinking about incentive alignment and skin in the game in these uncertain, and often unknowable markets, and really thinking back to some of the work of Friedrich Hayek because his analysis of why do centrally planned economies fail is one about our inability to get perfect information about a system. And one of the challenges with a world where people don’t put their own assessment into whether something is worth doing is that you actually lose part of the information, which is the embedded information in the black box that is my head about how I feel about where I am. And the way to capture that is through the staking process.
RICHARD CRAIB: Exactly. And one thing I think it’s worth noting, if you think about, say the organizational design of a company like Two Sigma versus Numerai. Two Sigma —
AZEEM AZHAR: Yeah. Two Sigma are a hedge fund, right?
RICHARD CRAIB: Yeah. Very successful, big hedge fund. They think about someone young who joins, who wants to use machine learning for some strategy at Two Sigma. How long do you think it will take for that young person who just got to the firm, for them to actually put billions of capital against that guy’s model? That won’t happen. They’ll just say “no, no, no, you got to do all your other work first.” So this fact that people are resistant to innovation because it’s risky. And so Numerai is the least resistant to innovation. We are the hedge fund that can capture advances in artificial intelligence faster than any other hedge fund because we’re open.
AZEEM AZHAR: Do you have a measure for that?
RICHARD CRAIB: Well, for us, it’s instant. At the beginning of every week, we don’t have any models. Our users have the models. If they don’t come back, we won’t have anything to trade. But if they do come back, it’s because they want to continue to stake that model. And so whatever they’re coming back with is something we still want to trade. So we’re starting fresh all the time in some way. And that’s why we’re always capturing new stuff.
AZEEM AZHAR: This comes back to some core tenets of economics. I think of Keynes’s beauty contest, where the game is not to predict who’s the prettiest, but to predict who everyone else is going to think is the prettiest. And what happens, I guess, in your world is that come Monday morning, you have no models. The risk that you will end the week with no models is very low because if I’m a data scientist and I know that it’s August and everyone else has gone on a holiday, then there’s an incentive for me to stake that week because there’s more takings for me to take. I’m going to share with fewer people. So you can construct this incentive model internally, which is pretty clever. I’m curious about what the minimum scale for a collective intelligence like this is. Is it 10 people? Is it a hundred? Is it a thousand? Where does it actually start to work?
RICHARD CRAIB: It’s always helpful to have more models. And, in fact, if you think about it, some models like decision trees or XGBoost, that we were talking about earlier, that is an ensemble model.
AZEEM AZHAR: It is, yeah.
RICHARD CRAIB: That is thousands of little trees that all have their own independent ideas of what’s going to happen. So even if you have one model, the right one model to have is a lot of models.
AZEEM AZHAR: So you’ve got some brilliant people, and they are coming in every week, maybe some of them come every week and some take a few days off, few weeks off. And how much money are they making?
RICHARD CRAIB: They have made quite a lot of money, and they make all the money in NMR tokens, which is very volatile. But we’ve paid out at least $20 million to our users, even at the current price of the token. And that’s a lot more than, I think, probably all Kaggle competitions put together. So because of the ability to use cryptocurrency, we’ve been able to pay a lot more.
AZEEM AZHAR: And let’s put that $20 million in context, because, of course, that sounds like a lot, but in a big asset manager, $20 million would disappear in a matter of seconds. So how big is the Numerai fund today? How big was it last year, and how big was it the year before? I guess AUM (assets under management) is the normal yardstick that we use for that.
RICHARD CRAIB: So it’s an interesting way the asset management industry works. It’s basically like, if your thing is not working, no one’s going to put money in. Bbut then once it starts working, people put money in really quickly. And they don’t know when it’s going to work, so they wait three years to see how it goes. And so that’s where we are now. So we had just $7 million in our fund last year, and now we have 70 million and we have $200 million in capacity rights that are allocated to investors to add more to the existing $70 [million] pool. So it’s going to scale quite fast, I think, especially in this market environment where I think we are maybe up three or four percent, five percent this year, but the NASDAQ’s down 30 percent. And so it’s like the perfect time for market neutral.
AZEEM AZHAR: We talked a lot about Medallion, and they’ve done really well. But quite often with quant funds, there are issues when clever mathematics and finance meet, especially when people say, “well, hey, there’s an arbitrage opportunity,” which means it’s riskless. And because it’s riskless, so we can start to excite our returns through leverage. We can borrow a bit of extra money to improve the returns. And I know that Numerai has started to take on some leverage as well. What makes you think that you are more like a Medallion that hasn’t blown up than LTCM (long term capital management), which was full of the smartest guys in the room and wasn’t hugely levered, it was levered 20 to one, and blew up horribly 20 years ago. What is it in your systems design, your incentive design that makes you think, “well, we actually can’t get there.”
RICHARD CRAIB: Well, there is nothing about our trading strategy that has any relationship with LTCM. The first thing is risk models are quite good. If you put our portfolio into a risk model and it gives you the expected volatility, that is actually often quite good. But also risk models are terrible. They’re bad in the sense that if you optimize to do well on a risk model, you’ll blow out all your unknown risks. And so Numerai’s optimizer, it doesn’t do that. It’s very like common sense. Like I said, literally if we have a tech stock that we go along, we have to short a tech stock. That’s just common sense that you don’t want to let that get out of hand — your sector risk. And we don’t really trust the risk models too much. To put a finite point on how precise you can get at this, we say in our terms, our expected volatility will be 10 percent. Okay. We’ve had some of the most volatile markets that you’ve seen in history. Our realized volatility was 9.88 percent. You can know about volatility. You might not have returns, but you can know about risk. I think we’re very good at that. And we’re much more conservative at risk than people might imagine. I think because we are associated with cryptocurrency and a startup, people think we might be crazy at risk. But if you look at the track record now, it starts to paint a very different story.
AZEEM AZHAR: It has been a really, really crazy market this year, this week, the week that we are recording, we’ve seen a really severe collapse in cryptocurrencies and Bitcoin is more than 50 percent off its all time high and Ethereum has had a terrible week, and some other cryptocurrencies have had even worse weeks. What do you observe in the collective intelligence with dislocations like this? Because you are talking about 30 years’ worth of data. We are seeing multi, multi-decade shifts in some asset prices in the matter of days in between the point at which the data was downloaded by your modelers. So how does that make its way?
RICHARD CRAIB: Well, there’s definitely nothing about, like, because of the current environment we change what we do. That’s the opposite of being market neutral really. What I would say is, “okay, here you have a tech crash.” That’s more like, tech is going down a lot faster than other things, like Berkshire Hathaway is up for the year. This is Warren Buffet’s time to brag. And so that’s a big dislocation, but that’s a normal dislocation in terms of 30 years, that type of stuff happens often. And so we’re always neutral to it. We’re neutral to it on the way up, and we were going up. And we’re neutral to it on the way down, and we’re growing up. Quant blew up in COVID, and I want to be sympathetic, I want to be like, “okay, yeah, I guess it was a tough time.” But I also want to be like, “you had bad risk control because we were down 1.5 percent and it looked bad, but come on, you also knew you were taking on risk that could reverse.”
AZEEM AZHAR: So what we have within Numerai is this collective intelligence. Machine learning models that are contributed by people — they get summed up together, and you’ve got it working in a particular data domain, which is equities with a longstanding heritage of being able to provide lots of rich data. How generalizable is this approach, do you think?
RICHARD CRAIB: I think it will absolutely work on every single asset class. I think if we wanted to, we could make a brilliant crypto beta one fund, if we wanted to, and we could make a corporate bond fund and we could do all that with the exact same system. We’ve just decided to focus on hard mode, which is like equity market neutral is hard because there’s so many quants looking at that space. If you can prove yourself in hard mode for three years, like we have, then you’re onto something. But I don’t think it will work in any other industry.
AZEEM AZHAR: Why wouldn’t it work in other industries where we could start to get risk data? Where we could, for example, start to evaluate various climate risk models, like solar radiation levels and water levels and so on. Why couldn’t that approach work?
RICHARD CRAIB: Okay. So that is Kaggle. Kaggle lets businesses host data science competitions, but here’s what’s different to Kaggle versus Numerai. Every model on Kaggle is basically a toy model that’s being used to win a competition. It is not going to be used in production. Models on Numerai are used in live trading on stock markets in 30 different markets around the world. So your model’s actually in production on Numerai and being used in real life. Whereas let’s say there was a Kaggle climate change competition, and someone had a great model. How do you put that model into production? You tell that guy, “hey, we need you to put this model into production.” And you hire that guy into your company, you can’t let him run the model remotely. It’s not very useful in that context, but what Numerai’s cracked is the ability to take signals from models without taking the models, without taking the IP of the users and actually apply that to the markets. Why it’s particularly useful for us is we don’t want one internal model that’s good at this problem. We really want all of them, all the time because that’s how you can be the best in this ultra-competitive space.
AZEEM AZHAR: The characteristics that we see of a trader in the market are, they’re not too different at some level to the characteristics of anything else that is exhibiting intelligence. At some level, the exhibition of intelligence is about having some kind of goal, an ability to sense your environment and understand where you are, and then an ability to select the best strategy or create a new strategy to then move towards your goal. And in a way that’s what Numerai does, your meta model senses the external environment from signals. It’s got this strategy and it knows what its actuation is through your portfolio optimizer, buy a stock, sell a stock, trim a position here, build it there. And it’s got a goal, which is number should go up. So there are some parallels.
RICHARD CRAIB: I think that’s a nice way of abstracting it, but what you’re really talking about is AI. AI can absolutely be used in every single industry and should be and will be. But this particular way of doing crowdsourced AI, I think AI is actually quite a centralized thing. It’s much better for Google to do it because they have all the data than for you to do it. But this particular problem, I think it does make sense to do it this way with multiple contributors.
AZEEM AZHAR: If I look at the performance of your fund over time, a couple of things have evolved, you have got more participants and you’ve got different participants, and they’ve built up their own levels of expertise. But the other thing that’s happened is that you’ve actually really enriched the data. You’ve got many, many more features in your data set today than you did a couple of years ago, but it does mean that a couple of years ago you had a much more parsimonious set of data, and you were still performing — well, maybe I’m being too optimistic. Maybe I’m reading too much into the analogy, but I was just wondering about whether that incentivized ensembling approach that you have tells us something about how we might be able to develop some sort of collective intelligence that’s an artificial collective intelligence?
RICHARD CRAIB: Yeah. It remains to be seen where else you could do it. I just have some doubts about practical realities, like if you had a Kaggle competition where someone figured out a model that could detect cancer from x-rays, the first thing you’d want to do is productionize that model and put it into a device. — not have it be remote, waiting for data scientists to reply with their results. But I think there’s many other things popping up that are interesting, especially the blockchain space is inspiring for that reason where you have people just taking all kinds of things on the blockchain and that can deliver some kind of signal and value potentially over time.
AZEEM AZHAR: You’ve said these things work on equities, and equities are ultimately connected to real things because an equity is an ownership stake in a General Motors, or in a Coca-Cola. But within the blockchain world, more widely ,we are, through the process of tokenization, we are tokenizing real things without a huge company in between them. So if you think about these resources networks, like Filecoin, which is a resource network for storage, or Helium, which is a resource network for radio, those could be assets that could be in a way optimized through a process like this.
RICHARD CRAIB: Yeah. Maybe.
AZEEM AZHAR: Maybe, you’re not so sure, Richard.
RICHARD CRAIB: I’ve been quite skeptical of some things lately. I think you said the word actuating, and if you think about control, you have an action that this has to take. It’s usually the action piece that is missing in a lot of the blockchain stuff. It’s like, yes, you stacked on this thing and you won the governance coin and you can vote on a protocol, but what happened in the real world? What physical reality was changed by this system? And usually it’s nothing.
AZEEM AZHAR: So you said you were feeling a bit more skeptical, and I’ve seen a couple of posts you’d made on Twitter that suggested that. What is the source of the skepticism, do you think?
RICHARD CRAIB: It’s just surprising how long the blockchain’s been around. Where are the revenues of this space after almost 13 years or so of having crypto? And I would argue it’s like 99.9 percent speculation. So you get fees from trading, you get fees from providing leverage to traders. Crypto assets go up because they’re claiming to be governance coins for protocols that will one day be very valuable. It’s very little usage of the coin for anything that matters in the real world. And I would submit that Numerai is staking a model to solve a business problem that we have — which is, we don’t know whether we can trust your model. And by staking it you’re telling us we can trust it. And now that affects the real world because we are trading real equities and making the markets a better place. And I think that’s a much more compelling story than so many other things I see in crypto.
AZEEM AZHAR: And, in a way, what we often hear and what I certainly often hear when I talk to people in crypto is they say, “it’s still early.”
RICHARD CRAIB: Yeah. Yeah.
AZEEM AZHAR: “It’s still early.” And I’ve been hearing that for seven or eight years.
RICHARD CRAIB: It’s still late. It’s still late.
AZEEM AZHAR: Yeah. It’s still late. Right. Right. In other words, it’s late to the party, it’s late to deliver the value. But outside of the example that you’ve given, are there other examples that you can think of that make you think, okay, we are actually showing some value?
RICHARD CRAIB: I like Uniswap. Uniswap lets people trade in a decentralized way, with no centralized exchange. And you could argue the fees are very cheap and the slippage is very low, and it’s a very impressive thing that they’ve done. But it’s mainly used for speculation. So it is useful to our users who are buying coins to stake, but it is also mainly used for speculation. And is that a good thing? I very much doubt it, very much doubt it.
AZEEM AZHAR: It’s a dose of realism. The market itself has certainly had a shock to its optimistic vibes, the crypto market, in the last few weeks.
RICHARD CRAIB: Well, I picked this time because no one would’ve listened to this a year ago, but I have been saying this kind of stuff for a while, but now’s I think the right time to have the reassessment.
AZEEM AZHAR: Well, thank you for that dose of realism and for the insights into building collective intelligence to tackle the financial markets. Richard Craib, great to speak to you.
RICHARD CRAIB: Thank you so much.
AZEEM AZHAR: Well, I hope you found this conversation as insightful and intriguing as I did. Now, if you are curious about the future of finance — whether it’s money in the metaverse, payments, or where DeFi [decentralized finance] is going, then please listen to some of my other conversations: Citi’s Ronit Ghose, Visa’s Charlotte Hogg, and the co-founder of Chainlink Sergey Nazarov — to name but a few. You can find them all in our archive. And do check out that podcast feed for those episodes and more. Today’s episode was produced by Fred Casella and Marija Gavrilov, and researched by Chantel Smith. Our sound editor is Bojan Sabioncello. The Exponential View podcast is a production E to the Pi i Plus One, Limited.