Author: Lynn Yang, Silicon Release
One evening last week, I was listening to a podcast on Spotify while eating dinner.
The guest of honor was Vinod Khosla, the first investor of OpenAI and founder of Khosla Ventures, a top technology venture capital firm in Silicon Valley.
Obviously, it’s not every day that you get the opportunity to get inside the mind of OpenAI’s first investor.
So I sorted out Khosla's core ideas and shared them with you. Here are Khosla's 7 core ideas about AI:
(one)
Related background: The moderator asked Khosla, the first investor in OpenAI, what AI use cases he primarily uses in his daily life.
Khosla: Mainly two, ChatGPT and Tesla's autonomous driving.
The amount of times I’ve said just go with Tesla, it’s amazing. It feels like full autopilot. You know the other night, I landed at 3 in the morning. And I was like: I’m so tired, I’m not going to be a safe driver. So I just said: Take me home. And that was an amazing experience.
These are the main uses of AI for me. And I use these two uses many times a day.
About ChatGPT, I'm using it to plan my spring garden.
I said to ChatGPT: I want plants that grow in zone 9a. I want the height of each zone because I’m stratifying them.
Then I also said: I want to have some flowers that bloom in the spring, some flowers that bloom in the early summer, some flowers that bloom in the late summer, and some flowers that bloom in the fall.
This is actually a design job. I had ChatGPT arrange 20 plants for me, and it gave me all this information, including: watering amount, climate zone, altitude, areas with no sun, areas with semi-shade, areas with bright shade.
So, ChatGPT does amazing things. Things that would have taken me 3-4 hours. So yes, I designed the garden all by myself, I don’t hire a designer. And I can promise you: my garden is blooming now, you probably won’t believe it.
(two)
Related background: The host asked Khosla, as the first investor in OpenAI, what he thought of Apple’s announcement of its AI strategy and cooperation with OpenAI, and what impact this cooperation would have on the AI startup ecosystem in the next few years?
Khosla: I think Apple needs to do something about AI, after all, Siri's reputation has started to deteriorate.
First, the smart thing for Apple to do is to keep it open and allow users to access any LLM. But Apple did choose to embed and build it into IOS, which was enough to make Elon Musk uneasy, and he threatened to ban Apple devices.
So I think what’s more important about this is that Apple is actually demonstrating something very important: how do we interact with computers?
I think over time Siri will evolve into the beginning of a real interface for humans. From that perspective, I think it's big news because we're seeing the beginning of that. It's exciting.
From OpenAI’s perspective, this collaboration has clearly identified OpenAI’s best position in the competition - direct interaction with users. In fact, many people want this business.
On the other hand, I do think that Apple should have thought carefully about this - where does Apple think the best AI will be in 1-2 years?
Therefore, in many ways, Apple's collaboration with OpenAI is a validation of OpenAI and a very important milestone in how humans interact with machines.
(three)
Related background: The host asked Khosla that Apple's case shows that a small model can do a lot of things. So what is the position of the big model in the future? And if everyone wants a small model, will the future become like this: you can talk to many people, some with an IQ of 50, some with an IQ of 100, and some with an IQ of 10,000. Then the key is, where do you want to spend your money, do you want to spend your money to ask a person with an IQ of 10,000 a question, or to ask a person with an IQ of maybe only 70 but who knows the content of your email? Here, it involves the balance of product processing direction and the cost calculated for the model. Do you think the future is such a competitive game?
Khosla: The small model and the large model are different and are not interchangeable.
Also, I might not agree with the future of IQ assumptions. In fact, what I think is going to happen is that computing is going to become really cheap in the future.
My bet is that in a year, compute will cost 1/5 to 1/10 of what it costs today. So my advice to all our startups is to ignore your compute costs, because any assumptions you make, any dollar you spend on optimizing software, will be worth nothing in a year.
The reason: Every owner of a big model is trying to reduce the cost of computing. And as engineers at OpenAI, Google, and cloud computing companies work to reduce the cost of expensive AI chips, computing will soon become very cheap.
So forget about it, and count on the competition between the various big models on the market, like Google’s Gemini and OpenAI, to drive costs down to the point where they don’t matter. In fact, as long as they get down to 10% or less of what they are currently, it won’t matter.
Also, currently the cost of training a large model that outperforms other large models is an order of magnitude higher. This is why I think open-sourcing models is not feasible, because the training cost is too high. But once you have trained them, you want to make them as widely available as possible for two reasons:
First, you want to get the most bang for your buck, and the model that has the lowest cost is going to get the most bang for your buck.
Second, but more importantly, there is a lot of data available for you to train the next generation of models.
So you want to maximize utilization for a number of reasons. If you're playing the long game, and I think the AI model games are mostly played on a 5-year timeframe, not a one-year timeframe. Over that timeframe, the costs go down.
Today, Nvidia extracts a pretty good tax from everyone, but each model will run on multiple types of GPUs or compute, and they require the most data generation. So I believe: in the next few years, revenue will not be an important metric for model companies.
Of course, you don’t want to lose too much money that you can’t afford. But you don’t want to make a lot of money because you’re trying to get a lot of user usage, and you’re trying to get a lot of data from user usage and learn to become a better model.
I do think that in terms of intelligence, there’s still a lot to be gained from models, whether it’s in reasoning, probabilistic thinking, or some kind of pattern matching, etc. There’s a lot of room for these models to get better.
So I think you're going to see amazing progress almost every year. Some companies execute better than others, and that's the main difference between companies: OpenAI is very good at execution, Google has great technology but lacks clarity in execution.
(Four)
Related background: The host asked Khosla, if you think about the five-year time frame, now, some people in the tech community really believe that the value of AI will all go into existing large companies. But even so, it has been commoditized. So what do you think the five-year outlook will be? And what are the AI themes you are more concerned about, and which themes are not involved in existing large companies?
Khosla: So I don’t believe that if you’re building base models and trying to compete with OpenAI and Google, that’s a good position to be in.
Because the large LLMs, will belong to large players that can run on very large clusters, and they are the ones who can pay for proprietary content/data, whether it's paying for Reddit, or for a company that has access to every scientific article.
So the biggest players do have an advantage.
But on the other hand, we recently announced an investment in Symbolica, a symbolic logic company. They take a very different approach to modeling. It doesn't rely on large amounts of data, and it doesn't rely on large amounts of computation. This is actually a high-risk, high-upside investment. If Symbolica succeeds, it will be dramatic.
So I think even at the model level, there are other approaches. If I call my friend Josh Tenenbaum at MIT, he would say the biggest contribution was probabilistic programming. Because human thinking is probabilistic, which is different from pattern matching. That's an important factor.
So I think the underlying technology is far from done. We are leveraging the Transformer model more and more, but there are other models to be developed. It's just that everyone is afraid to invest in something other than the Transformer model. And we didn't.
You know, I'm very into esoteric things. In fact, Symbolica is a theory called category theory, which most mathematicians have never heard of.
So we made a big bet probably about 15, 18 months ago. I thought it was stupid to invest in the cloud because people were buying GPUs to build clouds, but they were going to lose to Amazon, they were going to lose to Amazon’s scale and efficiency, and to Microsoft.
Both of those companies are doing custom chips so they don’t have to pay the Nvidia tax for a few years. Yes, there’s still a lot of catching up to do in chips, with AMD. But at the next level, at the application level, there’s a huge opportunity here.
(five)
Related background: In the content below, Khosla talked about what he thinks are the huge opportunities for AI applications, and listed many examples.
Khosla: One of my big predictions is that in the future, almost all expertise will be free.
So by that logic, whether you're talking about primary care physicians, teachers, structural engineers, or oncologists. There are hundreds, if not thousands, of specialty areas, and in each of them, there will be a very successful company.
We also recently invested in a company that’s building a structural engineer. Of course, we’ve invested in very popular things like Devin. Everybody knows Devin, they’re building an AI programmer, they’re not building a tool like Copilot for programmers, they’re building a programmer. But we also just invested in a company that’s building a structural engineer, and they’re called Hedral.
One thing that’s curious is, how many structural engineers are there today? How much do we spend on structural engineering? You give a building structure to a structural engineer, and two months later you get something and one variation. But you can get 5 variations from an AI structural engineer in 5 hours and save months on the building project. So that’s a great niche example. But this could be a multi-billion dollar niche.
So, my point is: …
