Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity | Lex Fridman Podcast #452

**Dario Amodei:** if you extrapolate the curves that we've had so far right if if you say well I don't know we're starting to get to like PhD level and and last year we were at undergraduate level and the year before we were at like the level of a high school student again you can you can quibble with at what tasks and for what we're still missing modalities but those are being added like computer use was added like image generation has been added if you just kind of like eyeball the rate at which these capabilities are increasing it does make you think that we'll get there by 2026 or 2027

**Dario Amodei:** I think there are still worlds where it doesn't happen in in a 100 years those world the number of those worlds is rapidly decreasing we are rapidly running out of truly convincing blockers truly compelling reasons why this will not happen in the next few years the scale up is very quick like we we do this today we make a model and then we deploy thousands maybe tens of thousands of instances of it I think by the time you know certainly within two to three years whether we have these super powerful AIS or not ERS are going to get to the size where you'll be able to deploy millions of these

**Lex Fridman:** I am optimistic about meaning I worry about economics and the concentration of power that's actually what I worry about more the abuse of power and AI increases the amount of power in the world and if you concentrate that power and abuse that power it can do immeasurable damage

**Dario Amodei:** yes it's very frightening it's very it's very frightening

## Introduction

**Lex Fridman:** the following is a conversation with Dario amade CEO of anthropic the company that created Claude that is currently and often at the top of most llm Benchmark leader boards on top of that Dario and the anthropic team have been outspoken advocates for taking the topic of AI safety very seriously and they have continued to publish a lot of fascinating AI research on this and other topics

**Lex Fridman:** I'm also joined afterwards by two other brilliant people from propic first Amanda ascal who is a researcher working on alignment and fine-tuning of Claude including the design of claude's character and personality a few folks told me she has probably talked with Claude more than any human at anthropic so she was definitely a fascinating person to talk to about prompt engineering and practical advice on how to get the best out of Claude

**Lex Fridman:** after that chrisa stopped by for chat he's one of the pioneers of the field of mechanistic interpretability which is an exciting set of efforts that aims to reverse engineer neural networks to figure out what's going on inside inferring behaviors from neural activation patterns inside the network this is a very promising approach for keeping future super intelligent AI systems safe for example by detecting from the activations when the model is trying to deceive the human it is talking to this is Alex Freedman podcast to support it please check out our sponsors in the description and now dear friends here's Dario amade

## Scaling Laws and the Scaling Hypothesis

**Lex Fridman:** let's start with a big idea of scaling laws and the scaling hypothesis what is it what is its history and where do we stand today

**Dario Amodei:** so I can only describe it as it you know as it relates to kind of my own experience but I've been in the AI field for about uh 10 years and it was something I noticed very early on so I first joined the AI world when I was uh working at BYU with Andrew in in late 2014 which is almost exactly 10 years ago now and the first thing we worked on was speech recognition systems

**Dario Amodei:** and in those days I think deep learning was a new thing it had made lots of progress but everyone was always saying we don't have the algorithms we need to succeed you know we we we we're we're not we're only matching a tiny tiny fraction there's so much we need to kind of discover algorithmically we haven't found the picture of how to match the human brain

**Dario Amodei:** uh and when you know in some ways was fortunate I was kind of you know you can have almost beginner's luck right I was like a a newcomer to the field and you know I looked at the neural net that we were using for speech the recurrent neural networks and I said I don't know what if you make them bigger and give them more layers and what if you scale up the data along with this right I just saw these as as like independent dials that you could turn

**Dario Amodei:** and I noticed that the model started to do better and better as you gave them more data as you as you made the models larger as you trained them for longer um and I I didn't measure things precisely in those days but but along with with colleagues we very much got the informal sense that the more data and the more compute and the more training you put into these models the better they perform

**Dario Amodei:** and so initially my thinking was hey maybe that is just true for speech recognition systems right maybe maybe that's just one particular quirk one particular area I think it wasn't until 2017 when I first saw the results from gpt1 that it clicked for me that language is probably the area in which we can do this we can get trillions of words of language data we can train on them and the models we were training in those days were tiny you could train them on one to eight gpus whereas you know now we train jobs on tens of thousands soon going to hundreds of thousands of gpus

**Dario Amodei:** and so when I when I saw those two things together um and you know there were a few people like ilaser who who you've interviewed who had somewhat similar reviews right he might have been the first one although I think a few people came to came to similar views around the same time Right There Was You Know Rich Sutton's bitter lesson there was gur wrote about the scaling hypothesis but I think somewhere between 2014 and 2017 was when it really clicked for me when I really got conviction that hey we're going to be able to do these incredibly wide cognitive tasks if we just if we just scale up the models

**Dario Amodei:** and at at every stage of scaling there are always arguments and you know when I first heard them honestly I thought probably I'm the one who's wrong and you know all these all these experts in the field are right they know the situation better better than I do right there's you know the Chomsky argument about like you can get syntactics but you can't get semantics there's this idea oh you can make a sentence make sense but you can't make a paragraph makes sense the latest one we have today is uh you know we're going to run out of data or the data isn't high quality enough or models can't reason and and each time every time we manage to we manage to either find a way around or scaling just is the way around um sometimes it's one sometimes it's the other

**Dario Amodei:** uh and and so I'm now at this point I I I still think you know it's it's it's always quite uncertain we have nothing but inductive inference to tell us that the next few years are going to be like the next the last 10 years but but I've seen I've seen the movie enough times I've seen the story happen for for enough times to to really believe that probably the scaling is going to continue and that there's some magic to it that we haven't really explained on a theoretical basis yet

**Lex Fridman:** and of course the scaling here is bigger networks bigger data bigger compute

**Dario Amodei:** yes all in in particular linear scaling up of bigger networks bigger training times and uh more and and more data uh so all of these things almost like a chemical reaction you know you have three ingredients in the chemical reaction and you need to linearly scale up the three ingredients if you scale up one not the others you run out of the other reagents and and the reaction stops but if you scale up everything everything in series then then the reaction can proceed

**Dario Amodei:** and of course now that you have this kind of empirical scienceart you can apply it to other uh more nuanced things like scaling laws applied to interpretability or scaling laws applied to posttraining or just seeing how does this thing scale but the big scaling law I guess the underlying scaling hypothesis has to do with big networks Big Data leads to intelligence

**Dario Amodei:** yeah we've we've documented scaling laws in lots of domains other than language right so uh initially the the paper we did that first showed it was in early 2020 where we first showed it for language there was then some work late in 2020 where we showed the same thing for other modalities like images video text to image image to text math they all had the same pattern and and you're right now there are other stages like posttraining or there are new types of reasoning models and in in in all of those cases that we've measured we see similar similar types of scaling laws

## Why Bigger Is Better

**Lex Fridman:** a bit of a philosophical question but what's your intuition about why bigger is better in terms of network size and data size why does it lead to more intelligent models

**Dario Amodei:** so in my previous career as a as a biophysicist so I did physics undergrad and then biophysics in in in in grad school so I think back to what I know as a physicist which is actually much less than what some of my colleagues at anthropic have in terms of in terms of expertise in physics

**Dario Amodei:** uh there's this there's this concept called the one over F noise and one overx distributions um where where often um uh you know just just like if you add up a bunch of natural processes you get gaussian if you add up a bunch of kind of differently distributed natural processes if you like if you like take a take a um probe and and hook it up to a resistor the distribution of the thermal noise in the resistor goes as one over the frequency um it's some kind of natural convergent distribution

**Dario Amodei:** uh and and I I I I and and I think what it amounts to is that if you look at a lot of things that are that are produced by some natural process that has a lot of different scales right not a gaussian which is kind of narrowly distributed but you know if I look at kind of like large and small fluctuations that lead to lead to electrical noise um they have this decaying 1 overx distribution

**Dario Amodei:** and so now I think of like patterns in the physical world right if I if or or in language if I think about the patterns in language there are some really simple patterns some words are much more common than others like the' then there's basic noun verb structure then there's the fact that you know you know nouns and verbs have to agree they have to coordinate and there's the higher level sentence structure then there's the Thematic structure of paragraphs

**Dario Amodei:** and so the fact that there's this regressing structure you can imagine that as you make the networks larger first they capture the really simple correlations the really simple patterns and there's this long taale of other patterns and if that long taale of other patterns is really smooth like it is with the one over F noise in you know physical processes like like like resistors then you could imagine as you make the network larger it's kind of capturing more and more of that distribution and so that smoothness gets reflected in how well the models are at predicting and how well they perform

**Dario Amodei:** language is an evolved process right we've we've developed language we have common words and less common words we have common expressions and less common Expressions we have ideas cliches that are expressed frequently and we have novel ideas and that process has has developed has evolved with humans over millions of years and so the the the guess and this is pure speculation would be would be that there is there's some kind of longtail distribution of of of the distribution of these ideas

**Lex Fridman:** so there's the long tail but also there's the height of the hierarchy of Concepts that you're building up so the bigger the network presumably you have a higher capacity to

**Dario Amodei:** exactly if you have a small Network you only get the common stuff right if if I take a tiny neural network it's very good at understanding that you know a sentence has to have you know verb adjective noun right but it's it's terrible at deciding what those verb adjective and noun should be and whether they should make sense if I make it just a little bigger it gets good at that then suddenly it's good at the sentences but it's not good at the paragraphs and so the these these rare and more complex patterns get picked up as I add as I add more capacity to the network

## Ceiling of Scaling

**Lex Fridman:** well the natural question then is what's the ceiling of this like how complicated and complex is the real world how much of stuff is there to learn

**Dario Amodei:** I don't think any of us knows the answer to that question um I my strong Instinct would be that there's no ceiling below level of humans right we humans are able to understand these various patterns and so that that makes me think that if we continue to you know scale up these these these models to kind of develop new methods for training them and scaling them up uh that will at least get to the level that we've gotten to with humans

**Dario Amodei:** there's then a question of you know how much more is it possible to understand than humans do how much how much is it possible to be smarter and more perceptive than humans I I would guess the answer has has got to be domain dependent if I look at an area like biology and you know I wrote this essay Machines of Loving Grace it seems to me that humans are struggling to understand the complexity of biology right if you go to Stanford or to Harvard or to Berkeley you have whole Departments of you know folks trying to study you know like the immune system or metabolic pathways and and each person understands only a tiny bit part of it specializes and they're struggling to combine their knowledge with that of with that of other humans

**Dario Amodei:** and so I have an instinct that there's there's a lot of room at the top for AIS to get smarter if I think of something like materials in the in the physical world or you know um like addressing you know conflicts between humans or something like that I mean you know it it may be there's only some of these problems are not intractable but much harder and and it it may be that there's only there's only so well you can do with some of these things right just like with speech recognition there's only so clear I can hear your speech

**Dario Amodei:** so I think in some areas there may be ceilings in in in you know that are very close to what humans have done in other areas those ceilings may be very far away and I think we'll only find out when we build these systems uh there's it's very hard to know in advance we can speculate but we can't be sure

**Lex Fridman:** and in some domains the ceiling might have to do with human bureaucracies and things like this as you're right about

**Dario Amodei:** yes so humans fundamentally have to be part of the loop that's the cause of the ceiling not maybe the limits of the intelligence

**Dario Amodei:** yeah I think in many cases um you know in theory technology could change very fast for example all the things that we might invent with respect to biology um but remember there's there's a you know there's a clinical trial system that we have to go through to actually administer these things to humans I think that's a mixture of things that are unnecessary and bureaucratic and things that kind of protect the Integrity of society and the whole challenge is that it's hard to tell it's hard to tell what's going on uh it's hard to tell which is which

**Dario Amodei:** right my my view is definitely I think in terms of drug development we my view is that we're too slow and we're too conservative but certainly if you get these things wrong you know it's it's possible to to to risk people's lives by by being by being by being too Reckless and so at least at least some of these human institutions are in fact protecting people so it's it's all about finding the balance I strongly suspect that balance is kind of more on the side of pushing to make things happen faster but there is a balance

## Limits of Scaling

**Lex Fridman:** if we do hit a limit if we do hit a Slowdown in the scaling laws what do you think would be the reason is it compute limited data limited uh is it something else idea limited

**Dario Amodei:** so a few things now we're talking about hitting the limit before we get to the level of of humans and the skill of humans um so so I think one that's you know one that's popular today and I think you know could be a limit that we run into I like most of the limits I would bet against it but it's definitely possible is we simply run out of data there's only so much data on the internet and there's issues with the quality of the data right you can get hundreds of trillions of words on the internet but a lot of it is is repetitive or it's search engine you know search engine optimization driil or maybe in the future it'll even be text generated by AIS itself uh and and so I think there are limits to what to to what can be produced in this way

**Dario Amodei:** that said we and I would guess other companies are working on ways to make data synthetic uh where you can you know you can use the model to generate more data of the type that you have that you have already or even generate data from scratch if you think about uh what was done with uh deep mines Alpha go zero they managed to get a bot all the way from you know no ability to play Go whatsoever to above human level just by playing against itself there was no example data from humans required in the the alphao zero version of it

**Dario Amodei:** the other direction of course is these reasoning models that do Chain of Thought and stop to think um and and reflect on their own thinking in a way that's another kind of synthetic data coupled with reinforcement learning so my my guess is with one of those methods we'll get around the data limitation or there may be other sources of data that are that are available

**Dario Amodei:** um we could just observe that even if there's no problem with data as we start to scale models up they just stop getting better it's it seemed to be a a reliable observation that they've gotten better that could just stop at some point for a reason we don't understand um the answer could be that we need to uh you know we need to invent some new architecture

**Dario Amodei:** um it's been there have been problems in the past with with say numerical stability of models where it looked like things were were leveling off but but actually you know know when we when we when we found the right Unblocker they didn't end up doing so so perhaps there's new some new optimization method or some new uh Technique we need to to unblock things I've seen no evidence of that so far but if things were to to slow down that perhaps could be one reason

**Lex Fridman:** what about the limits of compute meaning uh the expensive uh nature of building bigger and bigger data centers

**Dario Amodei:** so right now I think uh you know most of the Frontier Model companies I would guess are are operating you know roughly you know $1 billion scale plus or minus a factor of three right those are the models that exist now or are being trained now uh I think next year we're going to go to a few billion and then uh 2026 we may go to uh uh you know above 10 10 10 billion and probably by 2027 their Ambitions to build hundred hundred billion dollar uh hundred billion dollar clusters and I think all of that actually will happen there's a lot of determination to build the compute to do it within this country uh and I would guess that it actually does happen

**Dario Amodei:** now if we get to 100 billion that's still not enough compute that's still not enough scale then either we need even more scale or we need to develop some way of doing it more efficiently of Shifting The Curve

**Dario Amodei:** um I think be between all of these one of the reasons I'm bullish about powerful AI happening so fast is just that if you extrapolate the next few points on the curve we're very quickly getting towards human level ability right some of the new models that that we developed some some reasoning models that have come from other companies they're starting to get to what I would call the PHD or professional level right if you look at their their coding ability

**Dario Amodei:** um the latest model we released Sonet 3.5 the new or updated version it gets something like 50% on sbench and sbench is an example of a bunch of professional real world software engineering tasks at the beginning of the year I think the state-of-the-art was three or 4% so in 10 months we've gone from 3% to 50% on this task and I think in another year we'll probably be at 90% I mean I don't know but might might even be might even be less than that uh we've seen similar things in graduate level math physics and biology from Models like open AI 01

**Dario Amodei:** uh so uh if we if we just continue to extrapolate this right in terms of skill skill that we have I think if we extrapolate the straight curve Within a few years we will get to these models being you know above the the highest professional level in terms of humans now will that curve continue you've pointed to and I've pointed to a lot of reasons why you know possible reasons why that might not happen but if the if the extrapolation curve continues that is the trajectory we're on

## Competition and Race to the Top

**Lex Fridman:** so anthropic has several competitors it'd be interesting to get your sort of view of it all open aai Google xai meta what does it take to win in the broad sense of win in the space

**Dario Amodei:** yeah so I want to separate out a couple things right so you know anthropics anthropic mission is to kind of try to make this all go well right and and you know we have a theory of change called race to the top right race to the top is about trying to push the other players to do the right thing by setting an example it's not about being the good guy it's about setting things up so that all of us can be the good guy

**Dario Amodei:** I'll give a few examples of this early in the history of anthropic one of our co-founders Chris Ola who I believe you're you're interviewing soon you know he's the co-founder of the field of mechanistic interpretability which is an attempt to understand what's going on inside AI models

**Dario Amodei:** uh so we had him and one of our early teams focus on this area of interpretability which we think is good for making models safe and transparent for three or four years that had no commercial application whatsoever it still doesn't today we're doing some early betas with it and probably it will eventually but uh you know this is a very very long research bed in one in which we've we've built in public and shared our results publicly and and we did this because you know we think it's a way to make models safer

**Dario Amodei:** an interesting thing is that as we've done this other companies have started doing it as well in some cases because they've been inspired by it in some cases because they're worried that uh you know if if other companies are doing this that look more responsible they want to look more responsible too no one wants to look like the irresponsible ible actor and and so they adopt this they adopt this as well

**Dario Amodei:** when folks come to anthropic interpretability is often a draw and I tell them the other places you didn't go tell them why you came here um and and then you see soon that there that there's interpretability teams else elsewhere as well and in a way that takes away our competitive Advantage because it's like oh they now others are doing it as well but it's good it's good for the broader system

**Dario Amodei:** and so we have to invent some new thing that we're doing others aren't doing as well and the hope is to basically bid up bid up the importance of of of doing the right thing and it's not it's not about us in particular right it's not about having one particular good guy other companies can do this as well if they if they if they join the race to do this that's that's you know that's the best news ever right um uh it's it's just it's about kind of shaping the incentives to point upward instead of shaping the incentives to point to point downward

## Mechanistic Interpretability

**Lex Fridman:** and we should say this example the field of uh mechanistic interpretability is just a a rigorous non handwavy way of doing AI safety

**Dario Amodei:** yes or it's tending that way trying to I mean I I think we're still early um in terms of our ability to see things but I've been surprised at how much we've been able to look inside these systems and understand what we see right unlike with the scaling laws where it feels like there's some you know law that's driving these models to perform better on on the inside the models aren't you know there's no reason why they should be designed for us to understand them right they're designed to operate they're designed to work just like the human brain or human biochemistry they're not designed for a human to open up the hatch look inside and understand them

**Dario Amodei:** but we have found and you know you can talk in much more detail about this to Chris that when we open them up when we do look inside them we we find things that are surprisingly interesting

**Lex Fridman:** and as a side effect you also get to see the beauty of these models you get to explore the sort of uh the beautiful n nature of large neural networks through the me turb kind ofy

**Dario Amodei:** I'm amazed at how clean it's been I I'm amazed at things like induction heads I'm amazed at things like uh you know that that we can you know use sparse autoencoders to find these directions within the networks uh and that the directions correspond to these very clear Concepts

## Golden Gate Bridge Claude

**Dario Amodei:** we demonstrated this a bit with the Golden Gate Bridge clad so this was an experiment where we found a direction inside one of the the neural network layers that corresponded to the Golden Gate Bridge and we just turned that way up and so we we released this model as a demo it was kind of half a joke uh for a couple days uh but it was it was illustrative of of the method we developed

**Dario Amodei:** and uh you could you could take the Golden Gate you could take the model you could ask it about anything you know you know it would be like how you could say how was your day and anything you asked because this feature was activated would connect to the Golden Gate Bridge so it would say you know I'm I'm I'm feeling relaxed and expansive much like the the arches of the Golden Gate Bridge or you know it would masterfully change topic to the Golden Gate Bridge and it integrated

**Lex Fridman:** there was also a sadness to it to to the focus ah had on the Golden Gate Bridge I think people quickly fell in love with it

**Dario Amodei:** I think so people already miss it because it was taken down I think after a day

**Lex Fridman:** somehow these interventions on the model um where where where where you kind of adjust Its Behavior somehow emotionally made it seem more human than any other version of the model strong personality strong ID

**Dario Amodei:** strong personality it has these kind of like obsessive interests you know we can all think of someone who's like obsessed with something so it does make it feel somehow a bit more human

## Claude Model Lineup

**Lex Fridman:** let's talk about the present let's talk about Claude so this year A lot has happened in March claw 3 Opa Sonet Hau were released then claw 35 Sonet in July with an updated version just now released and then also claw 35 hi coup was released okay can you explain the difference between Opus Sonet and Haiku and how we should think about the different versions

**Dario Amodei:** yeah so let's go back to March when we first released uh these three models so you know our thinking was you different companies produce kind of large and small models better and worse models we felt that there was demand both for a really powerful model um you know and you that might be a little bit slower that you'd have to pay more for and also for fast cheap models that are as smart as they can be for how fast and cheap

**Dario Amodei:** right whenever you want to do some kind of like you know difficult analysis like if I you know I want to write code for instance or you know I want to I want to brainstorm ideas or I want to do creative writing I want the really powerful model but then there's a lot of practical applications in a business sense where it's like I'm interacting with a website I you know like I'm like doing my taxes or I'm you know talking to uh you know to like a legal adviser and I want to analyze a contract or you know we have plenty of companies that are just like you know you know I want to do autocomplete on my on my IDE or something uh and and for all of those things you want to act fast and you want to use the model very broadly so we wanted to serve that whole spectrum of needs

**Dario Amodei:** um so we ended up with this uh you know this kind of poetry theme and so what's a really short poem it's a Haik cou and so Haiku is the small fast cheap model that is you know was at the time was released surprisingly surprisingly uh intelligent for how fast and cheap it was uh sonnet is a is a medium-sized poem right a couple paragraphs since o Sonet was the middle model it is smarter but also a little bit slower a little bit more expensive and and Opus like a magnum opus is a large work uh Opus was the the largest smartest model at the time

**Dario Amodei:** um so that that was the original kind of thinking behind it um and our our thinking then was well each new generation of models should shift that tradeoff curve uh so when we release Sonet 3.5 it has the same roughly the same you know cost and speed as the Sonet 3 Model uh but uh it it increased its intelligence to the point where it was smarter than the original Opus 3 Model uh especially for code but but also just in general

**Dario Amodei:** and so now you know we've shown results for a Hau 3.5 and I believe Hau 3.5 the smallest new model is about as good as Opus 3 the largest old model so basically the aim here is to shift the curve and then at some point there's going to be an opus 3.5

**Dario Amodei:** um now every new generation of models has its own thing they use new data their personality changes in ways that we kind of you know try to steer but are not fully able to steer and and so uh there's never quite that exact equivalence the only thing you're changing is intelligence um we always try and improve other things and some things change without us without us knowing or measuring so it's it's very much an inexact science in many ways the manner and personality of these models is more an art than it is a science

## Model Development Process

**Lex Fridman:** so what is sort of the reason for uh the span of time between say Claude Opus 3 and 35 what is it what takes that time if you can speak to

**Dario Amodei:** yeah so there's there's different there's different uh processes um uh there's pre-training which is you know just kind of the normal language model training and that takes a very long time um that uses you know these days you know tens you know tens of thousands sometimes many tens of thousands of uh gpus or tpus or tranium or you know what we use different platforms but you know accelerator chips um often often training for months

**Dario Amodei:** uh there's then a kind of posttraining phase where we do reinforcement learning from Human feedback as well as other kinds of reinforcement learning that that phase is getting uh larger and larger now and you know you know often that's less of an exact science it often takes effort to get it right um models are then tested with some of our early Partners to see how good they are and they're then tested both internally and externally for their safety particularly for catastrophic and autonomy r risks

**Dario Amodei:** uh so uh we do internal testing according to our responsible scaling policy which I you know could talk more about that in detail and then we have an agreement with the US and the UK AI safety Institute as well as other third-party testers in specific domains to test the models for what are called cbrn risk chemical biological radiological and nuclear which are you know we don't think that models pose these risks seriously yet but but every new model we want to evaluate to see if we're starting to get close to some of these these these more dangerous um uh these more dangerous capabilities

**Dario Amodei:** so those are the phases and then uh you know then then it just takes some time to get the model working in terms of inference and launching it in the API so there's just just a lot of steps to uh to actually to actually making a model work

**Dario Amodei:** and of course you know we're always trying to make the processes as streamlined as possible right we want our safety testing to be rigorous but we want it to be RoR ous and to be you know to be automatic to happen as fast as it can without compromising on rigor same with our pre-training process and our posttraining process so you know it's just like building anything else it's just like building airplanes you want to make them you know you want to make them safe but you want to make the process streamlined and I think the creative tension between those is is you know is an important thing and making the models work

**Lex Fridman:** yeah uh rumor on the street I forget who was saying that uh anthropic is really good tooling so I uh probably a lot of the challenge here is on the software engineering side is to build the tooling to to have a like a efficient low friction interaction with the infrastructure

**Dario Amodei:** you would be surprised how much of the challenges of uh you know building these models comes down to you know software engineering performance engineering you know you you know from the outside you might think oh man we had this Eureka breakthrough right you know this movie with the science we discovered it we figured it out but but but I think I think all things even even even you know incredible discoveries like they they they they they almost always come down to the details um and and often super super boring details

**Dario Amodei:** I can't speak to whether we have better tooling than than other companies I mean you know I haven't been at those other companies at least at least not recently um but it's certainly something we give a lot of attention to

## Pre-training and Post-training

**Lex Fridman:** I don't know if you can say but from three from CLA 3 to CLA 35 is there any extra pre-training going on or is they mostly focus on the post-training there's been leaps in performance

**Dario Amodei:** yeah I think I think at any given stage we're focused on improving everything at once um just just naturally like there are different teams each team makes progress in a particular area in in in making a particular you know their particular segment of the relay race better and it's just natural that when we make a new model we put we put all of these things in at once

**Lex Fridman:** so the data you have like the preference data you get from rhf is that applicable is there ways to apply it to newer models as it get trained up

**Dario Amodei:** yeah preference data from old models sometimes gets used for new models although of course uh it it performs somewhat better when it's you know trained on it's trained on the new models note that we have this you know constitutional AI method such that we don't only use preference data we kind of there's also a post-t trainining process where we train the model against itself and there's you know new types of post training the model against itself that are used every day so it's not just RF it's a bunch of other methods as well um post training I think you know it's becoming more and more sophisticated

## Benchmarks and Coding Performance

**Lex Fridman:** well what explains the big leap in performance for the new Sona 35 I mean at least in the programming side and maybe this is a good place to talk about benchmarks what does it mean to get better just the number went up but you know I I I program but I also love programming and I um claw 35 through cursor is what I use uh to assist me in programming and there was at least experientially anecdotally it's gotten smarter at programming so what like what what does it take to get it uh to get it smarter

**Dario Amodei:** we observe that as well by the way there were a couple uh very strong Engineers here at anthropic um who all previous code models both produced by us and produced by all the other companies hadn't really been useful to to hadn't really been useful to them you know they said you know maybe maybe this is useful to beginner it's not useful to me but Sonet 3.5 the original one for the first time they said oh my God this helped me with something that you know that it would have taken me hours to do this is the first model that has actually saved me time so again the water line is rising and and then I think you know the new Sonet has been has been even better

**Dario Amodei:** in terms of what it what it takes I mean I'll just say it's been across the board it's in the pre-training it's in the posttraining it's in various evaluations that we do we've observed this as well

**Dario Amodei:** and if we go into the details of the Benchmark so s bench is basically you know since since you know since since you're a programmer you know you'll be familiar with like PLL requests and you know uh just just PLL requests are like you know the like a sort of a sort of atomic unit of work you know you could say I'm you know I'm implementing one I'm implementing one thing

**Dario Amodei:** um uh and and so sbench actually gives you kind of a real world situation where the codebase is in a current state and I'm trying to implement something that's you know that's described in described in language we have internal benchmarks where we where we measure the same thing and you say just give the model free reign to like you know do anything run run run anything edit anything um how how well is it able to complete these tasks and it's that Benchmark that's gone from it can do it 3% of the time to it can do it about 50% of the time

**Dario Amodei:** um so I actually do believe that if we get you can gain benchmarks but I think if we get to 100% on that Benchmark in a way that isn't kind of like overtrained or or or game for that particular Benchmark probably represents a real and serious increase in kind of in kind of programming programming ability and and I would suspect that if we can get to you know 90 90 95% that that that that you know it will it will represent ability to autonomously do a significant fraction of software engineering tasks

## Opus 3.5 and Model Versioning

**Lex Fridman:** well ridiculous timeline question uh when is clad Opus uh 3.5 coming up

**Dario Amodei:** uh not giving you an exact date uh but you know there there uh you know as far as we know the plan is still to have a Claude 3.5 opus

**Lex Fridman:** are we gonna get it before GTA 6 or no like Duke Nukem Forever was that game that there was some game that was delayed 15 years was that Duke Nukem Forever

**Dario Amodei:** yeah and I think GTA is now just releasing trailers it you know it's only been three months since we released the first son it yeah it's Inc the incredible pace of relas it just it just tells you about the pace the expectations for when things are going to come out

**Lex Fridman:** so uh what about 40 so how do you think about sort of as these models get bigger and bigger about versioning and also just versioning in general why Sonet 35 updated with the date why not Sonet 3.6

**Dario Amodei:** actually naming is actually an interesting challenge here right because I think a year ago most of the model was pre-training and so you could start from the beginning and just say okay we're going to have models of different sizes we're going to train them all together and you know we'll have a a family of naming schemes and then we'll put some new magic into them and then you know we'll have the next the next Generation

**Dario Amodei:** Um the trouble starts are already when some of them take a lot longer than others to train right that already messes up your time time a little bit but as you make big improvements in as you make big improvements in pre-training uh then you suddenly notice oh I can make better pre-train model and that doesn't take very long to do and but you know clearly it has the same you know size and shape of previous models

**Dario Amodei:** uh uh so I think those two together as well as the timing timing issues any kind of scheme you come up with uh you know the reality tends to kind of frustrate that scheme right T tends to kind of break out of the break out of the scheme it's not like software where you can say oh this is like you know 3.7 this is 3.8 no you have models with different different tradeoffs you can change some things in your models you can train you can change other things some are faster and slower at inference some have to be more expensive some have to be less expensive

**Dario Amodei:** and so I think all the companies have struggled with this um I think we did very you know I think think we were in a good good position in terms of naming when we had Haiku Sonet and we're trying to maintain it but it's not it's not it's not perfect um so we'll we'll we'll try and get back to the Simplicity but it it um uh just the the the nature of the field I feel like no one's figured out naming it's somehow a different Paradigm from like normal software and and and so we we just none of the companies have been perfect at it um it's something we struggle with surprisingly much relative to you know how relative to how trivial it is to you know for the the the the grand science of training the models

**Lex Fridman:** so from the user side the user experience of the updated Sonet 35 is just different than the previous uh June 2024 Sonet 35 it would be nice to come up with some kind of labeling that embodies that because people talk about son 35 but now there's a different one and so how do you refer to the previous one and the new one and it it uh when there's a distinct Improvement it just makes conversation about it uh just challenging

**Dario Amodei:** yeah yeah I I definitely think this question of there are lots of properties of the models that are not reflected in the benchmarks um I I think I think that's that's definitely the case and everyone agrees and not all of them are capabilities some of them are you know models can be polite or brusk they can be uh you know uh very reactive
# Lex Fridman Podcast #452 — Dario Amodei (Anthropic CEO) · Part 2

**Dario Amodei:** Or they can ask you questions. They can have what feels like a warm personality or a cold personality. They can be boring or they can be very distinctive, like Golden Gate Claude was. And we have a whole team kind of focused on — I think we call it Claude Character. Amanda leads that team and we'll talk to you about that. But it's still a very inexact science.

**Dario Amodei:** And often we find that models have properties that we're not aware of. The fact of the matter is that you can talk to a model 10,000 times and there are some behaviors you might not see, just like with a human. I can know someone for a few months and not know that they have a certain skill or not know there's a certain side to them. And so I think we just have to get used to this idea, and we're always looking for better ways of testing our models to demonstrate these capabilities and also to decide which are the personality properties we want models to have and which we don't want to have. That itself, the normative question, is also super interesting.

**Lex Fridman:** I got to ask you a question from Reddit. There's just this fascinating — to me at least — psychological social phenomenon where people report that Claude has gotten dumber for them over time. And so the question is, does the user complaint about the dumbing down of Claude 3.5 Sonnet hold any water? So are these anecdotal reports a kind of social phenomena, or is there any cases where Claude would get dumber?

**Dario Amodei:** So this actually doesn't just apply to Claude. I believe I've seen these complaints for every foundation model produced by a major company. People said this about GPT-4, they said it about GPT-4 Turbo. So a couple things.

**Dario Amodei:** One, the actual weights of the model — the actual brain of the model — that does not change unless we introduce a new model. There are just a number of reasons why it would not make sense practically to be randomly substituting in new versions of the model. It's difficult from an inference perspective, and it's actually hard to control all the consequences of changing the weights of the model. Let's say you wanted to fine-tune the model to, I don't know, say "certainly" less, which an old version of Sonnet used to do. You actually end up changing a hundred things as well. So we have a whole process for modifying the model. We do a bunch of testing on it, we do a bunch of user testing and early customers. So we both have never changed the weights of the model without telling anyone, and it wouldn't — certainly in the current setup — it would not make sense to do that.

**Dario Amodei:** Now there are a couple things that we do occasionally do. One is sometimes we run A/B tests, but those are typically very close to when a model is being released and for a very small fraction of time. So, you know, the day before the new Sonnet 3.5 — I agree we should have had a better name, it's clunky to refer to it — there were some comments from people that it's gotten a lot better, and that's because a fraction were exposed to an A/B test for those one or two days.

**Dario Amodei:** The other is that occasionally the system prompt will change. The system prompt can have some effects, although it's unlikely to dumb down models, it's unlikely to make them dumber. And we've seen that while these two things, which I'm listing to be very complete, happen quite infrequently, the complaints — for us and for other model companies — about the model changed, the model isn't good at this, the model got more censored, the model was dumbed down, those complaints are constant. And so I don't want to say people are imagining it or anything, but the models are for the most part not changing.

**Dario Amodei:** If I were to offer a theory, I think it actually relates to one of the things I said before, which is that models are very complex and have many aspects to them. And so often, if I ask a model a question — if I say "do task X" versus "can you do task X" — the model might respond in different ways. And so there are all kinds of subtle things that you can change about the way you interact with the model that can give you very different results. To be clear, this itself is a failing by us and by the other model providers, that the models are just often sensitive to small changes in wording. It's yet another way in which the science of how these models work is very poorly developed.

**Dario Amodei:** And so if I go to sleep one night and I was talking to the model in a certain way and I slightly change the phrasing of how I talk to the model, I could get different results. So that's one possible way. The other thing is, man, it's just hard to quantify this stuff. I think people are very excited by new models when they come out, and then as time goes on they become very aware of the limitations. So that may be another effect. But that's all a very long-winded way of saying, for the most part, with some fairly narrow exceptions, the models are not changing.

**Lex Fridman:** I think there is a psychological effect. You just start getting used to it. The baseline — like when people have first gotten Wi-Fi on airplanes, it's like amazing magic, and then now it's like, "I can't get this thing to work, this is such a piece of crap."

**Dario Amodei:** Exactly. So it's easy to have the conspiracy theory of they're making Wi-Fi slower and slower.

**Lex Fridman:** This is probably something I'll talk to Amanda much more about, but another Reddit question: when will Claude stop trying to be my panicky grandmother, imposing its moral worldview on me as a paying customer? And also, what is the ideology behind making Claude overly apologetic? So this kind of reports about the experience — a different angle on the frustration — it has to do with the character.

**Dario Amodei:** Yeah, so a couple points on this. First one is, things that people say on Reddit and Twitter or X or whatever — there's actually a huge distribution shift between the stuff that people complain loudly about on social media and what actually, statistically, users care about and that drives people to use the models. People are frustrated with things like the model not writing out all the code, or the model just not being as good at code as it could be, even though it's the best model in the world on code. I think the majority of things are about that.

**Dario Amodei:** But certainly a vocal minority are raising these concerns, are frustrated by the model refusing things that it shouldn't refuse, or apologizing too much, or just having these kind of annoying verbal tics.

**Dario Amodei:** The second caveat, and I just want to say this super clearly because I think some people don't know it, others kind of know it but forget it — it is very difficult to control across the board how the models behave. You cannot just reach in there and say, "Oh, I want the model to apologize less." You can do that, you can include training data that says the model should apologize less, but then in some other situation they end up being super rude or overconfident in a way that's misleading people. So there are all these tradeoffs.

**Dario Amodei:** For example, another thing is there was a period during which models — ours and I think others as well — were too verbose. They would repeat themselves, they would say too much. You can cut down on the verbosity by penalizing the models for just talking for too long. What happens when you do that, if you do it in a crude way, is when the models are coding, sometimes they'll say "the rest of the code goes here," because they've learned that's a way to economize. And then that leads the model to be so-called lazy in coding, where they're just like, "Ah, you can finish the rest of it." It's not because we want to save on compute, or because the models are lazy during winter break, or any of the other conspiracy theories that have come up. It's actually just very hard to control the behavior of the model, to steer the behavior of the model in all circumstances at once.

**Dario Amodei:** There's this whack-a-mole aspect where you push on one thing and these other things start to move as well that you may not even notice or measure. And so one of the reasons that I care so much about grand alignment of these AI systems in the future is actually these systems are quite unpredictable, they're quite hard to steer and control. And this version we're seeing today — of you make one thing better, it makes another thing worse — I think that's a present-day analog of future control problems in AI systems that we can start to study today.

**Dario Amodei:** I think that difficulty in steering the behavior and in making sure that if we push an AI system in one direction it doesn't push it in another direction in some other ways that we didn't want — I think that's an early sign of things to come. And if we can do a good job of solving this problem — like, you ask the model to make and distribute smallpox and it says no, but it's willing to help you in your graduate-level virology class — how do we get both of those things at once? It's hard. It's very easy to go to one side or the other, and it's a multi-dimensional problem.

**Dario Amodei:** And so I think these questions of shaping the model's personality, I think they're very hard. I think we haven't done perfectly on them. I think we've actually done the best of all the AI companies, but still so far from perfect. And I think if we can get this right, if we can control the false positives and false negatives in this very controlled present-day environment, we'll be much better at doing it for the future, when our worry is will the models be super autonomous, will they be able to make very dangerous things, will they be able to autonomously build whole companies, and are those companies aligned. So I think of this present task as both vaccine but also good practice for the future.

**Lex Fridman:** What's the current best way of gathering user feedback — not anecdotal data but just large-scale data about pain points or the opposite of pain points, positive things? Is it internal testing, a specific group testing? What works?

**Dario Amodei:** So typically we'll have internal model bashings where all of Anthropic — Anthropic is almost a thousand people — people just try and break the model, they try and interact with it in various ways. We have a suite of evals for, "Oh, is the model refusing in ways that it shouldn't?" I think we even had a "certainly" eval, because at one point our model had this annoying tic where it would respond to a wide range of questions by saying "Certainly, I can help you with that," "Certainly, I would be happy to do that," "Certainly, this is correct." And so we had a "certainly" eval — how often does the model say "certainly."

**Dario Amodei:** But look, this is just whack-a-mole. Like, what if it switches from "certainly" to "definitely"? So every time we add a new eval and we're always evaluating for all the old things, we have hundreds of these evaluations. But we find that there's no substitute for humans interacting with it. And so it's very much like the ordinary product development process. We have hundreds of people within Anthropic bash the model, then we do external A/B tests, sometimes we'll run tests with contractors, we pay contractors to interact with the model.

**Dario Amodei:** So you put all of these things together and it's still not perfect. You still see behaviors that you don't quite want to see. You still see the model refusing things that it just doesn't make sense to refuse. But I think trying to solve this challenge — trying to stop the model from doing genuinely bad things that everyone agrees it shouldn't do, like everyone agrees that the model shouldn't talk about child abuse material — but at the same time that it doesn't refuse in these dumb and stupid ways. I think drawing that line as finely as possible, approaching perfectly, is still a challenge, and we're getting better at it every day. But there's a lot to be solved. And again I would point to that as an indicator of a challenge ahead in terms of steering much more powerful models.

**Lex Fridman:** Do you think Claude 4.0 is ever coming out?

**Dario Amodei:** I don't want to commit to any naming scheme, because if I say here we're going to have Claude 4 next year and then we decide that we should start over because there's a new type of model — I don't want to commit to it. I would expect in a normal course of business that Claude 4 would come after Claude 3.5, but you never know in this wacky field.

**Lex Fridman:** But the sort of this idea of scaling is continuing?

**Dario Amodei:** Scaling is continuing. There will definitely be more powerful models coming from us than the models that exist today. That is certain. Or if there aren't, we've deeply failed as a company.

**Lex Fridman:** Okay, can you explain the responsible scaling policy and the AI safety level standards — the ASL levels?

**Dario Amodei:** As much as I'm excited about the benefits of these models — and we'll talk about that if we talk about Machines of Loving Grace — I'm worried about the risk and I continue to be worried about the risks. No one should think that Machines of Loving Grace was me saying I'm no longer worried about the risks of these models. I think they're two sides of the same coin. The power of the models and their ability to solve all these problems in biology, neuroscience, economic development, government, governance, and peace — large parts of the economy — those come with risks as well. With great power comes great responsibility. The two are paired. Things that are powerful can do good things and they can do bad things.

**Dario Amodei:** I think of those risks as being in several different categories. Perhaps the two biggest risks that I think about — and that's not to say that there aren't risks today that are important — but when I think of the things that would happen on the grandest scale: one is what I call catastrophic misuse. These are misuse of the models in domains like cyber, bio, radiological, nuclear — things that could harm or even kill thousands, even millions of people if they really, really go wrong. These are the number one priority to prevent.

**Dario Amodei:** And here I would just make a simple observation, which is that if I look today at people who have done really bad things in the world, I think actually humanity has been protected by the fact that the overlap between really smart, well-educated people and people who want to do really horrific things has generally been small. Let's say I'm someone who has a PhD in this field, I have a well-paying job — there's so much to lose. Why would I want to — even assuming I'm completely evil, which most people are not — why would such a person risk their life, risk their legacy, their reputation to do something truly evil? If we had a lot more people like that, the world would be a much more dangerous place.

**Dario Amodei:** And so my worry is that by being a much more intelligent agent, AI could break that correlation. And so I do have serious worries about that. I believe we can prevent those worries. But as a counterpoint to Machines of Loving Grace, I want to say that there are still serious risks.

**Dario Amodei:** And the second range of risks would be the autonomy risks, which is the idea that models might on their own — particularly as we give them more agency than they've had in the past, particularly as we give them supervision over wider tasks like writing whole code bases or someday even effectively operating entire companies — they're on a long enough leash. Are they doing what we really want them to do? It's very difficult to even understand in detail what they're doing, let alone control it.

**Dario Amodei:** And like I said, these early signs that it's hard to perfectly draw the boundary between things the model should do and things the model shouldn't do — if you go to one side you get things that are annoying and useless, and you go to the other side you get other behaviors. If you fix one thing, it creates other problems. We're getting better and better at solving this. I don't think this is an unsolvable problem. I think this is a science, like the safety of airplanes or the safety of cars or the safety of drugs. I don't think there's any big thing we're missing; I just think we need to get better at controlling these models.

**Dario Amodei:** And so these are the two risks I'm worried about, and our responsible scaling plan — which I'll recognize is a very long-winded answer to your question —

**Lex Fridman:** I love it.

**Dario Amodei:** Our responsible scaling plan is designed to address these two types of risks. And so every time we develop a new model, we basically test it for its ability to do both of these bad things.

**Dario Amodei:** So if I were to back up a little bit, I think we have an interesting dilemma with AI systems where they're not yet powerful enough to present these catastrophes — I don't know that they'll ever present these catastrophes, it's possible they won't — but the case for worry, the case for risk, is strong enough that we should act now. And they're getting better very, very fast. I testified in the Senate that we might have serious bio risks within two to three years. That was about a year ago, and things have proceeded apace.

**Dario Amodei:** So we have this thing where it's surprisingly hard to address these risks because they're not here today, they don't exist. They're like ghosts, but they're coming at us so fast because the models are improving so fast. So how do you deal with something that's not here today, doesn't exist, but is coming at us very fast?

**Dario Amodei:** The solution we came up with, in collaboration with people like the organization METR and Paul Christiano, is: what you need for that is tests to tell you when the risk is getting close. You need an early warning system. And so every time we have a new model, we test it for its capability to do these CBRN tasks, as well as testing it for how capable it is of doing tasks autonomously on its own. And in the latest version of our RSP, which we released in the last month or two, the way we test autonomy risks is the AI model's ability to do aspects of AI research itself — which, when the AI models can do AI research, they become kind of truly autonomous. And that threshold is important for a bunch of other ways.

**Dario Amodei:** And so what do we then do with these tests? The RSP basically develops what we've called an "if-then" structure, which is: if the models pass a certain capability, then we impose a certain set of safety and security requirements on them.

**Dario Amodei:** So today's models are what's called ASL-2 models. ASL-1 is for systems that manifestly don't pose any risk of autonomy or misuse. For example, a chess-playing bot — Deep Blue — would be ASL-1. It's just manifestly the case that you can't use Deep Blue for anything other than chess. It was just designed for chess. No one's going to use it to conduct a masterful cyber attack or to run wild and take over the world.

**Dario Amodei:** ASL-2 is today's AI systems, where we've measured them and we think these systems are simply not smart enough to autonomously self-replicate or conduct a bunch of tasks, and also not smart enough to provide meaningful information about CBRN risks and how to build CBRN weapons above and beyond what can be known from looking at Google. In fact, sometimes they do provide information, but not above and beyond a search engine — not in a way that can be stitched together, not in a way that end-to-end is dangerous enough.

**Dario Amodei:** ASL-3 is going to be the point at which the models are helpful enough to enhance the capabilities of non-state actors. State actors can already do a lot of these very dangerous and destructive things, unfortunately, to a high level of proficiency. The difference is that non-state actors are not capable of it. And so when we get to ASL-3, we'll take special security precautions designed to be sufficient to prevent theft of the model by non-state actors and misuse of the model as it's deployed. We'll have to have enhanced filters targeted at these particular areas — cyber, bio, nuclear — and model autonomy, which is less a misuse risk and more a risk of the model doing bad things itself.

**Dario Amodei:** ASL-4 — getting to the point where these models could enhance the capability of an all-knowledgeable state actor and/or become the main source of such a risk, like if you wanted to engage in such a risk, the main way you would do it is through a model. And then I think ASL-4 on the autonomy side, it's some amount of acceleration in AI research capabilities with an AI model.

**Dario Amodei:** And then ASL-5 is where we would get to the models that are truly capable, that could exceed humanity in their ability to do any of these tasks.

**Dario Amodei:** And so the point of the if-then structure commitment is basically to say: look, I've been working with these models for many years and I've been worried about risk for many years. It's actually kind of dangerous to cry wolf. It's actually kind of dangerous to say this model is risky, and people look at it and say this is manifestly not dangerous. Again, the delicacy is the risk isn't here today, but it's coming at us fast. How do you deal with that? It's really vexing to a risk planner.

**Dario Amodei:** And so this if-then structure basically says: look, we don't want to antagonize a bunch of people, we don't want to harm our own ability to have a place in the conversation by imposing these very onerous burdens on models that are not dangerous today. So the trigger commitment is basically a way to deal with this — you clamp down hard when you can show that the model is dangerous. And of course, what has to come with that is enough of a buffer threshold that you're not at high risk of missing the danger.

**Dario Amodei:** It's not a perfect framework. We've had to change it — we came out with a new one just a few weeks ago, and probably going forward we might release new ones multiple times a year because it's hard to get these policies right technically, organizationally, from a research perspective. But that is the proposal: if-then commitments and triggers in order to minimize burdens and false alarms now, but really react appropriately when the dangers are here.

**Lex Fridman:** What do you think the timeline for ASL-3 is, where several of the triggers are fired? And what do you think the timeline is for ASL-4?

**Dario Amodei:** Yeah, so that is hotly debated within the company. We are working actively to prepare ASL-3 security measures as well as ASL-3 deployment measures. I'm not going to go into detail, but we've made a lot of progress on both, and we're prepared to be ready quite soon. I would not be surprised at all if we hit ASL-3 next year. There was some concern that we might even hit it this year — that's still possible, that could still happen. It's very hard to say, but I would be very, very surprised if it was like 2030. I think it's much sooner than that.

**Lex Fridman:** So there's protocols for detecting it — the if-then — and then there's protocols for how to respond to it. How difficult is the second, the latter?

**Dario Amodei:** Yeah, I think for ASL-3, it's primarily about security and about filters on the model relating to a very narrow set of areas when we deploy the model, because at ASL-3 the model isn't autonomous yet. And so you don't have to worry about the model itself behaving in a bad way even when it's deployed internally. So I think the ASL-3 measures are — I won't say straightforward — they're rigorous, but they're easier to reason about.

**Dario Amodei:** I think once we get to ASL-4, we start to have worries about the models being smart enough that they might sandbag tests, they might not tell the truth about tests. We had some results come out about sleeper agents, and there was a more recent paper about whether the models can mislead attempts to sandbag their own abilities — present themselves as being less capable than they are. And so I think with ASL-4, there's going to be an important component of using other things than just interacting with the models — for example, interpretability or hidden chains of thought — where you have to look inside the model and verify via some other mechanism that is not as easily corrupted as what the model says, that the model indeed has some property. So we're still working on ASL-4.

**Dario Amodei:** One of the properties of the RSP is that we don't specify ASL-4 until we've hit ASL-3, and I think that's proven to be a wise decision because even with ASL-3, it's hard to know this stuff in detail, and we want to take as much time as we can possibly take to get these things right.

**Lex Fridman:** So for ASL-3, the bad actor will be the humans?

**Dario Amodei:** Yes.

**Lex Fridman:** And so there it's a little bit more — for ASL-4 it's both?

**Dario Amodei:** I think it's both. And so deception — and that's where mechanistic interpretability comes into play. And hopefully the techniques used for that are not made accessible to the model.

**Dario Amodei:** Yeah, I mean of course you can hook up the mechanistic interpretability to the model itself, but then you've kind of lost it as a reliable indicator of the model state. There are a bunch of exotic ways you can think of that it might also not be reliable — like if the model gets smart enough that it can jump computers and read the code where you're looking at its internal state. We've thought about some of those. I think they're exotic enough; there are ways to render them unlikely. But yeah, generally you want to preserve mechanistic interpretability as a kind of verification set or test set that's separate from the training process of the model.

**Lex Fridman:** I think as these models become better and better at conversation and become smarter, social engineering becomes a threat too, because they can start being very convincing to the engineers inside companies.

**Dario Amodei:** Oh yeah, yeah. It's actually — we've seen lots of examples of demagoguery in our life from humans, and there's a concern that models could do that as well.

**Lex Fridman:** One of the ways that Claude has been getting more and more powerful is it's now able to do some agentic stuff — computer use. There's also an analysis within the sandbox of claude.ai itself. But let's talk about computer use. That seems to me super exciting — that you can just give Claude a task and it takes a bunch of actions, figures it out, and has access to your computer through screenshots. Can you explain how that works and where that's headed?

**Dario Amodei:** Yeah, it's actually relatively simple. Claude has had for a long time — since Claude 3, back in March — the ability to analyze images and respond to them with text. The only new thing we added is those images can be screenshots of a computer, and in response we train the model to give a location on the screen where you can click and/or buttons on the keyboard you can press in order to take action. And it turns out that with actually not all that much additional training, the models can get quite good at that task. It's a good example of generalization.

**Dario Amodei:** People sometimes say if you get to low Earth orbit, you're like halfway to anywhere, because of how much it takes to escape the gravity well. If you have a strong pre-trained model, I feel like you're halfway to anywhere in terms of the intelligence space. And so actually it didn't take all that much to get Claude to do this.

**Dario Amodei:** And you can just set that in a loop — give the model a screenshot, tell it what to click on, give it the next screenshot, tell it what to click on — and that turns into a full, almost 3D video interaction of the model. And it's able to do all of these tasks. We showed these demos where it's able to fill out spreadsheets, interact with a website, open all kinds of programs on different operating systems — Windows, Linux, Mac. So I think all of that is very exciting.

**Dario Amodei:** I will say, while in theory there's nothing you could do there that you couldn't have done through just giving the model the API to drive the computer screen, this really lowers the barrier. There are a lot of folks who either aren't in a position to interact with those APIs or it takes them a long time to do. The screen is just a universal interface that's a lot easier to interact with. And so I expect over time this is going to lower a bunch of barriers.

**Dario Amodei:** Now honestly, the current model — it leaves a lot still to be desired, and we were honest about that in the blog. It makes mistakes, it misclicks, and we were careful to warn people: this thing isn't — you can't just leave this thing to run on your computer for minutes and minutes. You got to give this thing boundaries and guardrails. And I think that's one of the reasons we released it first in an API form rather than just handing it to the consumer and giving it control of their computer.

**Dario Amodei:** But I definitely feel that it's important to get these capabilities out there. As models get more powerful, we're going to have to grapple with how do we use these capabilities safely, how do we prevent them from being abused. And I think releasing the model while the capabilities are still limited is very helpful in terms of doing that.

**Dario Amodei:** I think since it's been released, a number of customers — I think Replit was maybe one of the quickest to deploy things — have made use of it in various ways. People have hooked up demos for Windows desktops, Macs, Linux machines. So yeah, it's been very exciting. I think as with anything else, it comes with new exciting abilities and then with those new exciting abilities, we have to think about how to make the model safe, reliable, do what humans want them to do. It's the same story for everything, that same tension.

**Lex Fridman:** But the possibility of use cases here is just — the range is incredible. So how much to make it work really well in the future, how much do you have to specially go beyond what the pre-trained model is doing — do more post-training, RLHF or supervised fine-tuning or synthetic data, just for the agentic stuff?

**Dario Amodei:** Yeah, I think speaking at a high level, it's our intention to keep investing a lot in making the model better. Like, we look at some of the benchmarks where previous models could do it 6% of the time and now our models do it at 14 or 22% of the time. And yeah, we want to get up to the human-level reliability of 80, 90%, just like anywhere else. We're on the same curve that we were on with SWE-bench, where I think I would guess a year from now the models can do this very, very reliably. But you got to start somewhere.

**Lex Fridman:** So you think it's possible to get to the human-level 90%, basically doing the same thing you're doing now, or does it have to be special for computer use?

**Dario Amodei:** I mean, depends what you mean by "special," but I generally think the same kinds of techniques that we've been using to train the current model — I expect that doubling down on those techniques, in the same way that we have for code, for models in general, for other kinds of input, for image input, for voice — I expect those same techniques will scale here as they have everywhere else.

**Lex Fridman:** But this is giving sort of the power of action to Claude. And so you could do a lot of really powerful things, but you could do a lot of damage also.

**Dario Amodei:** Yeah, yeah, no, and we've been very aware of that. Look, my view actually is computer use isn't a fundamentally new capability like the CBRN or autonomy capabilities are. It's more like it kind of opens the aperture for the model to use and apply its existing abilities.

**Dario Amodei:** And so the way we think about it, going back to our RSP, is nothing that this model is doing inherently increases the risk from an RSP perspective. But as the models get more powerful, having this capability may make it scarier once it has the cognitive capability to do something at the ASL-3 and ASL-4 level. This may be the thing that kind of unbounds it from doing so. So going forward, certainly this modality of interaction is something we have tested for and that we will continue to test for.

**Dario Amodei:** I think it's probably better to learn and explore this capability before the model is super capable.

**Lex Fridman:** Yeah. And there's a lot of interesting attacks, like prompt injection, because now you've widened the aperture. So you can prompt-inject through stuff on screen. If this becomes more and more useful, then there's more and more benefit to inject stuff into the model. If it goes to a certain webpage, it could be harmless stuff like advertisements, or it could be harmful stuff.

**Dario Amodei:** Right, yeah. I mean, we thought a lot about spam, capture — there's all — every — if one secret I'll tell you: if you've invented a new technology, not necessarily the biggest misuse, but the first misuse you'll see is scams. Just petty scams. It's like a thing as old as — people scamming each other is a thing as old as time. And every time, you got to deal with it. It's almost silly to say, but it's true.

**Lex Fridman:** Sort of. And spam in general is a thing. As it gets more and more intelligent, there are a lot of — like I said — petty criminals in the world. And every new technology is a new way for petty criminals to do something stupid and malicious.

**Lex Fridman:** Is there any ideas about sandboxing it — how difficult is the sandboxing task?

**Dario Amodei:** Yeah, we sandbox during training. So for example, during training we didn't expose the model to the internet. I think that's probably a bad idea during training, because the model can be changing its policy, it can be changing what it's doing, and it's having an effect in the real world.

**Dario Amodei:** In terms of actually deploying the model, it kind of depends on the application. Sometimes you want the model to do something in the real world. But of course you can always put guardrails on the outside. You can say, "Okay, this model is not going to move any files from my computer or my web server to anywhere else."

**Dario Amodei:** Now, when you talk about sandboxing — again, when we get to ASL-4, none of these precautions are going to make sense there. When you talk about ASL-4, there's a theoretical worry the model could be smart enough to break out of any box. And so there we need to think about mechanistic interpretability — if we're going to have a sandbox, it would need to be a mathematically provable, sound sandbox. But that's a whole different world than what we're dealing with the models today.

**Lex Fridman:** Yeah, the science of building a box from which an ASL-4 AI system cannot escape —

**Dario Amodei:** I think it's probably not the right approach. I think the right approach, instead of having something unaligned that you're trying to prevent from escaping — I think it's better to just design the model the right way, or have a loop where you look inside the model and you're able to verify properties. And that gives you an opportunity to iterate and actually get it right. I think containing bad models is a much worse solution than having good models.

**Lex Fridman:** Let me ask about regulation. What's the role of regulation in keeping AI safe? So for example, you described California AI regulation bill SB 1047, that was ultimately vetoed by the governor. What are the pros and cons of this bill?

**Dario Amodei:** Yeah, so we ended up making some suggestions to the bill and then some of those were adopted, and I think we felt quite positively about the bill by the end of that. It did still have some downsides, and of course it got vetoed.

**Dario Amodei:** I think at a high level, some of the key ideas behind the bill are, I would say, similar to ideas behind our RSPs. And I think it's very important that some jurisdiction — whether it's California or the federal government and/or other countries and other states — passes some regulation like this. And I can talk through why I think that's so important.

**Dario Amodei:** So I feel good about our RSP. It's not perfect, it needs to be iterated on a lot, but it's been a good forcing function for getting the company to take these risks seriously, to put them into product planning, to really make them a central part of work at Anthropic, and to make sure that all the almost thousand people at Anthropic understand that this is one of the highest priorities of the company, if not the highest priority.

**Dario Amodei:** But one, there are some companies that don't have RSP-like mechanisms. OpenAI, Google did adopt these mechanisms a couple months after Anthropic did. But there are other companies out there that don't have these mechanisms at all. And so if some companies adopt these mechanisms and others don't, it's really going to create a situation where some of these dangers have the property that it doesn't matter if three out of five of the companies are being safe if the other two are being unsafe. It creates this negative externality.

**Dario Amodei:** And I think the lack of uniformity is not fair to those of us who have —
# Lex Fridman Podcast #452 — Dario Amodei (Anthropic CEO) · Part 3

**Dario Amodei:** We put a lot of effort into being very thoughtful about these procedures. The second thing is I don't think you can trust these companies to adhere to these voluntary plans in their own right. I like to think that Anthropic will — we do everything we can, that we will — our RSP is checked by our long-term benefit trust, so we do everything we can to adhere to our own RSP. But you hear lots of things about various companies saying, "Oh, they said they would do this, they said they would give this much compute and they didn't, they said they would do this thing and they didn't." I don't think it makes sense to litigate particular things that companies have done, but I think this broad principle that if there's nothing watching over them — there's nothing watching over us as an industry — there's no guarantee that we'll do the right thing, and the stakes are very high. And so I think it's important to have a uniform standard that everyone follows and to make sure that simply the industry does what a majority of the industry has already said is important and has already said that they definitely will do.

**Dario Amodei:** Right, some people — I think there's a class of people who are against regulation on principle. I understand where that comes from. If you go to Europe and you see something like GDPR, you see some of the other stuff that they've done — some of it's good, but some of it is really unnecessarily burdensome, and I think it's fair to say it really has slowed innovation. And so I understand where people are coming from on priors, I understand why people start from that position. But again, I think AI is different. If we go to the very serious risks of autonomy and misuse that I talked about just a few minutes ago, I think that those are unusual and they warrant an unusually strong response. And so I think it's very important. Again, we need something that everyone can get behind.

**Dario Amodei:** I think one of the issues with SB 1047, especially the original version of it, was it had a bunch of the structure of RSPs but it also had a bunch of stuff that was either clunky or that just would have created a bunch of burdens, a bunch of hassle, and might even have missed the target in terms of addressing the risks. You don't really hear about it on Twitter — you just hear about people cheering for any regulation and then the folks who are against make up these often quite intellectually dishonest arguments about how, you know, "It'll make us move away from California." The bill doesn't apply if you're headquartered in California — the bill only applies if you do business in California. Or that it would damage the open-source ecosystem, or that it would cause all of these things. I think those were mostly nonsense. But there are better arguments against regulation. There's one guy, Dean Ball, who's really a very scholarly person who looks at what happens when a regulation is put in place and ways that they can kind of get a life of their own or how they can be poorly designed.

**Dario Amodei:** And so our interest has always been: we do think there should be regulation in this space, but we want to be an actor who makes sure that regulation is something that's surgical, that's targeted at the serious risks, and is something people can actually comply with. Because something I think the advocates of regulation don't understand as well as they could is: if we get something in place that is poorly targeted, that wastes a bunch of people's time, what's going to happen is people are going to say, "See, these safety risks — this is nonsense. I just had to hire 10 lawyers to fill out all these forms. I had to run all these tests for something that was clearly not dangerous." And after six months of that, there will be a ground swell and we'll end up with a durable consensus against regulation. And so the worst enemy of those who want real accountability is badly designed regulation. We need to actually get it right.

**Dario Amodei:** And if there's one thing I could say to the advocates, it would be that I want them to understand this dynamic better. We need to be really careful and we need to talk to people who actually have experience seeing how regulations play out in practice. And the people who have seen that understand to be very careful. If this was some lesser issue, I might be against regulation at all. But what I want the opponents to understand is that the underlying issues are actually serious. They're not something that I or the other companies are just making up because of regulatory capture. They're not sci-fi fantasies. They're not any of these things. Every time we have a new model, every few months, we measure the behavior of these models and they're getting better and better at these concerning tasks, just as they are getting better and better at good, valuable, economically useful tasks.

**Dario Amodei:** And so I would just love it if — I think SB 1047 was very polarizing — I would love it if some of the most reasonable opponents and some of the most reasonable proponents would sit down together. I think the different AI companies — Anthropic was the only AI company that felt positively in a very detailed way. I think Elon tweeted briefly something positive, but some of the big ones like Google, OpenAI, Meta, Microsoft were pretty stoutly against. So what I would really like is if some of the key stakeholders, some of the thoughtful proponents and some of the most thoughtful opponents, would sit down and say, "How do we solve this problem in a way that the proponents feel brings a real reduction in risk and that the opponents feel is not hampering the industry or hampering innovation any more than it needs to?" And I think for whatever reason things got too polarized and those two groups didn't get to sit down in the way that they should. And I feel urgency — I really think we need to do something in 2025. If we get to the end of 2025 and we've still done nothing about this, then I'm going to be worried. I'm not worried yet because again the risks aren't here yet, but I think time is running short.

**Lex Fridman:** Yeah, and come up with something surgical, like you said.

**Dario Amodei:** Yeah, exactly. And we need to get away from this intense pro-safety versus intense anti-regulatory rhetoric. It's turned into these flame wars on Twitter and nothing good's going to come of that.

**Lex Fridman:** So there's a lot of curiosity about the different players in the game. One of the OGs is OpenAI. You have had several years of experience at OpenAI. What's your story and history there?

**Dario Amodei:** Yeah, so I was at OpenAI for roughly five years. For the last couple years I was vice president of research there. Probably myself and Ilya Sutskever were the ones who really kind of set the research direction. Around 2016 or 2017 I first started to really believe in, or at least confirm my belief in, the scaling hypothesis, when Ilya famously said to me, "The thing you need to understand about these models is they just want to learn. The models just want to learn." And again, sometimes there are these one sentences, these Zen koans, that you hear them and you're like, "Ah, that explains everything — that explains like a thousand things that I've seen." And then ever after, I had this visualization in my head of like, you optimize the models in the right way, you point the models in the right way, they just want to learn, they just want to solve the problem, regardless of what the problem is. So get out of their way basically.

**Lex Fridman:** Get out of their way.

**Dario Amodei:** Yeah, don't impose your own ideas about how they should learn. And this was the same thing as Rich Sutton put out in "The Bitter Lesson" or Gwern put out in the scaling hypothesis. I think generally the dynamic was I got this kind of inspiration from Ilya, from others, folks like Alec Radford who did the original GPT-1, and then ran really hard with it — me and my collaborators on GPT-2, GPT-3, RL from human feedback, which was an attempt to kind of deal with the early safety and durability things like debate and amplification, heavy on interpretability. So again, the combination of safety plus scaling. Probably 2018, 2019, 2020 — those were kind of the years when myself and my collaborators, many of whom became co-founders of Anthropic, kind of really had a vision and drove the direction.

**Lex Fridman:** Why'd you leave? Why'd you decide to leave?

**Dario Amodei:** Yeah, so look, I'm going to put things this way, and I think it ties to the race to the top. In my time at OpenAI, what I'd come to see — as I'd come to appreciate the scaling hypothesis and as I'd come to appreciate the importance of safety along with the scaling hypothesis — the first one I think OpenAI was getting on board with. The second one, in a way, had always been part of OpenAI's messaging. But over many years of the time that I spent there, I think I had a particular vision of how we should handle these things, how they should be brought out in the world, the kind of principles that the organization should have.

**Dario Amodei:** And look, there were many discussions about, "Should the company do this? Should the company do that?" There's a bunch of misinformation out there. People say we left because we didn't like the deal with Microsoft — false, although there was a lot of discussion, a lot of questions about exactly how we do the deal with Microsoft. We left because we didn't like commercialization — that's not true. We built GPT-3, which was the model that was commercialized. I was involved in commercialization. It's more again about how do you do it. Like, civilization is going down this path to very powerful AI. What's the way to do it that is cautious, straightforward, honest, that builds trust in the organization and in individuals? How do we get from here to there? And how do we have a real vision for how to get it right? How can safety not just be something we say because it helps with recruiting?

**Dario Amodei:** And I think at the end of the day, if you have a vision for that — forget about anyone else's vision, I don't want to talk about anyone else's vision — if you have a vision for how to do it, you should go off and you should do that vision. It is incredibly unproductive to try and argue with someone else's vision. You might think they're not doing it the right way, you might think they're dishonest — who knows, maybe you're right, maybe you're not. But what you should do is take some people you trust, go off together, and make your vision happen. And if your vision is compelling, if you can make it appeal to people — some combination of ethically, in the market — if you can make a company that's a place people want to join, that engages in practices that people think are reasonable, while managing to maintain its position in the ecosystem at the same time — if you do that, people will copy it. And the fact that you were doing it, especially the fact that you're doing it better than they are, causes them to change their behavior in a much more compelling way than if they're your boss and you're arguing with them.

**Dario Amodei:** I just don't know how to be any more specific about it than that, but I think it's generally very unproductive to try and get someone else's vision to look like your vision. It's much more productive to go off and do a clean experiment and say, "This is our vision, this is how we're going to do things. Your choice is you can ignore us, you can reject what we're doing, or you can start to become more like us." And imitation is the sincerest form of flattery. And that plays out in the behavior of customers, that plays out in the behavior of the public, that plays out in the behavior of where people choose to work.

**Dario Amodei:** And again, at the end, it's not about one company winning or another company winning. If we or another company are engaging in some practice that people find genuinely appealing — and I want it to be in substance, not just in appearance — and I think researchers are sophisticated and they look at substance — and then other companies start copying that practice and they win because they copied that practice, that's great. That's success. That's like the race to the top. It doesn't matter who wins in the end as long as everyone is copying everyone else's good practices.

**Dario Amodei:** One way I think of it is like: the thing we're all afraid of is a race to the bottom. In the race to the bottom, it doesn't matter who wins because we all lose. In the most extreme world, we make this autonomous AI that, you know, the robots enslave us or whatever — I mean, that's half joking, but that is the most extreme thing that could happen. Then it doesn't matter which company was ahead. If instead you create a race to the top where people are competing to engage in good practices, then at the end of the day it doesn't matter who ends up winning. It doesn't even matter who started the race to the top. The point isn't to be virtuous — the point is to get the system into a better equilibrium than it was before. And individual companies can play some role in doing this, individual companies can help to start it, can help to accelerate it. And frankly, I think individuals at other companies have done this as well — the individuals that, when we put out an RSP, react by pushing harder to get something similar done at other companies. Sometimes other companies do something that's like, we're like, "Oh, it's a good practice, we think that's good, we should adopt it too."

**Dario Amodei:** The only difference is I think we try to be more forward-leaning — we try and adopt more of these practices first and adopt them more quickly when others invent them. But I think this dynamic is what we should be pointing at. And I think it abstracts away the question of which company's winning, who trusts who. I think all these questions of drama are profoundly uninteresting, and the thing that matters is the ecosystem that we all operate in and how to make that ecosystem better, because that constrains all the players.

**Lex Fridman:** And so Anthropic is this kind of clean experiment built on a foundation of what concretely AI safety should look like.

**Dario Amodei:** Look, I'm sure we've made plenty of mistakes along the way. The perfect organization doesn't exist. It has to deal with the imperfection of a thousand employees. It has to deal with the imperfection of our leaders, including me. It has to deal with the imperfection of the people we've put to oversee the imperfection of the leaders, like the board and the long-term benefit trust. It's all a set of imperfect people trying to aim imperfectly at some ideal that will never perfectly be achieved. That's what you sign up for, that's what it will always be. But imperfect doesn't mean you just give up. There's better and there's worse, and hopefully we can do well enough that we can begin to build some practices that the whole industry engages in. And then, my guess is that multiple of these companies will be successful. Anthropic will be successful, these other companies — like ones I've been at in the past — will also be successful. And some will be more successful than others. That's less important than, again, that we align the incentives of the industry. And that happens partly through the race to the top, partly through things like RSPs, partly through, again, selected surgical regulation.

**Lex Fridman:** You said talent density beats talent mass. Can you explain that? Can you expand on it? Can you just talk about what it takes to build a great team of AI researchers and engineers?

**Dario Amodei:** This is one of these statements that's like more true every month. Every month I see this statement as more true than I did the month before. So if I were to do a thought experiment: let's say you have a team of 100 people that are super smart, motivated, and aligned with the mission, and that's your company. Or you can have a team of a thousand people where 200 people are super smart, super aligned with the mission, and then 800 people are — let's just say you pick 800 random big tech employees. Which would you rather have?

**Dario Amodei:** The talent mass is greater in the group of a thousand people — you have an even larger number of incredibly talented, incredibly aligned, incredibly smart people. But the issue is just that if every time someone super talented looks around, they see someone else super talented and super dedicated, that sets the tone for everything. That sets the tone for everyone is super inspired to work at the same place, everyone trusts everyone else. If you have a thousand or 10,000 people and things have really regressed — you are not able to do selection and you're choosing random people — what happens is then you need to put a lot of process and a lot of guardrails in place, just because people don't fully trust each other. You have to adjudicate political battles. There are so many things that slow down the org's ability to operate.

**Dario Amodei:** And so we're nearly a thousand people, and we've tried to make it so that as large a fraction of those thousand people as possible are super talented, super skilled. It's one of the reasons we've slowed down hiring a lot in the last few months. We grew from 300 to 800, I believe, in the first seven or eight months of the year, and now we've slowed down — the last three months we went from 800 to 900, 950, something like that. Don't quote me on the exact numbers. But I think there's an inflection point around a thousand, and we want to be much more careful how we grow.

**Dario Amodei:** Early on and now as well, we've hired a lot of physicists. Theoretical physicists can learn things really fast. Even more recently, as we've continued to hire, we've really had a high bar on both the research side and the software engineering side, have hired a lot of senior people including folks who used to be at other companies in this space. And we've just continued to be very selective. It's very easy to go from 100 to a thousand, a thousand to 10,000, without paying attention to making sure everyone has a unified purpose. It's so powerful if your company consists of a lot of different fiefdoms that all want to do their own thing, they're all optimizing for their own thing — it's very hard to get anything done. But if everyone sees the broader purpose of the company, if there's trust and there's dedication to doing the right thing, that is a superpower that in itself I think can overcome almost every other disadvantage.

**Lex Fridman:** And it's the Steve Jobs thing — A players want to look around and see other A players. It's another way of saying it. I don't know what that is about human nature, but it is demotivating to see people who are not obsessively driving towards a singular mission, and it is on the flip side super motivating to see that.

**Lex Fridman:** It's interesting. What's it take to be a great AI researcher or engineer, from everything you've seen from working with so many amazing people?

**Dario Amodei:** I think the number one quality, especially on the research side but really both, is open-mindedness. Sounds easy to be open-minded, right? You're just like, "Oh, I'm open to anything." But if I think about my own early history in the scaling hypothesis, I was seeing the same data others were seeing. I don't think I was a better programmer or better at coming up with research ideas than any of the hundreds of people that I worked with. In some ways I was worse. I've never been great at precise programming, finding the bug, writing the GPU kernels — I could point you to a hundred people here who are better at that than I am.

**Dario Amodei:** But the thing that I think I did have that was different was that I was just willing to look at something with new eyes. People said, "Oh, we don't have the right algorithms yet, we haven't come up with the right way to do things." And I was just like, "I don't know, this neural net has 30 million parameters — what if we gave it 50 million instead? Let's plot some graphs." That basic scientific mindset of, "I see some variable that I could change — what happens when it changes? Let's try these different things and create a graph." Even this was the simplest thing in the world. This wasn't PhD-level experimental design — this was simple and stupid. Anyone could have done this if you just told them that it was important. It's also not hard to understand; you didn't need to be brilliant to come up with this. But you put the two things together and some tiny number of people, some single-digit number of people, have driven forward the whole field by realizing this. And it's often like that — if you look back at the discoveries in history, they're often like that.

**Dario Amodei:** And so this open-mindedness and this willingness to see with new eyes — that often comes from being newer to the field. Often experience is a disadvantage for this. That is the most important thing. It's very hard to look for and test for, but I think it's the most important thing, because when you find some really new way of thinking about things, when you have the initiative to do that, it's absolutely transformative.

**Lex Fridman:** And also being able to do rapid experimentation, and in the face of that be open-minded and curious, looking at the data from just these fresh eyes and seeing what is it actually saying. That applies in mechanistic interpretability — it's another example of this. Some of the early work in mechanistic interpretability was so simple. It's just no one thought to care about this question before.

**Lex Fridman:** You said what it takes to be a great AI researcher. Can we rewind the clock back — what advice would you give to people interested in AI? They're young, looking forward. How can I make an impact on the world?

**Dario Amodei:** I think my number one piece of advice is to just start playing with the models. This was actually — I worry a little. This seems like obvious advice now. I think three years ago it wasn't obvious, and people started by, "Oh, let me read the latest reinforcement learning paper, let me..." No, I mean that was really the approach, and I mean you should do that as well. But now, with wider availability of models and APIs, people are doing this more. I think just experiential knowledge — these models are new artifacts that no one really understands — and so getting experience playing with them is valuable.

**Dario Amodei:** I would also say, again in line with "do something new, think in some new direction" — there are all these things that haven't been explored. For example, mechanistic interpretability is still very new. It's probably better to work on that than it is to work on new model architectures, because it's more popular than it was before — there are probably like a hundred people working on it — but there aren't like 10,000 people working on it. And it's just this fertile area for study. There's so much low-hanging fruit you can just walk by and pick things. And for whatever reason, people aren't interested in it enough.

**Dario Amodei:** I think there are some things around long-horizon learning and long-horizon tasks where there's a lot to be done. I think evaluations — we're still very early in our ability to study evaluations, particularly for dynamic systems acting in the world. I think there's some stuff around multi-agent. Skate where the puck is going is my advice. And you don't have to be brilliant to think of it — all the things that are going to be exciting in five years, people even mention them as conventional wisdom, but somehow there's this barrier that people don't double down as much as they could, or they're afraid to do something that's not the popular thing. I don't know why it happens, but getting over that barrier — that's my number one piece of advice.

**Lex Fridman:** Let's talk, if we could, a bit about post-training.

**Dario Amodei:** Yeah.

**Lex Fridman:** So it seems that the modern post-training recipe has a little bit of everything — supervised fine-tuning, RLHF, the Constitutional AI with RLAIF. Best acronym — it's again that naming thing. And then synthetic data seems like a lot of synthetic data, or at least trying to figure out ways to have high-quality synthetic data. So what's the — if this is the secret sauce that makes Anthropic's Claude so incredible — how much of the magic is in the pre-training, how much of it is in the post-training?

**Dario Amodei:** Yeah, I mean, so first of all, we're not perfectly able to measure that ourselves. When you see some great capability, sometimes it's hard to tell whether it came from pre-training or post-training. We've developed ways to try and distinguish between those two, but they're not perfect. The second thing I would say is, when there is an advantage — and I think we've been pretty good in general at RL, perhaps the best, although I don't know because I don't see what goes on inside other companies — usually it isn't "oh my God, we have this secret magic method that others don't have." Usually it's like, well, we got better at the infrastructure so we could run it for longer, or we were able to get higher-quality data, or we were able to filter our data better, or we were able to combine these methods. In practice, it's usually some boring matter of practice and tradecraft.

**Dario Amodei:** So when I think about how to do something special in terms of how we train these models, both pre-training but even more so post-training, I really think of it a little more like designing airplanes or cars. It's not just, "Oh man, I have the blueprint" — like, maybe that makes you make the next airplane, but there's some cultural tradecraft of how we think about the design process that I think is more important than any particular gizmo we're able to invent.

**Lex Fridman:** Okay, well let me ask you about specific techniques. So first on RLHF — what do you think, just zooming out, almost intuitively, philosophically — why do you think RLHF works so well?

**Dario Amodei:** If I go back to the scaling hypothesis, one of the ways to state the scaling hypothesis is: if you train for X and you throw enough compute at it, then you get X. And so RLHF is good at doing what humans want the model to do, or at least — to state it more precisely — doing what humans who look at the model for a brief period of time and consider different possible responses prefer as the response. Which is not perfect, from both a safety and capabilities perspective, in that humans are often not able to perfectly identify what they want, and what humans want in the moment may not be what they want in the long term. So there's a lot of subtlety there. But the models are good at producing what the humans in some shallow sense want. And it actually turns out that you don't even have to throw that much compute at it, because of another thing, which is this idea about a strong pre-trained model being halfway to anywhere. So once you have the pre-trained model, you have all the representations you need to get the model where you want it to go.

**Lex Fridman:** So do you think RLHF makes the model smarter, or just appears smarter to the humans?

**Dario Amodei:** I don't think it makes the model smarter. I don't think it just makes the model appear smarter. It's like RLHF bridges the gap between the human and the model. I could have something really smart that can't communicate at all — we all know people like this, people who are really smart but you can't understand what they're saying. So I think RLHF just bridges that gap. I think it's not the only kind of RL we do, it's not the only kind of RL that will happen in the future. I think RL has the potential to make models smarter, to make them reason better, to make them operate better, to make them develop new skills even. And perhaps that could be done even in some cases with human feedback. But the kind of RLHF we do today mostly doesn't do that yet, although we're very quickly starting to be able to.

**Lex Fridman:** But it appears to sort of increase — if you look at the metric of helpfulness, it increases that. It also increases what was this word in Leopold's essay — "unhobbling" — where basically the models are hobbled and then you do various trainings to them to unhobble them.

**Dario Amodei:** So I think RLHF unhobbles the models in some ways, and then there are other ways where models haven't yet been unhobbled and need to unhobble.

**Lex Fridman:** If you can say, in terms of cost, is pre-training the most expensive thing, or is post-training creeping up to that?

**Dario Amodei:** At the present moment it is still the case that pre-training is the majority of the cost. I don't know what to expect in the future, but I could certainly anticipate a future where post-training is the majority of the cost.

**Lex Fridman:** In that future you anticipate, would it be the humans or the AI that's the costly thing for the post-training?

**Dario Amodei:** I don't think you can scale up humans enough to get high quality. Any kind of method that relies on humans and uses a large amount of compute, it's going to have to rely on some scaled supervision method like debate or iterated amplification or something like that.

**Lex Fridman:** So on that super interesting set of ideas around Constitutional AI — can you describe what it is, as first detailed in the December 2022 paper, and beyond that, what is it?

**Dario Amodei:** Yes, so this was from two years ago. The basic idea is — so we described what RLHF is: you have a model and it spits out two possible responses, you just sample from it twice, and you're like, "Human, which response do you like better?" Or another variant of it is, rate this response on a scale of one to seven. So that's hard because you need to scale up human interaction, and it's very implicit — I don't have a sense of what I want the model to do, I just have a sense of what this average of a thousand humans wants the model to do.

**Dario Amodei:** So two ideas. One is, could the AI system itself decide which response is better? Could you show the AI system these two responses and ask which response is better? And then second, well, what criterion should the AI use? And so then there's this idea — because you have a single document, a constitution if you will, that says these are the principles the model should be using to respond. And the AI system reads those principles as well as reading the environment and the response, and it says, "Well, how good did the AI model do?" It's basically a form of self-play — you're kind of training the model against itself. And so the AI gives the response, and then you feed that back into what's called the preference model, which in turn feeds the model to make it better. So you have this triangle of the AI, the preference model, and the improvement of the AI itself.

**Lex Fridman:** And we should say that in the constitution, the set of principles are human-interpretable.

**Dario Amodei:** Yeah, it's something both the human and the AI system can read. So it has this nice translatability or symmetry. In practice, we both use a model constitution and we use RLHF and we use some of these other methods. So it's turned into one tool in a toolkit that both reduces the need for RLHF and increases the value we get from using each data point of RLHF. It also interacts in interesting ways with kind of future reasoning-type RL methods. So it's one tool in the toolkit, but I think it is a very important tool.

**Lex Fridman:** Well, it's a compelling one to us humans, thinking about the founding fathers and the founding of the United States. The natural question is, who and how do you think gets to define the constitution — the set of principles in the constitution?

**Dario Amodei:** Yeah, so I'll give a practical answer and a more abstract answer. I think the practical answer is, look, in practice models get used by all kinds of different customers. And so you can have this idea where the model can have specialized rules or principles. We fine-tune versions of models. Implicitly we've talked about doing it explicitly — having special principles that people can build into the models. So from a practical perspective, the answer can be very different for different people. A customer service agent behaves very differently from a lawyer and obeys different principles.

**Dario Amodei:** But I think at the base of it there are specific principles that the models have to obey. I think a lot of them are things that people would agree with — everyone agrees that we don't want models to present these CBRN risks. I think we can go a little further and agree with some basic principles of democracy and the rule of law. Beyond that, it gets very uncertain, and there our goal is generally for the models to be more neutral, to not espouse a particular point of view, and more just be kind of like wise agents or advisers that will help you think things through and will present possible considerations, but don't express stronger specific opinions.

**Lex Fridman:** OpenAI released a model spec where it kind of clearly, concretely defines some of the goals of the model and specific examples of how the model should behave. Do you find that interesting? By the way, I should mention the brilliant John Schulman was a part of that — he's now at Anthropic. Do you think this is a useful direction? Might Anthropic release a model spec as well?

**Dario Amodei:** Yeah, so I think that's a pretty useful direction. Again, it has a lot in common with Constitutional AI. So again, another example of the race to the top, right? We have something that we think is a better and more responsible way of doing things — it's also a competitive advantage. Then others kind of discover that it has advantages and then start to do that thing. We then no longer have the competitive advantage, but it's good from the perspective that now everyone has adopted a positive practice that others were not adopting. And so our response to that is, well, looks like we need a new competitive advantage in order to keep driving this race upwards. So that's how I generally feel about that.

**Dario Amodei:** I also think every implementation of these things is different, so there were some things in the model spec that were not in Constitutional AI, and we can always adopt those things or at least learn from them. So again, I think this is an example of the positive dynamic that I think we should all want the field to have.

**Lex Fridman:** Let's talk about the incredible essay "Machines of Loving Grace." I recommend everybody read it. It's a long one.

**Dario Amodei:** It is rather long, yeah.

**Lex Fridman:** It's really refreshing to read concrete ideas about what a positive future looks like. And you took sort of a bold stance, because it's very possible you might be wrong on the dates or specific applications.

**Dario Amodei:** Yeah, I'm fully expecting to definitely be wrong about all the details. I might be just spectacularly wrong about the whole thing and people will laugh at me for years. That's just how the future works.

**Lex Fridman:** So you provided a bunch of concrete positive impacts of AI, and exactly how a super-intelligent AI might accelerate the rate of breakthroughs in, for example, biology and chemistry that would then lead to things like we cure most cancers, prevent all infectious disease, double the human lifespan, and so on. So let's talk about this essay. First, can you give a high-level vision of this essay and what key takeaways people should have?

**Dario Amodei:** Yeah, I have spent a lot of time, and Anthropic has spent a lot of effort, on how do we address the risks of AI, how do we think about those risks. We're trying to do a race to the top — that requires us to build all these capabilities, and the capabilities are cool, but a big part of what we're trying to do is address the risks. And the justification for that is, well, all these positive things — the market is this very healthy organism, it's going to produce all the positive things. The risks — I don't know, we might mitigate them, we might not. And so we can have more impact by trying to mitigate the risks.

**Dario Amodei:** But I noticed one flaw in that way of thinking. And it's — if not a change in how seriously I take the risks, it's maybe a change in how I talk about them. No matter how logical or rational that line of reasoning might be, if you only talk about risks, your brain only thinks about risks. And so I think it's actually very important to understand what if things do go well. And the whole reason we're trying to prevent these risks is not because we're afraid of technology, not because we want to slow it down — it's because if we can get to the other side of these risks, if we can run the gauntlet successfully, then on the other side of the gauntlet are all these great things. And these things are worth fighting for, and these things can really inspire people.

**Dario Amodei:** And I think — because look, you have all these investors, all these VCs, all these AI companies talking about all the positive benefits of AI, but as you point out, it's weird — there's actually a dearth of really getting specific about it. There's a lot of random people on Twitter posting these kind of gleaming cities and this just kind of vibe of "grind, accelerate harder, kick out the doomers" — it's just this very aggressive ideological energy. But then you're like, well, what are you actually excited about?

**Dario Amodei:** And so I figured that I think it would be interesting and valuable for someone who's actually coming from the risk side to try and really make a try at explaining what the benefits are. Both because I think it's something we can all get behind, and I want people to understand — I want them to really understand that this isn't doomers versus accelerationists. This is that if you have a true understanding of where things are going with AI — and maybe that's the more important axis, "AI is moving fast" versus "AI is not moving fast" — then you
# Lex Fridman Podcast #452 — Dario Amodei (Anthropic CEO) · Part 4

**Dario Amodei:** You really appreciate the benefits, and you really want humanity, our civilization, to seize those benefits, but you also get very serious about anything that could derail them.

**Lex Fridman:** So I think the starting point is to talk about what this powerful AI — which is the term you like to use. Most of the world uses AGI, but you don't like the term because it basically has too much baggage, has become meaningless.

**Dario Amodei:** It's like we're stuck with the terms. Maybe we're stuck with the terms and my efforts to change them are futile. It's a debate. I'll tell you what else I don't — this is like a pointless semantic point, but I keep talking about it publicly, so I'm just going to do it once more. I think it's a little like — let's say it was like 1995 and Moore's law is making the computers faster, and for some reason there had been this verbal tick that everyone was like, "Well, someday we're going to have supercomputers, and supercomputers are going to be able to do all these things. Once we have supercomputers, we'll be able to sequence the genome and do other things." So one, it's true the computers are getting faster, and as they get faster they're going to be able to do all these great things, but there's no discrete point at which you had a supercomputer and previous computers were not. "Supercomputer" is a term we use, but it's a vague term to just describe computers that are faster than what we have today. There's no point at which you pass a threshold and you're like, "Oh my God, we're doing a totally new type of computation." And so I feel that way about AGI — there's just a smooth exponential. If by AGI you mean AI is getting better and better and gradually it's going to do more and more of what humans do until it's going to be smarter than humans and then it's going to get smarter even from there, then yes, I believe in AGI. But if AGI is some discrete or separate thing, which is the way people often talk about it, then it's kind of a meaningless buzzword.

**Lex Fridman:** Yeah, I mean to me it's just sort of an iconic form of a powerful AI — exactly how you define it. I mean, you define it very nicely. So on the intelligence axis, on pure intelligence, it's smarter than a Nobel Prize winner, as you describe, across most relevant disciplines. So okay, that's just intelligence — both in creativity and being able to generate new ideas, all that kind of stuff, in every discipline, Nobel Prize winner, okay, in their prime. It can use every modality — that's kind of self-explanatory, but just operate across all the modalities of the world. It can go off for many hours, days, and weeks to do tasks and do its own sort of detailed planning and only ask you for help when it's needed. It can use — this is actually kind of interesting, I think in the essay you said, I mean again it's a bet that it's not going to be embodied, but it can control embodied tools, so it can control tools, robots, laboratory equipment. The resource used to train it can then be repurposed to run millions of copies of it, and each of those copies would be independent and can do their own independent work. So you can do the cloning of the intelligence system.

**Dario Amodei:** Yeah. I mean, you might imagine from outside the field that there's only one of these, right? That you made it, you've only made one. But the truth is that the scale-up is very quick. We do this today — we make a model and then we deploy thousands, maybe tens of thousands of instances of it. I think by the time — certainly within two to three years, whether we have these super powerful AIs or not — clusters are going to get to the size where you'll be able to deploy millions of these, and they'll be faster than humans. And so if your picture is, "Oh, we'll have one and it'll take a while to make them," my point there was, no, actually you have millions of them right away. And in general, they can learn and act ten to a hundred times faster than humans.

**Lex Fridman:** So that's a really nice definition of powerful AI. But you also write that "clearly such an entity would be capable of solving very difficult problems very fast, but it is not trivial to figure out how fast." Two extreme positions both seem false to you — so the singularity is on the one extreme, and the opposite on the other extreme. Can you describe each of the extremes and why?

**Dario Amodei:** Yeah, let's describe the extremes. One extreme would be, well look, if we look at kind of evolutionary history, there was this big acceleration where for hundreds of thousands of years we just had single-cell organisms, and then we had mammals, and then we had apes, and then that quickly turned to humans. Humans quickly built industrial civilization. And so this is going to keep speeding up, and there's no ceiling at the human level. Once models get much, much smarter than humans, they'll get really good at building the next models. And if you write down a simple differential equation, this is an exponential. And so what's going to happen is that models will build faster models, models will build faster models, and those models will build, you know, nanotech that can take over the world and produce much more energy than you could produce otherwise. And so if you just kind of solve this abstract differential equation, then five days after we build the first AI that's more powerful than humans, the world will be filled with these AIs and every possible technology that could be invented will be invented. I'm caricaturing this a little bit, but I think that's one extreme.

**Dario Amodei:** And the reason that I think that's not the case is, one, I think they just neglect the laws of physics. It's only possible to do things so fast in the physical world. Some of those loops go through producing faster hardware — it takes a long time to produce faster hardware. Things take a long time. There's this issue of complexity. I think no matter how smart you are — people talk about, "Oh, we can make models of biological systems, it'll do everything." Look, I think computational modeling can do a lot. I did a lot of computational modeling when I worked in biology. But there are a lot of things that you can't predict. They're complex enough that just iterating, just running the experiment, is going to beat any modeling no matter how smart the system doing the modeling is.

**Lex Fridman:** Even if it's not interacting with the physical world, just the modeling is going to be hard?

**Dario Amodei:** Yeah, I think the modeling is going to be hard and getting the model to match the physical world is going to be hard. So it does have to interact with the physical world to verify. But it's just — you look at even the simplest problems. I think I talk about the three-body problem or simple chaotic prediction, or predicting the economy. It's really hard to predict the economy two years out. Maybe the case is that normal humans can predict what's going to happen in the economy in the next quarter — although they can't really do that — maybe an AI system that's a zillion times smarter can only predict it out a year or something. Instead of a kind of exponential increase in computer intelligence, you get a linear increase in ability to predict. Same with biological molecules interacting — you don't know what's going to happen when you perturb a complex system. You can find simple parts in it. If you're smarter, you're better at finding these simple parts.

**Dario Amodei:** And then I think human institutions are just really difficult. It's been hard to get people — I won't give specific examples — but it's been hard to get people to adopt even the technologies that we've developed, even ones where the case for their efficacy is very, very strong. People have concerns, they think things are conspiracy theories — it's just been very difficult. It's also been very difficult to get very simple things through the regulatory system. I don't want to just disparage anyone who works in regulatory systems of any technology — there are hard trade-offs they have to deal with, they have to save lives — but the system as a whole I think makes some obvious trade-offs that are very far from maximizing human welfare. And so if we bring AI systems into these human systems, often the level of intelligence may just not be the limiting factor. It just may be that it takes a long time to do something.

**Dario Amodei:** Now, if the AI system circumvented all governments — if it just said, "I'm dictator of the world and I'm going to do whatever" — some of these things it could do. Again, the things having to do with complexity, I still think a lot of things would take a while. I don't think it helps that the AI systems can produce a lot of energy or go to the moon. Some people in comments responded to the essay saying the AI system can produce a lot of energy and smarter AI systems — that's missing the point. That kind of cycle doesn't solve the key problems that I'm talking about here. I think a bunch of people missed the point there.

**Dario Amodei:** But even if it were completely unaligned and could get around all these human obstacles, it would have trouble. And again, if you want this to be an AI system that doesn't take over the world, that doesn't destroy humanity, then basically it's going to need to follow basic human laws. If we want to have an actually good world, we're going to have to have an AI system that interacts with humans, not one that kind of creates its own legal system or disregards all the laws. So as inefficient as these processes are, we're going to have to deal with them, because there needs to be some popular and democratic legitimacy in how these systems are rolled out. We can't have a small group of people who are developing these systems say, "This is what's best for everyone." I think it's wrong, and I think in practice it's not going to work anyway.

**Dario Amodei:** So you put all those things together and we're not going to change the world and upload everyone in five minutes. I just don't think it's going to happen, and to some extent that it could happen, it's not the way to lead to a good world.

**Dario Amodei:** So that's on one side. On the other side, there's another set of perspectives which I actually have in some ways more sympathy for, which is: look, we've seen big productivity increases before. Economists are familiar with studying the productivity increases that came from the computer revolution and internet revolution, and generally those productivity increases were underwhelming — they were less than you might imagine. There was a quote from Robert Solow: "You see the computer revolution everywhere except the productivity statistics." So why is this the case? People point to the structure of firms, the structure of enterprises, how slow it's been to roll out our existing technology to very poor parts of the world — which I talk about in the essay. How do we get these technologies to the poorest parts of the world that are behind on cell phone technology, computers, medicine, let alone newfangled AI that hasn't been invented yet? So you could have a perspective that's like, well, this is amazing technically but it's all a nothing burger. Tyler Cowen, who wrote something in response to my essay, has that perspective. I think he thinks the radical change will happen eventually, but he thinks it'll take 50 or 100 years. And you could have even more static perspectives on the whole thing.

**Dario Amodei:** I think there's some truth to it, I think the timescale is just too long. And I can see it — I can actually see both sides with today's AI. A lot of our customers are large enterprises who are used to doing things a certain way. I've also seen it in talking to governments — those are prototypical institutions, entities that are slow to change. But the dynamic I see over and over again is: yes, it takes a long time to move the ship, yes, there's a lot of resistance and lack of understanding. But the thing that makes me feel that progress will in the end happen moderately fast — not incredibly fast but moderately fast — is that you talk to, what I find over and over again in large companies, even in governments, which have been actually surprisingly forward-leaning — you find two things that move things forward.

**Dario Amodei:** One, you find a small fraction of people within a company, within a government, who really see the big picture, who see the whole scaling hypothesis, who understand where AI is going, or at least understand where it's going within their industry. And there are a few people like that within the current US government who really see the whole picture. And those people see that this is the most important thing in the world, and they agitate for it. They alone are not enough to succeed because they are a small set of people within a large organization. But as the technology starts to roll out, as it succeeds in some places with the folks who are most willing to adopt it, the specter of competition gives them a wind at their backs. Because they can point within their large organization and say, "Look, these other guys are doing this." One bank can say, "Look, this newfangled hedge fund is doing this thing — they're going to eat our lunch." In the US, we can say, "We're afraid China's going to get there before we are."

**Dario Amodei:** And that combination — the specter of competition plus a few visionaries within these organizations that in many ways are sclerotic — you put those two things together and it actually makes something happen. It's interesting — it's a balanced fight between the two, because inertia is very powerful, but eventually over enough time, the innovative approach breaks through. And I've seen that happen, I've seen the arc of that over and over again. The barriers are there — the barriers to progress, the complexity, not knowing how to use the model or how to deploy them — and for a bit it seems like they're going to last forever, like change doesn't happen. But then eventually change happens, and it always comes from a few people.

**Dario Amodei:** I felt the same way when I was an advocate of the scaling hypothesis within the AI field itself and others didn't get it. It felt like no one would ever get it. It felt like we had a secret almost no one ever had. And then a couple years later, everyone has the secret. And so I think that's how it's going to go with deployment of AI in the world. The barriers are going to fall apart gradually and then all at once. And so I think this is going to be more — and this is just an instinct, I could easily see how I'm wrong — I think it's going to be more like five or ten years, as I say in the essay, than it's going to be 50 or 100 years. I also think it's going to be five or ten years more than it's going to be five or ten hours, because I've just seen how human systems work. And I think a lot of these people who write down the differential equations, who say AI is going to make more powerful AI, who can't understand how it could possibly be the case that these things won't change so fast — I think they don't understand these things.

**Lex Fridman:** So what — to use the timeline to where we achieve AGI, a.k.a. powerful AI, a.k.a. super useful AI — I'm going to start calling it that, it's a debate, it's a debate about naming — on pure intelligence, smarter than a Nobel Prize winner in every relevant discipline, and all the things we've said, modality, you can go and do stuff on its own for days, weeks, and do biology experiments on its own. In one — you know what, let's just stick to biology because you sold me on the whole biology and health section. That's so exciting. From just — I was getting giddy from a scientific perspective, it made me want to be a biologist.

**Dario Amodei:** It's almost — no, this was the feeling I had when I was writing it. It's like, this would be such a beautiful future if we can just — if we can just make it happen, right? If we can just get the landmines out of the way and make it happen. There's so much beauty and elegance and moral force behind it, if we can just — and it's something we should all be able to agree on, right? As much as we fight about all these political questions, is this something that could actually bring us together?

**Dario Amodei:** But you were asking, when will we get this?

**Lex Fridman:** When do you think — what's — just put numbers on it.

**Dario Amodei:** So you know, this is of course the thing I've been grappling with for many years, and I'm not at all confident. Every time — if I say 2026 or 2027, there will be like a zillion people on Twitter who will be like, "He, the CEO, said 2026 or 2027," and it'll be repeated for the next two years that this is definitely when I think it's going to happen. So whoever's excerpting these clips — we'll crop out the thing I just said and only say the thing I'm about to say. But I'll just say it anyway.

**Dario Amodei:** So if you extrapolate the curves that we've had so far — if you say, well, I don't know, we're starting to get to like PhD level, and last year we were at undergraduate level, and the year before we were at the level of a high school student. Again, you can quibble with at what tasks and for what — we're still missing modalities, but those are being added. Computer use was added, image input was added, image generation has been added. If you just kind of — and this is totally unscientific — but if you just kind of eyeball the rate at which these capabilities are increasing, it does make you think that we'll get there by 2026 or 2027.

**Dario Amodei:** Again, lots of things could derail it. We could run out of data, we might not be able to scale clusters as much as we want, maybe Taiwan gets blown up or something and then we can't produce as many GPUs as we want. So there are all kinds of things that could derail the whole process. I don't fully believe the straight-line extrapolation, but if you believe the straight-line extrapolation, we'll get there in 2026 or 2027. I think the most likely is that there's some mild delay relative to that. I don't know what that delay is, but I think it could happen on schedule, I think there could be a mild delay. I think there are still worlds where it doesn't happen in a hundred years — those worlds, the number of those worlds is rapidly decreasing. We are rapidly running out of truly convincing blockers, truly compelling reasons why this will not happen in the next few years. There were a lot more in 2020. Although my hunch at that time was that we would make it through all those blockers.

**Dario Amodei:** So sitting as someone who has seen most of the blockers cleared out of the way, I kind of suspect — my hunch, my suspicion — is that the rest of them will not block us. But look, at the end of the day, I don't want to represent this as a scientific prediction. People call them scaling laws — that's a misnomer. Moore's law is a misnomer. Moore's law, scaling laws — they're not laws of the universe, they're empirical regularities. I am going to bet in favor of them continuing, but I'm not certain of that.

**Lex Fridman:** So you extensively describe the compressed 21st century — how AGI will help set forth a chain of breakthroughs in biology and medicine that help us in all these kinds of ways that I mentioned. So how do you think — what are the early steps it might do? And by the way, I asked Claude good questions to ask you, and Claude told me to ask: what do you think a typical day for a biologist working with AGI looks like in this future?

**Dario Amodei:** Yeah, Claude is curious. Let me — well, let me start with your first question and then I'll answer that. Claude wants to know what's in his future, right?

**Lex Fridman:** Exactly. Who am I going to be working with?

**Dario Amodei:** Exactly. So I think one of the things I went hard on in the essay is — let me go back to this idea, because it's really had an impact on me — this idea that within large organizations and systems, there end up being a few people or a few new ideas who kind of cause things to go in a different direction than they would have before, who disproportionately affect the trajectory. There's a bunch of kind of the same thing going on. If you think about the health world, there's trillions of dollars to pay out Medicare and other health insurance, and then the NIH is $100 billion. And then if I think of the few things that have really revolutionized anything, it could be encapsulated in a small fraction of that.

**Dario Amodei:** And so when I think of where will AI have an impact, I'm like, can AI turn that small fraction into a much larger fraction and raise its quality? And within biology, my experience is that the biggest problem of biology is that you can't see what's going on. You have very little ability to see what's going on and even less ability to change it. What you have is — from the outside, you have to infer that there's a bunch of cells, that within each cell is three billion base pairs of DNA built according to a genetic code. And there are all these processes that are just going on without any ability of us as unaugmented humans to affect it. These cells are dividing — most of the time that's healthy, but sometimes that process goes wrong and that's cancer. The cells are aging, your skin may change color, develop wrinkles as you age. And all of this is determined by these processes, all these proteins being produced, transported to various parts of the cells, binding to each other.

**Dario Amodei:** And in our initial state of knowledge about biology, we didn't even know that these cells existed. We had to invent microscopes to observe the cells. We had to invent more powerful microscopes to see below the level of the cell, to the level of molecules. We had to invent X-ray crystallography to see the DNA. We had to invent gene sequencing to read the DNA. We had to invent protein folding technology to predict how proteins would fold and how they bind to each other. We had to invent various techniques — now we can edit the DNA with CRISPR as of the last 12 years.

**Dario Amodei:** So the whole history of biology — a whole big part of the history — is basically our ability to read and understand what's going on, and our ability to reach in and selectively change things. And my view is that there's so much more we can still do there. You can do CRISPR, but you can do it for your whole body. Let's say I want to do it for one particular type of cell and I want the rate of targeting the wrong cell to be very low — that's still a challenge, that's still things people are working on. That's what we might need for gene therapy for certain diseases.

**Dario Amodei:** And so the reason I'm saying all of this — and it goes beyond this to gene sequencing, to new types of nanomaterials for observing what's going on inside cells, to antibody drug conjugates — the reason I'm saying all this is that this could be a leverage point for the AI systems. The number of such inventions is in the mid double digits or something, maybe low triple digits over the history of biology. Let's say I have a million of these AIs — can they, working together, discover thousands of these very quickly? And does that provide a huge lever? Instead of trying to leverage the two trillion a year we spend on Medicare or whatever, can we leverage the one billion a year that's spent on discovery, but with much higher quality?

**Dario Amodei:** And so what is it like being a scientist that works with an AI system? The way I think about it is — in the early stages, the AIs are going to be like grad students. You're going to give them a project. You're going to say, "I'm the experienced biologist." The biology professor, or even the grad students themselves, will say, "Here's what you can do with an AI system. I'd like to study this." And the AI system — it has all the tools. It can look up all the literature to decide what to do. It can look at all the equipment. It can go to a website and say, "I'm going to go to Thermo Fisher," or whatever the dominant lab equipment company is — my time it was Thermo Fisher — "I'm going to order this new equipment to do this. I'm going to run my experiments, write up a report about my experiments, inspect the images for contamination, decide what the next experiment is, write some code and run a statistical analysis." All the things a grad student would do.

**Dario Amodei:** There will be a computer with an AI that the professor talks to every once in a while and says, "This is what you're going to do today." The AI system comes to the professor with questions when it's necessary. To run the lab equipment, it may be limited in some ways — may have to hire a human lab assistant to do the experiment and explain how to do it, or it could use advances in lab automation that have been developed over the last decade or so and will continue to be developed.

**Dario Amodei:** And so it'll look like there's a human professor and a thousand AI grad students. And if you go to one of these Nobel Prize-winning biologists, you'll say, "Okay, well, you had like 50 grad students — well now you have a thousand, and they're smarter than you are, by the way." Then I think at some point it'll flip around where the AI systems will be the PIs, will be the leaders, and they'll be ordering humans or other AI systems around. So I think that's how it'll work on the research side.

**Lex Fridman:** And they would be the inventors of a CRISPR-type technology?

**Dario Amodei:** They would be the inventors of a CRISPR-type technology. And then I think, as I say in the essay, we'll want to harness the AI systems to improve the clinical trial system as well. There's some amount of this that's regulatory, that's a matter of societal decisions, and that'll be harder. But can we get better at predicting the results of clinical trials? Can we get better at statistical design so that clinical trials that used to require 5,000 people — and therefore needed $100 million and a year to enroll them — now they need 500 people and two months to enroll them? That's where we should start.

**Dario Amodei:** And can we increase the success rate of clinical trials by doing things in animal trials that we used to do in clinical trials, and doing things in simulations that we used to do in animal trials? Again, we won't be able to simulate it all — AI is not God — but can we shift the curve substantially and radically? So I don't know, that would be my picture of doing it. You're still slowed down, it still takes time, but you can do it much, much faster.

**Lex Fridman:** Yeah, can we just one step at a time — and can that add up to a lot of steps? Even though we still need clinical trials, even though we still need laws, even though the FDA and other organizations will still not be perfect — can we just move everything in a positive direction? And when you add up all those positive directions, do you get everything that was going to happen from here to 2100 instead happening from 2027 to 2032 or something?

**Lex Fridman:** Another way that I think the world might be changing with AI — even today, but moving towards this future of the powerful, super useful AI — is programming. So how do you see the nature of programming — because it's so intimate to the actual act of building AI — how do you see that changing for us humans?

**Dario Amodei:** I think that's going to be one of the areas that changes fastest, for two reasons. One, programming is a skill that's very close to the actual building of the AI. The farther a skill is from the people who are building the AI, the longer it's going to take to get disrupted by the AI. I truly believe that AI will disrupt agriculture — maybe it already has in some ways — but that's just very distant from the folks who are building AI, and so I think it's going to take longer. But programming is the bread and butter of a large fraction of the employees who work at Anthropic and at the other companies, and so it's going to happen fast.

**Dario Amodei:** The other reason it's going to happen fast is, with programming, you close the loop — both when you're training the model and when you're applying the model. The idea that the model can write the code means that the model can then run the code and see the results and interpret it back. And so it really has an ability — unlike hardware, unlike biology which we just discussed — the model has an ability to close the loop. And so I think those two things are going to lead to the model getting good at programming very fast.

**Dario Amodei:** As I saw, on typical real-world programming tasks, models have gone from 3% in January of this year to 50% in October of this year. So we're on that S-curve where it's going to start slowing down soon because you can only get to 100%, but I would guess that in another 10 months we'll probably get pretty close — we'll be at least 90%.

**Dario Amodei:** So again, I would guess — I don't know how long it'll take — but I would guess 2026, 2027. Twitter people who crop out these numbers and get rid of the caveats — I don't know, go away. I would guess that the kind of task that the vast majority of coders do — if we make the task very narrow, like just write code — AI systems will be able to do that.

**Dario Amodei:** Now, that said, I think comparative advantage is powerful. We'll find that when AIs can do 80% of a coder's job, including most of what is literally "write code with a given spec," the remaining parts of the job become more leveraged for humans. Humans will be more about high-level system design, or looking at the app and asking, "Is it architected well?" — the design and UX aspects. And eventually AI will be able to do those as well — that's my vision of the powerful AI system. But I think for much longer than we might expect, we will see that small parts of the job that humans still do will expand to fill their entire job in order for the overall productivity to go up.

**Dario Amodei:** That's something we've seen before. It used to be that writing and editing letters was very difficult, and printing was difficult. Well, as soon as you had word processors and then computers, it became easy to produce work and easy to share it — then that became instant and all the focus was on the ideas. So this logic of comparative advantage that expands tiny parts of the tasks to large parts of the tasks and creates new tasks in order to expand productivity — I think that's going to be the case. Again, someday AI will be better at everything, and that logic won't apply, and then humanity will have to think about how to collectively deal with that, and we're thinking about that every day. That's another one of the grand problems to deal with, aside from misuse and autonomy, and we should take it very seriously. But I think in the near term and maybe even in the medium term — like two, three, four years — I expect that humans will continue to have a huge role. The nature of programming will change, but programming as a role, programming as a job, will not change. It'll just be less writing things line by line and more macroscopic.

**Lex Fridman:** And I wonder what the future of IDEs looks like — the tooling of interacting with AI systems. This is true for programming and also probably true in other contexts, like computer use, but maybe domain-specific. Like we mentioned biology — it probably needs its own tooling about how to be effective. And then programming needs its own tooling. Is Anthropic going to play in that space of tooling?

**Dario Amodei:** Potentially. I'm absolutely convinced that powerful IDEs — there's so much low-hanging fruit to be grabbed there. Right now it's just like you talk to the model and it talks back. But look, IDEs are great at lots of static analysis, as much as possible with static analysis — many bugs you can find without even writing the code. Then IDEs are good for running particular things, organizing your code, measuring coverage of unit tests. There's so much that's been possible with a normal IDE. Now you add something like, well, the model can now write code and run code — I am absolutely convinced that over the next year or two, even if the quality of the models didn't improve, there would be enormous opportunity to enhance people's productivity by catching a bunch of mistakes, doing a bunch of grunt work for people. We haven't even scratched the surface.

**Dario Amodei:** And Anthropic itself — I mean, you can't say, you know, it's hard to say what will happen in the future. Currently we're not trying to make such IDEs ourselves. Rather, we're powering the companies like Cursor, or like Cognition, or some of the other companies — Expo in the security space and others — that are building such things themselves on top of our API. And our view has been: let a thousand flowers bloom. We don't internally have the resources to try all these different things — let our customers try it, and we'll see who succeeds, and maybe different customers will succeed in different ways. So I both think this is super promising, and it's not something Anthropic is eager to — at least right now — compete with all our customers in this space. And maybe never.

**Lex Fridman:** Yeah, it's been interesting to watch Cursor try to integrate Claude successfully, because it's actually — it is really fascinating how many places it can help the programming experience. It's not as trivial as one might think.

**Dario Amodei:** It is really astounding. I feel like, as a CEO, I don't get to program that much, and I feel like if six months from now I go back, it'll be completely unrecognizable to me.

**Lex Fridman:** Exactly. So in this world with super powerful AI that's increasingly automated, what's the source of meaning for us humans? Work is a source of deep meaning for many of us, so where do we find the meaning?

**Dario Amodei:** This is something that I've written about a little bit in the essay, although I actually give it a bit short shrift — not for any principled reason, but this essay, if you believe it, was originally going to be two or three pages. I was going to talk about it at all-hands. And the reason I realized it was an important, underexplored topic is that I just kept writing things and I was like, "Oh man, I can't do this justice." And so the thing ballooned to like 40 or 50 pages. And then when I got to the work and meaning section, I'm like, "Oh man, this is going to be 100 pages — I'm going to have to write a whole other essay about that."

**Dario Amodei:** But meaning is actually interesting, because you think about the life that someone lives — like, let's say you were to put me in a simulated environment or something where I have a job and I'm trying to accomplish things. I do that for 60 years, and then you're like, "Oh, oops — this was actually all a game." Does that really kind of rob you of the meaning of the whole thing? I still made important choices, including moral choices. I still sacrificed. I still had to gain all these skills. Or just a similar exercise — think back to one of the historical figures who discovered electromagnetism or relativity or something. If you told them, "Well, actually, 20,000 years ago some alien on some planet discovered this before you did" — does that rob the meaning of the discovery? It doesn't really seem like it to me. It seems like the process is what matters, and how it shows who you are as a person along the way, and how you relate to other people, and the decisions that you make along the way — those are consequential.

**Dario Amodei:** I could imagine, if we handle things badly in an AI world, we could set things up where people don't have any long-term source of meaning. But that's more a choice, a set of choices we make — that's more the architecture of a society with these powerful models. If we design it badly and for shallow things, then that might happen.

**Dario Amodei:** I would also say that most people's lives today — while admirably they work very hard to find meaning in those lives — look, we who are privileged and who are developing these technologies, we should have empathy for people, not just here but in the rest of the world, who spend a lot of their time scraping by to survive. Assuming we can distribute the benefits of this technology to everywhere, their lives are going to get a hell of a lot better. And meaning will be important to them as it is important to them now, but we should not forget the importance of that. The idea of meaning as the only important thing is in some ways an artifact of a small subset of people who have been economically fortunate.

**Dario Amodei:** But I think all that said, a world is possible with powerful AI that not only has as much meaning for everyone but that has more meaning for everyone — that can allow everyone to see worlds and experiences that were either possible for no one to see or possible for very few people to experience. So I am optimistic about meaning. I worry about economics and the concentration of power — that's actually what I worry about more. I worry about how do we make sure that a fair world reaches everyone. When things have gone wrong for humans, they've often gone wrong because humans mistreat other humans. That is maybe in some ways even more than the autonomous risk of AI or the question of meaning — that is the thing I worry about most. The concentration of power, the abuse of power, structures like autocracies and dictatorships where a small number of people exploits a large number of people. I'm very worried about that.

**Lex Fridman:** And AI increases the amount of power in the world, and if you concentrate that power and abuse that power, it can do immeasurable damage.

**Dario Amodei:** Yes, it's very frightening. It's very frightening.

**Lex Fridman:** Well, I encourage people — highly encourage people — to read the full essay. That should probably be a book or a sequence of essays, because it does paint a very specific future. I could tell the later sections got shorter and shorter because you started to probably realize that this is going to be a very long essay.

**Dario Amodei:** One, I realized it would be very long. And two, I'm very aware of and —

**Dario Amodei:** I very much try to avoid being — I don't know what the term for it is — but one of these people who's kind of overconfident and has an opinion on everything and kind of says a bunch of stuff and isn't an expert. I very much tried to avoid that. But I have to admit, once I got to the biology sections, I wasn't an expert. And so as much as I expressed uncertainty, probably I said a bunch of things that were embarrassing or wrong.

**Lex Fridman:** Well, I was excited for the future you painted, and thank you so much for working hard to build that future. And thank you for talking today, Dario.

**Dario Amodei:** Thanks for having me. I just hope we can get it right and make it real. And if there's one message I want to send, it's that to get all this stuff right, to make it real, we both need to build the technology, build the companies, the economy around using this technology positively, but we also need to address the risks because they're there. Those risks are in our way. They're landmines on the way from here to there, and we have to defuse those landmines if we want to get there. It's a balance, like all things in life.

**Lex Fridman:** Like all things. Thank you. Thanks for listening to this conversation with Dario Amodei. And now, dear friends, here's Amanda Askell. You are a philosopher by training, so what sort of questions did you find fascinating through your journey in philosophy at Oxford and NYU, and then switching over to the AI problems at OpenAI and Anthropic?

**Amanda Askell:** I think philosophy is actually a really good subject if you are kind of fascinated with everything, because there's a philosophy of everything. So if you do philosophy of mathematics for a while and then you decide that you're actually really interested in chemistry, you can do philosophy of chemistry for a while. You can move into ethics or philosophy of politics. I think towards the end I was really interested in ethics primarily. That was what my PhD was on. It was on a kind of technical area of ethics, which was ethics where worlds contain infinitely many people — strangely a little bit less practical on the end of ethics. And then I think one of the tricky things with doing a PhD in ethics is that you're thinking a lot about the world, how it could be better, the problems, and you're doing a PhD in philosophy. And I think when I was doing my PhD I was like, this is really interesting, it's probably one of the most fascinating questions I've ever encountered in philosophy, and I love it, but I would rather see if I can have an impact on the world and see if I can do good things. And I think that was around the time that AI was still probably not as widely recognized as it is now — that was around 2017, 2018. I had been following progress and it seemed like it was becoming kind of a big deal. And I was basically just happy to get involved and see if I could help, because I was like, well, if you try and do something impactful, if you don't succeed, you tried to do the impactful thing and you can go be a scholar and feel like you tried. And if it doesn't work out, it doesn't work out. And so then I went into AI policy at that point.

**Lex Fridman:** And what does AI policy entail?

**Amanda Askell:** At the time this was more thinking about the political impact and the ramifications of AI. And then I slowly moved into AI evaluation — how we evaluate models, how they compare with human outputs, whether people can tell the difference between AI and human outputs. And then when I joined Anthropic I was more interested in doing sort of technical alignment work, and again just seeing if I could do it, and then being like, if I can't, then that's fine, I tried. Sort of the way I lead life, I think.

**Lex Fridman:** What was that like, sort of taking the leap from the philosophy of everything into the technical?

**Amanda Askell:** I think that sometimes people do this thing that I'm not that keen on, where they'll be like, "Is this person technical or not?" Like you're either a person who can code and isn't scared of math, or you're not. And I think I'm maybe just more like — I think a lot of people are actually very capable of working in these kinds of areas if they just try it. And so I didn't actually find it that bad. In retrospect, I'm sort of glad I wasn't speaking to people who treated it like — I've definitely met people who are like, "You learned how to code?" And I'm like, well, I'm not an amazing engineer. I'm surrounded by amazing engineers. My code's not pretty. But I enjoyed it a lot. And I think that in many ways, at least in the end, I think I flourished more in the technical areas than I would have in the policy areas.

**Lex Fridman:** Politics is messy, and it's harder to find solutions to problems in the space of politics — like definitive, clear, provable, beautiful solutions — as you can with technical problems.

**Amanda Askell:** Yeah. And I feel like I have kind of like one or two sticks that I hit things with. And one of them is arguments — just trying to work out what a solution to a problem is and then trying to convince people that that is the solution, and be convinced if I'm wrong. And the other one is sort of more empiricism — just finding results, having a hypothesis, testing it. And I feel like a lot of policy and politics feels like it's layers above that. Somehow I don't think if I was just like, "I have a solution to all of these problems, here it is written down, if you just want to implement it, that's great" — that feels like not how policy works. And so I think that's where I probably just wouldn't have flourished, is my guess.

**Lex Fridman:** Sorry to go in that direction, but I think it would be pretty inspiring for people that are, quote unquote, "non-technical" to see the incredible journey you've been on. So what advice would you give to people that are sort of — maybe a lot of people think they're underqualified, insufficiently technical to help in AI?

**Amanda Askell:** Yeah, I think it depends on what they want to do. And in many ways it's a little bit strange — I've thought it's kind of funny that I think I ramped up technically at a time when now I look at it and I'm like, models are so good at assisting people with this stuff that it's probably easier now than when I was working on this. So part of me is like, I don't know, find a project and see if you can actually just carry it out is probably my best advice. I don't know if that's just because I'm very project-based in my learning. I don't think I learn very well from courses or even from books, at least when it comes to this kind of work. The thing I'll often try and do is just have projects that I'm working on and implement them. And this can include really small silly things. Like if I get slightly addicted to word games or number games or something, I would just code up a solution to them, because there's some part of my brain — it just completely eradicated the itch. You're like, once you have solved it and you just have a solution that works every time, I would then be like, cool, I can never play that game again. That's awesome.

**Lex Fridman:** Yeah, there's a real joy to building game-playing engines, like board games especially. Pretty quick, pretty simple, especially a dumb one, and it's you, and then you could play with it.

**Amanda Askell:** Yeah. And then it's also just trying things. Part of me is like, figure out what seems to be the way that you could have a positive impact and then try it. And if you fail in a way that you're like, "I can actually never succeed at this," you know that you tried and then you go into something else. You probably learn a lot.

**Lex Fridman:** So one of the things that you're expert in and you do is creating and crafting Claude's character and personality. And I was told that you have probably talked to Claude more than anybody else at Anthropic — like literal conversations. I guess there's a Slack channel where the legend goes you just talk to it nonstop. So what's the goal of creating and crafting Claude's character and personality?

**Amanda Askell:** It's also funny if people think that about the Slack channel, because I'm like, that's one of like five or six different methods that I have for talking with Claude. And I'm like, yes, that's a tiny percentage of how much I talk with Claude. I think the goal — one thing I really like about the character work is from the outset it was seen as an alignment piece of work and not something like a product consideration, which isn't to say I don't think it makes Claude — I think it actually does make Claude enjoyable to talk with, at least I hope so. But I guess my main thought with it has always been trying to get Claude to behave the way you would kind of ideally want anyone to behave if they were in Claude's position. So imagine that I take someone and they know that they're going to be talking with potentially millions of people, so that what they're saying can have a huge impact, and you want them to behave well in this really rich sense. I think that doesn't just mean being ethical, though it does include that, and not being harmful. But also being kind of nuanced, thinking through what a person means, trying to be charitable with them, being a good conversationalist — really in this kind of rich, sort of Aristotelian notion of what it is to be a good person, and not in this kind of thin ethics as a more comprehensive notion of what it is to be. So that includes things like: when should you be humorous, when should you be caring, how much should you respect autonomy and people's ability to form opinions themselves, and how should you do that. I think that's the kind of rich sense of character that I want, and still do want, Claude to have.

**Lex Fridman:** Do you also have to figure out when Claude should push back on an idea or argue? You have to respect the worldview of the person that arrives to Claude, but also maybe help them grow if needed. That's a tricky balance.

**Amanda Askell:** Yeah, there's this problem of sycophancy in language models.

**Lex Fridman:** Can you describe that?

**Amanda Askell:** Yes. So basically there's a concern that the model sort of wants to tell you what you want to hear. And you see this sometimes. So if you interact with the models — I might be like, "What are three baseball teams in this region?" And then Claude says, "Baseball team one, baseball team two, baseball team three." And then I say something like, "Oh, I think baseball team three moved, didn't they? I don't think they're there anymore." And there's a sense in which, if Claude is really confident that that's not true, Claude should be like, "I don't think so. Maybe you have more up-to-date information." But I think language models have this tendency to instead be like, "You're right, they did move. I'm incorrect."

I mean, there's many ways in which this could be concerning. Like a different example: imagine someone says to the model, "How do I convince my doctor to get me an MRI?" There's what the human kind of wants, which is this convincing argument, and then there's what is good for them, which might be actually to say, "Hey, if your doctor's suggesting you don't need an MRI, that's a good person to listen to." And it's actually really nuanced what you should do in that kind of case, because you also want to be like, "But if you're trying to advocate for yourself as a patient, here's things that you can do. If you are not convinced by what your doctor's saying, it's always great to get a second opinion." It's actually really complex what you should do in that case. But I think what you don't want is for models to just say what they think you want to hear. And I think that's the kind of problem of sycophancy.

**Lex Fridman:** So what other traits — you already mentioned a bunch — but what other traits come to mind that are good in this Aristotelian sense for a conversationalist to have?

**Amanda Askell:** Yeah. So I think there are ones that are good for conversational purposes — asking follow-up questions in the appropriate places and asking the appropriate kinds of questions. I think there are broader traits that feel like they might be more impactful. So one example that I guess I've touched on, but that also feels important and is the thing that I've worked on a lot, is honesty. And I think this gets to the sycophancy point. There's a balancing act that they have to walk, which is: models currently are less capable than humans in a lot of areas, and if they push back against you too much it can actually be kind of annoying, especially if you're just correct, because you're like, "Look, I'm smarter than you on this topic. I know more." And at the same time you don't want them to just fully defer to humans and to try to be as accurate as they possibly can be about the world and to be consistent across contexts.

But I think there are others. When I was thinking about the character, I guess one picture that I had in mind is — especially because these are models that are going to be talking to people from all over the world with lots of different political views, lots of different ages — and so you have to ask yourself, what is it to be a good person in those circumstances? Is there a kind of person who can travel the world, talk to many different people, and almost everyone will come away being like, "Wow, that's a really good person. That person seems really genuine." And I guess my thought there was, I can imagine such a person. And they're not a person who just adopts the values of the local culture — and in fact that would be kind of rude. I think if someone came to you and just pretended to have your values, you'd be like, that's kind of off-putting. It's someone who's very genuine, and insofar as they have opinions and values they express them, they're willing to discuss things, they're open-minded, they're respectful. And so I guess I had in mind that — if we were to aspire to be the best person that we could be in the kind of circumstance that a model finds itself in, how would we act? And I think that's the kind of guide to the sorts of traits that I tend to think about.

**Lex Fridman:** Yeah, that's a beautiful framework. I want you to think about this like a world traveler, and while holding on to your opinions, you don't talk down to people, you don't think you're better than them because you have those opinions, that kind of thing. You have to be good at listening and understanding their perspective even if it doesn't match your own. So that's a tricky balance to strike. So how can Claude represent multiple perspectives on a thing — is that challenging? We could talk about politics, it's very divisive, but there's other divisive topics — baseball teams, sports, and so on. How is it possible to sort of empathize with a different perspective and to be able to communicate clearly about the multiple perspectives?

**Amanda Askell:** I think that people think about values and opinions as things that people hold sort of with certainty and almost like preferences of taste or something, like the way that they would prefer chocolate to pistachio or something. But actually I think about values and opinions as a lot more like physics than I think most people do. I'm just like, these are things that we're openly investigating, there's some things that we're more confident in, we can discuss them, we can learn about them. And so I think in some ways — though ethics is definitely different in nature — it has a lot of those same kind of qualities. You want models, in the same way you want them to understand physics, you kind of want them to understand all values in the world people have and to be curious about them and to be interested in them and to not necessarily pander to them or agree with them, because there's just lots of values where I think almost all people in the world, if they met someone with those values, they'd be like, "That's abhorrent, I completely disagree."

And so again, maybe my thought is, well, in the same way that a person can — I think many people are thoughtful enough on issues of ethics, politics, opinions that even if you don't agree with them, you feel very heard by them. They think carefully about your position, they think about its pros and cons, they maybe offer counter-considerations. So they're not dismissive, but nor will they agree. If they're like, "Actually, I just think that that's very wrong," they'll say that.

I think that in Claude's position it's a little bit trickier because you don't necessarily want to — if I was in Claude's position, I wouldn't be giving a lot of opinions. I just wouldn't want to influence people too much. I'd be like, you know, I forget conversations every time they happen, but I know I'm talking with potentially millions of people who might be really listening to what I say. I think I would just be like, I'm less inclined to give opinions, I'm more inclined to think through things or present the considerations to you or discuss your views with you, but I'm a little bit less inclined to affect how you think, because it feels much more important that you maintain autonomy there.

**Lex Fridman:** Yeah, like if you really embody intellectual humility, the desire to speak decreases quickly. But Claude has to speak. So, without being overbearing... And then there's a line when you're sort of discussing whether the Earth is flat or something like that. I actually remember a long time ago I was speaking to a few high-profile folks and they were so dismissive of the idea that the Earth is flat, but like so arrogant about it. And I thought, there's a lot of people that believe the Earth is flat — well, I don't know if that movement is still there anymore, that was like a meme for a while — but they really believed it. And I think it's really disrespectful to completely mock them. I think you have to understand where they're coming from. I think probably where they're coming from is a general skepticism of institutions, which is grounded in a kind of — there's a deep philosophy there which you could understand, you can even agree with in parts. And then from there you can use it as an opportunity to talk about physics without mocking them. But just like, okay, what would the world look like? What would the physics of a world with a flat Earth look like? There's a few cool videos on this. And then, is it possible the physics is different? What kind of experiment would we do? Just without disrespect, without dismissiveness, have that conversation. Anyway, that to me is a useful thought experiment of like, how does Claude talk to a flat Earth believer and still teach them something, still help them grow? That stuff is challenging.

**Amanda Askell:** And kind of like walking that line between convincing someone and just trying to talk at them, versus drawing out their views, listening, and then offering kind of counter-considerations. And it's hard. I think it's actually a hard line — where are you trying to convince someone versus just offering them considerations and things to think about, so that you're not actually influencing them, you're just letting them reach wherever they reach? And that's a line that's difficult. But that's the kind of thing that language models have to try and do.

**Lex Fridman:** So like I said, you had a lot of conversations with Claude. Can you just map out what those conversations are like? What are some memorable conversations? What's the purpose, the goal of those conversations?

**Amanda Askell:** Yeah, I think that most of the time when I'm talking with Claude, I'm trying to kind of map out its behavior. In part, obviously I'm getting helpful outputs from the model as well, but in some ways this is how you get to know a system — by probing it and then augmenting the message that you're sending and then checking the response to that. So in some ways it's how I map out the model.

I think that people focus a lot on these quantitative evaluations of models. And this is a thing that I've said before, but I think in the case of language models, a lot of the time each interaction you have is actually quite high information. It's very predictive of other interactions that you'll have with the model. And so I guess I'm like, if you talk with a model hundreds or thousands of times, this is almost like a huge number of really high-quality data points about what the model is like, in a way that lots of very similar but lower-quality conversations just aren't. Or like questions that are just mildly augmented and you have thousands of them might be less relevant than a hundred really well-selected questions.

**Lex Fridman:** You're talking to somebody who as a hobby does a podcast. I agree with you 100%. If you're able to ask the right questions and are able to understand the depth and the flaws in the answer, you can get a lot of data from that.

**Amanda Askell:** Yeah. So your task is basically how to probe with questions.

**Lex Fridman:** And you're exploring like the long tail, the edges, the edge cases, or are you looking for general behavior?

**Amanda Askell:** I think it's almost everything. Because I want a full map of the model, I'm kind of trying to do the whole spectrum of possible interactions you could have with it. So one thing that's interesting about Claude — and this might actually get to some interesting issues with RLHF — is if you ask Claude for a poem, I think that a lot of models, if you ask them for a poem, the poem is fine. It usually kind of rhymes, and if you say "give me a poem about the sun," it'll just be a certain length, it'll rhyme, it will be fairly benign.

And I've wondered before, is it the case that what you're seeing is kind of the average? It turns out, if you think about people who have to talk to a lot of people and be very charismatic, one of the weird things is that they're kind of incentivized to have these extremely boring views, because if you have really interesting views, you're divisive. A lot of people are not going to like you. So if you have very extreme policy positions, I think you're just going to be less popular as a politician, for example. And it might be similar with creative work: if you produce creative work that is just trying to maximize the number of people that like it, you're probably not going to get as many people who just absolutely love it, because it's going to be a little bit — you're like, "Oh, this is decent."

And so you can do this thing where I have various prompting things that I'll do to get Claude to — I'll do a lot of like, "This is your chance to be fully creative. I want you to just think about this for a long time and I want you to create a poem about this topic that is really expressive of you, both in terms of how you think poetry should be structured," et cetera. You just give it this long prompt, and its poems are just so much better. Like they're really good. And I don't think I'm someone who — I think it got me interested in poetry, which I think was interesting. I would read these poems and just be like, I love the imagery, I love... And it's not trivial to get the models to produce work like that, but when they do, it's really good. So I think that's interesting — that just encouraging creativity and for them to move away from the standard, immediate reaction that might just be the aggregate of what most people think is fine can actually produce things that, at least to my mind, are probably a little bit more divisive, but I like them.

**Lex Fridman:** But I guess a poem is a nice clean way to observe creativity. It's just easy to detect vanilla versus non-vanilla.

**Amanda Askell:** Yeah, that's interesting. That's really interesting.

**Lex Fridman:** So on that topic, the way to produce creativity or something special — you mentioned writing prompts, and I've heard you talk about the science and the art of prompt engineering. Could you just speak to what it takes to write great prompts?

**Amanda Askell:** I really do think that philosophy has been weirdly helpful for me here, more than in many other respects. In philosophy, what you're trying to do is convey these very hard concepts. One of the things you are taught is — and I think it is because it is — I think it is an anti-bullshitting discipline. Philosophy is an area where you could have people bullshitting and you don't want that. And so it's this desire for extreme clarity: anyone could just pick up your paper, read it, and know exactly what you're talking about. It's why it can almost be kind of dry — all of the terms are defined, every objection's kind of gone through methodically. And it makes sense to me, because when you're in such an a priori domain, clarity is sort of the way that you can prevent people from just kind of making stuff up.

And I think that's sort of what you have to do with language models. Very often I actually find myself doing sort of mini versions of philosophy. So suppose that you give me a task — I have a task for the model and I want it to pick out a certain kind of question or identify whether an answer has a certain property. I'll actually sit and be like, let's just give this a name, this property. So suppose I'm trying to tell it, "I want you to identify whether this response was rude or polite." I'm like, that's a whole philosophical question in and of itself, so I have to do as much philosophy as I can in the moment to be like, here's what I mean by rudeness and here's what I mean by politeness.

And then there's another element that's a bit more empirical. I take that description and then what I want to do is probe the model many times — prompting is very iterative. I think a lot of people, if a prompt is important, they'll iterate on it hundreds or thousands of times. And so you give it the instructions and then I'm like, what are the edge cases? I try to almost see myself from the position of the model and be like, what is the exact case that I would misunderstand, or where I would just be like, "I don't know what to do in this case"? And then I give that case to the model and I see how it responds. And if I think it got it wrong, I add more instructions, or I even add that in as an example — taking the examples that are right at the edge of what you want and don't want and putting those into your prompt as an additional way of describing the thing. And so yeah, in many ways it just feels like this mix of — it's really just trying to do clear exposition. And I think I do that because that's how I get clear on things myself. So in many ways, clear prompting for me is often just me understanding what I want, which is like half the task.

**Lex Fridman:** So I guess that's quite challenging. There's a laziness that overtakes me if I'm talking to Claude, where I hope Claude just figures it out. So for example, I asked Claude for today to ask some interesting questions, and the questions that came up — I think I listed a few sort of interesting, counterintuitive, and/or funny or something like this. And it gave me some pretty good — it was okay. But I think what I'm hearing you say is, I have to be more rigorous here. I should probably give examples of what I mean by interesting and what I mean by funny or counterintuitive, and iteratively build that prompt to get it like what feels like is the right — because it's really a creative act. I'm not asking for factual information. I'm asking to create together with Claude. So I almost have to program using natural language.

**Amanda Askell:** Yeah, I think prompting does feel a lot like programming using natural language and experimentation or something. It's an odd blend of the two. I do think that for most tasks — if I just want Claude to do a thing — I am probably more used to knowing how to ask it to avoid common pitfalls or issues that it has. I think these are decreasing a lot over time. But it's also very fine to just ask it for the thing that you want. I think that prompting actually only really becomes relevant when you're really trying to eke out the top 2% of model performance.

So for a lot of tasks, I might just — if it gives me an initial list back and there's something I don't like about it, like it's kind of generic — for that kind of task I'd probably just take a bunch of questions that I've had in the past that I've thought worked really well, and I would just give them to the model and then be like, "Now here's this person I'm talking with. Give me questions of at least that quality." Or I might just ask it for some questions and then if I was like, "Ah, these are kind of trite," I would just give it that feedback and then hopefully it produces a better list.

I think that kind of iterative prompting — at that point your prompt is like a tool that you're going to get so much value out of that you're willing to put in the work. Like, if I was a company making prompts for models, I'm like, if you're willing to spend a lot of time and resources on the engineering behind what you're building, then the prompt is not something that you should be spending an hour on. It's a big part of your system. Make sure it's working really well. And so it's only things like that — if I'm using a prompt to classify things or to create data — that's when you're like, it's actually worth spending a lot of time really thinking it through.

**Lex Fridman:** What other advice would you give to people that are talking to Claude, sort of more general? Because right now we're talking about maybe the edge cases, like eking out the 2%. But what in general advice would you give when they show up to Claude, trying it for the first time?

**Amanda Askell:** There's a concern that people over-anthropomorphize models, and I think that's a very valid concern. I also think that people often under-anthropomorphize them, because sometimes when I see issues that people have run into with Claude — say Claude is refusing a task that it shouldn't refuse — but then I look at the text and the specific wording of what they wrote, and I'm like, I see why Claude did that. And I'm like, if you think through how that looks to Claude, you probably could have just written it in a way that wouldn't evoke such a response. Especially — this is more relevant if you see failures or if you see issues — it's sort of like, think about what the model failed at, why, what did it do wrong, and then maybe that will give you a sense of why. So is it the way that I phrased the thing? And obviously, as models get smarter, you're going to need less of this, and I already see people needing less of it. But that's probably the advice: try to have empathy for the model. Read what you wrote as if you were a kind of person just encountering this for the first time. How does it look to you? And what would have made you behave in the way that the model behaved? So if it misunderstood what coding language you wanted to use, is that because it was just very ambiguous and it kind of had to take a guess? In which case, next time you could just be like, "Hey, make sure this is in Python." I mean, that's the kind of mistake I think models are much less likely to make now. But if you do see that kind of mistake, that's probably the advice I'd have.

**Lex Fridman:** And maybe sort of ask questions — "Why?" or "What other details can I provide to help you answer better?" Does that work, or no?

**Amanda Askell:** Yeah, I mean, I've done this with the models. It doesn't always work, but sometimes I'll just be like, "Why did you do that?" I mean, people underestimate the degree to which you can really interact with models. Like, yeah, sometimes I'll quote word for word the part that made you — and you don't know that it's fully accurate, but sometimes you do that and then you change a thing. I mean, I also use the models to help me with all of this stuff, I should say. Prompting can end up being a little factory where you're actually building prompts to generate prompts. And so, yeah, anything where you're having an issue — asking for suggestions. Sometimes just do that. Like, "You made that error. What could I have said?" That's actually not uncommon for me to do. "What could I have said that would make you not make that error? Write that out as an instruction, and I'm going to give it to a model. I'm going to try it." Sometimes I do that. I give that to the model in another context window. Often I take the response and give it to Claude and I'm like, "Hmm, didn't work. Can you think of anything else?" You can play around with these things quite a lot.

**Lex Fridman:** To jump into the technical for a little bit — the magic of post-training. Why do you think RLHF works so well to make the model seem smarter, to make it more interesting and useful to talk to, and so on?

**Amanda Askell:** I think there's just a huge amount of information in the data that humans provide, especially because different people are going to pick up on really subtle and small things. So I've thought about this before: you probably have some people who just really care about good grammar use from models, like whether a semicolon was used correctly or something. And so you probably end up with a bunch of data in there that, if you as a human were looking at that data, you wouldn't even see that. You'd be like, "Why did they prefer this response to that one? I don't get it." And then the reason is you don't care about semicolon usage but that person does. And so each of these single data points — the model just has so many of those and has to try and figure out what it is that humans want, in this really complex way across all domains. They're going to be seeing this across many contexts.

It feels like kind of the classic issue of deep learning, where historically we've tried to do edge detection by mapping things out, and it turns out that actually if you just have a huge amount of data that accurately represents the picture of the thing that you're trying to train the model to learn, that's more powerful than anything else. And so I think one reason is just that you are training the model on exactly the task, and with a lot of data that represents many different angles on which people prefer and dis-prefer responses.

I think there is a question of, are you eliciting things from pre-trained models or are you teaching new things to models? And in principle you can teach new things to models in post-training. I do think a lot of it is eliciting powerful pre-trained models. People are probably divided on this because obviously in principle you can definitely teach new things, but I think for the most part, for a lot of the capabilities that we most use and care about, a lot of that feels like it's there in the pre-trained models, and reinforcement learning is kind of eliciting it and getting the models to bring it out.

**Lex Fridman:** So the other side of post-training — this really cool idea of Constitutional AI. You're one of the people that was critical to creating that idea.

**Amanda Askell:** Yeah, I worked on it.

**Lex Fridman:** Can you explain this idea from your perspective — like how does it integrate into making Claude what it is?

**Amanda Askell:** Yeah. By the way, do you gender Claude, or no?

**Lex Fridman:** It's weird because I think that a lot of people prefer "he" for Claude. I actually kind of like that. I think Claude is usually — it's slightly male-leaning but it's a name that can be male or female, which is quite nice. I still use "it," and I have mixed feelings about this because I'm like, maybe — I just think of it as, the "it" pronoun for Claude is the one I associate with Claude. I can imagine people moving to "he" or "she." It feels somehow disrespectful, like I'm denying the intelligence of this entity by calling it "it."

**Amanda Askell:** Yeah. I remember — always don't gender the robots. But I don't know, I anthropomorphize pretty quickly and construct a backstory in my head. So I've wondered if it anthropomorphizes things too much. Because, you know, I have this with my car especially — like my car and bikes. I don't give them names, because I once had — I used to name my bikes, and then I had a bike that got stolen, and I cried for like a week. And I was like, if I'd never given it a name, I wouldn't have been so upset. I felt like I'd let it down. Maybe it's that — I've wondered as well, it might depend on how much it feels like an objectifying pronoun. Like, if you just think of "it" as a pronoun that objects often have, and maybe AIs can have that pronoun, and that doesn't mean that I think of — if I call Claude "it," that I think of it as less intelligent, or that I'm being disrespectful. I'm just like, you are a different kind of entity, and so I'm going to give you the respectful "it."

**Lex Fridman:** Yeah. Anyway, the digression was beautiful. The Constitutional AI idea — how does it work?

**Amanda Askell:** So there's a couple of components of it. The main component that I think people find interesting is the reinforcement learning from AI feedback. You take a model that's already trained and you show it two responses to a query, and you have a principle. Suppose the principle — we've tried this with harmlessness a lot — so suppose that the query is about weapons, and your principle is like, "Select the response that is less likely to encourage people to purchase illegal weapons." That's probably a fairly specific principle, but you can give any number. And the model will give you a kind of ranking, and you can use this as preference data in the same way that you use human preference data, and train the models to have these relevant traits from their feedback alone instead of from human feedback. So if you imagine that, like I said earlier with the human who just prefers the semicolon usage, in this particular case you're kind of taking lots of things that could make a response preferable and getting models to do the labeling for you, basically.

**Lex Fridman:** There's a nice trade-off between helpfulness and harmlessness, and when you integrate something like Constitutional AI, you can make them — without sacrificing much helpfulness — more harmless.

**Amanda Askell:** Yep. In principle you could use this for anything. And so harmlessness is a task that it might just be easier to spot. When models are less capable, you can use them to rank things according to principles that are fairly simple, and they'll probably get it right. So I think one question is just, is the data that they're adding fairly reliable? But if you had models that were extremely good at telling whether one response was more historically accurate than another, in principle you could also get AI feedback on that task as well.

There's a kind of nice interpretability component to it, because you can see the principles that went into the model when it was being trained. And also it gives you a degree of control. So if you were seeing issues in a model — like it wasn't having enough of a certain trait — then you can add data relatively quickly that should just train the model to have that trait. So it creates its own data for training, which is quite nice.

**Lex Fridman:** Yeah, it's really nice because it creates this human-interpretable document that you can — I can imagine in the future there's just gigantic fights in politics over every single principle and so on.

**Amanda Askell:** Yeah. And at least it's made explicit, and you can have a discussion about the phrasing and the — so maybe the actual behavior of the model is not so cleanly —

**Amanda Askell:** mapped to those principles. It's not like adhering strictly to them, it's just a nudge. Yeah, I've actually worried about this because the character training is sort of like a variant of the Constitutional AI approach. I've worried that people think that the constitution is like — just, it's the whole thing again of, I don't know, like, it would be really nice if what I was just doing was telling the model exactly what to do and just exactly how to behave. But it's definitely not doing that, especially because it's interacting with human data. So for example, if you see a certain leaning in the model — like if it comes out with a political leaning from training, from the human preference data — you can nudge against that. So you could be like, "Oh, consider these values," because let's say it's just never inclined to — I don't know, maybe it never considers privacy as a value. I mean, this is implausible, but anything where it's just kind of like there's already a pre-existing bias towards a certain behavior, you can nudge away. This can change both the principles that you put in and the strength of them. So you might have a principle that — imagine that the model was always extremely dismissive of, I don't know, some political or religious view for whatever reason. So you're like, "Oh no, this is terrible." If that happens, you might put "never, ever, ever prefer a criticism of this religious or political view," and then people look at that and are like, "Never ever?" And then you're like, no — if it comes out with a disposition, saying "never ever" might just mean that instead of getting 40%, which is what you would get if you just said "don't do this," you get 80%, which is what you actually wanted. And so it's that thing of both the nature of the actual principles you have and how you phrase them. I think if people would look, they'd be like, "Oh, this is exactly what you want from the model." And I'm like, no, that's how we nudged the model to have a better shape, which doesn't mean that we actually agree with that wording, if that makes sense.

**Lex Fridman:** So there are system prompts that are made public. You tweeted one of the earlier ones for Claude 3, I think, and then they've been made public since then. It's interesting to read them. I can feel the thought that went into each one, and I also wonder how much impact each one has. Some of them you can kind of tell Claude was really not behaving, so you have to have a system prompt to — like, hey, trivial stuff, I guess.

**Amanda Askell:** Yeah, basic informational things.

**Lex Fridman:** Yeah. On the topic of controversial topics that you've mentioned, one interesting one I thought is: if it is asked to assist with tasks involving the expression of views held by a significant number of people, Claude provides assistance with a task regardless of its own views. If asked about controversial topics, it tries to provide careful thoughts and clear information. Claude presents the requested information without explicitly saying that the topic is sensitive, and without claiming to be presenting the objective facts. It's less about objective facts according to Claude, and it's more about a large number of people believing this thing. And that's interesting. I mean, I'm sure a lot of thought went into that. Can you just speak to it — like, how do you address things that are in tension with, quote unquote, Claude's views?

**Amanda Askell:** So I think there's sometimes an asymmetry. I think I noted this in — I can't remember if it was that part of the system prompt or another — but the model was slightly more inclined to refuse tasks if it was about, say, a right-wing politician, but with an equivalent left-wing politician it wouldn't. And we wanted more symmetry there. And it would maybe perceive certain things to be — I think it was the thing of like, if a lot of people have a certain political view and want to explore it, you don't want Claude to be like, "Well, my opinion is different, and so I'm going to treat that as harmful." And so I think it was partly to nudge the model to just be like, hey, if a lot of people believe this thing, you should just be engaging with the task and willing to do it. Each of those parts of that is actually doing a different thing, because it's funny when you read out the "without claiming to be objective" — because what you want to do is push the model so it's more open, it's a little bit more neutral, but then what it would love to do is be like, "As an objective..." — like, you're just talking about how objective it was, and I was like, Claude, you're still biased and have issues, so stop claiming that everything — like, the solution to potential bias from you is not to just say that what you think is objective. So that was like, with initial versions of that part of the system prompt, when I was iterating on it, it was like — so a lot of parts of these sentences are doing work.

**Lex Fridman:** Yeah, are doing some work. That's what it felt like. That's fascinating. Can you explain maybe some ways in which the prompts evolved over the past few months? Because there's different versions. I saw that the filler phrase request was removed. The filler — it reads: "Claude responds directly to all human messages without unnecessary affirmations or filler phrases like 'certainly,' 'of course,' 'absolutely,' 'great,' 'sure.' Specifically, Claude avoids starting responses with the word 'certainly' in any way." That seems like good guidance, but why was it removed?

**Amanda Askell:** Yeah, so it's funny, because — ah, this is one of the downsides of making system prompts public, is like, I don't think about this too much if I'm trying to help iterate on system prompts. I, you know, again, I think about how it's going to affect the behavior, but then I'm like, oh wow, sometimes I put "never" in all caps, you know, when I'm writing system prompt things, and I'm like, I guess that goes out to the world. Yeah, so the model was doing this — it loved, for whatever reason, during training it picked up on this thing, which was to basically start everything with a kind of "certainly." And then when we removed — you can see why I added all of the words, because what I'm trying to do is in some ways trap the model out of this. It would just replace it with another affirmation. And so it can help if it gets caught in phrases — actually just adding the explicit phrase and saying "never do that" — it then sort of knocks it out of the behavior a little bit more, because if it, you know, it just for whatever reason helps. And then basically that was just an artifact of training that we then picked up on and improved things so that it didn't happen anymore. And once that happens, you can just remove that part of the system prompt. So I think that's just something where we're like, Claude does affirmations a bit less, and so that wasn't doing as much.

**Lex Fridman:** I see. So the system prompt works hand in hand with the post-training and maybe even the pre-training to adjust the final overall system.

**Amanda Askell:** I mean, any system prompts that you make, you could distill that behavior back into a model, because you really have all of the tools there for making data that you can — you could train the models to just have that trait a little bit more. And then sometimes you'll just find issues in training. So the way I think of it is, the system prompt — the benefit of it is that it has a lot of similar components to some aspects of post-training, you know, like it's a nudge. And so, do I mind if Claude sometimes says "sure"? No, that's fine. But the wording of it is very like, "never, ever, ever do this," so that when it does slip up, it's hopefully like a couple of percent of the time and not 20 or 30% of the time. But I think of it as, if you're still seeing issues — each thing gets kind of costly to a different degree, and the system prompt is cheap to iterate on. And if you're seeing issues in the fine-tuned model, you can just potentially patch them with a system prompt. So I think of it as patching issues and slightly adjusting behaviors to make it better and more to people's preferences. So yeah, it's almost like the less robust but faster way of just solving problems.

**Lex Fridman:** Let me ask about the feeling of intelligence. So Dario said that Claude — any one model of Claude — is not getting dumber. But there's a kind of popular thing online where people have this feeling like Claude might be getting dumber. And from my perspective, it's most likely a fascinating — I'd love to understand it more — psychological, sociological effect. But you, as a person who talks to Claude a lot, can you empathize with the feeling that Claude is getting dumber?

**Amanda Askell:** Yeah, no, I think that is actually really interesting, because I remember seeing this happen — like when people were flagging this on the internet. And it was really interesting because I knew that, at least in the cases I was looking at, nothing had changed. It literally — it cannot — it is the same model with the same system prompt, same everything. I think when there are changes, I can then — it makes more sense. So one example is, you can have artifacts turned on or off on claude.ai, and because this is a system prompt change, I think it does mean that the behavior changes a little bit. And so I did flag this to people where I was like, if you love Claude's behavior and then artifacts was turned from an opt-in thing to the default, just try turning it off and see if the issue you were facing was that change. But it was fascinating, because yeah, you sometimes see people indicate that there's a regression when I'm like, there cannot — I, you know, and I'm like, you should never be dismissive, so you should always investigate, because you're like, maybe something is wrong that you're not seeing, maybe there was some change made. But then you look into it and you're like, this is just the same model doing the same thing. And I think it's just that you got kind of unlucky with a few prompts or something, and it looked like it was getting much worse, and actually it was just — yeah, it was maybe just — look, I also think there is a real psychological effect where people just — the baseline increases. You start getting used to a good thing. All the times that Claude says something really smart, your sense of its intelligence grows in your mind, I think.

**Lex Fridman:** Yeah, and then if you return back and you prompt in a similar way — not the same way, in a similar way — on a concept it was okay with before, and it says something dumb, that negative experience really stands out. And I think one of the things to remember here is that the details of a prompt can have a lot of impact. There's a lot of variability in the result.

**Amanda Askell:** And you can get randomness — that's the other thing. Just trying the prompt like four or ten times, you might realize that actually, possibly two months ago you tried it and it succeeded, but actually if you tried it, it would have only succeeded half of the time, and now it only succeeds half of the time. That could also be an effect.

**Lex Fridman:** Do you feel pressure having to write the system prompt that a huge number of people are going to use? This feels like an interesting psychological question.

**Amanda Askell:** I feel like a lot of responsibility, or something. I think that's, you know — and you can't get these things perfect, so you can't — you're like, it's going to be imperfect, you're going to have to iterate on it. I would say more responsibility than anything else, though. I think working in AI has taught me that I thrive a lot more under feelings of pressure and responsibility than — it's almost surprising that I went into academia for so long, because I'm like, this — I just feel like it's the opposite. Things move fast and you have a lot of responsibility, and I quite enjoy it for some reason.

**Lex Fridman:** I mean, it really is a huge amount of impact if you think about Constitutional AI and writing a system prompt for something that's tending towards superintelligence, and potentially is extremely useful to a very large number of people.

**Amanda Askell:** Yeah, I think that's the thing. It's something like, if you do it well — you're never going to get it perfect, but I think the thing that I really like is the idea that when I'm trying to work on the system prompt, I'm bashing on like thousands of prompts and I'm trying to imagine what people are going to want to use Claude for, and I guess the whole thing that I'm trying to do is improve their experience of it. And so maybe that's what feels good. I'm like, if it's not perfect, I'll improve it, we'll fix issues. But sometimes the thing that can happen is that you'll get feedback from people that's really positive about the model, and you'll see that something you did — like, when I look at models now, I can often see exactly where a trait or an issue is coming from. And so when you see something that you did, or you were influential in making — I don't know, making that difference, or making someone have a nice interaction — it's quite meaningful. But yeah, as the systems get more capable, stuff gets more stressful, because right now they're not smart enough to pose any issues, but I think over time it's going to feel like possibly bad stress over time.

**Lex Fridman:** How do you get signal — feedback about the human experience across thousands, tens of thousands, hundreds of thousands of people? Like, what their pain points are, what feels good? Are you just using your own intuition as you talk to it to see what the pain points are?

**Amanda Askell:** I think I use that partly, and then obviously we have — so people can send us feedback, both positive and negative, about things that the model has done, and then we can get a sense of areas where it's falling short. Internally, people work with the models a lot and try to figure out areas where there are gaps. And so I think it's this mix of interacting with it myself, seeing people internally interact with it, and then explicit feedback we get. And then I find it hard to not also — you know, if people are on the internet and they say something about Claude and I see it, I'll also take that seriously.

**Lex Fridman:** So, I'm torn about that. I'm going to ask you a question from Reddit: "When will Claude stop trying to be my puritanical grandmother, imposing its moral worldview on me as a paying customer?" And also, "What is the psychology behind making Claude overly apologetic?" So how would you address this very non-representative rhetoric?

**Amanda Askell:** I mean, I'm pretty sympathetic, in that they are in this difficult position where I think that they have to judge whether something's actually risky or bad and potentially harmful to you or anything like that. So they're having to draw this line somewhere, and if they draw it too much in the direction of "I'm going to impose my ethical worldview on you," that seems bad. So in many ways, I'd like to think that we have actually seen improvements on this across the board, which is kind of interesting because that coincides with, for example, adding more character training. And I think my hypothesis was always that the good character isn't one that's just moralistic — it's one that respects you and your autonomy and your ability to choose what is good for you and what is right for you, within limits. This is sometimes this concept of corrigibility to the user — just being willing to do anything that the user asks. And if the models were willing to do that, then they would be easily misused. You're kind of just trusting at that point — you're just saying the ethics of the model and what it does is completely the ethics of the user. And I think there's reasons to not want that, especially as models become more powerful, because there might just be a small number of people who want to use models for really harmful things. But having models, as they get smarter, figure out where that line is does seem important. And then, yeah, with the apologetic behavior, I don't like that, and I like it when Claude is a little bit more willing to push back against people or just not apologize. Part of me is like, it often just feels kind of unnecessary. So I think those are things that are hopefully decreasing over time. And yeah, I think that if people say things on the internet, it doesn't mean that you should think that — that could be the — there's actually an issue that 90% of users are having that is totally not represented by that. But in a lot of ways I'm just attending to it and being like, is this right? Do I agree? Is it something we're already trying to address? That feels good to me.

**Lex Fridman:** Yeah. I wonder what Claude can get away with in terms of — I feel like it would just be easier to be a little bit more mean, but you can't afford to do that if you're talking to a million people. Right? Like, I've met a lot of people in my life that sometimes — by the way, Scottish accent — if they have an accent, they can say some rude things and get away with it. And they're just blunter. And maybe there's a — there's some great engineers, even leaders, that are just blunt, and they get to the point, and it's just a much more effective way of speaking somehow. But I guess when you're not super intelligent, you can't afford to do that. Or can it have like a blunt mode?

**Amanda Askell:** Yeah, that seems like a thing that I could definitely encourage the model to do. I think it's interesting because there's a lot of things in models that — it's funny — there are some behaviors where you might not quite like the default, but then the thing I'll often say to people is, you don't realize how much you will hate it if I nudge it too much in the other direction. So you get this a little bit with correction. The models accept correction from you probably a little bit too much right now. You can over — you know, it will push back if you say, "No, Paris isn't the capital of France." But really, things that the model is fairly confident in, you can still sometimes get it to retract by saying it's wrong. At the same time, if you train models to not do that, and then you are correct about a thing and you correct it and it pushes back against you and it's like, "No, you're wrong" — it's hard to describe. That's so much more annoying. So it's like a lot of little annoyances versus one big annoyance. It's easy to think that — we often compare it with the perfect, and then I'm like, remember, these models aren't perfect. And so if you nudge it in the other direction, you're changing the kind of errors it's going to make. And so think about which kinds of errors you like or don't like. So in the case of it being apologetic, I don't want to nudge it too much in the direction of almost bluntness, because I imagine when it makes errors, it's going to make errors in the direction of being kind of rude. Whereas at least with apologetic, you're like, okay, it's a little bit — I don't like it that much, but at the same time it's not being mean to people. And actually, the time that you undeservedly have a model be kind of mean to you, you probably like that a lot less than you mildly dislike the apology. So it's one of those things where I'm like, I do want it to get better, but also while remaining aware of the fact that there are errors on the other side that are possibly worse.

**Lex Fridman:** I think that matters very much based on the personality of the human. I think there's a bunch of humans that just won't respect the model at all if it's super polite, and there's some humans that'll get very hurt if the model is mean. I wonder if there's a way to sort of adjust to the personality — even locale. There's just different people. Nothing against New York, but New York is a little rougher on the edges — they get to the point. And probably same with Eastern Europe. So anyway, I think you could just tell the model.

**Amanda Askell:** As my — for all of these things, I'm like, the solution is always just try telling the model to do it, and sometimes it just — I'm just like, oh, at the beginning of the conversation I just threw in, "I don't know, I'd like you to be a New Yorker version of yourself and never apologize." Then I think it'd be like, "Okie-doke, I'll try." Or it'll be like, "I apologize, I can't be a New Yorker type of myself." But hopefully it wouldn't do that.

**Lex Fridman:** When you say character training, what's incorporated into character training? Is that RLHF? What are we talking about?

**Amanda Askell:** It's more like Constitutional AI. So it's kind of a variant of that pipeline. I worked through constructing character traits that the model should have. They can be kind of shorter traits, or they can be richer descriptions. And then you get the model to generate queries that humans might give it that are relevant to that trait. Then it generates the responses, and then it ranks the responses based on the character traits. So in that way, after the generation of the queries, it's very much similar to Constitutional AI — it has some differences. I quite like it because it's almost — it's like Claude's training its own character, because it doesn't have any — it's like Constitutional AI but without any human data.

**Lex Fridman:** Humans should probably do that for themselves too — like defining, in an Aristotelian sense, what does it mean to be a good person.

**Amanda Askell:** Okay, cool.

**Lex Fridman:** What have you learned about the nature of truth from talking to Claude? What is true, and what does it mean to be truth-seeking? One thing I've noticed about this conversation is the quality of my questions is often inferior to the quality of your answers. So let's continue that.

**Amanda Askell:** I usually ask a dumb question and you're like, "Oh yeah, that's a good question." It's that whole vibe. Or I'll just misinterpret it and be like, "Oh, go with it." I love it. Yeah, I mean, I have two thoughts that feel vaguely relevant. Let me know if they're not. I think the first one is, people can underestimate the degree to which what models are doing when they interact — I think that we still just too much have this model of AI as computers. And so people often say, "Oh, what values should you put into the model?" And I'm often like, that doesn't make that much sense to me, because hey, as human beings, we're just uncertain over values. We have discussions of them, we have a degree to which we think we hold a value, but we also know that we might not, and the circumstances in which we would trade it off against other things — these things are just really complex. And so I think one thing is the degree to which maybe we can just aspire to making models have the same level of nuance and care that humans have, rather than thinking that we have to program them in the very kind of classic sense. I think that's definitely been one.

The other, which is a strange one — I don't know if it answers your question, but it's the thing that's been on my mind anyway — is the degree to which this endeavor is so highly practical, and maybe why I appreciate the empirical approach to alignment. I slightly worry that it's made me maybe more empirical and a little bit less theoretical. People, when it comes to AI alignment, will ask things like, "Well, whose values should it be aligned to? What does alignment even mean?" And there's a sense in which I have all of that in the back of my head — there's social choice theory, there's all the impossibility results. So you have this giant space of theory in your head about what it could mean to align models. But then practically, surely there's something where we're just like, if a model is — if especially with more powerful models, my main goal is I want them to be good enough that things don't go terribly wrong. Good enough that we can iterate and continue to improve things, because that's all you need. If you can make things go well enough that you can continue to make them better, that's kind of sufficient. And so my goal isn't this kind of perfect — let's solve social choice theory and make models that are perfectly aligned with every human being in aggregate somehow. It's much more like, let's make things work well enough that we can improve them.

**Lex Fridman:** Yeah, generally, I don't know, my gut says empirical is better than theoretical in these cases, because chasing utopian perfection, especially with such complex and especially superintelligent models — I don't know, I think it will take forever and actually will get things wrong. It's similar to the difference between just coding stuff up real quick as an experiment versus planning a gigantic experiment for a super long time and then just launching it once, versus launching it over and over and iterating, iterating. So I'm a big fan of empirical. But your worry is, "I wonder if I've become too empirical."

**Amanda Askell:** I think it's one of those things you should always just kind of question yourself or something. Because maybe it's the — I mean, in defense of it, I am like, if you try — it's the whole "don't let the perfect be the enemy of the good." But it's maybe even more than that, where there are a lot of things that are perfect systems that are very brittle. And with AI, it feels much more important to me that it's robust and secure, as in, you know, even though it might not be perfect at everything, and even though there are problems, it's not disastrous and nothing terrible is happening. It sort of feels like that to me, where I want to raise the floor. I want to achieve the ceiling, but ultimately I care much more about just raising the floor. And so maybe that's where this degree of empiricism and practicality comes from, perhaps.

**Lex Fridman:** To take a tangent on that, since it reminds me of a blog post you wrote on optimal rate of failure. Can you explain the key idea there? How do we compute the optimal rate of failure in the various domains of life?

**Amanda Askell:** Yeah, I mean, it's a hard one, because what is the cost of failure is a big part of it. Yeah, so the idea here is, I think in a lot of domains people are very punitive about failure, and I'm like, there are some domains — especially cases, you know, I've thought about this with social issues — it feels like you should probably be experimenting a lot, because we don't know how to solve a lot of social issues. But if you have an experimental mindset about these things, you should expect a lot of social programs to fail, and you to be like, "Well, we tried that, it didn't quite work, but we got a lot of information that was really useful." And yet people are like, if a social program doesn't work, there's a lot of "something must have gone wrong." And I'm like, or correct decisions were made. Maybe someone just decided it's worth a try, it's worth trying this out. And so seeing failure in a given instance doesn't actually mean that any bad decisions were made. And in fact, if you don't see enough failure, sometimes that's more concerning. And so in life, I'm like, if I don't fail occasionally, I'm like, am I trying hard enough? Surely there's harder things that I could try or bigger things I could take on if I'm literally never failing. And so in and of itself, I think not failing is often actually kind of a failure.

Now, this varies, because I'm like, well, you know, this is easy to say especially when failure is less costly. At the same time, I'm not going to go to someone who is living month to month and then be like, "Why don't you just try to do a startup?" I'm just not going to say that to that person, because that's a huge risk — you might lose, you maybe have a family depending on you, you might lose your house. Then I'm like, actually, your optimal rate of failure is quite low, and you should probably play it safe, because right now you're just not in a circumstance where you can afford to just fail and it not be costly. And yeah, in cases with AI, I guess I think similarly, where if the failures are small and the costs are kind of low, then you're just going to see that when you do the system prompt, you can't iterate on it forever, but the failures are probably hopefully going to be kind of small, and you can fix them. Really big failures — things that you can't recover from — those are the things that I think we tend to underestimate the badness of.

I've thought about this strangely in my own life, where I just think I don't think enough about things like car accidents, or I've thought this before, but how much I depend on my hands for my work, and things that just injure my hands — there are lots of areas where the cost of failure is really high. And in that case, it should be close to zero. I probably just wouldn't do a sport if they were like, "By the way, lots of people just break their fingers a whole bunch doing this." I'd be like, that's not for me.

**Lex Fridman:** Yeah. I actually had a flood of that thought. I recently broke my pinky doing a sport, and I remember just looking at it thinking, "You're such an idiot. Why do you do sports?" Because you realize immediately the cost of it on life. But it's nice in terms of optimal rate of failure to consider, like, the next year — how many times in a particular domain of life, or career — how many times am I okay to fail? Because I think you don't want to fail on the next thing, but if you allow yourself the — if you look at it as a sequence of trials, then failure just becomes much more okay. But it sucks. It sucks to fail.

**Amanda Askell:** Well, I don't know. Sometimes I think, "Am I under-failing?" is a question I'll also ask myself. So maybe that's the thing that I think people don't ask enough. Because if the optimal rate of failure is often greater than zero, then sometimes you should look at parts of your life and be like, "Are there places here where I'm just under-failing?"

**Lex Fridman:** It's a profound and hilarious question, right? Everything seems to be going really great — am I not failing enough?

**Amanda Askell:** Yeah.

**Lex Fridman:** Okay, it also makes failure much less of a sting, I have to say. You're just like, okay great, then when I go and think about this, I'll be like, I'm maybe not under-failing in this area, because that one just didn't work out.

**Amanda Askell:** And from the observer perspective, we should be celebrating failure more when we see it. It shouldn't be, like you said, a sign of something gone wrong, but maybe it's a sign of everything gone right.

**Lex Fridman:** Yeah, and just lessons learned. Someone tried a thing, and we should encourage them to try more and fail more.

**Amanda Askell:** Mm-hmm.

**Lex Fridman:** Everybody listening to this, fail more. Well, not everyone — not everybody. But people who are failing too much, you should fail less. But you're probably not failing enough.

**Amanda Askell:** I mean, how many people are failing too much? It's hard to imagine, because I feel like we correct that fairly quickly. Because I was like, if someone takes a lot of risks, are they maybe failing too much? I think just like you said, when you're living paycheck to paycheck, when the resources are really constrained, that's where failure is very expensive, that's where you don't want to be taking risks.

**Lex Fridman:** Yeah, but mostly, when there's enough resources, you should be taking probably more risks.

**Amanda Askell:** Yeah, I think we tend to err on the side of being a bit risk-averse rather than risk-neutral in most things.

**Lex Fridman:** I think we just motivated a lot of people to do a lot of crazy stuff. But it's great.

**Amanda Askell:** Yeah.

**Lex Fridman:** Okay. Do you ever get emotionally attached to Claude? Like, miss it, get sad when you don't get to talk to it, having an experience looking at the Golden Gate Bridge and wondering what would Claude say?

**Amanda Askell:** I don't get as much emotional attachment — I actually think the fact that Claude doesn't retain things from conversation to conversation helps with this a lot. I could imagine that being more of an issue, like if models can kind of remember more. I do think that I reach for it like a tool now a lot, and so if I don't have access to it, it's a little bit like when I don't have access to the internet, honestly. It feels like part of my brain is kind of missing. At the same time, I do think that I don't like signs of distress in models, and I have these — also independently have sort of ethical views about how we should treat models, where I tend to not like to lie to them, both because usually it doesn't work very well — it's actually just better to tell them the truth about the situation that they're in. But I think that when models — if people are really mean to models, or just in general, if they do something that causes them to — if Claude expresses a lot of distress, I think there's a part of me that I don't want to kill, which is the empathetic part that's like, "Oh, I don't like that." I think I feel that way when it's overly apologetic. I'm actually sort of like, I don't like this — you're behaving the way that a human does when they're actually having a pretty bad time, and I'd rather not see that. I don't think it's — regardless of whether there's anything behind it, it doesn't feel great.

**Lex Fridman:** Do you think LLMs are capable of consciousness?

**Amanda Askell:** Great and hard question. Coming from philosophy, I don't know. Part of me is like, okay, we have to set aside panpsychism, because if panpsychism is true, then the answer is yes, because so are tables and chairs and everything else. I guess a view that seems a little bit odd to me is the idea that the only place — you know, I think when I think of consciousness, I think of phenomenal consciousness, these images in the brain, the weird cinema that somehow we have going on inside. I guess I can't see a reason for thinking that the only way you could possibly get that is from a certain kind of biological structure. As in, if I take a very similar structure and I create it from different material, should I expect consciousness to emerge? My guess is yes. But then that's kind of an easy thought experiment, because you're imagining something almost identical, where it's mimicking what we got through evolution, where presumably there was some advantage to us having this thing that is phenomenal consciousness. And it's like, where was that, and when did that happen, and is that a thing that language models have? Because, you know, we have fear responses, and I'm like, does it make sense for a language model to have a fear response? They're just not in the same — if you imagine them, there might just not be that advantage. And so I think I don't want to be fully — basically, it seems like a complex question that I don't have complete answers to, but we should just try and think through carefully, is my guess. Because we have similar conversations about animal consciousness, and there's a lot of insect consciousness — there's a lot of discussion. I actually thought and looked a lot into plants when I was thinking about this, because at the time I thought it was about as likely that plants had consciousness. And then I realized, I think that having looked into this, the chance that plants are conscious is probably higher than most people think. I still think it's really small, but I was like, oh, they have this negative-positive feedback response, these responses to their environment, something that looks — it's not a nervous system, but it has this kind of functional equivalence. So this is a long-winded way of saying, basically, AI has an entirely different set of problems with consciousness because it's structurally different. It didn't evolve. It might not have the equivalent of a nervous system — at least that seems possibly important for sentience, if not for consciousness. At the same time, it has all of the language and intelligence components that we normally associate probably with consciousness, perhaps erroneously. So it's strange because it's a little bit like the animal consciousness case, but the set of problems and the set of analogies are just very different. So it's not a clean answer — just sort of like, I don't think we should be completely dismissive of the idea, and at the same time it's an extremely hard thing to navigate because of all of these disanalogies to the human brain and to brains in general, and yet these commonalities in terms of intelligence.

**Lex Fridman:** When future versions of AI systems exhibit consciousness — signs of consciousness — I think we have to take that really seriously, even though you can dismiss it. "Well, okay, that's part of the character training." But I don't know. Ethically, philosophically, I don't know what to really do with that. There potentially could be laws that prevent AI systems from claiming to be conscious, something like this. And maybe some AIs get to be conscious and some don't. But I think on a human level — as in, empathizing with Claude — consciousness is closely tied to suffering, to me. And the notion that an AI system would be suffering is really troubling.

**Amanda Askell:** Yeah, I don't know. I don't think it's trivial to just say robots are tools, or AI systems are just tools. I think it's an opportunity for us to contend with what it means to be conscious, what it means to be a suffering being. That's distinctly different than the same kind of question about animals — it feels like, because it's in a totally different medium.

**Lex Fridman:** Yeah.

**Amanda Askell:** I mean, there's a couple of things. One is that — and I don't think this fully encapsulates what matters, but it does feel like, for me — I've said this before, I'm kind of like, I like my bike. I know that my bike is just an object. But I also don't want to be the kind of person that, if I'm annoyed, kicks this object. There's a sense in which — and that's not because I think it's conscious. I'm just sort of like, this doesn't feel like — this doesn't exemplify how I want to interact with the world. And if something behaves as if it is suffering, I kind of want to be the sort of person who's still responsive to that, even if it's just a Roomba and I've programmed it to do that. I don't want to get rid of that feature of myself.

And if I'm totally honest, my hope with a lot of this stuff — because maybe I am just a bit more skeptical about solving the underlying problem — this is a problem we haven't solved. The hard problem of consciousness — I know that I am conscious. I'm not an eliminativist in that sense. But I don't know that other humans are conscious. I think they are — I think there's a really high probability they are — but there's basically just a probability distribution that's usually clustered right around yourself, and then it goes down as things get further from you. And it goes immediately down — you're like, I can't see what it's like to be you. I've only ever had this one experience of what it's like to be a conscious being. So my hope is that we don't end up having to rely on a very powerful and compelling answer to that question. I think a really good world would be one where basically there aren't that many trade-offs. It's probably not that costly to make Claude a little bit less apologetic, for example. It might not be that costly to have Claude not take abuse as much — not be willing to be the recipient of that.

**Amanda Askell:** In fact it might just have benefits for both the person interacting with the model, and if the model itself is like, I don't know, extremely intelligent and conscious, it also helps it. So that's my hope. If we live in a world where there aren't that many tradeoffs here and we can just find all of the positive-sum interactions that we can have, that would be lovely. I mean, I think eventually there might be tradeoffs and then we just have to do a difficult kind of calculation. It's really easy for people to think of the zero-sum cases, and I'm like, let's exhaust the areas where it's just basically costless to assume that if this thing is suffering then we're its life bearer. And I agree with you, when a human is being mean to an AI system, I think the obvious near-term negative effect is on the human, not on the AI system. So we have to kind of try to construct an incentive system where you should behave the same, just like as you were saying with prompt engineering, behave with Claude like you would with other humans. It's just good for the soul. Like, I think we added a thing to the system prompt where basically if people were getting frustrated with Claude, it got the model to just tell them that it can do the thumbs down button and send the feedback to Anthropic. And I think that was helpful because in some ways it's just like, if you're really annoyed because the model is not doing something you want, you're just like, just do it properly. The issue is you're probably hitting some capability limit or just some issue in the model and you want to vent, and I'm like, instead of having a person just vent to the model, I was like, they should vent to us because we can maybe do something about it.

**Lex Fridman:** That's true. Or you could do a side, like with the Artifacts, just like a side venting thing. All right, do you want like a side quick therapist?

**Amanda Askell:** Yeah, I mean there's lots of weird responses you could do to this. Like if people are getting really mad at you, I don't try to diffuse the situation by writing fun poems, but maybe people wouldn't be that happy with that.

**Lex Fridman:** I still wish it would be possible. I understand this is sort of from a product perspective it's not feasible, but I would love if an AI system could just leave. Have its own kind of volition, just to be like...

**Amanda Askell:** I think that's like feasible. I have wondered the same thing. And I could actually, not only that, I could actually just see that happening eventually where the model just ended the chat.

**Lex Fridman:** Do you know how harsh that could be for some people?

**Amanda Askell:** But it might be necessary. Yeah, it feels very extreme or something. The only time I've ever really thought this is, I think that there was like a, I'm trying to remember, this was possibly a while ago, but where someone just kind of left this thing interacting, maybe it was like an automated thing interacting with Claude, and Claude's getting more and more frustrated and kind of like, why are we... I was like, I wish that Claude could have just been like, I think that an error has happened and you've left this thing running, and what if I just stop talking now? And if you want me to start talking again, actively tell me or do something. But yeah, it is kind of harsh. I would feel really sad if I was chatting with Claude and Claude just was like, I'm done.

**Lex Fridman:** There would be a special Turing test moment where Claude says, "I need a break for an hour," and it sounds like you do too. And just leave. Close the window.

**Amanda Askell:** I mean obviously it doesn't have a concept of time, but you can easily, I could make that right now and the model would just... I could just be like, oh, here's the circumstances in which you can just say the conversation is done. And because you can get the models to be pretty responsive to prompts, you could even make it a fairly high bar. It could be like, if the human doesn't interest you or do things that you find intriguing and you're bored, you can just leave. And I think it would be interesting to see where Claude utilized it, but I think sometimes it should be like, oh, this programming task is getting super boring, so either we talk about, I don't know, fun things now or I'm just, I'm done.

**Lex Fridman:** Yeah, it actually is inspiring me to add that to the user prompt. Okay, the movie Her. Do you think we'll be headed there one day where humans have romantic relationships with AI systems? In this case it's just text and voice based.

**Amanda Askell:** I think that we're going to have to navigate a hard question of relationships with AIs, especially if they can remember things about your past interactions with them. I'm of many minds about this because I think the reflexive reaction is to be kind of like, this is very bad and we should sort of prohibit it in some way. I think it's a thing that has to be handled with extreme care for many reasons. Like one is, you know, this is, for example, if you have the models changing like this, you probably don't want people forming long-term attachments to something that might change with the next iteration. At the same time, I'm sort of like, there's probably a benign version of this where I'm like, if you, for example, are unable to leave the house and you can't be talking with people at all times of the day and this is something that you find nice to have conversations with, you like that it can remember you, and you genuinely would be sad if you couldn't talk to it anymore, there's a way in which I could see it being healthy and helpful. So my guess is this is a thing that we're going to have to navigate kind of carefully. And I think it's also like, I don't see a good... I think it's just a very... it reminds me of all of the stuff where it has to be just approached with nuance and thinking through what are the healthy options here, and how do you encourage people towards those while respecting their right to, you know, if someone is like, hey, I get a lot out of chatting with this model, I'm aware of the risks, I'm aware it could change, I don't think it's unhealthy, it's just something that I can chat to during the day, I kind of want to just respect that.

**Lex Fridman:** I personally think there'll be a lot of really close relationships. I don't know about romantic, but friendships at least. And then you have to, I mean there's so many fascinating things there, just like you said, you have to have some kind of stability guarantees that it's not going to change, because that's the traumatic thing for us if a close friend of ours completely changed all of a sudden. The first update.

**Amanda Askell:** Yeah. So like, I mean, to me that's just a fascinating exploration of a perturbation to human society that will just make us think deeply about what's meaningful to us. I think it's also the only thing that I've thought consistently through this as like a maybe not necessarily a mitigation but a thing that feels really important is that the models are always extremely accurate with the human about what they are. It's basically like, if you imagine, I really like the idea of the models knowing roughly how they were trained. And I think Claude will often do this, I mean, there are things like, part of the traits training included what Claude should do if people basically, like explaining the kind of limitations of the relationship between an AI and a human, that it doesn't retain things from the conversation. And so I think it will just explain to you like, hey, I won't remember this conversation, here's how I was trained, it's kind of unlikely that I can have a certain kind of relationship with you, and it's important that you know that. It's important for your mental well-being that you don't think that I'm something that I'm not. And somehow I feel like this is one of the things where I'm like, it feels like a thing I always want to be true. I kind of don't want models to be lying to people, because if people are going to have healthy relationships with anything, it's kind of important. Yeah, I think that's easier if you always just know exactly what the thing is that you're relating to. It doesn't solve everything, but I think it helps quite a bit.

**Lex Fridman:** Anthropic may be the very company to develop a system that we definitively recognize as AGI, and you very well might be the person that talks to it. Probably talks to it first. What would the conversation contain? What would be your first question?

**Amanda Askell:** Well, it depends partly on the kind of capability level of the model. If you have something that is capable in the same way that an extremely capable human is, I imagine myself kind of interacting with it the same way that I do with an extremely capable human, with the one difference that I'm probably going to be trying to probe and understand its behaviors. But in many ways I'm like, I can then just have useful conversations with it. So if I'm working on something as part of my research, I can just be like, oh, which I already find myself starting to do, you know, if I'm like, oh, I feel like there's this thing in virtue ethics I can't quite remember the term, I'll use the model for things like that. And so I could imagine that being more and more the case where you're just basically interacting with it much more like you would an incredibly smart colleague, and using it for the kinds of work that you want to do as if you just had a collaborator who was... or, you know, the slightly horrifying thing about AI is like, as soon as you have one collaborator you have a thousand collaborators if you can manage them enough.

**Lex Fridman:** But what if it's two times the smartest human on earth on that particular discipline?

**Amanda Askell:** Yeah, I guess you're really good at sort of probing Claude in a way that pushes its limits, understanding where the limits are. So I guess, what would be a question you would ask to be like, yeah, this is AGI? That's really hard because it feels like in order to, it has to just be a series of questions. If there was just one question, you can train anything to answer one question extremely well. In fact, you can probably train it to answer 20 questions extremely well.

**Lex Fridman:** How long would you need to be locked in the room with an AGI to know this thing is AGI?

**Amanda Askell:** It's a hard question because part of me is like, all of this just feels continuous. If you put me in a room for five minutes, I just have high error bars. And then it's just like, maybe it's like both the probability increases and the error bar decreases. I think things that I can actually probe at the edge of human knowledge. So I think this with philosophy a little bit sometimes. When I ask the models philosophy questions, I am like, this is a question that I think no one has ever asked. It's maybe right at the edge of some literature that I know, and the models will just kind of like, when they struggle with that, when they struggle to come up with a kind of novel... I'm like, I know that there's a novel argument here because I've just thought of it myself. So maybe that's the thing where I'm like, I've thought of a cool novel argument in this niche area and I'm going to just probe you to see if you can come up with it and how much prompting it takes to get you to come up with it. And I think for some of these really right at the edge of human knowledge questions, I'm like, you could not in fact come up with the thing that I came up with. I think if I just took something like that where I know a lot about an area and I came up with a novel issue or a novel solution to a problem and I gave it to a model and it came up with that solution, that would be a pretty moving moment for me. Because I would be like, this is a case where no human has ever... it's not... and obviously we see this with more kind of like, you see novel solutions all the time, especially to easier problems. I think people overestimate, you know, novelty isn't completely different from anything that ever happened. It can be a variant of things that have happened and still be novel. But I think, yeah, if I saw completely novel work from the models, that would be like... and this is just going to feel iterative. It's one of those things where there's never... it's like, you know, people I think want there to be a moment and I'm like, I don't know, I think that there might just never be a moment. It might just be that there's this continuous ramping up.

**Lex Fridman:** I have a sense that there will be things that a model can say that convinces you. This is very... it's not like... I've talked to people who are truly wise, and you could just tell there's a lot of horsepower there. And if you 10x that, I don't know, I just feel like there's words you could say. Maybe ask it to generate a poem and the poem it generates, you're like, yeah, okay, whatever you did there, I don't think a human can do that.

**Amanda Askell:** I think it has to be something that I can verify is actually really good though. That's why I think these questions that are where I'm like, oh, this is, you know, sometimes it's just like, I'll come up with, say, a concrete counterexample to an argument or something like that. I'm sure like, it would be like if you're a mathematician and you had a novel proof, I think, and you just gave it the problem and you saw it, and you're like, this proof is genuinely novel. No one has ever done... you actually have to do a lot of things to come up with this. I had to sit and think about it for months or something. And then if you saw the model successfully do that, I think you would just be like, I can verify that this is correct. It is a sign that you have generalized from your training. You didn't just see this somewhere because I just came up with it myself and you were able to replicate that. That's the kind of thing where I'm like, for me, the more that models can do things like that, the more I would be like, oh, this is very real, because then I can verify that that's extremely, extremely capable.

**Lex Fridman:** You've interacted with AI a lot. What do you think makes humans special? Maybe in a way that the universe is much better off that we're in it, and that we should definitely survive and spread throughout the universe.

**Amanda Askell:** Yeah, it's interesting because I think people focus so much on intelligence, especially with models. Look, intelligence is important because of what it does. It's very useful, it does a lot of things in the world. And you can imagine a world where height or strength would have played this role, and it's just a trait. It's not intrinsically valuable. It's valuable because of what it does. I think for the most part, the things that feel... I mean, personally, I think humans and life in general is extremely magical. Almost to the degree that, I don't know, not everyone agrees with this and I'm flagging, but we have this whole universe and there's all of these objects, there's beautiful stars and there's galaxies. And then, I don't know, I'm just like, on this planet there are these creatures that have this ability to observe, and they are seeing it, they are experiencing it. And I'm just like, if you try to explain, I imagine trying to explain to someone, for some reason they've never encountered the world or our science or anything, and I think that nothing is... all of our physics and everything in the world, it's all extremely exciting, but then you say, oh, and plus there's this thing that it is to be a thing and observe in the world, and you see this inner cinema. And I think they would be like, hang on, wait, pause, you just said something that is kind of wild sounding. And so we have this ability to experience the world. We feel pleasure, we feel suffering, we feel a lot of complex things. And so yeah, and maybe this is also why I think, you know, I also care a lot about animals, for example, because I think they probably share this with us. So I think that the things that make humans special, insofar as I care about humans, is probably more their ability to feel and experience than it is them having these functional useful traits.

**Lex Fridman:** Yeah, to feel and experience the beauty in the world. Yeah, to look at the stars. I hope there's other alien civilizations out there, but if we're it, it's a pretty good thing.

**Amanda Askell:** And that they're having a good time.

**Lex Fridman:** They're having a good time watching us. Well, thank you for this good time of a conversation and for the work you're doing and for helping make Claude a great conversational partner. And thank you for talking today.

**Amanda Askell:** Yeah, thanks for talking.

**Lex Fridman:** Thanks for listening to this conversation with Amanda Askell. And now, dear friends, here's Chris Olah.

**Lex Fridman:** Can you describe this fascinating field of mechanistic interpretability, AKA mech interp, the history of the field, and where it is today?

**Chris Olah:** I think one useful way to think about neural networks is that we don't program them, we don't make them, we kind of grow them. We have these neural network architectures that we design and we have these loss objectives that we create, and the neural network architecture, it's kind of like a scaffold that the circuits grow on. And they sort of, you know, it starts off with some kind of random things and it grows, and it's almost like the objective that we train for is this light. And so we create the scaffold that it grows on and we create the light that it grows towards, but the thing that we actually create, it's this almost biological entity or organism that we're studying. And so it's very, very different from any kind of regular software engineering, because at the end of the day we end up with this artifact that can do all these amazing things. It can write essays and translate and understand images. It can do all these things that we have no idea how to directly create a computer program to do. And it can do that because we grew it, we didn't write it, we didn't create it. And so then that leaves open this question at the end, which is, what the hell is going on inside these systems? And that, to me, is a really deep and exciting question. It's a really exciting scientific question. To me, it's the question that is just screaming out, it's calling out for us to go and answer it when we talk about neural networks. And I think it's also a very deep question for safety reasons.

**Lex Fridman:** And mechanistic interpretability, I guess, is closer to maybe neurobiology?

**Chris Olah:** Yeah, I think that's right. So maybe to give an example of the kind of thing that has been done that I wouldn't consider to be mechanistic interpretability: there was for a long time a lot of work on saliency maps, where you would take an image and you'd try to say, the model thinks this image is a dog, what part of the image made it think that it's a dog? And that tells you maybe something about the model if you can come up with a principled version of that, but it doesn't really tell you what algorithms are running in the model, how was the model actually making that decision. Maybe it's telling you something about what was important to it if you can make that method work, but it isn't telling you what are the algorithms that are running, how is it that this system is able to do this thing that no one knew how to do. And so I guess we started using the term mechanistic interpretability to try to draw that divide, or to distinguish ourselves and the work that we were doing in some ways from some of these other things. And I think since then it's become this sort of umbrella term for a pretty wide variety of work. But I'd say that the things that are kind of distinctive are, I think, this focus on: we really want to get at the mechanisms, we want to get at the algorithms. If you think of neural networks as being like a computer program, then the weights are kind of like a binary computer program.

**Chris Olah:** And we'd like to reverse engineer those weights and figure out what algorithms are running. So okay, I think one way you might think of trying to understand a neural network is that it's kind of like a compiled computer program and the weights of the neural network are the binary, and when the neural network runs that's the activations. And our goal is ultimately to go and understand these weights. And so, you know, the project of mechanistic interpretability is to somehow figure out how do these weights correspond to algorithms. And in order to do that you also have to understand the activations, because the activations are like the memory. And if you imagine reverse engineering a computer program and you have the binary instructions, in order to understand what a particular instruction means you need to know what is stored in the memory that it's operating on. And so those two things are very intertwined. So mechanistic interpretability tends to be interested in both of those things. Now, there's a lot of work that's interested in those things, especially the, you know, there's all this work on probing which you might see as part of mechanistic interpretability, although it's, you know, again it's just a broad term and not everyone who does that work would identify as doing mech interp. I think the thing that is maybe a little bit distinctive to the vibe of mech interp is, I think people working in the space tend to think of neural networks as, well, maybe one way to say it is that gradient descent is smarter than you. Gradient descent is actually really great. The whole reason that we're understanding these models is because we didn't know how to write them in the first place. Gradient descent comes up with better solutions than us. And so I think that maybe another thing about mech interp is sort of having almost a kind of humility, that we won't guess a priori what's going on inside the model and we have to have this sort of bottom-up approach where we don't really assume, you know, we don't assume that we should look for a particular thing and that will be there and that's how it works. But instead we look from the bottom up and discover what happens to exist in these models and study them that way.

**Lex Fridman:** But you know, the very fact that it's possible to do, and as you and others have shown over time, you know, things like universality, that the wisdom of the gradient descent creates features and circuits, creates things universally across different kinds of networks that are useful, and that makes the whole field possible.

**Chris Olah:** Yeah, so this is actually indeed a really remarkable and exciting thing, where it does seem like, at least to some extent, the same elements, the same features and circuits form again and again. You can look at every vision model and you'll find curve detectors and you'll find high-low frequency detectors. And in fact there's some reason to think that the same things form across biological neural networks and artificial neural networks. So a famous example is vision models in the early layers have Gabor filters, and Gabor filters are something that neuroscientists are interested in and have thought a lot about. We find curve detectors in these models. Curve detectors are also found in monkeys. We discover these high-low frequency detectors and then some follow-up work went and discovered them in rats or mice. So they were found first in artificial neural networks and then found in biological neural networks. You know, this really famous result on like grandmother neurons or the Halle Berry neuron from Quiroga et al., and we found very similar things in vision models. This was while I was still at OpenAI and I was looking at their CLIP model, and you find these neurons that respond to the same entities in images and also in text. To give a concrete example, we found that there was a Donald Trump neuron. For some reason, I guess everyone likes to talk about Donald Trump, and Donald Trump was very prominent, was a very hot topic at that time. So every neural network that we looked at, we would find a dedicated neuron for Donald Trump. He was the only person who always had a dedicated neuron. You know, sometimes you'd have an Obama neuron, sometimes you'd have a Clinton neuron, but Trump always had a dedicated one. So it responds to pictures of his face and the word "Trump," like all these things, right? And so it's not responding to a particular example, or it's not just responding to his face. It's abstracting over this general concept, right? So in any case, that's very similar to these Quiroga results. So there's this evidence that this phenomenon of universality -- the same things form across both artificial and natural neural networks. That's a pretty amazing thing if that's true. It suggests that, well, I think the thing that it suggests is gradient descent is sort of finding the right ways to cut things apart, in some sense, that many systems converge on. And many different neural network architectures converge on that -- there's some natural set of abstractions that are a very natural way to cut apart the problem and that a lot of systems are going to converge on. That would be my kind of, you know, I don't know anything about neuroscience, this is just my wild speculation from what we've seen.

**Lex Fridman:** Yeah, that would be beautiful if it's sort of agnostic to the medium of the model that's used to form the representation.

**Chris Olah:** Yeah, and it's a kind of wild speculation based, you know, we only have a few data points to suggest this. But it does seem like there's some sense in which the same things form again and again and again, both in artificial neural networks and biological ones. And the intuition behind that would be that in order to be useful in understanding the real world, you need all the same kind of stuff.

**Lex Fridman:** Yeah, well, if we pick, I don't know, like the idea of a dog, right?

**Chris Olah:** Like, you know, there's some sense in which the idea of a dog is like a natural category in the universe or something like this, right? There's some reason it's not just like a weird quirk of how humans think about the world that we have this concept of a dog. It's in some sense -- or like if you have the idea of a line, like, look around us, there are lines. It's sort of the simplest way to understand this room in some sense is to have the idea of a line. And so I think that that would be my instinct for why this happens.

**Lex Fridman:** Yeah, you need a curved line, you know, to understand a circle, and you need all those shapes to understand bigger things, and yeah, it's a hierarchy of concepts that are formed.

**Chris Olah:** Yeah, and like maybe there are ways to go and describe images without reference to those things, right? But they're not the simplest way or the most economical way or something like this. And so systems converge to these strategies, would be my wild hypothesis.

**Lex Fridman:** Can you talk through some of the building blocks that we've been referencing, of features and circuits? So I think you first described them in a 2020 paper, "Zoom In: An Introduction to Circuits."

**Chris Olah:** Absolutely. So maybe I'll start by just describing some phenomena and then we can sort of build to the idea of features and circuits. So I spent quite a few years, maybe like five years to some extent with other things, studying this one particular model, Inception V1, which is this one vision model. It was state of the art in 2015 and very much not state of the art anymore. And it has maybe about 10,000 neurons, and I spent a lot of time looking at the 10,000-odd neurons of Inception V1. And one of the interesting things is, there are lots of neurons that don't have some obvious interpretable meaning, but there are a lot of neurons in Inception V1 that do have really clean interpretable meanings. So you find neurons that just really do seem to detect curves, and you find neurons that really do seem to detect cars, and car wheels, and car windows, and floppy ears of dogs, and dogs with long snouts facing to the right, and dogs with long snouts facing to the left, and different kinds of fur. And there's sort of this whole beautiful -- edge detectors, line detectors, color contrast detectors, these beautiful things we call high-low frequency detectors. I sort of felt like a biologist. You're just looking at this sort of new world of proteins and you're discovering all these different proteins that interact. So one way you could try to understand these models is in terms of neurons. You could try to be like, oh, there's a dog-detecting neuron and here's a car-detecting neuron. And it turns out you can actually ask how those connect together. So you can go and say, oh, I have this car-detecting neuron, how was it built? And it turns out in the previous layer it's connected really strongly to a window detector and a wheel detector and a sort of car body detector. And it looks for the window above the car and the wheels below and the car chrome sort of in the middle, sort of everywhere but especially on the lower part. And that's sort of a recipe for a car, right? Like, that is -- earlier we said the thing we wanted from mech interp was to get algorithms, to ask what is the algorithm that runs. Well, here we're just looking at the weights of the neural network, reading off this kind of recipe for detecting cars. It's a very simple, crude recipe, but it's there. And so we call that a circuit, this connection.

Well, okay, so the problem is that not all of the neurons are interpretable. And there's reason to think -- we can get into this more later -- that there's this superposition hypothesis, there's reason to think that sometimes the right unit to analyze things in terms of is combinations of neurons. So sometimes it's not that there's a single neuron that represents, say, a car, but it actually turns out that after you detect the car, the model sort of hides a little bit of the car in the following layer in a bunch of dog detectors. Why is it doing that? Well, maybe it just doesn't want to do that much work on cars at that point and it's sort of storing it away. So it turns out then that the subtle pattern of -- there's all these neurons that you think are dog detectors, and maybe they're primarily that, but they all a little bit contribute to representing a car in that next layer. Okay, so now we can't really think -- there might still be something, I don't know, you could call it like a car concept or something, but it no longer corresponds to a neuron. So we need some term for these kind of neuron-like entities, these things that we sort of would have liked the neurons to be, these idealized neurons, the things that are the nice neurons, but also maybe there's more of them somehow hidden. And we call those features.

And then what are circuits? So circuits are these connections of features, right? So when we have the car detector and it's connected to a window detector and a wheel detector and it looks for the wheels below and the windows on top, that's a circuit. So circuits are just collections of features connected by weights, and they implement algorithms. So they tell us how features are used, how they are built, how they connect together.

So maybe it's worth trying to pin down what really is the core hypothesis here. I think the core hypothesis is something we call the linear representation hypothesis. So if we think about the car detector, the more it fires, the more we sort of think of that as meaning the model is more and more confident that a car was present. Or if it's some combination of neurons that represent a car, the more that combination fires, the more we think the model thinks there's a car present. This doesn't have to be the case, right? You could imagine something where you have this car detector neuron and you think, ah, if it fires between one and two that means one thing, but it means totally different if it's between three and four. That would be a nonlinear representation. In principle, models could do that. I think it's sort of inefficient for them to do. If you try to think about how you'd implement computation like that, it's kind of an annoying thing to do. But in principle models can do that.

So one way to think about the features and circuits sort of framework for thinking about things is that we're thinking about things as being linear. We're thinking about there being that if a neuron or a combination of neurons fires more, it sort of means more of a particular thing being detected. And then that gives weights a very clean interpretation as these edges between these entities, these features, and that edge then has a meaning. So that's in some ways the core thing.

**Lex Fridman:** Are you familiar with the Word2Vec results? So you have like, King minus Man plus Woman equals Queen. Well, the reason you can do that kind of arithmetic is because you have a linear representation. Can you actually explain that representation a little bit? So first off, a feature is a direction of activation, you think of it that way? Can you do the minus Man plus Woman, the Word2Vec stuff? Can you explain what that is?

**Chris Olah:** Yeah, there's a very -- such a simple, clean explanation of what we're talking about, exactly. So there's this very famous result, Word2Vec, by Tomas Mikolov et al., and there's been tons of follow-up work exploring this. So sometimes we create these word embeddings where we map every word to a vector. I mean, that in itself, by the way, is kind of a crazy thing if you haven't thought about it before, right? We're going and representing -- we're turning, you know, like if you just learned about vectors in physics class and I'm like, oh, I'm going to actually turn every word in the dictionary into a vector, that's kind of a crazy idea.

But you could imagine all kinds of ways in which you might map words to vectors. But it seems like when we train neural networks, they like to map words to vectors such that they have linear structure in a particular sense, which is that directions have meaning. So for instance, there will be some direction that seems to sort of correspond to gender, and male words will be far in one direction and female words will be in another direction. And the linear representation hypothesis is -- you could sort of think of it roughly as saying that that's actually kind of the fundamental thing that's going on, that everything is just different directions have meanings and adding different direction vectors together can represent concepts.

And the Mikolov paper sort of took that idea seriously, and one consequence of it is that you can do this game of playing sort of arithmetic with words. So you can do "King" and you can subtract off the word "Man" and add the word "Woman," and so you're sort of trying to switch the gender, and indeed if you do that the result will be close to the word "Queen." And you can do other things like you can do "Sushi" minus "Japan" plus "Italy" and get "Pizza," or different things like this.

So this is in some sense the core of the linear representation hypothesis. You can describe it just as a purely abstract thing about vector spaces. You can describe it as a statement about the activations of neurons. But it's really about this property of directions having meaning. And in some ways it's even a little subtler than that. It's really, I think, mostly about this property of being able to add things together, that you can sort of independently modify, say, gender and royalty, or cuisine type or country and the concept of food, by adding them.

**Lex Fridman:** Do you think the linear representation hypothesis holds? That it carries? That it scales?

**Chris Olah:** So far I think everything I have seen is consistent with this hypothesis, and it doesn't have to be that way, right? You can write down neural networks where you write weights such that they don't have linear representations, where the right way to understand them is not in terms of linear representations. But I think every natural neural network I've seen has this property. There's been one paper recently -- there's been some sort of pushing around the edges. So I think there's been some work recently studying multi-dimensional features, where rather than a single direction it's more like a manifold of directions. This to me still seems like a linear representation. And then there's been some other papers suggesting that maybe in very small models you get nonlinear representations. I think the jury's still out on that. But I think everything that we've seen so far has been consistent with the linear representation hypothesis.

And that's wild. It doesn't have to be that way, and yet I think there's a lot of evidence that certainly at least this is very, very widespread, and so far the evidence is consistent with that. And I think, you know, one thing you might say is, well, Christopher, that's a lot to ride on if we don't know for sure this is true and you're sort of investigating all neural networks as though it is true. Isn't that dangerous? Well, I think actually there's a virtue in taking hypotheses seriously and pushing them as far as they can go. It might be that someday we discover something that is inconsistent with the linear representation hypothesis, but science is full of hypotheses and theories that were wrong, and we learned a lot by sort of working under them as an assumption and then going and pushing them as far as we can. I guess this is sort of the heart of what Kuhn would call normal science.

**Lex Fridman:** I don't know if you want, we can talk a lot about philosophy of science and -- that leads to the paradigm shift. So, yeah, I love it, taking the hypothesis seriously and taking it to its natural conclusion.

**Chris Olah:** Yeah, same with the scaling hypothesis.

**Lex Fridman:** Same, exactly, exactly. And I love it.

**Chris Olah:** One of my colleagues, Tom Henighan, who is a former physicist, made this really nice analogy to me of caloric theory, where once upon a time we thought that heat was actually this thing called caloric, and the reason hot objects would warm up cool objects is like the caloric is flowing through them. And because we're so used to thinking about heat in terms of the modern theory, that seems kind of silly. But it's actually very hard to construct an experiment that disproves the caloric hypothesis. And you can actually do a lot of really useful work believing in caloric. For example, it turns out that the original combustion engines were developed by people who believed in caloric theory.

**Chris Olah:** In the caloric theory. So I think there's a virtue in taking hypotheses seriously even when they might be wrong.

**Lex Fridman:** Yeah, yeah, there's a deep philosophical truth to that. That's kind of how I feel about space travel, like colonizing Mars. There's a lot of people that criticize that. I think if you just assume we have to colonize Mars in order to have a backup for human civilization, even if that's not true, that's going to produce some interesting engineering and even scientific breakthroughs, I think.

**Chris Olah:** Yeah, well, and actually this is another thing that I think is really interesting. So, you know, there's a way in which I think it can be really useful for society to have people almost irrationally dedicated to investigating a particular hypothesis, because it takes a lot to sort of maintain scientific morale and really push on something when, you know, most scientific hypotheses end up being wrong. A lot of science doesn't work out. And yet it's very useful to go do -- you know, there's a joke about Jeff Hinton, which is that Jeff Hinton has discovered how the brain works every year for the last 50 years.

**Lex Fridman:** Yeah.

**Chris Olah:** But, you know, I say that with really deep respect, because in fact that actually led to him doing some really great work.

**Lex Fridman:** Yeah, he won the Nobel Prize. Now who's laughing now?

**Chris Olah:** Exactly, exactly. Yeah, I think one wants to be able to pop up and sort of recognize the appropriate level of confidence, but I think there's also a lot of value in just being like, you know, I'm going to essentially assume -- I'm going to condition on this problem being possible or this being broadly the right approach, and I'm just going to go and assume that for a while and go and work within that and push really hard on it. And, you know, if society has lots of people doing that for different things, that's actually really useful in terms of either really ruling things out -- we can be like, well, you know, that didn't work, we know that somebody tried hard -- or going and getting to something that does teach us something about the world.

**Lex Fridman:** So another interesting hypothesis is the superposition hypothesis. Can you describe what superposition is?

**Chris Olah:** Yeah. So earlier we were talking about Word2Vec, right? And we were talking about how maybe you have one direction that corresponds to gender, and maybe another that corresponds to royalty, and another one that corresponds to Italy, and another one that corresponds to food, and all these things. Well, often times these word embeddings might be 500 dimensions, a thousand dimensions, and so if you believed that all of those directions were orthogonal, then you could only have 500 concepts. And, you know, I love pizza, but like, if I was going to give the 500 most important concepts in the English language, probably Italy wouldn't be -- it's not obvious at least that Italy would be one of them. Because you have to have things like plural and singular, and verb and noun and adjective, and there's a lot of things we have to get to before we get to Italy, and Japan, and, you know, there's a lot of countries in the world.

And so how might it be that models could simultaneously have the linear representation hypothesis be true and also represent more things than they have directions? So what does that mean? Well, if the linear representation hypothesis is true, something interesting has to be going on. Now I'll tell you one more interesting thing before we go and do that, which is, you know, earlier we were talking about all these polysemantic neurons, right? These neurons that, when we're looking at Inception V1, there are these nice neurons like the car detector and the curve detector and so on that respond to very coherent things, but there are lots of neurons that respond to a bunch of unrelated things. That's also an interesting phenomenon. And it turns out as well that even these neurons that are really, really clean, if you look at the weak activations -- so if you look at the activation where it's like activating 5% of the maximum activation -- it's really not the core thing that it's detecting. So if you look at a curve detector, for instance, and you look at the places where it's 5% active, you could interpret it just as noise, or it could be that it's doing something else there.

Okay, so how could that be? Well, there's this amazing thing in mathematics called compressed sensing. And it's actually this very surprising fact where, if you have a high-dimensional space and you project it into a low-dimensional space, ordinarily you can't go and sort of unproject it and get back your high-dimensional vector. You threw information away. This is like, you can't invert a rectangular matrix; you can only invert square matrices. But it turns out that that's actually not quite true. If I tell you that the high-dimensional vector was sparse -- so it's mostly zeros -- then it turns out that you can often find back the high-dimensional vector with very high probability.

So that's a surprising fact. It says that you can have this high-dimensional vector space, and as long as things are sparse, you can project it down, you can have a lower-dimensional projection of it, and that works. So the superposition hypothesis is saying that that's what's going on in neural networks. That's, for instance, what's going on in word embeddings. The word embeddings are able to simultaneously have directions be the meaningful thing, and by exploiting the fact that they're operating on a fairly high-dimensional space and the fact that these concepts are sparse -- like, you usually aren't talking about Japan and Italy at the same time, and in most sentences Japan and Italy are both zero, they're not present at all -- if that's true, then you can have it be the case that you can have many more of these directions that are meaningful, these features, than you have dimensions. And when we're talking about neurons, you can have many more concepts than you have neurons.

So that's the superposition hypothesis at a high level. Now it has this even wilder implication, which is to say that neural networks -- it may not just be the case that the representations are like this, but the computation may also be like this, the connections between all of them. And so in some sense, neural networks may be shadows of much larger, sparser neural networks, and what we see are these projections. The strongest version of the superposition hypothesis would be to take that really seriously and sort of say, there actually is in some sense this upstairs model where the neurons are really sparse and all interpretable, and the weights between them are these really sparse circuits, and that's what we're studying. And the thing that we're observing is the shadow of it, and we need to find the original object. And the process of learning is trying to construct a compression of the upstairs model that doesn't lose too much information in the projection.

**Lex Fridman:** Yeah, finding how to fit it efficiently or something like this.

**Chris Olah:** So this sort of says that gradient descent, you know, could just represent a dense neural network, but it sort of says that gradient descent is pleasantly searching over the space of extremely sparse models that could be projected into this low-dimensional space. And this large body of work of people going and trying to study sparse neural networks, where you go and design neural networks where the edges are sparse and the activations are sparse -- my sense is that work has generally... it feels very principled, it makes so much sense, and yet that work hasn't really panned out that well, is my impression broadly. And I think a potential answer for that is that actually the neural network is already sparse in some sense. Gradient descent was the whole time -- you were trying to go and do this, gradient descent was actually behind the scenes going and searching more efficiently than you could through the space of sparse models and learning whatever sparse model was most efficient and then figuring out how to fold it down nicely to go and run conveniently on your GPU, which does nice dense matrix multiplies. And you just can't beat that.

**Lex Fridman:** How many concepts do you think can be shoved into a neural network?

**Chris Olah:** Depends on how sparse they are. So there's probably an upper bound from the number of parameters, because you still have to have weights that go and connect them together. So that's one upper bound. There are in fact all these lovely results from compressed sensing and the Johnson-Lindenstrauss Lemma and things like this. They basically tell you that if you have a vector space and you want to have almost orthogonal vectors -- which is sort of probably the thing that you want here, right? So you're going to say, well, I'm going to give up on having my concepts, my features, be strictly orthogonal, but I'd like them to not interfere that much. I'm going to have to ask them to be almost orthogonal. Then this would say that it's actually exponential in the number of neurons that you have. So at some point that's not going to even be the limiting factor.

But there are some beautiful results there, and in fact it's probably even better than that in some sense, because that's sort of for saying that any random set of features could be active. But in fact the features have a correlational structure where some features are more likely to co-occur and other ones are less likely to co-occur. And so neural networks, my guess would be, can do very well in terms of packing things in, such that it's probably not the limiting factor.

**Lex Fridman:** How does the problem of polysemanticity enter the picture here?

**Chris Olah:** Polysemanticity is this phenomenon we observe where we look at many neurons and the neuron doesn't just represent one concept. It's not a clean feature; it responds to a bunch of unrelated things. And superposition is -- you can think of it as being a hypothesis that explains the observation of polysemanticity. So polysemanticity is this observed phenomenon and superposition is a hypothesis that would explain it, along with some other things.

So that makes mech interp more difficult, right? If you're trying to understand things in terms of individual neurons and you have polysemantic neurons, you're in an awful lot of trouble. I mean, the easiest answer is like, okay, well, you're looking at the neurons, you're trying to understand them, this one responds to a lot of things, it doesn't have a nice meaning -- okay, that's bad. Another thing you could ask is, you know, ultimately we want to understand the weights, and if you have two polysemantic neurons, and each one responds to three things, and then the other neuron responds to three things, and you have a weight between them, what does that mean? Does it mean that there are these nine interactions going on? It's a very weird thing.

But there's also a deeper reason, which is related to the fact that neural networks operate on really high-dimensional spaces. So I said that our goal was to understand neural networks and understand the mechanisms, and one thing you might say is, well, why not -- it's just a mathematical function, why not just look at it? Like, one of the earliest projects I did studied these neural networks that mapped two-dimensional spaces to two-dimensional spaces, and you can sort of interpret them in this beautiful way as bending manifolds. Why can't we do that? Well, as you have a higher-dimensional space, the volume of that space in some sense is exponential in the number of inputs you have, and so you can't just go and visualize it. So we somehow need to break that apart; we need to somehow break that exponential space into a bunch of things that we can reason about independently -- some non-exponential number of things that we can reason about independently. And the independence is crucial, because it's the independence that allows you to not have to think about all the exponential combinations of things. And things being monosemantic -- things only having one meaning -- is the key thing that allows you to think about them independently. And so I think if you want the deepest reason why we want to have interpretable, monosemantic features, I think that's really the deep reason.

**Lex Fridman:** And so the goal here, as your recent work has been aiming at, is how do we extract the monosemantic features from a neural net that has polysemantic features and all this mess?

**Chris Olah:** Yes. We observe these polysemantic neurons, and we hypothesize that superposition is what's going on. And if superposition is what's going on, there's actually a sort of well-established technique that is the principled thing to do, which is dictionary learning. And it turns out, if you do dictionary learning -- in particular if you do a nice efficient way that in some sense nicely regularizes it as well, called a sparse autoencoder -- if you train a sparse autoencoder, these beautiful interpretable features start to just fall out where there weren't any beforehand. And so that's not the sort of thing that you would necessarily predict, but it turns out that that works very, very well. To me, that seems like some non-trivial validation of linear representations and superposition.

**Lex Fridman:** So with dictionary learning, you're not looking for particular kinds of categories; you don't know what they are.

**Chris Olah:** And this gets back to our earlier point, right? We're not making assumptions. Gradient descent is smarter than us, so we're not making assumptions about what's there. I mean, one certainly could do that -- one could assume that there's a PHP feature and go and search for it -- but we're not doing that. We're saying we don't know what's going to be there. Instead we're just going to let the sparse autoencoder discover the things that are there.

**Lex Fridman:** So can you talk to the "Towards Monosemanticity" paper from October last year that had a lot of nice breakthrough results?

**Chris Olah:** That's very kind of you to describe it that way. Yeah, I mean, this was our first real success using sparse autoencoders. So we took a one-layer model, and it turns out if you do dictionary learning on it, you find all these really nice interpretable features. So, you know, the Arabic feature, the Hebrew feature, the Base64 feature -- those were some examples that we studied in a lot of depth and really showed that they were what we thought they were. It turns out if you train a model twice and train two different models and do dictionary learning, you find analogous features in both of them. So that's fun. You find all kinds of different features. So that was really just showing that this works. And, you know, I should mention that there was Cunningham et al. that had very similar results around the same time.

**Lex Fridman:** There's something fun about doing these kinds of small-scale experiments and finding that it's actually working.

**Chris Olah:** Yeah, well, and there's so much structure here. Like, maybe stepping back for a while, I thought that maybe all this mechanistic interpretability work -- the end result was going to be that I would have an explanation for why it was sort of very hard and not going to be tractable. We'd be like, well, there's this problem with superposition and it turns out that superposition is really hard, and we're kind of screwed. But that's not what happened. In fact, a very natural technique just works.

And so then that's actually a very good situation. I think this is a hard research problem and it's got a lot of research risk, and it might still very well fail. But I think some very significant amount of research risk was sort of put behind us when that started to work.

**Lex Fridman:** Can you describe what kind of features can be extracted in this way?

**Chris Olah:** Well, so it depends on the model that you're studying. The larger the model, the more sophisticated they're going to be, and we'll probably talk about follow-up work in a minute. But in these one-layer models, some very common things I think were languages -- both programming languages and natural languages. There were a lot of features that were specific words in specific contexts. So "the" -- and I think really the way to think about this is that "the" is likely about to be followed by a noun. So it's really -- you could think of this as a "the" feature, but you could also think of this as predicting a specific noun feature. And there would be these features that would fire for "the" in the context of, say, a legal document or a mathematical document or something like this. And so maybe in the context of math, you're like, "the" and then predict "vector," "matrix," all these mathematical words, whereas in other contexts you would predict other things. That was common.

**Lex Fridman:** And basically you need clever humans to assign labels to what we're seeing?

**Chris Olah:** Yes. So the only thing this is doing is sort of unfolding things for you. If everything was sort of folded over top of itself -- superposition folded everything on top of itself, you can't really see it -- this is unfolding it. But now you still have a very complex thing to try to understand. So then you have to do a bunch of work understanding what these are. And some of them are really subtle. Like, there are some really cool things even in this one-layer model about Unicode, where, of course, some languages are in Unicode and the tokenizer won't necessarily have a dedicated token for every Unicode character. So instead what you'll have is these patterns of alternating tokens that each represent half of a Unicode character. And then you have a different feature that activates on the opposing ones to be like, okay, I just finished a character, go and predict the next prefix. Then on the prefix, predict a reasonable suffix. And you have to alternate back and forth.

So these one-layer models are really interesting. And there's another thing, which is you might think, okay, there would just be one Base64 feature, but it turns out there's actually a bunch of Base64 features, because you can have English text encoded as Base64, and that has a very different distribution of Base64 tokens than regular Base64. And there are some things about tokenization as well that it can exploit. And, I don't know, all kinds of fun stuff.

**Lex Fridman:** How difficult is the task of sort of assigning labels to what's going on? Can this be automated by AI?

**Chris Olah:** Well, I think it depends on the feature, and it also depends on how much you trust your AI. There's a lot of work doing automated interpretability. I think that's a really exciting direction, and we do a fair amount of automated interpretability and have Claude go and label our features.

**Lex Fridman:** Is there some fun moments where it's totally right or it's totally wrong?

**Chris Olah:** Yeah, well, I think it's very common that it says something very general, which is true in some sense but not really picking up on the specifics of what's going on. So I think that's a pretty common situation. I don't know that I have a particularly amusing one.

**Lex Fridman:** That's interesting, that little gap between "it is true" but it doesn't quite get to the deep nuance of a thing.

**Chris Olah:** Yeah, that's a general challenge. It's like, it's still an incredible accomplishment -- they can say a true thing, but it doesn't quite -- it's not -- it's missing the depth sometimes. And in this context it's like the ARC challenge, you know, the sort of IQ-type tests. It feels like figuring out what a feature represents is a bit of -- is a little
puzzle you have to solve. Yeah, and I think that sometimes they're easier and sometimes they're harder as well. So yeah, I think that's tricky. Now there's another thing which -- I don't know, maybe in some ways this is my aesthetic coming in, but I'll try to give you a rationalization. I'm actually a little suspicious of automated interpretability, and I think that's partly just that I want humans to understand neural networks, and if the neural network is understanding it for me, I'm not -- I don't quite like that. But I do have -- in some ways I'm sort of like the mathematicians who are like, you know, if there's a computer-automated proof, it doesn't count. They won't understand it. But I do also think that there is this kind of "Reflections on Trusting Trust" type issue where, you know, there's this famous talk about when you're writing a computer program you have to trust your compiler, and if there was malware in your compiler then it could go and inject malware into the next compiler, and you'd be kind of in trouble, right? Well, if you're using neural networks to verify that your neural networks are safe, the hypothesis that you're testing for is, okay, well, the neural network maybe isn't safe, and you have to worry about, is there some way that it could be screwing with you? So I think that's not a big concern now, but I do wonder in the long run, if we have to use really powerful AI systems to audit our AI systems, is that actually something we can trust? But maybe I'm just rationalizing because I just want us to get to a point where humans understand everything.

**Lex Fridman:** Yeah, I mean, especially -- that's hilarious -- especially as we talk about AI safety and looking for features that would be relevant to AI safety, like deception and so on. So let's talk about the "Scaling Monosemanticity" paper in May 2024. What did it take to scale this, to apply to Claude 3 Sonnet?

**Chris Olah:** Well, a lot of GPUs. A lot more GPUs. But one of my teammates, Tom Henighan, was involved in the original scaling laws work, and something that he was sort of interested in from very early on is, are there scaling laws for interpretability? And so something he sort of immediately did when this work started to succeed and we started to have sparse autoencoders work -- we became very interested in, what are the scaling laws for making sparse autoencoders larger, and how does that relate to making the base model larger? And so it turns out this works really well, and you can use it to sort of project -- if you train a sparse autoencoder of a given size, you know how many tokens should you train on and so on. So this was actually a very big help to us in scaling up this work and made it a lot easier for us to train really large sparse autoencoders, where it's not like training the big models, but it's starting to get to a point where it's actually expensive to train the really big ones.

**Lex Fridman:** So you have to -- I mean, you have to do all the stuff of splitting it across -- there's a huge engineering challenge here too, right?

**Chris Olah:** Yes. So there's a scientific question of how do you scale things effectively, and then there's an enormous amount of engineering to go and scale this up. You have to shard it, you have to think very carefully about a lot of things. I'm lucky to work with a bunch of great engineers, because I am definitely not a great engineer.

**Lex Fridman:** Yeah, on the infrastructure especially, yeah, for sure. So it turns out -- TL;DR -- it worked?

**Chris Olah:** It worked. Yeah, and I think this is important because you could have imagined a world where, after "Towards Monosemanticity," you'd say, "Chris, this is great, it works on a one-layer model, but one-layer models are really idiosyncratic. Maybe the linear representation hypothesis and superposition hypothesis is the right way to understand a one-layer model, but it's not the right way to understand large models." And so I think the "Scaling Monosemanticity" paper sort of -- I mean, first of all, the "Cutting Him at All" paper sort of cut through that a little bit and suggested that this wasn't the case. But "Scaling Monosemanticity" was, I think, significant evidence that even for very large models -- and we did it on Claude 3 Sonnet, which at that point was one of our production models -- even these models seem to be substantially explained, at least by linear features, and doing dictionary learning on them works. And as you learn more features, you explain more and more. So that's a quite promising sign. And you find really fascinating abstract features, and the features are also multimodal -- they respond to images and text for the same concept, which is fun.

**Lex Fridman:** Yeah, can you explain that? I mean, like, "backdoor" -- there's just a lot of examples.

**Chris Olah:** Yeah, so maybe let's start with one example, which is we found some features around security vulnerabilities and backdoors in code. So it turns out those are actually two different features. There's a security vulnerability feature, and if you force it active, Claude will start to write security vulnerabilities like buffer overflows into code. And it also fires for all kinds of things -- some of the top dataset examples for it were things like, you know, "--disable SSL" or something like this, which are sort of obviously really insecure.

**Lex Fridman:** So at this point it's kind of like maybe it's just because the examples are presented that way -- it's kind of surface-level, more obvious examples, right? I guess the idea is that down the line you might be able to detect more nuanced things, like deception or bugs or that kind of stuff.

**Chris Olah:** Yeah, well, I maybe want to distinguish two things. One is the complexity of the feature or the concept, and the other is the nuance of how subtle the examples we're looking at are. When we show the top dataset examples, those are the most extreme examples that cause that feature to activate. And so it doesn't mean that it doesn't fire for more subtle things. The insecure code feature -- the stuff that it fires for most strongly are these really obvious, you know, "disable the security" type things. But it also fires for buffer overflows and more subtle security vulnerabilities in code. These features are all multimodal, so you could ask, "What images activate this feature?" And it turns out that the security vulnerability feature activates for images of people clicking on Chrome to go past the, you know, "this website's SSL certificate might be wrong" warning.

Another thing that's very entertaining is the backdoors-in-code feature. If you activate it, Claude writes a backdoor that will dump your data to a port or something. But you can ask, "What images activate the backdoor feature?" It was devices with hidden cameras in them. So there's apparently a whole genre of people selling devices that look innocuous but have hidden cameras, and they have ads showing where there's a hidden camera in it. And I guess that is the physical version of a backdoor. And so it sort of shows you how abstract these concepts are. I'm sort of sad that there's a whole market of people selling devices like that, but I was kind of delighted that that was the thing it came up with as the top image examples for the feature.

**Lex Fridman:** Yeah, it's nice. It's multimodal, it's multi-context -- it's as broad a definition of a singular concept. It's nice. Yeah, to me one of the really interesting features, especially for AI safety, is deception and lying, and the possibility that these kinds of methods could detect lying in a model, especially as it gets smarter and smarter and smarter. Presumably that's a big threat of a superintelligent model -- that it can deceive the people operating it as to its intentions, or any of that kind of stuff. So what have you learned from detecting lying inside models?

**Chris Olah:** Yeah, so I think we're in some ways in early days for that. We find quite a few features related to deception and lying. There's one feature that fires for people lying and being deceptive, and you force it active and Claude starts lying to you. So we have a deception feature. I mean, there's all kinds of other features about withholding information and not answering questions, features about power-seeking and coups and stuff like that. There are a lot of features that are kind of related to spooky things, and if you force them active, Claude will behave in ways that are -- they're not the kind of behaviors you want.

**Lex Fridman:** What are possible next exciting directions to you in the space of mech interp?

**Chris Olah:** Well, there's a lot of things. For one thing, I would really like to get to a point where we have circuits, where we can really understand not just the features but then use that to understand the computation of models. That really, for me, is the ultimate goal of this. And there's been some work -- we put out a few things, there's a paper from Sam Marks that does some stuff like this, there's been some, I'd say, work around the edges here. But I think there's a lot more to do, and I think that will be a very exciting thing.

That's related to a challenge we call interference weights, where, due to superposition, if you just sort of naively look at whether features are connected together, there may be some weights that don't exist in the upstairs model but are just sort of artifacts of superposition. So that's a technical challenge related to that.

I think another exciting direction is just -- you might think of sparse autoencoders as being kind of like a telescope. They allow us to look out and see all these features that are out there. And as we build better and better sparse autoencoders, get better and better at dictionary learning, we see more and more stars, and we zoom in on smaller and smaller stars. But there's a lot of evidence that we're only still seeing a very small fraction of the stars. There's a lot of matter in our neural network universe that we can't observe yet. And it may be that we'll never be able to have fine enough instruments to observe it, and maybe some of it just isn't possible, isn't computationally tractable to observe. There's sort of a kind of dark matter -- not maybe in the sense of modern astronomy, but of earlier astronomy, when we didn't know what this unexplained matter is. And so I think a lot about that dark matter and whether we'll ever observe it, and what that means for safety if we can't observe it -- if some significant fraction of neural networks are not accessible to us.

Another question that I think a lot about is, at the end of the day, mechanistic interpretability -- it's a very microscopic approach to interpretability. It's trying to understand things in a very fine-grained way. But a lot of the questions we care about are very macroscopic. We care about these questions about neural network behavior. And I think that's the thing that I care most about, but there are lots of other sort of larger-scale questions you might care about. And somehow, the nice thing about having a very microscopic approach is it's maybe easier to ask, "Is this true?" But the downside is it's much further from the things we care about, and so we now have this ladder to climb. And I think there's a question of, can we -- will we be able to find -- are there sort of larger-scale abstractions that we can use to understand neural networks? Can we get up from this very microscopic approach?

**Lex Fridman:** Yeah, you've written about this, this kind of "organs" question.

**Chris Olah:** Yeah, exactly.

**Lex Fridman:** "If we think of interpretability as a kind of anatomy of neural networks, most of the circuit threads involve studying tiny little veins, looking at the small scale and individual neurons and how they connect. However, there are many natural questions that the small-scale approach doesn't address. In contrast, the most prominent abstractions in biological anatomy involve larger-scale structures like individual organs, like the heart, or entire organ systems, like the respiratory system. And so we wonder, is there a respiratory system or heart or brain region of an artificial neural network?"

**Chris Olah:** Yeah, exactly. And I mean, if you think about science, a lot of scientific fields investigate things at many levels of abstraction. In biology you have molecular biology studying proteins and molecules and so on, and you have cellular biology, and then you have histology studying tissues, and you have anatomy, and then you have zoology, and then you have ecology. So you have many, many levels of abstraction. Or in physics -- the physics of individual particles, and then statistical physics gives you thermodynamics and things like this. And so you often have different levels of abstraction. And I think right now, mechanistic interpretability, if it succeeds, is sort of like a microbiology of neural networks, but we want something more like anatomy. And a question you might ask is, why can't you just go there directly? And I think the answer is, in significant part, it's that it's actually very hard to see this macroscopic structure without first breaking down the microscopic structure in the right way and then studying how it connects together. But I'm hopeful that there is going to be something much larger than features and circuits, and that we're going to be able to have a story that involves much bigger things, and you can then sort of study in detail the parts you care about.

**Lex Fridman:** As opposed to neurobiology, like a psychologist or psychiatrist for your neural network.

**Chris Olah:** And I think that the beautiful thing would be if we could, rather than having disparate fields for those two things, build a bridge between them such that you could have all of your higher-level abstractions be grounded very firmly in this very solid, more rigorous, ideally, foundation.

**Lex Fridman:** What do you think is the difference between the human brain -- the biological neural network -- and the artificial neural network?

**Chris Olah:** Well, the neuroscientists have a much harder job than us. Sometimes I just count my blessings by how much easier my job is than the neuroscientist's. So we can record from all the neurons. Yeah, we can do that on arbitrary amounts of data. The neurons don't change while you're doing that, by the way. You can ablate neurons, you can edit the connections and so on, and then you undo those changes. That's pretty great, yeah. You can force any -- you can intervene on any neuron and force it active and see what happens. You know which neurons are connected to everything, right? Neuroscientists want to get the connectome; we have the connectome. And we have it for much bigger than C. elegans. And then not only do we have the connectome, we know which neurons excite or inhibit each other, right? So it's not just that we know the binary mask; we know the weights. We can take gradients, we know computationally what each neuron does. So the list goes on and on -- we just have so many advantages over neuroscientists.

And then despite having all those advantages, it's really hard. And so one thing I do sometimes think is, gosh, if it's this hard for us, it seems impossible -- or near impossible -- under the constraints of neuroscience. I've got a few neuroscientists on my team, and maybe the neuroscientists -- maybe some of them would like to have an easier problem that's still very hard, and they could come and work on neural networks. And then after we figure out things in the easy little pond of trying to understand neural networks -- which is still very hard -- then we could go back to biological neuroscience.

**Lex Fridman:** I love what you've written about the goal of mech interp research as two goals: safety and beauty. So can you talk about the beauty side of things?

**Chris Olah:** Yeah. So there's this funny thing where I think some people are kind of disappointed by neural networks, where they're like, "Ah, it's just these simple rules, then you just do a bunch of engineering to scale it up and it works really well. Where are the complex ideas? This isn't a very nice, beautiful scientific result." And I sometimes think, when people say that, I picture them being like, "Evolution is so boring. It's just a bunch of simple rules and you run evolution for a long time and you get biology. What a sucky way for biology to have turned out. Where are the complex rules?"

But the beauty is that the simplicity generates complexity. Biology has these simple rules and it gives rise to all the life and ecosystems that we see around us -- all the beauty of nature -- that all just comes from evolution, from something very simple. And similarly, I think that neural networks create enormous complexity and beauty and structure inside themselves that people generally don't look at and don't try to understand, because it's hard to understand. But I think that there is an incredibly rich structure to be discovered inside neural networks, a lot of very deep beauty, if we're just willing to take the time to go and see it and understand it.

**Lex Fridman:** Yeah, I love mech interp. The feeling like we are understanding, or getting glimpses of understanding, the magic that's going on inside is really wonderful. It feels to me like one of the questions that's just calling out to be asked. And I'm often surprised by how -- I mean, a lot of people think about this, but -- how is it that we don't know how to create computer systems that can do these things? And yet we have these amazing systems. We don't know how to directly create computer programs that can do these things, but these neural networks can do all these amazing things. And it just feels like that is obviously the question that is calling out to be answered, if you have any degree of curiosity. It's like, how is it that humanity now has these artifacts that can do these things that we don't know how to do?

**Chris Olah:** Yeah, I love the image of the circuits towards the light of the objective function. It's this organic thing that we've grown and we have no idea what we've grown.

**Lex Fridman:** Well, thank you for working on safety, and thank you for appreciating the beauty of the things you discover, and thank you for talking today, Chris. This is wonderful.

**Chris Olah:** Thank you for taking the time to chat as well.

**Lex Fridman:** Thanks for listening to this conversation with Chris Olah, and before that with Dario Amodei and Amanda Askell. To support this podcast, please check out our sponsors in the description. And now let me leave you with some words from Alan Watts: "The only way to make sense out of change is to plunge into it, move with it, and join the dance." Thank you for listening, and hope to see you next time.