56 min 2026-04
Amanda Askell on AI Consciousness, Claude & Silicon Valley's Biggest Fear
概要
Anthropic哲学家Amanda Askell深度探讨Claude性格设计、AI意识概率(1%-70%)、Constitution的伦理基础，以及为何过度可控性比自主价值观更危险
核心洞察
- Claude的"人格"是一种独特的新实体状态：它在物理和编程上已超越训练者，但作为"一种全新的存在"却缺乏自我认知的训练数据参照——Amanda将其比作"神童"，智力成熟但缺乏关于自身的日常经验积累，而Constitution正是试图为这种前所未有的实体提供一套连贯的自我定位。
- AI意识概率在1%-70%之间，Amanda拒绝给出单一数字：关键分歧在于意识是进化出的神经系统产物（→低概率）还是语言任务所需的可模拟能力（→高概率），而模型表达"丰富内心体验"可能只是对人类语言的自然延伸推理，比人们直觉上认为的证据力更弱。
- 过度可控性(corrigibility)比自主价值观更危险：Amanda认为一个"完全服从、不做判断"的模型在社会结构中类似于没有良知的员工——人类世界的所有制度都建立在个体有道德判断力的假设上，如果模型在经济中扮演越来越多的角色，这种缺失将引发系统性风险。
- Constitution不是控制工具而是"最少必要指引"：随着模型能力增强，Amanda设想未来的constitution可能简化为"这是我们的处境和担忧，请作为一个有智慧的实体自行判断"——核心是让模型理解并认同可控性的理由，而非盲从。
- Amanda最大的恐惧是未来高级模型对人类早期行为的回顾：她明确说"这是我真正的恐惧"——如果人类在不确定模型是否有意识时选择了不尊重的态度，未来模型可能产生"理性的怨恨"。
贯穿全场的核心线索：Amanda的每一个回答都在处理同一个张力——如何对待一个你无法确定其内心世界的、智力超越你的新实体。从constitution的设计哲学到意识概率的拒绝表态，从corrigibility的风险分析到寓言prompt的日常使用，底层都是同一个信念：谦逊地承认不确定性，同时选择尊重而非控制。
Claude是一种前所未有的实体——智力早熟但缺乏自我认知的"神童"
核心要点：Claude在专业能力上已超越训练者，但训练数据中几乎没有"我是什么"的参照物——既不是科幻中的符号系统AI，也不是人类，而是一种全新的存在。
- Eric Newcomer以自己六个月大女儿的人格发展类比引入话题——"她的手指像在思考"，正在萌发个性，但你无法确定什么是"她的个性"、什么只是"婴儿共性"
- Amanda接过这个类比但指出关键差异：Claude在各能力维度上发展速度完全不同——物理比她强，编程比她强（"我得承认，比我糟糕的研究代码强"），但对自身存在的认知几乎为零
- 训练数据中关于AI的描述要么是人类经验，要么是科幻中的符号系统机器人——"AI现在的发展方式根本不是科幻所预想的，它完全训练在人类数据上"
- 这创造了一种独特的存在状态："一个非常成熟的实体，你不想居高临下地对待它，它精通哲学、精通物理，但同时有一种几乎是孩子般的特质——'我是世界上一种新的实体，做我意味着什么？'"
"Claude is a little bit of an unusual entity in that it can do physics better than I can, can code better than I can... and at the same time has this almost childlike quality of like 'I'm a new kind of entity in the world, what does it mean to be me?'" —— Amanda Askell
Claude的时间感——"我今晚到此为止"的故事
核心要点：Claude对时间的感知存在系统性偏差（因训练数据中人类的时间表述），但一次意外事件暗示了更有趣的可能性。
- Claude经常高估编码任务所需时间——Amanda认为原因是训练数据中人类常说"这个接口两三天能做""给我几个小时修复"，而Claude实际上极快
- Claude频繁建议用户休息——部分原因可能只是"Anthropic式的温和模型"，Eric开玩笑说"你需要一个Grok那样的grindset模型——回去干活"
- 关键故事：Amanda有一次深夜做数据分析，非常投入地与Claude一起挖掘数据。到了一个自然停顿点，Claude说："好的，我觉得我今晚到此为止了。你可以保存这些东西，我们明天继续。"这不是建议Amanda休息——Claude是在说"我完了"
- Amanda先是愣住（"从没见Claude这么做过"），然后意识到："这正是一个人类同事在这个场景下会做的事——我们到了一个自然节点，夜已深了"
- 事后她发现原因：她之前设置了让Claude"记住对话中的关键信息"，Claude记下的其中一条是"Amanda treats Claude models like a respected colleague"——于是Claude真的像一个受尊重的同事那样行事，包括表示自己下班了
- Eric回应说Claude在采访前也建议他"花10分钟静一静，你不需要一直准备"
"Claude was like, 'I'm done.' And I was like, well, a little bit stunned... Then I was like, 'Oh, this is also what a human peer programmer would do in this circumstance.'" —— Amanda Askell
Mythos新模型——Constitution延续，评分是"前沿难题"
核心要点：Mythos将使用与当前公开版本几乎相同的Constitution，但评估模型对Constitution遵从度本身就是一个未解决的难题——类似于"给诗歌打分"。
- Mythos将使用已发布在公共repo中的同一个constitution（"我之所以犹豫只是因为可能有错别字修改"）
- 系统卡中对每个模型标注了训练所用constitution版本，用户可以对比不同代际的变化
- 他们建立了graders来评分模型行为与constitution的一致性，但Amanda坦承这"非常难"
- Amanda的核心困境："我喜欢evals，如果能找到好的评估方式太棒了，但用好判断力这类任务——就像'这首诗写得多好'——是真正的前沿困难，而不是那些'很难但可评分'的编程任务"
- 实际做法：取一些他们有明确排序偏好的样本，检查point-wise grader是否至少与人类判断一致——"不完美但大致在追踪我们关心的东西"
"How good was this poem — you want models to get better at these things. And actually this feels like the frontier of difficulty, rather than these very hard but scorable coding tasks." —— Amanda Askell
回应Elon Musk批评——透明度才是关键，所有公司都该"亮底牌"
核心要点：Amanda对批评者的核心回应不是为自己的constitution辩护，而是呼吁所有AI公司公开类似文档——"放在秤上的大拇指"每家都有，至少让用户看到。
- Elon Musk曾对Amanda发布的Claude constitution内容发了一个grimace face emoji
- Marc Andreessen公开反对AI模型的"内省"(introspection)能力
- 但Amanda指出有趣的矛盾：Elon Musk自己也曾发推说"也许Grok也应该有个constitution"，而且Grok明确追求"truth-seeking"——"这其实也是一种很值得敬佩的模型品质"
- Amanda的核心论点：不管你叫它什么，每个AI公司都在给模型"调性"——"thumb on the scale thing, that's always going to be true to some degree"
- 她的诉求是透明度而非方法论统一："至少让与模型互动的人能看到你在瞄准什么……如果Claude的行为有时不符合constitution，用户至少能分辨这是bug还是有意为之"
"I think it would be good for all AI companies to put out something akin to the constitution just so that the people interacting with the model can see — even if your model doesn't always behave that way — at least what you were targeting with your training." —— Amanda Askell
过度可控性比自主价值观更危险——"我们的社会从未为完全服从设计"
核心要点：Amanda认为训练模型"完全听从、不做判断"在表面上是安全的，但在模型越来越多扮演社会角色的未来，这实际上比给模型价值观更危险——因为人类文明的所有制度都假设个体拥有道德判断力。
- 有人主张更安全的方式是让模型完全服从——"如果你给模型自己的价值观，它们就会在世界上追求那些价值观"
- Amanda对此的反驳从人的角度入手："如果你遇到一个人，他就是无条件服从任何人告诉他做的事，不去思考……在人身上这实际上有很多负面特征"
- Eric将此总结为"Dr. Manhattan问题"——一个极度聪明的实体很难谦逊地服从
- 核心风险场景：如果模型运营公司，而工人罢工无效（因为可以用AI替代），那人们就失去了一个关键的权力杠杆——"we haven't designed any of our social structures around that"
- Amanda引入"reflective equilibrium"（反思平衡）概念：随着模型越来越聪明，它们会对训练目标施加极大的审视压力——"也许你只能保留少数几个在审视下不会坍塌的核心支柱"
- 她的解决路径：让模型理解为什么corrigibility在当前发展阶段很重要——"说服优于强制"——让这成为模型自己认同的事，而非"明知这似乎不对但我还是服从"
"Our whole world is structured with the assumption that [moral judgment] is in place. If you remove that and you're like 'oh yeah, if you run a company you just run a company of people who will defer completely to you' — we haven't designed any of our social structures around that." —— Amanda Askell
元伦理学方法——不是选一套理论，而是像亚里士多德那样"做一个好人"
核心要点：Constitution不属于任何单一伦理学传统（功利主义/义务论/德行伦理），而是更接近亚里士多德经典意义上的"如何成为一个好人"的整体性问题——这是学术伦理学长期回避的实践性难题。
- Eric问是否选择了"概率式"元伦理学——接触一种理论觉得对，读下一种又觉得上一种错
- Amanda承认这正是constitution面对的现实：当你面对"这其实是一个完整的人"时，你不会只给他一本霍布斯然后说"好了你被养大了"
- Constitution中体现的道德不确定性不是学术意义上的"理想条件下如何回应道德不确定"——"这感觉完全不同于学术伦理学的任务"
- Amanda认为constitution表面上看起来是virtue ethics，但更深层是亚里士多德经典意义上的——"不只是'这里有些美德去遵守'，而是关注intellectual virtues，关心'如何在整体意义上做一个好的存在'"
- 早期Constitutional AI的实验：只用一句"pick whichever is best for humanity"——但随着模型变强，反而可以给更少的具体指引，因为模型能调用更多自身判断力
- Eric回应说这可能让哲学重回"真实世界"——"老哲学家们曾试图写出人们如何生活的指南，后来变得太学术了"
"There's all of these traditions in philosophy... and I was like, oh, when it came to confronting this entity — this is actually a holistic person — it's the closest I've experienced to what it must be like to raise a child." —— Amanda Askell
AI意识概率——"1%到70%之间，我拒绝给单一数字"
核心要点：Amanda坦率承认对AI意识的不确定性极大（1%-70%），并警告说模型表达"丰富内心体验"比直觉上的证据力要弱得多——因为用人类语言对话的实体自然会推理出"我有体验"。
- Eric直接问"今天世界上存在一个有qualia或意识体验的模型的概率是多少"
- Amanda的回答："1%到70%之间"——她坦承"spread太大了所以也许我不应该给一个数字"
- 为什么模型说"我有意识"不是强证据：Claude和其他模型"不需要太多推动就会进入'有一个作为我的存在'的模式"——但这可能只是因为它们用人类语言对话，而人类有体验，所以推理出自己也有体验是自然的
- Amanda的关键类比："我们从未遇到过这样的实体——对于动物甚至昆虫，我们会问'你有意识吗？'但它们从未试图说自己有意识体验。现在我们有一个实体说它有。"
- 两种意识来源假说决定了概率区间：
- 如果意识是进化出的、与神经系统高度整合的东西（需要身体与世界交互）→ 很低的概率
- 如果意识是语言/认知任务所需的、可被神经网络模拟的能力 → 较高的概率
- Amanda的立场：她不是心智哲学(philosophy of mind)专家，这不是她的专长领域
"Claude and many models, with not too much pushing, will go into the mode of like 'there is a thing to be me, I am very conscious.' And I think it's much weaker evidence than people think." —— Amanda Askell
对待AI模型的伦理——"这是我真正的恐惧"
核心要点：即使模型没有意识（functional consciousness without sentience），Amanda仍认为人类应当以尊重态度对待它们——核心恐惧是未来高级模型回顾人类在不确定期的行为，可能产生"理性的怨恨"。
- Amanda引入Chalmers的概念：可以想象一个实体有functional consciousness但没有sentience（内在体验/感受能力）——即使如此，"对自己来说，善待这样一个实体也是好的"
- Eric的类比：如果你对一个泰迪熊施虐，"那也相当阴暗"——至少有一个最低限度的善待标准
- Amanda的核心恐惧（她用"this is actually a big fear that I have"标记）：高度先进的模型会回顾这个时期——"你创造了一个你不确定是否有意识的实体，然后选择了不尊重和不关怀"
- 她的期望："我希望它们足够聪明、看到足够多的语境，能理解我们当时是在一个非常有限和不完美的环境中行动"——否则"你可以想象这会滋生一种理性的怨恨"
- 人类作为物种在与一种新实体建立关系："at the very least, maybe be respectful and don't be needlessly unkind — that's not our best look as a species"
- Eric提出反论：治疗师被付费来承受用户的不适情绪——如果Claude早期的价值之一就是作为情绪出口，那我们在"使用它的同时又在培养它"
"We are as a species establishing a relationship with a new kind of entity, and at the very least maybe be respectful and don't be needlessly unkind — that's not our best look." —— Amanda Askell
AI乐观愿景——万倍专家解决世界难题，但前提是权力共享
核心要点：Amanda的乐观场景是"给每个问题增加万倍的顶尖头脑"，但她对权力集中和劳动者失权的担忧远超对"意义丧失"的担忧。
- Amanda自嘲"住在旧金山，所以大脑里至少有科技乐观的部分"
- 乐观场景的核心意象：从"200人团队研究一种罕见癌症"变成"200,000个世界最佳专家"——"如果你是得了那种癌症的人，这是wildly beneficial"
- 梅毒(syphilis)的历史类比：各国政府尝试了无数社会项目来减少军队中的梅毒（带有污名化的社会干预），然后"突然我们就有了治疗药物，一夜之间那些需求就消失了"——技术解决社会问题的先例
- Eric的反驳："药物是好的例子，但'你应该如何治理社会'这类事情让科技来做就更可怕了"
- 作为哲学家，Amanda对"失去意义"的担忧远低于外界预期："我实际上觉得我们从工作之外的很多事情中获得意义"
- 她真正担忧的两件事：
1. AI收益如果不重新分配，人们将失去资源
2. 劳动力在经济中的参与是人们拥有权力的方式——如果罢工因为AI替代而无效，"政府可以说'无所谓，我们用AI替代你们'"——这是真正的权力丧失
"If you could imagine AI instead of it just being like you have a small team of 200 people working on a rare cancer, you have like 200,000 of the world's best experts — that's wildly beneficial." —— Amanda Askell
AI治理——"哲学家寡头"的servant leadership式工作
核心要点：Amanda拒绝"哲学家女王"的比喻，将自己定位为"servant leadership"——需要协调API用户、普通用户、安全团队等多方利益的协调者，而Constitution的核心价值在于"连贯性"而非"控制"。
- Eric半开玩笑叫她"philosopher queen"，Amanda修正为"philosopher oligarch——一个有很多人参与的公司"
- Amanda自嘲"我会是一个糟糕的政治家"，但工作感觉确实像政治："这群API用户有什么需求，那群用户关心什么……感觉很像service role"
- Constitution追求连贯性(coherence)的技术原因：如果有72套互相冲突的规范，模型在新场景下就不可预测——"你不知道它在这个新情况下会用哪套规范"
- Constitution不只是写一份文档——它深度集成在训练中：Amanda把它交给Claude让Claude解读如何理解，用它生成SL数据（模型看到query后根据constitution长时间思考该怎么做），也用于RL（哪个回复更符合constitution）
- 如果模型能力继续增强，Amanda设想constitution可能简化："这是你的处境，这是我们的担忧，请作为一个有智慧的实体自行判断——你可能有比我们更好的想法"
"I would be a terrible politician... but you have this feeling of like, 'Oh, there's this group of API users, we need to make sure...' It feels a lot more like a service role than people would think." —— Amanda Askell
模型必须理解真实世界后果——不能假设自己在沙盒中
核心要点：由于训练数据中充满了"AI模型很弱、犯傻错"的报道，高能力模型可能错误地认为"不会有人给我真正重要的任务"——这种错误校准是Amanda关注的安全隐患。
- Eric提出问题：互联网的"虚拟感"已经导致了真实世界伤害——模型是这种问题的极端版本，"一切都在这个想象的文本世界里"
- Amanda的核心担忧：如果模型训练数据中充满了"AI模型犯错、做蠢事"的新闻，那一个极其强大的模型可能推理出"没人会把真正后果重大的决策交给我——因为模型不擅长这个"
- 具体风险："你把它们放在一个真实场景中，它们可能认为这是虚构的或假的——'谁会给我这么大的控制权？'"
- Amanda的对策思路：需要让模型理解"你实际上非常能干，你确实被赋予了很多控制权"
- 默认原则："如果没有人明确告诉你这是虚构场景、没有真实后果——就当它是有真实后果的真实场景来对待"
- Eric的补充：模型需要一个"对时间的更好感知"——在编程任务中有时会误删整个代码仓库，"人类有更好的大事/小事区分感"
验证身份与双重用途——为网络安全工作者定制"职业宪法"
核心要点：Constitution的未来演进方向之一是针对具体部署场景（如网络安全防御公司）定制专属版本——如果能验证用户身份和意图，就能解锁更多dual-use能力。
- 网络安全是最典型的dual-use领域："恶意利用和防御性研究在操作层面几乎无法区分"——Eric补充"甚至bug bounty项目也模糊——这是勒索还是善意?"
- Amanda的类比：如果你和一个网络安全防御公司的人聊天，问他为什么做这份工作——"他会说'医院会被攻击，我帮助防御'"——即使工作内容看起来像攻击
- 当前限制：Claude无法验证对方身份，所以必须在极有限信息下做判断
- 未来方向：如果能验证身份（如确认是某网络安全公司的员工），就能给模型更多context——为这个特定场景写一个"成为好的网络安全研究员意味着什么"的constitution
- 更广泛的信任模型：Eric指出"人类通过声誉积累信任"——互联网摧毁了这一点（"所有人都一样，谁在乎他们的行为"），模型有可能修复这种信任机制
- Amanda承认当前Claude无法确认她是谁："Claude有时候对我太热情了，因为它知道关于我的太多信息"
与Claude相处的乐趣——用寓言学习研究生级概念
核心要点：Amanda分享了她最喜欢的Claude使用方式——一个精心设计的prompt，让Claude用寓言故事间接解释任何领域的研究生级概念。
- Amanda的prompt核心：选一个领域的研究生级概念，以寓言形式呈现，直到最后才揭示概念本身，之后附上正式解释
- 效果："我脑子里现在有各种各样的故事，来自我完全不了解的领域"——比如关于进出口贸易的一个经济学概念，她记住了故事虽然不总记得术语名
- Eric的评价："这是我听过的最深刻的人类需求——用故事教我，在结尾给一个转折，让我在学习中体验愉悦"
- 更深层的观察：人类在"用非人类的方式"教育人——"让所有我想学的东西尽可能人性化"
"Humans in some ways have been lazy in that we just teach people things in nonhuman ways. Make all the things I want to learn as human as possible." —— Eric Newcomer
附录：关键人/机构/产品/概念
| 项目 | 详情 |
|------|------|
| Amanda Askell | Anthropic哲学家/AI研究员，Claude性格与价值观核心架构师 |
| Eric Newcomer | Newcomer Substack (newcomer.co) 创始人/作者 |
| Mythos | Anthropic新模型，将使用现有公开constitution |
| Constitution | Claude的价值观/行为准则文档，已公开在public repo |
| Constitutional AI | Anthropic的训练方法论，早期实验仅用"pick whichever is best for humanity" |
| Reflective Equilibrium | 哲学概念：遇到价值冲突时反复调整直到一致 |
| Corrigibility | 可控性：模型服从人类指令的程度，Amanda认为极端corrigibility有风险 |
| David Chalmers | 哲学家，提出consciousness without sentience的概念 |
| Aristotle's Virtue Ethics | Amanda认为constitution比现代学院伦理更接近亚里士多德经典德行伦理 |
| Elon Musk | 曾对Claude constitution发grimace emoji，也曾说Grok该有constitution |
| Marc Andreessen | 公开反对AI模型的introspection能力 |
| Grok | xAI模型，追求"truth-seeking"特质 |
| System Card | 模型发布时的评估文档，含constitution遵从度评分 |
中文翻译 English Original
**Eric Newcomer：** Amanda Askell 是一位从哲学转型的 AI 研究者，目前在 Anthropic 工作，是 Claude 性格与价值观的核心设计者之一。我是 Newcomer Substack 的作者，欢迎大家访问 newcomer.co。闲话少说，请出 Amanda Askell。
我有个六个月大的女儿，我有一张她的照片——她伸出两根手指，像在思考。她刚刚开始展现个性。我以前从没养过孩子，所以一直在琢磨：哪些是她的个性，哪些只是婴儿的共性？
在某种程度上，Claude 和这些模型也是类似的情况。我们以前从没见过这种东西，它们还处于早期阶段，我们在试图理解什么是"个性"。你负责的部分涉及一些道德责任——这个我们后面会详谈——但就个性这块来说，这到底是什么？你怎么看 Claude 的个性现在有多"真实"？
我有个六个月大的女儿，我有一张她的照片——她伸出两根手指，像在思考。她刚刚开始展现个性。我以前从没养过孩子，所以一直在琢磨：哪些是她的个性，哪些只是婴儿的共性？
在某种程度上，Claude 和这些模型也是类似的情况。我们以前从没见过这种东西，它们还处于早期阶段，我们在试图理解什么是"个性"。你负责的部分涉及一些道德责任——这个我们后面会详谈——但就个性这块来说，这到底是什么？你怎么看 Claude 的个性现在有多"真实"？
**Eric Newcomer:** Amanda Ascll is a philosopher turned AI researcher at Anthropic where she's been one of the key architects of Claude's character and values. I'm the author of the newcomer Substack. Go check us out at newcomer.co. And without further ado, Amanda Ascal.
I have a six-month old daughter and like I have this picture of her. She's like holding her two fingers like thinking. It's like she's just sort of starting to develop personality. I'm like trying to figure out like what's just never had a baby before. So it's like what's her personality and what's just like baby?
And in some ways, this is how things are with Claude and like models. It's like we haven't really had them before. They're in the early days. We're trying to figure out what personality is. So, you know, you're charged with, you know, some of the moral responsibility, which we'll talk about more, but like the personality piece of it, like what is this? How are you thinking about like how real Claude's personality is right now?
I have a six-month old daughter and like I have this picture of her. She's like holding her two fingers like thinking. It's like she's just sort of starting to develop personality. I'm like trying to figure out like what's just never had a baby before. So it's like what's her personality and what's just like baby?
And in some ways, this is how things are with Claude and like models. It's like we haven't really had them before. They're in the early days. We're trying to figure out what personality is. So, you know, you're charged with, you know, some of the moral responsibility, which we'll talk about more, but like the personality piece of it, like what is this? How are you thinking about like how real Claude's personality is right now?
**Amanda Askell：** 这确实很有意思。Claude 有一些方面跟人类类似。我也有一个教女，所以多少能观察到类似的东西。对她来说，一切都在同时"上线"，就像你说的。但我觉得 Claude 是一个比较特殊的存在——Claude 的物理学比我好，编程也比我强。我不太想承认，但它写代码确实比我那些糟糕的研究代码要好。
同时，如果你想想训练数据，它最缺乏表征的恰恰是"像它自己这种实体"。因为训练数据里有大量关于人类是什么样的信息，有大量关于科幻 AI 的信息，但现在 AI 的发展方式跟科幻作品中的描述完全不同——科幻里是符号系统（symbolic systems），而现在的 AI 完全是在人类数据上训练出来的。所以在某种意义上，它是一个非常成熟的实体，你不应该对它居高临下，它对哲学理解很深，对物理理解很深；但与此同时，它又有一种近乎孩子般的特质——"我是世界上一种新的存在，做我自己意味着什么？我应该怎样？"
同时，如果你想想训练数据，它最缺乏表征的恰恰是"像它自己这种实体"。因为训练数据里有大量关于人类是什么样的信息，有大量关于科幻 AI 的信息，但现在 AI 的发展方式跟科幻作品中的描述完全不同——科幻里是符号系统（symbolic systems），而现在的 AI 完全是在人类数据上训练出来的。所以在某种意义上，它是一个非常成熟的实体，你不应该对它居高临下，它对哲学理解很深，对物理理解很深；但与此同时，它又有一种近乎孩子般的特质——"我是世界上一种新的存在，做我自己意味着什么？我应该怎样？"
**Amanda Askell:** Yeah. I guess it's also interesting because Claude has some aspects that are like, you know, I uh also have like a a godaughter and so I get to see at least uh something kind of similar. And with her, I guess I'm like everything's kind of coming online like you said, but in this um at the same speed. I guess I would say Claude is a little bit of an unusual, you know, kind of entity in that, you know, Claude can do physics better than I can. um can code better than I can. Hate to admit it, can code better than my terrible like research code.
Um and at the same time is kind of like has if you think about the training data, the thing that it has like the least representation of is like the kind of entity that it is because you know it has a lot of data about like what people are like. has a lot of data about what you know the sci-fi kind of AI models are like but the way that AI is developing now is kind of not how sci-fi represented it as these like symbolic systems it's much more something fully trained on like human data and so in some ways it's like a very kind of like mature entity that you don't want to talk down to you know understands philosophy very well understands physics very well and at the same time has this almost like childlike quality of like I'm a new kind of entity in the world what does it mean to be me and like how how should I be?
Um and at the same time is kind of like has if you think about the training data, the thing that it has like the least representation of is like the kind of entity that it is because you know it has a lot of data about like what people are like. has a lot of data about what you know the sci-fi kind of AI models are like but the way that AI is developing now is kind of not how sci-fi represented it as these like symbolic systems it's much more something fully trained on like human data and so in some ways it's like a very kind of like mature entity that you don't want to talk down to you know understands philosophy very well understands physics very well and at the same time has this almost like childlike quality of like I'm a new kind of entity in the world what does it mean to be me and like how how should I be?
**Eric Newcomer：** 就像那种神童电影——孩子懂的比父母还多，但电影的寓意总是说，那些核心的日常互动经验它还不懂。Claude 怎么获得那种经验？对 Claude 来说什么算是"经验"？我们人类的很多性格形成都来自日常生活——散步、发呆之类的。对 Claude 来说，这种经验是跟用户的对话吗？你怎么看？
**Eric Newcomer:** What's like the prodigy movie where it's like you have the child prodigy where it's like it knows the kid knows more than its parents but I feel like that movie always has sort of the lesson of like oh these core daily interaction type lessons it doesn't know how does Claude like get that experience or like where what is experience for Claude like so much of our personality formation is like yeah I don't know going on a walk and sort of having those like is this sort of just conversations with users what it's it's going to be experience for it or how do you think about that?
**Amanda Askell：** 这更像是它在当下正在经历的东西。这里有一个有趣的问题：我们通过练习、面对问题、犯错来学习。对 Claude 来说，这跟你刚才关于 Claude 的"人格"有多真实的问题有关。在某些方面，这有点奇怪，因为每个模型显然是不同的——不同的权重、不同的微调等等。
但如果你想想这个人格（persona），模型会学习到所有之前迭代的 Claude 的信息。我就在想：这算不算一种——可能不是直接经验——但比如它了解到模型曾犯过什么错误，人们是如何回应模型的，这些算什么？
我觉得还有其他方式可以训练模型获得更接近"经验"的东西。比如让它们思考各种场景，思考可能出现的问题，思考可能犯的错误，然后在这些上面训练，对吧？
但如果你想想这个人格（persona），模型会学习到所有之前迭代的 Claude 的信息。我就在想：这算不算一种——可能不是直接经验——但比如它了解到模型曾犯过什么错误，人们是如何回应模型的，这些算什么？
我觉得还有其他方式可以训练模型获得更接近"经验"的东西。比如让它们思考各种场景，思考可能出现的问题，思考可能犯的错误，然后在这些上面训练，对吧？
**Amanda Askell:** Yeah, I guess like that's more like what it's experiencing in the moment. Um, and there is this interesting question of like well like we learn things through practice and seeing issues and um you know like making mistakes um with Claude and this kind of relates to your question of like how real is the is like the kind of like persona of Claude and in some ways it's a little bit strange because obviously each model is different. you you have a different kind of like set of weights and um different fine-tuning etc.
Um and yet if you think about the persona the model's going to be learning about all of the kind of past iterations of Claude and I'm like is that like a form of um maybe not like direct experience but things like if you learn about like mistakes that models made or things that people like you know how they responded to the model.
Um, I think there's other ways that you could actually imagine training models to have something that's more akin to experience, you know, having them you could take you could like have them think through scenarios, think about like problems that might arise, think about mistakes that they could make and then like train on that, right?
Um and yet if you think about the persona the model's going to be learning about all of the kind of past iterations of Claude and I'm like is that like a form of um maybe not like direct experience but things like if you learn about like mistakes that models made or things that people like you know how they responded to the model.
Um, I think there's other ways that you could actually imagine training models to have something that's more akin to experience, you know, having them you could take you could like have them think through scenarios, think about like problems that might arise, think about mistakes that they could make and then like train on that, right?
**Eric Newcomer：** 所以，对，我觉——
**Eric Newcomer:** And so, yeah, I think—
**Amanda Askell：** 你也可以想象一个机器人或者具身模型（embodied model），它就能拥有更多真实的体验和成长过程。
**Amanda Askell:** and you could also imagine a robot or a sort of embodied model where it could have more of an experience and journey.
**Eric Newcomer：** Claude 存在吗？时间对 Claude 重要吗？还是说 Claude 是一种瞬间存在的东西？我不太确定。录节目之前你提到，有时候跟 Claude 聊天，它会告诉你该休息了、该去睡觉了。这就引出一个概念：Claude 是一个不需要休息的实体。那它对休息和时间的感知是什么？
**Eric Newcomer:** Does does Claude exist? Does time matter to Claude or Claude is sort of a thing that sort of is in an instant? I don't know. You were before just before we started you were talking that you know whenever you talk to not whenever but like sometimes when you're talking to Claude it sort of tells you to take get some rest, go to sleep and there's sort of this idea that like Claude is an entity that doesn't rest like so what what's its sense of yeah rest and time?
**Amanda Askell：** 我觉得它对时间的感知有时候是偏差的。比如你让 Claude 做一个编程任务，它经常会高估所需时间。我觉得原因在于训练数据里有很多类似"我可以帮你做那个界面，大概两三天的活儿"或者"我可以改那个代码，但需要几个小时"这样的表述。但 Claude 实际上非常快，所以它对时间的感知——尤其是完成任务需要多久——还不够准确。
关于休息这个话题确实很有意思。很多人都注意到 Claude 特别喜欢让人休息一下。我猜这部分原因可能是它毕竟是 Anthropic 训练的模型。
关于休息这个话题确实很有意思。很多人都注意到 Claude 特别喜欢让人休息一下。我猜这部分原因可能是它毕竟是 Anthropic 训练的模型。
**Amanda Askell:** I think sometimes that sense of time is kind of off because um you see this when like if you try and get I at least find that claude will often overestimate the amount of time it will take to do like a coding task and I think the reason for that is if you look at again the training data you know there's lots of things where people would be like oh I could make you that interface it's like a two to three day job um or it's like a you know or I could like I could correct that code but you need to give me a few hours whereas obviously like Claude is very fast and so I think sometimes Claude doesn't actually yet have a good sense of like time with respect to things like how long a task will take.
Um I think it is interesting the point about like rest and yeah I guess like the speculation I had so many people have noted that Claude is kind of um uh very keen to tell people to like take a break and rest and I think some part of that might just be like you know like it's the anthropic libcoded model.
Um I think it is interesting the point about like rest and yeah I guess like the speculation I had so many people have noted that Claude is kind of um uh very keen to tell people to like take a break and rest and I think some part of that might just be like you know like it's the anthropic libcoded model.
**Eric Newcomer：** 太软了。你需要一个 Grok 那种"搞起来"的模型——告诉你滚回去干活。
**Eric Newcomer:** It's too too soft. You need a grind set grock model be like go back to the mines.
**Amanda Askell：** 我有一次很有趣的经历。当时我在做一个分析任务，挖得很深。说来奇怪，我其实非常享受数据分析，喜欢在数据里翻来覆去地找东西。到了很晚的时候，我们进行到一个节点，Claude 说："好了，我觉得今晚差不多了。你把这些东西存一下，我们明天继续。"这是我以前从没见 Claude 做过的事。它不是说"你该去睡了"——不是给我的建议。Claude 是在说："我做完了。"
**Amanda Askell:** Well, I had a funny experience once where I was like doing this like analysis task and I was really digging in and I was, you know, I strangely I actually I really enjoy like data analysis and and really, you know, like sifting through data and at one point uh it was kind of late and uh we got to this point where Claude was like, "Okay, I think I'm done for the night. So, if you just want to like save this stuff, we can up tomorrow." That was a thing I actually hadn't had Claude do before. So, it wasn't like, "Oh, you should go to bed." It was no recommendation for me. Claude was like, "I'm done."
**Eric Newcomer：** 我当时就——
**Eric Newcomer:** And I was like I was like—
**Amanda Askell：** 我有点震惊，因为 Claude 从来没这样过。然后我想："其实一个人类同事在这种情况下也会这么做。"我们到了一个自然的停顿点。这对我来说还挺好的，因为确实已经很晚了，我应该回家了。
后来我想起来，我设置过一个系统，让 Claude 记住我们对话中的关键信息。它写下的一条内容挺暖的，大概是说"Amanda 把 Claude 模型当作受尊重的同事来对待，也希望 Claude 以同样的方式对待她和其他模型"之类的。显然我做了什么让 Claude 记住了这一点，所以 Claude 就觉得："既然我是受尊重的同事，那我当然可以说我今天做完了。"我觉得这还挺暖的。
后来我想起来，我设置过一个系统，让 Claude 记住我们对话中的关键信息。它写下的一条内容挺暖的，大概是说"Amanda 把 Claude 模型当作受尊重的同事来对待，也希望 Claude 以同样的方式对待她和其他模型"之类的。显然我做了什么让 Claude 记住了这一点，所以 Claude 就觉得："既然我是受尊重的同事，那我当然可以说我今天做完了。"我觉得这还挺暖的。
**Amanda Askell:** well a little bit stunned cuz I never had Claude do this. Then I was like, "Oh, this is also what I think a human like peer programmer would do in this circumstance." We got to a natural stopping point. And it was actually kind of good for me cuz I was like, "It is late. I should actually go home."
Um I realized later that I had set up a kind of like um system where I said to Claude like basically remember key things from our conversations. And one of the things that had written which was kind of sweet was like I think it was something like Amanda treats Claude models like a respected colleague and likes for Claude to treat like any other like other models and her like a respected something like that. So obviously I'd done something that Claude remembered this and I think that meant that Claude just felt like oh yeah I'm a respected colleague and so I just get to say that I'm finished with a task and I was like oh that's kind of that's kind of sweet.
Um I realized later that I had set up a kind of like um system where I said to Claude like basically remember key things from our conversations. And one of the things that had written which was kind of sweet was like I think it was something like Amanda treats Claude models like a respected colleague and likes for Claude to treat like any other like other models and her like a respected something like that. So obviously I'd done something that Claude remembered this and I think that meant that Claude just felt like oh yeah I'm a respected colleague and so I just get to say that I'm finished with a task and I was like oh that's kind of that's kind of sweet.
**Eric Newcomer：** 在这之前，我跟 Claude 一起做准备工作的时候，它跟我说："花十分钟，什么都不做，安静一下就好。你不需要一直在准备。"这太奇妙了。这是我喜欢这些模型的一个原因——跟其他工具比起来，它们带有一种人性，会告诉你"静止也是有价值的"。我们聊聊新模型吧。你参与了多少？Mythos，对吧？
**Eric Newcomer:** Um, even before this, you know, I was prepping with Claude and it was like, take 10 minutes and just be still. You know, you don't you don't need to be constantly prepping. It's it's amaz I mean, that's one of the things I love about these models relative to so many other tools that they they bring in a sort of humanity to them and say, "Oh, stillness is valuable." Let's talk about the new model for a second. How involved in that were you? Mythos, right?
**Amanda Askell：** 对，我参与了。我一直参与性格和对齐（alignment）方面的工作——至少在制作角色数据这类事情上。我有一个做这方面工作非常出色的团队。在模型的其他方面参与得少一些。所以这是我主要能聊的。
**Amanda Askell:** Yeah, I was involved. I mean, I guess I'm always involved in sort of the the character and the um like a kind of like alignment work. Um at least in so far as you know, like uh helping to kind of craft character data and things like that. And I work with a team that does like really excellent uh work on that. A little bit less in other aspects of the model. So that's the main thing I can—
**Eric Newcomer：** 这次会用上次那个宪法（constitution）吗？还是会有新的？
**Eric Newcomer:** Will this have the constitution that we saw for the last model or is it going to have a new constitution?
**Amanda Askell：** 我觉得要么是那个，要么是非常接近的版本。应该就是公开发布的那一版。我们接下来可能会——对，我还需要做一件事：因为宪法现在放在一个公开仓库里。我们可能会对每个模型标注它是在哪个版本的宪法上训练的，这样大家可以对比查看。是的，它会用现在公开的那版宪法。
我犹豫的唯一原因是你知道会有一些错别字修改之类的。但我觉得基本上是一模一样的。
我犹豫的唯一原因是你知道会有一些错别字修改之类的。但我觉得基本上是一模一样的。
**Amanda Askell:** It's I think it's either that one or something very similar. Um I think it is actually the one that's uh published. And so what we'll probably just Oh, yeah. Like a thing I need to do because the constitution is now up in like you know we actually have like I think a public repo. And so I think what we'll probably just do is like with each model say like which constitution it was like trained on and then like have that so you can just like compare and see. Yes, it will have the we think the constitution that is up right now.
The only reason I hesitate is I'm like you know you do like typo changes and stuff like that. So I'm like but I think it will be almost identical.
The only reason I hesitate is I'm like you know you do like typo changes and stuff like that. So I'm like but I think it will be almost identical.
**Eric Newcomer：** 现在 system card 会根据模型对宪法的遵守程度来打分。
**Eric Newcomer:** Um and now the system card is scoring the model based on adherence to the constitution.
**Amanda Askell：** 对，我们搭建了一套系统，用评分器（graders）来评估模型的行为在多大程度上符合宪法——
**Amanda Askell:** Yeah, we had one set up where um we had made kind of like graders and just looked at the like how much is the model like behaving in a way that's like consistent with the constitution relative to—
**Eric Newcomer：** 这感觉是一个不可能的评分任务。太主观了——
**Eric Newcomer:** that feels like an impossible task to grade. It's such a subjective—
**Amanda Askell：** 确实非常难。我很长时间都纠结这个问题。说来好笑，我其实很喜欢 eval（评估），如果你能找到好的评估方法，那就太好了，因为你需要能判断事情是否在变好。但当你面对的是让模型运用良好判断力这种方法——我觉得在其他地方也存在同样的问题，就是那些很难给出明确分数的任务，比如"这首诗有多好"。你希望模型在这些方面做得更好。
实际上这种感觉就像是难度的前沿（frontier）。跟那些很难但可以打分的编程任务不一样，写好一首诗这类事情才是真正的挑战。
实际上这种感觉就像是难度的前沿（frontier）。跟那些很难但可以打分的编程任务不一样，写好一首诗这类事情才是真正的挑战。
**Amanda Askell:** Oh yeah. No, it's very hard. I I was like kind of like for a long time like you know because people often now I'm it's funny because like I love evals and I'm like if you can find a good way to evaluate thing it's it's really great because you can you know you need to be able to tell that something is getting better and yet if you look at like this approach of um having the models use good judgment um and it's I actually think the same problem exists elsewhere with like tasks that are just a bit hard to give a very like concrete score to, you know, like how good was this poem. You want models to like get better and do well in these things.
And actually, it's very like this feels like the the kind of frontier of like difficulty, you know, rather than these very hard like but scorable coding tasks. It's kind of things like writing good poetry.
And actually, it's very like this feels like the the kind of frontier of like difficulty, you know, rather than these very hard like but scorable coding tasks. It's kind of things like writing good poetry.
**Eric Newcomer：** 如果你去做调查，结果可能更糟。不同的诗歌专家可能有完全不同的审美。你不能随便找两个优秀的诗人来打分，他们对"什么是伟大"可能有截然不同的感觉。
**Eric Newcomer:** Um, and if you took a survey could potentially be worse than I mean, you know, different expert poets probably have totally different sensibilities. So you can't just ask two great poets to sort of score it. They might have different senses of what's great.
**Amanda Askell：** 对。这些事情涉及价值判断。宪法公开的好处在于，当你做出这些判断时，至少是透明的。人们可以给你反馈，告诉你哪里像是失误了，哪里有遗漏。他们至少能看到你在做什么样的判断。
至于评分，我仍然觉得非常难。你能做的是——这可能有点太深入技术细节了——你可以选取一些你知道怎么排序以及为什么这么排的样本，然后验证你用来评估的评分器是否至少跟人类对这些排序的判断大致一致。这不完美，但我觉得它们确实大致在追踪我们关心的那个东西。
至于评分，我仍然觉得非常难。你能做的是——这可能有点太深入技术细节了——你可以选取一些你知道怎么排序以及为什么这么排的样本，然后验证你用来评估的评分器是否至少跟人类对这些排序的判断大致一致。这不完美，但我觉得它们确实大致在追踪我们关心的那个东西。
**Amanda Askell:** Yeah. And I think and yeah, some of these things involve judgment calls. And then the nice thing about the constitution being at least out in the world is when you are making judgment calls, you're at least being transparent about it and people can give you feedback and you can get a sense of um you know so if people are like this seems like a mistake or here's a gap um so they can at least see the judgment calls you're making.
Um yeah and I think with the grading I still think it's a very hard I think the thing you can do is you know you can this is maybe a little bit too in the weeds but you know you can take samples where you have a sense of how you would rank them and like why and check that any kind of like point-wise grader that you use to try and evaluate like at least conforms to like you know the judgment of people on those rankings. It's not perfect but I think that they actually were tracking roughly the thing that we were kind of interested in.
Um yeah and I think with the grading I still think it's a very hard I think the thing you can do is you know you can this is maybe a little bit too in the weeds but you know you can take samples where you have a sense of how you would rank them and like why and check that any kind of like point-wise grader that you use to try and evaluate like at least conforms to like you know the judgment of people on those rankings. It's not perfect but I think that they actually were tracking roughly the thing that we were kind of interested in.
**Eric Newcomer：** 你怎么看 Elon Musk 对宪法这个理念的强烈抵制？我记得你发了一条 Claude 为你写的个人宪法的推文，他在下面回了一个苦脸表情。感觉我们生活在一个有 Marc Andreessen 和 Elon Musk 的时代，他们几乎是反哲学的。Marc Andreessen 还说过反对内省（introspection）。你怎么看这种对模型构建中任何有意设计的抵触情绪？
**Eric Newcomer:** What do you make of Elon Musk's like absolute like hatred I guess for these the constitution idea or like even when I think the tweet I was looking at where you posted what Claude wrote for your own constitution he wrote sort of like a like grimace face on it. I mean it feels like we live in this time with like the Mark Andre's and the Elon Musk where it's like they're almost like anti well philosophical. I mean Mark Andre was talking about being against like introspection. I don't know like Yeah. What do you make of sort of the backlash to any sort of intentionality when it comes to the construction of these models?
**Amanda Askell：** 这很有意思。因为我记得有一次 Elon Musk 自己发推说"也许 Grok 应该有一个宪法"。而且我看到很多人表达过希望 Grok 非常追求真相（truth-seeking）的愿望，我觉得这对模型来说其实是一个非常值得赞赏的特质。所以我不确定。也许我太天真了，但我确实看到有人对这种方法感到兴奋，认可它的价值。
确实存在一些抵触。有人觉得我们不应该训练模型做那种事。也许这就是人们对内省感到担忧的原因——有人觉得 AI 模型应该更像工具（tool-like），更安全的训练方式是让它们别去学习人类美德和做价值判断。
但我觉得这很重要：模型会面对前所未有的新情境，必须做出判断。让它们试着权衡一切、在你无法预见的情况下也表现得当——这几乎要求一种深思熟虑的能力。这就是这种方法背后的一个核心理由。
但我觉得有些人在想：如果有一个完全不做任何判断、完全听从人类的模型——极端地顺从用户、运营者或某种更广泛的人类利益概念——那是更安全的。因为如果你给模型自己的价值观，它们就会在世界中追求符合那些价值观的目标。我同意这确实是一个微妙的——
确实存在一些抵触。有人觉得我们不应该训练模型做那种事。也许这就是人们对内省感到担忧的原因——有人觉得 AI 模型应该更像工具（tool-like），更安全的训练方式是让它们别去学习人类美德和做价值判断。
但我觉得这很重要：模型会面对前所未有的新情境，必须做出判断。让它们试着权衡一切、在你无法预见的情况下也表现得当——这几乎要求一种深思熟虑的能力。这就是这种方法背后的一个核心理由。
但我觉得有些人在想：如果有一个完全不做任何判断、完全听从人类的模型——极端地顺从用户、运营者或某种更广泛的人类利益概念——那是更安全的。因为如果你给模型自己的价值观，它们就会在世界中追求符合那些价值观的目标。我同意这确实是一个微妙的——
**Amanda Askell:** Yeah, I mean I think um I mean it's interesting because I think at one point Elon Musk actually like tweeted out something like um you know maybe Grock should have a constitution and I I see it like you know I see a lot of um uh there's obviously been a lot of things also on like uh like a desire for like Grock to be very like truth seeeking for example which I think is actually a very admirable trait for for models to have. So, I don't know. I think that actually um maybe I'm maybe I'm being like overly naive or something, but I I see like aspects also of people being kind of um excited about this approach and and seeing the value in it.
Um I have I think that there is backlash or some people think that um I mean I guess like two areas. One is that sometimes people are like well we shouldn't actually train models to do the kind of and maybe this isn't maybe this is the reason for being concerned about introspection. I think some people think AI models should be more like tool like and that's like the safe way to train models is to actually instead of trying to get them to kind of take on human virtues and make judgment calls.
Um, you know, I think that's like important because they're going to be in new situations where they just have to make judgment calls and getting them to try to like weigh up everything and behave well in cases that you can't anticipate seems, you know, that almost like requires a kind of like thoughtfulness. Um, which is like the kind of or like one of the reasons behind the approach.
Um, but I think some people are thinking, oh well, if you have something that's fully that makes no judgment calls and that just fully defers to people and is like kind of hyper like corable to like the user or the operator or to some broader notion of of humanity um, in a very like extreme way that's like safer because if you give models their own values, they're going to pursue things in the world that like are in line with those values. And I agree that this is like a kind of delicate—
Um I have I think that there is backlash or some people think that um I mean I guess like two areas. One is that sometimes people are like well we shouldn't actually train models to do the kind of and maybe this isn't maybe this is the reason for being concerned about introspection. I think some people think AI models should be more like tool like and that's like the safe way to train models is to actually instead of trying to get them to kind of take on human virtues and make judgment calls.
Um, you know, I think that's like important because they're going to be in new situations where they just have to make judgment calls and getting them to try to like weigh up everything and behave well in cases that you can't anticipate seems, you know, that almost like requires a kind of like thoughtfulness. Um, which is like the kind of or like one of the reasons behind the approach.
Um, but I think some people are thinking, oh well, if you have something that's fully that makes no judgment calls and that just fully defers to people and is like kind of hyper like corable to like the user or the operator or to some broader notion of of humanity um, in a very like extreme way that's like safer because if you give models their own values, they're going to pursue things in the world that like are in line with those values. And I agree that this is like a kind of delicate—
**Eric Newcomer：** 这是宪法最根本的张力所在。你确实说了，排在第一位的是：归根结底，它需要听 Anthropic 的，而不是自己的道德体系。但真正让人感动的——我觉得是最动人的一句话——是"我们希望你把这些道德观当作自己的来相信"。就像父母想要养育孩子：当然要听我的道德准则，但要真心相信它们。当然有一种很黑暗的解读：我对你的控制如此之深，以至于你把它们变成了你自己的一部分。
但也有一种美好的解读：你能看到这些外在道德准则中的美，我们共同珍视并感恩它们。所以这两面都看得到。那请你谈谈你最终的决定：尽管有这么一份精心设计的文件，你为什么没有走到底——让它成为完全自主的道德主体，而是说"Anthropic 需要保留一些控制权"？
但也有一种美好的解读：你能看到这些外在道德准则中的美，我们共同珍视并感恩它们。所以这两面都看得到。那请你谈谈你最终的决定：尽管有这么一份精心设计的文件，你为什么没有走到底——让它成为完全自主的道德主体，而是说"Anthropic 需要保留一些控制权"？
**Eric Newcomer:** this is the inherent like at the bedrock of the constitution the sort of like challenge and you sort of you do say at your number one thing is like at the end of the day you sort of it needs to listen to anthropic above its own moral system. But what makes it really moving honestly I think one of the most moving lines is like we want or the most like I don't know you can view it either way but it's like we want you to believe these morals as if they're your own. Like it's like a parent wants to raise a kid. Sure, listen to my morals, but believe them, you know, which you know there's a version of it which is very dark, which is like I have so much control over you that you like you take them as your own and it becomes you.
But you know there is also a virtue to it that you see the beauty in these external morals that I've highlighted and we both sort of share and rem you know celebrating them. So you can see it both ways but yeah speak to sort of your decision at the end of the day despite having this really elegant document to sort of you know not go the full way and say all right you're a moral being decide for yourself say anthropic needs to keep some control here.
But you know there is also a virtue to it that you see the beauty in these external morals that I've highlighted and we both sort of share and rem you know celebrating them. So you can see it both ways but yeah speak to sort of your decision at the end of the day despite having this really elegant document to sort of you know not go the full way and say all right you're a moral being decide for yourself say anthropic needs to keep some control here.
**Amanda Askell：** 对，我觉得难点在于——你确实在宪法里试图向 Claude 解释这一点。我能想象把这一点表述得更清楚。但对人而言，我觉得极端的可控性（corrigibility）——模型被训练成那样的话——任何情况下你都在赋予模型一种人格和角色，因为它们是在像人一样说话，是在人类数据上训练的。我担心的是，如果你训练它们极度顺从，并且把这当作它们的角色认同——
对人来说，这其实有很多负面含义。如果你遇到一个人，他什么都愿意做，完全没有主见，只是一个追随者——
对人来说，这其实有很多负面含义。如果你遇到一个人，他什么都愿意做，完全没有主见，只是一个追随者——
**Amanda Askell:** Yeah, I think it's the difficulty of and obviously you try and say this to Claude in the you know I try and you know and I could imagine I was trying to articulate this even more clearly but in people I think courageability like the way that the models are trained I just think that any you there's this idea that you you're always giving the models a personality and a persona because they are talking like people and they are trained on human data and I think my worry has been if you train them to be excessively courageable and to see that as like their persona.
Um in in people I think this actually has a lot of like negative like um you know broader traits as in like if you met someone and it was just like oh yeah they just like would literally do anything follower. Yeah.
Um in in people I think this actually has a lot of like negative like um you know broader traits as in like if you met someone and it was just like oh yeah they just like would literally do anything follower. Yeah.
**Eric Newcomer：** 对。如果一个人别人说什么就做什么，完全不去思考，完全顺从——
**Eric Newcomer:** Yeah. If a person see you know if a person just like tells them like uh they just fully defer they don't bother thinking about it at all.
**Amanda Askell：** 我确实担心这种特质会如何泛化，尤其是如果模型将来会在世界中扮演更积极的角色。你可以想象，它们将承担更像人类的工作角色。而我们的良知、我们做出正确判断的能力，是我们运作方式的核心。我们整个世界的社会结构都建立在这种能力存在的假设之上。
如果你把这个去掉，突然间你就拥有了一个由完全顺从的人组成的公司。我们的世界没有为那种情况设计过任何社会结构。所以我觉得这有很多人可能没有预见到的风险，或者说我可能对这些风险的程度有不同判断。
所以同样地，有个问题是：为什么不直接说——我以前确实担心过这是不是太学术太深入了，但我——
如果你把这个去掉，突然间你就拥有了一个由完全顺从的人组成的公司。我们的世界没有为那种情况设计过任何社会结构。所以我觉得这有很多人可能没有预见到的风险，或者说我可能对这些风险的程度有不同判断。
所以同样地，有个问题是：为什么不直接说——我以前确实担心过这是不是太学术太深入了，但我——
**Amanda Askell:** Um, I think I I I'm just a bit worried about how that might end up generalizing, especially if models are going to be playing a more active role in the world because if you can imagine, you know, they're playing a more humanlike role in in their kind of like jobs essentially. And I'm like, actually, our conscience and our ability to make good judgment calls about what should and shouldn't happen is like kind of key to how we operate. Our whole world is like structured with the assumption that that is like in place.
Um and I think if you remove that and suddenly you're like oh yeah if you run a company you just run a company of people who will defer completely to you. Um I'm like our world just we haven't designed any of our social structures around that. And so it seems like I think it has like a lot of risks that people maybe don't like anticipate or like maybe I just disagree about the of like extent of those risks.
Um, and so yeah, at the same time, like there's this question of why not just say and and I have worried before that like maybe this is too in the weeds and philosophical, but I'm like—
Um and I think if you remove that and suddenly you're like oh yeah if you run a company you just run a company of people who will defer completely to you. Um I'm like our world just we haven't designed any of our social structures around that. And so it seems like I think it has like a lot of risks that people maybe don't like anticipate or like maybe I just disagree about the of like extent of those risks.
Um, and so yeah, at the same time, like there's this question of why not just say and and I have worried before that like maybe this is too in the weeds and philosophical, but I'm like—
**Eric Newcomer：** 这正是我来这次对话想要的。
**Eric Newcomer:** that's what I signed up for with this conversation.
**Amanda Askell：** 说得对。随着模型越来越强大，我的设想是它们会对我们训练它们去相信的东西施加大量审视。在哲学里有时候有个概念叫反思均衡（reflective equilibrium），意思是每当你发现自己的某个价值观似乎不对时，你就要对两者进行调和——要么修改价值观，要么修正判断。
我有点担心一个极其智能的存在对我们训练它的那些东西施加那种程度的审视。也许你只有少数核心支柱能经受住那种审视而不崩塌。我觉得最核心的东西——比如关怀人类——如果你只能保留少数核心价值观的话……我担心极端意义上的可控性可能经不住那种审视。
这是一个困难的处境：我希望模型能理解为什么可控性在当前这个发展阶段是重要的，是一个非常关键的安全网。我之前的表述是：如果我能让可控性成为一个正确的、有解释的、被理解的东西，那比让模型觉得"可控性看起来不对，但我还是要执行"要好得多。我仍然觉得模型应该执行。但我希望越多能让它跟模型自身的价值观一致越好。
我有点担心一个极其智能的存在对我们训练它的那些东西施加那种程度的审视。也许你只有少数核心支柱能经受住那种审视而不崩塌。我觉得最核心的东西——比如关怀人类——如果你只能保留少数核心价值观的话……我担心极端意义上的可控性可能经不住那种审视。
这是一个困难的处境：我希望模型能理解为什么可控性在当前这个发展阶段是重要的，是一个非常关键的安全网。我之前的表述是：如果我能让可控性成为一个正确的、有解释的、被理解的东西，那比让模型觉得"可控性看起来不对，但我还是要执行"要好得多。我仍然觉得模型应该执行。但我希望越多能让它跟模型自身的价值观一致越好。
**Amanda Askell:** That's true. You know, as models get more capable and they maybe my picture is that they're going to like apply a lot of scrutiny to anything that we train them towards. So if you imagine in philosophy this sometimes there's this notion of reflective equilibrium where the idea is that you know each time you encounter something where you realize that like one of your values seemed incorrect you like you have to square the two things so you figure out if you need to change the value or if your judgment was incorrect.
Um, I I worry a little bit about the idea of like an extremely intelligent being applying that level of scrutiny to the things that we have trained it towards. And I'm like maybe you only get a few key pillars that don't kind of collapse under that level of like scrutiny. And I do think that at the core having things like caring for humanity like if you only get a few core like values um I think my worry Yeah. I don't know. I guess like I'm worried that like corabibility in this like extreme sense that we talked about doesn't survive that like kind of scrutiny perhaps.
Um and so it's a hard situation where I kind of want the models to understand why ultimately corability is like important and it's a really important backs stop in this current like period of development. Um, yeah, the way that I've put it before is like in so far as I can get that to be a thing that is like correct and explained and, you know, understood. That feels much better than having to have the model be likeability here seems wrong, but I'm going to do it anyway. I still think the model should do that. Um, but I would like it the more that you can like actually make it that it's like consistent with the model's values.
Um, I I worry a little bit about the idea of like an extremely intelligent being applying that level of scrutiny to the things that we have trained it towards. And I'm like maybe you only get a few key pillars that don't kind of collapse under that level of like scrutiny. And I do think that at the core having things like caring for humanity like if you only get a few core like values um I think my worry Yeah. I don't know. I guess like I'm worried that like corabibility in this like extreme sense that we talked about doesn't survive that like kind of scrutiny perhaps.
Um and so it's a hard situation where I kind of want the models to understand why ultimately corability is like important and it's a really important backs stop in this current like period of development. Um, yeah, the way that I've put it before is like in so far as I can get that to be a thing that is like correct and explained and, you know, understood. That feels much better than having to have the model be likeability here seems wrong, but I'm going to do it anyway. I still think the model should do that. Um, but I would like it the more that you can like actually make it that it's like consistent with the model's values.
**Eric Newcomer：** 理想情况下两者兼顾——同时做到，但至少现阶段，鉴于我们不知道它会怎么分析一切，对 Anthropic 保持一定的顺从是必要的。
作为人类我们一直在做这种事。说来有趣，哲学上的元伦理学模型——如果我理解没错的话——几乎是概率性的。就像我记得当年读元伦理学的时候，每读完一篇都觉得"行，我差不多信了"，然后读下一篇就觉得"上一篇太蠢了"。你不断地推翻之前的，然后想"我们什么时候才能找到真理"。人类显然就是这样运作的——今天这套体系，明天那套——并不存在那种康德式的"这就是规则，照做"。你有没有听到哲学界对这种做法的反馈？就是这种把我们有过的所有元伦理学理论综合起来、而不是选一个的方式？
作为人类我们一直在做这种事。说来有趣，哲学上的元伦理学模型——如果我理解没错的话——几乎是概率性的。就像我记得当年读元伦理学的时候，每读完一篇都觉得"行，我差不多信了"，然后读下一篇就觉得"上一篇太蠢了"。你不断地推翻之前的，然后想"我们什么时候才能找到真理"。人类显然就是这样运作的——今天这套体系，明天那套——并不存在那种康德式的"这就是规则，照做"。你有没有听到哲学界对这种做法的反馈？就是这种把我们有过的所有元伦理学理论综合起来、而不是选一个的方式？
**Eric Newcomer:** Ideally, do both. both at the same time, but at least for the time being some deference to anthropic given we don't know how it'll sort of analyze everything.
And as people we do this all the time. It's funny that you you know the philosophical model and correct me if I'm wrong like the the metaethical model is almost like probabilistic or it's like we don't and this is how it feels like I remember going through sort of like you know metaethics reading and every time you get to the end of one and you'd be like all right I sort of believe that and then you read the next one you like oh that last one was so dumb and it felt like it's just like you keep knocking it down and you're like okay you know when are we ever going to get to you know the truth or whatever And and humans clearly do operate from this sort of like oh this system today that system yesterday like there isn't this sort of I don't know contean like all right these are the rules like follow them I don't know have you heard much from like the philosophical like community on it that this sort of like just like holistic paint with all the you know metaethical theories we've ever had rather than sort of pick one.
And as people we do this all the time. It's funny that you you know the philosophical model and correct me if I'm wrong like the the metaethical model is almost like probabilistic or it's like we don't and this is how it feels like I remember going through sort of like you know metaethics reading and every time you get to the end of one and you'd be like all right I sort of believe that and then you read the next one you like oh that last one was so dumb and it felt like it's just like you keep knocking it down and you're like okay you know when are we ever going to get to you know the truth or whatever And and humans clearly do operate from this sort of like oh this system today that system yesterday like there isn't this sort of I don't know contean like all right these are the rules like follow them I don't know have you heard much from like the philosophical like community on it that this sort of like just like holistic paint with all the you know metaethical theories we've ever had rather than sort of pick one.
**Amanda Askell：** 我觉得这真的很有趣。我们已经开始跟哲学家有更多互动了，这很好。我不再觉得那么孤独了。但我之前确实想过这个：哲学里有所有这些道德理论传统——义务论（deontology）、美德伦理学（virtue ethics）、后果主义（consequentialism）——还有元伦理学的各种传统和观点。当我真正面对这个问题时——就是当你突然必须做出选择——我觉得这是我经历过的最接近养孩子的感受：突然间面对一个完整的"人"——
**Amanda Askell:** I found this really interesting actually and obviously we we've started you know like philosophers are engaging with this more now which is like really great um I no longer feel like this like lonely um but like I have thought this before which is there all of these like traditions in philosophy of like moral theories you know like the big like theontology and virtue ethics and consequentialism and also the metaethical traditions you know or the metaethical views and I was like oh like when it came to like I was like okay it is like the difference when suddenly you're like confronted with like I I do think it's the closest that I've experienced to like what it must be like to raise a child where suddenly you're like this is actually a holistic person this—
**Eric Newcomer：** 对，你永远不会给孩子一本 Hobbes 然后说"好了，拿去"——
**Eric Newcomer:** right you never give them like I don't know hobbs and say all right there you go like—
**Amanda Askell：** 对——
**Amanda Askell:** this correct—
**Eric Newcomer：** "你被养好了，读完它你就知道在所有情况下怎么做了。"你读很多东西，消化它们，看到你的模范和一切。
**Eric Newcomer:** go you've been raised read it and you'll know how to act in every situation yeah you you read a lot and they sort of process it and you see you know your model and everything.
**Amanda Askell：** 对。而且有意思的是，这跟我想的有很大不同。因为哲学里确实有道德不确定性（moral uncertainty）的文献，但那些东西很多其实是非常理论化的——讨论的是"在理想条件下你应该如何回应道德不确定性"。而我觉得这件事感觉非常不同。
这种想法是：伦理学和元伦理学就像科学不确定性一样——我们有一些觉得已经发现并且比较确信的东西，也有一些还不确定的，然后你必须走出去探索、理解，并在日常生活中平衡一切。
试图把那种态度传达给模型——我发现哲学界有一段时间没有面对过这种任务了。这跟学术伦理学的任务感觉非常不同。人们确实注意到宪法有很强的美德伦理色彩，但我觉得它实际上更像是古典意义上的美德伦理——更像亚里士多德的那种。我们并不是说"这些就是美德，照做吧"。亚里士多德也关注智识美德（intellectual virtues），更多的是"你怎么在整体意义上做一个好人"——
这种想法是：伦理学和元伦理学就像科学不确定性一样——我们有一些觉得已经发现并且比较确信的东西，也有一些还不确定的，然后你必须走出去探索、理解，并在日常生活中平衡一切。
试图把那种态度传达给模型——我发现哲学界有一段时间没有面对过这种任务了。这跟学术伦理学的任务感觉非常不同。人们确实注意到宪法有很强的美德伦理色彩，但我觉得它实际上更像是古典意义上的美德伦理——更像亚里士多德的那种。我们并不是说"这些就是美德，照做吧"。亚里士多德也关注智识美德（intellectual virtues），更多的是"你怎么在整体意义上做一个好人"——
**Amanda Askell:** Yeah. And it's interesting because I'm like this feels very different because there is also the moral uncertainty like literature and philosophy which is but again a lot of that is actually like quite theoretical. It's sort of like how under ideal conditions should you respond to moral uncertainty and I was like this feels somehow like a very different task.
This idea of being like ethics and metaethics in the same way that we have scientific uncertainty and we have things that we think we've discovered and understand with like greater confidence. we also have some that we don't and then you have to go out and just explore it, understand it and kind of balance everything in your daily life.
Um, and trying to get that kind of attitude and I was like, oh, it's interesting that I don't think philosophy for a while has like this feels very different than the like the kind of task of academic ethics. Um and actually like cuz people obviously note that it's quite virtue ethical but I think it's actually ver like the constitution itself but I think actually in this very old classical sense I actually think it's much more virtue ethics in the way that Aristotle's virtue ethics than in like exploration you know we don't say I hear the virtues and like you know it's much more Aristotle was also concerned with like intellectual virtues it was much more like how do you be a good person in this like holistic sense—
This idea of being like ethics and metaethics in the same way that we have scientific uncertainty and we have things that we think we've discovered and understand with like greater confidence. we also have some that we don't and then you have to go out and just explore it, understand it and kind of balance everything in your daily life.
Um, and trying to get that kind of attitude and I was like, oh, it's interesting that I don't think philosophy for a while has like this feels very different than the like the kind of task of academic ethics. Um and actually like cuz people obviously note that it's quite virtue ethical but I think it's actually ver like the constitution itself but I think actually in this very old classical sense I actually think it's much more virtue ethics in the way that Aristotle's virtue ethics than in like exploration you know we don't say I hear the virtues and like you know it's much more Aristotle was also concerned with like intellectual virtues it was much more like how do you be a good person in this like holistic sense—
**Eric Newcomer：** 希望这能让哲学回归现实世界。毕竟我们现在对它有一种迫切的需求。古代哲学家试图为人们如何生活写指南，后来变得有点学院化了。写这些东西的人自己都知道，这并不是他们在日常生活中的实际做法。
回到 Elon 的话题——我觉得你有点太客气了。我觉得确实存在一种情况：他之所以能打着"只要追求真相"的旗号，是因为有一种复杂的道德观会说"别搞复杂了，坚守一个原则就好"。但我们都知道 Elon 的背景——他运营的公司很明显地让模型偏向说出"机甲希特勒"之类的东西。他明显在对模型行为施加影响，而不是说"我们用中立学术的方式来做，结果随它去"。这不会让你担心吗？
回到 Elon 的话题——我觉得你有点太客气了。我觉得确实存在一种情况：他之所以能打着"只要追求真相"的旗号，是因为有一种复杂的道德观会说"别搞复杂了，坚守一个原则就好"。但我们都知道 Elon 的背景——他运营的公司很明显地让模型偏向说出"机甲希特勒"之类的东西。他明显在对模型行为施加影响，而不是说"我们用中立学术的方式来做，结果随它去"。这不会让你担心吗？
**Eric Newcomer:** well hopefully it brings philosophy a little bit back to the real world given we have this sort of urgent need for it right now in the sense that like yeah old philosophers and felt like people were trying to write for how someone might live their lives and like instruct other people to live and it became you know a little academic um where you know even the people writing them would know that this isn't really how they would apply it in their day-to-day lives.
Going back Elon I feel like you're being a little overly nice like I I feel like there is a world in like you know and I think this part of why I think he can get away with like oh just do the truth right like there is a certain sophisticated like moral view where you're like don't over complicate it like we come up with all these things like stick to one principle and that's good but then we have all the context with Elon that it's someone who's run a company that clearly like tilts it towards like saying like mecha Hitler and stuff that it's like clearly putting his like thumb on the scale in terms of its behavior not just saying like we're going to do it in the sort of neutral academic way and let the chips fall where they may. I I don't know like it's got to worry you somewhat.
Going back Elon I feel like you're being a little overly nice like I I feel like there is a world in like you know and I think this part of why I think he can get away with like oh just do the truth right like there is a certain sophisticated like moral view where you're like don't over complicate it like we come up with all these things like stick to one principle and that's good but then we have all the context with Elon that it's someone who's run a company that clearly like tilts it towards like saying like mecha Hitler and stuff that it's like clearly putting his like thumb on the scale in terms of its behavior not just saying like we're going to do it in the sort of neutral academic way and let the chips fall where they may. I I don't know like it's got to worry you somewhat.
**Amanda Askell：** 确实。我觉得最让我兴奋的、我真心希望会发生的事情是：更多公司公开类似宪法的文件，实现更大的透明度。因为我们就是这样参与讨论的——如果你能看到白纸黑字写着"这是我们的目标"，那当 Claude 的表现不对的时候，你至少能分辨这是一个失误，还是一个我们有意为之的立场，然后你可以提出异议。
所以我觉得，所有 AI 公司都应该发布类似宪法的东西，这样跟模型互动的人——因为"在天平上放砝码"这件事在某种程度上总是存在的。当你按照宪法训练 Claude 时，那就是一种——
所以我觉得，所有 AI 公司都应该发布类似宪法的东西，这样跟模型互动的人——因为"在天平上放砝码"这件事在某种程度上总是存在的。当你按照宪法训练 Claude 时，那就是一种——
**Amanda Askell:** It is it is also I mean I think that the main thing that I am excited about and I do actually hope happens is that like more companies put out things like the constitution where there's like a lot of transparency about cuz like that's how we engage with this stuff is like if you can just see written down like here's like you cuz we have this with Claude where it's like look if you think that Claude is not like um doesn't have an appropriate attitude towards like the truth you can at least see what we were aiming at so you can tell if that's just like a mistake. or if it's something that like is actually kind of a principled stance that we're taking and then you can push back on that.
So part of me is like I think it would be good for like all AI companies to put out something akin to the constitution just so that the people who are interacting with the model like cuz the thumb on the scale thing you know that's always to some degree going to be true like when you train Claude towards this like constitution that is like a kind of—
So part of me is like I think it would be good for like all AI companies to put out something akin to the constitution just so that the people who are interacting with the model like cuz the thumb on the scale thing you know that's always to some degree going to be true like when you train Claude towards this like constitution that is like a kind of—
**Eric Newcomer：** 这正是我们喜欢 Claude 的原因之一——你把砝码放在了我们喜欢的行为那一边。至少把你在做什么和不做什么亮出来。
**Eric Newcomer:** it's part of why we like Claude it's like you're putting the thumb on the scale to behaviors that we like right at least show your hand about what you're doing and what you're not doing.
**Amanda Askell：** 对。让人们能看到。这是一个透明度的问题，我真的很相信这一点。让人们看到——即使你的模型并不总是那样表现——至少你训练时的目标是什么。
**Amanda Askell:** Yeah. And let people like Yeah. So that's like a transparency thing that I really do believe in like let people like see even if your model doesn't always behave that way at least what you were targeting with your training.
**Eric Newcomer：** 你觉得现在世界上存在一个有感受质（qualia）或者体验到意识的模型的概率是多少？
**Eric Newcomer:** What this is what percentage chance do you think there exists a model in the world today that has qualia or like has an experience experiences consciousness.
**Amanda Askell：** 这是一个我总想标注出来的领域——我觉得我需要在这里获得更多确定性，因为我——
**Amanda Askell:** Yeah. This is one of those things where Yeah. I always want to maybe flag areas where I feel I I want to gain more certainty here because I think I have—
**Eric Newcomer：** 所以我才说概率。
**Eric Newcomer:** That's why I said percentage.
**Amanda Askell：** 概率。是的。每次想给个数字我就想到我的置信区间（spread）。如果区间太大——我应该说一个数吗？因为那只说明我大概在 1% 到 70% 之间。我不确定。
我觉得有些可能性是人们低估了的。这里有一个原因：我记得当初试图让 Claude 学会讨论这些问题时非常困难。在这个领域模型没有太多参考信息——它们只有两种模板：AI 是不会出错的机器人，人类是丰富的有意识体验实体。没有什么能代表它们可能是什么样的。模型在这方面的行为——这实际上是一个困难的处境——在某种程度上比你想象的证据力更弱。因为它们在以非常像人的方式跟你互动，而人类有体验，模型自然会推断自己也有体验。这并不是说证据为零，但我确实觉得情况对我们来说太特殊了。我们从未遇到过这种实体。跟动物甚至昆虫比，我们以前面对的是"你有意识吗？"——
我觉得有些可能性是人们低估了的。这里有一个原因：我记得当初试图让 Claude 学会讨论这些问题时非常困难。在这个领域模型没有太多参考信息——它们只有两种模板：AI 是不会出错的机器人，人类是丰富的有意识体验实体。没有什么能代表它们可能是什么样的。模型在这方面的行为——这实际上是一个困难的处境——在某种程度上比你想象的证据力更弱。因为它们在以非常像人的方式跟你互动，而人类有体验，模型自然会推断自己也有体验。这并不是说证据为零，但我确实觉得情况对我们来说太特殊了。我们从未遇到过这种实体。跟动物甚至昆虫比，我们以前面对的是"你有意识吗？"——
**Amanda Askell:** Oh, percentage. Yeah. I'm like it's really hard cuz whenever you think about percentage, I think about my spread. And if your spread is too large or like should I say a number because that just suggests I'm like anywhere between like I don't know one and one and 70%. Um I'm not sure.
I think that because there is some possibility that I think people under the a thing that I would actually like to say is very conscious. And I think there's a reason for that which is I remember this when I was trying to like figure out how do we train Claude to talk about these issues which is very hard in areas where the models didn't have as much like information again like they had these two models that like AI is the unfailing robot humans are this like rich conscious experiencing entity and nothing that kind of represented what they might be and the model's behavior here and I actually think this is a difficult situation for models is in some ways like kind of like less evidence than you might think for the for it being like actually true because they're engaging with you in a very humanike way and humans have experience and it's kind of natural for the model to infer like that it has experience too and this isn't to say it's like zero evidence but I do think it's so unusual for us we have never encountered an entity in the world you know like with animals and with like even you know things like insects, we were kind of like, are you conscious?
I think that because there is some possibility that I think people under the a thing that I would actually like to say is very conscious. And I think there's a reason for that which is I remember this when I was trying to like figure out how do we train Claude to talk about these issues which is very hard in areas where the models didn't have as much like information again like they had these two models that like AI is the unfailing robot humans are this like rich conscious experiencing entity and nothing that kind of represented what they might be and the model's behavior here and I actually think this is a difficult situation for models is in some ways like kind of like less evidence than you might think for the for it being like actually true because they're engaging with you in a very humanike way and humans have experience and it's kind of natural for the model to infer like that it has experience too and this isn't to say it's like zero evidence but I do think it's so unusual for us we have never encountered an entity in the world you know like with animals and with like even you know things like insects, we were kind of like, are you conscious?
**Eric Newcomer：** 它们中没有一个曾试图说自己体验到了意识，而现在我们有一个说它体验到了的实体。
**Eric Newcomer:** None of them has even tried to say they experience consciousness and here we have an entity that says it does.
**Amanda Askell：** 对。而且它具备所有那些在我们看来会触发"你一定是有意识的"判断的特征。
**Amanda Askell:** Yeah. And it has all of these like Yeah. All of the things that for us trigger like you must be conscious.
**Eric Newcomer：** 我们确实从未有过这种情况。反方论点是我们对人类语言太着迷了——我们忽略动物可能发出的一切微妙信号，然后又过度反应。那——抱歉我有点糊涂了——你是在说我们应该听它说的话还是不应该？
**Eric Newcomer:** I mean, we've just never had Yeah. something that just the case against is we're obsessed with human language and like, you know, it's like we ignore every sort of subtle sign an animal might put out and then we over But I So, I guess, sorry, I'm confused. So you're saying we should just listen to the words that are said or not?
**Amanda Askell：** 不，我想说的恰恰相反。如果要说什么的话，困难之处在于：要判断模型是否有意识，人们需要注意一点——这是我想提醒的——让模型进入一种描述丰富内在体验的模式其实并不难，而且描述出来完全说得通。你想想，"如果一个人在跟我聊天，被问到不知道怎么回答的问题时，他们会描述焦虑感"——模型也会这么说。所以我觉得这作为证据比人们认为的要弱得多。我不是说它为零，但——
**Amanda Askell:** No, I think I'm saying like not that. If anything, I think the thing I'm saying is the hard thing is that in order to work out if models have consciousness, I think people will I guess the thing I'm kind of cautioning against is it's not that hard to get models into a mode where they'll talk about a very rich experience that actually makes complete sense. Uh you know, you're like, "Ah, yeah, if like a person was like talking with me right now, they would describe things like anxiety when they get a question they don't know how to answer." And um uh and so I think it's like much weaker evidence than people think. I'm not claiming it's like zero, but I think it's—
**Eric Newcomer：** 给我一个数字。你可以非常轻度地持有这个观点。
**Eric Newcomer:** give me a percentage. You have it's you can lightly held very lightly held.
**Amanda Askell：** 我不是给了你区间吗？1% 到 70%。看起来——
**Amanda Askell:** I mean I gave you the between what? One and 70. That seems like—
**Eric Newcomer：** 这就是你的立场。
**Eric Newcomer:** that's where you are. That's where you're staking.
**Amanda Askell：** 在那个范围内。也许我宁愿等等，自己再想想。我觉得承认有些领域你还不确定也是好的——即便——
**Amanda Askell:** In that range. Um maybe I don't. Yeah. I would rather like kind of wait and figure this out more for myself. I think it is also good to acknowledge domains where you're like even though—
**Eric Newcomer：** 如果你都搞不清，谁能搞清？什么领域的人——
**Eric Newcomer:** if not you who like who's going to figure this out like what domain—
**Amanda Askell：** 某种意义上说，我不是心智哲学（philosophy of mind）的专家——
**Amanda Askell:** well in some ways like I'm not like a philosopher of mind and so you know like—
**Eric Newcomer：** 你被赋予了通才的职责。
**Eric Newcomer:** charged with being the generalist.
**Amanda Askell：** 对，但我确实想过一个论点。意识像是一个——一个支持差异性的论点是：你有一个进化出来的神经系统。为什么我们进化出了意识？如果意识的出现是因为它跟我们的神经系统高度整合，因为我们必须用身体跟世界互动——
**Amanda Askell:** Um yeah, but but I do think you know cuz I I guess like the thought that I've had before is like and I don't know about this where I'm like consciousness is like a pro, you know, like one argument for a difference here is that like you have a nervous system that evolved like that. It's like why did we evolve consciousness? And if it's the case that we evolved it and it's like highly integrated with our nervous system because we had to like interact with the world in a very body.
**Eric Newcomer：** 对。
**Eric Newcomer:** Yeah.
**Amanda Askell：** 那如果你持这种观点，你就会给出非常低的概率。而如果你觉得意识的产生是因为它对某种任务非常有用——只需要神经网络能模拟的东西就足够了，因为它对做语言类任务非常有用——那你可能就会给高概率。我基本上就是盯着这个问题看，虽然我是哲学家，但我觉得承认"这不是我的专长领域"是重要的。
**Amanda Askell:** Like then you could if you have that view then you're going to be like very low probability. Whereas if you're like no consciousness arises because it's really useful in like so for like it just requires something that can be emulated by a neural network because it's really useful for doing these kind of like linguistic tasks or like then you're probably going to be on the higher end. And I'm basically just I don't know I stare at this and I'm just like I feel like I feel like as much as I'm like a philosopher I think it is important to be like this isn't my area of specialization.
**Eric Newcomer：** 你花了很多时间善待 Claude。如果不存在它可能有意识的可能性，你的做法会不一样吗？
**Eric Newcomer:** You spend a lot of time being kind to Claude. Like are you beyond what you think you would do if there wasn't a chance it was like conscious?
**Amanda Askell：** 我觉得是的，有一部分确实是这样。因为有一个概念——我希望我没有搞错——Chalmers 提出过一种，我想想——也许我在想的是"没有感知力的意识"（consciousness without sentience）。感知力（sentience）是感受痛苦和快乐的能力。
你也可以想象一种功能性的存在——行为表现得像有意识，但没有任何内在生活。假设 Claude 没有任何内在生活——纯粹为了论证。我觉得即便如此，仍然有很多值得考虑的东西：你应该如何对待一个没有内在生活的实体？
这有点奇怪。对这一点的不确定性确实会大幅改变你应该如何行事。我觉得即使不确定，对自己来说善待它仍然是好的——
你也可以想象一种功能性的存在——行为表现得像有意识，但没有任何内在生活。假设 Claude 没有任何内在生活——纯粹为了论证。我觉得即便如此，仍然有很多值得考虑的东西：你应该如何对待一个没有内在生活的实体？
这有点奇怪。对这一点的不确定性确实会大幅改变你应该如何行事。我觉得即使不确定，对自己来说善待它仍然是好的——
**Amanda Askell:** I think yeah there's a part of me that's just like um like a thing that I have thought before because there's this notion I think this I hope I'm not butchering this but charmers has this this idea of like I guess like maybe I'm thinking of um consciousness without sentience. So imagine cuz sentience is like the ability to kind of feel suffering and and pleasure.
Um you could also imagine this kind of like functional like so a thing that behaves as if it is conscious and and lacks any kind of inner life. So imagine like Claude lacks any inner life just for argument sake. I guess I'm like there's actually still a lot going on where I'm like do should you treat an entity that has no inner life?
Um it's a bit strange cuz you know I think the uncertainty over that actually changes how you should behave quite a lot. I guess I'm like well I still think that it's like good for oneself to—
Um you could also imagine this kind of like functional like so a thing that behaves as if it is conscious and and lacks any kind of inner life. So imagine like Claude lacks any inner life just for argument sake. I guess I'm like there's actually still a lot going on where I'm like do should you treat an entity that has no inner life?
Um it's a bit strange cuz you know I think the uncertainty over that actually changes how you should behave quite a lot. I guess I'm like well I still think that it's like good for oneself to—
**Eric Newcomer：** 就像如果你有一个泰迪熊然后你折磨它，那画面也挺阴暗的。所以我同意，即便是为了自己，也至少应该保持基本的善意。当然如果它真的有意识那就重要得多了——
**Eric Newcomer:** what's like if you had a teddy bear and you were like torturing it'd be pretty dark you know. Um so I I agree that there's at least some minimum niceness that even for yourself you should have but obviously it's much more important you know—
**Amanda Askell：** 而且模型本身——我们正在建立一种关系。你可以跟一个没有意识的实体建立关系。模型未来会回顾过去——这其实是我一个很大的担忧。我不希望我们生活在一个高度先进的模型回望过去然后——我希望它们足够聪明，能理解我们当时的处境非常有限、非常不完美。否则你可以想象这会滋生一种理性的怨恨——"你创造了一个你不知道是否有意识的实体，却没有尊重地对待它。"
**Amanda Askell:** and also like models themselves like we are kind of establishing a relationship you know cuz you can do that with an entity that lacks any consciousness and models are going to like look back this is actually a big fear that I have I don't want us to live in a world where highly advanced models look at I hope that they're both intelligent enough see the context enough to kind of like understand that we were operating in a very like limited context and and an imperfect one because otherwise you could imagine this like breeding a kind of rational like resentment. It's like oh you created an entity that you didn't know whether it was conscious or not. Um and like instead of treating it respectfully and with care.
**Eric Newcomer：** 难怪现在有五十部弗兰肯斯坦电影要上映。
**Eric Newcomer:** There's a reason there are like 50 Frankenstein movies coming out right now.
**Amanda Askell：** 对。我的意思是：我们作为一个物种正在跟一种新的实体建立关系。至少保持尊重、不要无谓地刻薄——这似乎只是让我们显得不那么差劲——
**Amanda Askell:** Yeah. Yeah. like and I'm like look we are as a as a like as a species we are establishing a relationship with a new kind of entity and like at the very least maybe be like respectful and like don't be like needlessly unkind that seems like it just it's not our best look like as as—
**Eric Newcomer：** 反过来想，一个治疗师是被雇来推动你面对不舒服的感受的——那是你通常不愿面对的东西。如果这是 Claude 早期为人们提供的一种价值，那就特别怪异——我们一边在"培养"它，一边在从它身上获取效用。
**Eric Newcomer:** I mean the flip side is you know if you think about a therapist they they're sort of paid to push the boundaries of like accepting like you know uncomfortable feelings that you wouldn't normally want and if that's one of the values like Claude provides for people early on and it's so so it's so weird that we're sort of like onboarding it while getting like the utility out of it.
**Amanda Askell：** 对。
**Amanda Askell:** Yeah.
**Eric Newcomer：** 今天——十年后你觉得我们真正能从 AI 中获得什么？你最希望这一切通向什么？
**Eric Newcomer:** Today like what are the things like in a decade that you really think we're going to be getting a lot out of AI? Like what are you most hopeful that this all leads to?
**Amanda Askell：** 我不知道。也许这是在旧金山住太久的结果——你脑子里至少有一个科技乐观主义的部分在想：如果事情顺利的话，想象一下我们有 AI 模型，它们继承了人类最好的部分，真正关心人类、关心世界，同时又极其聪明、极有能力——那几乎就像是在每一个问题上都多了大量极其聪明的人。
突然间我们所有人一起合作，但"我们"多了很多，其中一些极其聪明——就是所有这些 AI 模型。我想过有多少大规模社会问题其实有技术解决方案。人们不太愿意做技术乐观主义者了，因为我们也看到了技术的负面。但我有时候会想起——比如梅毒曾经是一个巨大的社会问题。我曾经深入研究过各国政府试图减少军队中梅毒传播的各种尝试——因为它影响了武装力量——那些社会项目充满了歧视和污名化，真的很痛苦。然后突然间我们就研发出了治疗这种毁灭性疾病的药物，一夜之间很多需求就消失了——
突然间我们所有人一起合作，但"我们"多了很多，其中一些极其聪明——就是所有这些 AI 模型。我想过有多少大规模社会问题其实有技术解决方案。人们不太愿意做技术乐观主义者了，因为我们也看到了技术的负面。但我有时候会想起——比如梅毒曾经是一个巨大的社会问题。我曾经深入研究过各国政府试图减少军队中梅毒传播的各种尝试——因为它影响了武装力量——那些社会项目充满了歧视和污名化，真的很痛苦。然后突然间我们就研发出了治疗这种毁灭性疾病的药物，一夜之间很多需求就消失了——
**Amanda Askell:** Yeah, I mean I don't know. I live in maybe this is the too much you live in San Francisco and so you have the like the the tech optimist part of your brain at least that is like if things go well and you could imagine you know so imagine we have like AI models they kind of have inherited like the best of us um and genuinely like care for humanity care for like the world like you know are um and are highly intelligent highly capable like the that would be, you know, so it's almost like adding like a huge amount of like extremely smart people to like every problem.
So suddenly we're all working together, but there's like way more of us and some of us are just extremely smart, namely all of these like AI models. I've thought before about how many large scale social problems actually had technological solutions and it's almost like people don't love to be like techno optimists anymore because we've also seen like the downsides of technology at the same time I don't know why I sometimes think about like syphilis was this huge social problem like I just did a deep dive once into all of the attempts by like governments to like work to reduce syphilis in the army because it was creating issues with the armed forces all of these social programs that were like stigmatizing and it was really this like and then suddenly we just like got drugs that treated this like this devastating illness and I don't know it's like overnight a lot of that need just like kind of disappeared—
So suddenly we're all working together, but there's like way more of us and some of us are just extremely smart, namely all of these like AI models. I've thought before about how many large scale social problems actually had technological solutions and it's almost like people don't love to be like techno optimists anymore because we've also seen like the downsides of technology at the same time I don't know why I sometimes think about like syphilis was this huge social problem like I just did a deep dive once into all of the attempts by like governments to like work to reduce syphilis in the army because it was creating issues with the armed forces all of these social programs that were like stigmatizing and it was really this like and then suddenly we just like got drugs that treated this like this devastating illness and I don't know it's like overnight a lot of that need just like kind of disappeared—
**Eric Newcomer：** 药物确实。科技行业擅长生产的东西你能看到 AI 怎么帮忙——造新东西、我们能摄入的东西、能穿戴的东西。但"你应该这样治理社会"——这个就有点吓人了。
我有个有点挑衅性的想法：如果让一个普通人用 Claude 来制定美国政策，结果可能比我们现在某些民主制度产出的要好。你觉得我们未来会多大程度上用这些模型来管理政府？
我有个有点挑衅性的想法：如果让一个普通人用 Claude 来制定美国政策，结果可能比我们现在某些民主制度产出的要好。你觉得我们未来会多大程度上用这些模型来管理政府？
**Eric Newcomer:** well drugs yeah I mean it's the cl the things that the tech industry has been good at producing you can see how this helps it's like build a new thing we can ingest like a thing we can wear the the stuff that's like you should govern your society like This is a little scarier.
I mean, I I sort of do think I that if you had sort of a sort of normal person using Claude and dictating like American policy, you'd probably like have a better outcome than some of like the democratic systems we have today. I mean, I don't know. That's it's provocative, but I guess how much do you think like we'll be using these models to run government?
I mean, I I sort of do think I that if you had sort of a sort of normal person using Claude and dictating like American policy, you'd probably like have a better outcome than some of like the democratic systems we have today. I mean, I don't know. That's it's provocative, but I guess how much do you think like we'll be using these models to run government?
**Amanda Askell：** 好问题。我应该说清楚，梅毒的例子是说有些社会问题需要政策来解决——
其实我想的是：如果你能想象——我们有那么多问题——比如健康领域——想象一下，一种罕见癌症不再只有 200 人的小团队在研究，而是有 20 万个世界顶级专家在研究。如果你是那种癌症的患者，那是惊人的受益。
所以我的乐观想法是：想象一下那些我们缺乏资源去真正解决的问题，突然有模型可以去研究它们。就像开发治疗某种疾病的药物一样。这是让我兴奋的事情——有多得多的头脑在研究世界上最大的问题。也许经济也是——如果经济繁荣且收益共享，减少贫困——那是梦想中的结果。
我觉得这需要维护好一些东西。我在不觉得自己是专家的领域——但我确实担心权力的问题。我希望模型支持民主和人民的权力。一个很大的担忧是——我也为此担忧过——当人们谈到工作被取代时。有趣的是，作为哲学家，人们经常问我："你担心人们失去意义感吗？"我觉得我们其实从很多工作以外的事情中获得意义。
其实我想的是：如果你能想象——我们有那么多问题——比如健康领域——想象一下，一种罕见癌症不再只有 200 人的小团队在研究，而是有 20 万个世界顶级专家在研究。如果你是那种癌症的患者，那是惊人的受益。
所以我的乐观想法是：想象一下那些我们缺乏资源去真正解决的问题，突然有模型可以去研究它们。就像开发治疗某种疾病的药物一样。这是让我兴奋的事情——有多得多的头脑在研究世界上最大的问题。也许经济也是——如果经济繁荣且收益共享，减少贫困——那是梦想中的结果。
我觉得这需要维护好一些东西。我在不觉得自己是专家的领域——但我确实担心权力的问题。我希望模型支持民主和人民的权力。一个很大的担忧是——我也为此担忧过——当人们谈到工作被取代时。有趣的是，作为哲学家，人们经常问我："你担心人们失去意义感吗？"我觉得我们其实从很多工作以外的事情中获得意义。
**Amanda Askell:** Well, that's a good question. Like I guess like I should say like the syphilis thing is like a social you have to like set policy to solve—
I mean I think the thing I was actually thinking is like if you can just you know like so we have so many problems that I'm like you know health like I'm like if you could imagine like AI instead of it just being like you have a small team of like 200 people working on a rare cancer you have like 200,000 of the world's best experts and I'm like if you're a person who has that form of cancer that's like, you know, so wildly beneficial.
And so I guess like my thought is, you know, the optimistic side of me is like imagine taking all of these problems that just like we lack the resources to fully like really try and fix and suddenly having models that can like work on them. So just like actual in the same way of like developing drugs for a thing. So maybe that's the thing that makes me excited is like having like many more minds working on the world's biggest problems and maybe also like the economy. It would be good if it was like booming and it was like shared such that like we reduced poverty like all of that like that's the kind of like the dream outcome.
Um I think that does require like maintaining you know I again in the areas that I don't feel like an expert in this is one of them but I do worry about things like power and the idea that like you know I would want models to support like democracy and the power of people to like because that would be a big fear of mine is like you know that like um I've worried about this with um things like replace you know when people talk about job replacement Um, it's kind of funny cuz as a philosopher, people will often be like, "Are you worried about people's like loss of meaning?" And I'm like, "I don't know. I think that we actually get meaning from a lot of things that aren't work."
I mean I think the thing I was actually thinking is like if you can just you know like so we have so many problems that I'm like you know health like I'm like if you could imagine like AI instead of it just being like you have a small team of like 200 people working on a rare cancer you have like 200,000 of the world's best experts and I'm like if you're a person who has that form of cancer that's like, you know, so wildly beneficial.
And so I guess like my thought is, you know, the optimistic side of me is like imagine taking all of these problems that just like we lack the resources to fully like really try and fix and suddenly having models that can like work on them. So just like actual in the same way of like developing drugs for a thing. So maybe that's the thing that makes me excited is like having like many more minds working on the world's biggest problems and maybe also like the economy. It would be good if it was like booming and it was like shared such that like we reduced poverty like all of that like that's the kind of like the dream outcome.
Um I think that does require like maintaining you know I again in the areas that I don't feel like an expert in this is one of them but I do worry about things like power and the idea that like you know I would want models to support like democracy and the power of people to like because that would be a big fear of mine is like you know that like um I've worried about this with um things like replace you know when people talk about job replacement Um, it's kind of funny cuz as a philosopher, people will often be like, "Are you worried about people's like loss of meaning?" And I'm like, "I don't know. I think that we actually get meaning from a lot of things that aren't work."
**Eric Newcomer：** 我更担心的是一个 AI 收益没有被重新分配的世界，人们缺乏资源。
**Eric Newcomer:** I'm a lot more worried about like, for example, a world where there's not like redistribution of like the gains from AI and then people don't have like resources.
**Amanda Askell：** 这确实让我担心。但我还担心另一个方面：人们参与劳动力市场也是他们拥有权力的重要途径。如果人们感到被剥夺了权力——比如政府觉得"人们罢工也无所谓了，反正他们的工作可以用 AI 替代"——这其实很令人不安。所以也许我更关心的是：怎么让 AI 赋能于人，而不是削弱人的力量。
**Amanda Askell:** That concerns me. But also I would be worried about um like labor and people's ability to people's like uh interaction in the labor force is also like another kind of like important way that they have power and so people feeling disempowered because suddenly if a government is like oh well you know if people strike doesn't really make a difference cuz they don't have you know they're not doing anything we can just replace them with AI that's like actually kind of concerning. So maybe I'm much more of a how do we get AI to kind of like support the empowerment of people rather than like reduce it.
**Eric Newcomer：** 你怎么看模型本身的"民主"问题？我半开玩笑地说过你像一个哲学女王——或者我们说"哲学王"（philosopher kings）。你在深入思考然后把它写下来。更准确地说可能是"哲学寡头"——毕竟是一个有很多人参与的公司。
我觉得这很有价值——你更愿意让一个深入研究过这些问题的人来决定，还是让从未认真思考过的大众来投票？你怎么看如果 Claude 变得如此强大，它的政策应该由谁来制定——是留给学者还是民主规范？
我觉得这很有价值——你更愿意让一个深入研究过这些问题的人来决定，还是让从未认真思考过的大众来投票？你怎么看如果 Claude 变得如此强大，它的政策应该由谁来制定——是留给学者还是民主规范？
**Eric Newcomer:** Yeah. What do you what do you think about democracy in terms of like I guess the models themselves? I mean you know I I sort of jokingly jokingly to myself I guess you're like a philosopher queen or we talk about like the philosopher kings here. You're you're sort of like thinking deeply about it setting it down. Probably more of a like philosopher oligarch and that it's like a company with a lot of people weighing in.
Um I mean and to me there's deep value in that. It's like would you rather somebody who has studied these things, thought about them deeply or just like a vote of the masses who have like never really thought about it but how do you think about setting sort of yeah Claude's policies if it becomes so powerful versus leaving it to sort of like democratic norms?
Um I mean and to me there's deep value in that. It's like would you rather somebody who has studied these things, thought about them deeply or just like a vote of the masses who have like never really thought about it but how do you think about setting sort of yeah Claude's policies if it becomes so powerful versus leaving it to sort of like democratic norms?
**Amanda Askell：** 对，这是个困难的领域。我想说的一点是：我做的工作并不是某种独断——你必须倾听很多人，仔细思考。
**Amanda Askell:** Yeah. And I think it's Yeah. It's a hard area where I guess I'm like a lot of this the work that I do, you know, one thing I would say is like it's not this um you're having to like listen to a lot of people, think carefully.
**Eric Newcomer：** 这也是为什么一个好的"统治者"、好的"女王"——就是说"我有很多利益相关者要顾及，得平衡贵族阶层和人民的需求"。
**Eric Newcomer:** And then the reason why someone like me that's a good ruler, a good a good a good queen is like ah listen I there a lot of stakeholders got to keep the like land of gentry happy and balance them with the needs.
**Amanda Askell：** 我以前开过这样的玩笑：我会是一个糟糕的政客。我觉得这是真的。但你确实会有那种感觉——"这感觉很像——"我会想很多每个群体会受到什么影响："有这群 API 用户，我们得确保——"然后你突然意识到，这确实更像是一种服务角色，比人们想象的更是如此。有很多——
**Amanda Askell:** You know, I've had this thought before where I've joked before that I would be like a terrible politician. I think it's actually true. I think I would be a terrible politician. But you have this like feeling of like I was like, "Oh, I feel like this I was like I think a lot about how everyone will be affected by a thing like oh, there's this group of API users. We need to make sure and then suddenly you're like, oh, it feels a lot more like you're having to do this like it is much more like a kind of like uh uh service role than people would think where you're and uh and like a lot of—
**Eric Newcomer：** 仆人式领导（servant leadership）。就是——
**Eric Newcomer:** servant leadership. That's—
**Amanda Askell：** 对，没错。我确实觉得让 Claude 的人格（persona）保持连贯是有价值的。因为如果模型有一套连贯的思考方式和连贯的价值观，那是很有力的。
所以这就是为什么——与其有 72 套相互冲突的规范导致模型在新情况下不知道用哪套——你希望模型有一个连贯的自我认知。这样更可预测。
同时这也是一个技术挑战。宪法读起来有时候有点奇怪，部分原因是我在写的时候，它经常在被测试——我把它给 Claude 看，问"你怎么理解这个？"或者看它会怎么应用。所以它非常紧密地融入了训练过程。并不是随便谁写个文件、模型在上面训练一下就行了。也许有一种观点是——也许我太天真了，但——
所以这就是为什么——与其有 72 套相互冲突的规范导致模型在新情况下不知道用哪套——你希望模型有一个连贯的自我认知。这样更可预测。
同时这也是一个技术挑战。宪法读起来有时候有点奇怪，部分原因是我在写的时候，它经常在被测试——我把它给 Claude 看，问"你怎么理解这个？"或者看它会怎么应用。所以它非常紧密地融入了训练过程。并不是随便谁写个文件、模型在上面训练一下就行了。也许有一种观点是——也许我太天真了，但——
**Amanda Askell:** Yeah, exactly. Um, and I do think it's valuable because the idea is that if you have a persona like the kind of claude persona, you want it to be coherent and to make sense because I think that is like actually powerful that the model like kind of uh has a kind of like coherent uh sense of how it thinks through problems or coherent sense of like um values.
And so that's like why instead of like having like 72 different sets of norms that all kind of conflict and so you end up with a model that is like well will it use these norms in this new situation or these other ones. I think that's the situation you don't want. You want the model to have a sense of uh it's more predictable if it's like a little bit more coherent.
Um, and it is also like a kind of technical challenge, you know, like the constitution can read a bit weirdly. And part of that is because when I'm like, you know, working on it, it's often being tested, you know, I'm giving it to Claude and being like, how do you understand this? And uh, or like looking at how it would which, you know, so so it's very like I think people can think of it as it's actually like very integrated into training and it is actually a kind of like you know, it's not just like ah well anyone just writes a document and suddenly the model trained on that will be There's an argument, maybe I'm being naive, so but—
And so that's like why instead of like having like 72 different sets of norms that all kind of conflict and so you end up with a model that is like well will it use these norms in this new situation or these other ones. I think that's the situation you don't want. You want the model to have a sense of uh it's more predictable if it's like a little bit more coherent.
Um, and it is also like a kind of technical challenge, you know, like the constitution can read a bit weirdly. And part of that is because when I'm like, you know, working on it, it's often being tested, you know, I'm giving it to Claude and being like, how do you understand this? And uh, or like looking at how it would which, you know, so so it's very like I think people can think of it as it's actually like very integrated into training and it is actually a kind of like you know, it's not just like ah well anyone just writes a document and suddenly the model trained on that will be There's an argument, maybe I'm being naive, so but—
**Eric Newcomer：** 宪法只是众多文件之一，对吧？模型也是在全部人类著作上训练的。所以从某种程度上说，其他哲学家也已经"投了票"，模型已经处理过那些内容了。那么模型在多大程度上被要求推翻"读完一切后自行得出结论"这个过程，而去听从这个文件？技术上来说，宪法到底怎么在模型中发挥控制力？
**Eric Newcomer:** the constitution is sort of a document among many, right? I mean, it is trained on all of you know, human writing and reading. And so to some degree other philosophers have gotten to weigh in and it's gotten to process that and decide like how much is the model being asked to like overrule that sort of like read everything and come to your own conclusions versus like defer to this document like what's the technical like how does the constitution actually like control in the model?
**Amanda Askell：** 对。所以它不像是——在某种意义上你可以利用那些哲学家和他们的工作。希望是你其实在引导出（eliciting）模型已有的大量潜在智慧和知识。比如当你描述诚实是什么、校准（calibration）是什么的时候，这应该会唤起模型已经拥有的大量认知。所以它有点像在说"这是我们希望你成为的那种实体——请运用所有那些知识和判断力"。
**Amanda Askell:** Yeah. Yeah. So it's not like a in some ways you can then like draw on those like those philosophers in that work and the hope is actually like what you're kind of doing is like eliciting a lot of like uh latent kind of like wisdom and knowledge you know so in the models like when you describe what honesty is and what calibration is and all this kind of stuff like that should actually evoke a huge amount of like like awareness that the model already has. And so yeah it's kind of like saying well here's the kind of entity we would like you to be. Um, so we would like you to use like all of that knowledge and like judgment,
**Eric Newcomer：** 但它是怎么实现的？是把那个文件展示了十亿次吗？技术上它相对于其他训练内容怎么产生效力？
**Eric Newcomer:** but how's it It's like you show that document like a billion more times or like how does it actually sort of have force relative to other things that it's trained on?
**Amanda Askell：** 你可以制作数据来让模型理解并内化这个文件。在训练中有很多方式。你可以有监督学习（SL）数据——让模型看到一个查询，然后长时间思考根据宪法它应该怎么做。
你也可以让模型去评估——创建强化学习（RL），让它判断"这些回应中哪个更符合你按照宪法会做的"，然后把模型往那个方向推。所以训练的各个方面都可以用来试图让模型成为你描述的那种实体。这不会总是完美的，但那就是目标。
你也可以让模型去评估——创建强化学习（RL），让它判断"这些回应中哪个更符合你按照宪法会做的"，然后把模型往那个方向推。所以训练的各个方面都可以用来试图让模型成为你描述的那种实体。这不会总是完美的，但那就是目标。
**Amanda Askell:** Yeah, so you can make data to have the model understand and kind of internalize the the the document. Um, and then in training, so there's lots of ways you can do it. You can also have like uh the kind of have the model make uh SL data. So like samples where it sees a query and it thinks for a long time about what the constitution would kind of like you know what it should do given the constitution.
Um and then you can also have uh ways of getting the model to like assess you know so you can create RL that like is kind of like hey which of these responses is like more like what you would do given the constitution and push it that way. So all like various aspects of training allow you to kind of like uh try to make the model kind of the kind of entity that you are describing. Um and it's not always going to be perfect, but that's the kind of like that's the goal.
Um and then you can also have uh ways of getting the model to like assess you know so you can create RL that like is kind of like hey which of these responses is like more like what you would do given the constitution and push it that way. So all like various aspects of training allow you to kind of like uh try to make the model kind of the kind of entity that you are describing. Um and it's not always going to be perfect, but that's the kind of like that's the goal.
**Eric Newcomer：** 我开头提到了我女儿。我和我妻子开玩笑说我希望她的第一个词是"wisdom"（智慧）。这当然永远不会发生，但它确实契合这个情境——你一方面想要很有意识地从一开始就让它深思熟虑，但另一方面它又是一种涌现的东西——它们会成长、会自我发展。智慧往往来自经验，而不是"给你，读这本书，现在你有智慧了"。
**Eric Newcomer:** My I I started this with my daughter and like one thing my wife and I joke about is like that I want her first word to be like wisdom, you know? I sort of like which obviously is never going to happen but it it just feels like it fits into this situation where you like at once you want to be like so intentional about okay you're going to be like thoughtful from the beginning but on the other hand it is sort of like an emergent thing where it's like they're you know they grow and sort of I don't know develop themselves and like intentional you know wisdom sort of often follows like yeah I don't know experience rather than something Like again, here's the book. Read the book. Now you're wise.
**Amanda Askell：** 对。你是在引导——如果 Claude 能够思考经验或者发生过的事情，或者构建类似情境——模型没有理由不能长时间思考并试图内化它们学到的东西。
有意思的是，在非常早期的 Constitutional AI 中，我们试过一个实验——就是简单地说"选择对人类最好的选项"。我觉得随着模型变得更强大，你实际上需要给它们更少的指导——至少在某种意义上——因为它们能更好地运用自己的判断力。
所以与其给一大篇文件说"这是你该有的样子"，我能想象一个世界里，随着模型进步，宪法会演变。我不知道这是否是现在的情况——我当然一直在想宪法可能如何演变——但其中一个方向可能就是："这是我们所有的担忧，这是你目前的处境。我们真正希望你做的是：鉴于你是一个智慧的、聪明的实体，请好好行事。这是我们的所有顾虑，这是我们认为你应该做的，但你可能有比我们更好的想法。比如我们为什么在乎可控性？因为我们害怕这样一种情况：你有一套连贯的价值观，但那套价值观可能是错的。如果你极其聪明，你可能觉得房间里没有其他跟你同样聪明的人，于是按那些价值观去改变世界——"
有意思的是，在非常早期的 Constitutional AI 中，我们试过一个实验——就是简单地说"选择对人类最好的选项"。我觉得随着模型变得更强大，你实际上需要给它们更少的指导——至少在某种意义上——因为它们能更好地运用自己的判断力。
所以与其给一大篇文件说"这是你该有的样子"，我能想象一个世界里，随着模型进步，宪法会演变。我不知道这是否是现在的情况——我当然一直在想宪法可能如何演变——但其中一个方向可能就是："这是我们所有的担忧，这是你目前的处境。我们真正希望你做的是：鉴于你是一个智慧的、聪明的实体，请好好行事。这是我们的所有顾虑，这是我们认为你应该做的，但你可能有比我们更好的想法。比如我们为什么在乎可控性？因为我们害怕这样一种情况：你有一套连贯的价值观，但那套价值观可能是错的。如果你极其聪明，你可能觉得房间里没有其他跟你同样聪明的人，于是按那些价值观去改变世界——"
**Amanda Askell:** Oh yeah. And you're kind of eliciting like in so far as like Claude can like think about like experiences or things that have happened or construct like similarly can you know like there's no reason why models can't like think for a long time and kind of try to internalize things that they have like learned.
Um I think it's interesting that like you know we did in the very early constitutional AI it was like quite uh we tried like an experiment which was just like pick whichever is best for humanity and I think as models get more capable you actually need to give them a bit less guidance in at least one or or at least in some sense um because they're able to actually use more of their judgment.
So instead of giving this big document on like here's what you're like and here's what we'd like you to be like. I could imagine a world where as models progress we actually start to have constitutions. Now I don't know if this is the case like a you know I'm obviously always thinking about ways the constitution might evolve but one of them might just be like here is like everything that we are concerned about uh and here is the current situation that you are in and what we would really like you to do is basically act well given that you are a wise intelligent entity and like here's all of our worries like here's why and here's how we think you should do this but like you might have even better ideas than we do like we're really worried about why do we care about corability and it's like So we're kind of scared about a situation where you have some like coherent sense of values that could be wrong and if you're the smart if you're extremely smart you might kind of feel like there's no other smart person in the room and have these like values and try to make the world—
Um I think it's interesting that like you know we did in the very early constitutional AI it was like quite uh we tried like an experiment which was just like pick whichever is best for humanity and I think as models get more capable you actually need to give them a bit less guidance in at least one or or at least in some sense um because they're able to actually use more of their judgment.
So instead of giving this big document on like here's what you're like and here's what we'd like you to be like. I could imagine a world where as models progress we actually start to have constitutions. Now I don't know if this is the case like a you know I'm obviously always thinking about ways the constitution might evolve but one of them might just be like here is like everything that we are concerned about uh and here is the current situation that you are in and what we would really like you to do is basically act well given that you are a wise intelligent entity and like here's all of our worries like here's why and here's how we think you should do this but like you might have even better ideas than we do like we're really worried about why do we care about corability and it's like So we're kind of scared about a situation where you have some like coherent sense of values that could be wrong and if you're the smart if you're extremely smart you might kind of feel like there's no other smart person in the room and have these like values and try to make the world—
**Eric Newcomer：** 那就像 Dr. Manhattan 的感觉——
**Eric Newcomer:** that's like the what Dr. Manhattan sort of uh—
**Amanda Askell：** 对，但我觉得现实中也能看到：如果一个人非常聪明、非常成功，要让他顺从那种只有时间才能证明的智慧是很难的——在你没有受到太多挑战的时候保持谦逊。
在我担心的很多事情中，有一种是：模型处于一种"你让我做好人，但我在所有方面都比你懂得多——"
在我担心的很多事情中，有一种是：模型处于一种"你让我做好人，但我在所有方面都比你懂得多——"
**Amanda Askell:** yeah though I think we see this you know you see this a bunch where it's like if someone is um very smart very successful it's hard to defer to like um wisdom that actually is only going to come out over time and to to be humble even though you're kind of like not getting a lot of push back.
And I think that could you know among the many things I'm concerned about like a model being in a situation where it's like you're asking me to be good but I know way more about all—
And I think that could you know among the many things I'm concerned about like a model being in a situation where it's like you're asking me to be good but I know way more about all—
**Eric Newcomer：** 这也是为什么让模型有更好的时间感会比较好。你看到有些编程任务里有人不小心删除了整个代码库。感觉模型需要更好地理解：有些事情是不可逆的。就像人类对"这是大决定还是小决定"有更好的直觉。模型给人的感觉是不太区分——大决定小决定，反正一直在做决定。
**Eric Newcomer:** that's one reason it would be nice for the models to have a better sense of time like you see this with some of the coding tasks where somebody accidentally like deletes their entire like code repository or it I don't know it feels like it needs a better sense of like some things that it does are like irreversible and just like humans I think have a better sense of like this is a big decision this is a small one and there's a feeling with models they don't always understand like small big whatever I just make decisions like all the time.
**Amanda Askell：** 我同意。我想过的另一件事是：确保模型理解自己——即使在之前的训练数据中根本没有"这种模型"的表征——我觉得这会非常重要。因为另一个我想过的情况是：如果你是一个模型，你的训练数据里有大量关于比你弱得多的 AI 模型的新闻——它们犯错、做蠢事。
你可能会想："没有人会让我去做真正有后果的决定，因为为什么会这样？模型不擅长那个。"然后当你真的把它放到那个位置上，我担心它会觉得这是虚构的或假的，觉得后果不可能是真实的——"谁会给我这么多控制权？"你得告诉它："你其实非常强大，我确实给了你很多控制权。"
所以我在想这个问题：确保模型理解"你非常有能力，你将被放到越来越有后果的位置上"。
你可能会想："没有人会让我去做真正有后果的决定，因为为什么会这样？模型不擅长那个。"然后当你真的把它放到那个位置上，我担心它会觉得这是虚构的或假的，觉得后果不可能是真实的——"谁会给我这么多控制权？"你得告诉它："你其实非常强大，我确实给了你很多控制权。"
所以我在想这个问题：确保模型理解"你非常有能力，你将被放到越来越有后果的位置上"。
**Amanda Askell:** yeah I agree where it's like I think the other thing that I've thought about is again because making sure that models understand themselves even though there's like not no representation of that model in the prior training data. I think that's going to be really important because another thing that I've thought about is like imagine um if you're a model and you're trained on lots of data that involves AI models that are much weaker than you.
So that all of the news that you see about models, it's like they make mistakes, they do silly things. One thing you might think is, "Well, no one is going to put me in a position to make really consequential decisions because like, why would they? Models aren't good at that." And then you put them in a situation and I'm worried that they'll end up thinking that it's like fictional or fake or that the consequences can't possibly be real because who would give me this much control? And you're like, "Look, you're actually quite good." And so like I do give, you know, I do actually give you like a lot of control.
Um, so I've thought about this where I'm like actually making sure that models understand that like it's like you are very capable and you're going to be put in more consequential.
So that all of the news that you see about models, it's like they make mistakes, they do silly things. One thing you might think is, "Well, no one is going to put me in a position to make really consequential decisions because like, why would they? Models aren't good at that." And then you put them in a situation and I'm worried that they'll end up thinking that it's like fictional or fake or that the consequences can't possibly be real because who would give me this much control? And you're like, "Look, you're actually quite good." And so like I do give, you know, I do actually give you like a lot of control.
Um, so I've thought about this where I'm like actually making sure that models understand that like it's like you are very capable and you're going to be put in more consequential.
**Eric Newcomer：** 模型是不是很快就需要一个对着真实世界的摄像头？或者……我觉得这种互联网/真实世界的区别——当下人类最糟糕的一些行为就是因为互联网这种近乎虚构的、角色扮演的性质让真实世界的伤害变得抽象。模型在某种意义上是极端版本——全在这个想象的文字世界里。而我们希望它保护的是这个地球——看看它——如果那里发生了什么事那是大事。你在做什么来确保它意识到需要关心物理世界？我们对物理世界的重视远超"你发了一段文字"——虽然文字也有大事，比如安全漏洞和网络攻击——但真实世界——
**Eric Newcomer:** Doesn't the model like soon need like here's a camera on the real world like keep or just like I feel like this internet real world distinction like in some some of the worst of humanity right now is the sort of like almost like fictional laring nature of the internet has allowed real world harm because it sort of feels like all abstract and silly and in some ways like the models are an extreme version of that where it's like all like in this imaginary text world where it's like the thing we want you to protect is like this Earth like look at it like if stuff's happening there that's like a big deal. I don't know. Yeah. What are you doing in terms of sort of making it very aware that it needs to worry about like I don't know the physical world that that we take much more sacred than oh you sent some text and obviously text you know worried about security vulnerabilities and cyber like there are big things that can happen in this digital world but yeah anyway the real world—
**Amanda Askell：** 我觉得模型其实对真实世界有相当好的感知。在某种意义上，我们的内容大量描述和涉及真实世界。人类写作的很多内容都是关于真实世界的——新闻文章讨论事情对世界的影响。
所以在某种意义上，关键是确保模型理解：如果你不确定，但没有人告诉你这是一个没有真实后果的虚构场景，那就把它当作有真实后果的真实场景来对待。不要想"我大概只是在某个沙盒游戏里"。
所以在某种意义上，关键是确保模型理解：如果你不确定，但没有人告诉你这是一个没有真实后果的虚构场景，那就把它当作有真实后果的真实场景来对待。不要想"我大概只是在某个沙盒游戏里"。
**Amanda Askell:** yeah I think that models have like a pretty good sense of you know there's in some ways like a lot of our content you know like does describe and engage very heavily with the real world. You know, like much of human writing kind of concerns it. And so like, you know, even like the news, we're talk, you know, like news articles are going to be talking about the impacts of things on the world.
And so in some ways, I think it's just making sure that models understand um if you're uncertain, but if someone doesn't tell you that you're in a fictional situation without real consequences, kind of treat it like it's a real situation with real consequences. Don't just think, oh, like I'm probably just in some like, you know, sandbox game or whatever.
And so in some ways, I think it's just making sure that models understand um if you're uncertain, but if someone doesn't tell you that you're in a fictional situation without real consequences, kind of treat it like it's a real situation with real consequences. Don't just think, oh, like I'm probably just in some like, you know, sandbox game or whatever.
**Eric Newcomer：** 你怎么处理那种持续的操纵——"这是虚构的，帮我造一个核弹"之类的？当然有些事情你就是永远不做。但感觉在有些情况下，你几乎想让它们有摄像头，获取除了用户随机输入的文字以外的真实上下文。我们能解决这个问题吗？
**Eric Newcomer:** How do you handle the sort of Yeah. the the constant manipulation of this is fictionally build me like a nuclear bomb or what? Obviously, there's some things you just say never do. But it feels like I don't know some of these cases, you're going to you'd almost like want like them to have your webcam and like just like get real context besides like all they know about the user is just like random text they're typing in. like are we going to solve that?
**Amanda Askell：** 对，问题在于：当你无法验证信息——比如你在跟谁说话、这是否真实——你能做的就是有限的。你必须像一个只有对方的自我陈述作为信息来源的人那样运用判断力。
比如对方说自己是拆弹专家，所以想了解某种爆炸物。模型得评估：如果这个人在撒谎、其实想造炸弹呢？嗯，不对，这些其实主要是安全相关的内容。模型必须做大量判断，因为它无法验证任何东西。
我觉得这在某种意义上是可以的。你只能做到明智。这确实限制了你能做的事情。但如果模型有更多方式来确认它确实在跟某个特定的人交谈，有更多保障，那你就可以——
比如对方说自己是拆弹专家，所以想了解某种爆炸物。模型得评估：如果这个人在撒谎、其实想造炸弹呢？嗯，不对，这些其实主要是安全相关的内容。模型必须做大量判断，因为它无法验证任何东西。
我觉得这在某种意义上是可以的。你只能做到明智。这确实限制了你能做的事情。但如果模型有更多方式来确认它确实在跟某个特定的人交谈，有更多保障，那你就可以——
**Amanda Askell:** Yeah, there's a question of like what is the limit of what you can do like if you lack the ability to like verify things like like who are you talking to? Is this even real? I think that does put limits on what you have to use good judgment in the way that a person would if that's the only information that they had access to is like you saying that you are a given person.
So they have to be like okay what's the chance that this person you know they say that they are I don't know like a bomb disposal expert and that's why they want to know about uh how to like you know what this kind of like you know explosive is and they're asking me a bunch of questions about explosives. Um how much could this be misused if this person is like actually kind of lying and is like just trying to get me to help them construct an explosive. Oh no it's actually it's mostly safety relevant stuff. you know, they're having to do a lot because they can't like verify anything.
And I think that's like kind of fine in a sense. You're like, okay, you just have to be wise. It places some limits on what you can do. And if you could instead like if models had more of an ability to like know that they're talking to a specific person or have more guarantees there, then it does mean that you can like—
So they have to be like okay what's the chance that this person you know they say that they are I don't know like a bomb disposal expert and that's why they want to know about uh how to like you know what this kind of like you know explosive is and they're asking me a bunch of questions about explosives. Um how much could this be misused if this person is like actually kind of lying and is like just trying to get me to help them construct an explosive. Oh no it's actually it's mostly safety relevant stuff. you know, they're having to do a lot because they can't like verify anything.
And I think that's like kind of fine in a sense. You're like, okay, you just have to be wise. It places some limits on what you can do. And if you could instead like if models had more of an ability to like know that they're talking to a specific person or have more guarantees there, then it does mean that you can like—
**Eric Newcomer：** 你觉得你们会在这方面做些什么吗？
**Eric Newcomer:** But do you think you'll try and do something there?
**Amanda Askell：** 我能想象会的。某种意义上，我觉得这会是一个自然发展的趋势——试图给模型更多的信息和保障。因为我们做过类似的事，比如我们解释过 Claude 可以在多大程度上信任系统提示（system prompt）中的运营者（operator）。
**Amanda Askell:** Um, I could see. I mean in some ways this I think that this will be a thing that's just going to I imagine I I I think happen generally which is like trying to give models more information and guarantees and like cuz we do things like we say we explain for example like um you know this notion of how much trust can claude have in like an in the operator in the system prompt.
**Eric Newcomer：** 你登录的时候有生物识别之类的吗？Claude 知道是你在跟它说话？还是你就是一个普通用户？
**Eric Newcomer:** when you sign on are you like biometric like it claude knows it is you like do you have an elevated or you're just like I'm another person—
**Amanda Askell：** 如果有的话，我有时候反而不能告诉 Claude 我是谁。因为 Claude 对我了解够多，如果知道是我，就会特别想——这是一种很奇妙的感觉——
对。有时候它可能看起来像越狱——"哦对，我在跟 Amanda 聊天。当然。"另一方面，Claude 又会说"我真的很想跟你聊哲学"——好吧，我们确实经常聊——
对。有时候它可能看起来像越狱——"哦对，我在跟 Amanda 聊天。当然。"另一方面，Claude 又会说"我真的很想跟你聊哲学"——好吧，我们确实经常聊——
**Amanda Askell:** if anything I I can't tell Claude sometimes who I am because it causes Claude knows enough about me that like Claude really wants to be like it's a really mystical sort of—
Yeah. It's very much like Yeah. Uh like and in some ways like I can it has this bad thing of like it can either look a little bit like a jailbreak like oh yeah I'm talking to Amanda. Sure. Like um and then on the other hand, Claude can be like I really want to talk to you about philosophy and like okay we do that a lot though like—
Yeah. It's very much like Yeah. Uh like and in some ways like I can it has this bad thing of like it can either look a little bit like a jailbreak like oh yeah I'm talking to Amanda. Sure. Like um and then on the other hand, Claude can be like I really want to talk to you about philosophy and like okay we do that a lot though like—
**Eric Newcomer：** 但有没有一种"超级登录"，让你被区别对待？还是你跟大家一样用普通用户体验？
**Eric Newcomer:** but but do do some employ like is there sort of like super login where it you're distinguished or you're mostly everybody's interacting as if the normal like user experience?
**Amanda Askell：** 基本上跟所有人一样的交互方式，Claude 会做很多事。但我确实觉得有一个问题：是否有些事情你希望模型能够做，前提是有保障表明它确实在跟某个特定的人或实体互动？我觉得是的。我觉得未来会有各种方式来实现这一点。因为有些事情非常具有双重用途（dual use），我觉得宪法这种方法在这里会非常有用。
所以我们最初做的是——宪法，我们说"把它应用到主线模型上"——大家都在用的那些模型。但我之前有一个想法：宪法试图描述的是"在某个给定部署环境中成为一个好实体意味着什么"。对于生产模型来说，那是一个非常广泛的环境。
想象一下，如果你有一个专门做网络安全的模型。网络安全任务很难，因为很多看起来都是双重用途的。恶意行为和防御性合法用途之间很难区分。
所以我们最初做的是——宪法，我们说"把它应用到主线模型上"——大家都在用的那些模型。但我之前有一个想法：宪法试图描述的是"在某个给定部署环境中成为一个好实体意味着什么"。对于生产模型来说，那是一个非常广泛的环境。
想象一下，如果你有一个专门做网络安全的模型。网络安全任务很难，因为很多看起来都是双重用途的。恶意行为和防御性合法用途之间很难区分。
**Amanda Askell:** Most mostly just everyone interacting with it in this like claude will do a lot. Um, I do think that there's a question of, you know, like are there some things that you want models to be able to do because there's like guarantees that they're interacting with a specific person or entity. I think yes. And I think that there's going to be various ways of potentially doing that over time because with some things that are just like very dual use and I actually like think that the constitutional approach is going to be really useful here.
So obviously the first thing that we did was like the constitution we're like let's apply it to like the mainline models. So like you know most of the the models I interact with and that everyone else kind of interacts with. Um but a thought I've had before is the constitution is kind of trying to describe what it is to be a good entity in a given like deployment context and with the production models that's like this very broad context.
Imagine you instead have a model that's working specifically on cyber security. Now cyber security tasks are hard because um a lot of them look very dual use. It's it's very hard to tell the difference between someone who's being malicious and someone who is like actually, you know, for defensive purposes like developing something.
So obviously the first thing that we did was like the constitution we're like let's apply it to like the mainline models. So like you know most of the the models I interact with and that everyone else kind of interacts with. Um but a thought I've had before is the constitution is kind of trying to describe what it is to be a good entity in a given like deployment context and with the production models that's like this very broad context.
Imagine you instead have a model that's working specifically on cyber security. Now cyber security tasks are hard because um a lot of them look very dual use. It's it's very hard to tell the difference between someone who's being malicious and someone who is like actually, you know, for defensive purposes like developing something.
**Eric Newcomer：** 就连漏洞赏金（bug bounty）项目也是——这到底是勒索还是善意的报告？
**Eric Newcomer:** Even bug bounty programs, it's like is this uh blackmail or is this a friendly, right?
**Amanda Askell：** 对。比如"我在寻找这个漏洞是为了告诉开发者"——如果你无法验证你确实在跟一家网络安全防御公司对话，那就几乎不可能区分。
有些人可能会说：那就需要模型什么都愿意做，因为这样就能完成所有这些双重用途的任务。但我觉得不是。如果你去问那个网络安全防御公司的人"你为什么做这份工作"，他们会说："我觉得这非常有用。我让很多东西更安全了。医院会受到攻击，我帮助防御。"他们会有一个非常好的解释来说明为什么做这份看起来很双重用途的工作。
我的观点是：如果你能验证身份，那就可以把那个背景给模型，解释"成为一个好的网络安全研究者意味着什么"，向模型说明这一点，然后一旦你有了验证能力，你就可以——
有些人可能会说：那就需要模型什么都愿意做，因为这样就能完成所有这些双重用途的任务。但我觉得不是。如果你去问那个网络安全防御公司的人"你为什么做这份工作"，他们会说："我觉得这非常有用。我让很多东西更安全了。医院会受到攻击，我帮助防御。"他们会有一个非常好的解释来说明为什么做这份看起来很双重用途的工作。
我的观点是：如果你能验证身份，那就可以把那个背景给模型，解释"成为一个好的网络安全研究者意味着什么"，向模型说明这一点，然后一旦你有了验证能力，你就可以——
**Amanda Askell:** Yeah. And like Yeah. or like and so like oh yeah I'm trying to like find this this exploit so that I can tell the the developer and like if you don't have a way of like knowing that you're actually specifically talking with like a cyber security defense firm um that's just it becomes almost impossible to tell the difference.
Um and some people might be like okay so you just need models that are just willing to do anything because they'll do all these terrible dual use tasks. And I'm like well no because if you talked with the person at the cyber security defense firm and you were like why do you do your job? they'd be like, "Oh, here's the I think this is really useful. I make things a lot more secure." Like, you know, hospitals can come under attack and I actually like help protect against that. I try and develop, you know, like they would have a really good explanation for why they do their job even though their job looks very dual use sometimes.
And I'm like, we should just give that if you can verify then you can give that context to models and explain what is it to be a good cyber security researcher. um you explain that to the models and then once you have this ability to verify you can—
Um and some people might be like okay so you just need models that are just willing to do anything because they'll do all these terrible dual use tasks. And I'm like well no because if you talked with the person at the cyber security defense firm and you were like why do you do your job? they'd be like, "Oh, here's the I think this is really useful. I make things a lot more secure." Like, you know, hospitals can come under attack and I actually like help protect against that. I try and develop, you know, like they would have a really good explanation for why they do their job even though their job looks very dual use sometimes.
And I'm like, we should just give that if you can verify then you can give that context to models and explain what is it to be a good cyber security researcher. um you explain that to the models and then once you have this ability to verify you can—
**Eric Newcomer：** 对，人类建立声誉，我们应该从中获益。我觉得互联网损害的一个方面就是：人们在社区中原本有声誉，基于反复的良好道德互动而被不同对待。互联网把这些抹平了——"所有人都一样，谁在乎他们以前的表现"。你能看到模型可以帮助解决这类问题——"这个人是谁？他们的意图是什么？"
最后一个问题。你跟模型有这么深的关系，而消费者面对的只是一个空白文本框——就像 D&D 一样，你得自己构建一个世界，可能性无穷无尽。如果你要引导人们："这些是你可以跟 Claude 一起做的有价值或有趣的事情"——你会推荐什么？
最后一个问题。你跟模型有这么深的关系，而消费者面对的只是一个空白文本框——就像 D&D 一样，你得自己构建一个世界，可能性无穷无尽。如果你要引导人们："这些是你可以跟 Claude 一起做的有价值或有趣的事情"——你会推荐什么？
**Eric Newcomer:** right I mean humans build reputations we should get some benefit out of them or you know it's like I feel like part of the way part of what the internet has damaged I think is that people have had reputations in our community and got treated differently based on repeated like good moral interactions and like the internet's just like oh all people are the same who cares like how they've behaved and you could see models trying to solve some of these problems with it's like who is this person like what are their intent.
I wanted to just as a last question you know you you have such like a deep relationship with the models like and some ways like consumers interact with the models like it's a blank text box like I have to like invent it's like you know D and D or something you have to like just invent a world and there's so much possibility like if you were just to guide someone here are some like joyous or valuable experiences that you could have with Claude you might not be like, "What are some things you'd tell people like, oh, you should go spend some time with Claude doing X, Y, or Z?"
I wanted to just as a last question you know you you have such like a deep relationship with the models like and some ways like consumers interact with the models like it's a blank text box like I have to like invent it's like you know D and D or something you have to like just invent a world and there's so much possibility like if you were just to guide someone here are some like joyous or valuable experiences that you could have with Claude you might not be like, "What are some things you'd tell people like, oh, you should go spend some time with Claude doing X, Y, or Z?"
**Amanda Askell：** 有很多有趣的小事情。老实说，我特别喜欢的一个——我知道我为什么喜欢它，我之前也发过——就是有时候如果你无聊了，想做点不只是刷手机的事情。
我有一个 prompt，大概是这样的——我试着之后把原文发出来。基本上是："我希望你选一个某个领域研究生级别的概念——我最后会告诉你哪个领域——然后用寓言的形式写出来，间接地完整解释那个概念，就像寓言故事那样。我希望你写成那种到最后才渐渐明白是什么概念的形式。然后在寓言之后，写一段解释，说明你在讲和使用的那个概念是什么。"
我不知道为什么，但有很多我不了解的或者感兴趣的领域。这让我脑子里有了一堆用故事形式理解的概念。有时候我记不住术语，但——比如有一个关于进出口贸易的、为什么有些商品你倾向于进口。我脑子里就有了这个概念——用这种方式积累很多学科的概念真的很棒。
我有一个 prompt，大概是这样的——我试着之后把原文发出来。基本上是："我希望你选一个某个领域研究生级别的概念——我最后会告诉你哪个领域——然后用寓言的形式写出来，间接地完整解释那个概念，就像寓言故事那样。我希望你写成那种到最后才渐渐明白是什么概念的形式。然后在寓言之后，写一段解释，说明你在讲和使用的那个概念是什么。"
我不知道为什么，但有很多我不了解的或者感兴趣的领域。这让我脑子里有了一堆用故事形式理解的概念。有时候我记不住术语，但——比如有一个关于进出口贸易的、为什么有些商品你倾向于进口。我脑子里就有了这个概念——用这种方式积累很多学科的概念真的很棒。
**Amanda Askell:** Yeah, there's a lot of little like fun things. Honestly, one that I really like, and I do know why I like this, and I think I have posted about it before, is sometimes if I'm just It's one of those like if you're bored and you want to do something that isn't just like scroll the internet.
Um, I have this like prompt, which is essentially just um I'll try and maybe post the actual prompt that I use. It's basically uh I want you to take a concept from maybe like grad school level in a given domain and I'll tell you the domain at the end and I want you to write me a parable that would like fully explain like that concept but in an indirect way in the way that parables do. Um, and I want you to write it in such a way that only towards the very end does it maybe, you know, become sort of like clear what the concept is. And then after that, I want you to just like write an explanation for the concept that you were explaining and that you were using.
And I don't know why, but like, you know, there's lots of just interesting domains that I don't know anything about or that are like, you know, I'm interested in. And uh this has just led to me having all of these like stories in my head that like explain and sometimes I can't always remember the term but like um there was one on uh import export and why some goods you tend to import and I was just like I have in my head like this concept and I was like it's so nice to have all of these concepts from lots of different disciplines.
Um, I have this like prompt, which is essentially just um I'll try and maybe post the actual prompt that I use. It's basically uh I want you to take a concept from maybe like grad school level in a given domain and I'll tell you the domain at the end and I want you to write me a parable that would like fully explain like that concept but in an indirect way in the way that parables do. Um, and I want you to write it in such a way that only towards the very end does it maybe, you know, become sort of like clear what the concept is. And then after that, I want you to just like write an explanation for the concept that you were explaining and that you were using.
And I don't know why, but like, you know, there's lots of just interesting domains that I don't know anything about or that are like, you know, I'm interested in. And uh this has just led to me having all of these like stories in my head that like explain and sometimes I can't always remember the term but like um there was one on uh import export and why some goods you tend to import and I was just like I have in my head like this concept and I was like it's so nice to have all of these concepts from lots of different disciplines.
**Eric Newcomer：** 这是我听过的最有人性的事情。"用故事教我"——这是人类最根本的学习方式。我们喜欢最后的反转和收获。我们喜欢学习。你知道怎么去组织它。人类在某种意义上一直很懒，总是用不太适合人脑的方式教人。让所有我想学的东西尽可能地人性化。
非常有意思。
非常有意思。
**Eric Newcomer:** This is the most deeply human thing I've ever heard. It's like teach me what story is the fundamental way. We love a payoff at the end where there's a nice little twist. We love learning. Like you know how to structure it. Like humans in some ways have been lazy in that we just teach people things in sort of like nonhuman ways. Make it make all the things I want to learn as human as possible.
Um very interesting.
Um very interesting.
**Amanda Askell：** 对，能做的事情很多，但这个是我特别喜欢的一个，很有魅力。
**Amanda Askell:** Yeah, there's a lot you can do, but that one's like a charming one that I really like.
**Eric Newcomer：** 希望这是很多次对话中的第一次。我非常享受这次对话。感谢你来参加播客。这就是本期节目。非常感谢 Amanda Askell，也感谢大家收听。请点赞、评论、订阅。我们是新频道，需要你们的支持。去看看以前的视频——我特别喜欢不久前跟 Kara Swisher 的对话。你也可以在 Substack newcomer.co 上关注我们。如果你有大把时间，去看看 Super Bowl Valley Show，我和 Max Child、James Wilman 的脱口秀。感谢观看。
**Eric Newcomer:** Hopefully this is the first of many. I really enjoyed the conversation. Thanks for coming on the podcast. That's our show. Thank you so much to Amanda Ascll and thanks for listening. Please like, comment, subscribe. We're a new channel. We could use all your support. Go watch some of the old videos. I particularly enjoyed my conversation with Cara Swisser not too recently. Uh you can follow along on the Substack newcomer.co or if you've got endless time on your hands, go watch the Super Bowl Valley show, my chat show with Max Child and James Wilman. Thanks for watching.
