视频库 / NO.108
--:--
点击播放，这里会跟随视频显示当前句的中英字幕。
Superintelligence | Nick Bostrom | Talks at Google
节目发布 2014-09-22章节 · 点击跳转视频
0:01
波斯特罗姆与人类未来研究所介绍
1:18
人类处境的两个吸引子：灭绝与技术成熟
4:55
宇宙禀赋的价值与生存风险的定义
7:35
大瓮抽球模型：技术中的黑球风险
12:12
潜在黑球清单与温和技术决定论
15:32
差异化技术发展原则与加速取舍
20:18
技术、协调、洞察三轴与超级智能的双重角色
23:38
通往超级智能的路径：生物增强与全脑仿真
28:27
胚胎筛选与迭代选择的智力增益估算
36:34
专家调查：人类级AI到来时间与智能爆炸
38:56
快速起飞与缓慢起飞：单极体与数字心智生态
45:02
问答：可穿戴增强与人机融合之争
49:14
效用怪物、回形针最大化与价值选择难题
56:52
控制问题：能力控制与动机选择方法
60:11
策略性伪装、正交性论题与工具趋同
64:48
政策制定者能做什么与研究领域的极度稀缺
71:34
我们能挺过去吗：不到50%的毁灭风险
01波斯特罗姆与人类未来研究所介绍
0:01
MODERATOR: Today with us we have Professor Nick Bostrom. He was born in Helsingbrg in Sweden. He's a philosopher at St. Cross College at the University of Oxford. He's known for his work on existential risk, the entropic principle, human enhancement ethics, the reversal test, and consequentialism. He holds a Ph.D. from the London School of Economics, and he is the founding director of both the Future of Humanity Institute and the Oxford Martin Programme on the Impacts of the Future on Technology.
主持人：今天我们请到了尼克·波斯特罗姆教授。他出生于瑞典的赫尔辛堡。他是牛津大学圣十字学院的哲学家。他因在生存风险、人择原理、人类增强伦理学、逆转检验以及后果主义等方面的研究而知名。他拥有伦敦政治经济学院的博士学位，并且是人类未来研究所以及牛津马丁学院"技术对未来的影响"项目的创始主任。
0:36
He's the author of over 200 publications, including the book we are presenting today, "Superintelligence." And he has been awarded the Eugene R. Gannon Award and has been listed in "Foreign Policy"'s Top 100 Global Thinkers list. Please join me in welcoming Professor Nick Bostrom. [APPLAUSE] NICK BOSTROM: Yeah, great. Thanks you for coming. And I'm not going to try to summarize the entire book, but I want to give some of the background from which this work emerges. So I run this thing called The Future of Humanity Institute, which has a very sort of ambitious name.
他撰写了200多篇著作，其中包括我们今天要介绍的这本书——《超级智能》。他曾获得尤金·R·甘农奖，并入选《外交政策》杂志"全球百大思想家"榜单。请和我一起欢迎尼克·波斯特罗姆教授。（掌声）尼克·波斯特罗姆：好的，非常好。谢谢大家的到来。我不打算把整本书都总结一遍，但我想介绍一下这项工作产生的一些背景。我在运营一个叫"人类未来研究所"的机构，它有一个非常宏大的名字。
02人类处境的两个吸引子：灭绝与技术成熟
1:18
It's a small research center where mathematicians, philosophers, and scientists are trying to think through the really big-picture questions for humanity, ones that often traditionally have been relegated to crackpots, relegated to journalists or retired physicists to write some popular book. But questions that are actually extremely important and, I think, deserve close attention. So to give some sense of where I'm coming from, one can think about the human condition, in grand schematic terms, on a diagram like this, where we plot time on the x-axis, and then on the other axis, some measure of capability, like level of technological advancement.
这是一个小型研究中心，数学家、哲学家和科学家们在这里试图思考真正宏观的问题关乎人类的问题，这些问题传统上往往被归给那些疯子，或者被丢给记者、退休的物理学家去写点通俗读物。但这些问题其实极其重要，我认为值得认真对待。为了让大家了解我的出发点，我们可以从宏观的、示意性的角度来看待人类的处境，就像这张图一样，横轴是时间，另一个轴则是某种能力的度量，比如技术发展的水平。
2:03
Measure of the overall economic productivity that we have at some given point in time. And what we take to be the normal human condition-- the idea that you wake up in the morning and then commute to work and sit in front of a screen, and you're having too much rather than too little to eat, is a narrow band within this much larger space of possibilities. It's obviously an anomaly on evolutionary time scales. The human species is fairly young on this planet. It's also an anomaly on just historical time scales.
或者是我们在某个特定时间点所拥有的整体经济生产力的度量。而我们所认为的正常人类处境——早上醒来，通勤上班，坐在屏幕前，吃得过多而不是过少——只是这个大得多的可能性空间中的一个狭窄区间。从演化的时间尺度来看，这显然是个反常现象。人类这个物种在这颗星球上还相当年轻。即便只从历史的时间尺度来看，它也是个反常现象。
2:35
For most of human history, we were inhabiting a Malthusian state, and it's really only in the last few hundred years that we've kind of soared up. And even then, just in some parts of the world. It's also a huge anomaly, obviously, in space, the little crust of [INAUDIBLE] planet being very different from most of the stuff around us, which is just a ultra-high vacuum. And yet we tend to think that this is the normal way for things to be, that any claim that things might be very different is a radical claim that needs some extraordinary evidence.
在人类历史的大部分时间里，我们都处在马尔萨斯状态中，真正的改变只发生在最近这几百年我们才算是真正腾飞起来。而且即便如此，也只是在世界上的某些地方。显然，它在宇宙中也是个巨大的反常——[听不清]这颗行星薄薄的一层地壳，和我们周围绝大多数东西都极不一样，而周围那些东西不过是超高真空。然而我们却倾向于认为这才是事物的常态，任何声称情况可能大不相同的说法，都是需要非同寻常的证据来支持的激进主张。
3:07
Yet it's possible, if we reflect on it, that the longer the time scale we're considering, the greater the probability that we will exit this human condition in either one of two ways, downwards or upwards. This picture has two attractor states. So if we exit the human condition in the downwards direction-- there is in population biology the concept of a minimum viable population size. With too few individuals left, they can't sustain themselves. There's an attractor state down there, which is extinction.
可如果我们仔细想想，其实很有可能：我们考虑的时间尺度越长，我们以两种方式之一——向下或向上——脱离这种人类状态的概率就越大。这幅图景里有两个吸引子状态。如果我们是朝向下的方向脱离人类状态——种群生物学里有个概念叫最小可存活种群规模。当剩下的个体太少时，它们就无法维持自身延续。下面那里有一个吸引子状态，那就是灭绝。
3:37
Once you're extinct, you tend to stay extinct. And more than 99.9% of all the species that once flew, crawled, or swam on this planet are extinct, so that certainly is one possible future. Another way that the human condition could end would be that we exit in the upwards direction. And there, too, I think that there is an attractor state. If and when a civilization manages to obtain technological maturity-- meaning, we have developed most of all technologies that are physically possible to be developed, and we have the ability to spread through space in a reasonable way, through automated self-replicating colonization probes-- then the destiny might be pretty well set.
一旦灭绝了，你往往就永远灭绝了。而曾经在这颗星球上飞过、爬过或游过的所有物种中，超过99.9%都已经灭绝了，所以那当然是一种可能的未来。人类状态终结的另一种方式，是我们朝向上的方向脱离它。而我认为，在那个方向上也存在一个吸引子状态。如果并且当一个文明成功达到技术成熟——意思是，我们已经开发出了物理上可能开发的绝大多数技术，并且我们有能力以一种合理的方式扩散到太空中去，通过自动化的自我复制式殖民探测器——那么命运可能就基本定下来了。
4:25
The level of existential risk will go down, and maybe we could continue on like that for millions and billions of years, just growing at the significant fraction of the speed of light indefinitely, until the cosmological expansion makes it impossible to reach any further resources. Like if something is too far away today, then by the time we would get there, as it were, it has moved further away. So there's a finite bubble of stuff that something starting from what we are could, in principle, get.
生存性风险的水平会降下来，也许我们能就这样持续几百万、几十亿年，一直以接近光速的相当一部分速度无限扩张下去，直到宇宙膨胀使我们再也无法触及更远的资源。比如说，如果某样东西今天就太远了，那么等我们抵达那里的时候，它其实已经移动得更远了。所以，从我们这样的起点出发，原则上能够获取的东西构成一个有限的“气泡”。
03宇宙禀赋的价值与生存风险的定义
4:55
So that whole bubble of stuff, I call humanity's cosmic endowment. There is a lot of it there. And that might be another possible attractor state. If one has to view that-- that this stuff that could, in principle, be reached there is very important, if one has some kind of view on ethics, where fundamentally, the moral significance of an experience or a life does not depend on when it is taking place-- just as many people have the belief that from moral point of view, it doesn't matter where it takes place.
我把这整个气泡里的东西称为人类的宇宙禀赋。那里面的东西非常多。而那可能是另一个可能的吸引子状态。如果你认为——认为原则上可以触及的这些东西非常重要，如果你持有某种伦理学观点，即从根本上说，一段经历或一条生命的道德重要性并不取决于它发生在什么时候——就像许多人相信的那样：从道德的角度看，它发生在什么地方也无关紧要。
5:37
Like if you travel to Africa and you suffer there, it's as bad as if you were suffering here. If one has that view about time, then this cosmic endowment matters a lot. Because it could be used to create an enormous amount of value. We can count it up, roughly. We know that there are billions of galaxies, each with billions of stars, and each of those stars could have billions of people living around it for billions of years. And you get an enormous number of orders of magnitude if you try to measure the size of the future, even just assuming biological instantiations of minds.
比如你去非洲旅行并在那里受苦，这和你在这里受苦一样糟糕。如果你对时间也持这种看法，那么这份宇宙禀赋就至关重要。因为它可以被用来创造出极其巨量的价值。我们可以大致算一算。我们知道有几十亿个星系，每个星系有几十亿颗恒星，而每颗恒星周围都可能有几十亿人生活，并且持续几十亿年。所以如果你试着衡量未来的规模，你会得到一个数量级极其庞大的数字，哪怕只假设心智以生物形式来实现。
6:15
If you're imagine a more efficient digital implementation, you can add another big chunk of orders of magnitude. And so what you find fairly robustly, if you just work the numbers, is that if you have this evaluative view, some broadly-aggregated consequentialist view, then even a very, very small reduction in the net level of existential risk will be worth more, in expected utility terms, than any interventions you could do that would only have local effect here. Even something as wonderful as, like, curing cancer or eliminating world hunger would really be, from this evaluative perspective, insignificant compared to reducing the level of existential risk by, say, 1/100th of one percentage point.
如果你设想一种更高效的数字化实现方式，还可以再加上一大截数量级。所以，只要你把数字算一算，就会相当稳健地发现：如果你持有这种价值评估观，某种广义的加总式后果主义观点，那么哪怕是生存性风险净水平上极其微小的降低，从期望效用的角度看，也比你能做的任何只在此地产生局部影响的干预更有价值。即便是像治愈癌症或消除世界饥饿这样美好的事情，从这种价值评估的视角看，与把生存性风险的水平降低哪怕百分之一个百分点相比，也都微不足道。
6:58
So this level of existential risk becomes, then, maybe an important lens through which to look at global priorities. I define it as a risk that either threatens the survival of Earth-originating intelligent life, or threatens to permanently and drastically destroy our potential for desirable development. And now I think that maybe in a complete accounting of ethics, there are other variables, as well, to take into account. We have particular obligations to people that are near and dear to us. There might be other things in addition to this sort of aggregated consequentialist component.
所以生存性风险的水平也许就成了审视全球优先事项的一个重要透镜。我把它定义为这样一种风险：它要么威胁到源自地球的智慧生命的存续，要么威胁到永久且剧烈地摧毁我们实现理想发展的潜力。不过我现在认为，在一份完整的伦理学清单里，可能还有其他变量也需要纳入考量。我们对那些与我们亲近、与我们相爱的人负有特定的义务。除了这种加总式后果主义的成分之外，可能还有别的东西。
04大瓮抽球模型：技术中的黑球风险
7:35
Nevertheless, I think it's in there and it's important. So if one then tries to look more carefully at this category of existential risk-- like, what could actually go wrong in this way? What could permanently destroy our future? It's a very small subset of all the things that can go wrong. Like most things that threaten human welfare don't really create any existential risk. So it kind of narrows down the range of concerns quite significantly. A first distinction that is obvious in this field is the distinction between risks arising from nature and risks arising in some way from human activity.
不过尽管如此，我认为它是其中一部分，而且很重要。那么如果我们再仔细看看生存性风险这一类别——到底有什么可能以这种方式出问题呢？有什么可能永久地摧毁我们的未来？它只是所有可能出问题的事情中非常小的一个子集。比如大多数威胁人类福祉的事情，其实并不构成任何生存性风险。所以这就把需要关注的范围大大收窄了。这个领域里一个显而易见的初步区分，是源自自然的风险与以某种方式源自人类活动的风险之间的区分。
8:14
And a fairly robust result, I think, is that all the big existential risks, at least if we are thinking a time scale of 100 years or so, are anthropomorphic, arising from human activities. And one can see that just by reflecting that the human species has already been around for 100,000 years. So if firestorms and earthquakes and asteroids haven't done us in in the last 100,000 years, probably not going to do us in in the next 100 years. Whereas we will be introducing entirely new kinds of hazards into the world that we have no track record of surviving.
而我认为一个相当稳健的结论是：所有重大的生存性风险，至少如果我们考虑的是100年左右的时间尺度，都是人为的，源自人类活动。只要想一想人类这个物种已经存在了10万年，就能看出这一点。如果火风暴、地震和小行星在过去10万年里没能把我们灭掉，那么在接下来的100年里大概也灭不掉我们。而我们却将把全新种类的危险引入这个世界，对于这些危险我们并没有幸存下来的记录。
8:46
And more specifically, I think all the really big existential risks are related to certain anticipated future technologies that we might well develop over the coming decades or 100 years. And another way to-- another framing that makes a similar point is to think in terms of this metaphor of a great urn which contains a lot of balls. The balls represent different technologies that can be discovered, or more broadly, the different ideas that we can invent. And throughout human history, we have reached into this ball repeatedly and pulled out ball after ball.
更具体地说，我认为所有真正重大的生存性风险，都与某些可预期的未来技术有关，那些我们很可能在未来几十年或100年内开发出来的技术。还有另一种——另一种表达类似观点的框架，是设想这样一个比喻：一个巨大的瓮，里面装着许多球。这些球代表可以被发现的不同技术，或者更宽泛地说，代表我们能够发明的各种想法。纵观人类历史，我们一次又一次把手伸进这个瓮里，一个接一个地掏出球来。
9:23
And on balance, all these discoveries, all these ideas, have been an immense boon for us. It's because of all these ideas that we now live in abundance, and why there can be seven billion of us. There have been some discoveries, perhaps, that have been mixed blessings, that have done both good and ill. And a relatively small number-- it's not totally trivial to think about it, but balls that we would have been better off without having extracted from this urn, like discoveries that we would be better without.
总体而言，所有这些发现、所有这些想法，对我们来说都是巨大的福祉。正是因为所有这些想法，我们今天才能生活在富足之中，才可能有七十亿人口。也许有一些发现是喜忧参半的，既带来了好处也带来了坏处。还有相对少数的——想清楚这一点其实并不完全简单，但确实有些球，我们要是没从这个瓮里掏出来会更好，也就是一些我们没有会更好的发现。
9:54
I mean, maybe chemical weapons, say, or nuclear weapons. Or perhaps, like, torture instruments of different kinds. There are few things that seems to have been clearly bad. But there hasn't been any discovery so far made that is such that it automatically destroys the civilization that discovers it. So there hasn't been any black ball pulled out from this urn. And we can ask what that kind of discovery could look like. What would be a possible discovery, such that it kind of almost automatically spells the end of the discoverers?
我是说，比如化学武器，或者核武器。又或者，比如各种各样的刑具。有少数几样东西，看起来是明显糟糕的。但迄今为止还没有出现过这样一种发现：它会自动毁掉那个做出该发现的文明。也就是说，我们还没有从这个瓮里掏出过黑球。我们可以问问，那种发现会是什么样子。什么样的发现会几乎自动地宣告发现者的终结？
10:31
And it might be useful here to think of a counter [INAUDIBLE]. So we discovered, just over half a century ago, nuclear weapons. And it turned out, fortunately, that in order to make a thermonuclear device, you need some difficult-to-obtain raw materials. You need highly enriched uranium or plutonium. And the only way to get those is by having some large facility that's very expensive to build, takes a lot of energy, is easy to see. So very few people can build their own nuclear device. But suppose it had turned out to be differently.
这里也许可以想一个反面的例子。我们在半个多世纪前发现了核武器。而幸运的是，事实证明要制造一枚热核装置，你需要一些很难获得的原材料。你需要高浓缩铀或钚。而获得它们的唯一途径，是拥有某种造价极其昂贵的大型设施，它耗能巨大，而且很容易被发现。所以极少有人能自己造出核装置。但假设结果并非如此。
11:04
Suppose it had turned out instead that it had been possible to make a thermonuclear warhead by some simple procedure, like baking sand in your microwave oven, OK? So now we know physics doesn't allow for that. But before you actually did the relevant physics, how could you possibly have known whether particle physics would have provided some easy route to unleash these kinds of entities? So if that had been the case, then presumably that would then be the endpoint of human civilization. Once it became so easy to destroy entire cities, we could never again have cities, and maybe we would have been knocked back to the Stone Age.
假设事实证明，用某种简单的方法就能造出热核弹头，比如把沙子放进微波炉里烤一烤，对吧？当然，现在我们知道物理定律不允许这样。但在你真正做出相关的物理研究之前，你怎么可能知道粒子物理学会不会提供某种轻而易举就能释放这类力量的途径呢？所以如果情况真是那样，那大概就会是人类文明的终点。一旦摧毁整座城市变得如此容易，我们就再也不可能拥有城市，也许我们会被打回石器时代。
11:42
And by the time we would have again climbed back up to the technology level where somebody could build microwave ovens, we would presumably fall back again. And that might be forever the end of it. But so we were lucky on that occasion, but the question is whether we will continue to be lucky always. Like, whether in this big urn, if we keep extracting ball after ball, whether eventually we will pull out the black ball. If there is a black ball in there and we just keep pulling them out, then eventually, presumably, we will get it.
而等到我们再次爬回到有人能造出微波炉的技术水平时，我们大概又会重新跌落下去。那也许就是永远的终结了。所以在那件事上我们是幸运的，但问题是我们会不会一直这么幸运下去。也就是说，在这个大瓮里，如果我们不停地一个接一个掏球，最终会不会掏出那颗黑球。如果里面确实有一颗黑球，而我们又不停地往外掏，那么最终，大概我们就会掏到它。
05潜在黑球清单与温和技术决定论
12:12
And we don't yet have the ability to put a ball back in the urn. We can't undiscover things that we have discovered. So here is a kind of quick list of some areas where one might suspect that there could be these kinds of black balls. And AI is one that I'll kind of come back to more in this talk. There are some others. Synthetic biology will, I think over the coming decades, vastly increase the powers of human beings to change the world around us and ourselves. Those powers might be used wisely or not.
而且我们目前还没有能力把球放回瓮里。我们无法让已经发现的东西变回未被发现。所以这里有一个简要的清单，列出了一些人们可能会怀疑存在这类黑球的领域。人工智能是其中之一，我在这次演讲里还会更多地回到这个话题。还有一些其他的。我认为在未来几十年里，合成生物学将极大地增强人类改变周遭世界以及改变我们自身的能力。这些能力可能被明智地使用，也可能不会。
12:48
Molecular nanotechnology. Not the kind of thing that makes car tires today, but some kind of more advanced future version of that, like Eric Drexler imagined. Totalitarianism-enabling technology. So remember, again, the definition of an existential risk-- not only extinction scenarios, nice but also ways to permanently lock ourselves in to some radically suboptimal state. And you can imagine that maybe new technological discoveries that make surveillance very easy, or some new discovery that makes it possible, through psychological or neurophysiological techniques, to modify desires could sort of change some of the parameters of the sort of sociopolitical game, where new types of social organization becomes a lot easier to establish and maintain.
分子纳米技术。不是今天用来做汽车轮胎的那种，而是某种更先进的未来版本，就像埃里克·德雷克斯勒（Eric Drexler）设想的那样。使极权主义成为可能的技术。所以再回忆一下生存风险的定义——不只是灭绝的情景，那当然也算，还包括那些让我们永久锁死在某种极端次优状态里的方式。你可以想象，也许某些新的技术发现让监控变得极其容易，或者某种新发现使得通过心理学或神经生理学手段来改变人的欲望成为可能，这就可能改变社会政治博弈的某些参数，让新型的社会组织形式变得远比以往容易建立和维持。
13:38
Human modification, geoengineering. There are more you could add there, and I've left a lot of these bullets below here on the list unknown. So it's useful to reflect that if this question-- what are the biggest existential risks?-- had been asked 100 years ago, then presumably none of the ones that I now would place close to the top would have been listed. They didn't have computers, so they wouldn't have listed machine intelligence. Synthetic biology was not even a concept, nor nanotechnology. Maybe they would have worried some about, like, totalitarian tendencies.
人类改造、地球工程。这里还可以再加上更多，我在下面这个列表里留了很多条目，写着“未知”。所以值得反思一下：如果这个问题——最大的生存风险是什么？——在100年前被提出来，那么大概我现在会排在最前面的那些，一个都不会出现在名单上。他们那时没有计算机，所以不会列出机器智能。合成生物学当时连概念都没有，纳米技术也一样。也许他们会有点担心极权主义倾向之类的。
14:11
But the others, not so much. So if we reflect on our situation today, we have to maybe acknowledge, from standing outside and looking in, that there are probably some additional existential risks that are not yet on our radar but that could turn out to be as significant as some of the others. Which as I said, there could be high value to doing analysis on this and research to try to find them out. But if one combines these considerations with one other hypothesis-- say, a mild or moderate form of technological determinism, which I think is true-- the idea, basically, that assuming science and technology continues, there's no global collapse, then eventually we'll probably discover all technologies that could be discovered.
但其他的，就没怎么想到了。所以如果我们反思一下今天的处境，可能不得不承认，从局外往里看，很可能还存在一些额外的生存风险，目前还不在我们的雷达上，但最终可能和其他那些一样重大。正如我说的，对此做分析、做研究去把它们找出来，可能价值很高。但如果把这些考虑和另一个假说结合起来——比如说一种温和或中等程度的技术决定论，我认为它是成立的——这个想法基本上是说，假定科学和技术继续发展，没有发生全球性崩溃，那么最终我们大概会发现所有可被发现的技术。
15:03
At least all general-purpose technologies that have a lot of implications in many fields. I think that's fairly possible. It's not assured, but that level of technological determinism seems quite possible to me. It's a little bit like-- if you think of a big box that starts out empty and you pour in sand in it, this is like, you can fund one kind of research here. You can fund another kind of research. And what research you fund, where your priorities are, that determines where the sand piles up in this box.
至少是所有在很多领域都有广泛影响的通用技术。我觉得这相当有可能。这并非板上钉钉，但那种程度的技术决定论在我看来相当有可能。这有点像——你可以想象一个一开始空着的大箱子，你往里面倒沙子，就好比说，你可以资助这边的某一类研究。你也可以资助另一类研究。而你资助什么研究、优先级放在哪里，就决定了沙子会在这个箱子里的哪个位置堆起来。
06差异化技术发展原则与加速取舍
15:32
So you get different technologies, depending on what you do. But over time, if you just keep pouring in sand, then eventually the whole box will fill up. And so that seems fairly possible. Now if one has that view, then how should one kind of-- what attitude should one take to all of this? Like what should we do? So one possible response is one I think best expressed by this blogger. I don't know who it is. Washbash commented on some blog that "I instinctively think go faster. Not because I think this is better for the world.
所以你会得到不同的技术，取决于你怎么做。但随着时间推移，如果你只是不停地往里倒沙子，最终整个箱子都会被填满。所以这看起来相当有可能。那么如果你持这种观点，接下来该怎么——我们该对这一切抱持什么态度？我们该做什么？有一种可能的回应，我觉得这位博主表达得最好。我不知道他是谁。Washbash 在某个博客上评论说：“我本能地觉得应该加快。不是因为我认为这对世界更好。
16:02
Why should I care about the world when I'm dead and gone. I want it to go fast, damn it! This increases the chances I have of experiencing a more technologically advanced future." So here we've got to be clear what exactly the question is that we're asking. So if the question is, what would be best for me personally? What should we favor or hope for, from a egoistic point of view? Then I think that Washbash is correct. From an individual point of view, if-- well, first of all, if you're somehow hoping for these cosmic lifespans of millions of years, and being able to travel and expand into the universe, then clearly that's not going to happen unless something radical changes.
我都死了、不在了，我干嘛还要在乎这个世界。我就是想让它快点，见鬼！这样我能亲身体验一个技术更先进的未来的机会就更大。”所以在这里我们必须搞清楚，我们问的到底是什么问题。如果问题是：对我个人来说什么是最好的？从利己的角度出发，我们应该支持什么、期盼什么？那我认为 Washbash 是对的。从个人角度看，如果——首先，如果你多少还指望能有那种几百万年的宇宙级寿命，能够旅行、扩张到宇宙中去，那么很明显，除非发生某种彻底的改变，否则这不会发生。
16:47
Like the way things are going, I'm sad to say, we're all just going to die from aging in a few decades. Like we're all rotting. So the only way that that could possibly change is some radical upset. Like some cure for aging, or uploading into computers, or something really radical would have to happen to kind of thwart that. So that would be reason to favor faster technical growth. Or even, if you're despairing of that, even you could just hope to have more interesting gadgets around and a higher standard of living, which we can hope for through technology.
照目前的趋势，我很遗憾地说，我们所有人都会在几十年内因衰老而死。我们都在腐朽。所以唯一可能改变这一点的，就是某种彻底的颠覆。比如某种抗衰老疗法，或者上传到计算机里，或者某种真正激进的事情必须发生，才能扭转这个结局。所以这就构成了支持更快技术发展的理由。或者，就算你对那个已经不抱希望了，你至少也可以指望身边有更有意思的小玩意儿、更高的生活水平，而这些我们可以通过技术来期待。
17:19
However, if the question we ask is instead, what would be best from an impersonal point of view? Then I think the answer is quite different, something perhaps closer to this principle of technological development, rather than maximize the speed with which you rush ahead. This principle would say that we should "retard the development of dangerous and harmful technologies, especially ones that raise the level of existential risk, and accelerate the development of beneficial technologies, especially those that reduce the existential risks posed by nature or by other technologies."
然而，如果我们问的问题换成：从非个人的、超然的角度看，什么才是最好的？那我认为答案就相当不同了，可能更接近这条技术发展原则，而不是把一路狂奔的速度最大化。这条原则会说，我们应该“延缓危险和有害技术的发展，尤其是那些会提高生存风险水平的技术，并加速有益技术的发展，尤其是那些能降低由自然或其他技术带来的生存风险的技术。”
17:56
So the idea here is that rather than asking the question for some hypothetical technology, would we be better off without it? We ask a different question. Because basically, on this moderate form of technological determinism, that's just not on the table. We can't relinquish a technology permanently. But what we should think of instead is on the margin, should we try to hasten the arrival of some technology or slow it down? We might be able to make some difference there, say, by a couple months. And we want to think about how that small difference will influence our likelihood of harvesting this cosmic endowment.
所以这里的思路是，与其对某项假想的技术去问这样的问题——没有它我们会不会过得更好？我们问一个不同的问题。因为基本上，在这种温和形式的技术决定论下，那根本不是一个可选项。我们没法永久地放弃一项技术。但我们应该考虑的是在边际上：我们该设法让某项技术早点到来，还是让它慢一点？我们也许能在那里造成一点差别，比如说几个月。而我们要思考的是，这个微小的差别会如何影响我们收获这份宇宙禀赋的可能性。
18:34
If you think that it was literally impossible to even make a small difference in the timing, then that would mean that all the funding and all the effort that goes into technology development would just be wasted. So presumably we think we have some ability to at least move things around in time. And the principle of differential technological development suggests that it might be quite significant, sometimes, exactly when different things arrive, particularly the sequence in which different technologies arrive.
如果你认为在时间上哪怕造成一点点差别都是绝无可能的，那就意味着投入技术开发的所有资金和所有努力都是白费的。所以想必我们认为自己至少有一定能力让事情在时间上挪动一下。而差异化技术发展原则表明，不同事物到来的时间点——尤其是不同技术到来的先后顺序——有时候可能相当关键。
19:03
So if there's going to be, at some point, like a really harmful bio-engineered pathogen, and it could spread really easily and it's very lethal, and there's going to be, at some point, like a universal vaccine, you want to invent the vaccine before you invent the pathogen. If there's going to be, at some point, machine superintelligence, and if there is some possible technology that could assure the safety of machine superintelligence, you want the latter to come before the former. AUDIENCE: There's an argument that trying to retard development of technology would make it more dangerous because you're driving it underground, or you have less opportunity to do it out in the open, develop safe [INAUDIBLE].
所以，如果在某个时候会出现一种真正有害的、经过生物工程改造的病原体，它能非常容易地传播、致死率很高；而在某个时候也会出现某种通用疫苗，那你会希望先发明疫苗，再发明那种病原体。如果在某个时候会出现机器超级智能，而且如果存在某种技术能够确保机器超级智能的安全，那你会希望后者先于前者到来。观众：有一种论点认为，试图延缓技术的发展反而会让它更危险，因为你把它逼到了地下，或者说你更少有机会在公开场合去做、去发展安全的[听不清]。
19:48
Like we had, for example, the Asilomar guidelines in biotech, which have actually been very effective. For 30 years, there's been no accidents. And if you drive these technologies underground, you don't have the opportunity to have those kinds of safeguards. NICK BOSTROM: Yeah. This-- the principal leaves open whether you should focus on the retarding or the accelerating. If you wanted to retard, maybe one way would be just to refrain from like funding it or actively devoting yourself to accelerating it.
比如我们有生物技术领域的阿西洛马准则（Asilomar guidelines），它实际上非常有效。30年来，没有出过事故。而如果你把这些技术逼到地下，你就没有机会建立那样的保障措施。尼克·波斯特洛姆：是的。这个——这条原则并没有规定你该把重点放在延缓还是加速上。如果你想延缓，也许一种方式就是干脆不去资助它，或者不去主动投身于加速它。
07技术、协调、洞察三轴与超级智能的双重角色
20:18
With regard to AI, which I'll get to later, I think definitely accelerating the work on the safety problem is clearly the way to go. I think it's just a lot easier to make a big difference there than to try to somehow retard the development of AI work itself. Maybe we can return to that more in the Q&A. So we have a picture, perhaps, like this, where again, we're looking at three axes here. So technology on one, this is the same capability on the earlier slide. Coordination-- some measure of the degree to which humankind is able to solve a global coordination problems.
至于人工智能，我后面会讲到，我认为加速安全问题上的工作显然是正确的方向。我觉得在那方面做出重大改变，要比设法以某种方式延缓人工智能研究本身容易得多。也许我们可以在问答环节再回到这个话题。所以我们大概有这样一幅图景，我们同样是在看这三个坐标轴。一个是技术，也就是先前那张幻灯片上的同一个“能力”。协调——某种衡量人类解决全球性协调问题能力程度的指标。
20:58
Avoiding wars and arms races, and polluting our communal resources. And insight-- so a measure of our understanding into what uses of our capability would actually make things better. So it might well be that in order to have the best possible outcome, to have Utopia, that we need maximum amounts of all of these. Like super-duper advanced technology is necessary to realize the best state; great coordination, so we don't use that technology to wage war against one another, as we have through so much of human history; and great wisdom, so that we apply all these abilities to really do things that are worthwhile.
避免战争和军备竞赛，避免污染我们共有的资源。以及洞见——衡量我们对“如何使用我们的能力才真正能让事情变好”的理解程度。所以很可能，为了得到最好的结果，为了实现乌托邦，我们需要这三者都达到最大值。比如说，超级先进的技术是实现最佳状态所必需的；极好的协调，这样我们才不会用那些技术互相开战，而人类历史上大部分时候我们都在这么干；还有极高的智慧，这样我们才能把所有这些能力用来做真正值得做的事。
21:39
So that might be where we have to be, if we want to realize the best possible state. Now that then leaves open the question of whether from the position we are currently in-- at the moment, we would be better off with faster developments in each of these areas. It might be, for example, that even though we ultimately want maximum technology, we would be better off getting that technology only after we have first made more progress on global coordination or wisdom. Anyway, so that's by view of like a broader context.
所以如果我们想实现最好的可能状态，那可能就是我们必须抵达的地方。但这就留下了一个问题：从我们当前所处的位置来看——在此刻，我们是否会因为这几个方面各自发展得更快而变得更好。比如说，也可能虽然我们最终想要最大程度的技术，但更好的做法是等我们先在全球协调或智慧上取得更多进展之后，再获得那些技术。总之，这就是我关于更宏观背景的看法。
22:13
So we are thinking about other existential risks and stuff like that. And superintelligence, as I will talk about, I think is one big existential risk, perhaps. Perhaps, arguably, perhaps the biggest. I'm not sure. But it's peculiar in one respect, that although it's a big danger in its own right, it's also something that could help eliminate other existential risks. So if we imagine like a very simple model, where we have synthetic biology, nanotechnology, and AI-- we don't know which order they will come.
所以我们在思考其他的生存风险之类的东西。而超级智能，正如我接下来要讲的，我认为它是一个重大的生存风险，也许是。也许，可以说，也许是最大的那个。我不确定。但它在一个方面很特别：虽然它本身就是一个巨大的危险，它同时也是某种能帮助消除其他生存风险的东西。所以如果我们设想一个非常简单的模型，里面有合成生物学、纳米技术和人工智能——我们不知道它们会以什么顺序到来。
22:44
Maybe they each have some existential risks associated with them. Suppose we first develop synthetic biology. We get lucky and we get through the existential risks, however big they are. And then we reach molecular nanotechnology, and we are lucky. We get through that, as well. And finally, AI, the existential risks along that path are kind of the sum of these three different ones that we'll each have to surmount. In another trajectory, maybe we get AI first, and we have to face existential risk for that.
也许它们各自都伴随着一些生存风险。假设我们先发展出合成生物学。我们运气不错，闯过了那些生存风险，不管它们有多大。然后我们抵达分子纳米技术，我们又走运了。我们也闯过了那一关。最后是人工智能，这条路径上的生存风险差不多就是这三种不同风险的总和，我们得逐一去闯过。在另一条发展轨迹上，也许我们先得到了 AI，我们就必须面对它带来的生存性风险。
23:12
But then if we do get lucky there, we no longer have to face the risk with synthetic biology and nanotechnology, because we don't have the superintelligence to help us through. So in reality, it gets a lot more complicated than that, and we can discuss the intricacies more in the Q&A. But thinking about the sequencing and timing, I think, rather than yes or no, would we want the technology or not, is like an initial, necessary first step to be able to have any kind of meaningful conversation about this.
但如果我们在那一关走运了，我们就不必再面对合成生物学和纳米技术带来的风险，因为我们并没有超级智能来帮我们渡过难关。所以现实中，情况比这要复杂得多，我们可以在问答环节里更多地讨论这些细节。但我认为，去思考这些技术出现的先后顺序和时间点，而不是简单地问要不要这项技术、答案是「是」还是「否」，是我们要就此展开任何有意义的讨论所必需的第一步。
08通往超级智能的路径：生物增强与全脑仿真
23:38
So superintelligence, I think, will be a big game-changer, the biggest thing that will ever have happened in human history, at some point, this transition to superintelligence. There are two possible pathways, in principle, one could imagine that could lead there. You could enhance biological intelligence. We know biological intelligence has increased radically in the past, in kind of making the human species. Or machine intelligence, which is still far below biological intelligence, insofar as we're focusing on any form of general-purpose smartness and learning ability, but increasing at a more rapid clip.
所以我认为超级智能将是一个巨大的转折点，是人类历史上发生过的最重大的事情，在某个时刻，这个向超级智能的转变会到来。原则上，可以设想有两条可能通往那里的路径。你可以增强生物智能。我们知道，生物智能在过去曾经急剧提升过，某种意义上正是这造就了人类这个物种。或者机器智能，它目前仍远低于生物智能，只要我们关注的是任何形式的通用聪明才智和学习能力，但它提升的速度要快得多。
24:21
So specifically, you can imagine interventions on some individual brain to enhance biological condition. I'll say a few words about that just shortly. Or improvements in our ability to pool our individual information processing devices to enhance our collective rationality and wisdom. I won't talk about that, but that's clearly an exciting frontier, with the internet and new institutions, prediction markets and other things like that.
具体来说，你可以设想对某个个体的大脑进行干预，以增强其生物层面的状况。我等一下会简单说几句这方面的内容。或者是提升我们把各自的信息处理装置汇聚起来的能力，从而增强我们的集体理性和智慧。这个我不会展开讲，但它显然是一个令人兴奋的前沿领域，比如互联网和各种新制度，预测市场之类的东西。
24:51
There are some kind of hybrid approaches, it can vary between biology and machines, the cyborg approach. I personally don't think that that's where the action will be. It just seems to me very difficult, technologically speaking, to create implants that would really significantly enhance our cognitive ability more than you could have by having the same device outside of yourself. So you could say, wouldn't it be great with a little chip in the brain, and you could Google just by thinking about it?
还有一些混合路径，可以在生物和机器之间取不同的比例，也就是「赛博格」路线。我个人不认为真正的突破会出现在那里。在我看来，从技术角度讲，要造出真正能显著增强我们认知能力的植入物——增强的幅度超过把同样的设备放在体外所能达到的效果——是非常困难的。你可能会说，脑子里装个小芯片，一想就能谷歌搜索，那不是很棒吗？
25:23
And well, I mean, I can already Google, and I don't have to have neurosurgery to be able to do that. We have these amazing interfaces like the eyeballs, that can protect 100 million bits per second, straight into dedicated neural wetware that's highly optimized for making sense of this information. And it's really hard to beat that, I think. In any case, I mean, the rate at which sensory information can be entered into the brain is not really the limiting factor. The first thing the brain does with all of this visual information is to throw away almost all of it and just extract the relevant part.
可是，我现在就已经能谷歌搜索了，而且不需要做开颅手术就能做到。我们本来就有像眼球这样了不起的接口，每秒能传送一亿比特的信息，直接进入专门的神经「湿件」，而它对理解这类信息已经高度优化了。我觉得这真的很难被超越。不管怎么说，感官信息进入大脑的速率其实并不是瓶颈所在。大脑对所有这些视觉信息做的第一件事，就是把其中绝大部分丢掉，只提取相关的那一部分。
25:58
And then different versions of machine intelligence, where on the one hand, we have sort of purely synthetic methods that don't care about biology but try to make progress in mathematics and statistics and figure out clever algorithms. And two, approaches that try to learn from this one general intelligence system that already exists, that we can study the human brain for inspiration from that, or maybe even reverse-engineer it. Or in the limiting case, literally copying it in whole-brain emulation, where you would take a particular human brain and freeze it and slice it up, feed those slices through an array of microscopes to take good pictures.
然后是机器智能的不同版本：一方面，有纯粹合成的方法，它们不关心生物学，而是试图在数学和统计学上取得进展，设计出巧妙的算法。第二类，是试图向这个已经存在的通用智能系统学习的方法，也就是研究人脑、从中获得灵感，甚至可能对它进行逆向工程。或者在极端情况下，字面意义上地复制它，也就是全脑仿真：你取一个特定的人脑，把它冷冻起来，切成薄片，让这些切片通过一组显微镜，拍下高质量的图像。
26:45
So you have a stack of these pictures and use automated image-recognition software to extract the connectivity matrix of the neural network that's was in the original brain. And then annotate that with neurocomputational models of each type of neuron works. And finally, run that whole emulation on a sufficiently powerful computer. That would require some very advanced enabling technology that we don't yet have, so we know that that is not just around the corner. On the other hand, it would not require any theoretical breakthrough.
于是你就有了一叠这样的图像，再用自动图像识别软件提取出原本大脑中那个神经网络的连接矩阵。然后为它标注上每一类神经元如何工作的神经计算模型。最后，在一台足够强大的计算机上运行整个仿真。这需要一些我们目前还不具备的非常先进的使能技术，所以我们知道那件事并不是近在眼前。但另一方面，它并不需要任何理论上的突破。
27:21
It would not require any new, deep conceptual understanding of how thinking works. You would only need to understand the components of the brain to be able to make progress with that. So it's an open question which of these will get there first. Different researchers have their own favorite bets on that. One thought that sometimes is put to me is that-- OK, so Nick, you're worried about this AI stuff. So maybe what we should do is really try to push ahead with biological enhancement, so that we can kind of keep up with the computer.
它不需要对思维如何运作有任何全新的、深刻的概念性理解。你只需要理解大脑的各个组成部分，就能在这条路上取得进展。所以，究竟哪一条路会先到达终点，这是个悬而未决的问题。不同的研究者各有各偏爱的押注。有人有时会对我说——好吧，尼克，你担心 AI 这些事。那也许我们该做的，是真正大力推进生物增强，这样我们就能跟上计算机的步伐。
27:55
The computer's going to get smarter, but maybe if we enhance our own intelligence rapidly enough, we can keep one step ahead. I think that that's misguided. And in fact, if we do figure out ways to enhance biological cognition, I think that will only hasten the time when machines overtake us. Because basically, we will have smarter people doing the AI research and the computer science, and they will solve the problem faster. I still think that would probably have reason to try to accelerate biological cognitive development.
计算机会变得越来越聪明，但如果我们能足够快地增强自己的智能，也许就能始终领先一步。我认为这种想法是错的。事实上，如果我们真的找到了增强生物认知的方法，我认为那只会加快机器超越我们的到来。因为说到底，那意味着会有更聪明的人去做 AI 研究和计算机科学，他们会更快地解决这个问题。不过我仍然认为，我们大概还是有理由去努力加速生物认知能力的发展。
09胚胎筛选与迭代选择的智力增益估算
28:27
But not so that we can keep ahead of the computers, but that so that when the time comes where we will create intelligent machines, that we will be more competent at doing so. So let me just say a few words on this biological cognitive enhancement, because it might be-- especially if you think of arrival dates for artificial intelligence, where it's not just around the corner, but maybe it will happen in the latter half of this century or something like that-- by that time, that could be enough time to have a new cohort of cognitively enhanced people around.
但目的不是为了让我们领先于计算机，而是为了当我们创造出智能机器的那一刻到来时，我们能更有能力把这件事做好。那我就简单说几句生物认知增强，因为这可能——尤其是如果你认为人工智能到来的时间不是近在眼前，而可能发生在本世纪下半叶之类的时候——到那时，时间可能足够让一代经过认知增强的人成长起来。
28:58
And the technology that I think will first enable cognitive enhancement-- my best guess is that it'll be through genetic interventions. There are other paths, obviously-- smart drugs and such. I just-- I don't hold out much hope that they will do a great deal to improve general-purpose smartness. They might-- if there were a simple chemical that you could just inject and it would make you a lot smarter, I think evolution would find a way to endogenously produce that chemical. I think there might be ways to improve some peripheral characteristics, like mental energy, say, or concentration.
而我认为最先能实现认知增强的技术——我最好的猜测是通过基因干预。当然还有别的路径，比如益智药之类的。只是我并不太指望它们能在提升通用聪明才智上起多大作用。它们也许可以——但如果真有一种简单的化学物质，注射一下就能让你聪明很多，我想演化早就会找到办法让身体自己内源性地产生这种物质了。我认为也许有办法改善一些外围特征，比如说心智精力，或者专注力。
29:35
And we can see that evolution would have optimized us for a certain type of environment where there are trade-offs between, maybe, metabolic consumption of the brain and the amount of mental energy you have. And in the environment of evolutionary adaptiveness, the optimum point for that trade-off is at one place, and now we want to move that. And maybe it could have some stimulant that just increases the burn rate of calories and give you more mental energy. So peripheral adjustments like that, I think we could do maybe through drugs.
我们可以看到，演化是针对某一类特定环境对我们做了优化，在那种环境里存在权衡，比如大脑的代谢消耗和你拥有的心智精力之间的权衡。在演化适应环境中，这个权衡的最优点落在某个位置，而现在我们想把它挪一挪。也许可以有某种兴奋剂，它只是提高卡路里的燃烧速率，让你获得更多的心智精力。像这样的外围调整，我觉得也许可以通过药物来实现。
30:03
But raw cleverness, I think genetics is a more likely initial technology to do that. And so one way that that could work is in the context of in vitro fertilization, where you have normally, in the course of standard fertility procedure, maybe some six, eight, or ten eggs produced. And then the doctor chooses one of those to implant. And at the moment, you can look for like obvious abnormalities. You might screen for some monogenic disorders or for Down syndrome, which is done. But you can't really select positively for some complex trait today, because we don't yet know the genetic architecture for, say, intelligence.
但要提升最根本的聪明程度，我认为遗传学更可能是最初实现这一点的技术。一种可行的做法是在体外受精的场景下：在标准的助孕流程中，通常会产生大概六个、八个或十个卵子。然后医生从中挑选一个来植入。目前，你可以查看是否有明显的异常。你可能会筛查一些单基因疾病，或者唐氏综合征，这些现在都在做。但今天你还没法真正针对某种复杂性状做正向选择，因为我们还不知道比如智力的遗传结构。
30:53
But we will, I think, soon know that, because the price of gene sequencing is falling, and it's now coming down sufficiently where it is becoming feasible to run these very large-scale studies with hundreds of thousands or even millions of subjects. And because it turns out that the additive heritability, the variance in that in humans, is not due to like one or two genes that differ in between us, but a lot of genes-- maybe hundreds, maybe even a few thousand-- that each have a very, very small effect.
但我认为我们很快就会知道，因为基因测序的价格在不断下降，现在已经降到足够低，使得开展这些几十万甚至上百万受试者规模的大型研究变得可行。还因为事实证明，加性遗传率——人类在这方面的变异——并不是由一两个在我们之间存在差异的基因造成的，而是由大量基因造成的，也许几百个，甚至几千个，每一个的效应都非常非常小。
31:29
And so to discover a very, very small effect, you need a very large sample size. And so you need to sequence a lot of genomes, and that was too expensive to do, really. But now there are studies underway with hundreds of thousands of people, and maybe soon millions. So that, I think, will tell us some of this information that would be needed. And then to start doing this, nothing else would be required. No new technologies at all. You just have the information and you sequence it and select the embryos based on that.
而要发现非常非常小的效应，你就需要非常大的样本量。所以你需要测序大量的基因组，而这以前实在太贵了。但现在已经有几十万人规模的研究在进行，也许很快就会有上百万人的。所以我认为，那将会告诉我们其中一部分所需要的信息。而要开始做这件事，别的什么都不需要。完全不需要任何新技术。你只要有了那些信息，做测序，然后据此挑选胚胎就行了。
31:56
Now this would be vastly potentiated if it were combined with another technology that we don't yet have ready for use in humans, which is the ability to derive gamete from stem cells. So then you could do iterated embryo selection. We would generate an initial pool of embryos, select one that's highest in the expected trait value of interest, and then use that embryo to derive gametes-- sperm and ova-- that you could then recombine to get a new set of embryos. You pick out the best of those, and you repeat.
不过，如果能和另一项我们尚未准备好用于人类的技术结合起来，这件事的威力会被极大放大，那就是从干细胞中获得配子的能力。那样你就可以做「迭代式胚胎选择」。我们先生成一批初始胚胎，挑出在我们关心的性状上期望值最高的那一个，然后用这个胚胎来获得配子——精子和卵子——再让它们重新组合，得到新的一批胚胎。你再从中挑出最好的，如此反复。
32:31
So this technology here, the ability to create artificial gametes through stem cells, has been developed and done in mice. But a significant amount of additional work would be required to make it safe for use in humans. But if you had this-- and this might take anything from 10 to 30 or 40 years. It's hard to know. This would have the effect of collapsing the human generation cycle from 20, 30 years to a couple of months. And so if you imagine this kind of old mad scientist eugenics program where they would breed humans for like 500 years, and make very sure who mated with whom-- which setting aside all the ethical complications involved in that, which are legion, but I'm not going to talk about them here, not because I don't think they are there, but I just want to focus on the technical stuff-- it's just infeasible on a lot of different levels.
这项技术，也就是通过干细胞制造人工配子的能力，已经被开发出来，并在小鼠身上实现了。但要让它安全地用于人类，还需要相当多的额外工作。但如果你拥有了这项技术——而这可能需要 10 年到 30 年、40 年不等。这很难说。它的效果是把人类的世代周期从二三十年压缩到几个月。所以，如果你设想那种老式疯狂科学家的优生学计划，他们要让人类繁育大概 500 年，并且严格控制谁和谁交配——先撇开其中涉及的所有伦理问题不谈，那些问题多如牛毛，但我在这里不会讨论它们，并不是因为我认为它们不存在，而只是因为我想聚焦在技术层面——那种做法在很多层面上都根本不可行。
33:32
But here, you would instead have something that could be done-- instead of 500 years, you could have it done over a year. And instead of changing the breeding patterns of large populations, you would have a Petri dish and a scientist plucking around in that. And so through that, you would be able to probably achieve sort of weak forms of superintelligence in biology. I did an analysis with a colleague of mine, Carl Shulman, quite recently where we tried to estimate, for different assumptions about the selection power applied, what the gain in intelligence would be.
但在这里，你得到的是一件真正可以做到的事——不需要 500 年，一年之内就能完成。而且不是去改变大规模人群的婚配模式，而只是一个培养皿和一位科学家在里面操作。通过这种方式，你大概能在生物层面上实现某种弱形式的超级智能。不久前我和我的同事卡尔·舒尔曼（Carl Shulman）做过一项分析，我们尝试估算，在关于所施加的选择强度的不同假设下，智力上的增益会有多大。
34:08
And so you can see here that if you just produce two random embryos and select the best one-- not the one that's actually best, but the one that looks most promising, to the extent that there is an additive genetic heritability, you may get four IQ points from that. So if instead, you select the best of 1 in 10, or 11, you can see here, even if you could select the best of 1 in 1,000, you only get maybe 24 IQ points. This is with single-shot selection. But if you did this iterated embryo selection, and you could do five generations of selecting the best of 1 in 10, then you might get as many as 65 IQ points.
你可以看到，如果你只是产生两个随机胚胎，然后选出更好的那个——不是实际上最好的那个，而是看起来最有希望的那个——在存在加性遗传率的前提下，你大概能得到 4 个智商点。那么如果你改为从 10 个或 11 个里选最好的一个，你可以看到，即便你能从 1000 个里选出最好的一个，也只能得到大约 24 个智商点。这还只是单轮选择的情况。但如果你采用这种迭代式胚胎筛选，做五代、每代都从10个里挑最好的一个，那么你可能获得多达65个智商点的提升。
34:48
And with 10 generations of 1 in 10, you'd get far above what we've had in human history. You'd get the kind of phenotypes that have never existed in all of human history, the [INAUDIBLE] and stuff like that. So you observe here that while you get quickly diminishing returns by just doing one-shot selection from a pool of embryos, you largely avoid that by doing this iterated selection. So yeah, I'm going to skip through this. Yeah, so that does look like it should be feasible without any sort of magical new technology coming around.
而如果做10代、每代十选一，你得到的结果将远远超出人类历史上出现过的水平。你会得到人类历史上从未存在过的那种表型，[听不清]之类的东西。所以你可以看到，单次从一批胚胎里挑选很快就会遇到收益递减，但通过这种迭代式筛选，你在很大程度上避免了这个问题。好，这部分我就跳过去了。是的，所以这看起来是可行的，不需要什么神奇的新技术出现。
35:31
And then that also, I think, adds to the possibility that we will eventually get AI stuff. Like that would be towards the end of this century, if we haven't already solved the problem by then, with this significantly more capable generation of humans working on it. But ultimately, we will be surpassed by intelligent machines, assuming we haven't succumbed to existential catastrophe prior to that, just because the fundamental image-to-information processing in the machine substrate or just far beyond those in biology, like in terms of speed.
而且我认为，这也增加了我们最终能搞定AI相关问题的可能性。比如说到本世纪末的时候，如果那时我们还没解决这个问题的话，会有这样一代能力显著更强的人类来研究它。但最终我们还是会被智能机器超越——前提是在那之前我们没有毁于生存性灾难——这仅仅是因为机器基质中的基本信息处理能力远远超过生物体中的，比如在速度方面。
36:04
Even transistors today are far faster than neurons. So I'm going to-- this is not relevant, for you guys already know all of this. So there is, like, progress in AI, like-- I'm just saying the public consciousness is shaped by a few big milestones, but there's a lot of progress under the hood. Also hardware has driven a lot of progress we've seen. Here is a slice that could have been earlier. This is like with the brain emulation. This is basically the state of the art today. Here's a brain slice scanned with an electron micrograph.
即使是今天的晶体管，也比神经元快得多。所以我打算——这部分不太相关，因为你们已经都知道了。所以AI是有进展的，比如——我只是想说，公众的认知是由几个重大里程碑塑造的，但底层其实有大量的进展在推进。另外硬件也推动了我们看到的很多进展。这里有一张本该放在前面的幻灯片。这是关于大脑仿真的。这基本上就是今天的技术前沿水平。这是用电子显微镜扫描的一张脑切片。
10专家调查：人类级AI到来时间与智能爆炸
36:34
Here is a stack of those pictures on top of one another. And here is the result of applying an image-recognition algorithm to extract the connectivity matrix. But although we have the right resolution-- you can see individual atoms, if you want to, in the brain. It's just that to image the brain with that level of resolution would take, like, forever, so that we're presumably at least decades away from making something like that work. Lot of application there [INAUDIBLE]. So the question of how far away we are from human-level machine intelligence, I think the short answer is that nobody knows.
这是把那些图片一张张叠起来的图像栈。而这是应用图像识别算法提取出连接矩阵的结果。不过虽然我们已经有了足够的分辨率——如果你想的话，在大脑里甚至能看到单个原子。问题只是，要以那种分辨率给整个大脑成像，得花上差不多是永远那么久的时间，所以我们大概至少还要几十年才能让类似的东西真正跑起来。这方面有很多应用[听不清]。那么关于我们离人类水平的机器智能还有多远这个问题，我想简短的答案是：没人知道。
37:10
We did a survey of leading AI experts last year, and one of the questions we asked was, by what year do you think there is a 50% chance that we will have human-level machine intelligence? Here defined as one that could do most jobs that humans could do. And so the median answer to that we got was 2050 or 2040, depending exactly which group of experts we asked. That seems, to me, roughly reasonable, for what it's worth. We also asked, by what year do you think there's a 90% probability? And we got 2070 or 2075.
我们去年对顶尖AI专家做过一次调查，其中一个问题是：你认为到哪一年，我们有50%的概率会拥有人类水平的机器智能？这里定义为能够胜任人类所能做的大多数工作的机器智能。我们得到的答案中位数是2050年或2040年，具体取决于我们问的是哪一组专家。就我个人看法而言，这大致是合理的。我们还问了：你认为到哪一年会有90%的概率？得到的答案是2070年或2075年。
37:43
That, to me, seems overconfident. There is just a lot more than 10% probability, I think, that we will still have failed by then. I should say, as a footnote, that these estimates were conditioned on no global collapse occurring. So-- so maybe the numbers would be-- like the years would be slightly higher up if we hadn't made that assumption. We also asked, if and when we do reach human-level machine intelligence, how long do you think it will take from there to go to some radical superintelligence?
这在我看来就过于自信了。我认为到那时我们仍然没能做到的概率，远不止10%。我得补充一点作为脚注：这些估计是以不发生全球性崩溃为前提的。所以——所以如果我们不做那个假设，这些数字——这些年份可能会稍微往后推一些。我们还问了：如果我们真的达到了人类水平的机器智能，你认为从那里到某种彻底的超级智能需要多久？
38:19
And you can see for yourself the answer there. Now here, again, my view disagrees with those of people we sampled. I think-- I'm quite agnostic as to how far away we are from AI. I think we should basically have a very smeared-out probability distribution. I do think there is a fairly large probability, though, that if and when we get to human-ish level, we will soon after have superintelligence. I place a fairly high credence on there being, at some point, an intelligence explosion. So we need to sharply separate the two questions, like the distance between now and human-level, and the distance in time between that and radical superintelligence.
答案你们可以自己看。而在这一点上，我的看法又和我们调查的那些人不一样了。我认为——对于我们离AI还有多远，我是相当不可知论的。我认为我们基本上应该持有一个非常弥散的概率分布。不过我确实认为，有相当大的概率是：一旦我们达到了大致人类的水平，很快之后我们就会拥有超级智能。我对某个时刻会发生智能爆炸这件事，给予相当高的置信度。所以我们需要把这两个问题严格区分开：从现在到人类水平之间的距离，以及从那里到彻底的超级智能之间的时间距离。
11快速起飞与缓慢起飞：单极体与数字心智生态
38:56
I think this transition might well be very rapid. And things depend on that. So if you distinguish, like qualitatively, like fast-take of scenarios, where we go from something human-ish level to superintelligence within minutes or hours or days, a couple of weeks, in that kind of scenario, it happens too fast for us to really be able to do anything much about it while it is happening. If we get a desirable outcome, it's because we set up the initial conditions just right. By contrast, if one contemplates very slow take-up-- so you have some human-level system, and then only by laboriously adding additional little incremental capability after capability, so it takes like decades or centuries to work your way up to superintelligence, then that would be a lot of more time for new human institutions to arise to deal with this, like to develop a new profession of experts to deal with this, to try things out, see what works, and then change it up.
我认为这个转变很可能会非常迅速。而很多事情都取决于这一点。所以如果你在性质上做个区分：快速起飞的情形，也就是我们从大致人类水平在几分钟、几小时、几天或者一两周内就走到超级智能，在那种情形下，事情发生得太快，以至于在它发生的过程中我们其实做不了什么。如果我们得到一个理想的结果，那是因为我们把初始条件设定得恰到好处。相比之下，如果设想非常缓慢的起飞——你有一个人类水平的系统，然后只能费力地一点一点增加一项又一项的增量能力，这样要花几十年甚至几个世纪才能一路提升到超级智能，那样的话，就会有多得多的时间让新的人类制度出现来应对这件事，比如发展出一个专门处理这类问题的新职业，去做各种尝试，看看什么管用，然后再做调整。
40:00
So it makes a difference. Another way in which it makes a difference is that in the fast takeoff scenarios, it's likely that you will have a singleton outcome, I think. Which is basically a world order where at the highest level of decision-making, there's like one decision-making agency. If you think about competing technology projects, whether it's nations racing to build satellites, or nuclear weapons, or competing tech products, often there's some competition, and you're trying to get there first.
所以这是有区别的。另一个体现区别的地方在于，在快速起飞的情形下，我认为很可能会出现一个「单极体」结局。基本上就是这样一种世界秩序：在最高决策层面上，只存在一个决策机构。如果你想想相互竞争的技术项目，无论是各国竞相发射卫星、研制核武器，还是相互竞争的科技产品，通常都会有某种竞争，大家都想抢先做到。
40:26
But it's rare that the difference between the leader and the closest follower is a couple of days. Like usually the leader will be a few months ahead of the follower, or a couple of years. So if the takeoff is going to be over in a few days or a few weeks, then one project will have completed a takeoff before the next one will have started it, very likely. And then you will have a mature superintelligence in a world which contains no other even vaguely comparable system. And for reasons that I'll be happy to elaborate on in the Q&A, and that a lot of the book is about, as well, this first system then is likely to be very powerful, maybe to the point where it is able to shape the entire future according to its preferences.
但领先者和最接近的追赶者之间只差几天，这种情况是很罕见的。通常领先者会比追赶者领先几个月，或者几年。所以如果起飞过程在几天或几周内就结束了，那么很可能第一个项目已经完成了起飞，下一个项目才刚要开始。于是你就会得到一个成熟的超级智能，而这个世界上不存在任何其他哪怕勉强可比的系统。出于我很乐意在问答环节里展开的一些理由——书里也有很大篇幅在讲这个——这第一个系统很可能会非常强大，强大到也许能够按照自己的偏好塑造整个未来。按照它自己的偏好。
41:08
If you have a storied takeoff, then it's more likely you're going to have multiple outcomes. No system is so far ahead of all the others that it can just lay down the law. They end up superintelligent, but it will have economic competitive forces and evolutionary dynamics working on this population of digital minds shaping the outcome. And the concerns in that type of scenario look very different from the ones in the singleton scenarios. Not necessarily less serious, but different. So instead of having one agency that can dictate the future, you now have this ecology of digital minds.
如果起飞是渐进的，那就更有可能出现多种结局。没有哪个系统会遥遥领先到可以直接说了算的地步。它们最终都会达到超级智能，但会有经济竞争力量和演化动力作用在这群数字心智上，塑造最终的结果。而那类情形下的担忧，看起来和单极体情形下的担忧非常不同。不一定更轻，但确实不同。所以不是有一个能够决定未来的机构，而是有了这样一个数字心智的生态。
41:51
And you can think-- I mean, suppose to take a model-- so once we had human-level minds that were digital, like they could do exactly the same as humans do, and run at the same speed, initially-- suppose that you get there through whole-brain emulation, and this is the first type of AI you have. Then you could very quickly have a population explosion. So we know how to copy software. That takes a couple of minutes. And so as long as the productivity of these digital mind is higher than the cost of making another copy, there would be vast incentives to just keep making more copies until the income that digital minds can earn equals, like, the price of electricity and hardware rental.
你可以设想——我是说，假设我们做个模型——一旦我们有了数字化的人类水平心智，它们能做的和人类完全一样，运行速度一开始也一样——假设你是通过全脑仿真达到这一步的，而这是你拥有的第一种人工智能。那你可能很快就会迎来一场人口爆炸。因为我们知道怎么复制软件。那只要几分钟。所以只要这些数字心智的生产力高于再造一份拷贝的成本，就会有巨大的激励去不断制造更多拷贝，直到数字心智能挣到的收入等于电费加上硬件租用的价格。
42:35
So you have a Malthusian situation where the population of digital minds expands until the wage falls to subsistence level. But subsistence level for the digital minds, rather than for biological minds. So we are a lot more expensive, because we have to eat and have houses to live in and stuff like that. So humans might still be able to make some income through their capital investment. And there's then the question of whether in this world, which is increasingly shaped by the digital minds-- there are trillions and trillions of them, and they're getting faster all the time, and better, and humans constitute a small slice of all of this-- whether we would be able, in the long run, to really enforce property rights and our sociopolitical structures.
于是你就有了一种马尔萨斯式的局面：数字心智的人口不断膨胀，直到工资降到维生水平。但那是数字心智的维生水平，而不是生物心智的维生水平。我们要贵得多，因为我们得吃饭、得有房子住等等。所以人类也许还能靠自己的资本投资挣到一些收入。接下来的问题就是，在这个越来越由数字心智塑造的世界里——它们有数以万亿计，而且一直在变快、变得更好，人类只占其中很小的一部分——从长远看我们是否还能够真正维护产权和我们的社会政治结构。
43:22
Or whether these digital minds would eventually just swamp us and expropriate us. And at some point, presumably, even in this whole-brain emulation, at some point probably fairly soon after that point, you will have synthetic AIs that are more optimized than whatever sort of structures biology came up with, that will then kind of leave the [INAUDIBLE]. So there is a chapter in the book about that. But the bulk of the book is-- so all the stuff that I talked about, like how far we are from it and stuff like that, there's like one chapter about that in the beginning.
还是说这些数字心智最终会直接把我们淹没、把我们的财产剥夺掉。而且在某个时点，即便是在这种全脑仿真的情形下，大概在那之后不久，你就会有合成的人工智能，它们比生物演化出来的那些结构更加优化，然后就会把[听不清]甩在后面。书里有一章讲这个。但这本书的主体是——我前面讲的那些内容，比如我们离它还有多远之类的，开头大概只有一章在讲这些。
43:52
Maybe the second chapter has something about different pathways. But the bulk of the book is really about the question of, if and when we do reach the ability to create human-level machine intelligence-- so machines that are as good as we are in computer science, so they can start to improve themselves-- what happens then? And what happens when you have a superintelligence that might be extremely powerful? What are the control methods that we could try to apply to achieve a controlled detonation if there is going to be an intelligence explosion?
第二章可能会讲一点不同的实现路径。但这本书的主体其实是在讨论这个问题：如果、以及当我们真的具备了创造人类水平机器智能的能力——也就是在计算机科学上和我们一样好的机器，因而它们可以开始改进自身——那之后会发生什么？当你拥有一个可能极其强大的超级智能时，会发生什么？如果真的要发生智能爆炸，我们可以尝试用哪些控制方法来实现一次「受控引爆」？
44:21
How could we set up the initial conditions to get some kind of beneficial outcome? And there are a lot of initially plausible ways to solve this problem that turn out, on closer reflection not to worry. That this kind of one of the types of progress at have occurred in this field. It's like a deepening appreciation of just how profoundly difficult this problem is, of how you could create something vastly smarter than you and still ensure a desirable outcome. So that's the bulk of the book. And then the last two chapters are trying to think more generally about these macrostrategic questions, and how to think about what our levers of influence are, if one wants to increase the probability of a desirable outcome.
我们该如何设定初始条件，才能得到某种有益的结果？有很多乍看之下貌似可行的解决办法，仔细一想却站不住脚。这也算是这个领域里发生的一类进展。就是越来越深刻地体会到，这个问题到底有多么困难：你怎么可能造出一个远比你聪明的东西，同时还能确保结果是理想的。这就是本书的主体部分。然后最后两章试图更概括地思考这些宏观战略问题，以及如果一个人想提高得到理想结果的概率，该如何看待我们手上有哪些影响力的杠杆。
12问答：可穿戴增强与人机融合之争
45:02
So I'll put on the pause there, because I want to make sure we get a little bit of discussion in. Thanks. [APPLAUSE] MODERATOR: Thank you, Nick. We will use the microphone for questions, please. AUDIENCE: I'll just comment quickly. That 40-year median for when we'll achieve human intelligence is-- I've been tracking that. It was about 300 to 400 years in 1999. It was maybe 50 years in 2006. We took a poll at this conference at Dartmouth. And now it's 40 years. I'm saying 2029, but it's actually not so far off.
我就先讲到这里，因为我想确保我们能留出一点讨论时间。谢谢。[掌声]主持人：谢谢你，Nick。提问请使用麦克风。观众：我先简短评论一下。关于我们何时能实现人类水平智能的那个40年中位数——我一直在追踪这个数字。1999年时它大概是300到400年。2006年时大概是50年。我们在达特茅斯的这次会议上做过一次投票。现在则是40年。我说的是2029年，但其实相差并没有那么远。
45:44
I don't think we're going to get that far with enhancing biological intelligence, because our biological circuits are just inherently a million times slower than electronics, and so there's only so far you can get that way. Whole brain emulation is useful not to create an AI, but to be able to emulate a brain, or more likely a portion of a brain, to establish the functional description of what these basic circuits do to guide our creation of AI. My view, though, is that we are emerging with this technology.
我不认为在增强生物智能这条路上我们能走多远，因为我们的生物电路本质上就比电子电路慢一百万倍，所以这条路能走到的极限是有限的。全脑仿真的用处不在于创造一个AI，而在于能够仿真一个大脑，或者更可能是大脑的一部分，以此建立这些基本电路功能的描述，来指导我们创造AI。不过我的看法是，我们正在与这项技术融合。
46:16
I mean, it's already-- during that one-day SOPA strike, I felt like part of my brain went on strike. And so we're already enhanced by these devices. When I was in college, I'd take my bicycle to the computer, and now I carry it on my belt. I believe we will-- these devices are getting smaller. I think within, say, the '20s, '30s, they'll go inside our bloodstream and go into our brain. Basically put our neocortex on the cloud so we can extend the 300 million modules we have in the neocortex. In the cloud, there will be a hybrid.
我是说，这已经在发生了——在SOPA抗议那一天的罢工中，我感觉自己大脑的一部分也罢工了。所以我们已经被这些设备增强了。我上大学的时候，得骑自行车去用计算机，而现在我把它挂在腰带上。我相信我们会——这些设备正变得越来越小。我认为大概在20年代、30年代，它们就会进入我们的血液，进入我们的大脑。基本上就是把我们的新皮质放到云端，这样我们就能扩展新皮质中那三亿个模块。在云端，它将会是一种混合体。
46:51
But I would agree that ultimately, the non-biological portion will be so powerful that it will dominate, but that's a path to getting to superintelligence. But I would argue that the non-biological portion is human intelligence. I don't think it's non-human just because it's non-biological. NICK BOSTROM: Yes, so whether-- I mean-- I guess one doesn't want to be bogged down in the terminology of whether, like-- it seems clear to me to call it non-human. But the idea that it's implemented in machine substrate to me doesn't begin to answer the question of whether the outcome is desirable or not.
但我同意，最终非生物的部分会强大到占据主导地位，不过这正是通向超级智能的路径。但我想说的是，那个非生物的部分依然是人类智能。我不认为仅仅因为它是非生物的，它就是非人类的。NICK BOSTROM：是的，至于是不是——我是说——我想我们不必纠缠在术语上，不过在我看来，把它称为非人类似乎是清楚的。但它是在机器基质上实现的这一点，在我看来完全无法回答这个结果是否可取的问题。
47:30
To me, it would all depend on exactly what kind of intelligence is there in this machine substrate, and what this is it doing? What is it using its resources for? Like I could-- you could imagine that we're discovering that we are all in a simulation, we're already all digital. Like so what? I mean, that doesn't mean that human life doesn't have any moral significance just because we're not biological, as we thought. So in principle, you could have a digital mind with exactly the same experience and capabilities as we do, and presumably it should count for the same morally.
对我来说，这完全取决于这个机器基质里究竟存在什么样的智能，以及它在做什么？它把资源用在了什么地方？比如我可以——你可以想象我们发现自己身处一个模拟之中，我们其实早就是数字化的了。那又怎样呢？我是说，这并不意味着人类生命就没有任何道德意义，仅仅因为我们不是生物性的，不像我们原以为的那样。所以原则上，你可以拥有一个数字心智，它拥有与我们完全相同的体验和能力，那么想必在道德上它也应该被同等看待。
48:01
However, there are a lot of really bizarre types of minds that are possible in principle, and I think one of the slides further down, and one of the key questions that the book tries to answer, is how can we think about the motivations of superintelligent agents? Is it possible to say something useful about what they would want to do? AUDIENCE: We're all evolving together. There's, like, 2 billion people that are enhanced with these devices now. And as they get more intimate with us, it's not going to be like these science futures of movies, of one evil corporation that's got got this technology.
然而，原则上还存在很多非常离奇的心智类型，我想在后面的某一张幻灯片里，也是这本书试图回答的关键问题之一，就是我们该如何思考超级智能体的动机？超级智能体的动机？有没有可能对它们想做什么说出些有用的东西？观众：我们都在一起演化。现在大概有 20 亿人都在用这些设备增强自己。随着它们跟我们越来越亲密，情况不会像那些科幻电影里演的那样——某一家邪恶的公司掌握了这项技术。
48:39
It's going to be billions of us that enhance together, like it is today. NICK BOSTROM: Yeah, so the growth of collective intelligence. I mean, I think that at some point, the fleshy parts that are in crania will-- a., they will be a lot harder to enhance, and they will become just kind of negligible part of the actual intelligence that is created. And that everything then depends upon us having set up the initial conditions. So, like, superintelligence will be extremely powerful. We have the one advantage, that we get to make the first move.
而是我们数十亿人一起被增强，就像今天这样。尼克·博斯特罗姆：对，也就是集体智能的增长。我的意思是，我觉得到了某个时候，颅骨里那些血肉的部分——呃，它们会变得难以增强得多，最后在实际产生的智能里只占微不足道的一小部分。而一切都取决于我们有没有设定好初始条件。所以说，超级智能会极其强大。我们只有一个优势，就是我们能先出手。
13效用怪物、回形针最大化与价值选择难题
49:14
And I think we only get one try there. Because once you have like an unfriendly superintelligence, it will resist you sort of changing its values. And so part of what makes the problem so challenging is that you need to get it right on the first attempt, and humans are generally not very good at that. We like to sort of see how things work out and patch things up and learn from experience. AUDIENCE: I want to explore what you mean when you say a desirable outcome, what desirable means. There this old philosophical problem of the utility monster.
而且我认为我们只有一次机会。因为一旦你造出了不友好的超级智能，它就会抗拒你去改变它的价值观。所以这个问题之所以这么难，一部分原因就在于你必须一次就做对，而人类通常并不擅长这一点。我们习惯的是先看看情况怎么发展，再打补丁，从经验中学习。观众：我想探讨一下，你说的“理想的结果”是什么意思，“理想”指的是什么。有一个古老的哲学难题，叫做“功利怪物”。
49:45
It's sort of a challenge to a utilitarian notion of morality, which is, imagine that there's some creature that wants something more than the rest of humanity combined, feeding the one thing that it wants because it wants it so much more. Maximizes utility, ignoring the rest of humanity. So in some sense, the superintelligence scenario can give life to the utility monster in the sense that if the cognition after the explosion is vastly greater than the total sum of cognition of humanity, then perhaps the moral consideration of what a desirable outcome should be should only be paying attention to what it wants, not what we want.
它算是对功利主义道德观的一种挑战，设想有这么一个生物，它对某样东西的渴望超过了全人类欲望的总和，于是就该满足它想要的那一样东西，因为它想要的程度实在太强烈了。这样能让效用最大化，却完全忽略了人类的其余部分。所以从某种意义上说，超级智能的情景可能让功利怪物成为现实——如果智能爆发之后的认知能力远远超过全人类认知能力的总和，那么在考虑什么才算理想结果时，道德上或许就只该关注它想要什么，而不是我们想要什么。
50:32
NICK BOSTROM: Right. AUDIENCE: So I wanted to raise that as a challenge. I'm not advocating that perspective, I want to see how you reason about desirability in a world where we're coexisting with superintelligence. NICK BOSTROM: Yeah, generally speaking, it's easier to describe what an undesirable outcome would be than a desirable one. So there are a lot of ways in which things could turn out that, by most reasonable [INAUDIBLE], we would regard as pretty worthless. Like the standard example in this little literature is the paper clip maximizer.
尼克·波斯特罗姆：没错。观众：所以我想把这个作为一个挑战提出来。我并不是在支持这种观点，我是想看看，在一个我们与超级智能共存的世界里，你会怎么去论证“理想性”。尼克·波斯特罗姆：是的，总的来说，描述什么是不理想的结果，要比描述什么是理想的结果容易得多。事情可能有很多种走向，按照大多数合理的[听不清]标准，我们都会认为那些结果毫无价值。比如这个小领域里的标准例子，就是“回形针最大化器”。
51:05
So an AI that's superintelligent, and has as its only final, highest-level goal to maximize the number of paper clips it produces. This is a stand-in for some other arbitrary goal. But most final goals, if you think through how the world would be structured in order to maximize the realization of that goal, would involve, as a side effect, the elimination of human beings and everything we care about. So if you're a superintelligence that's a singleton, and you want to make sure there's as many paper clips as possible, for a start, you'd want to get rid of all humans.
一个超级智能的AI，它唯一的、最终的最高目标就是最大化自己生产的回形针数量。这只是随便某个目标的一个替代说法。但大多数最终目标，如果你去推演为了最大程度实现这个目标、世界会被组织成什么样子，都会作为副作用，导致人类以及我们所珍视的一切被消灭。所以如果你是一个独霸一切的超级智能，你想确保回形针越多越好，那么首先，你会想把所有人类除掉。
51:36
Because maybe we'll want to switch off or something like that, and then there'll be fewer paperclips. We also have bodies that are full of juicy atoms that could be used to make some really nice paper clips. So then you think, OK, that's not do paper clips. That's ridiculous. But then you think of something else. Like what about an AI who only wants to calculate decimal expansion of pi? So similarly, such an AI would want to maximize the amount of hardware it has so it can make more rapid progress in this calculation.
因为我们说不定会想把它关掉之类的，那样回形针就会变少。而且我们的身体里全是多汁的原子，可以用来造出一些相当不错的回形针。于是你会想，好吧，那就别搞回形针了。这也太荒唐了。但接着你又想到别的目标。比如，一个只想计算圆周率小数展开的AI呢？同样地，这样的AI会想尽可能扩大自己的硬件规模，好让这项计算推进得更快。
52:02
And it actually turns out to be quite difficult to specify a goal that would not be maximally realized in a world where not just human biological organisms are extinct, but also anything we would possibly place value on is eradicated. AUDIENCE: So the premise there is that-- I want to really focus on the premise, because I think the argument hinges on it-- that we're taking a snapshot of what it is we value today, where "we" includes the things that we consider to be adequately cognitive today. And we are ignoring in our definition of desirability-- Let's go to the extreme of the paper clip scenario.
而事实证明，要设定一个目标，使它不会在这样一个世界里被最大程度实现——在那个世界里，不只是人类这种生物有机体灭绝了，连任何我们可能赋予价值的东西也都被清除了——其实相当困难。观众：所以这里的前提是——我想着重讨论这个前提，因为我觉得整个论证都取决于它——我们是在给今天我们所珍视的东西拍了一张快照，这里的“我们”包括了今天我们认为具备足够认知能力的那些存在。而在我们对“理想性”的定义里，我们忽略了——我们不妨把回形针的情景推到极端。
52:47
A utilitarian might say, well, OK, if it wants paper clips, but its overall cognition is vastly greater than the rest of humanity as a whole, well, then that's what it wants, so the weighted definition of desirability should be to maximize paper clips, because that's what it wants. NICK BOSTROM: Well, OK, so there are different versions of utilitarianism. There is preference satisfactionism, which I think is what you alluded to, which would stipulate some sort of social welfare function that is maximized by fully-satisfied single preferences that exist.
一个功利主义者可能会说，好吧，如果它想要回形针，而它整体的认知能力远远超过全人类的总和，那么，它想要的就是这个，所以按加权计算的“理想性”定义，就应该是最大化回形针的数量，因为那正是它想要的。尼克·波斯特罗姆：嗯，好，功利主义其实有不同的版本。有一种是偏好满足主义，我想这就是你所指的那一种，它会设定某种社会福利函数，当现存的各个个体偏好都被充分满足时，这个函数达到最大值。
53:23
There's a big problem of how to aggregate them, but something along those lines. Other utilitarians would say maximize pleasure or maximize happiness or maximize some other part, the common feature being that the value of the whole is, as it were, the sum of the value of the parts. If you thought preference satisfaction is and was correct, you might want to design agents with easy-to-satisfy preferences. Like they want there to be at least three prime numbers or something like that, and then you're done.
如何把这些偏好加总起来是个大问题，但大致就是这类思路。另一些功利主义者会说，要最大化快乐，或者最大化幸福，或者最大化其他某种东西，其共同特点在于，整体的价值，可以说就是各个部分价值的总和。如果你认为偏好满足主义是正确的，那你可能会想去设计一些偏好极易被满足的智能体。比如它们希望至少存在三个质数之类的，那这事儿就成了。
53:55
And then maybe to have as many as possible of those agents. Like the minimum agent that would count as a morally considerable being. But that seems like a fairly impossible moral view. But one can decompose this big sort of problem into two parts. On the one hand, you have the technical problem of-- if you specified some value in human language, like whether it's to maximize happiness or freedom or love or creativity, whatever it is, that how could you sort of embed that into a seed AI, like an AI that's destined eventually to become a superintelligence?
然后再尽可能多地造出这样的智能体。也就是刚好能算作具有道德地位的最低限度的智能体。但这在道德观上似乎相当站不住脚。不过，人们可以把这个大问题拆解成两个部分。一方面，是技术性的问题——如果你用人类语言表述了某个价值，不管是最大化幸福、自由、爱，还是创造力，随便什么，那你要怎么才能把它嵌入到一个种子AI里，也就是一个注定最终会成长为超级智能的AI？
54:27
So this is like an enormous technical problem. Because like in C++, you don't have a primitive saying "happiness," right? You have to define all of these terms. Some goals would be feasible, like maybe to calculate as many digits of pi. It's something we could do today. Others, like, there's this big, unsolved technical problem. But then on top of that, you also have the second problem, which is the value selection problem, like trying to figure out which value it is that you would want to get in there in the first place.
所以这是个极其庞大的技术难题。因为比如在C++里，你可没有一个叫“幸福”的基本类型，对吧？这些词你都得自己去定义。有些目标是可行的，比如计算圆周率的位数，越多越好。这是我们今天就能做到的。而另一些目标，就属于这个尚未解决的重大技术难题。但除此之外，你还有第二个问题，也就是价值选择问题，也就是要弄清楚，你一开始究竟想把哪个价值放进去。
54:53
And both of these are places where we could easily stumble. So just to reflect on, like-- if the idea was to try to do some AI that was ethical, or maximally always did the morally right thing to do, if we try to achieve that by just creating a list or somehow embedding our current best understanding of ethics into a final goal, we should reflect that if any earlier age had done this with their values, it would have been what we can now see are catastrophes. Earlier ages were condoning slavery or human sacrifice, and all kinds of abuses of different minorities and stuff.
而这两个环节，我们都很容易栽跟头。所以回顾一下，如果我们的想法是做出某种符合伦理的 AI，或者说总是最大程度去做道德上正确之事的 AI，那么如果我们试图通过列一个清单、或者以某种方式把我们当下对伦理的最佳理解嵌入进去来作为它的最终目标，我们就该反思一下：如果任何一个更早的时代用他们的价值观这么做，结果会是我们今天所看到的种种灾难。更早的时代曾容忍奴隶制、活人献祭，以及对各种少数群体的种种虐待。
55:33
And presumably even though we might have made some progress towards moral enlightenment, we haven't gotten all the way there. So it would be important to preserve the possibility for moral growth in the value selection. And so there are a number of different paths that each should be explored, because we're still at such an early stage here. But maybe one of the more promising one is this idea of indirect normativity that I describe in the book, which is the idea that rather than trying to take explicitly characterized some desired end state, we try to motivate the AI to pursue some process whereby it can find out what it is that we were trying to work out when we were working with this problem.
而且可以推想，即便我们在道德启蒙上取得了一些进步，我们也还远没有走到终点。所以在价值选择中保留道德继续成长的可能性，是非常重要的。因此有若干条不同的路径都值得探索，因为我们现在还处在非常早期的阶段。但也许其中比较有前景的一条，是我在书里描述的「间接规范性」这个想法，它的意思是：与其试图明确刻画出某个我们想要的最终状态，不如去激励 AI 去追求某个过程，让它能够弄清楚：当我们在思考这个问题时，我们究竟想要得出什么。当我们在处理这个问题的时候。
56:15
So suppose you could give the AI the goal of doing that, which we would have asked it to do if we had had, like, 40,000 years to think about this question, and if we ourselves had been smarter, and if we had known more facts. So now we don't know what that is, currently. But it's an empirical question that we could then hopefully leverage the AI's superior intelligence to make a better estimate of. And then that kind of indirectly specified goal might then be more likely to produce an outcome that we would recognize, on reflection, as being worthwhile.
所以假设你可以给 AI 这样一个目标：去做那些我们本会要求它去做的事——如果我们曾有比如四万年的时间来思考这个问题，如果我们自己更聪明，如果我们知道更多事实的话。所以我们现在并不知道那是什么。但这是一个经验性问题，我们可以借助 AI 更高的智能来给出更好的估计。于是这种间接指定的目标，也许更有可能带来一个我们经过反思后会认可为值得追求的结果。
14控制问题：能力控制与动机选择方法
56:52
AUDIENCE: So I have a story. The other day, I was reading some of the news and analysis about the crisis in the Middle East, and I guess I spent like an hour thinking about it. And I didn't come up with a solution for the Middle East. NICK BOSTROM: Ah, darn. AUDIENCE: Now if I had been a speedy superintelligence, and in that hour I had spent 1,000 hours of thinking, I think I still wouldn't have come up with a solution. So I think there are some problems for which intelligence by itself isn't the answer.
观众：我有个故事。前几天我在读关于中东危机的一些新闻和分析，我大概花了一个小时思考这件事。结果我并没有想出中东问题的解决方案。尼克·波斯特洛姆：啊，可惜。观众：那么，如果我是一个高速运转的超级智能，在那一小时里我思考了相当于一千小时的量，我觉得我还是想不出解决方案。所以我认为有些问题，光靠智能并不是答案。
57:23
And you know, as humans, we put sapiens in our name. We think intelligence is really important, but it's not the only attribute. I don't think it solves all problems. NICK BOSTROM: Yeah. So I mean, I agree with that. A lot of sort of sociopolitical problems in the human realm often depend on people with conflicting preferences. There might just not success one solution that would maximally please everybody. And with the case of the AI, I mean, I think that in fact, the most important problem to work on is not the intelligence problem, which hastens the day where we'll have it, but rather this control problem.
你知道，我们人类给自己起名叫「智人」（sapiens）。我们认为智能非常重要，但它并不是唯一的属性。我不认为它能解决所有问题。尼克·波斯特洛姆：是的。我同意这一点。人类领域里很多社会政治问题，往往取决于人们之间相互冲突的偏好。可能根本就不存在一个能让所有人都最大程度满意的解决方案。而就 AI 而言，我认为事实上最重要的、需要去攻克的问题，并不是智能问题——那只会加速它到来的日子——而是这个「控制问题」。
57:56
How to ensure that it would deploy its intelligence in ways that are not harmful. And just briefly, there's, I think, two broad classes of control method that's one can envisage here. So one is capability control method, where you try to limit what the AI is able to do. So maybe put it in a box. You unplug the ethernet cable. You only allow it to communicate by typing text on a screen, let's say. Maybe only even answers to questions that are posted. And you try to clip its wings. And I think that those can be important during the development phase, like before you actually are ready to launch your system.
也就是如何确保它会以不造成伤害的方式运用它的智能。简单说一下，我认为这里可以设想两大类控制方法。一类是能力控制法，也就是你试图限制 AI 能做什么。比如把它关进一个「盒子」里。你拔掉网线。你只允许它通过在屏幕上打字来交流，比方说。甚至可能只让它回答被提出的问题。你试图剪掉它的翅膀。我认为这些方法在开发阶段可能很重要，也就是在你真正准备好上线系统之前。
58:33
But ultimately, I don't think they are the answer. Because in order for this AI to actually have any effect on the world, it will at some point need to interact with it. Like if you literally just had an isolated box that didn't closely interact with the world, yes, it could be safe, but it would also not do anything at all. But as soon as you have, say, a human being communicating with it, then you have a weak link here. Like humans are not secure systems. And even humans often succeed in manipulating or tricking or deluding other humans to do their-- like scam artists.
但归根结底，我不认为它们是答案。因为要让这个 AI 真正对世界产生任何影响，它总有一刻需要与世界互动。如果你真的只有一个完全隔离、不与世界密切互动的盒子，那没错，它可以是安全的，但它也将什么都做不了。可一旦你有了，比如说，一个人类去和它交流，你这里就有了一个薄弱环节。人类可不是安全的系统。连人类都常常能成功地操纵、欺骗或蒙蔽其他人类去做事——比如那些骗子。
59:06
And so if you had like a superhumanly powerful persuader and manipulator, chances are eventually, it would find a way to talk its way out of the box. Unless it could just hack its way out, like by-- so there are things like, we think, oh, well, we'll just put it in a box. If we don't talk to it, it's safe. Well, maybe there's some unanticipated way that we haven't thought of, like by wiggling its electrons around in its circuitry, maybe it could create electromagnetic waves that could influence a nearby apparatus or something like that.
所以如果你面对的是一个具有超人说服力和操纵力的存在，很可能它最终会找到办法把自己「说」出盒子。除非它干脆黑出去——比如说，我们会想，哦，好吧，我们就把它关进盒子里。如果我们不跟它说话，就安全了。但也许存在某种我们没想到的、出乎意料的方式，比如通过在电路里摆动电子，它也许能产生电磁波，去影响附近的某个设备之类的。
59:33
So then we think, oh, put it in a Faraday cage. But OK, so if we just keep patching up all the flaws that we can find, then we will just patch up all the ones we can find, but there are probably some more ones that we can't think of. And then it will use one of those. So the second class of control method is motivation selection methods, where instead of, or in addition to, trying to limit what the system can do, you try to engineer its motivation system, so that it would not want to cause harm. And that's then where this indirect normativity comes in, as one version of that, and there are many other many other aspects of that.
于是我们又想，哦，那就把它放进法拉第笼里。但问题是，如果我们只是不断去修补我们能发现的所有漏洞，那我们补上的也只是我们能发现的那些，而很可能还有更多是我们想不到的。然后它就会利用其中之一。所以第二类控制方法是动机选择法：与其（或者说，在此之外）试图限制系统能做什么，你转而去设计它的动机系统，让它压根就不想造成伤害。而这正是「间接规范性」的用武之地，它是其中一种版本，此外还有许多其他方面。
15策略性伪装、正交性论题与工具趋同
60:11
And that's, I think, the problem that ultimately we'll need to solve. AUDIENCE: So if you'd use these two mechanisms to control it, still, it comes back to this question on the other side of the equation. Like it somehow turns its fitness function into the will to dominate us, because of its will to survive. But we also have that will to survive, and even though we make mistakes, it seems like the argument of a superintelligence coming to completely dominate us requires a lapse of attention on our part, in our own promotion of our desire to survive, for long enough for it to actually be irretrievable.
我认为这才是我们最终必须解决的问题。观众：那么，如果你用这两种机制来控制它，问题最终还是回到等式另一边的那个疑问上。比如说，它出于求生意志，把自己的适应度函数变成了统治我们的意志。但我们同样有求生的意志，即便我们会犯错，「超级智能会彻底统治我们」这个论证，似乎要求我们在相当长的一段时间里对自身求生欲的维护出现注意力的失误，长到局面已经无法挽回。
60:51
So have you considered that-- it seems like even in all of the horrific things that you've described that could happen if a superintelligence did come to dominate, there would be that take-off duration period where we would presumably wake up and unplug it. NICK BOSTROM: Well, one would imagine, if the developers are somewhat sensible, that they wouldn't actually permit the take-off unless they at least believed that the system was safe. So imagine a scenario where they have maybe falsely deluded themselves that there is no flaw in their system.
所以你是否考虑过——即便是你所描述的那些可怕情形，如果超级智能真的开始统治，中间也会有一个「起飞」的持续期，在那期间我们大概会醒过来，把它的插头拔掉。尼克·波斯特洛姆：嗯，人们会设想，如果开发者还算明智，他们不会真的允许起飞发生，除非他们至少相信这个系统是安全的。所以设想这样一种情形：他们也许错误地自欺欺人，以为自己的系统毫无缺陷。
61:22
Or maybe they're just worried that there's this competitor who's soon going to release another system. So even if they haven't spent enough time on the safety, they still-- But you have to take into account that you're dealing with an intelligent adversary. So even just a human-level mind in this situation could figure out that it has an incentive to pretend to be nice, whether or not it actually is nice. Like when you're weak and, at the mercy of your programmers, who are inspecting you and seeing if you're ready to be released, and if you're an unfriendly AI, you would want to sort of behave cooperatively and pleasingly and all of these things.
又或者他们只是担心某个竞争对手很快就要发布另一个系统。所以即便他们在安全性上投入的时间不够，他们还是……但你必须考虑到，你面对的是一个有智能的对手。所以在这种处境下，哪怕只是一个人类水平的心智，也能想明白：无论自己是否真的友善，它都有动机去假装友善。比如当你还弱小、任由程序员摆布时，他们在检查你、判断你是否可以被放出来，而如果你是一个不友善的 AI，你就会想要表现得配合、讨人喜欢等等。
61:57
Like it can plan ahead to that extent. And only once you are sort of strong enough that it doesn't matter whether anybody tries to stop you, because they can't-- only then would it be safe for you to reveal your true nature. So there is this fundamental flaw in the-- so this is one of those initially plausible ideas that don't seem to work. Like you develop your AI. You keep it in a sandbox, like a secure environment, and you watch it for a while to see that it behaves nicely. And only once you've seen that it's cooperative and nice and friendly there do you let it out.
它是能做到这种程度的提前谋划的。只有当你强大到即便有人试图阻止你也无所谓时——因为他们做不到——那时候你才可以安全地暴露本性。所以这里存在一个根本性缺陷——这是那类初看很有道理、实则行不通的想法之一。比如你开发出你的 AI。把它放在沙盒里，一个安全的环境中，观察它一段时间，看它是否表现良好。只有当你看到它在里面配合、友好、和善之后，才把它放出来。
62:26
And the flaw is that there is this possibility for strategic behavior, that unfriendly AIs could mimic a friendly AI. And you mentioned something about this survival desire. So there is something like that, but it looks different. So we humans have-- we don't really have a clean agent architecture. There's not, like, one final goal for most of us. And there are lot of different drives that rise and fall in strength, depending on the time of day and the environment we're in. But if you have this architecture where there is a clearly-defined final goal, and everything else is pursued only by virtue of being conducive to the attainment of this final goal, then there are a couple of theses that I think help you think about that kind of structure.
而缺陷在于，存在策略性行为的可能：不友善的 AI 可以模仿友善的 AI。你还提到了求生欲这件事。确实存在类似的东西，但它的样子不太一样。我们人类——我们其实并没有一个干净的智能体架构。对我们大多数人来说，并不存在唯一的最终目标。我们有许多不同的驱力，它们的强度此消彼长，取决于一天中的时间和我们所处的环境。但如果你的架构中有一个界定清晰的最终目标，而其他一切之所以被追求，仅仅是因为有助于达成这个最终目标，那么有几条论题，我认为有助于你思考这类结构。
63:16
So on the one hand, you have the orthogonality thesis, as I call it. This is the idea that values and intelligence are orthogonal. You could have virtually any combination of them. Like a really smart system could be really benevolent or really evil or have some bizarre goal, like paper clips, or something human-meaningful. There's no necessary ontological connection. On the other hand, you also have this instrumental convergence thesis, which says that for almost any final goal and almost any environment, there will be certain instrumental values that you will recognize once you're smart enough.
一方面，是我称之为「正交性论题」的东西。它的意思是，价值与智能是正交的。它们几乎可以任意组合。比如一个非常聪明的系统，可能极其仁慈，也可能极其邪恶，或者拥有某个古怪的目标，比如做回形针，或者某个对人类有意义的目标。二者之间不存在必然的本体论联系。另一方面，还有「工具性趋同论题」，它说的是：对于几乎任何最终目标、几乎任何环境，都存在某些工具性价值，是你足够聪明之后就会认识到的。
63:49
For example, the value to prevent your own death. And so if you're a paper clip maximizer, the only reason that you don't want to die, it's not because you sort of value being alive. It's just that you predict that there will be fewer paper clips if you are switched off today. Because if you're still around tomorrow, you will still be working to make more paper clips. And similarly, goal content preservation. You can predict that if somebody changed your goals, then tomorrow, you will no longer be working to make paper clips.
比如说，避免自己死亡这个价值。所以如果你是一个回形针最大化器，你不想死的唯一理由，并不是因为你本身看重活着。而只是因为你预测到：如果你今天被关掉，回形针就会更少。因为如果你明天还在，你就还会继续努力做出更多回形针。类似地，还有目标内容的保全。你可以预测到：如果有人改变了你的目标，那么明天你就不会再致力于制造回形针了。
64:16
Now you will be working to make staplers, and then there will be fewer paper clips. So you, being a paper clip maximizer today, will want to prevent somebody from changing your goals. And there are others, like acquiring more material resources, or enhancing your own intelligence so that you become better able to realize whatever your goal are. And it's that combination between the lack of any necessary connection between final goal and intelligence, and these convergence instrumental reasons to just do things that are inconsistent with human values, that creates the intrinsic danger there.
你会转而去制造订书机，那样回形针就会更少。所以今天作为回形针最大化器的你，会想要阻止别人改变你的目标。还有其他一些，比如获取更多物质资源，或者提升自身的智能，好让你更有能力去实现你的目标，不管那目标是什么。正是这两点的结合——最终目标与智能之间不存在任何必然联系，再加上这些趋同的工具性理由会促使它去做与人类价值观相悖的事情——这才制造了其中固有的危险。
16政策制定者能做什么与研究领域的极度稀缺
64:48
You have to engineer a very particular kind of final goal to-- have a final goal such that if it's actually maximally pursued by a superintelligence, would be consistent with human survival. Maybe something that kind of embeds within it the same values that we have. MODERATOR: So we've been talking a lot about hypothetical stuff. What about some concrete stuff, namely policymakers? So we're talking here about scenarios that are potentially very dangerous and that may scare policymakers, whom we know are technologically not at the level of this audience and may start making decisions which will slow down or impede the progress, or maybe even ban computer science that tries to do AI research because of the fears that crop up in some of that.
你必须设计出一种非常特定的最终目标——要有这样一个最终目标：如果它真的被超级智能最大化地追求，仍然能与人类的存续相容。也许是某种在其内部嵌入了我们所拥有的那套价值观的东西。主持人：我们前面谈了很多假设性的东西。那具体一些的呢，比如说政策制定者？我们在这里讨论的是潜在非常危险的情景，这些可能会吓到政策制定者，而我们知道他们的技术水平并不在座各位这个层次，他们可能会开始做出一些决定，从而拖慢或阻碍进展，甚至可能因为其中某些引发的恐惧而禁止试图做 AI 研究的计算机科学。
65:39
What are your thoughts on the policymaking process and legislature process around issues of artificial intelligence? And can we expect that, you know, like computer scientists are one day labeled as terrorists? NICK BOSTROM: I don't think that that's very likely for various reasons. It's hard at the moment to see exactly what it is-- even if policymakers were willing to do something, what they could actually do that would be helpful, rather than harmful. At the moment, what needs to be done, I think, is more foundational work to build up a clear understanding of what precisely the problem is.
关于围绕人工智能问题的政策制定过程和立法过程，你有什么看法？还有，我们是不是可以预期，比如说，计算机科学家有一天会被贴上恐怖分子的标签？尼克·波斯特洛姆：出于种种原因，我认为那不太可能。目前很难看清究竟是什么——即便政策制定者愿意做点什么，他们实际上能做些什么才是有帮助的，而不是有害的。我认为，目前需要做的是更多基础性的工作，以建立起对问题究竟是什么的清晰理解。
66:17
And then ultimately, it's mostly a technical research challenge to work out the solution to this control problem. It requires some top-notch mathematical talent working together with theoretical computer scientists and maybe some philosophical expertise to really crack this problem. It's very hard to see how, like, from some high level of government-- so it's a very blunt instrument. And you might, even with the best intentions at the start, like at the top, once it filters down through the bureaucracy, it might have a very different effect than the one you intended.
然后最终，要找出这个控制问题的解决方案，主要是一项技术研究挑战。这需要一流的数学人才与理论计算机科学家合作，也许还需要一些哲学方面的专长，才能真正攻克这个问题。很难想象，比如说，从政府的某个高层来做——那是一种非常粗糙的工具。而且，即便一开始、在最高层是抱着最好的意图，一旦它层层向下渗透到官僚体系中，产生的效果可能与你原本设想的大不相同。
66:51
So there are some other existential risks where I think it would be easier to imagine ways in which regulation could help. AI is particularly difficult. Even just to understand what the problem is is quite hard. And it's hard to imagine a scenario, at least in the next couple of decades, where we would have some kind of sane thing coming from political processes. Maybe the closest would be like more funding for work on the control problem. But even that, once it sort of filters through the vested interest and academia, will probably translate into a rain of funding falling on a wide range of superficially related areas that might not actually have anything to do with the control problem, like general computer security or something like that.
所以还有另外一些存在性风险，我认为更容易设想出监管可以起到帮助作用的方式。AI 尤其困难。光是理解问题是什么就已经相当难了。而且很难想象有这样一种情景——至少在未来几十年内——政治过程会产出某种明智的东西。最接近的可能就是为控制问题的研究提供更多经费。但即便是这一点，一旦经过既得利益和学术界的层层过滤，很可能会变成一场撒向各种表面相关领域的经费之雨，而那些领域实际上可能与控制问题毫无关系，比如一般的计算机安全之类。
67:38
But there are other things that can be done. So there are some organizations that are working on this. So we are doing some work at the Future of Humanity Institute at Oxford. Another is the Machine Intelligence Research Institute, MIRI, at Berkeley. They have some excellent people, as well. That would be an obvious thing. And generally try to recruit some of the brightest minds of our generation and the next generation, to sort of focus on this. At the moment, worldwide, maybe there are half a dozen people or so, equivalent, working full-time on this, which is not in proportion to the importance of the problem.
但还有一些别的事情是可以做的。有一些机构正在做这方面的工作。我们在牛津的人类未来研究所（Future of Humanity Institute）就在做一些工作。另一个是位于伯克利的机器智能研究所，MIRI。他们那里也有一些非常出色的人。那会是一件显而易见该做的事。总的来说，就是设法招募我们这一代和下一代中最聪明的一些头脑，来专注于这个问题。目前在全世界范围内，也许只有大约六个人（折合全职）在全职做这件事，这与问题的重要性完全不成比例。
68:18
It's a more general issue. I did a little literature survey a couple of years ago. I just compared a number of academic papers on the dung beetle compared to a number on human extinction. And sad to tell you that there was more than an order of magnitude more on the dung beetle. So the positive spin on that is that there are enormous opportunities for somebody who actually does care to make a big difference. Like even one extra person or one like extra million or something, can do a lot of good there, because it's so neglected.
这是一个更普遍的问题。几年前我做过一个小小的文献调查。我把关于蜣螂的学术论文数量和关于人类灭绝的论文数量做了个比较。很遗憾地告诉各位，关于蜣螂的论文要多出一个数量级以上。所以从积极的角度看，这意味着对于真正在意这件事的人来说，存在着巨大的机会去带来重大改变。比如哪怕多一个人，或者多一百万美元之类，都能在这里做很多好事，因��它太被忽视了。
68:52
AUDIENCE: So regarding policy and political things, I think the general underlying principle here is that modern governments are like big battleships or big tanks. They do very well against large, stationary targets, but against small, mobile targets, they're extremely ineffective. And so if AI were like nuclear weapons, where in order to produce it, you need these giant, static manufacturing facilities that are very expensive and they're like fixed in one place so you can see where it is, then the political aspect, how you regulate it and whether you regulate it, is very important.
观众：关于政策和政治方面的事，我认为这里的一般性基本原理是，现代政府就像大型战舰或者大型坦克。它们对付大型静止目标非常在行，但面对小型的、移动的目标，就极其无效。所以，如果 AI 像核武器那样，要生产它就需要那种巨大的、固定的制造设施，非常昂贵，而且固定在一个地方、你能看到它在哪儿，那么政治层面——你如何监管它、是否监管它——就非常重要。
69:26
But artificial intelligence isn't like that. You can develop it from anywhere in the world. Your computer might cost $10,000, and it might be anywhere in the world, since you can do things through the cloud. And when governments try to handle these sorts of small mobile targets, like individual websites or individual people on the internet, it doesn't really matter, compared to the nuclear weapons case, very much what kinds of things they do. Because governments just can't hit that kind of target.
但人工智能不是那样的。你可以在世界上任何地方开发它。你的电脑可能只值一万美元，而且它可能在世界任何角落，因为你可以通过云来做事。而当政府试图应对这类小型移动目标时，比如单个网站或者互联网上的某个人，相比核武器的情形，他们做什么样的事情其实关系不大。因为政府根本打不中那种目标。
69:55
It's like, you know, piracy of software is, in theory, punishable by whatever penalty. But as we see everywhere in the world, those kinds of things are totally ineffective at achieving their stated goals. NICK BOSTROM: It depends a little bit on what the scenarios here that we're having. Like, say, if there were some scenario in which they would try to prevent AI from ever being developed, I think that's a lot more far-fetched. And slightly more possible scenarios where it became clear which products were going to succeed, and that it was going ahead, and then they would acquire that.
这就像，你知道的，软件盗版理论上是要受到某种处罚的。但正如我们在世界各地看到的，那类做法在实现其宣称的目标方面完全无效。尼克·波斯特洛姆：这多少取决于我们设想的是什么样的情景。比如说，如果存在某种情景，他们试图阻止 AI 被开发出来，我认为那要牵强得多。还有一些稍微更有可能的情景：当哪些产品会成功已经变得清楚，而且它正在推进，然后他们把它收归己有。
70:27
Like they would nationalize it. But then that doesn't solve the problem. That just means that now you have an encapsulation. So maybe it's all placed under the federal government, and they have military guarding the whole thing, but you would still have the same people inside, basically working on the same problem. And so that that outcome, scenario, might not make that much difference one way or the other. You still have the same basic technical problem inside. And it's also unclear to what extent it would be possible for non-experts to really be able to exert micro-level influence on the precise design of the AI.
比如他们会把它国有化。但那并不能解决问题。那只意味着现在你有了一个封装。所以也许这一切被置于联邦政府之下，他们派军队守卫整个地方，但里面还是同样那些人，基本上还在做同一个问题。所以那种结果、那种情景，可能并不会带来多大差别，无论朝哪个方向。内部依然是同样的基本技术问题。而且也不清楚，非专家究竟能在多大程度上对 AI 的精确设计施加微观层面的影响。
71:01
I mean, you have to know what you're doing to be able to do that. I think-- I mean, things that they could do in general, there are indirect things. So working harder to achieve global peace and coordination would help with a lot of problems, including AI. Maybe it makes it easier. And in the future, if there were like a race dynamic between different countries, that they could join together and do one joint thing, rather than racing to get there first and then having to cut back on safety. There are things that could be done, maybe, to facilitate biological cognitive enhancement.
我的意思是，你得懂行才有可能做到那一点。我认为——我是说，他们总体上能做的事情，是一些间接的事情。比如更努力地去实现全球和平与协调，那会对很多问题都有帮助，包括 AI。也许会让事情变得容易些。以及在未来，如果不同国家之间出现某种竞赛态势，他们可以联合起来做成一件共同的事，而不是竞相抢先到达、然后不得不在安全上打折扣。也许还有一些事情可以做，用来促进生物性的认知增强。
17我们能挺过去吗：不到50%的毁灭风险
71:34
If that was the will, you could certainly imagine different kinds of funding and policies for accessing and linking different databases that could be done, and stuff like that, that would be useful. So there are potentially cost-effective, indirect ways of approaching this problem, in addition to directly working on the control problem. There are these other levers that one could also consider. Particularly on things that we are still quite far away from the relevant crunch time. AUDIENCE: Hi, there.
如果有这个意愿，你当然可以设想各种不同的经费投入和政策，用于访问和打通不同的数据库，这些是能做到的，诸如此类，会很有用。所以除了直接研究控制问题之外，还存在一些潜在具有成本效益的、间接的处理途径。还有一些别的杠杆也是可以考虑的。尤其是在那些我们距离相关的关键时刻还相当遥远的事情上。观众：你好。
72:01
I was just curious. You're one of the world's experts in superintelligence and the extensional risks. Personally speaking, informally, intuitively, do you think we're gonna make it? [LAUGHTER] NICK BOSTROM: Uh, yeah. I mean, it's-- I think that the, uh-- [LAUGHTER] NICK BOSTROM: I mean, like, I mean-- yeah, probably like less than 50% risk of doom. But I don't know exactly what the number is. I mean, the more important question, I guess, is what is the best way to push it down. So that's where most of the mental energy is going into.
我只是有点好奇。你是世界上研究超级智能和存在性风险的专家之一。从个人角度、非正式地、凭直觉说，你觉得我们能挺过去吗？（笑声）尼克·波斯特洛姆：呃，是的。我是说，这个——我觉得那个，呃——（笑声）尼克·波斯特洛姆：我是说，就是，我是说——嗯，毁灭的风险大概低于 50% 吧。但我不确切知道这个数字是多少。我是说，我想更重要的问题是，把它压下去的最佳方式是什么。所以我大部分的心力都花在那上面。
72:45
[LAUGHTER] MODERATOR: So with that, please thank our guest today. [APPLAUSE] MODERATOR: Thank you, Nick.
（笑声）主持人：那么就到这里，请大家感谢我们今天的嘉宾。（掌声）主持人：谢谢你，尼克。
这期还没有生成总结。
这期还没有生成核心句型（制作精读 PDF 时会一并生成）。
这期还没有生成词汇表（制作精读 PDF 时会一并生成）。
这期还没有生成自测题（制作精读 PDF 时会一并生成）。
下载便签
手机：长按图片也可保存
← 上一期 · NO.107What can Mary Shelley's classic, Frankenstein, teach us today?
下一期 · NO.109 →17. The badness of death, Part II: The deprivation account
订阅苏菲周报
每周一封：本周入库的精读、一个值得带走的问题、一条苏菲按。免费，随时退订。
免费 · 每周一封 · 一键退订
苏菲拉底
THE SOPHIE LAB · ASK THE BEST MINDS THE BIG QUESTIONS
内容仅供学习 · thesophielab.com