Machine intelligence makes human morals more important

机器智能使人类道德更重要

by Zeynep Tufekci

So, I started my first job as a computer programmer in my very first year of college -- basically, as a teenager.

所以，我在大学一年级时就开始了我的第一份电脑程序员的工作，基本上是一个十几岁的孩子。

Soon after I started working, writing software in a company, a manager who worked at the company came down to where I was, and he whispered to me, "Can he tell if I'm lying?" There was nobody else in the room.

我开始工作后不久，在一家公司写软件，一位在公司工作的经理来到我所在的地方，他低声对我说：“他能告诉我我在撒谎吗？”房间里没有其他人。

"Can who tell if you're lying? And why are we whispering?"

“谁能告诉我你在撒谎？我们为什么要窃窃私语？”

The manager pointed at the computer in the room. "Can he tell if I'm lying?" Well, that manager was having an affair with the receptionist.

经理指着房间里的电脑。“他能告诉我我在撒谎吗？”嗯，那个经理和接待员有暧昧关系。

(Laughter)

（笑声）

And I was still a teenager. So I whisper-shouted back to him, "Yes, the computer can tell if you're lying."

我还是个十几岁的孩子。于是我小声地对他喊道：“是的，电脑能分辨出你在撒谎。”

(Laughter)

（笑声）

Well, I laughed, but actually, the laugh's on me. Nowadays, there are computational systems that can suss out emotional states and even lying from processing human faces. Advertisers and even governments are very interested.

嗯，我笑了，但事实上，我笑了。现在，有一些计算系统可以解决情绪状态，甚至可以从处理人脸上撒谎。广告商甚至政府都很感兴趣。

I had become a computer programmer because I was one of those kids crazy about math and science. But somewhere along the line I'd learned about nuclear weapons, and I'd gotten really concerned with the ethics of science. I was troubled. However, because of family circumstances, I also needed to start working as soon as possible. So I thought to myself, hey, let me pick a technical field where I can get a job easily and where I don't have to deal with any troublesome questions of ethics. So I picked computers.

我已经成为一名电脑程序员，因为我是一个对数学和科学着迷的孩子。但我在某个地方学到了核武器，我真的很关心科学的伦理学。我很烦恼。然而，由于家庭情况，我也需要尽快开始工作。因此，我想，嘿，让我选择一个技术领域，我可以轻松地找到一份工作，在那里我不需要处理任何棘手的道德问题。所以我选择了电脑。

(Laughter)

（笑声）

Well, ha, ha, ha! All the laughs are on me. Nowadays, computer scientists are building platforms that control what a billion people see every day. They're developing cars that could decide who to run over. They're even building machines, weapons, that might kill human beings in war. It's ethics all the way down.

哈，哈，哈！所有的笑声都在我身上。如今，计算机科学家正在构建一个平台，控制着每天有十亿人看到的东西。他们正在开发可以决定谁来跑的汽车。他们甚至制造机器，武器，可能会在战争中杀死人类。这是道德的一路下滑。

Machine intelligence is here. We're now using computation to make all sort of decisions, but also new kinds of decisions. We're asking questions to computation that have no single right answers, that are subjective and open-ended and value-laden.

机器智能在这里。我们现在使用计算来做所有的决定，但也有新的决定。我们问的问题是没有一个正确答案的计算，这是主观的，开放的和价值的。

We're asking questions like, "Who should the company hire?" "Which update from which friend should you be shown?" "Which convict is more likely to reoffend?" "Which news item or movie should be recommended to people?"

我们在问这样的问题：“公司应该雇佣谁？”“你应该从哪个朋友那里得到更新？”“哪一个犯人更有可能重新犯罪？”“应该向人们推荐哪种新闻或电影？”

Look, yes, we've been using computers for a while, but this is different. This is a historical twist, because we cannot anchor computation for such subjective decisions the way we can anchor computation for flying airplanes, building bridges, going to the moon. Are airplanes safer? Did the bridge sway and fall? There, we have agreed-upon, fairly clear benchmarks, and we have laws of nature to guide us. We have no such anchors and benchmarks for decisions in messy human affairs.

看，是的，我们已经使用了一段时间的电脑，但这是不同的。这是一个历史的转折，因为我们不能锚定计算这样的主观决定的方式，我们可以锚定计算的飞行飞机，建造桥梁，去月球。飞机安全吗？这座桥摇晃了吗？在那里，我们已经达成一致，相当明确的基准，我们有自然法则来指导我们。在混乱的人类事务中，我们没有这样的锚定和基准。

To make things more complicated, our software is getting more powerful, but it's also getting less transparent and more complex. Recently, in the past decade, complex algorithms have made great strides. They can recognize human faces. They can decipher handwriting. They can detect credit card fraud and block spam and they can translate between languages. They can detect tumors in medical imaging. They can beat humans in chess and Go.

为了使事情变得更复杂，我们的软件变得越来越强大，但它也变得越来越不透明，越来越复杂。最近，在过去的十年中，复杂的算法取得了很大的进步。他们可以识别人脸。他们能辨认笔迹。他们可以检测信用卡诈骗和阻止垃圾邮件，他们可以翻译之间的语言。他们可以在医学影像中发现肿瘤。他们可以在国际象棋中击败人类。

Much of this progress comes from a method called "machine learning." Machine learning is different than traditional programming, where you give the computer detailed, exact, painstaking instructions. It's more like you take the system and you feed it lots of data, including unstructured data, like the kind we generate in our digital lives. And the system learns by churning through this data. And also, crucially, these systems don't operate under a single-answer logic. They don't produce a simple answer; it's more probabilistic: "This one is probably more like what you're looking for."

这种进步很大程度上来自一种叫做“机器学习”的方法。机器学习不同于传统编程，在那里你给计算机详细、精确、细致的指令。它更像是你采取的系统，你喂它大量的数据，包括非结构化数据，像我们在我们的数字生活中产生的那种。系统通过这些数据来学习。而且，关键的是，这些系统不在一个单一的答案逻辑下运作。他们并没有给出一个简单的答案，而是更多的概率：“这一个可能更像你正在寻找的。”

Now, the upside is: this method is really powerful. The head of Google's AI systems called it, "the unreasonable effectiveness of data." The downside is, we don't really understand what the system learned. In fact, that's its power. This is less like giving instructions to a computer; it's more like training a puppy-machine-creature we don't really understand or control. So this is our problem. It's a problem when this artificial intelligence system gets things wrong. It's also a problem when it gets things right, because we don't even know which is which when it's a subjective problem. We don't know what this thing is thinking.

现在，好处是：这种方法真的很强大。谷歌的人工智能系统的负责人称之为“数据的不合理有效性”。缺点是，我们并不真正理解系统所学到的东西。事实上，这就是它的力量。这不像是给电脑指令，更像是训练 puppy-machine-creature 我们并不真正理解或控制。这就是我们的问题。当这种人工智能系统出错时，这是个问题。这也是一个问题，当它得到正确的东西，因为我们甚至不知道这是什么时候，这是一个主观的问题。我们不知道这是什么想法。

So, consider a hiring algorithm -- a system used to hire people, using machine-learning systems. Such a system would have been trained on previous employees' data and instructed to find and hire people like the existing high performers in the company. Sounds good. I once attended a conference that brought together human resources managers and executives, high-level people, using such systems in hiring. They were super excited. They thought that this would make hiring more objective, less biased, and give women and minorities a better shot against biased human managers.

因此，考虑一种雇佣算法——一种用来雇佣人的系统，使用机器学习系统。这样的系统将被培训在以前的员工的数据，并指示找到和雇用的人，如现有的高绩效的公司。听起来不错。我曾经参加过一个会议，汇集了人力资源经理和高级管理人员，高层人员，在招聘中使用这种系统。他们非常兴奋。他们认为这会使招聘更加客观，减少偏见，给女性和少数群体一个更好的机会来对付有偏见的人类管理者。

And look -- human hiring is biased. I know. I mean, in one of my early jobs as a programmer, my immediate manager would sometimes come down to where I was really early in the morning or really late in the afternoon, and she'd say, "Zeynep, let's go to lunch!" I'd be puzzled by the weird timing. It's 4pm. Lunch? I was broke, so free lunch. I always went. I later realized what was happening. My immediate managers had not confessed to their higher-ups that the programmer they hired for a serious job was a teen girl who wore jeans and sneakers to work. I was doing a good job, I just looked wrong and was the wrong age and gender.

看，人的雇佣是有偏见的。我知道。我的意思是，在我作为一名程序员的早期工作中，我的直属经理有时会到我真正早到的地方，或者是在下午很晚的时候，她会说：“Zeynep，我们去吃午饭吧！”我会被奇怪的时间所迷惑。下午四点。午餐？我破产了，所以免费的午餐。我总是去。后来我意识到发生了什么。我的直属经理们还没有承认他们的上司，他们雇用的一个严肃的工作是一个十几岁的女孩穿着牛仔裤和运动鞋上班。我做得很好，我只是看起来错了，是错误的年龄和性别。

So hiring in a gender- and race-blind way certainly sounds good to me. But with these systems, it is more complicated, and here's why: Currently, computational systems can infer all sorts of things about you from your digital crumbs, even if you have not disclosed those things. They can infer your sexual orientation, your personality traits, your political leanings. They have predictive power with high levels of accuracy. Remember -- for things you haven't even disclosed. This is inference.

因此，在一个性别和种族的盲目的方式雇用听起来对我很好。但是，随着这些系统，它是更复杂的，这就是为什么：目前，计算系统可以推断出各种各样的事情，你从你的数字面包屑，即使你没有透露这些东西。他们可以推断出你的性取向，你的个性特征，你的政治倾向。他们具有高精度的预测能力。记住--对于你还没有透露的事情。这是推理。

I have a friend who developed such computational systems to predict the likelihood of clinical or postpartum depression from social media data. The results are impressive. Her system can predict the likelihood of depression months before the onset of any symptoms -- months before. No symptoms, there's prediction. She hopes it will be used for early intervention. Great! But now put this in the context of hiring.

我有一个朋友开发了这样的计算系统，从社会媒体数据预测临床或产后抑郁症的可能性。结果令人印象深刻。她的系统可以预测几个月前出现任何症状之前的抑郁的可能性。没有症状，有预测她希望这将被用于早期干预。太棒了！但现在把这放在招聘的背景下。

So at this human resources managers conference, I approached a high-level manager in a very large company, and I said to her, "Look, what if, unbeknownst to you, your system is weeding out people with high future likelihood of depression? They're not depressed now, just maybe in the future, more likely. What if it's weeding out women more likely to be pregnant in the next year or two but aren't pregnant now? What if it's hiring aggressive people because that's your workplace culture?" You can't tell this by looking at gender breakdowns. Those may be balanced. And since this is machine learning, not traditional coding, there is no variable there labeled "higher risk of depression," "higher risk of pregnancy," "aggressive guy scale." Not only do you not know what your system is selecting on, you don't even know where to begin to look. It's a black box. It has predictive power, but you don't understand it.

所以在这次人力资源经理会议上，我找了一家非常大的公司的高级经理，我对她说：“看，如果你不知道，你的系统正在淘汰那些未来可能有抑郁症的人呢？他们现在不沮丧，只是可能在未来，更有可能。如果在接下来的一年或两年内淘汰妇女更可能怀孕，但现在又没有怀孕怎么办？如果它雇佣有侵略性的人，因为那是你的工作场所文化？”你不能通过看性别问题来判断。这些可能是平衡的。而且由于这是机器学习，而不是传统的编码，没有可变的标签“高风险的抑郁症，”“更高的风险怀孕，”“侵略性的家伙规模。”不仅你不知道你的系统在选择什么，你甚至不知道从哪里开始看。它是一个黑匣子。它具有预测力，但你不理解它。

"What safeguards," I asked, "do you have to make sure that your black box isn't doing something shady?" She looked at me as if I had just stepped on 10 puppy tails.

“什么保障措施，”我问，“你必须确保你的黑匣子没有做什么可疑的事情？”她看着我，好像我刚刚踩了10条小狗的尾巴。

(Laughter)

（笑声）

She stared at me and she said, "I don't want to hear another word about this." And she turned around and walked away. Mind you -- she wasn't rude. It was clearly: what I don't know isn't my problem, go away, death stare.

她盯着我，她说：“我不想再听到这个消息了。”她转身走开了。注意你--她并不粗鲁。很明显：我不知道的不是我的问题，走开，死亡凝视。

(Laughter)

（笑声）

Look, such a system may even be less biased than human managers in some ways. And it could make monetary sense. But it could also lead to a steady but stealthy shutting out of the job market of people with higher risk of depression. Is this the kind of society we want to build, without even knowing we've done this, because we turned decision-making to machines we don't totally understand?

看，这样的系统在某些方面甚至可能比人类管理者有更少的偏见。这可能会使货币变得有意义。但这也可能导致一个稳定但却悄无声息的退出就业市场的人，有更高的忧郁症风险。这是我们想要建立的社会，甚至不知道我们已经这样做了，因为我们把决策变成了我们不完全理解的机器？

Another problem is this: these systems are often trained on data generated by our actions, human imprints. Well, they could just be reflecting our biases, and these systems could be picking up on our biases and amplifying them and showing them back to us, while we're telling ourselves, "We're just doing objective, neutral computation."

另一个问题是：这些系统通常是由我们的行为、人类印记所产生的数据训练的。嗯，他们可能只是反映了我们的偏见，这些系统可能会发现我们的偏见，放大他们，并让他们回到我们，而我们告诉自己，“我们只是做客观的，中立的计算。”

Researchers found that on Google, women are less likely than men to be shown job ads for high-paying jobs. And searching for African-American names is more likely to bring up ads suggesting criminal history, even when there is none. Such hidden biases and black-box algorithms that researchers uncover sometimes but sometimes we don't know, can have life-altering consequences.

研究人员发现，在谷歌上，女性比男性更不可能在高薪职位上招聘广告。而寻找非裔美国人的名字更可能带来犯罪历史的广告，即使没有。研究人员有时会发现这种隐藏的偏见和黑箱算法，但有时我们不知道，会有改变生活的结果。

In Wisconsin, a defendant was sentenced to six years in prison for evading the police. You may not know this, but algorithms are increasingly used in parole and sentencing decisions. He wanted to know: How is this score calculated? It's a commercial black box. The company refused to have its algorithm be challenged in open court. But ProPublica, an investigative nonprofit, audited that very algorithm with what public data they could find, and found that its outcomes were biased and its predictive power was dismal, barely better than chance, and it was wrongly labeling black defendants as future criminals at twice the rate of white defendants.

在威斯康星州，一名被告因躲避警察被判六年监禁。你可能不知道这一点，但算法越来越多地用于假释和判决决定。他想知道：这是怎么计算的？它是一个商业黑匣子。该公司拒绝在公开法庭上对其算法提出质疑。但是ProPublica，一个调查性的非营利组织，对他们所能发现的公共数据进行了审计，发现其结果是有偏见的，其预测能力是令人沮丧的，几乎没有机会，并且错误地把黑人被告作为未来的犯罪分子，是白人被告的两倍。

So, consider this case: This woman was late picking up her godsister from a school in Broward County, Florida, running down the street with a friend of hers. They spotted an unlocked kid's bike and a scooter on a porch and foolishly jumped on it. As they were speeding off, a woman came out and said, "Hey! That's my kid's bike!" They dropped it, they walked away, but they were arrested.

所以，考虑一下这个案例：这个女人在佛罗里达州布劳沃德县的一所学校接她的教母，她和她的一个朋友在街上跑来跑去。他们发现一个未上锁的孩子的自行车和一辆滑板车在门廊和愚蠢地跳上它。当他们超速行驶时，一个女人跑出来说：“嘿！那是我孩子的自行车！”他们把它扔了，他们走开了，但他们被逮捕了。

She was wrong, she was foolish, but she was also just 18. She had a couple of juvenile misdemeanors. Meanwhile, that man had been arrested for shoplifting in Home Depot -- 85 dollars' worth of stuff, a similar petty crime. But he had two prior armed robbery convictions. But the algorithm scored her as high risk, and not him. Two years later, ProPublica found that she had not reoffended. It was just hard to get a job for her with her record. He, on the other hand, did reoffend and is now serving an eight-year prison term for a later crime. Clearly, we need to audit our black boxes and not have them have this kind of unchecked power.

她错了，她很愚蠢，但她也只有18岁。她有过两次未成年的轻罪。与此同时，那名男子因在家得宝商店行窃被逮捕——价值85美元的东西，类似的轻微犯罪。但他有两次持枪抢劫的前科。但该算法的得分高风险，而不是他。两年后，propublica发现她没有再生气。她很难找到一份有记录的工作。另一方面，他又重新犯罪，现在为以后的罪行服刑8年。显然，我们需要审计我们的黑匣子，而不是让他们拥有这种不受约束的权力。

(Applause)

（掌声）

Audits are great and important, but they don't solve all our problems. Take Facebook's powerful news feed algorithm -- you know, the one that ranks everything and decides what to show you from all the friends and pages you follow. Should you be shown another baby picture?

审计是伟大而重要的，但它们并不能解决我们所有的问题。采取Facebook的强大的新闻feed算法-你知道，一个排名一切，并决定什么显示你从所有的朋友和网页，你跟随。你应该再给我看一张婴儿照片吗？

(Laughter)

（笑声）

A sullen note from an acquaintance? An important but difficult news item? There's no right answer. Facebook optimizes for engagement on the site: likes, shares, comments.

一个熟人的闷闷不乐的便条？一个重要但困难的新闻？没有正确的答案。Facebook优化了对网站的参与：喜欢，分享，评论。

In August of 2014, protests broke out in Ferguson, Missouri, after the killing of an African-American teenager by a white police officer, under murky circumstances. The news of the protests was all over my algorithmically unfiltered Twitter feed, but nowhere on my Facebook. Was it my Facebook friends? I disabled Facebook's algorithm, which is hard because Facebook keeps wanting to make you come under the algorithm's control, and saw that my friends were talking about it. It's just that the algorithm wasn't showing it to me. I researched this and found this was a widespread problem.

2014年8月，在密苏里州的弗格森，在一名白人警官在黑暗的环境下杀害一名非洲裔美国少年后，发生了抗议活动。抗议的消息充斥了我的算法未过滤的Twitter feed，但在我的Facebook上却没有。是我的Facebook朋友吗？我禁用了Facebook的算法，这很难，因为Facebook一直想让你在算法的控制下，看到我的朋友们在谈论它。只是算法没有显示给我看。我研究了这一点，发现这是一个普遍的问题。

The story of Ferguson wasn't algorithm-friendly. It's not "likable." Who's going to click on "like?" It's not even easy to comment on. Without likes and comments, the algorithm was likely showing it to even fewer people, so we didn't get to see this. Instead, that week, Facebook's algorithm highlighted this, which is the ALS Ice Bucket Challenge. Worthy cause; dump ice water, donate to charity, fine. But it was super algorithm-friendly. The machine made this decision for us. A very important but difficult conversation might have been smothered, had Facebook been the only channel.

弗格森的故事不太友好。这不是“可爱”。谁会点击“喜欢”？这甚至不容易评论。没有喜欢和评论，算法可能会显示它更少的人，所以我们没有看到这一点。相反，在那个星期，Facebook的算法突出了这一点，这是ALS冰桶挑战。有价值的事业；倾倒冰水，捐给慈善机构，罚款。但它是超级算法友好。机器为我们做了这个决定。一个非常重要但困难的谈话可能被扼杀了，Facebook是唯一的渠道。

Now, finally, these systems can also be wrong in ways that don't resemble human systems. Do you guys remember Watson, IBM's machine-intelligence system that wiped the floor with human contestants on Jeopardy? It was a great player. But then, for Final Jeopardy, Watson was asked this question: "Its largest airport is named for a World War II hero, its second-largest for a World War II battle."

现在，最后，这些系统也可能是错误的方式，不象人类系统。你们还记得Watson，IBM的机器智能系统吗？这是一个伟大的球员。但是，最后的危险，沃森被问到这个问题：“它最大的机场被命名为一个二战英雄，它的第二次世界大战第二次战斗。”

(Hums Final Jeopardy music)

（人类最后的危险音乐）

Chicago. The two humans got it right. Watson, on the other hand, answered "Toronto" -- for a US city category! The impressive system also made an error that a human would never make, a second-grader wouldn't make.

芝加哥。这两个人是对的。华生，另一方面，回答“多伦多”-为美国城市类别！令人印象深刻的系统也犯了一个错误，一个人类永远不会做出，一个二年级学生不会作出。

Our machine intelligence can fail in ways that don't fit error patterns of humans, in ways we won't expect and be prepared for. It'd be lousy not to get a job one is qualified for, but it would triple suck if it was because of stack overflow in some subroutine.

我们的机器智能会以不符合人类错误模式的方式失败，以我们无法预料的方式来准备。如果没有一份工作是合格的，那就太糟糕了，但如果是因为某些子程序中的堆栈溢出，那么它将是三倍的。

(Laughter)

（笑声）

In May of 2010, a flash crash on Wall Street fueled by a feedback loop in Wall Street's "sell" algorithm wiped a trillion dollars of value in 36 minutes. I don't even want to think what "error" means in the context of lethal autonomous weapons.

2010年5月，华尔街“卖出”算法的反馈回路引发的华尔街闪电崩盘，在36分钟内抹去了1万亿美元的价值。我甚至不想在致命的自主武器的背景下思考“错误”的含义。

So yes, humans have always made biases. Decision makers and gatekeepers, in courts, in news, in war ... they make mistakes; but that's exactly my point. We cannot escape these difficult questions. We cannot outsource our responsibilities to machines.

是的，人类总是有偏见。决策者和看门人，在法庭上，在新闻里，在战争中……他们犯了错误，但这正是我的观点。我们无法逃避这些难题。我们不能把我们的责任外包给机器。

(Applause)

（掌声）

Artificial intelligence does not give us a "Get out of ethics free" card.

人工智能并没有给我们一个“摆脱道德自由”的卡片。

Data scientist Fred Benenson calls this math-washing. We need the opposite. We need to cultivate algorithm suspicion, scrutiny and investigation. We need to make sure we have algorithmic accountability, auditing and meaningful transparency. We need to accept that bringing math and computation to messy, value-laden human affairs does not bring objectivity; rather, the complexity of human affairs invades the algorithms. Yes, we can and we should use computation to help us make better decisions. But we have to own up to our moral responsibility to judgment, and use algorithms within that framework, not as a means to abdicate and outsource our responsibilities to one another as human to human.

数据科学家弗雷德本尼森称这是数学清洗。我们需要相反的东西。我们需要培养算法的怀疑、审查和调查。我们需要确保我们有算法问责、审计和有意义的透明度。我们需要承认，把数学和计算带入混乱、充满价值的人类事务并不会带来客观性；相反，人类事务的复杂性会侵入算法。是的，我们可以并且我们应该使用计算来帮助我们做出更好的决定。但我们必须承认我们的道德责任，并在这个框架内使用算法，而不是作为一种手段来放弃和外包我们的责任，作为人类对人类的责任。

Machine intelligence is here. That means we must hold on ever tighter to human values and human ethics.

机器智能在这里。这意味着我们必须更严格地坚持人类价值观和人类伦理。

Thank you.

谢谢。

(Applause)

（掌声）