So, could we start with you introducing yourself briefly?

[...] I’m Tuomas Sandholm. I’m Angel Jordan professor of computer science here at Carnegie
Mellon. I’m the founder and director of the Electronic Marketplaces Laboratory and co-director of
CMU AI. I’m also the founder, president, and CEO of three startups: Optimized Markets,
Strategic Machine, and Strategy Robot.

Okay, great. So, could you tell me a little bit about how your work in AI began?

Yeah, my research in AI began in 1989. I was doing a Master’s thesis in Finland and I built the
first automated negotiation system for self-interested agents that was truly distributed. It was in
the space of trucking which was the least sexy of applications, but they're actually very
combinatorial and that led to a multi-decade agenda for me, you know, to make it negotiation.

And what were some of the primary influences on that first work? Were you ever influenced at
all my popular culture, things like science fiction? Or was it really based within your work in
mathematics or within computer science?

It was a little bit of both. So, it was based on my AI studies and of course the popular movies,
watching Terminator, and I wrote some kind of little philosophical pieces on what the AI could be
as a student in the student newspaper, and the little things like that. And then my studies in
Operations Research, and it came together in that application.

Okay, interesting. So, this idea actually of writing for student newspaper, it sounds like you were
attune to or sensitive to communications in regards to AI from quite early? Is that the case?

Yeah, yeah, but I was doing AI communications before I really knew anything about AI. Like so
many people today.

That’s been the case since 1920 it seems. So in regards to communicating to broad publics in
regards to AI, can you think of an example either in your own work or in your colleague’s work
that has been either particularly useful or accurate in gaining, allowing a broad audience to gain
understanding of what these systems look like or what they do?

Yeah, so I think that the kidney exchange work that we’ve been doing has been communicated
pretty well for example by the National Science Foundation, by CMU, and so on, and hopefully
also by us. In that, you know people think about AI often times as a bad thing, and it’s going to
be terrible for humans, but the reality is totally the opposite. AI’s adding a lot of value and here
it’s really life and death things and there are hundreds of people, if not thousands, walking
around today because of it who wouldn't be otherwise.

Could you say a few things about that system?

Yeah, so we run, or our software runs the nationwide Kidney Exchange for United Network for
Organ Sharing, which is the government contractor that controls all of the transplantation in the
U.S. And we are optimizing the kidney exchange side of it, which is a form of live kidney
donation. Kidneys are by far the biggest transplant in terms of number, more prevalent than all
other transplants combined. More than all other solid organ transplants combined. And kidney
exchange really allows these willing but incompatible live donors to swap donors in clever ways
so that patients who don’t have a compatible donor actually get compatible donors. And we’ve
run the kidney exchange for UNOS, we’ve been running it since the outset in 2010 when it went
live. We optimized the plan for a hundred and fifty-nine transplant centers together every week.
So, it’s 69% of the transplant centers in the U.S. have already opted in.

That’s great. So, within your own work as an educator and as a researcher and it sounds like
also a founder of three different companies, what do you see is your responsibility in
communicating these complex systems to a broad public?

Yeah, that is very tricky. How do you communicate complex issues to the public? And I think,
you know, clearly there’s the absolute honesty that’s most important, that you have, that there’s
no hype, no exaggeration of the good sides or the bad sides, you try to be objective, but that’s
only half the challenge. I mean the other half of the challenge is to try to communicate these
very complex systems to people who don’t really have the background to understand them at
the deep level. And that’s always a challenge. And startup companies have faces that challenge
in selling complex systems. And then in this field of systems which is where we’ve successfully
fielded them, like in sourcing markets we’re done that commercially, in kidney exchange we’ve
done it pro bono, it’s hard, how to communicate it, and we have several ideas on kidney
exchange which are actually provably better than the status quo. And they’re not yet fielded,
and a big problem to what’s fielding them is communicating them to the public. Even to experts
in kidney exchange, communicating these mathematical things is quite difficult.

So...

Am I answering this at the right level?

Yeah, it’s been great so far.

I’m kind of making this up on the fly, maybe I should think a little more.

No, it’s okay I think, these are, these are pointed questions to get, you know, kind of honest
reactions. There’s been a lot of discussion and oftentimes speculation pertaining to AI systems
and the labor markets. So, I’m wondering, in your own work or as you observe as a practitioner
in the field, how do you think AI systems have changed the ways that people have worked up
until now into the present moment?

Yeah, not much. So, I don’t think AI has really changed labor that much yet. If you go back to
the kidney exchange, there it has certainly increased employment in that more transplants are
getting done, so you need more surgeons, more anesthesiologists, more nurses, more hospital
beds, all that. And you also need transplant coordinators. So, the transplant centers need to
have people who are talking to the exchange and people who are actually at the exchange
talking to the transplant centers. So, there it’s not a huge number of people, but still these are
new jobs that didn’t exist before. In the sourcing applications where we’ve fielded combinatorial
auctions for strategic sourcing where companies and governments buy goods and services from
their suppliers, there it became more, humans became kind of higher-level decision makers
instead of humans trying to sift through a number of solutions as what you could buy from whom
and combinations of those where the number of combinations is bigger than the number of
atoms in the universe. You let the computers do that. And that was one of the ways the public
can understand that idea of combinatorial auctions, you let the AI sift through those low-level
possibilities and the humans are talking more about hey what side constraints should we put on
this? What should be in the objective? How do we value these tradeoffs and so on? So, overall,
the idea is that you leave the humans to making more kind of value judgments and let the
computers be good at what they are good at, which is sifting through a lot of different
combinations. And I think that a lot of humans don’t really understand that line. They have a
hard time separating the means from the ends. The ends being the value system, and the
means being how you get there. And I think that the AI really helps on the means side. And it’s
important to separate those. We’ve seen that also in the kidney exchange. Oftentimes humans
confound those two things—the means and the ends. And we’re trying to always push the
humans to say “Okay, what is the ethical framework? What are the value judgments? You tell us
what that is, and the AI is going to figure out how to get there.” Well, humans have a tendency
to micromanage and go into this low-level things as to “Oh, in this situation you should do that,
and, in that situation, you should do that.” And humans are exceptionally poor at that.

Yeah, so as those systems become more sophisticated and as practitioners, who aren’t
necessarily developing the tools, but are still working with the tools, also develop in their
sophistication in terms of understanding these tools, how do you anticipate that might change
the way that people work, let’s say 20 years from now? So not necessarily one hundred years
from now, but really in that boundary space that we’re moving towards.

Yeah, that’s very interesting. So, one thing that I think it a very good prediction, I feel very
confident about this, is the separation of means and ends. I think humans will more focus on the
value judgments and you let the AI sift through the different combinations. And humans will not
so much be making the individual, low-level decisions at the operative level of here in this
situation you should do that, in that situation you should do that, but more giving these higher-
level value judgments and the AI is going to instantiate it into the practical domain. So that’s
one. Then more speculative is kind of people talk a lot about oversight, humans having
oversight over AI, but if you really go back and think about it, who’s making more mistakes,
humans or AI? Well I would argue it’s humans are making a lot more mistakes than AI, so
maybe there should be AI oversight over humans and maybe it should be a little bit of both
ways, and that remains to be seen, how that should work. And how do you hand over control.
But again, coming back to I think the low-level controls are going to shift more and more
towards the AI, humans just making the higher-level valued judgments.

Yeah, and that tension that you just described is I think quite interesting in terms of the crux
associated with human dignity. Right, and the ways in which the first scenario allows humans
really to be in the position of making those decisions and within the power negotiation,
continuing to use the system as a tool. But it sounds there’s at least a lurking possibility for the
inversion of those things when we start to think about the ways in which humans are susceptible
to mistakes. So, do you anticipate some tensions coming?

Yeah, maybe tensions and the notion of tool. If you have an AI system that oversees, make sure
people don’t make mistakes, in my opinion, that can still be a tool. And it doesn't really
undermine human dignity in any way. And another thing is that people said AI should never be
in charge of life and death decisions over humans. And I understand that argument sort of went
to autonomous weapons, but in the kidney exchange, the AI is making life and death decisions
every week. And that’s good because it’s making better decisions than any human could. So, I
don’t think it’s that simple that it’s somehow demeaning to humans when AI makes life and
death decisions over us. That can actually be a good thing.

Yeah, it’s an interesting counterexample because of the ways in which the rapidly developing
systems in these new domains also begins to particularize these questions.

To me it’s not so demeaning but we need to even think about what is demeaning. If the
automobile making some sort of little decision here and there is shifted to an AI why would that
be demeaning to a human? Yes, it reduces the humans autonomy in a way, but the other way to
look at it is it augments a human’s decision making capability, so that the humans can make
better decisions and the high-level decisions, the judgment calls that the humans are making,
really get instantiated into practice, the way that humans specified, instead of the way that
humans tried to do in these big combinatorial spaces where they can’t really do what they intend
to do.

Yeah.

There’s one more point that actually I should make.

Yeah, sure.

So historically humans really believe in humans. And more so than they should. And already
back in the 80s, I was writing a little bit about this, like what if there is a martian that comes here
and they’re smarter than us. Then we’re just the next animal, we’re the chimpanzee, we’re not
the best of the bunch. And with AI we’ve seen that in a lot of applications that you get
superhuman. One fun example was when we were playing this brains vs. AI rematch in heads-
up no-limit Texas Holdem against the best humans in January 2017. The morning of the match
the international betting sites were putting us at a four to one or five to one underdog. And we
ended up winning the match. Even three days in, when we had already beat the humans for
three days in a row, even then the betting sites were still putting us at even. So, humans really
believe in humans.

Yeah, it would be a fascinating conversation actually to have with some of our faculty in social
and decision sciences, the ways in which the...

My wife actually works in that department, so I’m having this discussion.

I’m telling you something new. That’s great. Maybe the kidney matching system or another
example could help us, but I’m wondering if you can describe an AI tool in which the power’s
been transferred from the human user to that system? Like a particular system that really has,
that you have seen that transfer.

Yeah, well kidney exchange is like that. So, the first kidney exchanges were manually matched,
and they were very ad hoc, and rules were not clear. Not transparent. And there’s still private
kidney exchanges are manually matched in the U.S. but the National Kidney Exchange then
went to optimization and actually simpler forms of optimization had already been used by some
of these private kidney exchanges as well. So, this sort of was a gradual transformation, but it’s
not complete. Some of the private kidney exchanges really do manual matching still today.

And, in regards, you said that there was an ethical framework. Just I’m not a technologist right,
so are there criteria that are part of the algorithm that are basically vetted by ethicists in terms of
what the variations are for criteria for the matching system to run and to optimize those
matches?

So, how it works today, the policy of, we call it the scoring rubric. How do you assign points on
the different potential transplants you could do? They have to do with, let’s say age. Young
people being preferred. They have to do with locality. They have to do with wait time. Tissue
type match. How good does the tissue type match, etc. etc. These are designed by politic
committees. I sit on those committees. We have ethicists, we have transplant surgeons, we
have nurses, we have all sorts of people on these politic committees and then they go out for
public comment, which typically take a year. And you get comments, you change your design,
and you go out again. So that is kind of the framework. Now, what’s an ethical framework?
That’s a good question. I mean I’ve published on ethics with ethicists and this is from an
engineering perspective, this is a little bit of frustration for me because the ethicists usually
evaluate things ex-post, as an engineer I would like to get the specification up front, like what
are the criteria? What are the ethical things that you want to measure? And then I want to
design something that’s as good as possible for those criteria. But that’s not how ethicists
typically work, so it’s a bit of a frustration. I think in the future where we have more of a kind of a
market design, AI system design need we’re going to get into kind of operational ethics, where
you really set down a framework and talk about the ethical trade-offs and issues up front before
you design, and then you design, and then you field. Right now, it’s more like you design, then
you have some feedback loop, and, and then you field, and then you have some more ethical
feedback loop there. But I think that the ethical framework is going to shift more into kind of an
ex-ante approach, which is a good thing.

So, many of the systems that we’ve discussed so far have actually been optimizing on manual
processes and to very good effect it sounds like, especially in terms of this organ donation
matching system. Are there examples that you’ve come in contact through your own research or
in, you know, peer review processes or what have it, where you’ve seen an undermining of
human decision-making as humans are working with a particular AI system?

Let me think about this. Well, I don’t really have a good answer to that. I haven’t really seen that,
but I can tell you another related thing which is that when we build better and better AIs for
solving imperfect information games and the leading benchmark for testing those system is
Poker. There we got to a point where human poker players would second-guess the decisions
of AI and would say “Okay, now the AI is making a mistake” and so on. We got to a point where
when humans would say that I would know from experience that chances are that actually the
human is wrong. That the AIs decision is right, and the human is wrong, while the human
actually thought that they were right. So, so you could call that an undermining, but in reality, in
almost all of those cases, the AI was right and taking to some human who knew even more
about the game and maybe it took three days to look at it and think through all of the issues.
This later it would come by and said okay, here’s a three pages essay why that was exactly the
right move. And it’s just that humans don’t think as deeply and this was thinking two levels
deeper in their words and you know this was exactly the right word, but it was, right action, but it
was a weird one.

Yeah.

So maybe that’s undermining if you want to call different actions undermining, it’s undermining,
but they were better.

Yeah, yeah. No, it’s interesting cause where I wanted to go next is considering the concept of
autonomy.

Yeah.

And, another way into parts of this question is whether or not you perceive the prospect of
machine autonomy as something that’s valuable and if it is valuable why, or if not, why?
Yeah, I perceive it as valuable. Myself I don’t really use the autonomy word that much cause I
really believe that there’s this hybrid where humans make value judgments or an ethical
framework if you like, even better, which leads to the value judgments. And then the AI
instantiates the ethical framework of value judgments into every case that you see.

Is it fair that in your mind is that a hierarchy? Or are they just multiple inputs?

I think it’s a hierarchy and its different time scales as well. You know, the ethical framework
comes to be into place over time with plenty of time to deliberate on those higher-level decisions
of which there are far fewer. And then it gets instantiated into much larger numbers of decisions
using the framework, and oftentimes, not always, under different time scales. So, like in kidney
exchange, we can talk about this ethical framework or even these value judgments oftentimes
for two-three years on an individual topic. And it goes through rounds and rounds of feedback
from all sorts of perspectives. So that’s years, while when the AI makes decisions, we’re talking
about making a decision once a week and the decision-making takes minutes. So, it’s a very
different time scale. And the same in autonomous cars. We could talk about what those cars
should do with plenty of time in an office, but when they have to be done, you’re talking
milliseconds sometimes.

Right. Right. So, to work into some of that specificity in terms of time scale, or if we’re thinking
about something like the prospect of the autonomous vehicle, for example, what do you see as
some of the, the most concerning pitfalls of those systems or developing systems?

Well I think there’s two kinds of pitfalls. One of two pitfalls where you actually feel something
that’s not ready and there’s pitfalls where you don’t feel something that’s better than human
already. So, people oftentimes talk about the first kind of pitfall. I’d like to bring both of those into
the discussion and perfection shouldn’t be the benchmark. It’s really, you know, are we safer
than humans. And that’s a benchmark cause nothing’s ever perfect.

So, even if the system’s not necessarily perfect, but it’s in a position oftentimes that will do a
better job in terms of making those decisions than, than a human individual or group, where do
you think responsibility should be ascribed if an autonomous system actually does cause harm?

Well, that’s a great question, and I don’t really know the answer to that there. But that is a
wonderful question.

So, can we mine into the kidney exchange example? So, within that system, the configuration is
such that the outputs are optimized beyond what a manual matching system has been able to
accomplish in terms of the number of matches made by virtue of the different variations. And
therefore, that’s having an immediate impact on individual but also scaled lives within the
system. What happens if there’s a mismatch, in terms of that or?

I’m not sure I understand what we mean by mismatch.

Is there ever the prospect of a mismatch or if someone can get in and hack the system let’s say
and play with that...

Oh, I see, I see, okay.

Where could the responsibility lie if it’s an inadvertent mismatch? Right, something has just
gone buggy with the algorithm, or if it was susceptible to hacking. Where might responsibility
lie?

Yeah, that’s a great question. So, where the software runs on secure servers at enrichment at
the UNOS headquarters. So, they’re very careful with HIPAA and all those types of things so
that their servers are secure. So that’s about the hacking. I guess, you know if they got hacked it
would be their responsibility. In terms of the AI decision-making, we actually had, when the
kidney exchange was small, it was so small on day one you could actually run a simpler solver
of the shelf to actually see that you get the same solutions or equally good, different solutions
out, so they were very careful of that as kind of having a shadow solver be there to make sure
the answers are right. And the same we’ve seen in the FCC spectrum auctions. They tend to
use shadow solvers, so they don’t just use one AI, they have multiple AIs and they check that
they come up with the same answer or at least equally good answers, optimal answers. Well
maybe, in the kidney exchange optimal answers, in the spectrum auctions maybe not exactly.
Yeah, so, we are finding optimal answers with respect to the value-judgment framework. Of
course, you could ask whether those judgments are right in some sense, but those are
manmade. And, another issue is that, or maybe interesting topic, not so much an issue, is that
what decisions humans would’ve made because they’re different from what the AI made, not as
good, but they may have saved different people. So, the same lives don’t get saved, and even
with the best decision-making, there’s not possibility to save all of the lives because there just
aren’t enough donors to go around. So, the solutions are different, just the AI solution being
better, but clearly, it's not better for the people who didn’t get selected.

Yeah, that really opens up that, the interpretation of harm.

Yeah.

Right, so the ways in which that data could be compared somewhat longitudinally even though
some of the decisions are made differently.

Yeah, yeah.

It would be a really interesting longitudinal project actually.

Yeah. So, you...

The manual selection process, the criteria associated with that, as well as the variations or
seeming kind of mutations in that decision-making tree...

Yeah.

Versus the way in which the AI system has built to optimize these decisions.

The human decision-making is hard to quantify. You can’t really look into their brain because it’s
not transparent you don’t know what they would do in various situations. You can look at what
they did in the situations that they faced if that data were public.

We’re actually on our last question. So, we’ve been asking everybody what their thoughts are in
the development of general artificial intelligence? So, perhaps we’re circling back to the science
fiction that we began with.
Okay, so I’ll start with something more mundane than science fiction which is the separation of
means and ends, of value judgements and instantiating the value judgements, I think that you
know, humans do this, AI does this, and we have this future match framework on which we have
a paper that does that. So, looking at how do you automatically instantiate the high-level value
judgments. And I think it’s not just for kidney exchange and not for matching problems but for AI
in general. I think that that’s a good one and it’s going to be more prevalent that we’re going to
come up with that type of separation of what humans do, what AI does. General intelligence, I’m
actually not a believer in artificial general intelligence. To me, AI is all about applications and the
benefits to humanity and different techniques for different applications. There’s some generality
in the techniques of course and maybe more generality in the future, but my goal has never
been to build (missing @ 1:29 2nd video) So, it’s never been to build an AI that does everything.
It’s, I mean, I think that’s only mildly interesting and maybe the more serious AI people aren’t
really talking about it as much as the populists.

Is there anything else that you’d like to share that we didn’t touch on today?

Well, I think it’s great that you’re doing this. I think AI ethics is very important going forward
cause AI is really changing the world for the better and will in the long run, and how do we make
sure that, you know, it’s not used for bad, that it’s used for good ways and what does good
mean? You know, I think that’s very important and how people will get used to working with AI
instead of thinking that an AI decision that’s superhuman you know that’s somehow a threat,
how do humans actually understand it more as a tool and an augmentation and a helpful thing.

Well, thank you for meeting with us today.