Frontiers in Machine Learning: Fireside Chat
Hello. And welcome to our frontiers, and machine learning online event. I'm sandy blythe, and it's a pleasure for me to introduce you to this microsoft. Research. Event. This is something new for msr, at a unique time for all of us, when so many of us cannot travel to have these discussions, in person. Frontiers, in machine learning continues, a long-standing. Investment. In engagement, and collaboration, with academia, and the broader research community. And replaces, our north america, faculty summit for 2020.. That event has for over 20 years been an opportunity, to gather together, and renew old acquaintances. Meet new people and discuss, research, of common interest. Now while this event cannot, fully replace the experience, of an in-person, summit we do hope to achieve many of the same objectives. We have had input from many members of the community, on the talks, and the topics of our agenda. And i hope you find the time worthwhile. You can see, that from the agenda, we're aiming for a balance, of presentations. And panel discussions. With opportunities. To ask. Live, questions of the participants. As well as explore, additional, materials. And do some virtual networking. As frontiers, continues. Our investment, in research connections, so too we continue, with and expand, our support for students, and faculty. In pursuit of state-of-the-art, research, across a wide variety, of disciplines. I'd like to take just a moment to overview, some of those award programs, from around the world, that i hope you will find of. Interest. Our research, fellows program, in our bangalore, india lab is a pre-doctoral. Program, that provides, one or two year roles, in our msr, india lab. And offers. Access, to state-of-the-art. Technology. And the chance to work side by side with world class researchers. From across, a variety, of disciplines. Our ada lovelace, fellowship. Offers a full, three-year, tuition. And stipend, for phd, students in their second year at a north american university. Who are from underrepresented. Groups in computer, science. It aims to address some of these structural, obstacles, to diverse students. By providing, an additional, year of support, beyond, that of our north america, phd, fellowship. The north american phd, fellowship, continues. And remains, a two-year, program, for third year students, in north american, universities. And like the ada lovelace, fellowship. Offers an opportunity, to interview for an internship, at msr, and attend our phd. Summit which is a two-day workshop. We held annually, in one of our north america. Labs. The msr, asia fellowship, offers a unique, program. Including, mentorship. Research. Networking. And other academic, opportunities. And includes a cash award. An optional, internship, in our beijing, lab, and attendance, at select, msr, asia events.
The Msr, dissertation. Grant, also recognizes. The value of diversity, in computing. And supports, phd, candidates, in their fourth year or later. From groups again that are underrepresented. In computing. And we're pleased to have recently, announced, the newest group, of 10 students. Who have been awarded, the dissertation, grants for 2020.. This is our fourth year of offering the dissertation, grant and this was by far the most competitive, year yet. Our congratulations. To this year's outstanding. Recipients. Our msr, faculty, fellowship, is a two-year. 100, 000. Per year award, which recognizes. Promising, new faculty, at north american, universities. Whose talent marks them as emerging, leaders in their fields. Here as well, we've recently announced, our 2020. Faculty, fellows. From nearly 200 nominations. And who are distinguishing. Themselves, across, a diverse, set of research interests. Congratulations. To our 2020. Fellowship, winners. Now we've recently, added, two new awards. In emea, and latin, america. For students in their third year or beyond. At a university. In those regions. Here are the first set of award winners from both emea, and latin america. And they each receive a cash reward. To help them complete their research. And also, to have travel, and accommodations. Provided, to attend. Our phd, summit. And we offer our congratulations. On the award. Finally, we continue, our phd, scholarship, program in emea, which provides, an annual payment of up to three years for supervisors. And students, to do collaborative, work on research, themes aligned to our msr, uk, lab in cambridge, england. Recipients, may also receive an internship, to one of our worldwide, labs during the term of that scholarship. These are just some of the faculty, and student scholarship, programs that are available, and if you'd like more information, i encourage, you to check out the academic, programs, tab on our microsoft, research website. Now frontiers. Is a virtual, event as a result of the coven 19, pandemic. Msr. Is currently supporting, a robust, research agenda, on the novel coronavirus. As well as extending, that research, to prepare, for future pandemics. And we're pleased to have announced, just last week, awards, of funding to nine different projects. Involving, collaboration. Between microsoft. And 19, different institutions. To advance, knowledge, on infection, prevention.
And Control. Treatment, and diagnosis. The ethical, allocation, of resources. Mental health. And return to work topics. Thank you, for the collaboration. And congratulations. To these institutions. Where researchers, continue, to work with msr. On pandemic, preparedness. Also of note. Microsoft, research, and our microsoft, ai for health team have partnered with cfar. The canadian, institute, for advanced, research, on their ai, and kovid 19. Catalysts. Grants, initiative. We're proud, to support the program, in accelerating, covid19. Research. By leveraging, our azure high performance, computing, resource. In addition to these programs, the pandemic, has brought into sharper relief the need to upgrade the resource, available, for research. Msr, supports world-class. Research, and echoes, the call, for a national, ai, research. Resource. As recommended, by a recent, national, security, commission, on ai. Eric horvitz, the microsoft. Chief scientist. Was a commissioner, on this report. In addition to providing, ai supercomputing. Cycles. Openly, available, data sets are necessary, to advance the state of the art. Microsoft. Research's. Open data repository. Has made available, curated, data sets that microsoft, researchers, have used in conjunction, with their own research. The site is enabled, to simplify, access to data sets and enable the reproducibility. Of our research. I'm pleased, that we have today, made the source for odr. Available, on github. By example. The emergent, alliance, which is a not-for-profit. Community, which aims to inform, future economic, decision-making. And to aid in societal, recovery, post-covid. 19. Has recently, used those sources from github, and instantiated. Their own open data repository. Of relevant resources. Machine learning is a rapidly, changing landscape. And within the larger scope, it's important for us to recognize. The dramatic, events, and profound, societal, change underway. As we all look to act, and address, systemic, racial injustice, around the world. I won't recount the things that i'm proud of microsoft, that is doing. Because, there's so much yet to be done. We must do better, and we will do the work. We'll sustain. This three company-wide. Multi-year, efforts. We'll be increasing, representation. And our culture of inclusion by adding over 150. Million dollars to our current, diversity, and inclusion, investments. And committing, to and reporting, against, specific. Representation. Of blacks and african americans. In leadership, positions, in the, u.s. We'll be engaging, our ecosystem. To use our balance sheet and engagement, with our suppliers, and partners. To create, new opportunities. And we'll be strengthening, our communities. With the power of data. Technology. Research. And partnership. To improve the lives of blacks and african american citizens, across our country. And the safety and well-being. In, of our own employees, in the communities, in which they live. The opportunity, and the obligation, for change is here. You expected, of us and we expect it from ourselves. We're going to act with intention. So clearly there's a lot going on around us and with msr, and i encourage, you to stay in connection with us. Through any of the. Web and social channels. And to subscribe, to our featured communication. We are at our best, when we're in close collaboration, with this community. And i hope you'll find. That some of the very best of that is on display, this week, during frontiers, of machine learning.
Thanks. And now. I'm pleased to introduce, a fireside, chat, between, christopher, bishop, and peter lee. Christopher bishop is a technical, fellow and the lab leader, of our msuk. Lab in cambridge. And peter lee is corporate vice president. Of research, and incubation, at. Microsoft. Thank you very much sandy for that nice introduction. Hello, and, a very warm welcome, to the frontiers, in machine learning event. My name is chris bishop, i'm a technical fellow at microsoft. And i'm also the lab director. Of the microsoft, research lab in cambridge, uk. Today, i'm delighted, to be talking with peter lee. Who is a corporate vice president and who is in charge, of research, and incubation, at microsoft. Worldwide. Peter it's a real pleasure to have you with us today. Uh thanks chris it's really, really exciting to be here. Great and congratulations. On your, relatively, new role now as head of research and incubation. I guess you started what two or three months ago maybe we could just sort of dive right in and perhaps you could tell us a bit about. Well first of all why did microsoft. Choose. To bring research, and incubation. Uh together, under the same roof as it. Were. Yeah, you know i think in in a way, uh. The that, question, is, really central, and it's left as an exercise, actually to all of us uh in microsoft, research and in the incubations. But i think there, are a couple of. Uh factors, that went into the, thought process behind this. One is that over the last few years. Certainly in the sacha nadella, era, for microsoft. Research. And research-powered. Ideas, and maybe most importantly. Researchers, themselves. Have been getting, increasingly. Involved. In. Creating, new technologies. New engineering. Capabilities. And. Actually new lines of business and new products. For microsoft. And i think that's been a direct response, to the way the, industry's, been going, so when you look at things like. Where silicon, is going in the cloud. Or. The whole idea of confidential, computing. Or. All, of the kind of, intensity, of activity. In the large-scale. Nlp, pre-trained, models. All of those things and more. Are all fundamentally, research powered. And furthermore. Require. The kind of mindset, and world view from researchers. And so. The idea, of, linking, together. What we do in microsoft, research. And. Sort of making it more possible. To capture, emerging, ideas, and turn them into new, possibilities. For. That's microsoft, so-called, incubation, function. There's a desire, to somehow, make that work even better than it has over the last few years. And. I think from the perspective. Especially for microsoft, research.
It's, Pretty, exciting, because. You know if you think about people like doug burger or galen hunt or lily chang or many many others. You know who. Have sort of. Gotten involved now in creating. Very significant, new possibilities. Or people like johannes, gerke, who started off as researchers, and then you know had stints. Leading very significant, engineering groups and now we're coming back to research. That sort of. Can interplay, between research. And, what the company is doing at a large scale. Is just getting. Incredibly. Important. And so it, has sort of just made logical, sense. To try, to create, a single organization. That. That just tries to, maximize. The benefit. Of that sort of thing. There's one other element too just. And i know i'm being slightly long-winded, here but. I think there's also been a desire, in this. Setup, to. Help, organize. And reunify. All of microsoft, research. Under one roof. As you know it's been a little bit fragmented, over the last few years. And, the hope is that. We'll, be able to build on the reputation. And the impact, and thought leadership, of microsoft research as a whole by. By bringing everything together. Thanks peter i mean i agree i think it is a very exciting development not only the reunification. But as you say. Bringing us close to incubation, many researchers. Love the fact that at microsoft, we have the opportunity, to reach hundreds of millions of people and just, anything that reduces, the friction of that has got to be a good thing. Um. Actually, you when you took on the role of course you had a rather unusual start i think you were what a few days into the role. And then, our chief executive, sachin adela asked you to sort of put things to one side and really focus, on the company's, response, to covet 19. And think about, how. There might be opportunities, for technology, to really help the world respond to this, pandemic, so can you tell us a little bit about that experience. And some of the projects, that were spun up and, and any sort of early results that we have. Yeah sure and uh first of all let me um. Just start by saying it it has been kind of a crazy time and i know um, you know crazy, is both the positive and negative, just given. Uh the seriousness, of this, pandemic, crisis, um and also uh chris let me thank you for your patience, because. You know we. Made this change and then you and i uh, you know were put back together, um, working very closely, and then you know five days. After that change happened. I had to uh kind of put. Things on hold, for a little while and um your patience with me i think is, i really do appreciate. A great deal. Um, because you know what happened was.
We Made this change and we created this new. Research and incubation, organization. I think that happened on a monday. And then on, that, same week, on thursday. Satya. Kevin scott. And curtail benny. Asked me. To put things on hold. And focus, on coordinating. Microsoft's. Response. Science and technology, response. To the covid19. Crisis. And, there was frustration, because you know it was hard to know, how to get your ideas, heard and seen and how to recruit and mobilize. Uh resources. So one of the things that, um. Just to try to get handle on this, um, was um. We worked with the garage. To stand up the hackbox, platform, that's used for our annual hackathons. And we created, a hackbox, site. So that people who just wanted to volunteer, their time, uh could could browse, projects, that were being posted, um and join them. If you had an idea for a project. Uh you could write them down, uh their descriptions. And and, get them surfaced. And and. Recruit people. And then we, had a. V team set up. Involving mostly people from microsoft, research. To. Triage. Through all of those. Projects. And try to surface ones that might, benefit. From more focused attention. And that whole process. I think was really good, one thing that microsoft, has really trained itself to do is, is to do hackathons. And so. By the time we, close that down, there were 1100. Microsoft, employees. Uh who had joined projects. And there were 186. Projects. That were. Set up, um, and each project. Had on average about five people. Uh or so. Out of that. There were a couple dozen, that. Kind of got plucked, out. And really. Got a lot of very focused attention. And some of them have had tremendous, impact. One. Of course was in direct. Crisis, response. For, hospitals, and clinics. That involve the health bot. Technology, that's built on the bot framework. And. You know the problem there was, you know people were flooding, emergency, departments, in hot spot areas. Were calling. Call centers you know that were manned by nurses and it was just. You know people were getting overwhelmed. The health bot, uh was set up. In, collaboration. With the u.s, uh cdc. To have a self-assessment. Protocol, and then a smart handoff. To a telehealth, session. Uh, to. To a call center. Or. Possibly, to an emergency department. Or a drive up testing center. And. By now, over. 2100. Hospitals and clinics around the world. Have installed, this. And, the official. Self-assessment. Bot for the cdc. Uh is using this and so that's been tremendous, and the hospitals, that we talked to are reporting, a 30 to 40, reduction, in call center or telehealth, volume. As a result of that. Another project. Is. Has to do with diagnostics. Working with adaptive, biotechnologies. We've been. Involved. In, the deep analysis, using machine learning. Of t-cell. Response. To. Covid19. And. That's resulting, in a data set called, immune, code, that's an open dataset, for researchers. And it, is already, looking like in the first tranche of data that's been published. That there is a new type of diagnostic. That's based on t cell response. As opposed to antibody, measurements. Or direct, virus. Pcr, analysis. That would be. Much more precise. Much, higher sensitivity, and specificity. As well as. Catch. And diagnose, disease early. On, and that's been sort of, in the realm of a large number of other activities. In support of new. Drug therapies. In vaccines. And then. There's been just tremendous, amount that just has to do with the public health, just, analyzing. Capacity, and utilization. For things like. You know where are. The next hot spot areas where are the vulnerable, populations. In, various countries. And and, how well do those things match up with. Supply, of, intensive, care units, ventilators. Pbe equipment, and so on. Um and so uh just a tremendous, number of projects, like this that. I think we should all feel really. Proud. Of you know i think microsoft's, response, just. Just really. Has made a difference. And continues, to make a difference. And out of that whole hackathon, effort, um. Over a third of the participants. And over a third of the projects, came out of microsoft, research. Which i think is just. Just really amazing, and it's really brought microsoft, research. Front and center. Uh, in in the company's, response. To covet 19.. Um. Clearly healthcare.
More Generally, is a major. Uh opportunity. Really for machine learning to have, impact, i know something that's very close to your heart and i wonder if we could just. Just step back a little bit um because of course before you took on your current role, you spent several years building, up the healthcare. Activities. In microsoft. And and can you share with us some of your thinking, about. Microsoft's. Strategy, in the healthcare, space or even, you know why microsoft, why should microsoft, be involved in healthcare at all. Yeah, and uh chris you shouldn't give me too much credit for this because, uh you know, you yourself, have been very, much involved in this, and um, and, the cambridge lab in particular. And it has been. Something that has involved not just. Research. But also, the commercial, business. Teams in azure, and. In the experiences. And devices. You know for me i think, the way i thought about this, was in three stages. Relevance. Value. And transformation. And, and and they kind of came in stages. You know when. Satya. Asked. Us to take on healthcare. The first. Order of business. Was this issue of relevance, and what i mean by relevance, is. Um. How would the, stakeholders. In the world of healthcare. Understand. That microsoft, had something to offer, so how could we be relevant, to the healthcare industry, to. Healthcare, providers. Hospitals, clinics, health systems. To insurance. Providers, and other pairs. To the biopharma, industry. You know to medical, technology. Companies, startups, and so on so there's a relevance, there that we had to somehow, figure out. Uh how to. Earn. Because, by doing that then we could get into, collaborations. And partnerships, and start to learn, more. But then there's also relevance, within. Microsoft. Because. Healthcare, is one of these, areas. Where. You know. Everyone, has, direct experience with healthcare everyone has an opinion. That experience, tends to be colored, by, people's, personal, contact, with. Doctors, and nurses and hospitals. But is largely, ignorant, of the much much bigger world. Of healthcare. That happens behind the scenes, and so we had to work to earn. Credibility, and relevance, internally. Um and so. To do that you know, that really meant trying to find the right partners across microsoft, and early on. I made a decision, to partner closely, with the commercial, business. Led by judson, aldhoff. And by. Jean-philippe, courtois. To really. Identify. Key early partners. That we could work with, you know like the nhs. Like humana. Like walgreens, and so on. And by doing that we earned both internal external, relevance. Then, the second stage was value, and. That was largely, about. Data, and ai. There's a tremendous, amount happening with healthcare, data today. Around, this problem what's called interoperability. You know really trying to get, health data, flowing. Uh in a standardized, format. To where it needs to be. And making it more susceptible. To machine learning and data analysis. So we've done. A huge amount of work. To sort of, evolve, azure. Dynamics. And microsoft, 365. So that they speak the language, of health data. So you hear things like, fire. And. Smart. And so these are sort of new emerging, standards for health data. And then. The ai, is tremendously. Fundamental. And, and important. Um. You know a huge amount of that health data is unstructured. Text so nlp, and machine reading. Become. Incredibly, important. Computer, vision. To really, understand. Medical imaging. Understanding, molecules. Understanding. The human genome. All of these things, understanding. The immune system. And the immune, all of these things end up being fundamentally. Machine learning. And ai problems. And so. So that's another area. That we've really been focused, on and. You know, trying to build up that technology, stack. For each of these things. And then get things out in as products. Has been. Uh, the the big challenge. And um. You know and and your question chris. Why are we doing this. My favorite example of this is um. To. Try to get, people within microsoft, to understand. The global. Market for healthcare. Is estimated, to be about 7.5. Trillion, dollars. Now you know what what does that mean. Let's just take, one. Company that we work pretty closely with, in the u.s it's a company called optum.
And What optum, does is, they handle the data. For medical, claims. To route them from, healthcare providers. To payers. And then the remittances. From payers back to providers. And so that data stream that goes back and forth. Is a very important, function, in the health care system in the united states. And there's a lot of data analytics. That, helps facilitate, that in both directions. And optum is, the second largest. Provider. Of that service. In the u.s healthcare, system. So that. Niche. Market. In optum. Sustains, a company, optim. That has the same. Head count. And the same annual revenues. As microsoft. All up. And so if you think about the possibilities. In this massive shift that's happening right now of healthcare. To the cloud. There is no reason why healthcare. In our cloud. Shouldn't be bigger, than, all of microsoft's, current business combined, and of course one of the. Most interesting. Points of our. Collaboration, together has been, about with, novartis. And so um. It's just, something that, we've all been so excited about so it'd be great to. Hear from you uh, a little bit more about that. Yeah so this is a really. Exciting opportunity, it's also a very different, um, sort of mode of operation, i suppose, for microsoft, research, i've been privileged, to be part of msr, for. Over 23, years now and. Historically, we would do a lot of basic research, from time to time we would, transfer, technology, into products, the products would ship customers would get to use them and so we'd have real world impact but it was this very. This long chain of, process. Uh. By which we connected, with the real world as it were. And now in this partnership with novartis, working directly with a customer. And, it's uh it's very exciting i think very, very relevant, in the world of machine learning in this new data-driven, world because. We're no longer, thinking about. One-size-fits-all. Technology, that sort of put on a disk and shrink wrapped and and. Sent around the world. Things are much more bespoke, now bespoke to a particular domain and even to a particular. Collaborator. And a particular. Application. And so we're working very closely with novartis, under this new. Five-year, partnership, it was signed last year it kicked off in, january, of this year. There are different components, to it but the piece that i'm looking after is the research, component, the the longer term. Uh. Component, of this uh collaboration. And it's really a a. Peer-to-peer, partnership, between scientists, in microsoft research and scientists, in novartis. Bringing together their amazing, expertise, in in pharma. And their. Amazing. Data that they've been building up along with our expertise, in machine learning and of course leveraging. Microsoft's, cloud, with the the storage, and the very powerful compute that we have. And seeing. How together, we can go after some really tough challenges, that it just wouldn't be possible for, either organization. To do on their own. So we've we've started a number of projects, we're probably not yet talking. Publicly in detail about those projects but i can share a sense of the the kinds of things that we're working on together. One of the things we've done, is to think about. Leveraging, the sort of expertise, and technology, that we already have in microsoft, research and seeing how that's applicable, to some of the challenges, in novartis.
For Example, we have a project called inner eye, that was started, a few years ago. And this really looks at medical imaging, and in particular the segmentation. Of three-dimensional. Medical images, such as cat and mri. And there are various applications. For this but, one important, one is, so-called radiation, therapy planning. So if somebody has a solid tumor, that's going to be treated with radiation. Then there's some software, that has to optimize. The three-dimensional. Shape of the beams, to sort of maximize, the damage to the tumor. And minimize. Damage to surrounding, tissue and especially to vital organs. And so in order for that software to work it needs a three-dimensional. Map of that tumor and that's where inner eye comes in, so at the moment. Radiation, oncologists. Will will take the 3d, scan. And then literally, go through this slice, by slice. With a stylus, on the on the computer, screen, marking, out the boundary. And that can take you know 20 minutes for a simple case or if it's metastasized. There are multiple, tumors it can take several hours, it's painstaking. It's tedious, you kind of have to be accurate. And this is where nri, can really help. The workflow, for the radiation oncologists. By. Automatically. And in the space of a few seconds producing a sort of candidate, segmentation. And the the human expert can then go over and fix up any any little, details that they want to change. But it really speeds up that, that process. That that technology, is actually being used in, in a research environment. It quite extensively, in a local hospital in cambridge called adam brooks a very one of europe's largest teaching and research hospitals, and we're, we're delighted, to see that technology. Being being explored, in that way in, in. Clinical practice. Effectively. You know chris the thing that i think, that particular, example. Highlights, is. Again how important, microsoft, research, is because. It's it's not, that. It wasn't, possible. For the inter-i application, to just take, an existing, off-the-shelf. Machine learning or computer vision system, or even an, off-the-shelf.
Machine Learning algorithm or computer vision algorithm, something, new had to be developed, specifically, for that application, to work well. And you know it really, takes a world-class. Research, organization, to be able to do anything like that and, and so it's, it, really highlights, just how important the partnership, with. An organization, like microsoft, research, is. Yeah i think that's right i think it's that that intersection. Of both deep research, and real-world, application. That. Gets a lot of researchers, excited, i mean for me it's the thing that gets me out of bed in the morning the fact that we. We we have an opportunity, to have impact on the real world in the case of healthcare. Improve lives save lives. And, and yet that's enabled, by first of all tackling some very hard research problems so it's that it's that combination, of deep research and real world impact that for me at least is tremendously, exciting. Yeah for sure. Another nice example of something we're doing with, novartis. Really goes to the heart of their business which of course is creating new drugs new therapies, which, which effectively, means discovering, new molecules. What's interesting, here is that, the nature of the data is rather different, from many other applications. If you think about, let's say imaging, again. Images tend to come in a fixed size or you can re-sample, to a fixed size so the neural network, always gets data, in the same, format, the same dimension, as it were. But molecules, are interesting because they're, obviously they vary in size and shape and configuration. And so you can't take a simple representation. Of a molecule. And treat that as the input to a neural network because it's it has this variable, variable, configuration. There's some. Techniques, called, graph neural networks that were really pioneered, in in microsoft, research. Which, addressed that problem of how to, take machine learning and apply it to data which it has variable. Size and structures, things like molecules. And so again that's a really nice example, of taking some of the deep research, from microsoft, research. And combining, that with the expertise, that novartis, has in understanding, the relationships, between the structures of molecules, and their biological, activity. And that's a project, which, it would be difficult, to imagine. Either either group on their own doing such a good job but together we can do things that, i think are really unique and very very interesting. Yeah and again you know i think. One of the interesting, scientific, challenges. Is the. Uh. You can't hope to solve this. Uh, problem. Purely, based on the data. Nor, purely on the basis, of our understanding, of the chemical. Uh. Processes. You really need a combination, of the two. Sure, and i and i. I think one of the really interesting things about healthcare, of course is the opportunity. For, for real world impact, and and and. Benefiting, society. I think also healthcare, really throws a spotlight, on many many, deep challenges, research, challenges, in machine learning and in fact i know we're going to be, looking at many of those during this frontiers, and machine learning, event. And, this may be a good moment then just to play a little taster video, so this is a little clip by best miranushi. Who's a senior researcher, at microsoft. And she's going to be leading a session, in this event on machine learning reliability. And robustness. So let's just have a little look at that clip. Hi everyone, my name is bismillah. And i am a researcher, in the adaptive, systems, and interaction, group at microsoft, research. This year at the frontiers, in machine learning event we are organizing, a session, on machine learning reliability. And robustness.
To Discuss, recent work on reliability. Guarantees. Of machine learning algorithms. But also on how such guarantees, translate, to the real world. This session, will review, properties, of machine learning algorithms. That make them more preferable, than others. From a reliability. Lens such as consistency. Interpretability. Or generalization. To new instances. And, second we will also review, tooling support, that is needed for machine learning developers. To verify. And build with these properties, in mind. We will have three well-known, researchers, in the field to share their work and thoughts. Tom dietrich, is a distinguished, professor, in oregon, state university. And he will talk about, novel category. Detection. In machine learning and vision. Educamar. Is a senior principal, researcher, in msr. And she will share her work on blind spot detection, in the open world. And finally, such issaria, who is a professor, at john hopkins, university. Will talk about, the work that is done in her lab for safe deployment, of machine learning. Under various, data shifts with applications. In healthcare. In the last 30 minutes we will also have an open discussion. Among the speakers, and the audience. Thanks so much for attending, and very much looking forward, to meeting you virtually, at the session. Okay so that's a pretty impressive, lineup of participants. That bismir, has assembled, for that and i'm really looking forward to that session. So, um, peter we've talked a lot about covet 19, and of course another, big impact, from this global pandemic. Has been this amazing, shift towards. Remote working and working from home. And, an associated. Sort of explosion. In the use of remote collaboration, technologies. Things such as microsoft, teams. Now johannes, gurka, is a technical fellow at microsoft. And he recently, transitioned. Into our org. To become the managing, director. Of our complete portfolio, of research, activities, in redmond.
Johannes, Prior to that was responsible. For things like large-scale. Engineering, activities, in microsoft, office. And specifically. In the area of the scalability, of ai, and microsoft, teams. Before joining microsoft, he was a professor, at cornell, university. Where his, scholarly, work and many accolades. Including, election as an acm, fellow. And an ieee, fellow. And. He also won the 2011. Ieee. Computer society, technical achievement, award. So i think johannes, is probably the ideal, person to share with us his thoughts, on the changes, that we're seeing, in productivity. And, the technology, that supports, that, and more specifically, how machine learning, can further empower us in in that world. So a few days ago i spoke with johannes, about his new role. And i asked him about the impact of remote working, and also the power of machine, learning. To improve this technology. So let's have a little look at that interview. We're delighted, that you've joined, microsoft, research. I know you've only been enrolled, a very short time but can you tell us a little bit about the new role and what it will involve. Yes i'm, in the role now for about four weeks and it's really a privilege to be here i mean there's so many amazing, people there's, really a. Huge breadth of research, going on and you'll see this um you know throughout this conference as well. Um. So i think what it can bring to the role is that i have seen, the different. Parts, of research, both at a university, and now i'm experiencing, it here at microsoft, research. But i also bring with. A good understanding, of what it means to be both in a startup. As well as in a product team. So especially here in the last seven years while i was at microsoft. I've gained a deep understanding. And a lot of empathy, of what it means to ship products. And how to scale software development to hundreds of engineers. How to think, when you're in a product group when you have to develop new features when you have to talk to customers. And what are the challenges, in working with all the existing, systems. Because. Every new system that you build here at microsoft, often has some legacy components, to it i need to bring these legacy components, into the new world. So i hope to leverage. This breadth of knowledge both from the research side, as well as from the microsoft, internal side here in manual. So, our focus. Here of course is on machine learning so, what are your thoughts on the role that machine learning could play in terms of productivity. Tools and technology. Yeah so there, are quite a few um. Interesting. Ideas, i think, the first one of course is to look at the audio and video stack, and and look at. Where there's often, a lot of old control theory whether we can replace that with machine learning. Um you know we're recently, and we're about to ship noise suppression. Where we basically, take an um an old-style, gaussian noise suppressor. And, replace it with machine learning and the advances, there are really amazing. And this this is also, a good example, where. Machine learning research is playing a big role but also the gap between. The papers that were published. And actually what can be shipped, um was really big and so we had to do a lot more work to make the mods more performant. And also work for the large variety of noises that we actually see in practice. So so basically the whole. Control, plane maybe even the data plane of the audio media stack can be replaced with machine learning in my opinion. Um second of all there might be very interesting, user-facing. Features. You know if you think right now we have a feature where i can raise my hand. But then people forget, to, take their hands down or when they're done speaking.
So I think there's a lot of user facing features where we can just ease the. Level of interaction, through the subtle signals that we usually see when we talk with each other one-on-one. Or you know in a physical setting, that we don't really have in a virtual setting yet here at all. Yeah i think it's really interesting isn't it to see just how ubiquitous. Machine learning is becoming, and as you say how many of these, sort of more traditional, problems are now being, replaced, with machine learning solutions, that, many times work better because they're tuned to the particular, data or the particular environment, in which they're actually used instead of being sort of general purpose. I think that's i think that is one of the the big frontiers, of machine learning these days. Thank you very much johannes. And if you'd like to hear more from johannes, then the full 15-minute. Interview. Will be available in the highlight, section. Of the website. So uh peter i think it's pretty exciting that we've managed to persuade johannes, to come and join us in microsoft, research don't you think. Yeah well let's face it uh johannes. Is genetically. A researcher. And. So he belongs, in our org and so it's i think it's great and i know, uh we're all very excited. I've, actually tracked johannes's, career ever since he started. His professorship, you know he finished his, phd, at the university of, wisconsin. Actually his. Phd, advisor is raghu ramakrishnan. Who's now in azure of course, also a technical, fellow. And. You know johannes, did some, wonderful, work, as a professor, at cornell. And then, joined, microsoft, now i think the intent when he joined microsoft, was always to be a part of microsoft, research but, he quickly got sucked into. Some, significant, engineering. Leadership. Opportunities. In various. Product groups and and, so it's just, thrilling to have him. Be here, and you know as. Always, almost everything is infused. With machine learning. And so. Maybe, chris this brings us back to you. Because you've been. Of course. One of the pioneers. And, one of the worldwide, leaders in machine learning for the, past 30 years or more. It would be interesting to get your thoughts on. You know how has the field changed and evolved, over the 30 years that you've. Been in it. I think for me though the biggest. Shift. Um over those 30 years has been. Really in the in terms of the emphasis, of the field because for most of those 30 years certainly for the first 20 of those 30 years. If i'm honest machine learning didn't really work, that well there was a lot of excitement. Everybody understood the potential. Intellectually. It was fascinating. But the reality, was that the performance. Of of many machine learning systems, was really, not adequate, for real world use there were maybe a few niche applications. But mostly it didn't really live up to the, to the promise or the excitement. And of course that's really changed in the last decade, and especially, through the development. Of deep neural networks, and deep learning. And and scaling, up to large data sets and large amounts of compute. And so today we're in a world where there are literally, thousands of applications, of machine learning i mean most people have used several already today probably even without knowing it, um it's becoming ubiquitous.
And That means that. Although we continue. To, have a strong focus, on the on the performance, in the sense of the accuracy. Of machine learning we always want to make it more accurate. Because we're now using. Machine, learning in real world applications. It opens up a whole raft, of of new challenges, i think of as a sort of a penumbra. Of research, questions, that surround, the core question of just getting the machine learning to work at all. I think things like. Biases, that creep into the predictions, because of biases in the data set for example. Thinking about fairness, thinking about explainability. Thinking about, causality. If we actually want to make interventions, on the basis of predictions. Um. Adversarial. Issues you know, nobody was going to attack my my neurops paper 20 years ago. But, once you put something online, and you've got 100 million people using it, there are then adversarial. Uh, agents, out there, people with ill intent will attack it in all sorts of ways for a variety of different reasons and we have to, we have to worry about those issues as well. And, actually this is probably a good point to show another of the taster videos, um because i think this is uh very relevant to this discussion. This is one by rich caruana, he's a senior, principal researcher, at microsoft, research. And. He's, in this video he's talking about an upcoming session that he's organizing, at this event. Called interpretable. Machine learning so let's hear from rich. Hi, i'm rich carwana, a machine learning researcher, at microsoft, research in redmond. This session is about saving lives, by using interpretable, machine learning, in healthcare. Because healthcare data is very complex. It's critical to use interpretable, machine learning methods, to make sure that the models we train, are safe to deploy, and use on real patients. One challenge is that most patients are already receiving, treatment. And that can cause confounding, in the data. For example. A model might learn that high blood pressure, looks like it's good for you, because the treatment given to you when your blood pressure is high. Lowers your risk compared to healthier, patients who had lower blood pressure to begin with. There are many ways this kind of confounding, can cause models to learn to predict, very risky things. In the first presentation. I'll talk about problems like this that we see in healthcare, data, thanks to the interpretable, machine learning methods we're using. In the second presentation. Ankara turo desai, from the university of washington. Is going to talk about fairness, in machine learning when it's used for healthcare. And in the last presentation. Marzia, gassami, from the university of toronto. Will talk about how interpretable. Explainable. And transparent, ai. Can actually be dangerous, when used in healthcare. Looks like an exciting lineup so please join us. Wow, chris you know that's just awesome, and you know what a what's an exciting, session, and um some amazing, speakers. Um you know the the whole, event, is about. An effect called, frontiers, of machine learning. And so it'd. Be interesting, i think for people to hear what are your, you know what are chris bishop's views, about the important frontiers, and where is the field heading over the next few years. Okay thanks pete that's a great question. Um, well of course, in one sense my answer is that this whole event as you say is about those frontiers, and really it's not for me to provide the answers, i i really encourage people. Um to you know dive deep into this event engage with all of the different activities, we've got an incredible, lineup, of. Amazing, people external, people and great people from from within microsoft. And. Between them i don't think we'll arrive at all the answers but we'll certainly touch on many of the key issues and hear some some very interesting, uh viewpoints, on these on these many frontiers. Um. I certainly don't have all the answers but i'll just offer a couple of thoughts maybe, um on things which i think are, are. Trends, that we're seeing right now that i think are very exciting. One of them, is i suppose in a sense fairly obvious, and it's it's the scaling.
One Of the reasons why machine learning works so much better today. Is because we've learned to scale, to scale the size of the data sets to learn that to scale the size of the learning algorithms, the models in terms of the number of parameters. And of course we've had to scale up the compute, in order to be able to train large models on large data sets. And and that trend looks set to continue when we think about the developments, in in natural language models for example. There's no sense that we've reached some sort of asymptote, there's every indication, that as we bring larger data sets bigger models more compute to bear, we'll see more and more, improvements, in performance, more and more of these emerging, properties. Has really been quite remarkable. So real challenge, for the field is how we stay, on that curve how we continue, to, see these massive, increases. In. In in the performance, of machine learning, hardware. And, of course that's something which of great interest to microsoft, and we're doing a lot of work in that space. At the moment. So i think that's one, very important trend i think that's set to continue. The other one. Really. Relates, to the fact that machine, learning, is really about data data sits at the heart of of machine learning and as we seek to. Bring the power of machine learning to more and more domains, we talked a lot about healthcare. Is a great example, of many other domains, where, the data. That's being collected the data is available the data we could potentially, gather, in the future. Clearly has a lot of potential. To bring great benefit to society. But much of that data is also very sensitive, it may be very personal in the case of healthcare data is a great example, but data generally, we need to be very careful about data both from a privacy, point of view and from a security, point of view. And this is a, i think a very exciting, and very important, frontier, it's one where microsoft. In in many ways has has taken a lead in terms of the. The ability, to provide, confidentiality. For machine learning within the cloud we're. The first cloud provider, to, deploy. Technology. That allows. Data to remain, encrypted. Not only when it's being transmitted, over the internet and stored. But right up to the point where it's actually inside, the processor. So the decryption. Only happens, inside the processor. And it means even somebody in the data center. With physical access to the chip. Would only see encrypted, data going on and off the chip they still wouldn't have access to the data so very high levels of. Security, and privacy. And that allows, some really interesting scenarios, so. We know that. Machine learning not only benefits from more data but it benefits, from diverse, data sometimes, you can bring several data sets together, and you can get more than the sum of the parts. And. The question is, how can, let's say different organizations. Different people how can they bring their data together and pool that data, for machine learning, without, simply having to give, other people or other organizations. At direct access, to the data. Well this confidential. Machine learning. Opens up that possibility. The idea that data can be brought together. It's only decrypted, on the chip. It's used within the chip to train a machine learning model. And that machine learning model is then made available, or its predictions, are made available to the providers, of the data. It was trained, on the pool data, so it's more effective, more capable. And yet, at no stage did any of those entities, have access to the data from the other entities. In fact at no stage did microsoft, have access to any of the data. So, i think that. Uh intersection. Of privacy. With, machine learning, is going to be a very important. Area in the years to come. But those are just examples, of frontiers, in machine learning and we'll see many more important. Frontiers, over the next few days. Which actually i think is a good moment to play our final. Taster video. This one comes from staff ross volos he's a senior, researcher, in microsoft, research. And he's going to be leading a session, on accelerating. Machine learning, with confidential, computing. So here's stavros. So hi everyone, i'm starbuzz, boss a researcher. In the microsoft, research cambridge. And i'm sharing, the session on, accelerating. Machine learning with conventional, computing. So in this, session we have an exciting.
Agenda. With topics. Across. The whole, confessional, computing, stack. Okay so let's, let's, talk more about confidential. Machine learning. So today's clouds are spending, an increasing amount of compute, cycles. On machine learning tasks. One key concern, of these systems, is the privacy, of the data. Being analyzed. As well as the results of such analysis. So the, these concerns, have raised the need for confessional, ml, uh platforms. Now the goal of this system is to provide strong security, and privacy, concerns. To cloud tenants. A key block in these systems, is conventional, computing, hardware, which is trusted, by. Cloud tenants. In turn, the hardware, provides, the assurance. To remote entities. That their, data, code. And, models, can remain. Protected, from privileged, attackers. And cloud administrators. Throughout, the computation. Of the job. Now in this session, our speakers, will present applications, and advances, in conventional, ai platforms. First we'll find out how, entities. Can securely, collaborate, and train accurate models. Using sensitive, data, while relying on conventional, computing hardware. Then we'll learn how cloud accelerator, systems, can be designed, to provide strong security. Guarantees. Overcoming. Their performance, limitations, of cpu-based. Confessional, computing. So i hope you enjoyed this, session and would like to hear, your questions at the end of this talk thanks. Well, you know again it just seems so interesting, and you know as you were saying earlier, chris before playing that video. There's, so much, happening, just in terms of scale and in fact i think even specialists. Have a hard time appreciating. Uh just. This the scale that we're operating, at right now it's, just, stunning, and you know by the way this also brings us back to. The beginning of our conversation. About, why bring research, and incubations. Together. Yet i think it does seem extremely natural in this new world to bring bring research and incubation, so close together, as you say i think it's a, very natural thing to to do and very exciting. And because of this ubiquity, of machine learning it means that machine learning is is, not only showing up in lots of different places but it's really impacting, society. In, in very new ways that we haven't seen before. In fact. Microsoft, researcher, mary gray, she's a senior, principal researcher, in microsoft, research and she's going to be leading, a panel discussion, tomorrow, in fact, which will be talking about how. We can push on the machine learning frontiers, in ways that deliver. Better social, equity, which is a a topic of the course is very much on our minds these days so i'm very excited about that uh. So peter it looks like we've pretty much used up our time, um i think for the last 10 minutes or so we'd like to just open this up to questions. Now my colleague, rachel, howard has been monitoring the feed. So, rachel do we have any. Questions. Thanks chris thanks peter. So chris i think i'll come to you first as we've had a few questions related to data, privacy. And perhaps i'll read out a couple and you can. Cover them both at the same time. So we have, since healthcare data is sensitive, and private. There is a trade-off, between maintaining, privacy, while explaining. Any high-level, insight on how to approach this. And the other is. Is there any research, on secure, multi-party, computation. To maintain data privacy. Thanks that's a great question i think it's actually true in general there is this tension, between the desire to create, value out of data. And the need to protect, data and preserve, privacy. And. There isn't a, sort of one-size-fits-all. Answer to this but some research that we're doing in microsoft, research. Um, really, aims to get to the heart of this and address, that trade-off, and you heard heard me talk a little bit about it already there, the idea, that, of course it's very easy to protect data when it's at rest or when it's being, transferred from one place to another because it's encrypted. But to get value out of the data you need to decrypt, it so the idea of this uh, secure, computation.
Is To decrypt, the data only, on chip and the the goal really is to be at the stage where, um, even if somebody were in the data center and even if they had access to all the passwords. And even if they had clips, and could could measure the, signals, going in and out of the pins, on the chip, they still wouldn't be able to see the data they would just see what appeared to be random noise just encrypted data, so that's the goal and that's very powerful. In general. Is particularly powerful in machine learning when you want to do this aggregation, of data from different sources, different people different providers. And, uh and train up models on aggregated, data because those models are often better than, data than models just trained on single sources of data. But there are still open research questions, so we've made a lot of advance the technology, that we developed, in microsoft, research is now deployed. In azure, uh microsoft, was the world's first company to have this technology. Deployed, live in its cloud. But there are still open questions there are interesting questions about leakage, of. Information. Via train models so there's a lot of research still to be told to be done in this space, peter i don't know if you want to add anything. Yeah i think you know. All of the leadership, and leading research, now that is deployed. And is a standard, part of microsoft, azure i think has been tremendous. But as you were saying there's still a lot more that has to be done. Um, and you know there's also a range of things you know if you have healthcare data that's in the fire. Standard fire format, we have, anonymization. Apis. That meet the, legal standards. For anonymizing. But it's not really the same as really kind of locking down and then, protecting. Uh people's identities, and so, uh the need for more. Uh, research, both in silicon, architectures. In uh, core cryptographic. Algorithms. And protocols. All the way to, ai, i think. Is still a major focus especially for for you and your lab chris. Yeah absolutely, i i, saw there's also a question about homomorphic, encryption, which is also very interesting, it's a sort of a i view it as a complimentary. Technique, it's uh, it's one that. Produces, very very high levels, of security, and privacy. Um but it perhaps lacks the generality, in the scaling, that confidential, computing, offers so i think right now confidential computing looks like a very practical, technology, that we're already, using in real scenarios, but a lot more work to be done in this space. Well there's another aspect that i think about with research, because even if we don't necessarily. Feel we can have a general homomorphic, encryption. Uh, deployed. Let's say in product form today, it, dramatically. Influences, our thinking, it makes us think, a little bit differently about the whole problem and how we might approach it. And so, um, it gives us sort of more, room to be creative than uh with you know how would we go about this. Yeah i also think actually it's very beautiful it's surprising, that you can do it at least to me the fact that you can do tomorrow, encryption, and, data without decrypting, it it's sort of magical, so it's kind of inspiring, as well. Yep. Thanks rachel do we have anything else. We do so peter can you perhaps um share a little bit more about microsoft's, approach, to fairness in ai. Well, um, there again there's a range, of, uh aspects. Um and in the chat you know i posted, a paper that's, on my reading list about. Biases. Analysis, biases, and nlp, trained, models. Um but, stepping back for a moment, you know of course, uh, the, technologist. In all of us, is looking for tools. Uh, and, in tools.
Uh Things, uh. Frameworks, like shap, and lime. Where we have very intensive. Research and development, going on. Give us an ability, to create. Models. That then can be analyzed. For different kinds of biases, so if you wanted to ask a question. Is this model. Let's say biased. Um. With respect to age. Uh you know ageism, or. Race. Or gender. These. Shep and lyme and similar kinds of frameworks, give you an ability, to, ask those questions models do an analysis, and and get some insights into whether that's, true or not. And that has actually already, started to have an impact, for example. As microsoft, works for example with the. Financial, services, industry. You know where ageism. For example in the denial, of. Of a credit application. Is is actually illegal. And, that has then. Created, a great deal of interest, across microsoft research and whether applications, of frameworks, like that and tools like this could be useful say in healthcare, settings. And and so that's, one aspect but then, popping up a level there's just also generally. The policy, how should we. Behave. And think, and conduct our research. And deploy, technology. In a responsible. Way. In a way that really, gives us, a chance, that these technologies. As they develop. Are used in the most, ethical. Way. With the most positive societal, implications. And so we try, to work in that span. Of just, actual concrete, tools. That researchers, and developers can use all the way. To thinking, uh about the influence, of these technologies, uh. On our policy thinking, um chris uh i i know you've also been really pushing. A lot of this. Uh as well in your own direct research. Yeah absolutely, and it's interesting i mean healthcare, is one of the fastest growing areas in in the cambridge lab for example but in, the redmond lab and others as well and it's a it's a great, domain, that highlights, i think really brings all of these issues, into, it's a very sharp focus. And not least because, you know the potential, upside when we can address them is so enormous, the opportunity, to improve, lives and improve healthcare outcomes, and so on, so although a lot of these issues are very broad and very generic, i. Personally feel particularly passionate about the, healthcare space is a great domain, to to stimulate, research simply because it's so motivating, at least that's that's something i find personally. So uh i know this time is getting on a little bit i know we haven't addressed all of the questions, but um you know perhaps in the interest of time and with the next session we should uh, perhaps draw this to a close, um first of all we say a big thank you to peter it's been enormous fun chatting with you uh, we could uh obviously chat all day, it's been quite a lot of fun, a big thank you to to the team for pulling us together and for rachel, for triaging. And, asking the questions. So uh we'll move shortly, to the the next session, um this is going to be led by susan demay, she's a world leading researcher. And also a technical fellow at microsoft. And she's going to be leading a panel. Uh called machine learning conversations. So i do hope you can. Join us for that and meanwhile just a big thank you again to peter, and i hope you'll all have, a very stimulating, and informative, event over the next few days thank you very much. Thanks chris thanks to everyone for being here.
2020-08-02 04:43
