Tag: Microsoft Research

Due to recent events, millions of office workers have needed to rapidly adjust to working from home—learning new collaboration tools and best practices, re-thinking how to stay connected with colleagues outside the office, and adapting to new social norms around meetings. Working remotely presents both technical and social challenges, and researchers at Microsoft have been working across disciplines to understand and support both aspects of this challenge for decades.

Below is just a small sample of the work researchers at Microsoft and their colleagues have produced to improve the remote work experience. For those seeking to build better remote work products and services—or for anyone who wants to be more productive at home—we hope this research can provide some guidance, insight, and inspiration.

Although these are truly challenging times, we can benefit from a strong foundation of interdisciplinary research that can help us all stay productive and connected—with the hope of emerging from this crisis better-equipped to work together.

• Paying attention can be harder in remote meetings. Sean Rintel at Microsoft Research Cambridge recently published two papers that can inform the design of features to support remote participants’ attention. One paper models how we ‘see’ attention in meetings. It suggests that machine perception may help us gather, signal, and follow attention when remote. The second paper suggests that low engagement in meetings may not always be a problem. Not every meeting requires our full engagement, but until we develop technological support for more nuanced roles, it is good practice to be up front about your engagement level. Together, these papers suggest that AI supported attention personalization could make future remote meetings more inclusive and effective by helping us overcome constraints and assumptions.
• One benefit to everyone attending a meeting virtually is that it can be easier to review missed content if you show up to a meeting late or have to step out for a moment. For instance, Kori Inkpen, Sasa Junuzovic, and John Tang from Microsoft Research Redmond have explored using “accelerated instant replay” (AIR) to help people catch up quickly and then jump (back) into the real-time meeting.
• In a world without business travel, negotiating time zone constraints becomes even more important. John Tang at Microsoft Research Redmond and Kori Inkpen at the Microsoft Research AI Lab have catalogued strategies for mitigating time zone-related obstacles to productivity and provided guidelines to help overcome these obstacles. Tang and Inkpen also worked with Asta Roseway, Mary Czerwinski, and Paul Johns from Microsoft Research Redmond to explore novel uses of asynchronous video to support collaboration across time zones and developed two prototype systems, Time Travel Proxy and Video Threads.
• Some features in Microsoft Teams were inspired by work at Microsoft Research Cambridge on utilizing multiple devices to support better presentation and collaboration over video; researchers’ participation in an internal hackathon led to close collaboration on development of the product. Their previous research into ad hoc adaptability in video calling, wireless smartphone mirroring in video calls, and shared slideware control helped enhance users’ ability to join meetings on multiple devices and use a phone as a companion device. Learn the inside story of the hackathon and product collaboration in this article on companion experiences for Microsoft Teams.

Looking forward, there will be an unprecedented opportunity for researchers to learn from the current situation to figure out not only how to manage future disruption, but also to incorporate new ways of working at home or in the office. Microsoft is committed to investing in research internally and externally to make this happen. For example, in more typical times, remote work often involves meetings with both remote and co-located colleagues. Better understanding and supporting productivity in these hybrid meetings is the subject of one of the academic projects Microsoft funds through the Microsoft Productivity Research program in collaboration with Dr. Mirjam Augstein and Dr. Thomas Neumayr at the University of Applied Sciences Upper Austria.

If you’d like to do a deeper dive into the literature on remote work, below is a selection of additional papers from Microsoft researchers on the subject.

Researchers at Microsoft have been working across disciplines to address the unique and complex challenges of meeting remotely, such as improving the quality, fidelity, and utility of meetings; addressing design issues; merging physical and virtual collaboration; and exploring the use of avatars:

“Hybrid meetings” (meetings with remote and co-located participants)

Spatialized audio and video for video calling

“Accelerated instant replay” during a video call

Avatars in remote meetings

Remotely collaborating when completing tasks in the physical world

Researchers are also addressing the challenges of remote team building by helping people maintain connections across time zones, welcome new remote team members, and e engage in shared experiences:

Working across time zones

Remote employee onboarding

Co-watching video

Long before remote work became a way of life, researchers at Microsoft exploring the social and technical aspects of collaborating remotely. Below is a selection of work on the subject dating back nearly thirty years:

Microsoft researchers have also made substantial contributions to researcher areas adjacent to remote work that are increasingly relevant in today’s context—in how remote work technologies can support family life and play. For instance:

Connecting family across distance

Remote Play

Video Communication for First Responders

Thanks to Kori Inkpen, Sean Rintel, Abi Sellen, and John Tang, who also contributed to this post.

Episode 109 | March 4, 2020

Over the past decade, the healthcare industry has undergone a series of technological changes in an effort to modernize it and bring it into the digital world, but the call for innovation persists. One person answering that call is Dr. Peter Lee, Corporate Vice President of Microsoft Healthcare, a new organization dedicated to accelerating healthcare innovation through AI and cloud computing.

Today, Dr. Lee talks about how MSR’s advances in healthcare technology are impacting the business of Microsoft Healthcare. He also explains how promising innovations like precision medicine, conversational chatbots and Azure’s API for data interoperability may make healthcare better and more efficient in the future.

Related:

---

Transcript

Peter Lee: In tech industry terms, you know, if the last decade was about digitizing healthcare, the next decade is about making all that digital data good for something, and that good for something is going to depend on data flowing where it needs to flow at the right time.

Host: You’re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I’m your host, Gretchen Huizinga.

Host: Over the past decade, the healthcare industry has undergone a series of technological changes in an effort to modernize it and bring it into the digital world, but the call for innovation persists. One person answering that call is Dr. Peter Lee, Corporate Vice President of Microsoft Healthcare, a new organization dedicated to accelerating healthcare innovation through AI and cloud computing.

Today, Dr. Lee talks about how MSR’s advances in healthcare technology are impacting the business of Microsoft Healthcare. He also explains how promising innovations like precision medicine, conversational chatbots and Azure’s API for data interoperability may make healthcare better and more efficient in the future. That and much more on this episode of the Microsoft Research Podcast.

(music plays)

Host: Peter Lee, welcome to the podcast!

Peter Lee: Thank you. It’s great to be here.

Host: So you’re a Microsoft Corporate Vice President and head of a relatively new organization here called Microsoft Healthcare. Let’s start by situating that within the larger scope of Microsoft Research and Microsoft writ large. What is Microsoft Healthcare, why was it formed, and what do you hope to do with it?

Peter Lee: It’s such a great question because when, we were first asked to take this on, it was confusing to me! Healthcare is such a gigantic business in Microsoft. You know, the number that really gets me is, Microsoft has commercial contracts with almost 169,000 healthcare organizations around the world.

Host: Wow.

Peter Lee: I mean, it’s just massive. Basically, anything from a one-nurse clinic in Nairobi, Kenya, to Kaiser Permanente or United Healthcare, and everything in-between. And so it was confusing to try to understand, what is Satya Nadella thinking to ask a “research-y” organization to take this on? But, you know, the future of healthcare is so vibrant and dynamic right now, and is so dependent on AI, on Cloud computing, big data, I think he was really wanting us to think about that future.

Host: Let’s situate you.

Peter Lee: Okay.

Host: You cross a lot of boundaries from pure to applied research, computer science to medicine. You’ve been head of Carnegie Mellon University’s computer science department, but you were also an office director at DARPA, which is the poster child for applied research. You’re an ACM fellow and on the board of directors of the Allen Institute for AI, but you’re also a member of the National Academy of Medicine, fairly newly minted as I understand?

Peter Lee: Right, just this year.

Host: And on the board of Kaiser Permanente’s School of Medicine. So, I’d ask you what gets you up in the morning, but it seems like you never go to bed… So instead, describe what you do for a living, Peter! How you choose what hat to wear in the morning and what’s a typical day in your life look like?

Peter Lee: Well, you know, this was never my plan. I just love research, and thinking hard about problems, being around other smart people and thinking hard about problems, getting real depth of understanding. That’s what gets me up. But I think the world today, what’s so exciting about it for anyone with the research gene, is that research, in a variety of areas, has become so important to practical, everyday life. It’s become important to Microsoft’s business. Not just Microsoft, but all of our competitors. And so I just feel like I’m in a lucky position, as well as a lot of my colleagues, I don’t think any of us started with that idea. We just wanted to do research and now we’re finding ourselves sort of in the middle of things.

Host: Right. Well, talk a little bit more about computer science and medicine. How have you moved from one to the other, and how do you kind of envision yourself in this arena?

Peter Lee: Well, my joke here is, these were changes that, actually, Satya Nadella forced me to make! And it’s a little bit of a joke because I was actually honored that he would think of me this way, but it was also painful because I was in a comfort zone just doing my own research, leading research teams, and then, you know, Satya Nadella becomes the CEO, Harry Shum comes on board to drive innovation, and I get asked to think about new ways to take research ideas and get them out into the world. And then, three years after that, I get asked to think about the same thing for healthcare. And each one of those, to my mind, are examples of this concept that Satya Nadella likes to talk about, “growth mindset.” I joke that growth mindset is actually a euphemism because each time you’re asked to make these changes, you just get this feeling of dread. You might have a minute where you’re feeling honored that someone would ask you something, but then…

Host: Oh, no! I’ve got to do it now!

Peter Lee: …and boy, I was, you know, on a roll in what I was doing before, and you do spend some time feeling sorry for yourself… but when you work through those moments, you find that you do have those periods in your life where you grow a lot. And my immersion with so many great people in healthcare over the last three or four years has been one of those big growth periods. And to be recognized, then, let’s say, by the National Academies is sort of validation of that.

Host: All right, so rewind just a little bit and talk about that space you were in just before you got into the healthcare situation. You were doing Microsoft Research. Where, on the spectrum from pure, like your Carnegie Mellon roots, to applied, like your DARPA roots, did that land? There’s an organization called NeXT here I think, yeah?

Peter Lee: That’s right. You know, when I was in academia, academia really knows how to do research.

Host: Yeah.

Peter Lee: And they really put the creatives, the graduate students and the faculty, at the top of the pyramid, socially, in the university. It’s just a great setup. And it’s organized into departments, which are each named after a research area or a discipline and within the departments there are groups of people organized by sub-discipline or area, and so it’s an organizing principle that’s tried and true. When I went to DARPA, it was completely different. The departments aren’t organized by research area, they’re organized by mission, some easily assessable goal or objective. You can always answer the question, have we accomplished it yet or not?

Host: Right.

Peter Lee: And so research at DARPA is organized around those missions and that was a big learning experience for me. It’s not like saying we’re going to do computer vision research. We’ll be doing that for the next fifty years. It’s, can we eliminate the language barrier for all internet-connected people? That’s a mission. You can answer the question, you know, how close are we?

Host: Right.

Peter Lee: And so the mix between those two modes of research, from academia to DARPA, is something that I took with me when I joined Microsoft Research and, you know, Microsoft Research has some mix, but I thought the balance could be slightly different. And then, when Satya Nadella became the CEO and Harry Shum took over our division, they challenged me to go bigger on that idea and that’s how NeXT started. NeXT tried to organize itself by missions and it tried to take passionate people and brilliant ideas and grow them into new lines of business, new engineering capabilities for Microsoft, and along the way, create new CVPs and TFs for our company. There’s a tension here because one of the things that’s so important for great research is stability. And so when you organize things like you do in academia, and in large parts of Microsoft Research, you get that stability by having groups of people devoted to an area. We have, for example, say, computer networking research groups that are best in the world.

Host: Right.

Peter Lee: And they’ve been stable for a long time and, you know, they just create more and more knowledge and depth, and that stability is just so important. You feel like you can take big risks when you have that stability. When you are mission-oriented, like in NeXT, these missions are coming and going all the time. So that has to be managed carefully, but the other benefit of that, management-wise, is more people get a chance to step up and express their leadership. So it’s not that either model is superior to the other, but it’s good to have both. And when you’re in a company with all the resources that Microsoft has, we really should have both.

Host: Well, let’s zoom out and talk, somewhat generally, about the promise of AI because that’s where we’re going to land on some of the more specific things we’ll talk about in a bit, but Microsoft has several initiatives under a larger umbrella called AI for Good and the aim is to bring the power of AI to societal-scale problems in things like agriculture, broadband accessibility, education, environment and, of course, medicine. So AI for Health is one of these initiatives, but it’s not the same thing as Microsoft Healthcare, right?

Peter Lee: Well, the whole AI for Good program is so exciting and I’m just so proud to be in a company that makes this kind of commitment. You can think of it as a philanthropic grants program and it is, in fact, in all of these areas, providing funding and technical support to really worthy teams, passionate people, really trying to bring AI to bear for the greater good.

Host: Mm-hmm.

Peter Lee: But it’s also the case that we devote our own research resources to these things. So it’s not just giving out grants, but it’s actually getting into collaborations. What’s interesting about AI for Health is that it’s the first pillar in the AI for Good program that actually overlaps with a business at Microsoft and that’s Microsoft Healthcare. One way that I think about it is, it’s an outlet for researchers to think about, what could AI do to advance medicine? When you talk to a lot of researchers in computer science departments, or across Microsoft research labs, increasingly you’ll see more and more of them getting interested in healthcare and medicine and the first things that they tend to think about, if they’re new to the field, are diagnostic and therapeutic applications. Can we come up with something that will detect ovarian cancer earlier? Can we come up with new imaging techniques that will help radiologists do a better job? Those sorts of diagnostic and therapeutic applications, I think, are incredibly important for the world, but they are not Microsoft businesses. So the AI for Health program can provide an outlet for those types of research passions. And then there are also, as a secondary element, four billion people on this planet today that have no reasonable access to healthcare. AI and technology have to be part of the solution to creating that more equitable access and so that’s another element that, again, doesn’t directly touch Microsoft’s business today in Microsoft Healthcare, but is so important we have a lot to offer so AI for Health is just, I think, an incredibly visionary and wonderful program for that.

Host: Well, let’s zoom back out… um, no, let’s zoom back in. I’ve lost track of the camera. I don’t know where it is! Let’s talk about the idea of precision medicine, or precision healthcare, and the dream of improving those diagnostic and therapeutic interventions with AI. Tell us what precision medicine is and how that plays out and how are the two rather culturally diverse fields of computer science and medicine coming together to solve for X here?

Peter Lee: Yeah, I think one of the things that is sometimes underappreciated is, over the past ten to twenty years, there’s been a massive digitization of healthcare and medicine. After the 2008 economic collapse, in 2009, there was the ARA… there was a piece of legislation attached to that called the HITECH Act, and HITECH actually required healthcare organizations to digitize health records. And so for the past ten years, we’ve gone from something like 15% of health records being in digital form, to today, now over 98% of health records are in digital form. And along with that, medical devices that measure you have gone digital, our ability to sequence and analyze your genome, your proteome, have gone digital and now the question is, what can we do with all the digital information? And on top of that, we have social information.

Host: Yeah.

Peter Lee: People are carrying mobile devices, people talk to computers at home, people go to their Walgreens to get their flu shots.

Host: Yeah.

Peter Lee: And all of this is in digital form and so the question is, can we take all of that digital data and use it to provide highly personalized and precisely targeted diagnostics and therapeutics to people.

Host: Mm-hmm.

Peter Lee: Can we get a holistic, kind of, 360-degree view of you, specifically, of what’s going on with you right now, and what might go on over the next several years, and target your wellness? Can we advance from sick care, which is really what we have today…

Host: Right.

Peter Lee: …to healthcare.

Host: When a big tech company like Microsoft throws its hat in the healthcare ring and publicly says that it has the goal of “transforming how healthcare is experienced and delivered,” I immediately think of the word disruption, but you’ve said healthcare isn’t something you disrupt. What do you mean by that, and if disruption isn’t the goal, what is?

Peter Lee: Right. You know, healthcare is not a normal business. Worldwide, it’s actually a $7.5 trillion dollar business. And for Microsoft, it’s incredibly important because, as we were discussing, it’s gone digital, and increasingly, that digital data, and the services and AI and computation to make good use of the data, is moving to the cloud. So it has to be something that we pay very close attention to and we have a business priority to support that.

Host: Right.

Peter Lee: But, you know, it’s not a normal business in many, many different senses. As a patient, people don’t shop, at least not on price, for their healthcare. They might go on a website to look at ratings of primary care physicians, but certainly, if you’re in a car accident, you’re unconscious. You’re not shopping.

Host: No.

Peter Lee: You’re just looking for the best possible care. And similarly, there’s a massive shift for healthcare providers away from what’s called fee-for-service, and toward something called value-based care where doctors and clinics are being reimbursed based on the quality of the outcomes. What you’re trying to do is create success for those people and organizations that, let’s face it, they’ve devoted their lives to helping people be healthier. And so it really is almost the purest expression of Microsoft’s mission of empowerment. It’s not, how do we create a disruption that allows us to make more money, but instead, you know, how do we empower people and organizations to deliver better – and receive better – healthcare? Today in the US, a primary care doctor spends almost twice as much time entering clinical documentation as they do actually taking care of patients. Some of the doctors we work with here at Microsoft call this “pajama time,” because you spend your day working with patients and then, at home, when you crawl into bed, you have to finish up your documentation. That’s a big source of burn out.

Host: Oh, yeah.

Peter Lee: And so, what can we do, using speech recognition technologies, natural language processing, diarization, to enable that clinical note-taking to be dramatically reduced? You know, how would that help doctors pay more attention to their patients? There is something called revenue-cycle management, and it’s sort of sometimes viewed as a kind of evil way to maximize revenues in a clinic or hospital system, but it is also a place where you can really try to eliminate waste. Today, in the US market, most estimates say that about a trillion dollars every year is just gone to waste in the US healthcare system. And so these are sort of data analysis problems, in this highly complex system, that really require the kind of AI and machine learning that we develop.

Host: And those are the kinds of disruptions we’d like to see, right?

Peter Lee: That’s right. Yeah.

Host: We’ll call them successes, as you did.

Peter Lee: Well, and they are disruptions though, they’re disruptions that help today’s working doctors and nurses. They help today’s hospital administrators.

(music plays)

Host: Let’s talk about several innovations that you’ve actually made to help support the healthcare industry’s transformation. Last year – a year ago – at the HIMSS conference, you talked about tools that would improve communication, the healthcare experience and interoperability and data sharing in the cloud. Tell us about these innovations. What did you envision then, and now, a year later, how are they working out?

Peter Lee: Yeah. Maybe the one I like to start with is about interoperability. I sometimes have joked that it’s the least sexy topic, but it’s the one that is, I think, the most important to us. In tech industry terms, you know, if the last decade was about digitizing healthcare, the next decade is about making all that digital data good for something and that good for something is going to depend on data flowing where it needs to flow…

Host: Right.

Peter Lee: …at the right time. And doing that in a way that protects people’s privacy because health data is very, very personal. And so a fundamental issue there is interoperability. Today, while we have all this digital data, it’s really locked into thousands of different incompatible data formats. It doesn’t get exposed through modern APIs or microservices. It’s oftentimes siloed for business reasons, and so unlocking that is important. One way that we look at it here at Microsoft is, we are seeing a rising tidal wave of healthcare organizations starting to move to the cloud. Probably ten years from now, almost all healthcare organizations will be in the cloud. And so, with that historic shift that will happen only once, ever, in human history, what can we do today to ensure that we end up in a better place ten years from now than we are now? And interoperability is one of the keys there. And that’s something that’s been recognized by multiple governments. The US government, through the Centers for Medicare and Medicaid Services, has proposed new regulations that require the use of specific interoperable data standards and API frameworks. And I’m very proud that Microsoft has participated in helping endorse and guide the specific technical choices in those new rules.

Host: So what is the API that Microsoft has?

Peter Lee: So the data standard that we’ve put a lot of effort behind is something called FHIR. F-H-I-R, Fast Healthcare Interoperability Resources. And for anyone that’s used to working in the web, you can look at FHIR and you’ll see something very familiar. It’s a modern data standard, it’s extensible, because medical science is advancing all the time, and it’s highly susceptible to analysis through machine learning.

Host: Okay.

Peter Lee: And so it’s utterly modern and standardized, and I think FHIR can be a lingua franca for all healthcare data everywhere. And so, for Microsoft, we’ve integrated FHIR as a first-class data type in our cloud, in Azure.

Host: Oh, okay.

Peter Lee: We’ve enabled FHIR in Office. So the Teams application, for example, it can connect to health data for doctors and nurses. And there’s integration going on into Dynamics. And so it’s a way to convert everything that we do here at Microsoft into great healthcare-capable tools. And once you have FHIR in the cloud, then you also, suddenly, unlock all of the AI tools that we have to just enable all that precision medicine down the line.

Host: That’s such a Biblical reference right then! The cloud and the FHIR.

Peter Lee: You know, there are – there’s an endless supply of bad puns around FHIR. So thank you for contributing to that.

Host: Well, it makes me think about the Fyre Festival, which was spelt F-Y-R-E, which was just the biggest debacle in festival history…

Peter Lee: I should say, by the way, another thing that everyone connected to Microsoft should be proud of is, we have really been one of the chief architects for this new future. One of the most important people in the FHIR development community is Josh Mandel, who works with us here at Microsoft Healthcare, and he has the title Chief Architect, but it’s not Chief Architect for Microsoft, it’s Chief Architect for the cloud.

Host: Oh, my gosh.

Peter Lee: So he spends time talking to the folks at Google, at AWS, at Salesforce and so on.

Host: Right.

Peter Lee: Because we’re trying to bring the entire cloud ecosystem along to this new future.

Host: Tell me a little bit about what role bots might play in this arena?

Peter Lee: Bots are really interesting because, how many listeners have received a lab test result and have no idea what it means? How many people have received some weird piece of paper or bill in the mail from their insurance company? It’s not just medical advice, you know, where you have a scratch in your throat and you’re worried about what you should do. That’s important too, but the idea of bots in healthcare really span all these other things. One of the most touching, in a project led by Hadas Bitran and her team, has been in the area of clinical trials. So there’s a website called clinicaltrials.gov and it contains a registry describing every registered clinical trial going on. So now, if you are desperate for more experimental care, or you’re a doctor treating someone and you’re desperate for this, you know, how do you find, out of thousands of documents, and they’re complicated…

Host: Right.

Peter Lee: …technical, medical, science things.

Host: Jargon-y.

Peter Lee: Yeah, and it’s difficult. If you go to clinicaltrials.gov and type into the search box ‘breast cancer’ you get hundreds of results. So the cool project that Hadas and her team led was to use machine reading from Microsoft Research out of Hoifung Poon’s team, to read all of those clinical trials documents and create a knowledge graph and use that knowledge graph then to drive a conversational chatbot so that you can engage in a conversation. So you can say, you know, “I have breast cancer. I’m looking for a clinical trial,” and the chatbot will start to ask you questions in order to narrow down, eventually, to the one or two or three clinical trials that might be just right for you. And so this is something that we just think has a lot of potential.

Host: Yeah.

Peter Lee: And business-wise, there are more mundane, but also important things. Just call centers. Boy, those nurses are busy. What would happen if we had a bot that would triage and tee up some of those things and really give superpowers to those call center nurses. And so it’s that type of thing that I think is very exciting about conversational tech in general. And of course, Microsoft Research and NeXT should be really proud of really pioneering a lot of this bot technology.

Host: Right. So if I employed a bot to narrow down the clinical trials, could I get myself into one? Is that what you’re explaining here?

Peter Lee: Yeah, in fact, the idea here is that this would help, tremendously, the connection between perspective patients and clinical trials. It’s so important because pharmaceutical companies, in clinics that are setting up clinical trials, more than 50% of them fail to recruit enough participants. They just never get off the ground because they don’t get enough. The recruitment problem is so difficult.

Host: Wow.

Peter Lee: And so this is something that can really help on both ends.

Host: I didn’t even think about it from the other angle. Like, getting people in. I always just assumed, well, a clinical trial, no biggie.

Peter Lee: It’s such a sad thing that most clinical trials fail. And fail because of the recruitment problem.

Host: Huh. Well, let’s talk a little bit more about some of the really interesting projects that are going on across the labs here at Microsoft Research. So what are some of the projects and who are some of the people that are working to improve healthcare in technology research?

Peter Lee: Yeah. I think pretty much every MSR lab is doing interesting things. There’s some wonderful work going on in the Cambridge UK lab, in Chris Bishop’s lab there, in a group being led by Aditya Nori. One of the things there has been a set of projects in collaboration with Novartis really looking at new ideas about AI-powered molecule design for cellular therapies, as well as very precise dosing of therapies for things like macular degeneration and so these are, sort of, bringing the very best machine learning and AI researchers shoulder-to-shoulder with the best researchers and scientists at Novartis to really kind of innovate and invent the future. In the MSR India lab, Sriram Rajamani’s team, they’ve been standing up a really impressive set of technologies and projects that have to do with global access to healthcare and this is something that I think is just incredibly, incredibly important. You know, we really could enable, through more intelligent medical devices for example, much less well-trained technicians and clinicians to be able to deliver healthcare at a distance. The other thing that is very exciting to me there is just looking at data. You know, how do we normalize data from lots of different sources?

Host: Right.

Peter Lee: And then MSR Asia in Beijing, they’ve increasingly been redirecting some of the amazing advances that that lab is famous for in computer vision to the medical imaging space. And there are just amazing possibilities in taking images that might not be high resolution enough for a precise diagnosis and using AI to, kind of, magically improve the resolution. And so just across board, you go from, kind of, lab to lab you just see some really inspiring work going on.

Host: Yeah, some of the researchers have been on the podcast. Antonio Criminisi with InnerEye, umm…  haven’t had Ethan Jackson from Premonition yet…

Peter Lee: No, Premonition… Well, Antonio Criminisi and the work that he led on InnerEye, you know, we actually went all the way to an FDA 510(k) approval on the tumor segmentations…

Host: Wow.

Peter Lee: …and the components of that now are going into our cloud. Really amazing stuff.

Host: Yeah.

Peter Lee: And then Premonition, this is one of these things that is, in the age of coronavirus…

Host: Right?

Peter Lee: …is very topical.

Host: I was just going to refer to that, but I thought maybe I shouldn’t…

Peter Lee: The thing that is so important is, we talked of precision medicine before…

Host: Yeah.

Peter Lee: …but there is also an emerging science of precision population health. And in fact, the National Academy of Medicine just recently codified that as an official part of medical research and it’s bringing some of the same sort of precision medicine ideas, but to population health applications and studies. And so when you look at Premonition, and the ability to look at a whole community and get a genetically precise diagnosis of what is going on in that community, it is something that could really be a game-changer, especially in an era where we are seeing more challenging infectious disease outbreaks.

Host: I think a lot of people would say, can we speed that one up a little? I want you to talk for a minute about the broader tech and healthcare ecosystem and what it takes to be a leader, both thought and otherwise, in the field. So you’ve noted that we’re in the middle of a big transformation that’s only going to happen once in history and because of that, you have a question that you ask yourself and everyone who reports to you. So what’s the question that you ask, and how does the answer impact Microsoft’s position as a leader?

Peter Lee: Right. You know, healthcare, in most parts of the world, is really facing some big challenges. It’s at a financial breaking point in almost all developed countries. The spread of the latest access to good medical practice has been slowing in the developing world and as you, kind of, look at, you know, how to break out of these cycles, increasingly, people turn to technology. And the kind of shining beacon of hope is this mountain of digital data that’s being produced every single day and so how can we convert that into what’s called the triple aim of better outcomes, lower costs and better experiences? So then, when you come to Microsoft, you have to wonder, well, if we’re going to try to make a contribution, how do you do it? When Satya Nadella asked us to take this on, we told ourselves a joke that he was throwing us into the middle of the Pacific Ocean and asking us to find land, because it’s such a big complex space, you know, where do you go? And, we had more jokes about this because you start swimming for a while and you start meeting lots of other people who are just as lost and you actually feel a little ashamed to feel good about seeing other people drowning. But it fundamentally it doesn’t help you to figure out what to work on, and so we started to ask ourselves the question, if Microsoft were to disappear today, in what ways would healthcare be harmed or held back tomorrow and into the future? If our hyperscale cloud were to disappear today, in what ways would that matter to healthcare? If all of the AI capabilities that we can deploy so cheaply on that cloud were to disappear, how would that matter? And then, since we’re coming out of Microsoft Research, if Microsoft Research were to disappear today, in what ways would that matter? And asking ourselves that question has sort of helped us focus on the areas where we think we have a right to play. And I think the wonderful thing about Microsoft today is, we have a business model that makes it easy to align those things to our business priorities. And so it’s really a special time right now.

(music plays)

Host: Well, this is – not to change tone really quickly – but this is the part of the podcast where I ask what could possibly go wrong? And since we’ve actually just used a drowning in the sea metaphor, it’s probably apropos… but when you bring nascent AI technologies, and I say nascent because most people have said, even though it’s been going on for a long time, we’re still in an infancy phase of these technologies. When you bring that to healthcare, and you’re literally dealing with life–and–death consequences, there’s not any margin for error. So… I realize that the answer to this question could be too long for the podcast, but I have to ask, what keeps you up at night, and how are you and your colleagues addressing potential negative consequences at the outset rather than waiting for the problems to appear downstream?

Peter Lee: That’s such an important question and it actually has multiple answers. Maybe the one that I think would be most obvious to the listeners of this podcast has to do with patient safety. Medical practice and medical science has really advanced on the idea of prospective studies and clinical validation, but that’s not how computer science, broadly speaking, works. In fact, when we’re talking about machine learning it’s really based on retrospective studies. You know, we take data that was generated in the past and we try to extract a model through machine learning from it. And what the world has learned, in the last few years, is that those retrospective studies don’t necessarily hold up very well, prospectively. And so that gap is very dangerous. It can lead to new therapies and diagnoses that go wrong in unpredictable ways, and there’s sort of an over-exuberance on both sides. As technologists, we’re pretty confident about what we do and we see lots of problems that we can solve, and the healthcare community is sometimes dazzled by all of the magical machine learning we do and so there can be over-confidence on both sides. That’s one thing that I worry about a lot because, you know, all over our field, not just all over Microsoft, but across all the other major tech companies and universities, there are just great technologists that are doing some wonderful things and are very well-intentioned, but aren’t necessarily validated in the right way. And so that’s something that, really, is worrisome. Going along with safety is privacy of people’s health data. And while I think most people would be glad to donate their health data for scientific progress, no one wants to be exploited. Exploited for money, or worse, you know, denied, for example, insurance.

Host: Right.

Peter Lee: And you know, these two things can really lead to outcomes, over the next decade, that could really damage our ability to make good progress in the future.

Host: So that said, we’re pretty good at identifying the problem. We may be able to start a good “conversation,” air quotes, on that, but this is, for me, like, what are you doing?

Peter Lee: Yeah.

Host: Because this is a huge thing, and…

Peter Lee: I really think, for real progress and real transformation, that the foundations have to be right and those foundations do start with this idea of interoperability. So the good thing is that major governments, including the US government, are seeing this and they are making very definitive moves to foster this interoperable future. And so now, our role in that is to provide the technical guidance and technologies so that that’s done in the right way. And so everything that we at Microsoft are doing around interoperability, around security, around identity management, differential privacy, all of the work that came out of Microsoft Research in confidential computing…

Host: Yeah.

Peter Lee: …all of those things are likely to be part of this future. As important as confidential computing has been as a product of Microsoft Research, it’s going to be way, way more important in this healthcare future. And so it’s really up to us to make sure that regulators and lawmakers and clinicians are aware and smart about these things. And we can provide that technical guidance.

Host: What about the other companies that you mentioned? I mean, you’re not in this alone and it’s not just companies, it’s nations, and, I dare say, rogue actors, that are skilled in this arena. How do you get, sort of, agreement and compliance?

Peter Lee: I would say that Microsoft is in a good position because it has a clear business model. If someone is asking us, well what are you going to with our data? We have a very clear business model that says that we don’t monetize on your data.

Host: Right.

Peter Lee: But everyone is going to have to figure that out. Also, when you are getting into a new area like healthcare, every tech company is a big, complicated place with lots of stakeholders, lots of competing internal interests, lots of politics.

Host: Right.

Peter Lee: And so Microsoft, I think, is in a very good position that way too. We’re all operating as one Microsoft. But it’s so important that we all find ways to work together. One point of contact has been engineered by the White House in something called the Blue Button Developers Conference. So that’s where I’m literally holding hands with my counterparts at Google, at Salesforce, at Amazon, at IBM, making certain pledges there. And so the convening power of governments is pretty powerful.

Host: It’s story time. We’ve talked a little about your academic and professional life. Give us a short personal history. Where did it all start for Peter Lee and how did he end up where he is today?

Peter Lee: Oh, my.

Host: Has to be short.

Peter Lee: Well, let’s see, so uh, I’m Korean by heritage. I was born in Ohio, but Korean by heritage and my parents immigrated from Korea. My dad was a physics professor. He’s long retired now and my mother a chemistry professor.

Host: Wow.

Peter Lee: And she passed away some years ago. But I guess as an Asian kid growing up in a physical science household, I was destined to become a scientist myself. And in fact, they never said it out loud, but I think it was a disappointment to them when I went to college to study math! And then maybe an even the bigger disappointment when I went from math to computer science in grad school. Of course they’re very proud of me now.

Host: Of course! Where’d you go to school?

Peter Lee: I went to the University of Michigan. I was there as an undergrad and then I was planning to go work after that. I actually interviewed at a little, tiny company in the Pacific Northwest called Microsoft…

Host: Back then!

Peter Lee: … and …but I was wooed by my senior research advisor at Michigan to stay on for my PhD and so I stayed and then went from grad school right to Carnegie Mellon University as a professor.

Host: And then worked your way up to leading the department…

Peter Lee: Yeah. So I was there for twenty four years. They were wonderful years. Carnegie Mellon University is just a wonderful, wonderful place. And um..

Host: It’s almost like there’s a pipeline from Microsoft Research to Carnegie Mellon. Everyone is CMU this, CMU that!

Peter Lee: Well, I remember, as an assistant professor, when Rick Rashid came to my office to tell me that he was leaving to start this thing called Microsoft Research and I was really sad and shocked by that. Now here I am!

Host: Right. Well, tell us, um, if you can, one interesting thing about you that people might not know.

Peter Lee: I don’t know if people know this or not, but I have always had an interest in cars, in fast cars. I spent some time, when I was young, racing in something called shifter karts and then later in open wheel Formula Ford, and then, when I got my first real job at Carnegie Mellon, I had enough money that I spent quite a bit of it trying to get a sponsored ride with a semi-pro team. I never managed to make it. It’s hard to kind of split being an assistant professor and trying to follow that passion. You know, I don’t do that too much anymore. Once you are married and have a child, the annoyance factor gets a little high, but it’s something that I still really love and there’s a community of people, of course, at a place like Microsoft, that’s really passionate about cars as well.

Host: As we close, Peter, I’d like you to leave our listeners with some parting advice. Many of them are computer science people who may want to apply their skills in the world of healthcare, but are not sure how to get there from here. Where, in the vast sea of technology and healthcare research possibilities, should emerging researchers set their sights and where should they begin their swim?

Peter Lee: You know, I think it’s all about data and how to make something good out of data. And today, especially, you know, we are in that big sea of data silos. Every one of them has different formats, different rules, most of them don’t have modern APIs. And so things that can help evolve that system to a true ocean of data, I think anything to that extent will be great. And it is not just tinkering around with interfaces. It’s actually AI. To, say, normalize the schemas of two different data sets, intelligently, is something that we will need to do using the, kind of, latest machine learning, latest program synthesis, the kind of, latest data science techniques that we have on offer.

Host: Who do you want on your team in the coming years?

Peter Lee: The thing that I think I find so exciting about great researchers today is their intellectual flexibility to start looking at an idea and getting more and more depth of understanding, but then evolve as a person to understanding, you know, what is the value of this in the world, and understanding that that is a competitive world. And so, how willing are you to compete in that competitive marketplace to make the best stuff? And that evolution that we are seeing over and over again with people out of Microsoft Research is just incredibly exciting. When you see someone like a Galen Hunt or a Doug Burger or a Lili Cheng come out of Microsoft Research and then evolve into these world leaders in their respective fields, not just in research, but spanning research to really competing in a highly competitive marketplace, that is the future.

Host: Peter Lee, thank you for joining us on the podcast today. It’s been an absolute delight.

Peter Lee: Thank you for having me. It’s been fun.

(music plays)

To learn more about Dr. Peter Lee and how Microsoft is working to empower healthcare professionals around the world, visit Microsoft.com/research

Cecily Morrison wants to build technology that enables people to live their lives the way they want and accomplish things they wouldn’t otherwise be able to.

The Principal Researcher landed on the personal mission years before joining Microsoft Research. Experiences first teaching children robotics and then working with the United Kingdom’s National Health Service applying existing technology to healthcare scenarios ignited the desire. Since then, she’s helped develop a system for monitoring the progression of multiple sclerosis in patients and a physical programming language for children who are blind or have low vision. She’s currently working on visual agent technology, including a new project in which she’ll explore computer vision and spatialized audio to help children born blind develop social and learning skills that children with sight cultivate through visual cues.

“I am passionate about demonstrating in the real how AI can fundamentally change people’s lives,” says Morrison, who is collaborating on the work with Oussama Metatla of the University of Bristol in the United Kingdom.

Advancing research like Morrison’s through collaboration and strong relationships between Microsoft Research Cambridge researchers and academia is at the core of the Microsoft Research PhD Scholarship Programme in EMEA (Europe, the Middle East, and Africa). Morrison and Metatla’s “Using AI to Develop Joint Attention in Blind Children” is one of 15 projects that have been selected this year to receive the scholarship, which provides financial support for up to three years. Their work joins the more than 400 projects since the program’s inception in 2004 that have helped drive innovation in the lab’s key research areas: All Data AI, Cloud Infrastructure, Confidential Computing, Future of Work, Game Intelligence, Healthcare Intelligence, and Biological Computation. This year’s other projects include conversational user interfaces for mental health status reporting, failure detection in machine learning models, and learning local forward models in complex 3D games.

The value of collaboration

As part of the program selection process, Cambridge researchers invite a PhD supervisor at an EMEA institution to collaborate. Often, they have a preexisting relationship with the supervisor or an interest in working with them. Together, the pair writes and submits a proposal. Proposals are reviewed and selected by Microsoft researchers in a two-stage review process. The researchers and supervisors of the selected proposals then choose a PhD student to work on the project.

The collaborative nature of the program supports all parties involved in important ways. The program, for example, gives researchers a chance to conduct more exploratory research than they might normally be able to undertake in their day to day and to draw on the unique perspectives and knowledge of and students. PhD supervisors  and students—encouraged to attend meetings at Cambridge’s lab, where they get to experience firsthand the breadth and depth of research at Microsoft—are introduced to such professional opportunities as visiting researcher positions for faculty and internships for students.

The program also allows for resource sharing. PhD supervisors and students can benefit from Microsoft technology and computational power such as cloud services while academia offers facilities and equipment researchers don’t have access to.

“I see this as a way to do longer-term research than we can do in Microsoft Research, but that couldn’t be done in academia because they don’t have access to the technology,” says Morrison.

For Microsoft Principal Scientist Sara-Jane Dunn of the Biological Computation Group, who was selected to the program this year with Graziano Martello of the University of Padua, Italy, working with PhD students is particularly valuable.

“Being able to co-supervise brilliant young researchers is rewarding in many ways: being able to foster new research, mentor talent, develop new ideas, and learn new techniques,” she says. “It’s invaluable as a seasoned scientist.”

Dunn and Martello’s project, “The Pluripotency Program in Human Embryonic Stem Cells,” builds on the pair’s work in stem cell research. They’ll focus on how personalized stem cells are generated, a greater understanding of which could help inform new medical diagnostics and treatments.

Over the next several months, selected researchers and PhD supervisors will be recruiting students for their projects. For more information or to apply, visit the program home page. Positions will be posted as they become available.

EMEA PhD Award

As a complement to the EMEA PhD Scholarship Programme, Microsoft Research is excited to announce the Microsoft Research EMEA PhD Award, a new research grant for PhD students in computing-related fields who are in their third year or beyond at universities in EMEA.

Award recipients will receive the following:

• $15,000 to put toward their doctoral thesis work for the upcoming academic year
• an invitation, including travel and accommodations, to attend the two-day Microsoft Research PhD Summit workshop in North America, where they will present their work and be mentored by Microsoft researchers
• an offer to intern at the Cambridge lab

Applications are due by 11:59 UTC on April 1. To find out more, including how to apply, visit the EMEA PhD Award home page.

Research is about achieving long-term goals, often through incremental progress. As the year comes to an end, it’s a good time to step back and reflect on the work that researchers at Microsoft and their collaborators have done to advance the state of the art in computing, particularly by increasing the capabilities and reach of AI and delivering technology experiences that are more inclusive, secure, and accessible. This covers only a sliver of all the amazing work Microsoft Research has accomplished this year, and we encourage you to discover more of the hundreds of projects undertaken in 2019 by exploring our blog further.

Improving the reach and accessibility of AI and machine learning

Machine learning has made a tremendous impact on people’s everyday lives, especially in the latter half of this decade, while also raising significant policy and societal issues for research to address. This year, Microsoft researchers and their collaborators worked to improve the capabilities of machine learning systems and also explored new models that can take the discipline further. They used unique approaches that can make these systems more accessible and inclusive.

In deep learning, Jianfeng Gao’s team released MT-DNN, a model for learning universal language embeddings that combines the strengths of multi-task learning and the language model pre-training of BERT, helping systems quickly develop the semantic understanding necessary for natural language processing. And Xu Tan and his collaborators at Microsoft Research Asia developed MASS, a pre-training method that outperforms existing models at sequence-to-sequence language generation.

In the coming years, breakthroughs in machine learning will emerge by exploring new models beyond the current foundation of using Markov decision processes, particularly as reinforcement learning—a data-hungry approach generally suited to simulation scenarios—becomes more applicable to real-world scenarios. In this podcast, John Langford and Rafah Hosn discuss these new directions in reinforcement learning and their applications to everyday computing, including the real-world RL now deployed in Personalizer, an Azure Cognitive Service. Langford and Alekh Agarwal also hosted a webinar on the foundations of real-world reinforcement learning.

Many machine learning applications benefit from training with very large datasets; however, many potential uses simply do not have enough data for typical approaches to be effective. Enter machine teaching, where domain experts can build bespoke AI models with little data—and no machine learning expertise. In this podcast, Riham Mansour discusses (among other things) LUIS, one of the first Microsoft products to deploy machine teaching concepts in real-world scenarios.

Another project aimed at further democratizing AI is the Decentralized & Collaborative AI on Blockchain framework with Justin Harris, which enables users to train and maintain models and datasets on the Ethereum network. At NeurIPS 2019, Debadeepta Dey and collaborators presented Project Petridish, an efficient forward neural architecture search algorithm that helps identify suitable neural architectures for a given machine learning task. And Adith Swaminathan and Emre Kiciman’s February blog post explores researchers’ work to improve causal inference modeling, which helps AI better understand “what if” scenarios in a wide variety of contexts.

Enabling responsible, inclusive, human-centered innovation

2019 kicked off with the inaugural ACM FAT* Conference in Atlanta, which focused on fairness, accountability, and transparency in socio-technical systems. Microsoft Research presented four papers at the conference. They covered gender bias in occupation classification, the role of data-driven decision making in reinforcing or amplifying injustices, strategic manipulation of algorithmic decision systems, and the fair allocation of items in scenarios without money, respectively. This work came from the FATE research group at Microsoft, which studies the complex social implications of AI, machine learning, data science, large-scale experimentation, and automation.

At May’s CHI Conference on Human Factors in Computing Systems, Saleema Amershi and her collaborators presented a set of guidelines for human-AI interaction design that brings together more than 20 years of research, recommendations, and best practices around effective interaction with AI-infused systems. Bringing this work together will help designers manage user expectations, moderate the level of autonomy, resolve ambiguity, and provide users with awareness of how systems learn from their behavior.

To ensure that machine learning systems effectively do the jobs we deploy them to do, we must develop a deeper understanding of how they succeed and fail. This paper, from researchers at Microsoft Ram Shankar Siva Kumar and Jeffrey Snover and their collaborators at Harvard, articulates the various ways machine learning systems can fail—either through intentional adversarial attacks or unintentional failures in which the output is formally correct, but unwanted.

Helping to train autonomous systems that can be trusted in real-world applications, the open-source simulator AirSim provides realistic and detailed testing environments. This year, it played host to the NeurIPS competition Game of Drones. In the drone race challenge, participants competed against a Microsoft Research opponent on the same track, working with a level of strategy and maneuvering not generally offered by contests of its kind. Microsoft researchers and collaborators who organized the competition plan to keep it open and add new racing environments. Visit the GitHub repository for more information.

In January, Jenn Wortman Vaughan and Hanna Wallach hosted a webinar on Fairness in Machine Learning, demonstrating how to make detecting and mitigating biases a first-order priority in the development and deployment of machine learning systems.

Creating human-computer interaction that works for all

This year, Microsoft researchers continued their work to make computing more natural, comfortable, and accessible for everyone. At the ACM CHI Conference on Human Factors in Computing Systems, researchers presented a number of papers and demos exploring how to support accessibility for users with cognitive or sensory disabilities. These include studies on whether web browsers’ “reading mode” is truly helpful for people with dyslexia and tools to help VR be more accessible for people with low vision (including tunnel vision, brightness sensitivity, and low visual acuity).

Also presented at CHI was Microsoft Soundscape, a project that uses 3D audio cues to enhance situational awareness and assist with navigation. (You can try the app yourself here.) In this op-ed in the Toronto Sun, Microsoft researcher Bill Buxton elaborates on the importance of work like this, noting that 1 billion people worldwide have some form of disability, making it imperative that we create technologies that support personal autonomy.

Speaking of sound, Nikunj Raghuvanshi’s podcast explores the physics of audio and discusses Project Triton, an acoustic system that models how sound waves behave so that the audio in 3D game environments can be as rich and immersive as the graphics. Project Triton is available for any game via the Unity and Unreal game engine plugins, as part of Project Acoustics, powered by Azure.

At the ACM Symposium on User Interface Software and Technology, Microsoft researchers presented a number of projects that make virtual environments more realistic, tactile, and navigable. Dreamwalker is a VR project that can augment a real-world walking experience with virtual reality—a virtual environment that detects the user’s surroundings in real time and generates a virtual world that accounts for their path and any obstacles—so that you can walk to work in Seattle, but through a virtual Manhattan. Mise-Unseen is a project that uses gaze detection to modify or replace elements of a virtual world while the user’s attention is directed elsewhere. And CapstanCrunch is a VR controller that leverages centuries-old technology, once used to control ropes on sailing ships, to provide effective and inexpensive haptic feedback.

Meanwhile in the physical world, Microsoft researchers partnered in May with students at the Brooklyn Public Library’s Fashion Academy to weave technology into their designs using Project Alava, which aims to develop microcontroller-based systems that are simple to build and code for people with a limited computer science background. At their end-of-program fashion show, students showed garments that incorporated LEDs, motion sensors, and motors. You can read about other areas where Microsoft researchers are working at the intersection of art and science here, including Ada, a first-of-its-kind architectural pavilion that incorporates AI, on display at Microsoft Research Redmond.

Breakthroughs in security, storage, systems, and applications

2019 saw continued progress in the development and adoption of homomorphic encryption, which enables computation on encrypted data, helping to preserve privacy. Microsoft SEAL has become one of the world’s most popular homomorphic encryption libraries, with broad adoption in both academia and industry. In February, Microsoft took the next step in democratizing homomorphic encryption by releasing SEAL for .NET. (The Microsoft SEAL library is open source and available on GitHub.)

In August 2019, Microsoft researchers joined their industry and academic peers for the Homomorphic Encryption Standards Meeting. The group will reconvene at Microsoft Research in Redmond for next year’s meeting this February. Take our webinar to learn more about homomorphic encryption, and listen to this October podcast with Craig Costello for an overview of the year’s developments in cryptography generally, including efforts to prepare for a post-quantum future.

In April, Project Everest took another step forward in its work to build a verified, secure HTTPS ecosystem with the release of EverCrypt, the first fully verified cryptographic provider to meet the security needs of the TLS protocol. Project Everest is a collaboration between Microsoft, Inria, and Carnegie Mellon University.

By 2023, it’s expected that more than 100 zettabytes of data will be stored in the cloud. To meet that need, Project Silica is developing the first-ever storage technology designed from the media up for use in the cloud. This year, the team collaborated with Warner Bros. on a proof of concept, storing the 1978 film Superman on a nearly indestructible piece of glass roughly the size of a drink coaster. This work is part of the Optics for the Cloud Research Alliance, which you can learn more about here or on the Microsoft Research Podcast. Meanwhile, researchers at Microsoft and the University of Washington achieved a “Hello, World!” moment in April for a different way to meet our growing storage needs: They demonstrated the first fully automated system to store and retrieve data in manufactured DNA. (For more on the intersection between computing and biology, listen to this podcast featuring Andrew Phillips, who leads the Biological Computation Group at Microsoft Research Cambridge.)

Cambridge researchers Andy Gordon and Simon Peyton Jones demonstrated the practical impact of fundamental research by exploring how ideas from programming language research could improve one of the world’s most common business applications: the spreadsheet. In this January blog post, they detail how their collaboration with the Microsoft Excel team led to product improvements such as cells that can contain first-class records linked to external data sources and formulas that can compute array values that “spill” into adjacent cells.

At the ACM International Conference on Web Search and Data Mining, Microsoft researchers presented new work in extreme classification, a research area that promises to dramatically improve the speed and quality of algorithms that can answer multiple-choice questions involving uncertainty, where there could be multiple correct answers. Among other things, this work can lead to more relevant recommendations and search results. In this blog post from February, Manik Varma of Microsoft Research India provides a deep dive into extreme classification.

Thanks to gains in computer vision, particularly object detection and classification, video analysis has become far more accurate; however, fast and affordable real-time video analysis is lagging. In December, Microsoft researchers Ganesh Ananthanarayanan and Yuanchao Shu hosted a webinar on Project Rocket, an extensible software stack that leverages the edge and cloud to meet the needs of video analytic applications.

In April, the Microsoft Research Podcast turned its attention to databases—particularly the need for imperative programming that allows for good software engineering practices like modularity, readability, and reusability. In this episode, Karthik Ramachandra discusses Froid, an extensible and language-agnostic framework for imperative functions in databases, which is available as “Scalar UDF Inlining” in Microsoft SQL Server 2019.

Open-source tools and data for the research community

Throughout the year, researchers from Microsoft made a number of projects open source for the benefit of the academic community, including the following:

• • SandDance, a data visualization tool included in Azure Data Studio, Visual Studio Code, and Power BI
  • TensorWatch, an AI debugging and visualization tool
  • PhoneticMatching, a component of Maluuba’s natural language understanding platform
  • SpaceFusion, a learning paradigm that brings together a palette of different deep learning models for conversational AI
  • Icecaps, a toolkit for conversation modeling
  • Icebreaker, a deep generative model that minimizes the amount and cost of data required to train a machine learning model

Building on last year’s announcement of Microsoft Research Open Data—an Azure-based repository for sharing datasets—the company developed a set of data use agreements, released them on GitHub, and adopted them for a number of public datasets. This work aims to make research data more readily available in the cloud and to encourage the reproducibility of research.

Supporting and honoring the research community

This year, Microsoft Research introduced the Ada Lovelace Fellowship to support diverse talent from underrepresented groups pursuing doctorates in computing-related fields. You can read about the fellows and their research here. Ten doctoral students were also awarded two-year fellowships as part of the PhD Fellowship program, supporting research in photonics, systems and networking, and AI. Additionally, Microsoft Research awarded Microsoft Research Faculty Fellowships to five early-career faculty members pursuing high-impact breakthrough research. You can read about their work here.

A number of researchers at Microsoft received awards and honors throughout 2019—check out the full list of recipients here.

Finally, we are saying goodbye to Harry Shum, who is leaving the company in February after 23 years, and hello to Microsoft CTO and EVP Kevin Scott, who has assumed Shum’s responsibilities as head of the Microsoft Artificial Intelligence and Research Group. Listen to Scott on the Microsoft Research Podcast here.

We hope you had a good year, and we look forward to a 2020 full of collaboration and exciting breakthroughs. Happy holidays.

To stay up to date on all things research at Microsoft, follow our blog and subscribe to our newsletter and the Microsoft Research Podcast. You can also follow us on Facebook, Twitter, YouTube, and Instagram.

Drone racing has transformed from a niche activity sparked by enthusiastic hobbyists to an internationally televised sport. In parallel, computer vision and machine learning are making rapid progress, along with advances in agile trajectory planning, control, and state estimation for quadcopters. These advances enable increased autonomy and reliability for drones. More recently, the unmanned aerial vehicle (UAV) research community has begun to tackle the drone-racing problem. This has given rise to competitions, with the goal of beating human performance in drone racing.

At the thirty-third Conference on Neural Information Processing Systems (NeurIPS 2019), the AirSim research team is working together with Stanford University and University of Zurich to further democratize drone-racing research by hosting a simulation-based competition, Game of Drones. We are hosting the competition on Microsoft AirSim, our Unreal Engine-based simulator for multirotors. The competition focuses on trajectory planning and control, computer vision, and opponent drone avoidance. This is achieved via three tiers:

• Tier 1 – Planning only: The participant’s drone races tête-à-tête with a Microsoft Research opponent racer. The goal is to go through all gates in the minimum possible time, without hitting the opponent drone. Ground truth for gate poses, the opponent drone pose, and the participant drone are provided. These are accessible via our application-programming interfaces (APIs). The opponent racer follows a minimum jerk trajectory, which goes through randomized waypoints selected in each gate’s cross section.
• Tier 2 – Perception only: This is a time trial format where the participants are provided with noisy gate poses. There’s no opponent drone. The next gate will not always be in view, but the noisy pose returned by our API will steer the drone roughly in the right direction, after which vision-based control would be necessary.
• Tier 3 – Perception and Planning: This combines Tier 1 and 2. Given the ground truth state estimate for participant drone and noisy estimate for gates, the goal is to race against the opponent racer without colliding with it.

The animation on the left below shows the ground truth gate poses (Tier 1), while the animation on the right shows the noisy gate poses (Tier 2 and Tier 3). In each animation, the drone is tracking a minimum jerk trajectory using one of our competition APIs.

The following animation shows a segment of one of our racing tracks with two drones racing against each other. Here “drone_2” (pink spline) is the opponent racer going through randomized waypoints in each gate cross section, while “drone_1” (yellow spline) is a representative competitor going through the gate centers.

The competition is being run in two stages—an initial qualification round and a final round. A set of training binaries with configurable racetracks was made available to the participants initially, for prototyping and verification of algorithms on arbitrary racetracks. In the qualification stage (Oct 15^th to Nov 21^st), teams were asked to submit their entries for a subset or all of the three competition tiers.  117 teams registered for the competition worldwide, with 16 unique entries that have shown up on the qualification leaderboard.

We are now running the final round of the competition and the corresponding leaderboard is available here. All of the information for the competition is available at our GitHub repository, along with the training, qualification, and final race environments.

Engineering-wise, we introduced some new APIs in AirSim specifically for the competition, and we’re continually adding more features as we get feedback. We highlight the main components below:

In the long term, we intend to keep the competition open, and we will be adding more racing environments after NeurIPS 2019. While the first iteration brought an array of new features to AirSim, there are still many essential ingredients for trustable autonomy in real-world scenarios and effective simulation-to-reality transfer of learned policies. These include reliable state estimation; camera sensor models and motion blur; robustness to environmental conditions like weather, brightness, and diversity in texture and shape of the drone racing gates; and robustness against dynamics of the quadcopter. Over the next iterations, we aim to extend the competition to focus on these components of autonomy as well.

For more of the exciting work Microsoft is doing with AirSim, see our blog post on Ignite 2019.

Acknowledgements: This work would not have been possible without the substantial team effort behind the scenes by all members of the organizing team—Ratnesh Madaan, Nicholas Gyde, Keiko Nagami, Matthew Brown, Sai Vemprala, Tim Taubner, Eric Cristofalo, Paul Stubbs, Jim Piavis, Guada Casuso, Mac Schwager, Davide Scaramuzza, and Ashish Kapoor.

Last summer we announced Microsoft Research Open Data—an Azure-based repository-as-a-service for sharing datasets—to encourage the reproducibility of research and make research data assets readily available in the cloud. Among other things, the project started a conversation between the community and Microsoft’s legal team about dataset licensing. Inspired by these conversations, our legal team developed a set of brand new data use agreements and released them for public comment on Github earlier this year.

Today we’re excited to announce that Microsoft Research Open Data will be adopting these data use agreements for several datasets that we offer.

Diving a bit deeper on the new data use agreements

The Open Use of Data Agreement (O-UDA) is intended for use by an individual or organization that is able to distribute data for unrestricted uses, and for which there is no privacy or confidentiality concern. It is not appropriate for datasets that include any data that might include materials subject to privacy laws (such as the GDPR or HIPAA) or other unlicensed third-party materials. The O-UDA meets the open definition: it does not impose any restriction with respect to the use or modification of data other than ensuring that attribution and limitation of liability information is passed downstream. In the research context, this implies that users of the data need to cite the corresponding publication with which the data is associated. This aids in findability and reusability of data, an important tenet in the FAIR guiding principles for scientific data management and stewardship.

We also recognize that in certain cases, datasets useful for AI and research analysis may not be able to be fully “open” under the O-UDA. For example, they may contain third-party copyrighted materials, such as text snippets or images, from publicly available sources. The law permits their use for research, so following the principle that research data should be “as open as possible, as closed as necessary,” we developed the Computational Use of Data Agreement (C-UDA) to make data available for research while respecting other interests. We will prefer the O-UDA where possible, but we see the C-UDA as a useful tool for ensuring that researchers continue to have access to important and relevant datasets.

Datasets that reflect the goals of our project

The following examples reference datasets that have adopted the Open Use of Data Agreement (O-UDA).

Location data for geo-privacy research

Microsoft researcher John Krumm and collaborators collected GPS data from 21 people who carried a GPS receiver in the Seattle area. Users who provided their data agreed to it being shared as long as certain geographic regions were deleted. This work covers key research on privacy preservation of GPS data as evidenced in the corresponding paper, “Exploring End User Preferences for Location Obfuscation, Location-Based Services, and the Value of Location,” which was accepted at the Twelfth ACM International Conference on Ubiquitous Computing (UbiComp 2010). The paper has been cited 147 times, including for research that builds upon this work to further the field of preservation of geo-privacy for location-based services providers.

Hand gestures data for computer vision

Another example dataset is that of labeled hand images and video clips collected by researchers Eyal Krupka, Kfir Karmon, and others. The research addresses an important computer vision and machine learning problem that deals with developing a hand-gesture-based interface language. The data was recorded using depth cameras and has labels that cover joints and fingertips. The two datasets included are FingersData, which contains 3,500 labeled depth frames of various hand poses, and GestureClips, which contains 140 gesture clips (100 of these contain labeled hand gestures and 40 contain non-gesture activity). The research associated with this dataset is available in the paper “Toward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines,” which was published in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems.

Question-Answer data for machine reading comprehension

Finally, the FigureQA dataset generated by researchers Samira Ebrahimi Kahou, Adam Atkinson, Adam Trischler, Yoshua Bengio and collaborators, introduces a visual reasoning task for research that is specific to graphical plots and figures. The dataset has 180,000 figures with 1.3 million question-answer pairs in the training set. More details about the dataset are available in the paper “FigureQA: An Annotated Figure Dataset for Visual Reasoning” and corresponding Microsoft Research Blog post. The dataset is pivotal to developing more powerful visual question answering and reasoning models, which potentially improve accuracy of AI systems that are involved in decision making based on charts and graphs.

The data agreements are a part of our larger goals

Microsoft Research Open Data project was conceived from the start to reflect Microsoft Research’s commitment to fostering open science and research and to achieve this without compromising the ethics of collecting and sharing data. Our goal is to make it easier for researchers to maintain provenance of data while having the ability to reference and build upon it.

The addition of the new data agreements to Microsoft Research Open Data’s feature set is an exciting step in furthering our mission.

Acknowledgements: This work would not have been possible without the substantial team effort by — Dave Green, Justin Colannino, Gretchen Deo, Sarah Kim, Emily McReynolds, Mario Madden, Emily Schlesinger, Elaine Peterson, Leila Stevenson, Dave Baskin, and Sergio Loscialo.

With inputs from: Elizabeth Bondi (Harvard University), Bob DeBortoli (Oregon State University), Balinder Malhi (Microsoft) and Jim Piavis (Microsoft)

Autonomous systems have the potential to improve safety for people in dangerous jobs, particularly first responders. However, deploying these systems is a difficult task that requires extensive research and testing.

In April, we explored complexities and challenges present in the development of autonomous systems and how technologies such as AirSim provide a pragmatic way to solve these tasks. Microsoft believes that the key to building robust and safe autonomous systems is providing a system with a wide range of training experiences to properly expose it to many scenarios before it can be deployed in the real world. This ensures training is done in a meaningful way—similar to how a student might be trained to tackle complex tasks through a curriculum curated by a teacher.

With autonomous systems, first responders gain sight into the unknown

One way Microsoft trains autonomous systems is through participating in unique research opportunities focused on solving real-world challenges, like aiding first responders in hazardous scenarios. Recently, our collaborators at Carnegie Mellon University and Oregon State University, collectively named Team Explorer, demonstrated technological breakthroughs in this area during their first-place win at the first round of the DARPA Subterranean (SubT) Challenge.

The DARPA SubT Challenge aspires to further the technologies that would augment difficult operations underground. Specifically, the challenge focuses on the methods to map, navigate, and search complex underground environments. These underground environments include human-made tunnel systems, urban underground, and natural cave networks. Imagine constrained environments that are several kilometers long and structured in unique ways with regular or irregular geological topologies and patterns. Weather or other hazardous conditions, due to poor ventilation or poisonous gasses, often make first responders’ work even more dangerous.

Team Explorer engaged in autonomous search and detection of several artifacts within a man-made system of tunnels. The end-to-end solution that the team created required many different complex components to work across the challenging circuit including mobility, mapping, navigation, and detection.

Microsoft’s Autonomous Systems team worked closely with Team Explorer to provide a high-definition simulation environment to help with the challenge. The team used AirSim to create an intricate maze of man-made tunnels in a virtual world that was representative of such real-world tunnels, both in complexity as well as size. The virtual world was a hybrid synthesis, where a team of artists used reference material from real-world mines to modularly generate a network of interconnected tunnels spanning two kilometers in length spread over a large area.

Additionally, the simulation included robotic vehicles—wheeled robots as well as unmanned aerial vehicles (UAVs)—and a suite of sensors that adorned the autonomous agents. AirSim provided a rich platform that Team Explorer could use to test their methods along with generate training experiences for creating various decision-making components for the autonomous agents.

At the center of the challenge was the ability for the robots to perceive the underground terrain and discover things (such as human survivors, backpacks, cellular phones, fire extinguishers, and power drills) while adjusting to different weather and lighting conditions. Multimodal perception is important in challenging environments, as well as AirSim’s ability to simulate a wide variety of sensors, and their fusion can provide a competitive edge. One of the most important sensors is a LIDAR, and in AirSim, the physical process of generating the point clouds are carefully reconstructed in software, so the sensor used on the robot in simulation uses the same configuration parameters (such as number-of-channels, range, points-per-second, rotations-per-second, horizontal/vertical FOVs, and more) as those found on the real vehicle.

It is challenging to train perception modules based on deep learning models to detect the target objects using LIDAR point clouds and RGB cameras. While curated datasets, such as ScanNet and MS COCO, exist for more canonical applications, none exist for underground exploration applications. Creating a real dataset for underground environments is expensive because a dedicated team is needed to first deploy the robot, gather the data, and then label the captured data. Microsoft’s ability to create near-realistic autonomy pipelines in AirSim means that we can rapidly generate labeled training data for a subterranean environment.

Detecting animal poaching through drone simulations

With autonomous systems, the issues with collection data are further exacerbated for applications that involve first line responders since the collection process is itself dangerous. Such challenges were present in our collaboration with Air Shepherd and USC to help counter wildlife poaching.

The central task in this collaboration was the development of UAVs equipped with thermal infrared cameras that can fly through national parks at night to search for poachers and animals. The project had several challenges, the largest of which was building such a system that requires data for both training as well as testing purposes. For example, labeling a real-world dataset, which was provided by Air Shepherd, took approximately 800 hours over the course of 6 months to complete. This produced 39,380 labeled frames and approximately 180,000 individual poacher and animal labels on those frames. This data was used to build a prototype detection system called SPOT but did not produce acceptable precision and recall values.

AirSim was then used to create a simulation, where virtual UAVs flew over virtual environments like those found in the Central African savanna at an altitude from 200 to 400 feet above ground level. The simulation took on the difficult task of detecting poachers and wildlife, both during the day and at night, and ultimately ended up increasing the precision in detection through imaging by 35.2%.

Driving innovation through simulation

Access to simulation environments means that we have a near-infinite data generation machine, where different simulation parameters can be chosen to generate experiences at will. This capability is foundational to test and debug autonomous systems that eventually would be provably robust and certified. We continue to investigate such fuzzing and falsification framework for various AI systems.

Holistic challenges such as the DARPA SubT Challenge, and partnerships with organizations like Air Shepherd allow researchers and developers to build complete solutions that cover a wide array of research topics. There are many research challenges at the intersection of robotics, simulations, and machine intelligence that we continue to invest in our journey to build toolchains, enabling researchers and developers to build safe and useful simulations and robots.

We invite readers to explore AirSim on our GitHub repository and invest in our journey to build toolchains in collaboration with the community. The AirSim network of man-made caves environment was co-created with Team Explorer for the DARPA SubT Challenge and is publicly available for researchers and developers.

Microsoft is expanding its support for academic researchers through the new Microsoft Investigator Fellowship. This fellowship is designed to empower researchers of all disciplines who plan to make an impact with research and teaching using the Microsoft Azure cloud computing platform.

From predicting traffic jams to advancing the Internet of Things, Azure has continued to evolve with the times, and this fellowship aims to keep Azure at the forefront of new ideas in the cloud computing space. Similarly evolving, Microsoft fellowships have a long history of supporting researchers, seeking to promote diversity and promising academic research in the field of computing. This fellowship is an addition to this legacy that highlights the significance of Azure in education, both now and into the future.

Full-time faculty at degree-granting colleges or universities in the United States who hold PhDs are eligible to apply. This fellowship supports faculty who are currently conducting research, advising graduate students, teaching in a classroom, and plan to or currently use Microsoft Azure in research, teaching, or both.

Fellows will receive $100,000 annually for two years to support their research. Fellows will also be invited to attend multiple events during this time, where they will make connections with other faculty from leading universities and Microsoft. They will have the opportunity to participate in the greater academic community as well. Members of the cohort will also be offered various training and certification opportunities.

When reviewing the submissions, Microsoft will evaluate the proposed future research and teaching impact of Azure. This will include consideration of how the Microsoft Azure cloud computing platform will be leveraged in size, scope, or unique ways for research, teaching, or both.

Candidates should submit their proposals directly on the fellowship website by August 16, 2019. Recipients will be announced in September 2019.

We encourage you to submit your proposal! For more information on the Microsoft Investigator Fellowship, please check out the fellowship website.

Episode 82, June 26, 2019

If you’re like me, you’re no longer amazed by how all your technologies can work for you. Rather, you’ve begun to take for granted that they simply should work for you. Instantly. All together. All the time. The fact that you’re not amazed is a testimony to the work that people like Dr. Lidong Zhou, Assistant Managing Director of Microsoft Research Asia, do every day. He oversees some of the cutting-edge systems and networking research that goes on behind the scenes to make sure you’re not amazed when your technologies work together seamlessly but rather, can continue to take it for granted that they will!

Today, Dr. Zhou talks about systems and networking research in an era of unprecedented systems complexity and what happens when old assumptions don’t apply to new systems, explains how projects like CloudBrain are taking aim at real-time troubleshooting to address cloud-scale, network-related problems like “gray failure,” and tells us why he believes now is the most exciting time to be a systems and networking researcher.

Related:

---

Transcript

Lidong Zhou: We have seen a lot of advances in, for example, machine learning and deep learning. So, one thing that we have been looking into is how we can leverage all those new technologies in machine learning and deep learning and apply it to deal with the complexity in systems.

Host: You’re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I’m your host, Gretchen Huizinga.

Host: If you’re like me, you’re no longer amazed by how all your technologies can work for you. Rather, you’ve begun to take for granted that they simply should work for you. Instantly. All together. All the time. The fact that you’re not amazed is a testimony to the work that people like Dr. Lidong Zhou, Assistant Managing Director of Microsoft Research Asia, do every day. He oversees some of the cutting-edge systems and networking research that goes on behind the scenes to make sure you’re not amazed when your technologies work together seamlessly but rather, can continue to take it for granted that they will!

Today, Dr. Zhou talks about systems and networking research in an era of unprecedented systems complexity and what happens when old assumptions don’t apply to new systems, explains how projects like CloudBrain are taking aim at real-time troubleshooting to address cloud-scale, network-related problems like “gray failure,” and tells us why he believes now is the most exciting time to be a systems and networking researcher. That and much more on this episode of the Microsoft Research Podcast.

Host: Lidong Zhou, welcome to the podcast.

Lidong Zhou: Yes. It’s great to be here.

Host: As the Assistant Managing Director of MSR Asia, you are, among other things, responsible for overseeing research in systems and networking, and I know you’ve done a lot of research in systems and networking over the course of your career as well. So, in broad strokes, what do you do and why do you do it? What gets you up in the morning?

Lidong Zhou: Yeah, I think, you know, this is one of the most exciting times to do research in systems and networking. And we already have seen advances of, you know, systems and networking have been pushing the envelopes in many technologies. We’ve seen the internet, the web, web search, big data, and all the way to the artificial intelligence and cloud computing that, you know, everybody kind of relies on these days.

Host: Yeah.

Lidong Zhou: All those advances have created challenges of unprecedented complexity, scale and a lot of dynamism. So, my understanding, you know, of systems is always, you know, a system is about bringing order to chaos, right? The chaotic situation. So, we are actually in a very chaotic situation where things change so fast and there are a lot of, you know, new technologies coming. And so, when we talk about systems research, it’s really about transforming all those unorganized pieces into a unified whole, right? That’s why, you know, we’re very excited about all those challenges. And also, we realized over the years that it’s actually not just the typical systems expertise – when we talk about distributed systems, operating systems or networking – that’s actually not enough to address the challenges we’re facing. Like, you have to actually also master other fields like, you know, database systems and programming languages, compilers, hardware, and also in artificial intelligence and machine learning and deep learning. And what I do at Microsoft Research Asia, is to put together a team with a diverse set of expertise and inspire the team to take on those big challenges together by, you know, working together, and, you know, that’s a very exciting job to have.

Host: I love the “order out of chaos” representation… if you’ve ever been involved in software code writing, you write this here and someone else is writing that there, and it has to work together, and you’ve got ten other people writing… and we all just take for granted, on my end, it’s going to work. And if it doesn’t, I curse my computer!

Lidong Zhou: Yes, that’s our problem!

Host: Well, I had Hsiao-Wuen Hon on the podcast in November for the 20th anniversary of the lab there, and he talked about the mission to, in essence, both advance the theory and practice of computing, in general. Your own nearly twenty-year career has been about advancing the theory and practice of distributed systems, particularly. So, talk about some of the initiatives you’ve been part of and technical contributions you’ve made to distributed systems over the years. You’ve just come off the heels of talking about the complexities. Now, how have you seen it evolve over those years?

Lidong Zhou: You know, I think we are getting into the year of distributed systems. Being a distributed systems person, we always believe, you know, what we’re working on is the most important piece. You know, I think Microsoft Research is really a great place to connect theory and practice, because we are constantly exposed to very difficult technical challenges from the product teams. They’re tackling very difficult problems, and we also have the luxury of stepping back and thinking deeply about the problems we’re facing and thinking about what kinds of new theories we want to develop, what new methodologies we can develop to address those problems. I remember, you know, in early 2000, when Microsoft started doing web search, and we had a meeting with the dev manager, who was actually in charge of architecting the web search system. And so, we had a, you know, very interesting discussion. We talked about, you know, how we were doing research in distributed systems, how we had to deal with, you know, a lot of problems when services fail. So, we have to make sure that the whole service actually stays correct in the face of all kinds of problems that you can see in a distributed system. I remember at that time, we had Roy Levin, Leslie Lamport, you know, a lot of colleagues, and we talked about protocols. And, at the beginning, the dev manager basically said, oh yeah, I know, you know, it’s complicated to deal with all these failures, but it’s actually under control. And a couple months later, he came back and said, oh, you know, there’s so many corner cases. It’s just beyond our capability of reasoning about the correctness. And we need the protocols that we were talking about. But it’s also interesting that, you know, in developing those protocols, we tend to make some assumptions. Say, okay, you know, we can tolerate a certain number of failures. And one question that the general manager asked was, you know, what happens if we have more than that number of failures in the system, right? And from a practical point of view, you have to deal with those kinds of situations. In theory, when you work on theory, then, you know, you can say, okay, let’s make an assumption and let’s just work under that assumption. So, we see that there’s a difference between theory and practice. The nice thing about working at Microsoft Research is you can actually get exposed to those real problems and keep you honest about what assumptions are reasonable, what assumptions are not reasonable. And then you think about, you know, what is the best way of solving those problems in a more general sense rather than just solving a particular problem?

Host: Your work in networked computer systems is somewhat analogous to another passion of yours that I’m going to call “networked human systems.” In other words, your desire to build community among systems researchers. How are you going about that? I’m particularly interested in your Asia Pacific Systems workshop and the results you’ve seen come out of that.

Lidong Zhou: So, I moved to Microsoft Research Asia in late 2008, and, when I was in the United States, clearly there is a very strong systems community. And, over the years, we’ve also seen that community sort of expanding into Europe. So, the European systems community sort of started the systems workshop, and eventually it evolved into a conference called EuroSys, and very successfully. And you know we see a lot of people getting into systems and networking because of the community, because of the influence of those conferences. And the workshop has been very successful in gathering momentum in the region. And so, in 2010, I remember it was Chandu Thekkath and Rama Kotla who were my colleagues at Microsoft Research, and they basically had this idea that maybe we should start something also in the Asia Pacific region. At that time, I was already working in Beijing, and I thought, you know, this is also part of my obligation. So, in 2010, we started the first Asia Pacific systems workshop. And it was a humble beginning. We had probably about thirty submissions and accepted probably a dozen. It was a good workshop, but it was a very humble beginning, as I said. But what happened after that was really beyond our expectation. It’s like, you know, we just planted a seed, and the community sort of picked it up and grew with it. And, you know, it’s very satisfying to see that we’re actually going to have the tenth workshop in Hangzhou in August. If you look at the organizing committee, they are really you know all world-class researchers from all over the world. It’s not just from a particular region, but you know really, all the experts across the world contributed to the success of this workshop over the last, you know, almost ten years now. And the impact that this workshop has is actually pretty tremendous.

Host: What would you attribute it to?

Lidong Zhou: I think it’s really, first of all, this is the natural trend, right? You go from… the U.S. was leading in systems research and, and then expanded to Europe. And it’s just a natural trajectory to expand further to Asia Pacific given, you know, a lot of, you know, technological advances are happening in Asia. And the other, you know, reason is because the community really came together. There are a lot of top systems researchers that originally, just like me, came from the Asia Pacific region. So, we have a lot of incentives and commitment to give back.

Host: Right.

Lidong Zhou: And all those enthusiasms, passion, or the willingness to help young researchers in the region, I mean those actually contributed to the success of the workshop, in my view.

Host: Well, you were recently involved in hosting another interesting workshop, or conference: The Symposium on Operating Systems Principles, right?

Lidong Zhou: Right.

Host: SOSP?

Lidong Zhou: SOSP.

Host: And this was in Shanghai in 2017. It’s the premier conference for computer systems technology. And as I understand, it’s about as hard to win the bid for as the Olympics!

Lidong Zhou: Yes, almost.

Host: So why was it important to host this conference for you, and how do you think it will help broaden the reach of the systems community worldwide?

Lidong Zhou: So, SOSP is one of the most important systems conferences and traditionally, it has been held in the U.S. and later on, they started rotating into Europe. And it was really a very interesting journey that we went through, along with Professor Haibo Chen who is from Shanghai Jiao Tong University. We started pitching for having SOSP in the Asia Pacific region in 2011. That was like six years before we actually succeeded! We pitched three times. But overall, even for the first time, the community was very supportive in many ways, so that we’d be very careful to make sure that the first one is going to be a success. And in 2017, when Haibo and I opened the conference, I was actually very happy that I didn’t have to be there to make another pitch! I was essentially opening the conference. And it was very successful in the sense that we had a record number of attendees, over eight hundred people…

Host: Wow.

Lidong Zhou: …and we had almost the same number, if not a little bit more, from the U.S. and Europe. And we had, you know, many more people from the region, which was what we intended.

Host: Mm-hmm.

Lidong Zhou: And having the conference in the Asia Pacific is actually very significant to the region. We’re seeing more and more high-quality work and papers in those top conferences from the Asia Pacific region, you know, from Korea, India, China, and many other countries.

Host: Right.

Lidong Zhou: And I’d like to believe that what we have done sort of helped a little bit in those regards.

(music plays)

Host: Let’s talk about the broader topic of education for a minute. This is really, really important for the systems talent pipeline around the world. And perhaps the biggest challenge is expanding and improving university-level education for this talent pipeline. MSRA has been hosting a systems education workshop for the past three years. The fourth is coming up this summer, and none other than Turing Award winner John Hopcroft has praised it as “a step toward improving education and cultivating world-class talent.” And he also said a fifth of the world’s talent is in the Asia Pacific region, so we’d better get over there. Tell us about this ongoing workshop.

Lidong Zhou: Yeah, actually John really inspired us to get this started I think more than three years ago.

Host: Mm-hmm.

Lidong Zhou: And I think we’re seeing a need to improve, you know, systems education. But more importantly, I think, for MSR Asia, one of the things that we’re very proud of doing is connecting educators and researchers from all over the world, especially connecting people from, you know, the U.S. and Europe with those in the Asia Pacific region. And the other thing that we are also very proud of doing is cultivating the next generation of computer scientists. And certainly, as you said, you know, the most important thing is education. And during the process, what we found, is that there are a lot of professors who share the same passion. And we’re talking about, you know, a couple of professors, Lorenzo Alvisi from Cornell and Robbert van Renesse from Cornell and Geoff Voelker from UCSD… they actually came all the way from the U.S. just to be at the workshop, talking to all the systems professors from all over the country in China. And so, I attended those workshops myself. The first one was five days, and the next two were, like, three days. It’s a huge time commitment.

Host: Yeah.

Lidong Zhou: But you see all the passion from those professors. They’re really into improving teaching. They’re trying to figure out, you know, how to make students more engaged, how to get them excited about systems, even how to design experiments, all those aspects. And, you know, we’re really optimistic that with those passionate professors, we’re going to see a very strong new generation of systems researchers. And this is, you know, I think the kind of impact we really want to see from a perspective of, you know, Microsoft Research Asia. It’s not just about making the lab successful, but, if we can make an impact in the community in terms of talent, in terms of the quality of education, that’s much more satisfying.

Host: Before we get into specific work, I’d like you to talk about what you’d referred to as a fundamental shift in the way we need to design systems – and by we, I mean you – in the era of cloud computing and AI. You’ve suggested that things have changed enough that the older methodologies and principles aren’t valid anymore. So, unpack that for us. What’s changed and what needs to happen to build next-gen systems?

Lidong Zhou: Yeah, that’s a great question. I’ll continue with the story about building fault-tolerant systems. So, in the last thirty years, we have been working on systems reliability, and we have developed a lot of techniques, a lot of protocols, and we think it will solve all the problems. But if you look at how this thread of work started, it really started in the late seventies when we were looking at the reliability of airplanes, and so on. Of course, you know, there are assumptions we make about the kinds of failures in those kinds of systems. And we sort of generalize those protocols so that it can be applicable up until now. But if you look at the cloud, it’s much more complicated, in many dimensions. And the system also evolves very quickly. And a lot of assumptions we make actually start to break. And even though we have applied all these well-known techniques, that’s just not enough. So, that’s one aspect. The other aspect is, it used to be that, you know, the system we build, we can sort of understand how it works, right? And now, the complexity has already gone beyond our own understanding, you know. We can’t reason about how the system behaves. On the other hand, we have seen a lot of advances in, for example, machine learning and deep learning. So, one thing that we have been looking into is how we can leverage all those new technologies in machine learning and deep learning and apply it to deal with the complexity in systems. And that’s, you know, another very fascinating area that we’re looking into as well.

Host: Yeah. Well, let’s get specific now. Another super interesting area of research deals with exceptions and failures in the cloud-scale era and how you’re dealing with what you call “gray failure.” And you’ve also called it the gray swan (which I want you to explain) or the Achilles heel of cloud-scale systems. So how did you handle exceptions and failures in a somewhat less complex, pre-cloud era and what new methodologies are you trying to implement now?

Lidong Zhou: Right. So, as I mentioned, in the older days, we are targeting those systems with assumptions about failures, right? Like crash failures, you know, a component can fail… when it fails, it crashes. It stops working. And nowadays, we realize, you know, this kind of assumption no longer holds. So, this is why we define a new type of failures called gray failures. So, thinking about what kind of name to give to this very interesting new line of research that we’re starting so we called it gray swan. People already know about black swan or gray rhino. So first of all, because we’re talking about the cloud, we want something not as heavy as a rhino!

Host: Right.

Lidong Zhou: We want something that can fly. And the reason we call it gray is because, you know a systems component is no longer just black or white. It could be in a weird state where, from some of the observers it’s actually behaving correctly, but from the others, it’s actually not. And that turns out to be behind many of the issues that major problems that we’re seeing in the cloud. And it has sort of some components of black swan in the sense that some of the assumptions we’re making break. So that’s why everything we build on top of that assumption starts to break down. So, for example, I mentioned the assumption about failure, right? If you think that it either crashed or it’s correct, then it’s a very simple kind of world, right? But if it’s not the case, then all the protocols that will work under that assumption will cease to work. It also has this connection with gray rhino because gray rhino is this problem that everybody sort of sees coming, and it’s a very major problem, but people tend to ignore it for the wrong reason. And in our case, we know that, for the cloud, all those service disruptions happen all the time, and there are actually failures all over the place. It’s just very hard to figure out which ones are important. But we know something big is going to happen at some point, right? So, we try to use this notion of gray swan to describe this new line of thinking where, you know, we really think about failures that are not just crash failures or not even, you know, Byzantine failures where it’s essentially arbitrary failures. But there’s something in between that we should reason about, and then using those to reason about the correctness of the whole service.

Host: So, does the word catastrophic enter into this at all? Or is it…

Lidong Zhou: Yes! That could be catastrophic. Eventually.

Host: How does that kind of thinking playing into what you’re doing?

Lidong Zhou: If you look at the cloud system, it’s like in a rhino sort of charging towards you, and before it hits you, there are a lot of dusts, and you know noise and other things. But you just don’t know when and how something bad is going to happen, right? And it could be catastrophic. It happens actually a couple times already. And so, one of the things we try to do is to try to figure out when and how bad things could happen to prevent catastrophic failures…

Host: Right.

Lidong Zhou: …from all the dust and maybe, you know, other signals we have in the system. There are signals. It’s just we don’t know how to leverage them.

Host: Part of your approach to coping with gray failures is a line of research you call CloudBrain.

Lidong Zhou: Right.

Host: And it’s all about automatic troubleshooting for the cloud. It’s actually a huge issue because of the remarkable complexity of the systems. So, tell us how CloudBrain, and what you call DeepView, is actually helping operators – the people that have to deal with it on the ground – simplify how they write troubleshooting algorithms.

Lidong Zhou: Mm-hmm. So, I think CloudBrain is one of the efforts that we have to deal with gray failures. And remember, you know, we talked about the challenges that come from the complexity of the system or the scale of the system. It would really have, you know, a huge number of components interacting with each other. But on the other hand, we can really leverage the scale of the system to help us in terms of, you know, diagnosis and all, detecting problems, even figuring out where the problem is. And this is the premise of the CloudBrain project. So, it has actually three components, three ideas. The first one is really the notion of near, real-time monitoring. And so instead of trying to look at the logs after the fact and then analyze what happened, we try to have a pulse on what the system is doing, how it’s doing, and so on. So that’s the first component. And the second component is we really want to form a global view. So, it’s not just one observation we make about a system, but really observations for all over the systems combined, so we can actually understand how a system is behaving and which part is actually having a problem. And then, the third part is, once you have, you know, all these global observations that come in real time, then we can use statistical methods to really reason about, you know, what’s abnormal and so on. So, this is where we really leverage the scale, the huge amount of data…

Host: Right.

Lidong Zhou: …that used to be a challenge and now it becomes an opportunity for us to actually come up with new solutions to handle the complexity of the system.

Host: So how does that help an operator simplify writing an algorithm?

Lidong Zhou: Right, so now, the operator actually has all the data in near real time. And, you know, you can write this very simple algorithm that operates on the data sort of like a SQL query.

Host: Right.

Lidong Zhou: Right? And then it can emit signals and you know tell people that something’s wrong or something’s correct, or maybe we have to pay attention to part of the system that seems to have some problems.

Host: So where is this gray failure research, with all its pieces and parts, in the pipeline for production?

Lidong Zhou: Overall, we are not at the stage where we solve all the problems, but we have pieces of the technology we developed to solve some specific problems like DeepView and CloudBrain are, you know, the two projects that have already been incorporated in Azure to deal with network-related problems, for example.

Host: Mm-hmm.

Lidong Zhou: But, you know, we are far from solving the problem. It’s really sort of a research agenda that we set out probably for years to come. And one idea that we have been working on, which is actually very interesting, is that we really have to change how we view programs. In the past, for defensive programming, we have been trained to handle exceptions, and it turns out that handling exceptions in a large, complex system is not enough. So, one of the ideas that we’ve been thinking about is changing exception handling into exception or error reporting. So, you start to collect all those signals. We talked about, you know, the dust when the…

Host: Right.

Lidong Zhou: …rhino comes charging at you. So, you have to really collect those dusts towards one place so that you can actually reason about the behavior of the system. And that’s, you know, one of those major shifts…

Host: Yeah.

Lidong Zhou: …that, you know, we see coming even in how we develop systems.

Host: Right.

Lidong Zhou: Not just, you know, after the fact, we already have this beast and now we need to understand what’s going on.

Host: Right.

Lidong Zhou: So those methodologies, I think, is where we’re pushing. You know, it’s not just solving a specific problem. We have an incident; we try to solve this problem. Yeah, we can do that. But more importantly… this goes back to the theory meets practice…

Host: Right.

Lidong Zhou: …so, we need to come out of looking at the specific instances, but think about, you know, what methodologies we should adopt to change the status completely.

Host: So how do you implement, then, a brand-new thing? I mean, we talked about the beast that already exists, and is growing. What are you proposing with your research?

Lidong Zhou: Right, so, this is always a hard problem. We already have something running, and it has to keep running, and now it has a lot of problems we need to solve. So, one of the ways we deal with those challenges is trying to solve the current problems. You know, like CloudBrain and DeepView sort of try to fit into the current practice. But for some other projects, what we do is like, you know, what I talked about, changing from exception handling to error reporting – that actually is a system we build that we can transform automatically a piece of code that does error handling in the traditional way into a piece of code that actually does error reporting in the way that we desire.

Host: Right.

Lidong Zhou: And that helps because we don’t want everybody to rewrite the whole code base.

Host: No.

Lidong Zhou: It’s just not possible. So, we have to find ways to help developers to sort of do the transformation and also live with the current boundaries of the system. And we hopefully, gradually, we’ll move towards the right direction.

Host: Yeah, I think you see that in just about every place software exists is there’s a legacy system. You’ve got to retrofit some stuff that added complexity to it.

Lidong Zhou: That’s right.

Host: But you can’t just make everyone throw out what they’re already using. So, this is a big challenge. I’m glad you’re on the job.

(music plays)

Host: Well, we talked about what gets you up in the morning and all the work you’re doing to make sure that everything goes right… that is basically what you’re doing, is trying to make everything go right…

Lidong Zhou: Right.

Host: …but as we know – as you know more than I know – something always goes wrong!

Lidong Zhou: Right, unfortunately.

Host: The rhino… So, given what you see in your work every day, is there anything that keeps you up at night?

Lidong Zhou: Yes, I think we’re realizing that the kinds of distributed systems we’re designing, or building, are becoming more and more important. They’re becoming part of the sort of critical infrastructure of our society. And that puts a lot of burden on us to make sure that whatever we’re building can be mission critical.

Host: Right.

Lidong Zhou: And you know we have a lot of researchers working on formal methods, verification, just to make sure that the core of the system can be verifiable, will give some assurance that it’s actually working correctly. And, you know, we talked about applying machine learning and deep learning mechanisms, but it’s statistical. So sometimes – actually, naturally – there are cases where it breaks. So how we can safeguard this kind of system from what you call catastrophic issues, and this is also another thing that we have been putting a lot of thought into. And we’re not short of challenges, especially on making the cloud infrastructure really, you know, mission critical!

Host: Lidong, tell us your story. How did you end up at Microsoft Research, and how did you develop your path to the positions you hold right now?

Lidong Zhou: Yeah, looking back, I remember when I finished my PhD, I started job hunting and I got, you know, a couple of offers, and I talked to my advisor. Of course, that’s what you do when you’re a graduate student. And he basically gave me a very simple piece of advice. He basically said, well, just go where you can find the best colleagues, the colleagues with maybe, you know, Turing-Award caliber. So, I ended up going to Microsoft Research Lab where, at that time, we didn’t have a Turing Award winner, but within ten years, we had two! So that was how things started. Looking back, what’s really important is the quality of colleagues you have, especially in the early stages of my career. I learned how to do research in some sense. It’s not about getting papers published. It’s internal passion that drives research and I think the first phase of my career is more on personal development. I remember being pushed by my manager at the time, Roy Levin, to get out of my comfort zone. We started as a sort of technical contributor, but then, I was pushed to lead a project and there are always new challenges that you face. And you get a lot of support from your colleagues to get to the next stage, and that’s very satisfying. And then I went to MSR Asia, where I later became a manager of a research group, and I think that’s sort of the second phase of my career, where it’s not about my personal career development. It’s also about building a team and how you can contribute to other people’s success. And that turns out to be even more satisfying to see the impact you can have on other people’s careers and their success. And also, during that period of time, I also realized that it’s not just about your own team. You know, we can build the best systems research team in Asia Pacific, but it’s more satisfying if you can contribute to the community. And we talked about starting the workshop and getting the conference into Asia Pacific, and, you know, a lot of other things that we do to contribute to society, including, you know, the talent fostering and many other things. And those, in my mind, are becoming even more critical as we move on in our career.

Host: Yeah.

Lidong Zhou: So, I view this as sort of the three stages of my career. It started with personal development, learning, you know, what it means to love what you do and do what you love. And then you think about how you can contribute to other people’s success and increase your ability to influence others and impact others, and positively. And finally, in what you can contribute to the society, to the community. And I’ve been very fortunate to have been working with a lot of great, you know, leaders and colleagues, and I’ve learned a lot along the way. And I remember you know I worked with a lot of product teams as well. And they also offered a lot of career advice and support. So, this is just, you know, my story, I guess.

Host: You know, it sounds to me like almost a metaphor. You know, you start with yourself, you grow and mature outwards to others, and then the broader community impact that ultimately a mature person wants to see happen, right?

Lidong Zhou: I hope so!

Host: I get the sense that it is!

Lidong Zhou: It’s just about seeking the truth. It’s not about, you know, getting papers published. It’s not about, you know, chasing fame or, you know, all those things that we start to lose sight of, you know, what the true meaning of research is. It’s not about all these results that we try to get, but truly, it’s about finding the truth and enjoying the process along the way.

Host: At the end of each podcast, I ask my guests to give some parting advice to our listeners. What big, unsolved problems do you see on the horizon for researchers who may just be getting their feet wet with systems and networking research?

Lidong Zhou: Well, I think they are very fortunate to be a young researcher in systems and networking now. I remember I was talking to But[ler] Lampson when I started my career in 2003, and he said, you know, he was feeling lucky that he was doing all the work in the late seventies and early eighties because it was the right time to see a paradigm shift. And I think, now, we are at the point that we’re going to see another major paradigm shift, just like, you know, folks in Xerox PARC. What they did was, essentially, to define computing for the next thirty years. Even now, we’re sort of living in the world that they defined, looking at the PC, even with the phone. I mean, that’s just a different form factor, right? They sort of defined the mouse, the laser printer, all the things that we know about, and the user interface. And the reason that happened at that time was because the computing was becoming, you know, more powerful from supercomputers now to personal computing, because…

Host: Right.

Lidong Zhou: …you know, we can pack so much computation power into a small machine. And now, I think, you know, the computation power has reached another milestone where computing capability is going to be everywhere. And we’re going to have intelligence everywhere around us. The boundary between sort of the virtual world in computers and our physical world will disappear. And that will lead to really paradigm-shifting opportunities where we figure out, you know, what computing really means in the next, you know, ten years, twenty years. And this is what I would encourage everyone focus on rather than just incremental improvements to the protocols and so on. Because we are really seeing a lot of assumptions being invalidated. And we really have to look at the world in a very different view and from, you know, how we interact with sort of the computing capability and how we expose computing capability to do what we need to do. And it’s not just doing computing in front of a computer but, you know, doing everything with sort of the computing capability around us. And that’s just exciting to imagine. And I can’t even describe what the future will look like, but it’s up to our young researchers to really make it a reality.

Host: Lidong Zhou, it’s been an absolute pleasure. Thanks for joining us in the booth today.

Lidong Zhou: Thank you, Gretchen. Really a pleasure.

(music plays)

To learn more about Dr. Lidong Zhou and how researchers are working to bring order out of systems and networking chaos, visit Microsoft.com/research

Episode 68, March 20, 2019

If you’ve ever played video games, you know that for the most part, they look a lot better than they sound. That’s largely due to the fact that audible sound waves are much longer – and a lot more crafty – than visual light waves, and therefore, much more difficult to replicate in simulated environments. But Dr. Nikunj Raghuvanshi, a Senior Researcher in the Interactive Media Group at Microsoft Research, is working to change that by bringing the quality of game audio up to speed with the quality of game video. He wants you to hear how sound really travels – in rooms, around corners, behind walls, out doors – and he’s using computational physics to do it.

Today, Dr. Raghuvanshi talks about the unique challenges of simulating realistic sound on a budget (both money and CPU), explains how classic ideas in concert hall acoustics need a fresh take for complex games like Gears of War, reveals the computational secret sauce you need to deliver the right sound at the right time, and tells us about Project Triton, an acoustic system that models how real sound waves behave in 3-D game environments to makes us believe with our ears as well as our eyes.

Related:

---

Final Transcript

Nikunj Raghuvanshi: In a game scene, you will have multiple rooms, you’ll have caves, you’ll have courtyards, you’ll have all sorts of complex geometry and then people love to blow off roofs and poke holes into geometry all over the place. And within that, now sound is streaming all around the space and it’s making its way around geometry. And the question becomes how do you compute even the direct sound? Even the initial sound’s loudness and direction, which are important? How do you find those? Quickly? Because you are on the clock and you have like 60, 100 sources moving around, and you have to compute all of that very quickly.

Host: You’re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I’m your host, Gretchen Huizinga.

Host: If you’ve ever played video games, you know that for the most part, they look a lot better than they sound. That’s largely due to the fact that audible sound waves are much longer – and a lot more crafty – than visual light waves, and therefore, much more difficult to replicate in simulated environments. But Dr. Nikunj Raghuvanshi, a Senior Researcher in the Interactive Media Group at Microsoft Research, is working to change that by bringing the quality of game audio up to speed with the quality of game video. He wants you to hear how sound really travels – in rooms, around corners, behind walls, out doors – and he’s using computational physics to do it.

Today, Dr. Raghuvanshi talks about the unique challenges of simulating realistic sound on a budget (both money and CPU), explains how classic ideas in concert hall acoustics need a fresh take for complex games like Gears of War, reveals the computational secret sauce you need to deliver the right sound at the right time, and tells us about Project Triton, an acoustic system that models how real sound waves behave in 3-D game environments to makes us believe with our ears as well as our eyes. That and much more on this episode of the Microsoft Research Podcast.

Host: Nikunj Raghuvanshi, welcome to the podcast.

Nikunj Raghuvanshi: I’m glad to be here!

Host: You are a senior researcher in MSR’s Interactive Media Group, and you situate your research at the intersection of computational acoustics and graphics. Specifically, you call it “fast computational physics for interactive audio/visual applications.”

Nikunj Raghuvanshi: Yep, that’s a mouthful, right?

Host: It is a mouthful. So, unpack that! How would you describe what you do and why you do it? What gets you up in the morning?

Nikunj Raghuvanshi: Yeah, so my passion is physics. I really like the mixture of computers and physics. So, the way I got into this was, many, many years ago, I picked up this book on C++ and it was describing graphics and stuff. And I didn’t understand half of it, and there was a color plate in there. It took me two days to realize that those are not photographs, they were generated by a machine, and I was like, somebody took a photo of a world that doesn’t exist. So, that is what excites me. I was like, this is amazing. This is as close to magic as you can get. And then the idea was I used to build these little simulations and I was like the exciting thing is you just code up these laws of physics into a machine and you see all this behavior emerge out of it. And you didn’t tell the world to do this or that. It’s just basic Newtonian physics. So, that is computational physics. And when you try to do this for games, the challenge is you have to be super-fast. You have 1/60th of a second to render the next frame to produce the next buffer of audio. Right? So, that’s the fast portion. How do you take all these laws and compute the results fast enough that it can happen at 1/60th of a second, repeatedly? So, that’s where the computer science enters the physics part of it. So, that’s the sort of mixture of things where I like to work in.

Host: You’ve said that light and sound, or video and audio, work together to make gaming, augmented reality, virtual reality, believable. Why are the visual components so much more advanced than the audio? Is it because the audio is the poor relation in this equation, or is it that much harder to do?

Nikunj Raghuvanshi: It is kind of both. Humans are visual dominant creatures, right? Because visuals are what is on our conscious mind and when you describe the world, our language is so visual, right? Even for sound, sometimes we use visual metaphors to describe things. So, that is part of it. And part of it is also that for sound, the physics is in many ways tougher because you have much longer wavelengths and you need to model wave diffraction, wave scattering and all these things to produce a believable simulation. And so, that is the physical aspect of it. And also, there’s a perceptual aspect. Our brain has evolved in a world where both audio/visual cues exist, and our brain is very clever. It goes for the physical aspects of both that give us separate information, unique information. So, visuals give you line-of-sight, high resolution, right? But audio is lower resolution directionally, but it goes around corners. It goes around rooms. That’s why if you put on your headphones and just listen to music at the loud volume, you are a danger to everybody on the street because you have no awareness.

Host: Right.

Nikunj Raghuvanshi: So, audio is the awareness part of it.

Host: That is fascinating because you’re right. What you can see is what is in front of you, but you could hear things that aren’t in front of you.

Nikunj Raghuvanshi: Yeah.

Host: You can’t see behind you, but you can hear behind you.

Nikunj Raghuvanshi: Absolutely, you can hear behind yourself and you can hear around stuff, around corners. You can hear stuff you don’t see, and that’s important for anticipating stuff.

Host: Right.

Nikunj Raghuvanshi: People coming towards you and things like that.

Host: So, there’s all kinds of people here that are working on 3D sound and head-related transfer functions and all that.

Nikunj Raghuvanshi: Yeah, Ivan’s group.

Host: Yeah! How is your work interacting with that?

Nikunj Raghuvanshi: So, that work is about, if I tell you the spatial sound field around your head, how does it translate into a personal experience in your two ears? So, the HRTF modeling is about that aspect. My work with John Snyder is about, how does the sound propagate in the world, right?

Host: Interesting.

Nikunj Raghuvanshi: So, if there is a sound down a hallway, what happens during the time it gets from there up to your head? That’s our work.

Host: I want you to give us a snapshot of the current state-of-the-art in computational acoustics and there’s apparently two main approaches in the field. What are they, and what’s the case for each and where do you land in this spectrum?

Nikunj Raghuvanshi: So, there’s a lot of work in room acoustics where people are thinking about, okay, what makes a concert hall sound great? Can you simulate a concert hall before you build it, so you know how it’s going to sound? And, based on the constraints on those areas, people have used a lot of ray tracing approaches which borrow on a lot of literature in graphics. And for graphics, ray tracing is the main technique, and it works really well, because the idea is you’re using a short wavelength approximation. So, light wavelengths are submicron and if they hit something, they get blocked. But the analogy I like to use is sound is very different, the wavelengths are much bigger. So, you can hold your thumb out in front of you and blot out the sun, but you are going to have a hard time blocking out the sound of thunder with a thumb held out in front of your ear because the waves will just wrap around. And, that’s what motivates our approach which is to actually go back to the physical laws and say, instead of doing the short wave length approximation for sound, we revisit and say, maybe sounds needs the more fundamental wave equation to be solved, to actually model these diffraction effects for us. The usual thinking is that, you know, in games, you are thinking about we want a certain set of perceptual cues. We want walls to occlude sound, we want a small room to reverberate less. We want a large hall to reverberate more. And the thought is, why are we solving this expensive partial differential equation again? Can’t we just find some shortcut to jump straight to the answer instead of going through this long-winded route of physics? And our answer has been that you really have to do all the hard work because there’s a ton of information that’s folded in and what seems easy to us as humans isn’t quite so easy for a computer and and there’s no neat trick to get you straight to the perceptual answer you care about.

(music plays)

Host: Much of the work in audio and acoustic research is focused on indoor sound where the sound source is within the line of sight and the audience and the listener can see what they were listening to…

Nikunj Raghuvanshi: Um-hum.

Host: …and you mentioned that the concert hall has a rich literature in this field. So, what’s the gap in the literature when we move from the concert hall to the computer, specifically in virtual environments?

Nikunj Raghuvanshi: Yeah, so games and virtual reality, the key demand they have is the scene is not one room, and with time it has become much more difficult. So, a concert hall is terrible if you can’t see the people who are playing the sound, right? So, it allows for a certain set of assumptions that work extremely nicely. The direct sound, which is the initial sound, which is perceptually very critical, just goes in a straight line from source to listener. You know the distance so you can just use a simple formula and you know exactly how loud the initial sound is at the person. But in a game scene, you will have multiple rooms, you’ll have caves, you’ll have courtyards, you’ll have all sorts of complex geometry and then people love to blow off roofs and poke holes into geometry all over the place. And within that, now sound is streaming all around the space and it’s making its way around geometry. And the question becomes, how do you compute even the direct sound? Even the initial sound’s loudness and direction, which are important? How do you find those? Quickly? Because you are on the clock and you have like 60, 100 sources moving around, and you have to compute all of that very quickly. So, that’s the challenge.

Host: All right. So, let’s talk about how you’re addressing it. A recent paper that you’ve published made some waves, sound waves probably. No pun intended… It’s called Parametric Directional Coding for Pre-computed Sound Propagation. Another mouthful. But it’s a great paper and the technology is so cool. Talk about this… research this that you’re doing.

Nikunj Raghuvanshi: Yeah. So, our main idea is, actually, to look at the literature in lighting again and see the kind of path they’d followed to kind of deliver this computational challenge of how you do these extensive simulations and still hit that stringent CPU budget in real time. And one of the key ideas is you precompute. You cheat. You just look at the scene and just compute everything you need to compute beforehand, right? Instead of trying to do it on the fly during the game. So, it does introduce the limitation that the scene has to be static. But then you can do these very nice physical computations and you can ensure that the whole thing is robust, it is accurate, it doesn’t suffer from all the sort of corner cases that approximations tend to suffer from, and you have your result. You basically have a giant look-up table. If somebody tells you that the source is over there and the listener is over here, tell me what the loudness of the sound would be. We just say okay, we this a giant table, we’ll just go look it up for you. And that is the main way we bring the CPU usage into control. But it generates a knock-off challenge that now we have this huge table, there’s this huge amount of data that we’ve stored and it’s 6-dimensional. The source can move in 3-dimensions and the listener can move in 3-dimensions. So, we have the giant table which is terabytes or even more on data.

Host: Yeah.

Nikunj Raghuvanshi: And the game’s typical budget is like 100 megabytes. So, the key challenge we’re facing is, how do we fit everything in that? How do we take this data and extract out something salient that people listen to and use that? So, you start with full computation, you start as close to nature as possible and then we’re saying okay, now what would a person hear out of this? Right? Now, let’s do that activity of, instead of doing a shortcut, now let’s think about okay, a person hears the directional sound comes from. If there is a doorway, the sound should come from the doorway. So, we pick out these perceptual parameters that are salient for human perception and then we store those. That’s the crucial way you kind of bring down this enormous data set and do a sort of memory budget that’s feasible.

Host: So, that’s the paper.

Nikunj Raghuvanshi: Um-hum.

Host: And how has it played out in practice, or in project, as it were?

Nikunj Raghuvanshi: So, a little bit of history on this is, we had a paper SIGGRAPH 2010, me and John Snyder and some academic collaborators, and at that point, we were trying to think of just physical accuracy. So, we took the physical data and we were trying to stay as close to physical reality as possible and we were rendering that. And around 2012, we got to talking with Gears of War, the studio, and we were going through what the budgets will be, how things would be. And we were like we need… this needs to… this is gigabytes, it needs to go to megabytes…

Host: Really?

Nikunj Raghuvanshi: …very quickly. And that’s when we were like, okay, let’s simplify. Like, what’s the four like most basic things that you really want from an acoustic system? Like walls should occlude sound and thing like that. So, we kind of re-winded and came to it from this perceptual viewpoint that I was just describing. Let’s keep only what’s necessary. And that’s how we were able to ship this in 2016 in Gears of War 4 by just re-winding and doing this process.

Host: How is that playing in to, you know… Project Triton is the big project that we’re talking about. How would you describe what that’s about and where it’s going? Is it everything you’ve just described or is there… other aspects to it?

Nikunj Raghuvanshi: Yeah. Project Triton is this idea that you should precompute the wave physics, instead of starting with approximations. Approximate later. That’s one idea of Project Triton. And the second is, if you want to make it feasible for real games and real virtual reality and augmented reality, switch to perceptual parameters. Extract that out of this physical simulation and then you have something feasible. And the path we are on now, which brings me back to the recent paper you mentioned…

Host: Right.

Nikunj Raghuvanshi: …is, in Gears of War, we shipped some set of parameters. We were like, these make a big difference. But one thing we lacked was if the sound is, say, in a different room and you are separated by a doorway, you would hear the right loudness of the sound, but its direction would be wrong. Its direction would be straight through the wall, going from source to listener.

Host: Interesting.

Nikunj Raghuvanshi: And that’s an important spatial cue. It helps you orient yourself when sounds funnel through doorways.

Host: Right.

Nikunj Raghuvanshi: Right? And it’s a cue that sound designers really look for and try to hand-tune to get good ambiances going. So, in the recent 2018 paper, that’s what we fixed. We call this portaling. It’s a made-up word for this effect of sounds going around doorways, but that’s what we’re modeling now.

Host: Is this new stuff? I mean, people have tackled these problems for a long time.

Nikunj Raghuvanshi: Yeah.

Host: Are you people the first ones to come up with this, the portaling and…?

Nikunj Raghuvanshi: I mean, the basic ideas have been around. People know that, perceptually, this is important, and there are approaches to try to tackle this, but I’d say, because we’re using wave physics, this problem becomes much easier because you just have the waves diffract around the edge. With ray tracing you face the difficult problem that you have to trace out the rays “intelligently” somehow to hit an edge, which is like hitting a bullseye, right?

Host: Right.

Nikunj Raghuvanshi: So, the ray can wrap around the edge. So, it becomes really difficult. Most practical ray tracing systems don’t try to deal with this edge diffraction effect because of that. Although there are academic approaches to it, in practice it becomes difficult. But as I worked on this over the years, I’ve kind of realized, these are the real advantages of this. Disadvantages are pretty clear: it’s slow, right? So, you have to precompute. But we’re realizing, over time, that going to physics has these advantages.

Host: Well, but the precompute part is innovative in terms of a thought process on how you would accomplish the speed-up…

Nikunj Raghuvanshi: There have been papers on precomputed acoustics, academically before, but this realization that mixing precomputation and extracting these perceptual parameters? That is a recipe that makes a lot of practical sense. Because a third thing that I haven’t mentioned yet is going to the perceptual domain, now the sound designer can make sense of the numbers coming out of this whole system. Because it’s loudness. It’s reverberation time, how long the sound is reverberating. And these numbers that are super-intuitive to sound designers, they already deal with them. So, now what you are telling them is, hey, you used to start with a blank world, which had nothing, right? Like the world before the act of creation, there’s nothing. It’s just empty space and you are trying to make things reverberate this way or that, now you don’t need to do that. Now physics will execute first ,on the actual scene with the actual materials, and then you can say I don’t like what physics did here or there, let me tweak it, let me modify what the real result is and make it meet the artistic goals I have for my game.

(music plays)

Host: We’ve talked about indoor audio modeling, but let’s talk about the outdoors for now and the computational challenges to making natural outdoor sounds, sound convincing.

Nikunj Raghuvanshi: Yeah.

Host: How have people hacked it in the past and how does your work in ambient sound propagation move us forward here?

Nikunj Raghuvanshi: Yeah, we’ve hacked it in the past! Okay. This is something we realized on Gears of War because the parameters we use were borrowed, again, from the concert hall literature and, because they’re parameters informed by concert halls, things sound like halls and rooms. Back in the days of Doom, this tech would have been great because it was all indoors and rooms, but in Gears of War, we have these open spaces and it doesn’t sound quite right. Outdoors sounds like a huge hall and you know, how do we do wind ambiances and rain that’s outdoors? And so, we came up with a solution for them at that time which we called “outdoorness.” It’s, again, an invented word.

Host: Outdoorness.

Nikunj Raghuvanshi: Outdoorness.

Host: I’m going to use that. I like it.

Nikunj Raghuvanshi: Because the idea it’s trying to convey is, it’s not a binary indoor/outdoor. When you are crossing a doorway or a threshold, you expect a smooth transition. You expect that, I’m not hearing rain inside, I’m feeling nice and dry and comfortable and now I’m walking into the rain…

Host: Yeah.

Nikunj Raghuvanshi: …and you want the smooth transition on it. So, we built a sort of custom tech to do that outdoor transition. But it got us thinking about, what’s the right way to do this? How do you produce the right sort of spatial impression of, there’s rain outside, it’s coming through a doorway, the doorway is to my left, and as you walk, it spreads all around you. You are standing in the middle of rain now and it’s all around you. So, we wanted to create that experience. So, the ambient sound propagation work was an intern project and now we finished it up with our collaborators in Cornell. And that was about, how do you model extended sound sources? So, again, going back to concert halls, usually people have dealt with point-like sources which might have a directivity pattern. But rain is like a million little drops. If you try to model each and every drop, that’s not going to get you anywhere. So, that’s what the paper is about, how to treat it as one aggregate that somebody gave us? And we produce an aggregate sort of energy distribution of that thing along with this directional characteristics and just encode that.

Host: And just encode it.

Nikunj Raghuvanshi: And just encode it.

Host: How is it working?

Nikunj Raghuvanshi: It works nice. It sounds good. To my ears it sounds great.

Host: Well you know, and you’re the picky one, I would imagine.

Nikunj Raghuvanshi: Yeah. I’m the picky one and also when you are doing iterations for a paper, you also completely lose objectivity at some point. So, you’re always looking for others to get some feedback.

Host: Here, listen to this.

Nikunj Raghuvanshi: Well, reviewers give their feedback, so, yeah.

Host: Sure. Okay. Well, kind of riffing on that, there’s another project going on that I’d love for you to talk as much as you can about called Project Acoustics and kind of the future of where we’re going with this. Talk about that.

Nikunj Raghuvanshi: That’s really exciting. So, up to now, Project Triton was an internal tech which we managed to propagate from research into actual Microsoft product, internally.

Host: Um-hum.

Nikunj Raghuvanshi: Project Acoustics is being led by Noel Cross’s team in Azure Cognition. And what they’re doing is turning it into a product that’s externally usable. So, trying to democratize this technology so it can be used by any game audio team anywhere backed by Azure compute to do the precomputation.

Host: Which is key, the Azure compute.

Nikunj Raghuvanshi: Yeah, because you know, it took us a long time, with Gears of War to figure out, okay, where is all this precompute going to happen?

Host: Right.

Nikunj Raghuvanshi: We had to figure out the whole cluster story for themselves, how to get the machines, how to procure them, and there’s a big headache of arranging compute for yourself. And so that’s, logistically, a key problem that people face when they try to think of precomputed acoustics. The run-time side, Project Acoustics, we are going to have plug-ins for all the standard game audio engines and everything. So, that makes things simpler on that side. But a key blocker in my view was always this question of, where are you going to precompute? So, now the answer is simple. You get your Azure badge account and you just send your stuff up there and it just computes.

Host: Send it to the cloud and the cloud will rain it back down on you.

Nikunj Raghuvanshi: Yes. It will send down data.

Host: Who is your audience for Project Acoustics?

Nikunj Raghuvanshi: Project Acoustics, the audience is the whole game audio industry. And our real hope is that we’ll see some uptake on it when we announce it at GDC in March, and we want people to use it, as many teams, small, big, medium, everybody, to start using this because we feel there’s a positive feedback loop that can be set up where you have these new tools available, designers realize that they have these new tools available that have shipped in Triple A games, so they do work. And for them to give us feedback. If they use these tools, we hope that they can produce new audio experiences that are distinctly different so that then they can say to their tech director, or somebody, for the next game, we need more CPU budget. Because we’ve shown you value. So, a big exercise was how to fit this within current budgets so people can produce these examples of novel possible experiences so they can argue for more. So, to increase the budget for audio and kind of bring it on par with graphics over time as you alluded to earlier.

Host: You know, if we get nothing across in this podcast, it’s like, people, pay attention to good audio. Give it its props. Because it needs it. Let’s talk briefly about some of the other applications for computational acoustics. Where else might it be awesome to have a layer of realism with audio computing?

Nikunj Raghuvanshi: One of the applications that I find very exciting is for audio rendering for people who are blind. I had the opportunity to actually show the demo of our latest system to Daniel Kish, who, if you don’t know, he’s the human echo-locator. And he uses clicks from his mouth to actually locate geometry around him and he’s always oriented. He’s an amazing person. So that was a collaboration, actually, we had with a team in the Garage. They released a game called Ear Hockey and it was a nice collaboration, like there was a good exchange of ideas over there. That’s nice because I feel that’s a whole different application where it can have a potential social positive impact. The other one that’s very interesting to me is that we lived in 2-D desktop screens for a while and now computing is moving into the physical world. That’s the sort of exciting thing about mixed reality, is moving compute out into this world. And then the acoustics of the real world being folded into the sounds of virtual objects becomes extremely important. If something virtual is right behind the wall from you, you don’t want to listen to it with full loudness. That would completely break the realism of something being situated in the real world. So, from that viewpoint, good light transport and good sound propagation are both required things for the future compute platform in the physical world. So that’s a very exciting future direction to me.

(music plays)

Host: It’s about this time in the podcast I ask all my guests the infamous “what keeps you up at night?” question. And when you and I talked before, we went down kind of two tracks here, and I felt like we could do a whole podcast on it, but sadly we can’t… But let’s talk about what keeps you up at night. Ironically to tee it up here, it deals with both getting people to use your technology…

Nikunj Raghuvanshi: Um-hum.

Host: And keeping people from using your technology.

Nikunj Raghuvanshi: No! I wanted everybody to use the technology. But I’d say like five years ago, what used to keep me up at night is like, how are we going to ship this thing in Gears of War? Now what’s keeping me up at night is how do we make Project Acoustics succeed and how do we you know expand the adoption of it and, in a small way, try to improve, move the game audio industry forward a bit and help artists do the artistic expression they need to do in games? So, that’s what I’m thinking right now, how can we move things forward in that direction? I frankly look at video games as an art form. And I’ve gamed a lot in my time. To be honest, all of it wasn’t art, I was enjoying myself a lot and I wasted some time playing games. But we all have our ways to unwind and waste time. But good games can be amazing. They can be much better than a Hollywood movie in terms of what you leave them with. And I just want to contribute in my small way to that. Giving artists the tools to maybe make the next great story, you know.

Host: All right. So, let’s do talk a little bit, though, about this idea of you make a really good game…

Nikunj Raghuvanshi: Um-hum.

Host: Suddenly, you’ve got a lot of people spending a lot of time. I won’t say wasting. But we have to address the nature of gaming, and the fact that there are you know… you’re upstream of it. You are an artist, you are a technologist, you are a scientist…

Nikunj Raghuvanshi: Um-hum.

Host: And it’s like I just want to make this cool stuff.

Nikunj Raghuvanshi: Yeah.

Host: Downstream, it’s people want people to use it a lot. So, how do you think about that and the responsibilities of a researcher in this arena?

Nikunj Raghuvanshi: Yeah. You know, this reminds me of Kurt Vonnegut’s book, Cat’s Cradle? He kind of makes – what there’s scientist who makes Ice 9 and it freezes the whole planet or something. So, you see things about video games in the news and stuff. But I frankly feel that the kind of games I’ve participated in making, these games are very social experiences. People meet on the games a lot. Like Sea of Thieves is all about, you get a bunch of friends together, you’re sitting on the couch together, and you’re just going crazy like on these pirate ships and trying to just have fun. So, they are not the sort of games where a person is being separated from society by the act of gaming and just is immersed in the screen and is just not participating in the world. They are kind of the opposite. So, games have all these aspects. And so, I personally feel pretty good about the games I’ve contributed to. I can at least say that.

Host: So, I like to hear personal stories of the researchers that come on the podcast. So, tell us a little bit about yourself. When did you know you wanted to do science for a living and how did you go about making that happen?

Nikunj Raghuvanshi: Science for a living? I was the guy in 6th grade who’d get up and say I want to be a scientist. So, that was then, but what got me really hooked was graphics, initially. Like I told you, I found the book which had these color plates and I was like, wow, that’s awesome! So, I was at UNC Chapel Hill, graphics group, and I studied graphics for my graduate studies. And then, in my second or third year, my advisor, Ming Lin, she does a lot of research in physical simulations. How do we make water look nice in physical simulations? Lots of it is CGI. How do you model that? How do you model cloth? How do you model hair? So, there’s all this physics for that. And so, I took a course with her and I was like, you know what? I want to do audio because you get a different sense, right? It’s simulation, not for visuals, but you get to hear stuff. I’m like okay, this is cool. This is different. So, I did a project with her and I published a paper on sound synthesis. So, like how rigid bodies, like objects rolling and bouncing around and sliding make sound, just from physical equations. And I found a cool technique and I was like okay, let me do acoustics with this. It’s going to be fun. And I’m going to publish another paper in a year. And here I am, still trying to crack that problem of how to do acoustics in spaces!

Host: Yeah, but what a place to be. And speaking of that, you have a really interesting story about how you ended up at Microsoft Research and brought your entire PhD code base with you.

Nikunj Raghuvanshi: Yeah. It was an interesting time. So, when I was graduating, MSR was my number one choice because I was always thinking of this technology as, it would be great if games used this one day. This is the sort of thing that would have a good application in games. And then, around that time, I got hired to MSR and it was a multicore incubation back then, my group was looking at how do these multicore systems enable all sorts of cool new things? And one of the things my hiring manager was looking at was how can we do physically based sound synthesis and propagation. So, that’s what my PhD was, so they licensed the whole code base and I built on that.

Host: You don’t see that very often.

Nikunj Raghuvanshi: Yeah, it was nice.

Host: That’s awesome. Well, Nikunj, as we close, I always like to ask guests to give some words of wisdom or advice or encouragement, however it looks to you. What would you say to the next generation of researchers who might want to make sound sound better?

Nikunj Raghuvanshi: Yeah, it’s an exciting area. It’s super-exciting right now. Because even like just to start from more technical stuff, there are so many problems to solve with acoustic propagation. I’d say we’ve taken just the first step of feasibility, maybe a second one with Project Acoustics, but we’re right at the beginning of this. And we’re thinking there are so many missing things, like outdoors is one thing that we’ve kind of fixed up a bit, but we’re going towards what sorts of effects can you model in the future? Like directional sources is one we’re looking at, but there are so many problems. I kind of think of it as the 1980s of graphics when people first figured out that you can make this work. You can make light propagation work. What are the things that you need to do to make it ever closer to reality? And we’re still at it. So, I think we’re at that phase with acoustics. We’ve just figured out this is one way that you can actually ship in practical applications and we know there are deficiencies in its realism in many, many places. So, I think of it as a very rich area that students can jump in and start contributing.

Host: Nowhere to go but up.

Nikunj Raghuvanshi: Yes. Absolutely!

Host: Nikunj Raghuvanshi, thank you for coming in and talking us today.

Nikunj Raghuvanshi: Thanks for having me.

(music plays)

To learn more about Dr. Nikunj Raghuvanshi and the science of sound simulation, visit Microsoft.com/research