Posted on Leave a comment

How Microsoft Premonition can help in the global fight against disease outbreaks

By the time Zika emerged in 2016 in the Americas, the team had been researching new monitoring approaches for about a year. They quickly produced a small fleet of prototypes. These early robotic smart traps, resembling scale models of circular, high rise condos, were designed to lure, autonomously identify and capture mosquitoes, providing public health officials with data streams that weren’t previously available. The goal was to help them decide when and where disease-transmitting mosquitoes will be – to better understand the risk of Zika.

Harris County, home to the city of Houston, Texas, was the location for the initial deployment of Project Premonition, which has now matured into Microsoft Premonition. Fast-forward four years, and now Microsoft Premonition and Harris County Public Health will begin building one of the most advanced biothreat detection networks in an expanded partnership.

“Game-changing,” is how Douglas E. Norris, an entomologist and Johns Hopkins University professor of molecular microbiology and immunology, describes Premonition’s impact.

“Everything we do now in terms of mosquito treatment is reactive – we see a lot of mosquitoes, we go spray a lot of mosquitoes,” says Norris who worked on the project. “Imagine if you had a forecasting system that shows, in a few days you’re going to have a lot of mosquitoes based on all this data and these models – then you could go out and treat them earlier before they’re biting, spray, hit them early so you don’t get those big mosquito blooms which then might result in disease transmission.”

It is a healthier approach for humans and for the environment, Norris says. And it’s also a more cost-effective approach, especially with COVID-19 stressing the staffing and budget limits of public health departments around the world.

Building on the idea of One-Health, a concept that reinforces the idea that the health of the humans depends on the environment they live in, Premonition aids public health systems to better measure the efficacy of interventions and the costs of varying approaches.

“Most of the things that can impact our health, the health of our societies and our economies are small,” says Jackson. “They’re things like arthropods – such as mosquitoes and ticks, or microbes and viruses that are even smaller; they are at the scales of millimeters, microns and nanometers.”

Microsoft Premonition’s robotic sensing platform will capture, collect, aggregate and analyze data about these tiny and often seemingly unseen threats.

Ethan Jackson stands with plants and shelves in the background
“Microsoft Premonition obviously has a very different perspective than when we first started this project,” says Ethan Jackson, director of Microsoft Premonition. Photo by Microsoft.

“All of the sensor networks that we have today – networks that do things like collect data to predict weather, collect data about the power grid so that we can load balance it, collect information about what traffic is doing so that we can predict it – all of those sensor networks, which are really hundreds and hundreds of millions of sensors – can’t see these important species,” Jackson says.

“These life forms we’re talking about are invisible to basically all of those sensors that we’ve deployed across the globe. And that’s pretty incredible when you think about it, that we have such a huge blind spot about what’s in the environment.”

In 2016, during the peak risk of Zika transmission, 10 robotic smart traps were trained in Harris County to identify and selectively capture relevant mosquitoes and did so with about 90% accuracy. In addition, metagenomic analyses detected microorganisms and viruses in mosquito specimens, and identified the types of animals on which they fed.

Now, with the upcoming deployment of Microsoft Premonition, Harris County will have a sensor network at scale, providing “continuous biological situational awareness,” Jackson says. “So, they should be able to look at a map and see in real time what is happening now. Which, from the weather analogy, just doesn’t exist today. A 24-hour forecast allows them to plan early for specific interventions in the environment.”

“We want a future where emerging pathogens like Zika can be detected and suppressed quickly and equitably across Harris County,” says Dr. Umair Shah, executive director of Harris County Public Health.

“This partnership will also evaluate new genomic capabilities to detect known and emerging pathogens from environmental samples – which we now know is especially important for diseases like COVID-19.”

The future of public health “depends on innovation – innovative science, innovative engineering and innovative policy,” he says. “We are excited to continue this journey with Microsoft as we learn together.”

The next step is to be able to forecast “when and where the threat might emerge, not just 24 hours from now, but say, a month from now,” Jackson says. “And to do that, we’ll be refactoring, redesigning epidemiological models so that we can tell Harris County – ‘This is a location a month from now where there is a high possibility of an outbreak of West Nile virus,’” the primary mosquito-borne virus in Harris County.

Nicolas Villar hovers over a piece of lab equipment as part of Project Premonition
“We are trying to solve really important problems that face humanity today, and on a scale that is incredibly exciting,” says Nicolas Villar, a principal hardware architect. Photo by Microsoft.

Over the last five years, Premonition’s technologies have been tested in a variety of habitats – ranging from the sands of the Florida Keys to the remote forests of Tanzania, Africa. “Biology is hard, and we want to do it right,” Jackson says. Science can’t be rushed.

Premonition systems are developed in the “Premonition Proving Ground,” a state-of-the-art Arthropod Containment Level 2 (ACL-2) facility, where wild mosquitoes can be raised, digitized, and observed to develop identification algorithms and evaluate device designs. The Microsoft Redmond campus is also their hub for computationally scanning environmental samples – obtained and sequenced by collaborating partners – for pathogens.

Stopping disease threats before they cause outbreaks is a cross-disciplinary and cross-industry challenge.

Deep collaboration with academia is important to develop the right technology based on the best understanding of the underlying science. The National Science Foundation (NSF) recently awarded an NSF “Convergence Accelerator” grant, which includes academic partners from Vanderbilt University, Johns Hopkins University, Institute for Health Metrics and Evaluation at the University of Washington and the University of Pittsburgh. The NSF Convergence Accelerator is one of the NSF’s “10 big ideas” to rapidly accelerate innovation.

“This project will provide long lasting contributions to human health and pandemic preparedness,” the NSF said in its award, noting that “as deep biome data exponentially scales, the life sciences will become overwhelmed with genomic information. Convergence must lead to new methods to efficiently harness these data and autonomously derive insights.”

Industrial collaborators are also crucial to success. Bayer, perhaps best known for its aspirin – but also one of the world’s leading agricultural companies – plays a critical role by providing public health with treatments to suppress mosquito populations. For example, in partnership with other leading companies in vector control, they are working to eradicate malaria by the year 2040. According to the World Health Organization, in 2018, nearly half of the world’s population was at risk of malaria.

“That’s why we need to have in our toolbox a diverse set of solutions, including those that give us insights,” says Jacqueline Applegate, president of Global Vegetable Seeds and Environmental Science at Bayer. “Microsoft Premonition gives us the opportunity to be able to get a much more realistic perspective.”

Premonition will allow Bayer “to utilize data, information tools, and resources in new ways so we can be even more prescriptive and optimize our vector control strategies so that they have the greatest impact,” she says. “And as intervention becomes more accurate, we can help free up capacity for countries – with often limited resources – for other public health issues.”

Posted on Leave a comment

Microsoft Expressive Pixels: new platform for creativity, inclusion and innovation

“Emojis are already ubiquitous in digital communication platforms, including text-based messaging, email and social media. They require fewer ‘clicks’ to express intent, add context, or set tone. A single emoji can dramatically alter or enhance the interpretation of a message. We think they can be leveraged as a supplemental expressive proxy for people who may not have access to speech or the muscles that drive facial expression,” Paradiso says. “Our PALS collaborators have been some of the funniest, most thoughtful and creative people I’ve known, but their expressive range can be limited by the constraints of disability as well as today’s speech devices and their underlying technologies. We know that people want to express a lot more with their AAC devices than basic transactional communication. We wanted to create something that would help people stay more actively engaged in conversations, be visible in low light and from a distance, and provide another avenue for unique expression, playfulness and connection.”

The team looked at different types of secondary displays, but they kept coming back to LED displays, for several reasons: they’re low-cost, they work reasonably well and “they have kind of a cool factor.” The team’s collaborators in PALS also made it clear they didn’t want to use anything that could have unintended negative social consequences for the users.

“Watching somebody living with ALS and being able to empower them to do something that they previously had given up all hope of being able to do is enough to inspire you to want to do more,” says Dwayne Lamb, a developer who specializes in User Experience and User Interface creation who joined the Enable Group in early 2017. “Most of the time when you’re trying to communicate with somebody who can only use their eyes to communicate, they’re using their eyes to type into a keyboard on a device right in front of them, and although it’s not socially good etiquette, you commonly find yourself kind of looking over their shoulder to try and see what they’re typing.”

Expressive Pixels evolved in part from wanting to solve that problem.

Face emoji
Emoji in the Expressive Pixels app

And with Expressive Pixels, it’s possible to create animations on displays of many sizes, up to 64 x 64 pixels, says Christopher O’Dowd, who helped fill the hardware gaps on the project. He points out that LED displays are ubiquitous at Maker Faires, on houses during the holidays, etc. LED displays are so versatile, you often even find them on fabric (face masks, caps and backpacks, for instance) or banners.

One early way the team incorporated LED displays was through hands-free music, an award-winning SXSW project.

There, they used a custom midi-enabled, music synced LED array as a supplemental visualization to an eye controlled, physical drum rig designed for one of their collaborators, a Seattle area musician living with ALS. The project, which won the 2018 SXSW Interactive Innovation Award: Music and Audio Innovation, features a suite of novel eye-controlled applications for music performance, collaboration and composition.

“How does someone without access to speech or movement compose or perform music, command a stage, or connect with a live audience? What about collaborating with other musicians in rehearsed or improvisational scenarios? How can we lower the barriers to making school music programs more inclusive without ‘othering’ or minimizing the students with disabilities? These were some of our foundational questions,” Paradiso recalled. “We wanted to adapt our technology and designs to align with a person’s creative goals and real-life scenarios, rather than the way around.”

Lamb came up with the idea to add Musical Instrument Digital Interface (MIDI) capability, with the core idea to send out different signals to different instruments – an idea that would transfer to Expressive Pixels.