Posted on Leave a comment

From predicting performance to preventing injuries: How machine learning is unlocking the secrets of human movement

Launched in 2006, P3 is the first facility to apply a more data-driven approach to understanding how elite competitors move. It uses advanced sports-science strategies to assess and train athletes in ways that will revolutionize pro sports – and, eventually, the bodies and abilities of weekend warriors, Elliott says.

“We are challenging them and measuring them. But we’re not interested in how high they jump or how fast they accelerate,” Elliott says. “We’re interested in the mechanics of how they jump, how they accelerate and decelerate. It’s helping us unlock the secrets of human movement.”

Working directly with players and their agents or families, P3 has evaluated members of the past six NBA draft classes, amassing a database of more than 600 current and former NBA athletes.

An athlete jumps in a gym, reaching for the ceiling with her right hand.
Volleyball player Cassandra Strickland leaps at P3.

Some of P3’s clients include NBA stars Luka Doncic and Zach LaVine plus athletes from the NFL, Major League Baseball, international soccer, track and field and more.

Many of those NBA clients, like Philadelphia 76ers guard Josh Richardson, return to P3 each summer for re-testing to pinpoint whether their movement patterns have gained asymmetries that could cause injury, or to reconfirm the health of physical systems they use to leap, land, stop and start, fueling their on-court edge.

“This is my fifth off-season now at P3,” Richardson says. “When I started with them during my NBA draft preparation, I immediately saw that their approach was different and that it could help me have the best chance to improve my athleticism. Every off-season I get to see exactly where I am physically compared to where I was before – and compared to other NBA players.

“They are able to help me identify where I might be at risk of injury and where I can improve physically. It’s important for me to know that the training I am doing is specific to my unique needs,” Richardson says.

To collect all that granular data, P3 outfitted its lab with a high-speed camera system manufactured by Simi Reality Motion Systems GmbH, a German company from the ZF Group and a Microsoft partner.

Simi offers markerless, motion-capture software that removes the need for athletes to wear tracking sensors while they play or train. Simi also works with seven Major League Baseball clubs, deploying high-speed camera systems to those stadiums to record every pitch during every game since the 2017 season.

Simi’s software digitizes the pitchers’ arm angles and related body movements, spanning 42 different joint centers across 24,000 pitches thrown per team per season. That produces hundreds of billions of data points that are uploaded and processed on Microsoft Azure, enabling teams to create in-depth biomechanical analyses for the players, says Pascal Russ, Simi’s CEO.

A laptop screen shows a player in mid-stride on a track and the movement data he is producing.
An athlete’s workout at P3 produces data on his body angles and movements.

“The first team that deploys this effectively on the field to pick lineups or to see which pitch angles worked well against which batters is going to see a huge separation between them and the other teams not using this,” Russ says.

“It’s freakishly accurate.”

While Russ foresees this technology eventually remaking baseball, such seismic shifts already are occurring in the NBA through P3’s player assessments, says Benedikt Jocham, Simi’s U.S. chief operations officer.

“We provide the software solution that can quantify the movement and analyze, for example, how much pressure and torque a person is putting on various body parts,” Jocham says. “P3 adds the magic sauce. They are wizards at figuring out what it all means and making sense out of it for athletes.”

After the cameras record a player’s movements in the P3 lab, those datasets are loaded into Azure where machine-learning algorithms reveal how that player’s physical systems are most related to other NBA players who were similarly assessed. The algorithm then assigns that player into one of several clusters or branches that predict how their basketball career may unfold, Elliott says.

One branch, for example, contains athletes who had a brief NBA experience and never became significant players. Another branch encompasses players who were impactful during their first three or four seasons then sustained serious injuries that depleted their skills. In still another branch, players share rare combinations of length, power and force that fed elite careers – and they remained healthy.

“The human eye is good at measuring size and maybe estimating weight, and very bad at comparing athletes’ physical systems and movement symmetries to one another,” Elliot says. “But we can measure those things in the lab and the machine tells us how young athletes are most alike.

“It’s a solid foothold into an area of sports science that has been out of sight until now,” he says.

An athlete looks at a big screen projection of his workout and the data is produced while a coach talks with him.
Cleveland Indians prospect Will Benson scans his workout data with P3 biomechanist Ben Johnson.

The data is also helping to shatter long-held theories that successful NBA players who, at first glance, lack the size, jumping ability or quickness of traditional stars are merely compensating by tapping unmeasurable intangibles such as “intuition” or “IQ” or “heart.”

“That’s how people once would have defined (2017-18 NBA most valuable player) James Harden, as somebody who just has this super-high basketball IQ,” Elliott says. “Maybe he does. But he also has a better stopping or braking system than anybody we’ve ever assessed in the NBA.

“That creates competitive advantages,” he adds. “There’s Newtonian physics behind these advantages.”

Case in point: Dallas Mavericks rookie Luka Doncic. In its pre-draft assessment of Doncic one year ago, P3 identified that same hidden performance metric – the elite ability to stop quickly. P3 knew, before his NBA Draft, that Doncic and Harden were in the same player branch. Doncic posted a stunning first pro season.

An athlete is shown moving sideways on an indoor track.
Aaron Gordon training on the P3 indoor track.

The insights also help athletes avoid injuries by adopting new training techniques to change unhealthy movement patterns revealed in the data, says Elliott, who previously served as the first director of sports science in MLB (for the Seattle Mariners) and as the first director of sports science in the NFL (for the New England Patriots).

Every NBA player or draft prospect assessed by P3 receives a report that highlights their injury risks and compares them to league peers based on performance.

“Athletes come to us because they trust us to take better care of their bodies than would happen anywhere else,” Elliott says. “Traditionally, and still today, when these bad things happen to players, everyone says, ‘Oh, that was a freak injury.’ I’m just telling you that the machine learning models predict a whole lot of these.

“I can’t imagine a world where out of nowhere you suffer, say, a right tibial stress fracture – not your left one, not your femur, it’s your tibia, out of nowhere,” he adds. “Without a doubt, these are not random events. Sports science just has not been very good about identifying them.”

Eventually, this same information may become available to amateur athletes and everyone else, Elliott says. The same technologies could predict, for example, that a weekend warrior has too much force going through the left leg while jumping or landing plus a tiny but unhealthy rotation of the left knee and femur, causing too much friction, and, eventually, an erosion of the left knee cartilage.

“What if you identified that when you were 30 or 20, instead of learning when you’re 50 that your cartilage is gone? That really is the future,” Elliott says.

“The power of machine learning and (Microsoft) Artificial Intelligence are going to help us unlock these secrets in ways that have never existed. We’re already doing it but it’s only in the early days of what I think is going to be a revolution in this space,” he says. “It’s coming. It’s definitely coming.

Top photo: Stanley Johnson, a forward with the NBA’s New Orleans Pelicans, moves laterally inside an exercise band at the P3 lab. (All photos courtesy of P3.)

Posted on Leave a comment

Driving lessons for autonomous vehicles

Paul Shieh, Founder and CEO of Linker Networks, says his company is now working with global auto manufacturers that are trying to create AI systems that can drive vehicles with flawless image recognition functionality. To attain that, the systems use machine learning to recognize millions of digital images of other objects, including other vehicles, roads, signs, pedestrians, and a myriad of other features and objects.

To do that, images of all these things must first be identified and labeled.

Shieh explains, “At present, many companies are finding it difficult to hire thousands of workers that want to manually do this image work. It is labor-intensive and time-consuming. Moreover, each worker must maintain unrelenting focus on the task, leaving open the possibility of natural human error. A single mistake is all it takes to affect a dataset’s quality and drag down the overall performance, and therefore the safety level, of a model.”

Manual tagging is labor-intensive and time-consuming. For example, labeling a single car takes a worker up to 30 seconds to complete.

As an example, Shieh says labeling a single car takes a worker up to 30 seconds to complete – placing the duration needed for a thousand workers to process larger quantities of images, say 100 million, at more than a year.

But imagine being able to label all that data in a single click. That is the promise of auto-labeling – Linker Networks’ latest AI venture.

Inventing the fast track

Using a pre-trained model to label digital images, the system recognizes objects using transfer learning technology – a method that lets machines apply existing knowledge to various similar scenarios. For example, systems trained to recognize cars can apply the same algorithm to recognize other vehicles, like buses or trucks.

“If you input an image with about a hundred cars in it and hit the auto-label button, most of them will be auto-labeled in just a few seconds with very high accuracy,” Shieh says. “That saves a lot of time and improves image recognition quality.”

Employees like Cindy Chao, who used to do manual labeling have been upskilled to do quality control of the auto labeling algorithms, also known as machine teaching.

Accuracy rates have also increased. At the same time, manual inspections and corrections are still carried out, to ensure close to 100 percent data accuracy.

The process allows millions of images to be labeled in less than a day, which is a 70 percent reduction in time compared to manual labeling. The company is also seeing cost savings of more than 60 percent.

Shieh shares, “Linker’s auto-labeling model uses Microsoft Azure Machine service to reduce costs, boost productivity and improve accuracy by enabling customers to handpick images to auto-label and store.”

Ultimately with AI, Linker Networks’ goal is for auto manufacturers to build smarter, safer vehicles.

Employees that used to do manual labelling have been upskilled to do quality control of the auto labelling algorithms, also known as machine teaching. The AI model seeks to gain knowledge from people rather than extracting knowledge from data alone. With people guiding the AI systems to learn the things that they already know, the job requires critical thinking and fewer repetitive and monotonous tasks.

“Linker’s data scientists are able to focus on developing the AI and let Azure take care of scaling their AI training jobs,” Shieh explained.

Other possibilities

Ultimately with AI, the company’s goal is for auto manufacturers to build smarter, safer vehicles. With auto labelling technology, Linker Networks envisions safe self-driving capability in the near future.

Besides autonomous driving, auto-labelling can be used in factories to detect product defects, identify theft at retail stores and profile vehicles to strengthen security. Shieh said, “the auto-labeling system allows us to take advantage of all the benefits of AI, empowering humans to do what they do best, while improving efficiency and safety.”

Posted on Leave a comment

Deep Learning Indaba 2018 conference strengthens African contributions to machine learning

deep leaning indaba, participants at conference

Images ©2018 Deep Learning Indaba.

At the 30th conference on Neural Information Processing in 2016, one of the world’s foremost gatherings on machine learning, there was not a single accepted paper from a researcher at an African institution. In fact, for the last decade, the entire African continent has been absent from the contemporary machine learning landscape. The following year, a group of researchers set out to change this, founding a world-class machine learning conference that would strengthen African machine learning – the Deep Learning Indaba.

The first Deep Learning Indaba took place at Wits University in South Africa. The indaba (a Zulu word for a gathering or meeting) was a runaway success, with almost 300 participants representing 22 African countries and 33 African research institutes. It was a week-long event of teaching, sharing and debate around the state of the art in machine learning and artificial intelligence that aimed to be a catalyst for strengthening machine learning in Africa.

indaba group picture

Attendees at Deep Learning Indaba 2017, held at Wits University, South Africa.

Now in its second year, Microsoft is proud to sponsor Deep Learning Indaba 2018, to be held September 9-14 at Stellenbosch University in South Africa.

The conference offers an exciting line-up of talks, hands-on workshops, poster sessions and networking/mentoring events. Once again it has attracted a star-studded guest speaker list – Google Brain lead and Tensorflow co-creator Jeff Dean; DeepMind lead Nando de Freitas; and AlphaGo lead, David Silver. Microsoft is flying in top researchers as well; Katja Hofmann will speak about reinforcement learning and Project Malmo (check out her recent podcast episode). Konstantina Palla will present on generative models and healthcare. And Timnit Gebru will talk about fairness and ethics in AI.

The missing continent

The motivation behind this conference really resonated with me. When I heard about it, I knew I wanted to contribute to the 2018 Indaba, and I was excited that Microsoft was already signed-up as a headline sponsor, and had our own Danielle Belgrave on the advisory board.

African Map - Indaba 2017 attendance

African countries represented at the 2017 Deep Learning Indaba.

Dr Tempest van Schaik, Software Engineer, AI & Data Science

Dr. Tempest van Schaik, Software Engineer, AI & Data Science

I graduated from University of the Witwatersrand (“Wits”) in Johannesburg, South Africa, with a degree in biomedical engineering, and a degree in electrical engineering, not unlike some of the conference organizers. In 2010, I came to the United Kingdom to pursue my PhD at Imperial College London and stayed on to work in the UK, joining Microsoft in 2017 as a software engineer in machine learning.

In my eight years working in the UK in the tech community, I have seldom come across African scientists, engineers and researchers sharing their work on the international stage. During my PhD studies, I was acutely aware of the Randlord monuments flanking my department’s building, despite the absence of any South Africans inside the department. At scientific conferences in Asia, Europe and the USA, I scanned the schedule for African institutions but seldom found them. Fellow Africans that I do find are usually working abroad. I have come to learn that Africa, a continent bigger than the USA, China, India, and Europe put together, has little visible global participation in science and technology. The reasons are numerous, with affordability being just one factor. I have felt the disappointment of trying to get a Tanzanian panelist to a tech conference in the USA. We realized that even if we could raise sufficient funds for his participation, the money would have achieved so much more in his home country that he couldn’t justify spending it on a conference.

Of all tech areas, perhaps it is artificial intelligence in particular that needs African participation. Countries such as China and the UK are gearing-up for the next industrial revolution, creating plans for re-retraining and increasing digital skills. Those who are left behind could face disruption due to AI and automation and might not be able to benefit from the fruits of AI. Another reason to increase African participation in AI is to reduce algorithmic bias that can arise when a narrow section of society develops technology.

A quote from the Indaba 2017 report perhaps says it best: “The solutions of contemporary AI and machine learning have been developed for the most part in the developed-world. As Africans, we continue to be receivers of the current advances in machine learning. To address the challenges facing our societies and countries, Africans must be owners, shapers and contributors of the advances in machine learning and artificial intelligence. “

Attendees at Deep Learning Indaba 2017

Attendees at Deep Learning Indaba 2017

Diversity

One of the goals of the conference is to increase diversity in the field. To quote the organizers, “It is critical for Africans, and women and black people in particular, to be appropriately represented in the advances that are to be made.” The make-up of the Indaba in its first two years is already impressive and leads by example to show how to organize a diverse and inclusive conference. From the Code of Conduct to the organizing committee, the advisory board, the speakers and attendees, you see a group of brilliant and diverse people in every sense.

Women in Machine Learning session

The 2018 Women in Machine Learning lineup.

The 2018 Women in Machine Learning lineup.

The Indaba’s quest for diversity aligns with another passion of mine, that of increasing women’s participation in STEM. Since my days of being the lonely woman in electrical engineering lectures, things have been improving. There seems to be more awareness today about attracting and retaining women in STEM, by improving workplace culture. However, there’s still a long way to go, and in the UK where I work, only 11% of the engineering workforce is female according to a 2017 survey. I have found great support and encouragement from women-in-tech communities and events such as PyLadies/RLadies London and AI Club For Gender Minorities, and saw the Indaba as an opportunity to pay it forward and link up with like-minded women globally. So, I’m very pleased to say that on the evening of September 10 at the Indaba, Microsoft is hosting a Women in Machine Learning event.

Indaba – a gathering.

Indaba – a gathering.

The aim of our evening is to encourage, support and unite women in machine learning. Our panelists each will describe her personal career journey and her experiences as a woman in machine learning. As there will be a high number of students in attendance, our panel also highlights diverse career paths, from academia to industrial research, to applied machine learning, to start-ups. Our panel consists of Sarah Brown (Brown University, USA), Konstantina Palla (Microsoft Research, UK), Muthoni Wanyoike (InstaDeep, Kenya), Kathleen Siminyu (Africa’s Talking, Kenya) and myself from Microsoft Commercial Software Engineering (UK). We look forward to seeing you there!

Posted on Leave a comment

The power of machine learning to change—and maybe even save—the world

In the last two decades, the impact of artificial intelligence (AI) has grown from a very small community of data scientists to something that is woven into many people’s daily lives. Machine learning, computer vision, and other AI disciplines—supported by the cloud—are helping people achieve more, from mundane tasks, like avoiding a traffic jam, to revolutionary breakthroughs, like curing cancer.

Over the past year, Microsoft has been on a journey to apply these transformative technologies to the world’s biggest environmental challenges. On July 12, 2017, Microsoft launched AI for Earth as a $2 million program in London, with a goal of providing AI and cloud tools to researchers working on the frontlines of environmental challenges in the areas of agriculture, water, biodiversity, and climate change.

Since that time, AI for Earth has grown into a $50 million over five-year program, with 112 grantees in 27 countries and seven featured projects. People are using machine learning and computer vision to learn more than previously possible about our planet and how it’s changing, and increasingly using these insights to chart a better future.

These are big goals, but we’re confident in our ability to get there because we know how advanced our tools like machine learning and computer vision already are. Consider machine learning. We have come a long way from the simple pattern-matching of ELIZA. Fifteen years ago, when I got my degree in artificial intelligence, problems like facial recognition, machine translation, and speech recognition were dreams of the field, and now they are solved problems. Among other things, machine learning can group similar items together, detect unusual occurrences, and construct mathematical models of historical data to make future predictions.

These techniques are incredibly helpful for sorting through large amounts of data. Today, we’re excited to share a new story about the power of this technology that also helps answer a basic question: what is the value of AI when we don’t have massive amounts of data already waiting to be processed? This is an issue for many individuals and organizations working in the field of biodiversity, especially when the species are very small, travel great distances, and are hidden from public view.

That’s precisely the challenge we set out to address recently at the most magical place in the world – Walt Disney World Resort. Purple martins are yearly visitors to Disney, nesting at the park before returning their journey to the Brazilian Amazon. Disney scientists have been working with the purple martin community and have provided homes for the families for the past 20 years, studying the conservation of the species with more than 170 nests each year. Despite their annual visits, there is still lots to be learned about nesting behavior of these birds, in part because they nest in enclosed structures known as gourds. Some of what is known is troubling – the species is in decline, with an estimated population drop of 40 percent since 1966.

How do you close this data gap quickly to better understand the species to protect their future? Enter AI. Tiny connected homes, including cameras and cloud-connected sensors were installed, and those combined with computer vision began to deliver data on behaviors that were infrequently observed, like hatching, the caring for and growth of purple martins. External factors, like temperature, humidity, and air pressure were also recorded. Disney and Microsoft hope to expand this work, and AI will help pull all this data together to deliver insights in hopes of inspiring the next generation of conservationists to protect the purple martins for the future.

While this is our newest story, this work is happening across the world. We’re proud to support AI-enabled solutions for biodiversity, including:

PAWS: Machine learning to predict poaching. Spearheaded by a team of researchers at USC, an AI for Earth partner, with additional work being done by a member of the team now at Carnegie Mellon University, an AI for Earth grantee, the Protection Assistant for Wildlife Security (PAWS) processes data about previous poaching activities in an area and creates optimized routes for rangers to patrol based on where poaching is most likely to occur. These routes are also randomized to keep poachers from learning and adapting to patrol patterns. Currently, the PAWS algorithm is being improved so that it can incorporate new information that rangers see while on patrol—such as human footprints—to alter the proposed patrol route in real-time.

Access to ranger patrol data is key. That’s why PAWS partnered with the Uganda Wildlife Authority at Queen Elizabeth National Park. They had collected 14 years of patrol data and more than 125,000 observations on animal sightings, snares, animal remains, and other signs of poaching. PAWS is now being used in several parks, and the system has led to more observations of poacher activities per kilometer than were possible without technology.

Wildbook: Machine learning and computer vision to identify species. One of our newest featured projects, Wild Me, is showing what is possible by pushing the limits of computer vision, with an AI tool that smartly identifies, captions, and moderates pictures. Researchers often have little meaningful data on species. But computer vision makes it possible to tap into an explosion of images, available for free or at a low cost from camera traps, drones, professional photographers, safari-goers, and citizen scientists. Wild Me is not only using computer vision to identify images of zebras, for example, but is also identifying the individual animals in photos—helping to address a fundamental problem in conservation. If we can identify individual animals, then this eliminates the need for physically tagging them, which can harm the animal.

This new data on animals then goes into Wildbook, the platform developed by Wild Me. Using machine learning, it’s possible to either match an animal within the database or determine that the individual is new. Once an animal is identified, it can be tracked in other photographs. Wildbook stores information about the animals, such as their location at a specific time, in a fully developed database. This combination of AI tools and human ingenuity makes it possible to connect information about sightings with additional relevant data, enabling new science, conservation, and education at unprecedented scales and resolution. With a much more detailed and useful picture of what is happening, researchers and other decision-makers are able to implement new, more effective conservation strategies.

We see incredible potential and tremendous progress in our grantees’ work and in the explosive pace at which new algorithms are being built, refined, and made publicly available. And these are just a few of the grantees, featured projects, and partners we’re working with in the area of biodiversity; there’s equally exciting work in water, agriculture, and climate change that we look forward to sharing in the near future on this blog. Check out the amazing organizations and individuals we’re supporting, apply for a grant to join us or our new partnership with National Geographic Society, or just follow our progress on Twitter by following @Microsoft_Green, or me at @jennifermarsman.

Tags: , , ,

Posted on Leave a comment

How Microsoft uses machine learning to fight social engineering attacks

Machine learning is a key driver in the constant evolution of security technologies at Microsoft. Machine learning allows Microsoft 365 to scale next-gen protection capabilities and enhance cloud-based, real-time blocking of new and unknown threats. Just in the last few months, machine learning has helped us to protect hundreds of thousands of customers against ransomware, banking Trojan, and coin miner malware outbreaks.

But how does machine learning stack up against social engineering attacks?

Social engineering gives cybercriminals a way to get into systems and slip through defenses. Security investments, including the integration of advanced threat protection services in Windows, Office 365, and Enterprise Mobility + Security into Microsoft 365, have significantly raised the cost of attacks. The hardening of Windows 10 and Windows 10 in S mode, the advancement of browser security in Microsoft Edge, and the integrated stack of endpoint protection platform (EPP) and endpoint detection and response (EDR) capabilities in Windows Defender Advanced Threat Protection (Windows Defender ATP) further raise the bar in security. Attackers intent on overcoming these defenses to compromise devices are increasingly reliant on social engineering, banking on the susceptibility of users to open the gate to their devices.

Modern social engineering attacks use non-portable executable (PE) files like malicious scripts and macro-laced documents, typically in combination with social engineering lures. Every month, Windows Defender AV detects non-PE threats on over 10 million machines. These threats may be delivered as email attachments, through drive-by web downloads, removable drives, browser exploits, etc. The most common non-PE threat file types are JavaScript and VBScript.

Figure 1. Ten most prevalent non-PE threat file types encountered by Windows Defender AV

Non-PE threats are typically used as intermediary downloaders designed to deliver more dangerous executable malware payloads. Due to their flexibility, non-PE files are also used in various stages of the attack chain, including lateral movement and establishing fileless persistence. Machine learning allows us to scale protection against these threats in real-time, often protecting the first victim (patient zero).

Catching social engineering campaigns big and small

In mid-May, a small-scale, targeted spam campaign started distributing spear phishing emails that spoofed a landscaping business in Calgary, Canada. The attack was observed targeting less than 100 machines, mostly located in Canada. The spear phishing emails asked target victims to review an attached PDF document.

When opened, the PDF document presents itself as a “secure document” that requires action – a very common social engineering technique used in enterprise phishing attacks. To view the supposed “secure document”, the target victim is instructed to click a link within the PDF, which opens a malicious website with a sign-in screen that asks for enterprise credentials.

Phished credentials can then be used for further attacks, including CEO fraud, additional spam campaigns, or remote access to the network for data theft or ransomware. Our machine learning blocked the PDF file as malware (Trojan:Script/Cloxer.A!cl) from the get-go, helping prevent the attack from succeeding. 

Figure 2. Phishing email campaign with PDF attachment

Beyond targeted credential phishing attacks, we commonly see large-scale malware campaigns that use emails with archive attachments containing malicious VBScript or JavaScript files. These emails typically masquerade as an outstanding invoice, package delivery, or parking ticket, and instruct targets of the attack to refer to the attachment for more details. If the target opens the archive and runs the script, the malware typically downloads and runs further threats like ransomware or coin miners.

Figure 3. Typical social engineering email campaign with an archive attachment containing a malicious script

Malware campaigns like these, whether limited and targeted or large-scale and random, occur frequently. Attackers go to great lengths to avoid detection by heavily obfuscating code and modifying their attack code for each spam wave. Traditional methods of manually writing signatures identifying patterns in malware cannot effectively stop these attacks. The power of machine learning is that it is scalable and can be powerful enough to detect noisy, massive campaigns, but also specific enough to detect targeted attacks with very few signals. This flexibility means that we can stop a wide range of modern attacks automatically at the onset.

Machine learning models zero in on non-executable file types

To fight social engineering attacks, we build and train specialized machine learning models that are designed for specific file types.

Building high-quality specialized models requires good features for describing each file. For each file type, the full contents of hundreds of thousands of files are analyzed using large-scale distributed computing. Using machine learning, the best features that describe the content of each file type are selected. These features are deployed to the Windows Defender AV client to assist in describing the content of each file to machine learning models.

In addition to these ML-learned features, the models leverage expert researcher-created features and other useful file metadata to describe content. Because these ML models are trained for specific file types, they can zone in on the metadata of these file types.

Figure 4. Specialized file type-specific client ML models are paired with heavier cloud ML models to classify and protect against malicious script files in real-time

When the Windows Defender AV client encounters an unknown file, lightweight local ML models search for suspicious characteristics in the file’s features. Metadata for suspicious files are sent to the cloud protection service, where an array of bigger ML classifiers evaluate the file in real-time.

In both the client and the cloud, specialized file-type ML classifiers add to generic ML models to create multiple layers of classifiers that detect a wide range of malicious behavior. In the backend, deep-learning neural network models identify malicious scripts based on their full file content and behavior during detonation in a controlled sandbox. If a file is determined malicious, it is not allowed to run, preventing infection at the onset.

File type-specific ML classifiers are part of metadata-based ML models in the Windows Defender AV cloud protection service, which can make a verdict on suspicious files within a fraction of a second.

Figure 5. Layered machine learning models in Windows Defender ATP

File type-specific ML classifiers are also leveraged by ensemble models that learn and combine results from the whole array of cloud classifiers. This produces a comprehensive cloud-based machine learning stack that can protect against script-based attacks, including zero-day malware and highly targeted attacks. For example, the targeted phishing attack in mid-May was caught by a specialized PDF client-side machine learning model, as well as several cloud-based machine learning models, protecting customers in real-time.

Microsoft 365 threat protection powered by artificial intelligence and data sharing

Social engineering attacks that use non-portable executable (PE) threats are pervasive in today’s threat landscape; the impact of combating these threats through machine learning is far-reaching.

Windows Defender AV combines local machine learning models, behavior-based detection algorithms, generics, and heuristics with a detonation system and powerful ML models in the cloud to provide real-time protection against polymorphic malware. Expert input from researchers, advanced technologies like Antimalware Scan Interface (AMSI), and rich intelligence from the Microsoft Intelligent Security Graph continue to enhance next-generation endpoint protection platform (EPP) capabilities in Windows Defender Advanced Threat Protection.

In addition to antivirus, components of Windows Defender ATP’s interconnected security technologies defend against the multiple elements of social engineering attacks. Windows Defender SmartScreen in Microsoft Edge (also now available as a Google Chrome extension) blocks access to malicious URLs, such as those found in social engineering emails and documents. Network protection blocks malicious network communications, including those made by malicious scripts to download payloads. Attack surface reduction rules in Windows Defender Exploit Guard block Office-, script-, and email-based threats used in social engineering attacks. On the other hand, Windows Defender Application Control can block the installation of untrusted applications, including malware payloads of intermediary downloaders. These security solutions protect Windows 10 and Windows 10 in S mode from social engineering attacks.

Further, Windows Defender ATP endpoint detection and response (EDR) uses the power of machine learning and AMSI to unearth script-based attacks that “live off the land”. Windows Defender ATP allows security operations teams to detect and mitigate breaches and cyberattacks using advanced analytics and a rich detection library. With the April 2018 Update, automated investigation and advance hunting capabilities further enhance Windows Defender ATP. Sign up for a free trial.

Machine learning also powers Office 365 Advanced Threat Protection to detect non-PE attachments in social engineering spam campaigns that distribute malware or steal user credentials. This enhances the Office 365 ATP comprehensive and multi-layered solution to protect mailboxes, files, online storage, and applications against threats.

These and other technologies power Microsoft 365 threat protection to defend the modern workplace. In Windows 10 April 2018 Update, we enhanced signal sharing across advanced threat protection services in Windows, Office 365, and Enterprise Mobility + Security through the Microsoft Intelligent Security Graph. This integration enables these technologies to automatically update protection and detection and orchestrate remediation across Microsoft 365.

Gregory Ellison and Geoff McDonald
Windows Defender Research


Talk to us

Questions, concerns, or insights on this story? Join discussions at the Microsoft community and Windows Defender Security Intelligence.

Follow us on Twitter @WDSecurity and Facebook Windows Defender Security Intelligence.