Posted on Leave a comment

New Garage project Earth Lens pairs AI with aerial imagery to aid disaster relief, environment

This summer, Microsoft’s AI for Earth team and the office of the AI CTO decided to challenge a group of Garage interns to pair AI with aerial images to make a difference. Today, we announce the release of their summer project: Earth Lens, a Microsoft Garage project, a Xamarin-based, open source project for iPad that identifies, tracks, and analyzes objects in aerial imagery to assist in scenarios such as disaster relief and environmental conservation. Earth Lens releases on the heels of Microsoft’s announcement to commit over $40 Million to new initiative to leverage AI in efforts to save lives, joining a suite of programs known as AI for Good, including AI for Earth, AI for Accessibility, and now, AI for Humanitarian Action.

Earth_Lens_Screenshot_2Imagine yourself as a data analyst working to identify objects in aerial imagery. Maybe you’re tracking fishing vessels through the ocean to support ocean sustainability and protect marine ecosystems, or perhaps you’re looking at cities that have been hit by a natural disaster to determine where to allocate your relief efforts. In either case, you would need a dedicated team to comb through thousands of satellite images, manually identifying, tracking, and analyzing relevant objects. Extracting insights from imagery data is an expensive process that could take upwards of days, delaying the decision-making process in mission critical situations.

Inspired to make this process more efficient, the two co-sponsors decided to team up to pitch Garage interns on tackling this problem by leveraging AI. The Garage Internship Program offers teams of 5-6 university students the opportunity to design and build their own project in response to a challenge by sponsoring Microsoft groups. The Vancouver-based group of interns loved the AI for Good pitch. “In the start of May, our sponsors came to us with an idea that immediately sparked interest among the six of us: an opportunity to leverage AI to help with environmental efforts headed by researchers and conservationists,” shared Michelle Chen, Program Manager intern for Earth Lens.

AI for Good

The team was especially inspired by the work of the organizations helping victims of natural disasters by providing satellite imagery to hasten relief. She continues, “Our vision for Earth Lens was to use technology and AI to transform the way humanitarian work is conducted. The rise of automated image recognition has empowered humanitarian organizations to triage damage, and prioritized areas that require immediate help.”

Equipped with a trained machine learning model from the AI for Earth team, the interns set out to build a minimally viable product over the course of their remaining 16-week internship. They approached design with a customer obsessed mindset and interviewed prospective users like to the Red Cross, OceanMind, and FarmBeats to identify useful features and iterate on their prototype. The team quickly realized that, in addition to disaster relief, such a tool might be useful in a variety of social impact and sustainability applications such as agriculture, forestry, fishing, and land conservation.

The team built an iOS app that offers a host of features that humanitarians and environmental researchers can leverage to accelerate their analyses:

• Automatically identify, classify, and label objects in satellite images, encircling like objects in color-coded bounding boxes
• Count and aggregate objects identified by the machine learning model
• Toggle classes on/off to focus in on specific objects
• View images over time alongside a data visualization chart to identify trends and patterns in a Time Series mode
• Use the app remotely, without an internet connection

With the Earth Lens source code and corresponding instructions now available on GitHub, researchers will be able to build their own iOS apps that automatically extract information from a large dataset and scan that analysis in a useful and interactive view. As a result, this open source project can be useful in various industrial or agricultural applications to further Microsoft’s commitment to sustainability. In a disaster relief scenario, another minute could mean another life lost.

An Internship Where You Can Make an Impact

Earth_Lens_Team_ImageThis Garage project illustrates how developers can use AI to make a difference and the team hopes they’ve built something that will spark humanitarian creativity in others. “We all had the privilege and opportunity to experience what it means to work at Microsoft and be a part of something larger than us,” notes Seara Chen, a Software Developer intern who worked on model integration. “Our entire team agrees that this internship was an amazing experience filled with learning, surprises and a lot of problem-solving.”

Interested in trying Earth Lens? Check out the source code on GitHub, where you can also find instructions on how to build the iPad application and leverage the machine learning model that powers the project. You can learn more or provide feedback via GitHub.

The Garage Internship Program is hiring. You can learn more about openings and explore past Garage intern projects: Seeing AI, Ink to Code, Mobile Chest X-ray.

Posted on Leave a comment

MileIQ team launches new Microsoft Garage project to simplify expense management

In the fall of 2015, a startup team committed to making mileage logging easier joined Microsoft to expand their ability to empower the self-directed worker. Today, the team that created MileIQ now announces an iOS app to simplify expense management. Spend, a Microsoft Garage project, is now available for download in the US App Store.

The Microsoft Garage has released dozens of projects in the four years following the October 2014 launch of its Garage project program. Several projects have gone on to become features of flagship Microsoft products or new, branded products in their own right. Both self-organized, grassroots teams and arms of Microsoft product groups have leveraged the program to collect user feedback, add new features, and refine their approach.

Spend joins a host of Garage projects that have narrowed in on a specific target customer to understand their needs and build scenarios that tackle acute problems. Sports Performance Platform worked with the Seattle Seahawks, Benfica, Cricket Australia, and Real Madrid among other major sports teams and organizations. Video Indexer—the once-titled Video Breakdown and Garage project alum—synthesized multiple Microsoft Cognitive Services into an improved experience and is now offered as a standalone Azure Media Service. Microsoft Kaizala began its journey focusing on mobile collaboration scenarios, getting its big break working with the Indian state government to organize a massive holiday event attended by over 20 million people.

Hassle-Free, On-the-go Expense Management

Spend is mobile-first and built with the user in mind, making it simple to track expenses for reimbursements or taxes

• Quickly manage all your purchases for your expense reports with automatically tracked expenses from a connected credit card, debit card or bank account
• See purchases in a feed and easily classify expenses as business or personal with a single swipe
• Edit purchases or bulk classify expenses through the web dashboard
• Create accurate reports with only a few clicks for the week, month, or another customer period
• Fully customize reports which are available in either spreadsheet or PDF, commonly-used formats compatible with leading accounting and expense management software
• Snap pictures of receipts and attached to purchase with additional features for easy cash purchase management
• Track confidently: Spend uses 256-bit encryption, bank-level security, with Microsoft certifications

Spend, a Microsoft Garage project

The team has long focused on creating solutions that simplify work and empower self-directed workers, and built its first product MileIQ to enhance the mileage logging experience. When you’re on the road a lot, it can be time-consuming and chaotic to organize mileage and gas information for reimbursements or tax deductions. Now, the team is also turning its attention to another pain point for this audience: expense management.

“Keeping, sorting, and tracking paper receipts is annoying and inefficient. Spend uses intelligent features to bring receipt and expense tracking to the modern era,” says Heman Chawda, Product Manager for Spend. Built for mobile and designed around those always on the go, Spend takes a fresh approach to expense management. “It’s a great opportunity for us to be able to explore this space,” shared Nat Robinson, General Manager of MileIQ. “We are a growing team at Microsoft working on a number of experiences that empower small and medium businesses. The Microsoft Garage gives us the chance to offer a simplified expense management experience and really hone the value we provide. This is an important area of investment for us—we’re really excited to take this first step.”

Posted on Leave a comment

Microsoft New England teams up with customers to create high-impact product features

Microsoft Garage has been a force in affecting positive change within the company, practicing new ways of thinking and spreading a growth mindset culture to employees as well as partners outside the company. Microsoft’s global Hackathon is a key player in CEO Satya Nadella’s culture change priorities driving innovations that empower customers to achieve more. The Garage at NERD, located in Cambridge, MA is one of many Garage locations worldwide making big strides in challenging conventional ways while furthering business impact.

Garage NERD Hackathon 2018 Science Fair

The Garage at NERD during the Hackathon 2018 Science Fair

The New England Research and Development Center (NERD) is steeped in a rich community of forward-thinking institutions that are molding some of the best minds of the future. One of the newer Garage locations, The Garage – NERD is in a unique position of having the perfect vantage point to measure the pulse of research giants like MIT right next door and invite these institutions and local Boston communities into the Garage space to collaborate and share learnings.

“It’s about the people,” said Linda Thackeray, Director of The Garage at NERD. “We’re here to help employees and teams drive employee-led innovation at the company, for products and other areas employees are passionate about. We do that by creating opportunities to experiment and collaborate, not only within the Microsoft family, but with the local community as well. Look where we are! The greater Boston area is loaded with unique and robust talent and industry tech.”

The product development teams at NERD are at the forefront of the customer obsession initiative. The Docs Collaboration team took advantage of Microsoft’s annual Hackathon to create a proof of concept that could change the way Office builds app features. Ben Wilde, principal program manager on the Docs team, led the hack project which focused on adding key features to Office apps to enable richer and smarter collaboration. “A handful of us have been on a bit of a crusade to ensure that we’re solving real, high-impact customer problems that we’ve identified during actual conversations with our users, understanding what challenges they face when collaborating and coming up with creative solutions to eliminate their friction.” Though the project was about Office collaboration features, their team’s journey started months before Hackathon when the Docs team as a larger group began talking with customers in a more concentrated effort to get to the root of customer needs.

“For me personally the Hackathon has emboldened me and proved that the process works – I feel even more convinced now. This is something we’re all passionate about and we truly believe this is how we should be driving features.”

Wilde and team documented all their learnings and research, including customer conversations, and how they approached customer research as a process. A playbook of how to build features that solve actual customer problems, with data and feedback to validate what features should be prioritized. “At a high level, the process we went through started with building empathy and understanding user needs – the foundational sit-down-and-talk-to-customers approach. And more important than talking is listening, understanding where they feel friction, where things work and don’t work,” Wilde explained. From those conversations the team created feature concepts that they showed to customers and investigated which were the most impactful for collaboration. Those ideas became the key features that were prototyped during Hackathon and showcased at The Garage NERD Science Fair at the end of Hackathon, where their project won the Business Impact award.

Hackathon Project Team Photo- Cambridge, MA Science Fair

Hackathon 2018 Science Fair, Business Impact award winners at Cambridge, MA: Terrell Cox (NERD Site GM), Lyndsy Stopa, Ben Wilde, Daniel Chattan, Anunaya Pandey, Garrigan Stafford, with David Ku, CVP

Lyndsy Stopa, a senior software engineer on the Docs team, was excited to talk about how the work they did began a larger conversation around what would need to happen to get these customer-backed features into product. “The more people that see this successful proof of concept the better. If you asked us, we’d love to build these features into the product tomorrow. At the same time, we also know the limitations.” While details of the features can’t be revealed, the work they did exploring the existing codebase and hooking up other Microsoft technologies for a working prototype, in true hack fashion, allowed them to think about possible implementation paths.

Anunaya Pandey, software engineer on the project, recalled the many discussions they had while showing their work to colleagues. “The fact that it was a hackathon meant that we could focus on what the experience for the customer was going to be like, versus how this was all to be implemented as part of a product. And that’s generally the opposite of how we traditionally do things. We wanted to be able to show how this was going to look inside of an app.”

“Hackathon for us is one step in a larger journey where we’re trying to shift culture and get people on board with this approach to building features.” Daniel Chattan, principal software engineer also on the Docs team, shared other takeaways. “We had an intern, Garrigan Stafford, who jumped in to work on the hack project. He was from a different product team, didn’t know our code base at all – and he was able to make changes very quickly. It challenged a lot of our preconceptions that there was a big on-ramp to be able to do these things and for new people to get into this space.”

The project also used Microsoft cognitive services and machine learning to interpret and infer user intent while collaborating and suggest actions at opportune moments. More recently as a larger group, the Docs team has expanded to include more data scientists with a machine learning focus. Chattan explained how they got more out of cognitive services than functional impact. “This was a great way to demonstrate thinking differently about how you can pull these things like AI and ML into thinking about a feature. Rather than think, ‘OK, we have this feature, now how do we fit cognitive services into it?’ the approach that we presented is ‘We have a feature we’re building that’s trying to accomplish something specific for the user, how can cognitive services help the user as opposed to helping the feature?’ It leads to a more natural connection of those two things.”

Project team photo, casual talking in Garage Maker space

Garrigan Stafford, Anunaya Pandey, Ben Wilde, and Daniel Chattan in the Garage – NERD Maker space

It’s about a different mindset, open to change, embracing experimentation, and validating with customers. All are values that the project team is championing within their immediate organization, and values that Garage continues to foster within the company.

Post-hackathon, the team is also seeing a shift happening in their work group in how people are approaching feature development by being more incremental. “Let’s get to a minimum bar functionality,” explained Chattan, “then iterate again, then iterate again, which allows us to move very quickly and see the results of what we’re building and do fast course-corrections. We’re building more incrementally to incorporate user learning into the process, so we can learn as we go instead of creating a big monolith that a user sees for the first time only after it’s finished.”

It may sound like a simple idea, practicing this level of customer-obsession, but when trying to steer feature development there are other factors that come into play, such as demand for resources, and prior commitments that must be brought to completion before any new work can begin. The project team will continue their customer-validation approach for new iterations of the hack project, while relentlessly pursuing the same in their day jobs for current features they’re working on.

“For me personally the Hackathon has emboldened me and proved that the process works – I feel even more convinced now. This is something we’re all passionate about and we truly believe this is how we should be driving features.” Wilde explained. This begs the question then – how does management feel about it? “Management is in spiritual agreement. Everyone we’ve talked to is supportive and clearly sees the value of the features and the approach. But it’s difficult to say right now if we can pull together the necessary resources.”

With time, the team remains optimistic that their features, and their processes, will be embraced more widely as they continue their journey. Just as Wilde and his team are on a mission to affect culture-change in product development within Office, The Garage pursues culture-change for the entire company, so individuals, groups and organizations can provide greater business impact using Garage programs. This week, The Garage at NERD opened its doors to HUBWeek, a weeklong Boston festival where entrepreneurs, artists, and academics come together to collaborate on creating an inclusive and diverse future for art, science, and tech. Another example of the connective force of The Garage, and how a vibrant community continues to grow, full of doers, thinkers, and change-makers that will shape the next big ideas.

Read more about The Garage – NERD, check out a project shipped by Garage interns, and learn more about internship opportunities.

Posted on Leave a comment

Microsoft Hackathon 2017 winner powers Mixer’s massively successful HypeZone

HypeZone, released in December 2017, rapidly gained millions of new users to livestream community Mixer. HypeZone’s secret weapon? The 2017 Hackathon Grand Prize Winner, Watch For, a Microsoft Garage project.

Last month, Microsoft’s fifth annual One Week Hackathon wrapped up with astounding numbers. This year, during the largest private hackathon on the planet, over 23,000 employees registered to hack, and ultimately created 5,800 projects. As judging for this year’s projects begins and eager hackers await the winner announcements, it’s the perfect time to reconnect with last year’s Grand Prize Winner.

Originally called Lookout, the project team now known as Watch For has made tremendous strides in both personal growth and Microsoft business growth. Over the past year, team members Lenin Ravindranath Sivalingam, Matthai Philipose and Peter Bodik have been working as an incubation startup within Microsoft Research with autonomy and ownership to steer their project in a direction they desire.

The team’s original idea, which won the 2017 Hackathon, was an app to monitor live video streams on behalf of a user and notify him or her when specified events occur. Such a seemingly simple idea can be very powerful using artificial intelligence with many different applications.

2017 Hackathon winning team: Hackathon 2017 winning team: Matthai Philipose, Lenin Sivalingam, Yifan Wu, Peter Bodik and Victor Bahl. (Photo by Elizabeth Ong)
Hackathon 2017 winning team: Matthai Philipose, Lenin Sivalingam, Yifan Wu, Peter Bodik and Victor Bahl. (Photo by Elizabeth Ong)

As part of Microsoft Research, the project team members previously worked on video analytics for enterprise scenarios in their day jobs. One of their biggest partners was working with the city to monitor and analyze traffic cameras for a better understanding of how pedestrians, bikes, and vehicles crossed intersections.

Not surprisingly, livestreams are big in enterprise settings, and that translates as well to consumer settings. For Lenin, Matthai, and Peter, the most interesting part of working on a hack project was experimenting with how best to apply video analysis to consumer scenarios.

“What attracted me to this hackathon project was the chance to apply AI in large scale and at low-cost to the consumer setting. Our project really pushes the envelope on how efficient the AI systems would need to be, and it’s also meaningful in that my kids and mother can understand it and use it.” Matthai explained, adding, “And I love the idea of working with Lenin and Peter.”

The team took what they learned over the years about video analytics and traffic cams, and created such a compelling project that not only did Microsoft CEO Satya Nadella put his influence behind them, but the senior leadership team took notice and became excited about the possibilities. Ed Essey, principal program manager of Microsoft Garage, helped prepare the team to think and work like a lean startup.

Over the course of several months, they fine-tuned a business strategy for their product – including the team’s special blend of expertise, knowledge, experience, and idea-leadership – that led the team to work on Watch For full time.
In September 2017, a few weeks after the team’s Hackathon win, the Mixer group reached out to the team, having seen their project video. Mixer, acquired by Microsoft in 2016 as Beam, is a next-generation, interactive live streaming platform with a large gaming audience.

Taking a community-first focus on features, Matt Salsamendi, principal software engineering lead, Mixer and Chad Gibson, general manager, Mixer saw huge opportunity to accelerate Mixer’s vision in the computer vision space and were excited to partner with other Microsoft teams working in this area.

HypeZone Fortnite

The more popular games on Mixer tend to be multiplayer battle-royale style competitions where the last person standing wins. “Games like PlayerUnknown’s Battlegrounds (PUBG) and Fortnite are pretty new. For these games, a very simple thing works very well to light up Mixer scenarios.” Peter explained.

The scenario that Matt and Chad of Mixer wanted to execute on was how best to surface the most interesting parts of streams to a bigger audience. There are thousands of streams at any given time, of which only a couple hundred get viewed by most people. How do the rest of the streamers get any visibility and how do you avoid wasting those assets? How do Mixer fans discover those hidden gems? “The game streaming ecosystem has lots of undiscovered content, people wanting to be discovered, and viewers wanting to discover more compelling moments.”

“The game streaming ecosystem has lots of undiscovered content, people wanting to be discovered, and viewers wanting to discover more compelling moments.”

Lenin, Matthai, and Peter started to work closely with the Mixer team last September, and an ambitious goal organically formed, of launching new channels in winter of 2017 tailored with content discovered by AI models trained to “Watch For” specific events in streams. The timing coincided with PUBG’s release on Xbox One, which was fast becoming one of the most popular games on Mixer.

Mixer already had a front-end design where a single channel could host many different people’s streams continuously – they took advantage of that, and queried Watch For’s backend to determine when to switch between streams for the most interesting content. Thus, HypeZone was born – channels on Mixer using Watch For algorithms to highlight the final, nail-biting rounds of last-person standing games like PUBG that viewers found so engaging to watch.

“Matt already had the idea of HypeZone itself, to switch from stream to stream within a channel – but the experience of HypeZone evolved very quickly during our collaboration.” Lenin recalled. “We met with Matt and Chad early September. Two weeks later we had a prototype that we showed them. Then we kept improving its accuracy. By mid-October we had another prototype that they could use to run their HypeZone experience. We tested it for another 3 weeks. Then, 2 days before release, PUBG changed their UI. 1 day before release, we had to completely change all our models.”

Despite the whirlwind of activity, the Watch For team appreciated Mixer’s style of working fast and friendly. “As a business group, Mixer is very agile and easy to work with. We work close and we work well together.”


“The choice of content for HypeZone is determined by all the analysis Watch For does. Which is one of the reasons why we were able to move so fast,” Peter explained. Peter and team had to tailor their AI models for HypeZone by building core video analytics skills specific to each game.

Over the last several months, HypeZone channels were among the most popular channels on Mixer. “It’s a win-win product. Viewers love it because it shows only the most exciting content, and streamers love it because they get featured on Mixer’s front page and get new followers. They start streaming more because they want to be featured on HypeZone and gain followers.” Game producers can also be counted among the many fans as HypeZone provides more exposure for their games.

The biggest challenge – and the team’s biggest accomplishment – was how to get HypeZone to scale, and at low-cost.

“HypeZone is driven by Watch For’s large-scale video analysis of every stream that’s coming into Mixer. Every stream we try to understand what’s on the screen. We look for various metadata that tell us the game is exciting. Text on the screen, icons that tell you state of the game, player stats and score. Over time we have evolved to understand more and more.” Lenin explained.

The secret sauce is very much a combination of Matthai’s AI expertise and Lenin and Peter’s end-to-end distributive systems knowledge that allows them to deeply and efficiently analyze and understand each stream’s content in real-time.

“This is one of the advantages of being in a company like Microsoft. The Garage and Hackathon gave us visibility, but there was a product group (Mixer) out there looking around who had a great understanding of their customers, and that Watch For might light up their market.” Matthai recalled how it all came together. “There was an element of luck that battle royale type games came into vogue around the same time. It’s a combination of all of these things that made this partnership work so well.”

“It’s one thing to have cool demos and enthusiasm from senior leadership, but it’s another thing to see our customers enjoying, laughing and crying , wanting to see more. That’s what really lit a fire under the whole project, that connection.”

A game-changer for streaming content platforms and how content can be surfaced and consumed – Watch For is a stellar example of using artificial intelligence for consumer scenarios. What’s next for Watch For? The team continues to work with Mixer, and other groups, to create awesome experiences yet to come using the power of AI.

Story by Meixia Huang

Check out HypeZone on Mixer
Get videos on the Mixer Channel One on YouTube
Follow Mixer on Twitter:
Read more about this Hackathon team:
Artificial intelligence eclipses cloud and mobile projects to win the day at Microsoft 2017 Hackathon

Posted on Leave a comment

New: Use your computer and phone together in video meetings

Research into Computer-Supported Collaborative Work has explored problems of disengagement in video meetings and device conflict since the 1990s, but good solutions that could work at scale have been elusive. Microsoft Research Cambridge UK had been working on these issues when the 2015 Hackathon arose as an opportunity to highlight for the rest of the company that just a few simple and dynamic device combinations might provide users with the means to solve the issues themselves.

While we had explored some research prototypes in late 2014 and early 2015, for the Hackathon we decided to use a vision video with the goal of getting the attention of the Skype product group, because we knew that the idea would have the most impact as an infrastructural feature of an existing product rather than as a new stand-alone product. We called the video “Skype Unleashed” to connote breaking free of the traditional one person per endpoint model.

team in a conference room
Turning the hackathon video into a working proof-of-concept

When we won the Business category, our prize was meeting with the sponsor of the Business category, then-COO Kevin Turner.  We scrambled to build a proof-of-concept prototype, which at first we jokingly referred to as “Skype Skwid”, a deliberate misspelling of “squid”, because it was like a body that had lots of tentacles that could reach out to different other things. However, we realized that we needed an official project name, so we became “Project Wellington”. This was a related inside joke, because the largest squid in the world is the Colossal Squid, and the largest specimen in the world is in the Museum of New Zealand Te Papa Tongarewa… in Wellington, New Zealand.

So as Project Wellington we went to meet Kevin Turner, who also invited Gurdeep Singh Pall, then-CVP for Skype, in November 2015. Both immediately saw the relevance of the concepts and Gurdeep connected us to Brian MacDonald’s incubation project that would become Microsoft Teams.

Brian also understood right away that Companion Experiences could be an innovative market differentiator for meetings and a mobile driver for Teams. He championed the integration of our small Cambridge group with his Modern Meetings group as a loose v-team. The Modern Meetings group was exceptionally welcoming, graciously showing us the ropes of productization and taking on the formidable challenge of helping us socialize the need for changes at all levels of the product, from media stack, middle tier, and all clients. We, in turn, learned a lot about the cadence of production, scoping, aligning with the needs of multiple roadmaps, and the multitude of issues required to turn feature ideas into releasable code.Through 2016 and 2017 we worked on design iterations, usability testing, and middle tier and client code. We were thrilled when first glimpses of roving camera and proximity joining were shown at Build 2017, and then announced as officially rolling out at Enterprise Connect 2018.

a group of people in a conference room
The combined research and product team

We are very excited to see these features released. We are also excited to close the research loop by evaluating our thesis that dynamic device combinations will improve hybrid collaboration in video meetings, and doing research ‘in the wild’ at a scale unimaginable by most research projects. Microsoft is one of only a handful of institutions that can make research possible that will improve the productivity of millions of people daily. So as well as releasing product features, we are exceptionally proud of the model of collaboration itself. And, indeed, we are continuing to collaborate with Microsoft Teams even after these features are released, as we now have a tremendous relationship with a product group that understands how we work and values our help.

To come full circle, then, it was Satya Nadella’s emphasis on the Hackathon as a valuable use of company time, and The Garage’s organization of the event itself, that allowed ideas well outside a product group to be catapulted to the attention of people who could see its value and then provide a path to making it happen.

If you would like to find out more about this project, connect with Sean Rintel on LinkedIn or follow @seanrintel on twitter.