post

By devs, for devs: Meet new Garage projects XAML Studio and Team Retrospectives

Today, we announce two new Garage projects built by developers, for developers. Two teams, inspired by their roles building developer tools, worked on ideas to enhance the developer experience during Microsoft’s annual Hackathon. XAML Studio, a Microsoft Garage project, makes iterative prototyping easy with real-time UI module changes. Team Retrospectives, a Microsoft Garage project, closes the loop on continuous iteration, enabling an Agile retrospectives experience within Azure DevOps. Whether devs are starting a new project or reflecting on how to work more efficiently as a team, they can use these projects to get a little more productive.

From Design to Development

XAML Studio enables developers to rapidly prototype UWP XAML by previewing and adjusting XAML in real-time before easily copying it over into Visual Studio.
Having spent the better part of a decade listening to developers around the world and as a developer himself, Michael Hawker has spent a lot of time thinking about how to make dev tools even more useful. Michael is a Senior Software Engineer working on the Partner, Analytics, and Essential Experiences, or PAX, team and contributes to the Windows Community Toolkit. He found himself wanting a faster way to edit XAML UI and set out to build a project that could do so in Microsoft’s 2017 Hackathon. The annual event gives Microsoft employees the opportunity to explore and build new ideas. Several teams continue working on the projects beyond the week-long experience to make their way into the hands of customers via Garage projects.
Xaml Studio projectMichael created an interactive XAML sandbox to test out small code changes without having to compile or rebuild projects first, much like what WPF developers had access to for years. “When you’re in Visual Studio, you have a variety of tools available to make coding and laying out your UI easy, but it’s not always straightforward to visualize the final result. So, I thought to myself, ‘What could I do to add to this experience?’” shared Michael, the creator of the project. “XAML Studio is about that quick prototyping loop. Using this tool, I can type my XAML and see how it renders right away. Then, I can interact with it and explore how it behaves before it goes in my app.”
XAML Studio follows a number of Garage projects that facilitate rapid prototyping. Launched in the January 2018, Garage intern project Ink to Code, enables developers to sketch wire frames and kickstart their app in Visual Studio. Both Ink to Code and XAML Studio help developers transition from design to code by creating concrete development assets that can be used in Visual Studio. Where Ink to Code provides a way to create several prototypes very quickly, reducing repitive steps across multiple UI sketches, XAML Studio accelerates prototyping through easy-to-see iteration. Developers can now quickly fine-tune their prototypes, aided by an interactive preview window.
XAML Studio also hosts a variety of tools, including a binding debugger that can illustrate how UI is going to use the data behind it and if problems arise, where those might be occurring. Check out the full set of features.

By Developers, For Developers

The team behind Team Retrospectives also got their start at the One Week Hackathon, inspired by their roles in the Azure Production Infrastructure Engineering organization, or Azure PIE, where they focus on ways to improve the Azure developer experience. “We knew we wanted to work on something that would help devs and tackle something that we could actually create a working prototype for during the timeframe of the Hackathon,” shared Andy Stumpp, Senior Software Engineering Lead in Azure PIE and the development lead on the project.
Photo of the Team Retrospectives team at the Microsoft Hackathon
True to the “One Week” name, the team landed on a simple idea that could make their own projects more productive and could be created in the space of a few days: building an experience that integrated Agile retrospectives into Azure DevOps.
Many Microsoft developer teams use Agile methods to build, deploy, and continuously improve upon their products, as well as create tools that empower developers to do the same. Azure DevOps is a set of modern development services that make it easier to plan, collaborate, and ship faster, and many of its features support Agile. “I love Azure DevOps. It’s a one-stop shop,” shares Perth Charernwattanagul, a Senior Software Engineer in Azure PIE and the technical lead for Team Retrospectives. “It has everything I need from starting a project to releasing it; I don’t need 5 or 10 different products, I can just use Azure DevOps, and everything is streamlined and coordinated.”

Closing the Agile Loop

The team loved using Azure DevOps to organize their iterations, the first phase of Agile in which developer teams determine and execute work items, but they found that they also wanted to integrate retrospectives, the closing step in Agile, designed to fuel improvement and upcoming work items in the next iteration. “Retrospectives are a key part of the Agile process,” adds Andy. “They help you figure out what you can do better and the next steps in your new sprint.”

Team Retrospectives Screenshot

The Team Retrospective hackers created a project in Microsoft’s internal Hackathon tool and quickly attracted other developers around the company who shared a similar problem. They achieved their goal of creating a working prototype during the hackathon and immediately began testing it during their everyday roles. The team found that not only did Team Retrospectives streamline work items within Azure DevOps, it also made collaboration more efficient. “I’m based in Vancouver, and the rest of our team sits in our headquarters in Redmond,” Perth explains. “Our tool made it easy to collaborate digitally, and with the option to add our feedback in advance, we didn’t have to reschedule our meeting if someone couldn’t make it.”

Increased Collaboration and Reflection

Team Retrospectives is an Azure DevOps extension that enhances collaboration and productivity within the Azure Boards service:
  • Collect – Create a board, connect from any device and start submitting positive or negative feedback. The interface makes it easy to manage boards and feedback cards.
  • Group – Organize similar items into categories to facilitate discussion.
  • Vote – Prioritize feedback when team members vote on the items that are most important to them.
  • Act – Create follow-up action items in your team’s Azure DevOps project. Track previous work items and view roll-up status to ensure your team is progressing.
Since building and using Team Retrospectives, the team has found themselves becoming more effective. For example, shares Perth, “Having this extension baked right into Azure DevOps makes it effortless to have these retrospective conversations, and we’re simply prioritizing it more. We’ve become faster, more impactful, and more observant. We were able to hit a difficult deadline by quickly noticing that we were distributing too many work items to one person.”
“Team Retrospectives makes sure that we’re not just shipping features, but always learning from them.” –Jeff Braunstein, Senior Program Manager, Azure PIE and PM, Team Retrospectives
The team also rolled the project out to other Microsoft groups for feedback and testing, both within their own organization as well as to several teams working on the Microsoft Store. With this release, the team is eager to get feedback from developers beyond Microsoft, especially around how the team can support different Agile styles and methods. “Every team conducts retrospectives their own way; we want to know what customizable features can help teams perform at their best,” continues Perth.

Try XAML Studio & Team Retrospectives

post

Meet Ear Hockey: Newest Garage project infuses inclusive design into the classic arcade game

Three years ago, a team of Microsoft employees joined together on a quest to make gaming more fun and accessible to people who are blind or have low vision. After years of hacking on machine learning algorithms in their spare time, talking with customers in the blind and low vision community, and partnering with gaming experts, the team releases their first title: Ear Hockey, a Microsoft Garage project. With in-game audio cues that enable gamers to play with or without for visuals, Ear Hockey was crafted using inclusive design to promote accessibility in game development.

When spatial audio meets accessible game development

Ear Hockey is a Pong-style, first-person paddle game, where players use their keyboard or controller to hit the ball back and forth with a partner or computer player—but with a twist on the arcade classic air hockey: it uses unique spatial audio cues, powered by Microsoft’s HRTF spatial audio engine for Unity, to signal to the player where they are in the play-space and where the ball is traveling. The team was inspired by  and worked with Daniel Kish, an expert in echolocation and President of World Access for the Blind, to craft special sounds for ball movement and contact with the paddle or other play space surfaces to help orient players. For example, a fine-tuned combination of low-frequency humming and high-frequency ticking, along with other contextual sound effects, makes the ball easier to locate with audio alone.
As they developed Ear Hockey, the team also consulted with Senior Program Manager Brannon Zahad, who has over 16 years of experience in the gaming space and works on Accessibility R&D. “One of the biggest challenges we have in the industry right now is: when a developer wants to make a game, it’s really overwhelming to think about all the different things they need to do to make their game accessible,” describes Brannon. “For the longest time, there’s been a perception that it would be impossible to build a first-person perspective game accessible to people who are blind, for example. The work this team is doing to evolve those perceptions and build technology that empowers game developers is, frankly, incredible.”
The grassroots team, which goes by Audio Augmented Reality Gaming, or AARG, originally set out to tackle making games more accessible for people who are blind or have low vision. After winning the Ability Hack category at Microsoft’s annual Hackathon in 2017, they met with several people working towards similar goals across Microsoft, including the Microsoft Researchers who developed Project Triton. The spatial audio solution creates realistic reverb effects based on objects in a video game’s map using sophisticated machine learning algorithms. The AARG team recognized the potential for this kind of audio capability for accessible game development, and has since been partnering closely to explore how this could improve not only the gamer’s experience, but also the developer’s experience in building a more inclusive game. The game studio who developed Gears of War 4, The Coalition, used Project Triton to bring next-level sound to life, featuring the possibilities of advanced reverb effects; the AARG team quickly realized they could help inspire even more innovation in this area by building their own title.

Inclusive gaming

The Ear Hockey attends the CSUN Assistive Technlogy ConferenceAARG is comprised of a passionate set of team members from every corner of the company, many of whom have never formally worked on game development. To ensure Ear Hockey was a fun experience for both people who are blind or low vision, as well as people who are sighted, the team attended one of the world’s largest accessibility summits, the CSUN Assistive Technology Conference, where they showcased a few experimental games to members of the blind and low vision community. Players of all experience types provided feedback that helped shape the game. “For me, I realized the importance of orienting the player. We’d taken for granted how critical it is to provide that information at the beginning of the experience,” shared Patrick Farley, a Content Developer in the Cloud and AI group and one of the core architects of Ear Hockey. “Since then, we’ve enhanced our tutorial with a lot more detail.” For Boris Baracaldo, a Software Engineer and the other primary game developer, he noticed there was room for more customization with respect to ambient noise. Ear Hockey gamers can now adjust the volume of background music because of the invaluable customer feedback the team received.
In addition to soliciting feedback from customers and employees from the blind and low vision community, the team also designed Ear Hockey with a visual experience, subtitles, and support for the Xbox Adaptive Controller, another Hackathon project, in an effort to make the game more inclusive for a wide range of players. This has influenced how the team approaches developing for accessibility in their day-to-day activities. “I’m a Software Developer in Dynamics and this has completely shifted my approach to making products accessible,” continued Boris, who has found new ways to apply inclusive design to the products he works on in his full-time role.

Try it Out

With Ear Hockey now available worldwide, the team is eager to get feedback from players and drive a larger conversation around accessible gaming and game development. When asked about the team’s journey over the past 3 years, original organizer and Senior Mechanical Engineer Brandon Arteaga spoke of their excitement to offer this fun experience and help promote awareness about the power of echolocation. “We support Daniel Kish’s mission to teach echolocation as a skillset which can empower members of the blind and low-vision community to explore the world more freely. Anything we can do to shine a light on his work would be a dream come true.” Wilson Dreewes, a Senior Software Developer in AARG added, “Our goal is to showcase feature sets and what can be done in this space. We hope that the games we’re releasing now would be fun in and of themselves, but also that they inspire more innovation in game development.” So, have fun playing! Please try out Ear Hockey and share any feedback to the team on the game or accessible game development via UserVoice. You can also read about the full game features on the Garage Workbench.

post

New Garage project Earth Lens pairs AI with aerial imagery to aid disaster relief, environment

This summer, Microsoft’s AI for Earth team and the office of the AI CTO decided to challenge a group of Garage interns to pair AI with aerial images to make a difference. Today, we announce the release of their summer project: Earth Lens, a Microsoft Garage project, a Xamarin-based, open source project for iPad that identifies, tracks, and analyzes objects in aerial imagery to assist in scenarios such as disaster relief and environmental conservation. Earth Lens releases on the heels of Microsoft’s announcement to commit over $40 Million to new initiative to leverage AI in efforts to save lives, joining a suite of programs known as AI for Good, including AI for Earth, AI for Accessibility, and now, AI for Humanitarian Action.

Earth_Lens_Screenshot_2Imagine yourself as a data analyst working to identify objects in aerial imagery. Maybe you’re tracking fishing vessels through the ocean to support ocean sustainability and protect marine ecosystems, or perhaps you’re looking at cities that have been hit by a natural disaster to determine where to allocate your relief efforts. In either case, you would need a dedicated team to comb through thousands of satellite images, manually identifying, tracking, and analyzing relevant objects. Extracting insights from imagery data is an expensive process that could take upwards of days, delaying the decision-making process in mission critical situations.

Inspired to make this process more efficient, the two co-sponsors decided to team up to pitch Garage interns on tackling this problem by leveraging AI. The Garage Internship Program offers teams of 5-6 university students the opportunity to design and build their own project in response to a challenge by sponsoring Microsoft groups. The Vancouver-based group of interns loved the AI for Good pitch. “In the start of May, our sponsors came to us with an idea that immediately sparked interest among the six of us: an opportunity to leverage AI to help with environmental efforts headed by researchers and conservationists,” shared Michelle Chen, Program Manager intern for Earth Lens.

AI for Good

The team was especially inspired by the work of the organizations helping victims of natural disasters by providing satellite imagery to hasten relief. She continues, “Our vision for Earth Lens was to use technology and AI to transform the way humanitarian work is conducted. The rise of automated image recognition has empowered humanitarian organizations to triage damage, and prioritized areas that require immediate help.”

Equipped with a trained machine learning model from the AI for Earth team, the interns set out to build a minimally viable product over the course of their remaining 16-week internship. They approached design with a customer obsessed mindset and interviewed prospective users like to the Red Cross, OceanMind, and FarmBeats to identify useful features and iterate on their prototype. The team quickly realized that, in addition to disaster relief, such a tool might be useful in a variety of social impact and sustainability applications such as agriculture, forestry, fishing, and land conservation.

The team built an iOS app that offers a host of features that humanitarians and environmental researchers can leverage to accelerate their analyses:

• Automatically identify, classify, and label objects in satellite images, encircling like objects in color-coded bounding boxes
• Count and aggregate objects identified by the machine learning model
• Toggle classes on/off to focus in on specific objects
• View images over time alongside a data visualization chart to identify trends and patterns in a Time Series mode
• Use the app remotely, without an internet connection

With the Earth Lens source code and corresponding instructions now available on GitHub, researchers will be able to build their own iOS apps that automatically extract information from a large dataset and scan that analysis in a useful and interactive view. As a result, this open source project can be useful in various industrial or agricultural applications to further Microsoft’s commitment to sustainability. In a disaster relief scenario, another minute could mean another life lost.

An Internship Where You Can Make an Impact

Earth_Lens_Team_ImageThis Garage project illustrates how developers can use AI to make a difference and the team hopes they’ve built something that will spark humanitarian creativity in others. “We all had the privilege and opportunity to experience what it means to work at Microsoft and be a part of something larger than us,” notes Seara Chen, a Software Developer intern who worked on model integration. “Our entire team agrees that this internship was an amazing experience filled with learning, surprises and a lot of problem-solving.”

Interested in trying Earth Lens? Check out the source code on GitHub, where you can also find instructions on how to build the iPad application and leverage the machine learning model that powers the project. You can learn more or provide feedback via GitHub.

The Garage Internship Program is hiring. You can learn more about openings and explore past Garage intern projects: Seeing AI, Ink to Code, Mobile Chest X-ray.

post

MileIQ team launches new Microsoft Garage project to simplify expense management

In the fall of 2015, a startup team committed to making mileage logging easier joined Microsoft to expand their ability to empower the self-directed worker. Today, the team that created MileIQ now announces an iOS app to simplify expense management. Spend, a Microsoft Garage project, is now available for download in the US App Store.

The Microsoft Garage has released dozens of projects in the four years following the October 2014 launch of its Garage project program. Several projects have gone on to become features of flagship Microsoft products or new, branded products in their own right. Both self-organized, grassroots teams and arms of Microsoft product groups have leveraged the program to collect user feedback, add new features, and refine their approach.

Spend joins a host of Garage projects that have narrowed in on a specific target customer to understand their needs and build scenarios that tackle acute problems. Sports Performance Platform worked with the Seattle Seahawks, Benfica, Cricket Australia, and Real Madrid among other major sports teams and organizations. Video Indexer—the once-titled Video Breakdown and Garage project alum—synthesized multiple Microsoft Cognitive Services into an improved experience and is now offered as a standalone Azure Media Service. Microsoft Kaizala began its journey focusing on mobile collaboration scenarios, getting its big break working with the Indian state government to organize a massive holiday event attended by over 20 million people.

Hassle-Free, On-the-go Expense Management

Spend is mobile-first and built with the user in mind, making it simple to track expenses for reimbursements or taxes

• Quickly manage all your purchases for your expense reports with automatically tracked expenses from a connected credit card, debit card or bank account
• See purchases in a feed and easily classify expenses as business or personal with a single swipe
• Edit purchases or bulk classify expenses through the web dashboard
• Create accurate reports with only a few clicks for the week, month, or another customer period
• Fully customize reports which are available in either spreadsheet or PDF, commonly-used formats compatible with leading accounting and expense management software
• Snap pictures of receipts and attached to purchase with additional features for easy cash purchase management
• Track confidently: Spend uses 256-bit encryption, bank-level security, with Microsoft certifications

Spend, a Microsoft Garage project

The team has long focused on creating solutions that simplify work and empower self-directed workers, and built its first product MileIQ to enhance the mileage logging experience. When you’re on the road a lot, it can be time-consuming and chaotic to organize mileage and gas information for reimbursements or tax deductions. Now, the team is also turning its attention to another pain point for this audience: expense management.

“Keeping, sorting, and tracking paper receipts is annoying and inefficient. Spend uses intelligent features to bring receipt and expense tracking to the modern era,” says Heman Chawda, Product Manager for Spend. Built for mobile and designed around those always on the go, Spend takes a fresh approach to expense management. “It’s a great opportunity for us to be able to explore this space,” shared Nat Robinson, General Manager of MileIQ. “We are a growing team at Microsoft working on a number of experiences that empower small and medium businesses. The Microsoft Garage gives us the chance to offer a simplified expense management experience and really hone the value we provide. This is an important area of investment for us—we’re really excited to take this first step.”

post

Microsoft New England teams up with customers to create high-impact product features

Microsoft Garage has been a force in affecting positive change within the company, practicing new ways of thinking and spreading a growth mindset culture to employees as well as partners outside the company. Microsoft’s global Hackathon is a key player in CEO Satya Nadella’s culture change priorities driving innovations that empower customers to achieve more. The Garage at NERD, located in Cambridge, MA is one of many Garage locations worldwide making big strides in challenging conventional ways while furthering business impact.

Garage NERD Hackathon 2018 Science Fair

The Garage at NERD during the Hackathon 2018 Science Fair

The New England Research and Development Center (NERD) is steeped in a rich community of forward-thinking institutions that are molding some of the best minds of the future. One of the newer Garage locations, The Garage – NERD is in a unique position of having the perfect vantage point to measure the pulse of research giants like MIT right next door and invite these institutions and local Boston communities into the Garage space to collaborate and share learnings.

“It’s about the people,” said Linda Thackeray, Director of The Garage at NERD. “We’re here to help employees and teams drive employee-led innovation at the company, for products and other areas employees are passionate about. We do that by creating opportunities to experiment and collaborate, not only within the Microsoft family, but with the local community as well. Look where we are! The greater Boston area is loaded with unique and robust talent and industry tech.”

The product development teams at NERD are at the forefront of the customer obsession initiative. The Docs Collaboration team took advantage of Microsoft’s annual Hackathon to create a proof of concept that could change the way Office builds app features. Ben Wilde, principal program manager on the Docs team, led the hack project which focused on adding key features to Office apps to enable richer and smarter collaboration. “A handful of us have been on a bit of a crusade to ensure that we’re solving real, high-impact customer problems that we’ve identified during actual conversations with our users, understanding what challenges they face when collaborating and coming up with creative solutions to eliminate their friction.” Though the project was about Office collaboration features, their team’s journey started months before Hackathon when the Docs team as a larger group began talking with customers in a more concentrated effort to get to the root of customer needs.

“For me personally the Hackathon has emboldened me and proved that the process works – I feel even more convinced now. This is something we’re all passionate about and we truly believe this is how we should be driving features.”

Wilde and team documented all their learnings and research, including customer conversations, and how they approached customer research as a process. A playbook of how to build features that solve actual customer problems, with data and feedback to validate what features should be prioritized. “At a high level, the process we went through started with building empathy and understanding user needs – the foundational sit-down-and-talk-to-customers approach. And more important than talking is listening, understanding where they feel friction, where things work and don’t work,” Wilde explained. From those conversations the team created feature concepts that they showed to customers and investigated which were the most impactful for collaboration. Those ideas became the key features that were prototyped during Hackathon and showcased at The Garage NERD Science Fair at the end of Hackathon, where their project won the Business Impact award.

Hackathon Project Team Photo- Cambridge, MA Science Fair

Hackathon 2018 Science Fair, Business Impact award winners at Cambridge, MA: Terrell Cox (NERD Site GM), Lyndsy Stopa, Ben Wilde, Daniel Chattan, Anunaya Pandey, Garrigan Stafford, with David Ku, CVP

Lyndsy Stopa, a senior software engineer on the Docs team, was excited to talk about how the work they did began a larger conversation around what would need to happen to get these customer-backed features into product. “The more people that see this successful proof of concept the better. If you asked us, we’d love to build these features into the product tomorrow. At the same time, we also know the limitations.” While details of the features can’t be revealed, the work they did exploring the existing codebase and hooking up other Microsoft technologies for a working prototype, in true hack fashion, allowed them to think about possible implementation paths.

Anunaya Pandey, software engineer on the project, recalled the many discussions they had while showing their work to colleagues. “The fact that it was a hackathon meant that we could focus on what the experience for the customer was going to be like, versus how this was all to be implemented as part of a product. And that’s generally the opposite of how we traditionally do things. We wanted to be able to show how this was going to look inside of an app.”

“Hackathon for us is one step in a larger journey where we’re trying to shift culture and get people on board with this approach to building features.” Daniel Chattan, principal software engineer also on the Docs team, shared other takeaways. “We had an intern, Garrigan Stafford, who jumped in to work on the hack project. He was from a different product team, didn’t know our code base at all – and he was able to make changes very quickly. It challenged a lot of our preconceptions that there was a big on-ramp to be able to do these things and for new people to get into this space.”

The project also used Microsoft cognitive services and machine learning to interpret and infer user intent while collaborating and suggest actions at opportune moments. More recently as a larger group, the Docs team has expanded to include more data scientists with a machine learning focus. Chattan explained how they got more out of cognitive services than functional impact. “This was a great way to demonstrate thinking differently about how you can pull these things like AI and ML into thinking about a feature. Rather than think, ‘OK, we have this feature, now how do we fit cognitive services into it?’ the approach that we presented is ‘We have a feature we’re building that’s trying to accomplish something specific for the user, how can cognitive services help the user as opposed to helping the feature?’ It leads to a more natural connection of those two things.”

Project team photo, casual talking in Garage Maker space

Garrigan Stafford, Anunaya Pandey, Ben Wilde, and Daniel Chattan in the Garage – NERD Maker space

It’s about a different mindset, open to change, embracing experimentation, and validating with customers. All are values that the project team is championing within their immediate organization, and values that Garage continues to foster within the company.

Post-hackathon, the team is also seeing a shift happening in their work group in how people are approaching feature development by being more incremental. “Let’s get to a minimum bar functionality,” explained Chattan, “then iterate again, then iterate again, which allows us to move very quickly and see the results of what we’re building and do fast course-corrections. We’re building more incrementally to incorporate user learning into the process, so we can learn as we go instead of creating a big monolith that a user sees for the first time only after it’s finished.”

It may sound like a simple idea, practicing this level of customer-obsession, but when trying to steer feature development there are other factors that come into play, such as demand for resources, and prior commitments that must be brought to completion before any new work can begin. The project team will continue their customer-validation approach for new iterations of the hack project, while relentlessly pursuing the same in their day jobs for current features they’re working on.

“For me personally the Hackathon has emboldened me and proved that the process works – I feel even more convinced now. This is something we’re all passionate about and we truly believe this is how we should be driving features.” Wilde explained. This begs the question then – how does management feel about it? “Management is in spiritual agreement. Everyone we’ve talked to is supportive and clearly sees the value of the features and the approach. But it’s difficult to say right now if we can pull together the necessary resources.”

With time, the team remains optimistic that their features, and their processes, will be embraced more widely as they continue their journey. Just as Wilde and his team are on a mission to affect culture-change in product development within Office, The Garage pursues culture-change for the entire company, so individuals, groups and organizations can provide greater business impact using Garage programs. This week, The Garage at NERD opened its doors to HUBWeek, a weeklong Boston festival where entrepreneurs, artists, and academics come together to collaborate on creating an inclusive and diverse future for art, science, and tech. Another example of the connective force of The Garage, and how a vibrant community continues to grow, full of doers, thinkers, and change-makers that will shape the next big ideas.

Read more about The Garage – NERD, check out a project shipped by Garage interns, and learn more about internship opportunities.

post

Microsoft Hackathon 2017 winner powers Mixer’s massively successful HypeZone

HypeZone, released in December 2017, rapidly gained millions of new users to livestream community Mixer. HypeZone’s secret weapon? The 2017 Hackathon Grand Prize Winner, Watch For, a Microsoft Garage project.

Last month, Microsoft’s fifth annual One Week Hackathon wrapped up with astounding numbers. This year, during the largest private hackathon on the planet, over 23,000 employees registered to hack, and ultimately created 5,800 projects. As judging for this year’s projects begins and eager hackers await the winner announcements, it’s the perfect time to reconnect with last year’s Grand Prize Winner.

Originally called Lookout, the project team now known as Watch For has made tremendous strides in both personal growth and Microsoft business growth. Over the past year, team members Lenin Ravindranath Sivalingam, Matthai Philipose and Peter Bodik have been working as an incubation startup within Microsoft Research with autonomy and ownership to steer their project in a direction they desire.

The team’s original idea, which won the 2017 Hackathon, was an app to monitor live video streams on behalf of a user and notify him or her when specified events occur. Such a seemingly simple idea can be very powerful using artificial intelligence with many different applications.

2017 Hackathon winning team: Hackathon 2017 winning team: Matthai Philipose, Lenin Sivalingam, Yifan Wu, Peter Bodik and Victor Bahl. (Photo by Elizabeth Ong)
Hackathon 2017 winning team: Matthai Philipose, Lenin Sivalingam, Yifan Wu, Peter Bodik and Victor Bahl. (Photo by Elizabeth Ong)

As part of Microsoft Research, the project team members previously worked on video analytics for enterprise scenarios in their day jobs. One of their biggest partners was working with the city to monitor and analyze traffic cameras for a better understanding of how pedestrians, bikes, and vehicles crossed intersections.

Not surprisingly, livestreams are big in enterprise settings, and that translates as well to consumer settings. For Lenin, Matthai, and Peter, the most interesting part of working on a hack project was experimenting with how best to apply video analysis to consumer scenarios.

“What attracted me to this hackathon project was the chance to apply AI in large scale and at low-cost to the consumer setting. Our project really pushes the envelope on how efficient the AI systems would need to be, and it’s also meaningful in that my kids and mother can understand it and use it.” Matthai explained, adding, “And I love the idea of working with Lenin and Peter.”

The team took what they learned over the years about video analytics and traffic cams, and created such a compelling project that not only did Microsoft CEO Satya Nadella put his influence behind them, but the senior leadership team took notice and became excited about the possibilities. Ed Essey, principal program manager of Microsoft Garage, helped prepare the team to think and work like a lean startup.

Over the course of several months, they fine-tuned a business strategy for their product – including the team’s special blend of expertise, knowledge, experience, and idea-leadership – that led the team to work on Watch For full time.
In September 2017, a few weeks after the team’s Hackathon win, the Mixer group reached out to the team, having seen their project video. Mixer, acquired by Microsoft in 2016 as Beam, is a next-generation, interactive live streaming platform with a large gaming audience.

Taking a community-first focus on features, Matt Salsamendi, principal software engineering lead, Mixer and Chad Gibson, general manager, Mixer saw huge opportunity to accelerate Mixer’s vision in the computer vision space and were excited to partner with other Microsoft teams working in this area.

HypeZone Fortnite

The more popular games on Mixer tend to be multiplayer battle-royale style competitions where the last person standing wins. “Games like PlayerUnknown’s Battlegrounds (PUBG) and Fortnite are pretty new. For these games, a very simple thing works very well to light up Mixer scenarios.” Peter explained.

The scenario that Matt and Chad of Mixer wanted to execute on was how best to surface the most interesting parts of streams to a bigger audience. There are thousands of streams at any given time, of which only a couple hundred get viewed by most people. How do the rest of the streamers get any visibility and how do you avoid wasting those assets? How do Mixer fans discover those hidden gems? “The game streaming ecosystem has lots of undiscovered content, people wanting to be discovered, and viewers wanting to discover more compelling moments.”

“The game streaming ecosystem has lots of undiscovered content, people wanting to be discovered, and viewers wanting to discover more compelling moments.”

Lenin, Matthai, and Peter started to work closely with the Mixer team last September, and an ambitious goal organically formed, of launching new channels in winter of 2017 tailored with content discovered by AI models trained to “Watch For” specific events in streams. The timing coincided with PUBG’s release on Xbox One, which was fast becoming one of the most popular games on Mixer.

Mixer already had a front-end design where a single channel could host many different people’s streams continuously – they took advantage of that, and queried Watch For’s backend to determine when to switch between streams for the most interesting content. Thus, HypeZone was born – channels on Mixer using Watch For algorithms to highlight the final, nail-biting rounds of last-person standing games like PUBG that viewers found so engaging to watch.

“Matt already had the idea of HypeZone itself, to switch from stream to stream within a channel – but the experience of HypeZone evolved very quickly during our collaboration.” Lenin recalled. “We met with Matt and Chad early September. Two weeks later we had a prototype that we showed them. Then we kept improving its accuracy. By mid-October we had another prototype that they could use to run their HypeZone experience. We tested it for another 3 weeks. Then, 2 days before release, PUBG changed their UI. 1 day before release, we had to completely change all our models.”

Despite the whirlwind of activity, the Watch For team appreciated Mixer’s style of working fast and friendly. “As a business group, Mixer is very agile and easy to work with. We work close and we work well together.”

VictoryRoyale

“The choice of content for HypeZone is determined by all the analysis Watch For does. Which is one of the reasons why we were able to move so fast,” Peter explained. Peter and team had to tailor their AI models for HypeZone by building core video analytics skills specific to each game.

Over the last several months, HypeZone channels were among the most popular channels on Mixer. “It’s a win-win product. Viewers love it because it shows only the most exciting content, and streamers love it because they get featured on Mixer’s front page and get new followers. They start streaming more because they want to be featured on HypeZone and gain followers.” Game producers can also be counted among the many fans as HypeZone provides more exposure for their games.

The biggest challenge – and the team’s biggest accomplishment – was how to get HypeZone to scale, and at low-cost.

“HypeZone is driven by Watch For’s large-scale video analysis of every stream that’s coming into Mixer. Every stream we try to understand what’s on the screen. We look for various metadata that tell us the game is exciting. Text on the screen, icons that tell you state of the game, player stats and score. Over time we have evolved to understand more and more.” Lenin explained.

The secret sauce is very much a combination of Matthai’s AI expertise and Lenin and Peter’s end-to-end distributive systems knowledge that allows them to deeply and efficiently analyze and understand each stream’s content in real-time.

“This is one of the advantages of being in a company like Microsoft. The Garage and Hackathon gave us visibility, but there was a product group (Mixer) out there looking around who had a great understanding of their customers, and that Watch For might light up their market.” Matthai recalled how it all came together. “There was an element of luck that battle royale type games came into vogue around the same time. It’s a combination of all of these things that made this partnership work so well.”

“It’s one thing to have cool demos and enthusiasm from senior leadership, but it’s another thing to see our customers enjoying, laughing and crying , wanting to see more. That’s what really lit a fire under the whole project, that connection.”

A game-changer for streaming content platforms and how content can be surfaced and consumed – Watch For is a stellar example of using artificial intelligence for consumer scenarios. What’s next for Watch For? The team continues to work with Mixer, and other groups, to create awesome experiences yet to come using the power of AI.

Story by Meixia Huang

Check out HypeZone on Mixer https://mixer.com/
Get videos on the Mixer Channel One on YouTube
Follow Mixer on Twitter: https://twitter.com/WatchMixer
Read more about this Hackathon team:
Artificial intelligence eclipses cloud and mobile projects to win the day at Microsoft 2017 Hackathon

post

New: Use your computer and phone together in video meetings

Research into Computer-Supported Collaborative Work has explored problems of disengagement in video meetings and device conflict since the 1990s, but good solutions that could work at scale have been elusive. Microsoft Research Cambridge UK had been working on these issues when the 2015 Hackathon arose as an opportunity to highlight for the rest of the company that just a few simple and dynamic device combinations might provide users with the means to solve the issues themselves.

While we had explored some research prototypes in late 2014 and early 2015, for the Hackathon we decided to use a vision video with the goal of getting the attention of the Skype product group, because we knew that the idea would have the most impact as an infrastructural feature of an existing product rather than as a new stand-alone product. We called the video “Skype Unleashed” to connote breaking free of the traditional one person per endpoint model.

team in a conference room
Turning the hackathon video into a working proof-of-concept

When we won the Business category, our prize was meeting with the sponsor of the Business category, then-COO Kevin Turner.  We scrambled to build a proof-of-concept prototype, which at first we jokingly referred to as “Skype Skwid”, a deliberate misspelling of “squid”, because it was like a body that had lots of tentacles that could reach out to different other things. However, we realized that we needed an official project name, so we became “Project Wellington”. This was a related inside joke, because the largest squid in the world is the Colossal Squid, and the largest specimen in the world is in the Museum of New Zealand Te Papa Tongarewa… in Wellington, New Zealand.

So as Project Wellington we went to meet Kevin Turner, who also invited Gurdeep Singh Pall, then-CVP for Skype, in November 2015. Both immediately saw the relevance of the concepts and Gurdeep connected us to Brian MacDonald’s incubation project that would become Microsoft Teams.

Brian also understood right away that Companion Experiences could be an innovative market differentiator for meetings and a mobile driver for Teams. He championed the integration of our small Cambridge group with his Modern Meetings group as a loose v-team. The Modern Meetings group was exceptionally welcoming, graciously showing us the ropes of productization and taking on the formidable challenge of helping us socialize the need for changes at all levels of the product, from media stack, middle tier, and all clients. We, in turn, learned a lot about the cadence of production, scoping, aligning with the needs of multiple roadmaps, and the multitude of issues required to turn feature ideas into releasable code.Through 2016 and 2017 we worked on design iterations, usability testing, and middle tier and client code. We were thrilled when first glimpses of roving camera and proximity joining were shown at Build 2017, and then announced as officially rolling out at Enterprise Connect 2018.

a group of people in a conference room
The combined research and product team

We are very excited to see these features released. We are also excited to close the research loop by evaluating our thesis that dynamic device combinations will improve hybrid collaboration in video meetings, and doing research ‘in the wild’ at a scale unimaginable by most research projects. Microsoft is one of only a handful of institutions that can make research possible that will improve the productivity of millions of people daily. So as well as releasing product features, we are exceptionally proud of the model of collaboration itself. And, indeed, we are continuing to collaborate with Microsoft Teams even after these features are released, as we now have a tremendous relationship with a product group that understands how we work and values our help.

To come full circle, then, it was Satya Nadella’s emphasis on the Hackathon as a valuable use of company time, and The Garage’s organization of the event itself, that allowed ideas well outside a product group to be catapulted to the attention of people who could see its value and then provide a path to making it happen.

If you would like to find out more about this project, connect with Sean Rintel on LinkedIn or follow @seanrintel on twitter.