Tag: Artificial Intelligence

“I think with some confidence I can say that 2023 is going to be the most exciting year that the AI community has ever had,” writes Kevin Scott, chief technology officer at Microsoft, in a Q&A on the company’s AI blog. He acknowledges that he also thought 2022 was the most exciting year for AI, but he believes that the pace of innovation is only increasing. This is particularly true with generative AI, which doesn’t simply analyze large data sets but is a tool people can use to create entirely new works. We can already see its promise in systems like GPT-3, which can do anything from helping copyedit and summarize text to providing inspiration, and DALL-E 2, which can create useful and arresting works of art based on text inputs. Here are some of Scott’s predictions about how AI will change the way we work and play.

1. It Will Unleash Our Creativity

As generative AI becomes more popular and accessible, more people will be able to use the technology for creative expression, whether it’s helping them produce sophisticated artworks or write moving poetry. In his blog post, Scott describes how new AI tools are democratizing access to design. “An AI system such as DALL-E 2 doesn’t turn ordinary people into professional artists, but it gives a ton of people a visual vocabulary that they didn’t have before—a new superpower they didn’t think they would ever have.” DALL-E 2 already shows up in tools like Microsoft Designer, but there’s exciting potential for it to help many more people unleash creative ideas in ways that were once only available to trained professionals.

2. It Will Make Coding Much More Accessible

Generative AI innovations like GitHub Copilot, an AI pair programmer built using OpenAI’s Codex AI system, can translate natural human language into programming code, essentially turning our practical intentions into complex pieces of software. Among Copilot users, 40 percent of the code in some popular programming languages is being generated by Copilot, a figure that is set to increase. In a recent talk at the Fortune Brainstorm AI conference, Scott pointed to the example of people noodling around with the capabilities of ChatGPT (which is powered by GPT-3.5) to hint at the future potential. “It really opens up the aperture of who can actually use AI now,” he says. “We’ll need new sorts of specialties in the future, but you don’t need to have a PhD in computer science anymore to build an AI application, which I think is really, really exciting.”

3. It Will Become Our Copilot in Other Ways Too

In an essay for Wired UK, Scott sketches a scenario in which AI helps us do our jobs better. Like coding assistance with GitHub Copilot, industries from construction to healthcare, technology to law, could potentially benefit from a form of AI assistance. “The applications are potentially endless, limited only by one’s ability to imagine scenarios in which productivity-assisting software could be applied to complex cognitive work, whether that be editing videos, writing scripts, designing new molecules for medicines, or creating manufacturing recipes from 3D models,” he writes. While there’s concern about how AI will impact human jobs, Scott describes in his post how, with thoughtful application, these AI tools have the potential to augment and amplify human capability, enabling people to spend less time on repetitive tasks. These models will also “democratize access to AI,” he writes, so “you’ll have a more diverse group of people being able to participate in the creation of technology.”

4. It Will Unlock Faster Iteration

Generative AI may significantly reduce the legwork of the creative process by helping designers iterate on product concepts and helping writers generate first drafts of press releases, essays, and scripts, assisting with graphic design–heavy posters, video edits, and more. Scott notes in his Wired essay that it has the potential to “allow knowledge workers to spend their time on higher order cognitive tasks, and effectively trans­forming how a great many of us interact with technology to get things done.” The upshot is an acceleration of the iterative cycle, as human beings tweak and refine the AI’s work in a virtuous, back-and-forth collaborative process. We will become adept at developing techniques to edit and modify generated images, text, drawings, and even molecules or proteins to be used in new medicines, creating better results more quickly through careful collaboration with AI.

5. It Will Make Work More Enjoyable

In the AI blog post, Scott observes that AI tools for programmers have the potential to vastly improve the overall work experience. “People now have new and interesting and fundamentally more effective tools than they’ve had before,” he notes. “This is exactly what we’re seeing with the experiences developers are having with Copilot; they are reporting that Copilot helps them stay in the flow and keeps their minds sharper during what used to be boring and repetitive tasks.” This also extends to low-code and no-code tools in products like Power Platform that are opening new potential across job functions, roles, and processes. “We did a study that found using no-code or low-code tools led to more than an 80 percent positive impact on work satisfaction, overall workload, and morale.”

Generative AI has the capacity to profoundly alter the working practices of a range of vocations, giving rise to new professions and transforming established ones. With ethical and thoughtful deployment, it is a tool that could help precipitate a revolution in creativity—one that enables everyone to better express their humanity.

Microsoft is proud to be a platinum sponsor of the 36th annual conference on Neural Information Processing Systems (NeurIPS), which is widely regarded as the world’s most prestigious research conference on artificial intelligence and machine learning.

Microsoft has a strong presence at NeurIPS again this year, with more than 150 of our researchers participating in the conference and 122 of our research papers accepted. Our researchers are also taking part in 10 workshops, four competitions and a tutorial.

In one of the workshops, AI for Science: Progress and Promises, a panel of leading researchers will discuss how artificial intelligence and machine learning have the potential to advance scientific discovery. The panel will include two Microsoft researchers: Max Welling, Vice President and Distinguished Scientist, Microsoft Research AI4Science, who will serve as moderator, and Peter Lee, Corporate Vice President, Microsoft Research and Incubations.

Of the 122 Microsoft research papers accepted for the conference, seven have been selected for oral presentations during the virtual NeurIPS experience the week of December 4^th. The oral presentations provide a deeper dive into each of the featured research topics.

In addition, two other Microsoft research papers received Outstanding Paper Awards for NeurIPS 2022. One of those papers, Gradient Estimation with Discrete Stein Operators, explains how researchers developed a gradient estimator that achieves substantially lower variance than state-of-the-art estimators with the same number of function evaluations, which has the potential to improve problem solving in machine learning. In the other paper, A Neural Corpus Indexer for Document Retrieval, researchers demonstrate that an end-to-end deep neural network that unifies training and indexing stages can significantly improve the recall performance of traditional document retrieval methods.

Below we have provided the titles, authors and abstracts for all seven of the Microsoft research papers chosen for oral presentations at NeurIPS, with links to additional information for those who want to explore the topics more fully:

Uni[MASK]: Unified Inference in Sequential Decision Problems

Micah Carroll, Orr Paradise, Jessy Lin, Raluca Georgescu, Mingfei Sun, David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht, Anca Dragan, Sam Devlin

Abstract: Randomly masking and predicting word tokens has been a successful approach in pre-training language models for a variety of downstream tasks. In this work, we observe that the same idea also applies naturally to sequential decision making, where many well-studied tasks like behavior cloning, offline RL, inverse dynamics, and waypoint conditioning correspond to different sequence maskings over a sequence of states, actions, and returns. We introduce the UniMASK framework, which provides a unified way to specify models which can be trained on many different sequential decision-making tasks. We show that a single UniMASK model is often capable of carrying out many tasks with performance similar to or better than single-task models. Additionally, after fine tuning, our UniMASK models consistently outperform comparable single-task models.

---

K-LITE: Learning Transferable Visual Models with External Knowledge

Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell, Anna Rohrbach, Jianfeng Gao

Abstract: The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, based on the broad concept coverage achieved through large-scale data collection process. Alternatively, we argue that learning with external knowledge about images is a promising way which leverages a much more structured source of supervision and offers sample efficiency.

In this paper, we propose K-LITE (Knowledge-augmented Language-Image Training and Evaluation), a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in natural language with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts; In evaluation, the natural language is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge-augmented models show significant improvement in transfer learning performance over existing methods. Our code is released at https://github.com/microsoft/klite.

---

Extreme Compression for Pre-trained Transformers Made Simple and Efficient

Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, Yuxiong He

Abstract: Extreme compression, particularly ultra-low bit precision (binary/ternary) quantization, has been proposed to fit large NLP models on resource-constraint devices. However, to preserve the accuracy for such aggressive compression schemes, cutting-edge methods usually introduce complicated compression pipelines, e.g., multi-stage expensive knowledge distillation with extensive hyperparameter tuning. Also, they oftentimes focus less on smaller transformer models that have already been heavily compressed via knowledge distillation and lack a systematic study to show the effectiveness of their methods.

In this paper, we perform a very comprehensive systematic study to measure the impact of many key hyperparameters and training strategies from previous. As a result, we find out that previous baselines for ultra-low bit precision quantization are significantly under-trained. Based on our study, we propose a simple yet effective compression pipeline for extreme compression.

Our simplified pipeline demonstrates that:

(1) we can skip the pre-training knowledge distillation to obtain a 5-layer \bert while achieving better performance than previous state-of-the-art methods, like TinyBERT;

(2) extreme quantization plus layer reduction is able to reduce the model size by 50x, resulting in new state-of-the-art results on GLUE tasks.

---

On the Complexity of Adversarial Decision Making

Dylan J Foster, Alexander Rakhlin, Ayush Sekhari, Karthik Sridharan

Abstract: A central problem in online learning and decision making—from bandits to reinforcement learning—is to understand what modeling assumptions lead to sample-efficient learning guarantees. We consider a general adversarial decision-making framework that encompasses (structured) bandit problems with adversarial rewards and reinforcement learning problems with adversarial dynamics. Our main result is to show—via new upper and lower bounds—that the Decision-Estimation Coefficient, a complexity measure introduced by Foster et al. in the stochastic counterpart to our setting, is necessary and sufficient to obtain low regret for adversarial decision making. However, compared to the stochastic setting, one must apply the Decision-Estimation Coefficient to the convex hull of the class of models (or, hypotheses) under consideration. This establishes that the price of accommodating adversarial rewards or dynamics is governed by the behavior of the model class under convexification, and recovers a number of existing results –both positive and negative. En route to obtaining these guarantees, we provide new structural results that connect the Decision-Estimation Coefficient to variants of other well-known complexity measures, including the Information Ratio of Russo and Van Roy and the Exploration-by-Optimization objective of Lattimore and György.

---

Maximum Class Separation as Inductive Bias in One Matrix

Tejaswi Kasarla, Gertjan J. Burghouts, Max van Spengler, Elise van der Pol, Rita Cucchiara, Pascal Mettes

Abstract: Maximizing the separation between classes constitutes a well-known inductive bias in machine learning and a pillar of many traditional algorithms. By default, deep networks are not equipped with this inductive bias and therefore many alternative solutions have been proposed through differential optimization. Current approaches tend to optimize classification and separation jointly: aligning inputs with class vectors and separating class vectors angularly.

This paper proposes a simple alternative: encoding maximum separation as an inductive bias in the network by adding one fixed matrix multiplication before computing the softmax activations. The main observation behind our approach is that separation does not require optimization but can be solved in closed-form prior to training and plugged into a network. We outline a recursive approach to obtain the matrix consisting of maximally separable vectors for any number of classes, which can be added with negligible engineering effort and computational overhead. Despite its simple nature, this one matrix multiplication provides real impact. We show that our proposal directly boosts classification, long-tailed recognition, out-of-distribution detection, and open-set recognition, from CIFAR to ImageNet. We find empirically that maximum separation works best as a fixed bias; making the matrix learnable adds nothing to the performance. The closed-form implementation and code to reproduce the experiments are available on GitHub.

---

Censored Quantile Regression Neural Networks for Distribution-Free Survival Analysis

Tim Pearce, Jong-Hyeon Jeong, Yichen Jia, Jun Zhu

Abstract: This paper considers doing quantile regression on censored data using neural networks (NNs). This adds to the survival analysis toolkit by allowing direct prediction of the target variable, along with a distribution-free characterization of uncertainty, using a flexible function approximator. We begin by showing how an algorithm popular in linear models can be applied to NNs. However, the resulting procedure is inefficient, requiring sequential optimization of an individual NN at each desired quantile. Our major contribution is a novel algorithm that simultaneously optimizes a grid of quantiles output by a single NN. To offer theoretical insight into our algorithm, we show firstly that it can be interpreted as a form of expectation-maximization, and secondly that it exhibits a desirable `self-correcting’ property. Experimentally, the algorithm produces quantiles that are better calibrated than existing methods on 10 out of 12 real datasets.

---

Learning (Very) Simple Generative Models Is Hard

Sitan Chen, Jerry Li, Yuanzhi Li

Abstract: Motivated by the recent empirical successes of deep generative models, we study the computational complexity of the following unsupervised learning problem. For an unknown neural network \(F:\mathbb{R}^d\to\mathbb{R}^{d’}\), let \(D\) be the distribution over \(\mathbb{R}^{d’}\) given by pushing the standard Gaussian \(\mathcal{N}(0,\textrm{Id}_d)\) through \(F\). Given i.i.d. samples from \(D\), the goal is to output \({any}\) distribution close to \(D\) in statistical distance.

We show under the statistical query (SQ) model that no polynomial-time algorithm can solve this problem even when the output coordinates of \(F\) are one-hidden-layer ReLU networks with \(\log(d)\) neurons. Previously, the best lower bounds for this problem simply followed from lower bounds for \(supervised\) \(learning\) and required at least two hidden layers and \(poly(d)\) neurons [Daniely-Vardi ’21, Chen-Gollakota-Klivans-Meka ’22].

The key ingredient in our proof is an ODE-based construction of a compactly supported, piecewise-linear function \(f\) with polynomially-bounded slopes such that the pushforward of \(\mathcal{N}(0,1)\) under \(f\) matches all low-degree moments of \(\mathcal{N}(0,1)\).

For instance, TaylorMade Golf Company turned to Microsoft Syntex for a comprehensive document management system to organize and secure emails, attachments and other documents for intellectual property and patent filings. At the time, company lawyers manually managed this content, spending hours filing and moving documents to be shared and processed later.

With Microsoft Syntex, these documents are automatically classified, tagged and filtered in a way that’s more secure and makes them easy to find through search instead of needing to dig through a traditional file and folder system. TaylorMade is also exploring ways to use Microsoft Syntex to automatically process orders, receipts and other transactional documents for the accounts payable and finance teams.

Other customers are using Microsoft Syntex for contract management and assembly, noted Teper. While every contract may have unique elements, they are constructed with common clauses around financial terms, change control, timeline and so forth. Rather than write those common clauses from scratch each time, people can use Syntex to assemble them from various documents and then introduce changes.

“They need AI and machine learning to spot, ‘Hey, this paragraph is very different from our standard terms. This could use some extra oversight,’” he said.

“If you’re trying to read a 100-page contract and look for the thing that’s significantly changed, that’s a lot of work versus the AI helping with that,” he added. “And then there’s the workflow around those contracts: Who approves them? Where are they stored? How do you find them later on? There’s a big part of this that’s metadata.”

When DALL∙E 2 gets personal

The availability of DALL∙E 2 in Azure OpenAI Service has sparked a series of explorations at RTL Deutschland, Germany’s largest privately held cross-media company, about how to generate personalized images based on customers’ interests. For example, in RTL’s data, research and AI competence center, data scientists are testing various strategies to enhance the user experience by generative imagery.

RTL Deutschland’s streaming service RTL+ is expanding to offer on-demand access to millions of videos, music albums, podcasts, audiobooks and e-magazines. The platform relies heavily on images to grab people’s attention, said Marc Egger, senior vice president of data products and technology for the RTL data team.

“Even if you have the perfect recommendation, you still don’t know whether the user will click on it because the user is using visual cues to decide whether he or she is interested in consuming something. So artwork is really important, and you have to have the right artwork for the right person,” he said.

Imagine a romcom movie about a professional soccer player who gets transferred to Paris and falls in love with a French sportswriter. A sports fan might be more inclined to check out the movie if there’s an image of a soccer game. Someone who loves romance novels or travel might be more interested in an image of the couple kissing under the Eiffel Tower.

Combining the power of DALL∙E 2 and metadata about what kind of content a user has interacted with in the past offers the potential to offer personalized imagery on a previously inconceivable scale, Egger said.

“If you have millions of users and millions of assets, you have the problem that you simply can’t scale it – the workforce doesn’t exist,” he said. “You would never have enough graphic designers to create all the personalized images you want. So, this is an enabling technology for doing things you would not otherwise be able to do.”

Egger’s team is also considering how to use DALL∙E 2 in Azure OpenAI Service to create visuals for content that currently lacks imagery, such as podcast episodes and scenes in audiobooks. For instance, metadata from a podcast episode could be used to generate a unique image to accompany it, rather than repeating the same generic podcast image over and over.

Along similar lines, a person who is listening to an audiobook on their phone would typically look at the same book cover art for each chapter. DALL∙E 2 could be used to generate a unique image to accompany each scene in each chapter.

Using DALL∙E 2 through Azure OpenAI Service, Egger added, provides access to other Azure services and tools in one place, which allows his team to work efficiently and seamlessly. “As with all other software-as-a-service products, we can be sure that if we need massive amounts of imagery created by DALL∙E, we are not worried about having it online.”

The appropriate and responsible use of DALL∙E 2

No AI technology has elicited as much excitement as systems such as DALL∙E 2 that can generate images from natural language descriptions, according to Sarah Bird, a Microsoft principal group project manager for Azure AI.

“People love images, and for someone like me who is not visually artistic at all, I’m able to make something much more beautiful than I would ever be able to using other visual tools,” she said of DALL∙E 2. “It’s giving humans a new tool to express themselves creatively and communicate in compelling and fun and engaging ways.”

Her team focuses on the development of tools and techniques that guide people toward the appropriate and responsible use of AI tools such as DALL∙E 2 in Azure AI and that limit their use in ways that could cause harm.

To help prevent DALL∙E 2 from delivering inappropriate outputs in Azure OpenAI Service, OpenAI removed the most explicit sexual and violent content from the dataset used to train the model, and Azure AI deployed filters to reject prompts that violate content policy.

In addition, the team has integrated techniques that prevent DALL∙E 2 from creating images of celebrities as well as objects that are commonly used to try to trick the system into generating sexual or violent content. On the output side, the team has added models that remove AI generated images that appear to contain adult, gore and other types of inappropriate content.

FARMINGTON, Wash. – The gently rolling hills here in eastern Washington have long grown rich harvests of wheat, barley and lentils.

Fifth-generation farmer Andrew Nelson is adding a new bumper crop to that bounty: Data.

He gathers it from sensors in the soil, drones in the sky and satellites in space. They feed Nelson information about his farm at distinct points, every day, all year long — temperature variations, soil moisture and nutrient levels, plant health and more.

Nelson in turn feeds that data into Project FarmVibes, a new suite of farm-focused technologies from Microsoft Research. Starting today, Microsoft will open source these tools so researchers and data scientists — and the rare farmer like Nelson, who is also a software engineer — can build upon them to turn agricultural data into action that can help boost yields and cut costs.

The first open-source release is FarmVibes.AI. It is a sample set of algorithms aimed at inspiring the research and data science community to advance data-driven agriculture. Nelson is using this AI-powered toolkit to help guide decisions at every phase of farming, from before seeds go into the ground until well after harvest.

FarmVibes.AI algorithms, which run on Microsoft Azure, predict the ideal amounts of fertilizer and herbicide Nelson should use and where to apply them; forecast temperatures and wind speeds across his fields, informing when and where he plants and sprays; determine the ideal depth to plant seeds based on soil moisture; and tell him how different crops and practices can keep carbon sequestered in his soil.

“Project FarmVibes is allowing us to build the farm of the future,” said Nelson, who has partnered with Microsoft Research to turn his 7,500 acres into a proving ground for Project FarmVibes. “We’re showcasing the impact technology and AI can have in agriculture. For me, Project FarmVibes is saving a lot in time, it’s saving a lot in costs and it’s helping us control any issues we have on the farm.”

The new tools sprouted from Microsoft’s work with large customers like Land O’ Lakes and Bayer to integrate and analyze data. Project FarmVibes reflects more recent research in precision and sustainable agriculture.

By open sourcing its latest research tools, Microsoft wants to spread them far beyond Washington to help tackle the world’s urgent food problem, said Ranveer Chandra, managing director of Research for Industry.

By 2050, we’ll need to roughly double global food production to feed the planet, Chandra said. But as climate change accelerates, water levels drop and arable lands vanish, doing that sustainably will be a huge challenge.

“We believe one of the most promising approaches to address this problem is data-driven agriculture,” he said.