Create an account


Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Microsoft - Latest AI breakthrough describes images as well as people do

#1
Latest AI breakthrough describes images as well as people do

Novel object captioning


Image captioning is a core challenge in the discipline of computer vision, one that requires an AI system to understand and describe the salient content, or action, in an image, explained Lijuan Wang, a principal research manager in Microsoft’s research lab in Redmond.

“You really need to understand what is going on, you need to know the relationship between objects and actions and you need to summarize and describe it in a natural language sentence,” she said.

Wang led the research team that achieved – and beat – human parity on the novel object captioning at scale, or nocaps, benchmark. The benchmark evaluates AI systems on how well they generate captions for objects in images that are not in the dataset used to train them.

Image captioning systems are typically trained with datasets that contain images paired with sentences that describe the images, essentially a dataset of captioned images.

“The nocaps challenge is really how are you able to describe those novel objects that you haven’t seen in your training data?” Wang said.

To meet the challenge, the Microsoft team pre-trained a large AI model with a rich dataset of images paired with word tags, with each tag mapped to a specific object in an image.

Datasets of images with word tags instead of full captions are more efficient to create, which allowed Wang’s team to feed lots of data into their model. The approach imbued the model with what the team calls a visual vocabulary.

The visual vocabulary pre-training approach, Huang explained, is similar to prepping children to read by first using a picture book that associates individual words with images, such as a picture of an apple with the word “apple” beneath it and a picture of a cat with the word “cat” beneath it.

“This visual vocabulary pre-training essentially is the education needed to train the system; we are trying to educate this motor memory,” Huang said.

The pre-trained model is then fine-tuned for captioning on the dataset of captioned images. In this stage of training, the model learns how to compose a sentence. When presented with an image containing novel objects, the AI system leverages the visual vocabulary to generate an accurate caption.

“It combines what is learned in both the pre-training and the fine-tuning to handle novel objects in the testing,” Wang said.

When evaluated on nocaps, the AI system created captions that were more descriptive and accurate than the captions for the same images that were written by people, according to results presented in a research paper.



https://www.sickgaming.net/blog/2020/10/...people-do/
Reply



Possibly Related Threads…
Thread Author Replies Views Last Post
  Microsoft - Wayve’s self-driving solution seeks to protect people and the planet xSicKxBot 0 13 10-23-2020, 02:18 PM
Last Post: xSicKxBot
  Microsoft - New to Microsoft Teams: Microsoft Lists now generally available xSicKxBot 0 81 09-04-2020, 04:22 AM
Last Post: xSicKxBot
  Microsoft - Growing Azure’s capacity to help people during the COVID-19 pandemic xSicKxBot 0 112 06-16-2020, 07:02 PM
Last Post: xSicKxBot
  Microsoft - New app lets people contribute photos to help developers build AI models xSicKxBot 0 120 06-03-2020, 08:20 AM
Last Post: xSicKxBot
  Microsoft - New app lets people contribute photos to help developers build AI models xSicKxBot 0 113 05-29-2020, 03:19 AM
Last Post: xSicKxBot
  Microsoft - Fairness and interpretability in AI: putting people first xSicKxBot 0 122 05-23-2020, 08:24 AM
Last Post: xSicKxBot
  Microsoft - Microsoft adds 5 languages of India to Microsoft Translator xSicKxBot 0 178 04-17-2020, 01:48 PM
Last Post: xSicKxBot
  Microsoft - CMO Chris Capossela: Taking a people-first response to COVID-19 xSicKxBot 0 232 04-14-2020, 01:39 AM
Last Post: xSicKxBot
  Microsoft - Globe-trotting photographer connects people to wildlife xSicKxBot 0 196 02-13-2020, 09:13 PM
Last Post: xSicKxBot
  Microsoft - Using AI to advance the health of people and communities around the world xSicKxBot 0 323 01-29-2020, 09:03 PM
Last Post: xSicKxBot

Forum Jump:

Become a Patron!
[-]
Upcoming Events

[-]
Latest Threads
Ultimate Fantasy Bundle Giveaway
Last Post: xSicKxBot
Today 10:01 PM
» Replies: 0
» Views: 1
Unified Blazor UI in the Mobile Blazor B...
Last Post: xSicKxBot
Today 10:01 PM
» Replies: 0
» Views: 0
AppleInsider - Apple TV+, now one year o...
Last Post: xSicKxBot
Today 10:01 PM
» Replies: 0
» Views: 0
News - Poll: Box Art Brawl: Duel #66 – R...
Last Post: xSicKxBot
Today 10:01 PM
» Replies: 0
» Views: 0
News - Good Smile Releasing Pokémon Swor...
Last Post: xSicKxBot
Today 10:01 PM
» Replies: 0
» Views: 0
News - Newegg Reveals Its Month-Long Bla...
Last Post: xSicKxBot
Today 09:47 PM
» Replies: 0
» Views: 1
[Tut] How Many Freelance Developers Are ...
Last Post: xSicKxBot
Today 08:27 PM
» Replies: 0
» Views: 1
(Indie Deal) HALLOWEEN SCRATCHY SALE, Be...
Last Post: xSicKxBot
Today 08:26 PM
» Replies: 0
» Views: 2
Xbox Wire - Celebrate Halloween in Dead ...
Last Post: xSicKxBot
Today 06:19 PM
» Replies: 0
» Views: 1
News - Don’t Miss: Looking back at Left ...
Last Post: xSicKxBot
Today 06:19 PM
» Replies: 0
» Views: 2

[-]
Twitter

[-]
Sponsored
Get the Deal of the Week at RefurBees.com

Copyright © SickGaming.net 2012-2020