Microsoft Edge adds support for automatic image descriptions

A picture is worth a thousand words. Microsoft is taking the old adage to heart, it has introduced a new feature in Edge, called Automatic Image Descriptions, to help people with visual impairments.

Before we discuss how the technology works, let’s first take a look at how websites use images. When blogs post articles, writers attach screenshots to the posts and set an attribute for the image. The attribute is called alt text. This description acts as a caption or description, which is recognized by search engines. When a user searches for keywords that fit the image alt text, the search engine will highlight the appropriate image from the results.

Automatic image descriptions in Microsoft Edge

Screen readers, like Narrator in Windows 10 and 11, are often used by people with visual impairments. This software uses text-to-speech algorithms to help them understand what is displayed on the screen and select/execute various options etc.

Microsoft Edge supports Narrator to read the text content on web pages and helps users navigate websites, links, etc. When the browser loads a page that contains images, Narrator will check to see if the image has alt text assigned, and if so, it will read it out loud.

According to Microsoft, many websites do not include alt text for images. This means that your descriptions are blank and skipped by the screen reader entirely, and the user will miss out on useful information that the image might contain.

This is where the new Automatic Image Descriptions in Microsoft Edge come into play. It combines optical image recognition and text-to-speech. When Microsoft Edge detects that an image doesn’t have an alt text caption, it will send the media to its machine learning algorithm, which works with the Azure Cognitive Service Computer Vision API.

AI technology analyzes the content of the images, creates a description for it in one of the supported languages, and returns it to the browser for Narrator to read aloud. It is also capable of Optical Character Recognition (OCR) to detect text within images and supports 120 languages. Automatic image descriptions support common image formats like JPEG, GIF, PNG, WebP to name a few.

There are some exceptions that the Vision API will ignore, ie it will not read aloud to the reader. This includes images that the website states as descriptive, images smaller than 50 x 50 pixels, very large images, and photos that may contain gore or adult content.

How to enable automatic image descriptions in Microsoft Edge?

Automatic image descriptions are now available in the stable release of Microsoft Edge for Windows, Linux, and macOS. The option is not enabled by default, because it has its own privacy policy. To turn the option on, go to the Edge Settings > accessibility page, and click the button next to “Get image descriptions from Microsoft for screen readers,” you’ll need to agree to the Privacy Policy to use the feature.

Enable Windows Narrator using the Ctrl + Win + Enter hotkey, and the screen reader will read the image descriptions as you browse the Internet using Microsoft Edge. You can toggle the feature from the browser’s context menu.

Here is an official video that illustrates how the trait works.

It’s really amazing how accessibility features like live captions and color blind filters can help people with disabilities. Automatic image descriptions are a great addition to your arsenal.

advertising