Bing Visual Search: search in images

Martin Brinkmann
Jun 1, 2017
Updated • Jun 1, 2017
Internet, Search

Bing Visual Search is a new feature of Bing's Image Search tool that allows you to select objects in images to run searches for them.

The new feature, which is live already on Bing, provides you with options to run searches for objects or people that are displayed in images that are returned by Bing's Image Search tool.

The process itself works similarly to how you would capture a rectangular screenshot on the screen. You draw a rectangle around the object, Bing identifies it, and returns results that match it.

Lets take a closer look at how this all works.

Bing Visual Search

bing visual image search

First thing you do is head over to Bing, and run a search using the search engine's image search tool.

Select one of the results that you are interested in. You will notice the new "search within this image" icon that Bing Images displayed in the top left corner of the image. When you click on it, the Visual Search tool is loaded so that you can select an object or person on the screen to run that search on Bing.

The Visual Search tool on Bing displays a selection rectangle on the screen which you can move around, and change in size. This works via dragging and dropping. You may also draw a new rectangle on the screen instead; the old is removed when you do so.

Bing returns information whenever you move the rectangle, or change its size. It is not necessarily the case that the "related images" change when you do, but they may depending on your selection.

visual search

The Visual Search tool displays only related images, which means that you cannot use it for other things. An option to identify elements on a picture would certainly be useful as well.

All you get right now however is a list of image that Bing believes are related to the selected area of the source image.

The identification works surprisingly well for some queries. While you should not expect 100% percent matches, especially when multiple images are returned, it works better than reverse image search engines.

Microsoft notes that this works really well for shopping related searches. Mark a chandelier in your dream home, and get chandeliers that look like it (or similar), in the results. All you have to do then is follow the links to find out how much it costs, and where you can buy it.

Microsoft reveals how this new Visual Search tool processes the queries:

  1. The first step is the image understanding step. Microsoft uses Image Processing services to "perform object detection, extraction of various image features including DNN features, recognition features, and additional features used for duplicate detection".
  2. Then comes the Text Query Inference step. Bing tries to describe the image using test based on the results of the image processing services.
  3. A triggering scenario is run then which identifies different scenarios for search by image.
  4. Next up is image matching using Bing's Visual Words feature which " allows us to quantize a dense feature vector into a set of discrete visual words".
  5. Image candidates are then ranked in the last step, and the best matching images are returned as results.


Bing Visual Search is a promising feature that worked really well during tests. Your mileage may vary depending on what you search for and highlight in the images though. If you run some searches, let me know how that turned out for you.

I wish Microsoft would improve the basic feature in the future to return text results as well.

Now You: What's your take on Bing Visual Search?


Bing Visual Search: search in images
Article Name
Bing Visual Search: search in images
Bing Visual Search is a new feature of Bing's Image Search tool that allows you to select objects in images to run searches for them.
Ghacks Technology News

Tutorials & Tips

Previous Post: «
Next Post: «


  1. Ano Nymus said on June 3, 2017 at 2:00 am

    I see a post that starts with Bing and I just skip it. I guess I’m not the only one, looking at ‘all’ the others that have posted here.

  2. Ken Saunders said on June 2, 2017 at 3:34 am

    It’s pretty impressive.
    Thanks, I wouldn’t have known about this.

    I’ve been using Bing images over Google for a few years. It can be frustratingly slow sometimes, but the results make it worth it.

Leave a Reply

Check the box to consent to your data being stored in line with the guidelines set out in our privacy policy

We love comments and welcome thoughtful and civilized discussion. Rudeness and personal attacks will not be tolerated. Please stay on-topic.
Please note that your comment may not appear immediately after you post it.