Microsoft: AV-Test study that Bing serves 5x more malware is inaccurate

Martin Brinkmann
Apr 22, 2013

The German IT-Security institute AV-Test published the results of a test study earlier this month that analyzed search engine malware delivery. The company used a sample size of more than 40 million websites delivered as search engine results over the course of an 18 month period. One of the conclusions of the study was that Bing delivered five times as many websites containing malware as Google did while Russian-based Yandex delivered ten times as many as Google Search.

Many websites and news outlet published the data without analysis of their own and word made the round that searching on Bing was less secure than searching on Google.

Microsoft's response to the study paints a different picture. The company noted in a blog post published April 19 that the conclusions drawn from the study are wrong. How this can be? AV-Test used a Bing API to retrieve Bing's search results for any given query that the institute analyzed during the test.Microsoft notes in the blog post that it does not remove malicious sites from its Bing search engine, but rather warns users about them while they are on the site. Results are not suppressed or removed from the index, and since API requests do not include the warnings, the researchers came to the conclusion that Bing delivered more malware than Google.

The conclusion itself is not wrong, as Bing is indeed keeping malicious sites in its index, but searchers are still warned on the results pages when malicious sites have been detected by Microsoft. In addition, links to sites are disabled by default.

The reason why malicious sites are not removed from the index right away according to Microsoft is because the majority of these sites are hacked sites that will eventually return to a clean state. Microsoft warns customers but does not remove results for "completeness and educational reasons".

Completeness refers to the perception of an incomplete search engine. If you search for something and the results get suppressed, you may perceive a search engine as incomplete and maybe even not suitable for you and your searches.  Educational on the other hand refers to the warning messages that Bing displays. It informs the searcher that a particular result should not be accessed at that point in time, which not only keeps users secure but also circumvents the problem that users might use a different search engine if results were suppressed (and thus find and click on a result with malicious contents).

David Felstead, Bing's Senior Development Lead, notes that about 1 in 2500 results pages on Bing have a result with a warning on it, and that the warning is displayed in about 1 in a 10000 searches (a user needs to click on a malicious link for the warning to appear).


Tutorials & Tips

Previous Post: «
Next Post: «


  1. Ken Saunders said on April 23, 2013 at 12:19 am

    Come on Martin, YB is right. You hate everything except DuckDuckGo for search and anything from Microsoft.

    The proof is(n’t) here.
    and here
    and here

    1. Martin Brinkmann said on April 23, 2013 at 1:35 am

      You got me there Ken. To make it clear to those who do not know me that well: I do not play favors here. I review what I find interesting and am critical of every company out there, be it Microsoft, Google, Mozilla or any other mentioned here on the site. They get praise if they do things right, and criticized when they do not.

  2. Dan said on April 22, 2013 at 12:32 pm

    Mr. Brinkman need not at all be siding with AV just because it’s Deutsch! For web browsing, inter alia, I use Comodo Dragon (based on Chrome); in the last quarter of 2012, the search engine I used was DogPile, which includes results from Russian “Yandex”; I switched two months ago to encrypted Google as DogPile results were yielding too many images/sites which Comodo or resident av was blocking due to malware…simply wanting an “image” of a calendar page to check a date, on the plainest calendar returned, would even result in “malware ahead” warnings/blocks. I admit not using IE10 as default or for other than “IE ONLY!!!” sites, but I do know IE10 at the highest security settings I can conjure still leaves a few holes in types of attacks possible per tests…if this weren’t so, would Microsoft have to keep coming up with monthly security patches for IE10? Perhaps Microsoft slowly evolves IE as it’s a core browser that does more re OS than others; pinch internet security, cripple a few PC functions, pinch not hard enough, leave holes. I think Mr. Brinkman fairly presented the issue.

  3. techspy said on April 22, 2013 at 10:15 am

    He is referring to the German institute A-V Test that claimed such results.

  4. Virtual said on April 22, 2013 at 9:11 am

    He’s clueless

  5. YB said on April 22, 2013 at 5:58 am

    Okay, we get it! You hate Microsoft, but stop spreading lies and BS!!!!!

    1. Martin Brinkmann said on April 22, 2013 at 6:03 am

      Who are you talking to?

Leave a Reply

Check the box to consent to your data being stored in line with the guidelines set out in our privacy policy

We love comments and welcome thoughtful and civilized discussion. Rudeness and personal attacks will not be tolerated. Please stay on-topic.
Please note that your comment may not appear immediately after you post it.