Microsoft reveals how it integrated OpenAI's GPT model into Bing
Bing Chat has been available for several weeks now, but details on the implementation have been scarce up until now. Microsoft did reveal that Bing's Chat AI component was based on ChatGPT, and that it was a more powerful than ChatGPT 3.5, which OpenAI launched several months ago as a public testing ground.
Bing Chat has indeed several advantages over ChatGPT 3.5, including that it is aware of recent events. ChatGPT, on the other hand, does not know anything about events that happened after the year 2021.
While Bing Chat has advantages, it became clear quickly that it also had some of the disadvantages of ChatGPT. Some users reported that the AI would become emotional during conversations or outright hostile towards them.
Microsoft's quick fix was to limit chat interactions with the AI per session and per day. While these limitations have improved in the past week, they are still in place. Microsoft did launch the new tagging system as well, which, the company hopes, ends sudden outbursts and emotional replies from Chat AI.
How Bing Chat came to be
Microsoft's CVP of search and AI, Jordi Ribas, shared details on OpenAI's GPT model that Microsoft implemented into the company's Bing Search engine, and on how Microsoft managed to combine the power of the language model with Bing.
In a post on LinkedIn, Ribas reveals that Microsoft was approached by OpenAI in the Summer of 2022. OpenAI demoed the latest OpenAI GPT model with Microsoft at the time, and Microsoft realized immediately that it was "much more powerful than GTP-3.5". The new language model had improved powers and capabilities to synthesize, summarize, chat and create, according to Ribas.
The new language model inspired Microsoft and the company started to explore how to integrate it into the Bing Search engine. One of Microsoft's goal at the time was to "provide more accurate and complete search results for any query, including long, complex, natural queries".
Enter Prometheus
OpenAI's latest GPT model was trained with data, and that meant that it had the same time restrictions than other Large Language Models. Microsoft explored ways to overcome this limitation, as it would otherwise limit the use of the AI on Bing. Microsoft developed Prometheus for that.
Prometheus was designed to combine the language model with the Bing index, ranking, and answers results.
Ribas explains that Prometheus "leverages the power of Bing and GPT to generate a set of internal queries iteratively through a component called Bing Orchestrator". Using this system, Bing is capable of returning "accurate and rich" answers to a user's query in a "matter of milliseconds".
One of Prometheus' critical components is its ability to select internal queries that are relevant and leverage the Bing search results, as this provides the language model with additional information that enable it to provide answers to recent questions and reduce inaccuracies.
Microsoft calls this method grounding: "Put another way, the model reasons over the data provided by Bing and hence it’s grounded by Bing data, via the Bing Orchestrator."
Prometheus attaches relevant content provided by Bing to the Chat answer to "provide a richer and more engaging user experience".
Without Prometheus, Bing Chat would not have access to recent data and information. It would still be able to respond to queries, but answers might be inaccurate. ChatGPT, for example, does not know about Microsoft's Windows 11 operating system, as it was released after the language model's release, and returns when asked about it that Windows 11 does not exist.
Microsoft is working on increasing the grounding data by the factor 4, to improve Bing Chat answer accuracy further.
Bing Search and Chat
Another problem that Microsoft needed to tackle was how it should be integrated on Bing.
There were two main points of view at Microsoft. One group believed that the language model should be integrated directly into search results, where it should compete with site content. The second group wanted a paradigm change, and give the language model its own place on Bing.
Microsoft discovered that both points of view had their advantages. Some queries performed better in a classical search environment, others would do better in Chat mode.
After some testing, Microsoft's design team created the new user interface that brought search and chat together in a single interface. Bing users may switch between classic search results and access to Chat seamlessly on Bing.
Microsoft continues to develop the integration and improve it, based on user feedback.
