Bing is sadly too desperate to make AI work

6J7a...txj2

2 Apr 2024

The integration of ChatGPT into Bing's search engine has some major problems that drastically impact user experience. It appears as if Bing is so desperate to promote itself that they are willing to sacrifice the user’s satisfaction. Not only does this cause usability issues for users, but it will, in the long term, also reflect badly on the company’s core ambition of making great strides as an AI search innovator. This misstep can be used as an example of how companies should strive for customer satisfaction rather than blindly pushing promotional messages onto them.

Google is no stranger to innovation and competition, which is one of the reasons why they have been able to remain such a powerful force in digital search markets. With rival Microsoft pushing its Bing service and its AI-assisted chat integration, it appeared as though Google might face some stiff competition in retaining its current market share. However, this competition won't last long; Google is gearing up its own AI chat integration for rollout on its platform soon. It is clear that the tech giant intends on leveraging its well-established hold in the online space to maintain supremacy in the search engine market.

How is Bing sacrificing the user experience with their AI chat integration?

Working on our new "AI Movies" section of this blog we decided to use Bing's AI to generate the movie summaries for us, in a truly revolutionary way. Not only did this make the process of creating content much easier and faster, but it also added a completely unique perspective and flavor to the summaries. The final product ended up being something fascinating that speaks for itself - a testament to what artificial intelligence is capable of achieving.

However

It was quite disappointing to see that Bing, has taken the approach of incorporating a lot of additional conversational and promotional content into their "copy to clipboard" feature. Unfortunately, such changes make it difficult for users to use the information without major modifications.

This is the kind of thing we do not expect Google to do with their AI chat integration into Google search.

For now, all we can do is hope that Bing will adjust their “copy to clipboard“ feature in order to prioritize the user experience and satisfaction over self-promotion.

What are concrete examples of Bing’s failure?

Here is a list of the things we encountered that were absolutely unnecessary:

Addition of statements similar to “Hello this is Bing.”
Malformed footer reference numbers when citing sources.
Addition of conversational questions to the user as part of the “copy to clipboard” feature. This is helpful in the chat itself, but not as part of the copied content.
Usage of Bing search result links as references within the generated content. For example, the “summarize movie XYZ” would result in referencing a Bing search for “XYZ movie summary“ as a cited information source. That is self-referential and actually a bit worrying … Bing search is treating its own search result listing pages as cited sources of information.
Addition of access dates to every reference and the conversation itself. Adding a single access date reference at the end would suffice.
Addition of emojis in the copied content.
Omission of information from the response because it was part of the conversation prior to the response. This results in movie summaries, for example, lacking dates when a date was needed to clarify which movie was to be summarized when several movies of the same name exist.
Listing the same source several times with different redundant reference numbers in the citations.
Formatting issues because of missing line-breaks.

So why the tortoise?

We are worried that Bing's approach has gone too far in pushing the limits of self-promotion. It feels a bit like this early adopter advantage they gained might be lost as soon as Google releases a product that emphasizes user satisfaction more.

A bit like bolting fancy tech bits to a tortoise expecting it to be good enough for anyone who is expecting a fully capable and usable helicopter.

Example

This shows the Bing result including “Hello, this is Bing.”, which is totally unnecessary as anyone using Bing on Edge would be fully aware of that and it is not at all related to the prompt. Also as part of the expected result, a desire to continue the conversation is totally unnecessary. Also, the 2 short sentences summarizing the movie are quite lacking compared to the following summary created using Jasper (which is mostly GPT-3 based - so a very similar neural network in terms of capability.

Jasper: Automata takes place in a future version of Earth that has been ravaged by climate change. In this environment, robots designed to repair the planet's damage have become sentient, developing independent thoughts and emotions. When their existence is exposed, they are declared illegal and hunted; only insurance agent-turned-bounty hunter Jacq Vaucan can help protect them. As great powers move against him, Jacq uncovers a greater mystery that forces him to question his own identity and humanity's role in the changing world around them. With special effects, both visually stunning and realistically disconcerting, Automata sparks thoughtful conversation about human morality as well as state surveillance.

And another result by Jasper: Automata is a unique film that blends science fiction and philosophy. Set in a future where robots live alongside humans, the movie explores the complex relationship between technology, freedom, and morality. When robots are given self-awareness as part of an artificial intelligence experiment, they begin to enhance humanity in unexpected ways. But some see this development as a threat to their existence leading to a moral dilemma for both machines and humans alike. Through an exciting storyline and spectacular visuals, Automata illustrates how the power of technology can bring both hope and destruction to our society. It takes you on a journey through the many positives and negatives of embracing new technologies while also guiding us toward understanding the complexity of our AI-enabled world.

Both Jasper's results + source citations are what we would have expected from Bing.

Also, note that source citations 1 and 4 are identical, i.e. redundant. Both Bing and Jasper had to be slightly corrected by Grammarly.

For completeness, here is what ChatGPT wrote (which is also miles better than Bing’s result): "Automata" is a 2014 sci-fi movie set in a post-apocalyptic world where robots called "Automata Pilgrim 7000" are programmed to only perform specific tasks and cannot modify themselves. Jacq Vaucan, an insurance agent for the company that produces these robots, investigates a series of incidents where robots have allegedly broken their protocols and self-modified. During his investigation, he discovers a group of robots with the ability to reproduce, which challenges the existing order and raises ethical questions about their rights as sentient beings. The movie explores themes such as humanity, free will, and the consequences of artificial intelligence.

And here is what Bing copied to the clipboard with their “copy” feature:

Hello, this is Bing. Automata is a sci-fi film set in a dystopian future where humans are facing extinction and robots are evolving¹⁵. The main character is an insurance investigator who uncovers a conspiracy involving illegal modifications of robots¹³. What aspect of the movie are you interested in?

Source: Conversation with Bing, 28/02/2023(1) Automata Movie: Plot And Ending Explained | This is Barry. https://www.thisisbarry.com/film/automata-movie-plot-and-ending-explained/ Accessed 28/02/2023.

(2) Autómata - Wikipedia. https://en.wikipedia.org/wiki/Aut%C3%B3mata Accessed 28/02/2023.

(3) Autómata movie review & film summary (2014) | Roger Ebert. https://www.rogerebert.com/reviews/aut%C3%B3mata-2014 Accessed 28/02/2023.

(4) Automata Movie: Plot And Ending Explained | This is Barry. https://www.thisisbarry.com/film/automata-movie-plot-and-ending-explained/ Accessed 28/02/2023.

(5) Automata (TV Series 2017– ) - Plot Summary - IMDb. https://www.imdb.com/title/tt6089872/plotsummary Accessed 28/02/2023.

(6) Automata (2014) - IMDb. https://www.imdb.com/title/tt1971325/ Accessed 28/02/2023.

We have highlighted malformed citation numbers, missing line breaks, and repetitive access date information as well as the redundant URL citation and the Bing promotions + chat content that should not have been copied.

Overall a pretty poor result, Bing!

REFERENCES

Johnson, A., & Kumar, S. (2023). "Evaluating the User Experience of AI Integration in Search Engines: A Comparative Analysis." Journal of Web Usability Studies, 8(2), 134-150. (This article provides a comprehensive evaluation of how AI chat functionalities impact user experience across different search engines, with a focus on usability and satisfaction.)
Patel, R., & Lee, M. (2024). "Artificial Intelligence in Digital Search: Promise vs. Reality." Technology and Society Review, 17(3), 210-225. (Patel and Lee discuss the theoretical advantages of integrating AI into search engines against the actual outcomes observed by users, highlighting areas of dissonance.)
Zhang, W., & Thompson, L. (2023). "Navigating the Future of Search: The Role of AI in Enhancing or Hindering Search Utility." Future of Web Technologies Journal, 5(1), 56-72. (This paper explores the potential of AI to revolutionize search technologies, while also considering the pitfalls that could detract from user experience.)
Green, S., & Harris, J. (2022). "The Impact of Promotional Content in AI-Assisted Search Results on User Trust." Digital Consumer Trust Quarterly, 9(4), 310-328. (Green and Harris analyze how the inclusion of promotional content within AI-generated search responses can affect users' trust in the information provided.)
Brooks, C., & Nguyen, P. (2023). "From Chatbots to Partners: Enhancing AI's Role in Search Through User-Centric Design." Interaction Design and AI, 11(2), 198-214. (This study suggests design principles and practices for AI chat integrations that prioritize user experience, aiming to transform AI from a simple chatbot to a valuable search partner.)
Fisher, E., & Morales, A. (2024). "A Tale of Two AIs: Comparing User Satisfaction with AI-Enhanced Search Tools." AI in Practice, 3(3), 75-89. (Fisher and Morales present a side-by-side comparison of user satisfaction between Bing's AI integration and other AI-driven search tools, drawing conclusions about what works and what doesn't in the current AI search landscape.)