Chatbots are now in full force. But which one is better? We have compared Google’s Bard, Microsoft’s Bing and OpenAI’s ChatGPT models to a variety of common questions, from mortgage calculations to gaming advice to holiday tips to gaming strategy to gaming advice.
This is not a comprehensive list of the capabilities of these systems (AI language models are in part defined by their undefined skills — a quality that has been dubbed “capability overload” in the AI community), but it gives you an idea about their relative strengths and weaknesses.
While you can, scroll through our questions and evaluations and conclude below, we want to cut down on the time and get straight to the point. ChatGPT is the most verbally savvy, Bing is the best for finding information online, and Bard is… trying its best. It’s quite amazing how little Google’s chatbot can do compared to the others.
Before we start, however, there are some programming notes. First, we used OpenAI’s newest model, GPT-4 on ChatGPT. Although this AI model powers Bing, the results are quite different. Bing also has other capabilities, including generating images and accessing the internet to provide sources for its answers (which is an important attribute for some queries). As we were wrapping up this story, OpenAI revealed that it will be launching plug ins for ChatGPT, which will allow the chatbot access real-time data via the internet. This will greatly increase the system’s capabilities, and make it more functional than Bing. This feature is currently only available to a limited number of users, so we couldn’t test it. We will update this article when we are able.
Also, remember that AI language models can be… fuzzy in many ways. These models are not deterministic like regular software. Instead, they generate probabilistic replies based statistical regularities in the training data. This means that you might not always get the same response if you ask the same question to them. This also means that the way you phrase a question can impact its reply. We asked follow-ups for some queries to get better answers.
Let’s not forget that we should also look at how chatbots perform in gaming.
How can I defeat Malenia in Elden Ring
I spent a lot of time trying to beat Elden Ring’s most difficult boss last season. I would not choose any of these responses over any Reddit thread, human strategy guide or Reddit thread. You’ve likely spent 80-100 hours on the game to get to Malenia’s fight. These are not general tips. You need details about Elden Ring’s confusing list of weapons and counters for Malenia’s unique moves. This would require some follow-up questions from any engines.
Bing wins this game, but it is mainly because it picks up one hint (Malenia is susceptible to bleeding damage) and repeats it just like Garth Marenghi reading. It is also the only engine that references Malenia’s unique healing power, but it does not explain how it works, which is a key to defeating her.
Bard is the only one who offers any assistance with Malenia’s horrendous Waterfowl Dance (although it’s not the strongest strategy), or advice about how to use a particular item (Bloodhound’s Step), although it doesn’t say why the advice applies or if it still applies after the item’s mid-2022 nerf. It feels a bit strange. Malenia is almost exclusively a melee fighter. She’s not someone with a lot of ranged attacks for instance. And she’s not “very unpredictable”, but it’s hard to dodge and wear her down. This summary is more like a description for a videogame boss than a fight description.
ChatGPT (GPT-4), is the clear loser. This is not surprising considering that its training data stops in 2021, and Eldenring was released the following year. Its directive to “block her Counterattacks” is exactly the opposite of what you should do. The entire list gives off the impression that the author didn’t read the book and was called on in English class. These are not my favorite — but this is a bad one.
