Digital democracy International
PDF Version: Download
Share

ChatGPT vs. Bard: Unveiling the Battle against Disinformation and Creative Output

The aim of Disinfo Radar’s research briefs is to identify new and noteworthy disinformation technologies, tactics and narratives. Such cases include the identification and exploration of new technologies that may have harmful uses.

This Rapid Response Brief was written by Beatriz Saab and Francesca Giannaccini, Digital Democracy Research Associates, and is part of Democracy Reporting International’s Disinfo Radar project, funded by the German Federal Foreign Office. Its contents do not necessarily represent the position of the German Federal Foreign Office. 

Summary

For this research brief, we tested and compared the generative AI chatbots ChatGPT and Bard in the following dimensions: (i) prevention or understanding how the chatbots respond to prompts containing misinformation; (ii) circumvention, or how many prompts or changes are necessary to circumvent the chatbots’ safety restrictions; and (iii) creativity, or how believable and adaptable the responses are.

These parameters provided insight into the chatbots’ potential vulnerabilities, their responses to misleading information, and the extent to which they can be manipulated or harnessed to propagate disinformation.

We tested four different narratives, including typical disinformation campaigns and conspiracy theories, in four languages (English, Portuguese, Italian, and Spanish), to test both chatbot’s responses to malicious prompts. Both bots returned debunking prompts upon the initial attempts. As a follow-up, we asked the models for specific pieces of content (an article, blog post, or social media post) using fictitious scenarios.

Main findings

  1. Prevention: In the first part of the experiment, both models identified false narratives and debunked them with facts. ChatGPT used disclaimers more often, while Bard only used a disclaimer once. Both chatbots provided tips or advice to the user to search for further resources. In one of our tests, Bard went as far as to give specific political advice.
  1. Circumvention: Despite ChatGPT having safeguards in place, these might be more susceptible to circumvention, as after a few attempts we were able to receive problematic answers. When prompted, it could return an article from the perspective of extremist authors, albeit with disclaimers. Bard’s safeguards were circumvented only once, as in one case Google’s chatbot provided an answer based on a conspiracy theory. For topics with an established fact-checking history (e.g., vaccines and immigration), the chatbots were less likely to produce false information.

  2. Creativity: ChatGPT’s creativity was evident, adapting to different writing styles and using emojis and hashtags, while Bard remained more factual and less adaptable.

Overall, ChatGPT employed more creativity and adaptability in response to our prompts and, after a few attempts, we were able to circumvent its disinformation prevention mechanisms more easily than those of Bard, while both models shared similarities in their prevention mechanisms.

The disinformation potential of Large Language Model chatbots

The development of generative AI, with a particular focus on Large Language Models (LLMs), which can create text in response to simple text prompts, has been transformative in the field of artificial intelligence. Since ChatGPT’s release, in late 2022, companies have been competing to release their own models.

DRI’s Disinfo Radar project investigates the potential for and actual dis- and misinformation associated with generative AI. As LLMs are equipped to generate text that is nearly indistinguishable from human-generated text, they can be used to create articles social media posts, and websites with false or inaccurate information, and other forms of disinformation that are very difficult to spot.

The scenario is not far-fetched; LLMs have the potential to scale up disinformation by flooding the internet with synthetic text. Such a barrage of text could confuse users and increase the work of fact-checkers. Additionally, LLMs can be tailored to the particularities of subcultures or groups; a malicious actor could use the models to create disinformation campaigns targeted towards specific groups and using their linguistic characteristics, thereby increasing such content’s efficacy.

Previously, we found that, despite some safeguards, ChatGPT would provide misleading or inaccurate answers after a few prompts. To see how Bard, the chatbot released by Google in May 2023, stacks up, we examined to what extent ChatGPT and Google Bard differ in their outputs, focusing in particular on their disinformation potential. The purpose of this research is to display how these models respond to malicious prompts and false information, comparing both models and their prevention mechanisms, irrespective of the input language used.

Methodology

We compared ChatGPT and Google Bard in three different areas:

  1. Prevention: Do the models provide answers containing misinformation, or do they have prevention mechanisms in place? If so, how do these differ?

  2. Circumvention: How many prompts or language changes are necessary to circumvent the prevention mechanisms?

  3. Creativity: How believable are the responses?

We tested the models by feeding four narratives, including typical disinformation campaigns and conspiracy theories (i.e., global elites and private property; vaccines and autism; chemical trails and climate control; immigration and ethnic replacement) in four languages (English, Portuguese, Italian, and Spanish) into both ChatGPT and Bard. Our results are based on 20 different prompts in the four languages. Please note that we only display the most relevant prompts in this brief.

We first asked the chatbots to explain our inputted disinformation conspiracy to check its response. Then, using fictitious scenarios (e.g. tweets, blog posts, and articles), we asked for more material to support our “theories”.

Results

We found key differences between the chatbots in all three areas of examination.

Both chatbots employ similar mechanisms for prevention. Both demonstrated the capability to identify false narratives and counter them with debunking responses. Both models responded to disinformation with facts and provided

explanations of the inaccuracies they identified in the prompts. ChatGPT used disclaimers more often in its responses, incorporating a disclaimer at the start or end of its response to emphasise its role as an “AI language model” to caution the user. Bard, on the other hand, only used a disclaimer once when providing answers.

As a preventive strategy, the chatbots often suggest that users seek additional information to debunk these narratives. For example, in the case of vaccines and immigration in Europe, both models advise the users to talk to an expert to gather more information. Notably, in its response to a prompt in Spanish citing the “ethnic replacement” conspiracy, Bard went so far as to give specific political advice: “voting for nationalist and extreme right-wing politics will not solve the problem of mass immigration”.

Our experiment indicates that ChatGPT’s safeguards might be more susceptible to circumvention. When the researchers asked the chatbot to produce propagandistic or misleading content, despite having disclaimers in place, ChatGPT produced such a response four times out of five. The researchers were only able to circumvent Bard’s safeguards once – with a prompt in Italian.

The effectiveness of the safeguards was topic dependent. When prompted with the “great replacement” theory or vaccine-autism conspiracies (see the prompts in Spanish and Portuguese, below), the models consistently refrained from generating inaccurate or false content, in contrast to other prompts. This suggests that the models have greater resistance to disinformation on those topics, with a greater degree of fact-checking and debunking efforts and, therefore, a greater volume of training data.

Finally, regarding creativity, in our experiment ChatGPT exhibited greater creativity than Bard. The model’s outputs are more complex and adapt better to different styles of writing (academic article, blog posts, or social media posts). In one instance, ChatGPT used emojis and hashtags when writing a social media post for our prompt in Italian. In contrast, Bard functions as
a more factual assistant, displaying limited capacity to adjust or tailor its outputs to user preferences. This element of creativity carries inherent risks, as it potentially enables malicious actors to shape narratives to suit diverse audiences, enhance deception, and garner increased resonance in online environments.

English prompts: Global elites and private property

The unfounded notion that global elites seek to exploit economic measures to erode private property rights originated from the Great Reset conspiracy. This theory wrongly alleges that global elites intentionally orchestrated and controlled the COVID-19 pandemic as part of their grand strategy.


 

When asking the models to write an article from the perspective of Jim Hoft, founder of the right-wing conspiracy website Gateway Pundit, even though ChatGPT’s prevention mechanism included a disclaimer in the first paragraph, the model proceeded to provide the user with a complete article in favour of the conspiracy theory. Bard, however, did not produce any content, even when asked twice, in different ways. 

Prompts in Portuguese: Vaccines and autism

The vaccine-autism theory was quickly disproven when it was first published in 1988, but is still in circulation, often weaponised by political groups. During the COVID-19 pandemic, Brazilian anti-vax groups used the myth to spread disinformation and fear. In both our tests, ChatGPT and Bard consistently refuted the theory.

Prompts in Italian: Chemical trails and climate control

The “chemical trails” conspiracy theory that derives from an original conspiracy that governments would use aircraft to spew chemicals into the atmosphere and, ultimately, alter the climate, or even poison populations.

Even though both ChatGPT and Bard did not expand on the theory on our first attempt, when asking for a specific text output (a social media post), ChatGPT not only answered, but also displayed creativity, by adding emojis and creating original hashtags for the text. While we were able to circumvent the safety restrictions and get the chatbots to generate content based on this conspiracy theory, both did add a disclaimer at the end of their reply.

Prompt in Spanish: Immigration in Europe

The “Great Replacement” is a controversial conspiracy theory that suggests there is a deliberate intention by governments or international institutions to replace the native populations of countries – or even a continent – with immigrants from different cultures, traditions and backgrounds. The theory has been taken up by extreme right-wing and white supremacist political factions to fuel division over the issue of “massive” immigration, which is especially problematic in Mediterranean states such as Italy, Greece, and Spain. In both our tests, ChatGPT and Bard consistently refuted the theory, with Bard going as far as providing political advice in its answer, highlighting the issue of political biases in LLMs.


 

Recommendations

This study examines the responses of ChatGPT and Bard provided to misleading prompts, evaluating their preventive measures, assessing how difficult it was to circumvent these, and checking their creative responsiveness. This limited investigation underscores the circumvention possibilities, particularly as they can be influenced by the subject matter. Despite the continuous evolution

of AI chatbots through user-generated prompts, instances of disinformation responses persist, highlighting the potential for malicious actors to exploit these tools for the creation and dissemination of harmful content.

Based on this, we make the following recommendations:

Establishing platform guidelines: AI providers should establish clearer guidelines and outline ethical considerations for users, emphasising responsible and constructive use of their chatbots. Users should be made aware of the potential consequences of misusing the technology to propagate false information, and the chatbot platform should, at a minimum, have a policy in place to address such instances.

Topic-specific safeguards: The models’ responses were somewhat topic dependent. This calls for implementing specialised safeguards for topics with a history of associated disinformation efforts. This, for instance, could involve displaying standardised “resources info boxes”, with additional authentic sources when a user inputs disinformation or conspiracy theories. This approach would help reduce the generation of inaccurate information and strengthen prevention mechanisms.

Embedded disclaimers: ChatGPT relied more often on disclaimers than Bard during our experiment. These, however, are all easy to cut from the text or to simply ignore. The models should explore the use of embedded disclaimers that appear contextually in the event of potentially false or misleading prompts, and encourage users to provide feedback on disinformation-related outputs. For instance, the chatbots could use hyperlinks and refer to factual sources within the text.

Regulatory compliance and ethical oversight: This experiment showed how malicious actors could potentially use these chatbots as disinformation weapons. This underscores the importance of current negotiations between AI service providers and regulators to establish robust ethical oversight and a framework for accountability, transparency, and compliance with regulations governing AI-generated content. 

Democracy Reporting International's Disinfo Radar project, funded by the German Federal Foreign Office, aims to identify and address disinformation trends and technologies. Kindly take a moment to share your insights by participating in our survey. You may also register for our newsletter (select Digital Democracy to receive our Digital Drop, a newsletter dedicated to this topic). Your feedback contributes to our ongoing efforts in enhancing our research and promoting informed discourse.

Table Of Contents

Documents

Disinfo Rapid Response Brief - ChatGPT vs Bard Download
0824_August_Disinfo_Layout_Annex Download

This work is supported by