The AI threat landscape is rapidly evolving. Natural Language Processing (NLP) enables seamless interaction with AI systems through conversational interfaces. However, as we increasingly rely on AI for productivity, new risks emerge.
Recent research from the University of Sheffield has shown that NLP models like ChatGPT can be misused to produce harmful malware code, posing new risks of cyberattacks [1]. The study is the first demonstration that these models can be exploited as attack vectors in the wild.
The researchers conducted both blackbox and backdoor attacks on six different commercial text-to-SQL systems [2], which included ChatGPT, BAIDU-UNIT, AI2SQL, AIHELPERBOT, Text2SQL, and ToolSKE. The research uncovered security flaws in all the systems tested, which could lead to data breaches and disruption of databases, including deletion or corruption and denial-of-service (DoS) attacks to overwhelm databases and cause outages. For instance, an individual, such as a nurse, might request ChatGPT to generate SQL commands designed to carry out malicious actions against a vital database, like a healthcare database system. This could result in the system crashing or revealing confidential and sensitive information.
The researchers also successfully created a proof of concept demonstrating how backdoor attacks on databases could be achieved by poisoning Text-to-SQL algorithms. They inserted ‘Trojan Horse’ backdoors into the algorithms by contaminating the training data with malicious samples.
As we look to the future, there is a need for collaboration in the AI community to develop cybersecurity strategies that can keep pace with emerging attack tactics. Furthermore, integrating cybersecurity principles within the machine learning pipeline will be critical to avoiding AI misuse. Of course, ongoing human oversight of AI systems is a fundamental requirement, ensuring that what begins as innocent exploration does not transform into criminal exploitation.
We must remain vigilant about potential downsides, all the while embracing the significant promises and benefits that AI offers.
[1] https://arxiv.org/pdf/2211.15363.pdf
[2] Text-to-SQL systems use a Natural Language Processing (NLP) interface that allows users to query databases using plain English instead of SQL.