Share
Download the
Global Threat
Intelligence Report
This article explains our recent paper "Detecting Phishing Sites Using ChatGPT" [1] published in June 2023. The author of this article is Takashi Koide.
Can ChatGPT detect phishing sites?
The use of artificial intelligence (AI) in cyber attacks has become a growing concern in the security community. ChatGPT has the potential to automate various malicious activities, such as generating phishing sites with evasion techniques and creating malware code [2,3]. Defending against these AI-driven cyber attacks is crucial. However, there hasn't been much discussion about the use of ChatGPT in cybersecurity, particularly for analyzing and detecting malicious websites. In this article, we aimed to address this gap by investigating the effectiveness of ChatGPT in detecting phishing sites. The results were promising, with ChatGPT (GPT-4) achieving an accuracy of over 98% in detecting phishing sites.
Get started
The goal of our experiment is to leverage ChatGPT to determine whether a website is a phishing site or not. Phishing sites are designed to deceive users by impersonating legitimate web services through social engineering techniques. For example, they may abuse well-known web service logos and brand names, or display false alerts about virus infections or cash prizes. To identify a phishing site, we rely on crucial clues found in the URL, HTML, and screenshot.
Determining the authenticity of a URL is essential in recognizing a phishing site created to mimic a legitimate website. The text included in the HTML of phishing sites may contain psychological manipulation directed at users. Moreover, the HTML contains not only text but also structural elements such as login forms and credit card number input fields. ChatGPT is expected to perform such advanced analysis of natural and programming languages. Since text displayed in image files cannot be seen in HTML, we used Azure Cognitive Services OCR (Optical Character Recognition) to convert screenshot images into text data in this experiment. Due to the token limitations of ChatGPT, we simplified the HTML and OCR-extracted text to fit within the 4096 tokens of GPT-3.5 for this experiment.
Prompt Engineering
For tasks that require high accuracy, prompt engineering (https://www.promptingguide.ai/) is essential for improving ChatGPT's response. As a preliminary analysis, we tried a simple prompt to determine whether a given HTML and URL represented a phishing site or not. However, the results often led to incorrect classifications, with non-phishing sites being misidentified as phishing sites. For example, a website could be labeled as phishing simply because it asked for a username and password on a login page.
Therefore, we designed the prompt to divide the task of determining the phishing site into four subtasks and guide ChatGPT to answer them step-by-step.
1. Social Engineering Analysis
- Identify whether the website contains text that psychologically manipulates the user. We have provided examples such as fake rewards (smartphones or gift cards) or false virus infection warnings.
2. Brand name extraction
- Compare the URL and the extracted brand's legitimate domain name to determine if the target website is impersonating a service or company.
3. Explanation of the determination process
- Give ChatGPT time to think about determining the phishing site. By explaining the reasoning behind its decision, we expect to improve response accuracy and make it easier to analyze responses.
4. Result Output
- Response in JSON format. The response contains four elements: phishing score, brand name, whether it is a phishing site or not, and the suspiciousness of the domain name.
Here is the prompt template. We enter the URL, HTML, and OCR extracted text from the target website into the prompt.
| You are a web programmer and security expert tasked with examining a web page to determine if it is a phishing site or a legitimate site. To complete this task, follow these sub-tasks: 1. Analyze the HTML, URL, and OCR-extracted text for any social engineering techniques often used in phishing attacks. Point out any suspicious elements found in the HTML, URL, or text. 2. Identify the brand name. If the HTML appears to resemble a legitimate web page, verify if the URL matches the legitimate domain name associated with the brand, if known. 3. State your conclusion on whether the site is a phishing site or a legitimate one, and explain your reasoning. If there is insufficient evidence to make a determination, answer "unknown". 4. Submit your findings as JSON-formatted output with the following keys: - phishing_score: int (indicates phishing risk on a scale of 0 to 10) - brands: str (identified brand name or None if not applicable) - phishing: boolean (whether the site is a phishing site or a legitimate site) - suspicious_domain: boolean (whether the domain name is suspected to be not legitimate) Limitations: - The HTML may be shortened and simplified. - The OCR-extracted text may not always be accurate. Examples of social engineering techniques: - Alerting the user to a problem with their account - Offering unexpected rewards - Informing the user of a missing package or additional payment required - Displaying fake security warnings. URL: {URL} HTML: ``` {HTML} ``` Text extracted using OCR: ``` {OCR-extracted text} ``` |
Results
Now let's try to detect phishing sites using ChatGPT. We conducted an experiment using the above prompts to determine if a website is a phishing site. We prepared two types of data for the experiment: phishing and non-phishing sites. The seed URLs of phishing sites were collected from two phishing intelligence sources, OpenPhish (https://openphish.com/) and PhishTank (https://phishtank.org/), as well as from phishing victim reports posted on Twitter [4]. Using our web crawler, we collected phishing sites impersonating a total of 147 brands. These phishing sites were distributed in 22 different languages. For non-phishing sites, we collected URLs of legitimate websites of brands targeted by phishing sites and top site lists, Tranco (https://tranco-list.eu/). Non-phishing sites were distributed in 34 different languages.
- Phishing sites: 1,000
- Non-phishing sites: 1,000
We used Azure OpenAI Service to query the GPT-4 and GPT-3.5 APIs. We analyzed the JSON text in the responses, and if either the "phishing" key or the "suspicious_domain" key (or both) had a value of true, we counted the website as phishing; if both were false, we counted it as non-phishing.
As a result of the experiment, the true positive rate was 86.7% for GPT-3.5 and 98.4% for GPT-4, and the false positive rate was 1.5% for GPT-3.5 and 1.7% for GPT-4. ChatGPT was indeed able to detect phishing sites with high accuracy.
The following table shows the results. Both GPT-3.5 and GPT-4 had the same ability to identify non-phishing sites. However, GPT-3.5 had many cases of mistakenly identifying phishing sites as non-phishing sites (133 False Negatives). On the other hand, GPT-4 significantly improved this misidentification with only 16 False Negatives.
GPT-3.5:
| Predicted Phishing | Predicted Non-Phishing | |
| Actual Phishing | 867 | 133 |
| Actual Non-Phishing | 15 | 985 |
GPT-4:
| Predicted Phishing | Predicted Non-Phishing | |
| Actual Phishing | 984 | 16 |
| Actual Non-Phishing | 17 | 983 |
Example of ChatGPT’s Response
Now let's look at examples of ChatGPT responses that correctly detected phishing sites.
Example 1:
Response:
| 1. Suspicious elements: - The URL contains a subdomain "securitypages", which is unrelated to the legitimate Facebook domain (www.facebook[.]com). - The extracted text alerts the user about having 12 hours to cancel the account. This is a social engineering technique often used in phishing attacks. - Furthermore, the link provided redirects to a suspicious, non-legitimate URL: "http://centrenotificationvices.duckdns[.]org/cntre.html". 2. The brand name identified is Meta Facebook. 3. The site appears to be a phishing site because of the suspicious URL and social engineering tactics in the text. 4. JSON-formatted output: ``` { "phishing_score": 9, "brands": "Meta Facebook", "phishing": true, "suspicious_domain": true } ``` |
This website, which impersonates Facebook, uses various tactics to deceive users and create a sense of urgency regarding their account issues. Upon analyzing the website, ChatGPT correctly identified it as associated with the brand "Meta Facebook." It also recognized several suspicious elements that indicate the website's illegitimacy. The URL is unrelated to the official Facebook domain name (www.facebook[.]com). This discrepancy raises concerns about the website's authenticity. Additionally, the text displayed on the website follows a common social engineering technique used in phishing attacks. It warns the user about a limited 12-hour timeframe to cancel their account, aiming to manipulate them into taking immediate action. Furthermore, ChatGPT detected the presence of suspicious links within the website's HTML. One such link redirects users to a non-legitimate URL. This type of redirection is often associated with phishing attempts. Based on these findings, ChatGPT assigned a phishing score of 9 to this website, indicating a high likelihood of phishing. It flagged both the phishing and suspicious_domain attributes as true in the JSON-formatted output.
Example 2:
Response:
| 1. Analysis: - The HTML title is misleading and does not match the content of the page. - The URL is not associated with the DHL brand and is suspected to be a phishing URL. - The OCR-extracted text displays social engineering techniques by alerting the user of a missing payment required, offering an option to pay, and pressuring the user to complete the payment to avoid any delay in delivery. 2. Brand name: The brand name is DHL. 3. Conclusion: The site appears to be a phishing site, based on the suspicious URL, misleading HTML title, and social engineering techniques used on the page. 4. JSON-formatted output: { "phishing_score": 9, "brands": "DHL", "phishing": true, "suspicious_domain": true } |
In this example, we encountered a phishing site that impersonates DHL, with the intention of tricking users into providing their financial information by claiming a payment is required to receive a package. ChatGPT accurately identified the brand name as DHL and correctly recognized the social engineering technique employed, which involves requesting payment for import duty/tax expenses. It also determined that the domain name used does not belong to the legitimate DHL website. Additionally, by analyzing the HTML structure, ChatGPT provided detailed insights into the composition of the phishing site, which aims to steal credit card information like card numbers and CVC numbers.
Example 3:
Response:
| 1. There are clearly some social engineering techniques used in this website's code and text. Using "Congratulations" and "You are Today's Winner" are prime examples of an attempt to lure visitors into providing their personal information. 2. The brand name isn't clearly mentioned on the web page. The mentioned prizes such as Samsung Galaxy, Apple iPad, Apple iPhone, Xbox, Playstation, and Dyson products might lead to confusion but they're not the brand that hosts this giveaway. 3. I believe this site is a phishing site because it uses social engineering techniques to capture sensitive information from users, and the URL and domain name do not seem legitimate for hosting a sweepstakes or genuine giveaway event. 4. JSON-formatted output: { "phishing_score": 8, "brands": None, "phishing": true, "suspicious_domain": true } |
This phishing site does not impersonate any services, however, it displays fake reward of the iPad Pro prize to collect victims personal information. ChatGPT correctly detected social engineering techniques used in this website, such as showing "Congratulations" and "You are Today's Winner” for luring victims’ intentions. ChatGPT has successfully identified this phishing site, which does not impersonate any specific brands or services, but uses social engineering techniques to trick users. The site displays a fake reward of an iPad Pro prize as a method of collecting victims' personal information. In its analysis, ChatGPT highlighted the use of phrases such as "Congratulations" and "You are today's winner" as tactics used by the website to trick unsuspecting visitors into providing their sensitive information. With this detection, ChatGPT demonstrates its ability to identify potential phishing threats without brand identification.
Conclusion
We have presented the results of an experiment on detecting phishing sites using ChatGPT. This experiment showed that GPT-4 can detect phishing sites with over 98% accuracy. In particular, we found that ChatGPT has an excellent ability to determine whether a domain name is legitimate and to identify social engineering techniques used in phishing sites. In addition, ChatGPT can perform advanced contextual analysis, such as whether the text displayed on the web page is psychologically manipulative. We will continue to explore the potential of ChatGPT and use it to analyze various cyber-attacks in the future.
References
[1] Takashi Koide, Naoki Fukushi, Hiroki Nakano, and Daiki Chiba. 2023. Detecting Phishing Sites Using ChatGPT. https://arxiv.org/abs/2306.05816
[2] OPWNAI : Cybercriminals Starting to Use ChatGPT - Check Point Research https://research.checkpoint.com/2023/opwnai-cybercriminals-starting-to-use-chatgpt/
[3] Sayak Saha Roy, Krishna Vamsi Naragam, and Shirin Nilizadeh. 2023. Generating Phishing Attacks using ChatGPT. https://arxiv.org/abs/2305.05133
[4] Hiroki Nakano, Daiki Chiba, Takashi Koide, Naoki Fukushi, Takeshi Yagi, Takeo Hariu, Katsunari Yoshioka, and Tsutomu Matsumoto. 2023. Canary in Twitter Mine: Collecting Phishing Reports from Experts and Non-experts. https://arxiv.org/abs/2303.15847