Phishing Maturity Comparison Project

Author: Jack Kelly, Work Experience Student

In the 2018 Verizon annual data breach investigations report[1], it is stated that ransomware has increased by 100% in the last year and is now sitting at 39% of all malware attacks. Of these ransomware attacks only 1 out of 4 come from vulnerabilities in the computer systems, the rest originate from people as 96% of modern malware attacks are now delivered through the inbox of a user rather than using a vulnerability in the computer system, this figure comes as no surprise.

This threat is a reoccurring reason to prevent a new wave of attacks against computer networks and to use our most important asset as a defence strategy – our people.

I am Jack Kelly and I am spending two weeks on a work experience placement at cyber security firm Sapphire. One of the tasks I have been working on, is to analyse a segment of anonymised data which comprises of 1,200 users from the 25,000 users of Sapphire’s managed phishing awareness service to test the phishing vulnerabilities of modern companies.

The industry average for phishing varies between industries and can often depend on the size of the organisation and its maturity in relation to phishing awareness and training. When carrying out an initial phishing test to an organisation who has yet to have delivered any sustained phishing and awareness training programme to its users, the average click rate can vary between 15%-34%.

The company whose data I analysed as part of this project was chosen specifically as a mature phishing testing and training company who had been carrying out a sustained training programme over a period of two years. My task was to analyse their data to compare their phish-prone average against the industry averages for organisations who have not adopted phishing testing and training.

In the initial run of each test on the test company we sent every user a phishing email before any training was given and the average result or click through rate from all staff was 6.45%. This is used as a baseline throughout the rest of the test and analysis of the data. The following actions were also taken into consideration: clicking on a link embedded in the email, opening an email attachment and allowing a macro to execute on computer due to opening an attachment.

After these results the users received staff training against phishing attacks and when analysing results, I saw a significant decrease in the click rate of data analysed. Within 4 months the click rate dropped to 3.24% and with this 1 in 2 users who had failed the test previously now passed. By month ten, and almost at the end of the phishing training, the click rate for all users had reduced to 1.21%. With this 5-fold improvement overall all test subjects should be far more resistance to phishing attacks as emails now have to reach 1.21% of users rather than 6.45% of users to have an impact. The improvement means that only 1 in 5 people who failed the test at the beginning of the phishing exercise are still vulnerable to phishing attacks.

As part of this project, I was able to separate the users into two groups, those that completed the phishing training and awareness programme and those who did not and then analyse this further. The data shows that users who remained untrained had their click rates fluctuates between 2%-5%. They started the test on 4.65% and ended on 2.01% but had peaks of 4.79% just the month before and had little consistency in results. This shows little to no improvement in the results of the untrained group.

The trained group on the other hand show consistent improvement on aggregate. They started with a 7.71% click rate and ended with a 0.92% click rate. The trained users’ data and the data of all the users for failure rate has a correlation coefficient of 98.67%. This shows the effectiveness of the staff training as with each decrease in the trained users we see the result mirrored to the over the all user data. They had a strong negative correlation which shows improvement overtime the more training they were given.

I have also tested the significance of the data and it has been shown to be mathematically significant with a probability of happening by chance of less than 0.2%. The click rate was reduced to 1.21% which is a drastic improvement in the capability of our human firewall will lead to a large security improvement without requiring extra IT hardware or software.

This exercise clearly shows a strong correlation between training users and phishing failure rate and the negative trend of the failure rate with training compared to the lack of trend in the untrained data shows that the training is effective at reducing the phishing failure rate. It also demonstrates the benefits in carrying out a sustained phishing testing and awareness programme rather than a one-off test or short-term project.

I would personally recommend any business that can should receive phishing training and testing to reduce their chance of adding to the 3 out of 4 companies that get infected with malware due to phishing attacks.  Verizon’s report recommends that you make your staff the first line of defence through training and this is backed up by Sapphire whose data shows that training is highly effective at reducing the failure rate of the company.





Tags: ,