How to choose the right way to test security of website or application? When are the automated scan, penetration test or bug bounty program the right choice? And why is not internal testing sufficient? You can find the answers in our comprehensive article on online security testing.
In the world of information technologies, there are just a few ways you can test the security of websites or mobile apps. While companies neglecting security consider installing an antivirus program to be the utmost cyber-protection, advanced companies know that their security needs to be examined into depth. This is the job of:
- “Code reviews” and code security testing,
- automated scans of vulnerabilities,
- penetration tests – automated or manual, in-house or external,
- bug bounty programs – private or public,
- monitoring, static and dynamic analytical tools.
In this article, we compare the most common form of testing – penetration tests (and their cheaper version of automated vulnerability scans) with modern bug bounty programs.
What is a penetration test?
The penetration test is a one-time test of your IT security, and it is done in automated way – using specialized scans and tools, and / or manually – focusing on a group of the most common and most dangerous vulnerabilities. The result is a detailed report on security vulnerabilities in your system, including checking their deeper impact.
Don’t mistake it for a vulnerability scan
Vulnerability Assessment / Vulnerability Scan is often coupled with the concept of penetration testing. This is an automated test to discover the known vulnerabilities of your product or service resulting in a list of vulnerabilities with their location. However, it is not a full penetration test.
You can imagine the penetration test as a rather complex black-hat hacker attack on your system. With the difference that the pen-test is performed by an ethical hacker that works for you.
Pen-test has a wider and deeper purpose – besides the list of vulnerabilities, it will also tell you what the system can do for you. Additionally, the penetration test also provides verification of the vulnerabilities found that is done manually.
The most common vulnerabilities discovered in penetration tests
There is a number of automated penetration testing tools (Netsparker, Burp, Kali Linux, etc.) including EdgeScan, which has also compiled a list of the most commonly discovered vulnerabilities and bugs in online applications and on the web:
- Vulnerabilities associated with web browser attacks (61%):
- XSS (86%),
- CSRF (5%),
- HTML Injection (3%),
- DOM (3%),
- Open Redirection (2%),
- Response Splitting (1%),
- cryptographic vulnerabilities (17%),
- session management (9%),
- attacks by inserting malicious code (4%),
- authorization issues (4%),
- information leakage (3%),
- incorrect use (1%),
- unavailability of services (1%).
Advantages and disadvantages of penetration tests
| Advantages | Disadvantages |
|---|---|
| Complexity. | High price. |
| Possibility to test unpublished software / hardware. | Describes the safety of the product / service only as of the date of the pen-test. |
| Output quality including manual verification. | Time-limited testing. |
| Stricter contractual relationships and greater legal guarantees. | It tested by only a small number of testers. |
| Especially suitable for testing in a test environment. | |
| When using an inexpensive scan without manual verification, the output is insufficient and contains a large number of fake reports. |
Advantages and disadvantages of penetration testing.
Complexity can be considered one of the biggest advantages of more expensive penetration tests (doesn’t apply to use of cheaper automated vulnerability scans). One or more ethical hackers employed by the pen-test company will test the different levels of your product (within a range set by you). Penetration tests usually follow the OWASP standards.
A time limit is the downside, as it restricts the scope of testing – the pen-test describes the level of product or service security only at a given time. Manual control after automatic penetration is also largely subject to the individual experience of the tester. And skills varies among the ethical hackers.
The advantage of penetration testing is also the possibility to test those parts of your system that you do not want to have publicly tested by ethical hackers in bug bounty programs. This allows for stronger contracts between the company that has its own testing system and the penetration testing firm. Such contracts also include an NDA with precise test specifications, including fines for non-compliance. This is, in the case of public bug bounty programs impossible (private programs allow it, however).
Are pen-tests worth their price?
A very important parameter of penetration tests is their cost. According to Security Metrics, price for complex penetration tests ranges from $ 4,000 to $ 20,000. By comparison, the average penetration test in Slovakia or Czech Republic is about EUR 6,000.
The high price of penetration tests is a major problem for some companies. Especially smaller companies can not afford the pen-tests and security audits because of the high price. However, in the Anglo-Saxon world, it is a popular way of testing security. Last year, companies from around the world spent approximately $ 600 million for penetration tests. According to forecasts, by 2021 this amount should be threefold.
What is a bug bounty program?
Our readers are aware of the bug bounty program concept. It is a continuous security test that allows businesses to prevent cyber attacks, theft of data and abuse. Security testing is carried out by ethical hackers who receive pre-specified rewards for found errors and vulnerabilities related to the vulnerability of services and applications.
| Advantages | Disadvantages |
|---|---|
| Variety of outputs over a longer period of time. | You can test only software/hardware which is available online. |
| Low price (you set the amount of the rewards and you pay only for valid vulnerabilities). | The range of the test is set by a company which might be not properly experienced. |
| Wide range of testers (ethical hackers) with different experiences and education. | A lot of false vulnerability reports. |
| Long-term (continuous) testing of your security. | Ethical hackers may not be interested in bug bounty programs with low rewards. |
| Possibility of testing in production and/or test environment. | Weaker legal guarantees. |
| Bug bounty program can find rarity outputs that pentest can’t detect. | Less complexity in compare with penetration tests. |
Advantages and disadvantages of bug bounty programs
The biggest difference with the penetration test is that the bug bounty program offers continuous security testing at a price that the company itself sets. The company will determine the amount of rewards for ethical hackers for those types of vulnerabilities that it finds interesting. Additionally, individual vulnerabilities can be tested sequentially, with lower payouts, and so the cost is divided over time.
In bug bounty programs, rewards are paid only if the ethical hacker actually finds relevant vulnerability in the system. For comparison, the company must pay for the penetration test even if it does not detect any safety vulnerability.
Bug Bounty testing itself is not limited to one ethical hacker or a smaller set of testers, as is the case with a pen-test. Your product in the bug bounty program is tested by tens of hundreds of ethical hackers who even compete with each other – finding the error as the first to guarantees them a reward.
The disadvantage of bug bounty programs is, however, in most cases is their public character. It determines that, in the vast majority of cases, ethical hackers can only test publicly accessible parts of websites, applications, or interfaces. Applications with no outer access can not usually be tested. On the other hand, it is purely the decision of the firm to let the hackers test the production or testing environment.
The most common vulnerabilities discovered in bug bounty programs
Annually, tens of thousands of vulnerabilities are reported to bug bounty programs. The well-known bug bounty platforms speak of more than 44,000 reported vulnerabilities (Hackerone) or 37,227 vulnerabilities through the Bugcrowd program. The first-named platform paid the ethical hackers more than $ 10 million a year, and in the case of BugCrowd, the value of the highest hacker reward was $ 10,000 for a found vulnerability.
According to this year’s Hacktrophy statistics, the following types of vulnerabilities are reported most frequently:
- Other – vulnerabilities outside the TOP 10 OWASP (10.35%),
- SPF configuration problems (5.52%),
- XSS (4.83%),
- CSRF (4.16%),
- Session fixation (4.16%),
- SSL / TLS (4.16%).
When we look at the most commonly reported types of critical vulnerabilities, according to American Bugcrowd, these are the following:
- SQL Injection (63%),
- XSS (18%),
- CSRF (14%),
- vulnerabilities of mobile apps (6%).
Not every vulnerability found means a reward to hacker
The security and quality of bug bounty programs is guaranteed by their rules. These include, among other things, the commonly used principle of paying the reward to the first discoverer. In order to receive a reward, the vulnerability must be verified and confirmed. Therefore, the ratio of valid vulnerabilities to the total number of reported vulnerabilities, which are often fake or duplicate, or those that do not belong to the assignment, is important:
| Bug bounty program/ platform | Hacktrophy | Bugcrowd | Hackerone | ||
|---|---|---|---|---|---|
| Ratio of valid vulnerabilities | 23,3% | 18,5% | 23% | 5% | 4% |
How should the company choose a form of online security testing?
Quality penetration tests are costly and time-limited. They are performed by security specialists, whose time is expensive and limited (salary pen-tester is up to 200 EUR / hour). In addition, they require relatively demanding preparation – both formal and technical setting of test conditions (as they are usually tested in a test environment rather than a production environment).
However, they present a comprehensive picture of the current state of the security of your online project and of the infrastructure on which the project stands. The security company usually has 1-2 weeks to prepare a pen-test and its output is a manually validated report with vulnerabilities, their location, and possible impact in the case of abuse. The advantage is that even unpublished sites, applications, and products can be tested.
Even purely internal security testing is not very cost-effective. If you would like to replace pen-test with your own security expert, the penetration test or the bug bounty program is just a fraction of the amount you would pay for your services. According to PayScale, the average annual salary of a security tester / ethical hacker is about $ 80,000. And we have not yet included the price of a high-quality automatic penetration scan license.
By contrast, bug bounty programs are more cost-effective. It is assumed that they will take a longer period (at least several months), which will spread over time the cost of rewarding white hat hackers. It’s up to the business what the ethical hackers will test and how much money they will pay for their security flaws.
However, a period without found vulnerabilities can occur – especially with badly set rewards that would be unattractive for ethical hackers. On the other hand, relatively high-quality vulnerability output often created by detailed manual testing means the benefit to a business. This is a fundamental difference compared to cheaper automated scanning – the bug bounty program involves a number of ethical hackers from around the world with diverse experiences and education. They can also find rare security vulnerabilities that sometimes even the most expensive penetration tests fail to detect.
The biggest benefit to businesses is the guarantee that they will pay for detecting real security vulnerabilities. In addition, bug bounty project setup is easier than penetration testing, and the production version of the website or application can be tested. Moreover, a qualified moderator can help the company, including the help with evaluation of the report.
Basic cycle of online project security testing.
Pen-test + bug bounty program = higher security
It would be a big mistake to perceive bug bounty programs, penetration tests and internal testing as opposed forms of online security checking. Neither of them is able to reveal all potential risks and vulnerabilities through which it is possible to penetrate the system and steal data.
In reality, it should be a complementary way of using which will bring a higher level of online project security. Before an official website or online service is published (and always after a major update), it is advisable to do internal testing and then a complex penetration test that will reveal the greatest security vulnerabilities (see graph).
Once removed, the product or service is available to the public in the form of a bug bounty program. It tests publicly available components of the system. It is up to the firm to decide whether to allow ethical hackers to a production or testing environment. It is important that testing in the bug bounty program takes place over a long time (at least 3 to 6 months, ideally even longer). This will allow the hackers who have recently registered to alert you of security bugs caused, for example, by minor updates on your system.
If the company’s security testing budget is limited, the alternative process is: 1. Internal security test, 2. Basic vulnerability scan (possibly without it), and 3. Creation of bug bounty program. In this case, however, the company must count with the possibility that their system may contain more serious vulnerabilities for some time (until the detection of vulnerabilities by ethical hackers).
| Basic Vulnerability Scan | Penetration test | Bug bounty program | |
|---|---|---|---|
| Depth of testing | Finding of only basic vulnerabilities,that are in well-known databases | It is determined by the client. Variable, time-limited. It usually goes into great depth. | Determined by the extent of the client’s interest. Suitable for testing online accessible applications. |
| Time range of testing | Time-limited. This is a one-time test. | Time-limited. This is a one-time test. | Specified by the client. This is usually a long-term continuous test. |
| Test subjects | Private or public. | Private or public. | Public. |
| Expertise of the test | Automated tool with or without manual verification of vulnerabilities found. | Combination of automated scanning and manual verification. | Spectroscopic testing according to the expertise of individual hackers. Typically, a combination of scans and manual verification. |
| Place of testing | A test environment is recommended. | A test environment is recommended. | Suitable for both test and production environments. |
| The price | Approximately 150 to 3000 Euros. | 3000 to 20,000 Euros. In Slovakia and in the Czech Republic, one test is worth an average of 6000 euros. | Variable, paid according to the number and type of vulnerabilities found + the cost of using the service and advice. |
Comparison of automated vulnerability scan, penetration test, and bug bounty program.
Not sure? Recognizing all the features of penetration tests, vulnerability scans and bug bounty programs is not an easy task at all. If you’re not sure what is the best form of online security testing, ask us! We will be happy to advise you on IT security issues and, if necessary, help you with the creation of your first bug bounty program.
