Automated Security Testing Using ZAP Python API

By Amit Kulkarni

Automated Security Testing using ZAP API can help in finding early vulnerabilities. The security tool and API used is OWASP ZAP, which stands for open web application security project zed attack proxy. OWASP ZAP will help automate security tests to include in the Continuous Integration/Continuous Delivery (CI/CD) pipeline for your application, using the existing functional regression test-suites and ZAP Python API. 

Please see the reference section for further reading.

A Definition Of Security Testing

Security Testing is a process of identifying weakness in the security mechanisms of an application that protects data and maintains specified functionality. Purpose of security testing is to be aware of possible threats from SQL injections, cross-site scripting, and sensitive data exposure. It can also help to protect against Unauthorized and Unauthenticated users from changing or disrupting access to an application.

SQL(Structured Query Language) injection is inserting queries into a system which tamper with data. Cross-site scripting (XSS) is using malicious scripts to change or modify normal scripting traffic from a trusted application. 

Fig.1: What is Security Testing [10]

 

OWASP ZAP

OWASP ZAP security tool is an open source. It is OWASP’s flagship project which means it’s the most mature and most suitable for people to adopt for security testing purposes. It is ideal for beginners because the UI is very easy to use.  

ZAP Features

ZAP is built with a Swing [12] based UI for desktop. It also has a comprehensive rest API for daemon mode which means ZAP can be started or enabled by running .bat file with daemon flag letting it run as a background process. 

ZAP works as spider [13]. It crawls the web, explores the specified site and finds the URLs.  There are two kinds of spiders: traditional and ajax spiders. Ajax spider types are mainly for JavaScript applications. ZAP has two scanners, Passive and Active, that are used for scanning and finding vulnerabilities. 

  • Passive scanner monitors the requests-responses and identifies vulnerabilities
  • Active scanner attacks and manipulates the header for finding vulnerabilities. 

Both scanners are highly configurable. There are scan policies, in other words scan rules for active scanning. These are highly scriptable as various configuration parameters can be passed. The active scan policies can be configured in UI as shown in below fig.2.

Fig.2: Active Scan Policy

Below is an example of the OWASP ZAP UI. In the tools option tab you can set the proxy, address, and port you want ZAP to monitor. ZAP acts as a Man-in-the-middle [14] proxy which uses the concept of an attack proxy.

Fig.3: ZAP UI and ZAP UI Proxy

Using ZAP UI 

To start using this security testing tool, Open ZAP UI and set a proxy, address and port. Enter the Application URL to perform security testing on and then click attack. 

The UI tool will start the spider process which will crawl the application to find all the URLs/Pages in that application. It will then perform passive scan and use a default scan policy for active scans to find vulnerabilities. A security report is generated and can be viewed under Report tab in the UI tool.

How To Automate ZAP

Automating Security Testing is achieved in three sequential steps: 1. Enable/start zap via API in daemon mode. It will be running as a background process so it can proxy the browser. 2. Start the UI regression test-suites (already developed using any automation tool) so that zap can proxy all the URL’s/pages opened by regression tests, and 3. Use spider ZAP API to find any additional pages and then use scan and reporting functions provided by the ZAP API to find vulnerabilities and generate a security report for the same.

Fig.4: Security Testing in CI/CD

Step 1: Enabling ZAP In Daemon Mode

Fig.5: ZAP Batch Command 

ZAP is enabled by running the ZAP.bat file. This .bat file passes the daemon configuration parameters. 

Similar to the UI, we can set the parameters by passing configuration parameters such as host and port info. Similarly other parameters can be passed like ‘apikey’. Every zap ui/tool installed will have unique api key which can be generated within the tool in options tab. One has to use that specific api key to use zap api.

If you are using outgoing or corporate proxy then a proxychain is enabled and once it’s enabled you can specify hostname and port by passing parameters. If you are using authentication then you would need to specify a username and password.

By using this ZAP.bat file, it starts ZAP and runs it as a background process.

Step 2: Running UI Functional Tests

Once ZAP is started, then kick-off UI functional tests. For better scan results, it is important to have exhaustive regression tests exploring the application.Tests implemented in any UI automation tool can be run. 

Step 3: Scan & Report

Once regression test suite is finished, run spider to find additional pages not covered by functional tests and then all these pages/url’s will be scanned by both passive and active scanner to generate a report.  Enabling ZAP, Spidering and scanning is all done by calling ZAP Python API.

Once the ZAP is enabled and proxy is set say for e.g. localhost:8090. Open browser and type localhost:8090 and ZAP API UI will open as shown in Fig.6 below. As you can see the API calls like ‘MaxChildren’, ‘MaxDepth’ can be called and set in API calls similar to API UI. The ZAP API client is available in various languages such as java, python and nodejs.

Fig.6: ZAP API UI

 

ZAP Python API – Install

To get the Python API package, install Python2.7 or higher which contains the pip package. Pip stands for preferred installer program. 

Pip is a package management system used to install and manage software packages written in Python. ZAP Python API can be installed using pip install command and specifying python-owasp-zap version as explained here [4].

ZAP Python API – Import

Fig.7: ZAP Import

Once the ZAP Python package is installed, you can import it by using the command:

 `from zapv2 import ZAPv2` 

Similarly, set the target URL for the scan and then set the `apikey`, host, and port via the proxies as shown. 

`zap.urlopen(target)` - requests url proxy through zap.

ZAP Python API – Spider

Fig.8: ZAP Spider Scan

Once you have set up a proxy, then make a simple spider API call by calling ‘zap.spider.scan’ and pass the UI parameters needed. The API takes all the parameters such as ‘url’, ‘apikey’, ‘MaxDepth’, ‘MaxChildren’. 

Once the spider API is called it waits for its completion by pooling status API. When status equals 100, spidering process is complete. Spider doesn’t do any scan. It crawls the web application to follow links, sources, anchors in HTML, JS and CSS and adds to the source tree, creating a hierarchical data structure. This tree is then traversed by passive or active scan to perform vulnerability analysis. 

ZAP Python API – Passive Scan

 Fig.9: ZAP Passive Scan

Similarly, passive scan API is called by ‘zap.pscan.records_to_scan’. This API waits till all the records are scanned. Passive scan just looks at the requests and responses rather than making any additional requests. This is good for finding problems like missing security headers or missing anti CSRF tokens (Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application). But is no good for finding vulnerabilities like XSS which require malicious requests to be sent – that’s the job of the active scanner.

ZAP Python API – Active Scan

Fig.10: ZAP Active Scan

Similar to spider, active scan API is called by ‘zap.ascan.scan’ API which starts the active-scan process. Once the active scan API is called it waits for its completion by pooling status API. Active-scan is complete when status equals 100. Active scanner performs a wide range of attacks. Active scan policies can be set similar to UI tool shown in Fig.2 by calling ZAP API which attacks and make requests to the server by modifying the header. Active scanning is a real attack on those targets and can put the targets at risk, so do not use active scanning against targets you do not have permission to test.

ZAP Python API – Report

Fig.11: ZAP Report

After the spider and scans are complete, call the report API by ‘zap.core.htmlreport’ and passing ‘apikey’ parameter.  A report can be generated in both HTML and XML format.

ZAP Typical Report

 Fig.12: ZAP Scanning Report

This is how a typical ZAP report will look. It mentions things like the Risk Levels and the number of alerts associated with it. In the detail section, it lists the affected URL and possible solution to fix it. 

SQL injection is at high risk because SQL injection attacks allow attackers to tamper with existing data, change transactions, allow complete disclosure of data, can destroy the data or make it unavailable, and also can become administrators of the database server.

Top 10 Application Security Risks

Fig.13: Top 10 Security Risks [9]

These are the top 10 application security risks. Over the last few years, the fundamental technology and architecture of applications has changed significantly which has caused some of these risks to move between 2013 and 2017. SQL Injection and Broken Authentication are still top 2 security risks as it gives complete control of the system.

ZAP Python API : Closing

Fig.14: ZAP Close

Once the report is generated, ZAP can be closed by calling ‘zap.core.shutdown’ API which makes ZAP exit from daemon mode and is no longer listening or proxying the browser.

Advantages Of Automated Security Testing

Fig.15: Continuous Integration Continuous Deployment [11]

Finding vulnerabilities early in development cycle helps the team reduce security issues and cost. As an Agile testing strategy, Security testing needs to be integrated into continuous integration and delivery cycle to find and prevent basic vulnerabilities. This allows penetration testers to address more critical issues. This will not remove the need for additional security testing, or third party security testing, but it might serve as an early warning system while the product is being developed.

References

You can look here [1] for details about ZAP UI. There are many ZAP API clients [3] provided like Java, Python, Node.js, PHP and Ruby. Of which only Java[7], Python and Node.js[8] are Official API. I have chosen Python as it is highly maintained and the most popular one based on GitHub stars. There is an existing article covering automated security test using ruby [6].

  1. https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project
  2. https://github.com/zaproxy/zaproxy
  3. https://github.com/zaproxy/zaproxy/wiki/ApiDetails
  4. https://github.com/zaproxy/zap-api-python
  5. https://github.com/AmitKulkarni9/AutomatedSecurity_PenetrationTesting
  6. https://www.swtestacademy.com/automated-security-testing-using-zap
  7. https://github.com/zaproxy/zap-api-java
  8. https://github.com/zaproxy/zap-api-nodejs
  9. https://www.owasp.org/images/7/72/OWASP_Top_10-2017_%28en%29.pdf
  10. http://www.locateqa.com/wp-content/uploads/2018/06/Untitled-3-9.jpg
  11. https://galilsoftware.com/wp-content/uploads/2015/05/Continuous-Integration.png
  12. https://en.wikipedia.org/wiki/Swing_(Java)
  13. https://en.wikipedia.org/wiki/Web_crawler
  14. https://www.owasp.org/index.php/Man-in-the-middle_attack

Author Bio:

Amit is a test automation specialist and has over 10 years of experience delivering quality product for clients in Australia, Europe, Japan and USA. Amit is passionate about test automation for both functional and non-functional including security and performance testing. Amit has a diversified experience in Semiconductor technology, Embedded and Software industry in the field of Banking, Healthcare and Consumer Electronics working for companies such as Fujitsu, Mitsubishi, Qualcomm and Telstra. Amit is currently working as a Senior Quality Engineer in Bankwest and driving best quality practices in test automation and continuous testing to deliver quality product faster.
LinkedIn: https://www.linkedin.com/in/amtoya/
Twitter: https://twitter.com/amtoya
Website: https://amtoya.com/