Integrating fuzzing into DevSecOps

Fuzz testing is a highly effective technique for finding weaknesses in software. It’s performed by delivering malformed and unexpected inputs to target software while monitoring it to detect unwanted behavior and log failures. Fuzzing is widely recognized as a valuable technique for improving the security, robustness, and safety of software, and it’s an integral part of some of the best-known secure software development life cycle (SDLC) frameworks.

Over the last decade, software development tools and methodologies have become more integrated into continuous integration / continuous delivery (CI/CD) pipelines and security testing processes. Some testing methodologies integrate more easily than others though, so adoption and integration depth can vary widely.

The fuzz testing maturity model (FTMM) was created to measure the extent to which software builders and buyers have used fuzzing. FTMM describes six levels of testing maturity:

• Immature (level 0): This is software that has never been fuzzed.
• Initial (level 1): In this level, there has been initial exposure to fuzz testing. Although it’s not comprehensive, it’s an excellent first step for organizations looking to make quick improvements in robustness and security.
• Defined (level 2): In this level, companies are using a more defined approach for performing fuzz testing. The starting point is an attack surface analysis of the target.
• Managed (level 3): This level emphasizes completeness and documentation so it’s easy to observe and improve the fuzzing process. This is an excellent baseline for builders.
• Integrated (level 4): This level increases the fuzzing time per fuzzer type to one week. Fuzzing must be incorporated in the organization’s automated testing.
• Optimized (level 5): This level increases fuzz testing time to 30 days for each fuzzing type, and requires the use of at least two different fuzzers per fuzzing type.

FTMM sets the requirements for the types of fuzzers, the amount of testing, instrumentation, acceptable failures, processes, and documentation for each level. This blog post focuses on the benefits of leveraging fuzzing in existing SDLC processes and extending its use in CI/CD pipelines. It also addresses integration concerns with fuzzing technology.

Benefits of fuzz testing maturity model

In the Managed level of FTMM (level 3), fuzz testing is integrated into the secure software development process to detect vulnerabilities before the software is released. However, the process to set up the target software, configure and run the test tool, and report the findings is entirely manual. Fuzzing, as well as other dynamic application security testing (DAST) tools, require runtime verification of compiled, packaged, configured, and running software. The fuzz test tool must also be configured—it’s a good amount of work and multiple steps. If that work is left on human shoulders, chances are the first round of tests may not match the second, which will slow down subsequent diagnosis, fixing, and testing.
 

Repeatability

Therefore, repeatability is the #1 benefit of moving to the Integrated level of FTMM (level 4). The CI pipeline can be used to perform the build, static analysis, and software composition analysis of the code to locate quality issues and known vulnerabilities before the target software is deployed in the test infrastructure for fuzzing. The CI pipeline ensures the deployment is done in a repeatable manner—it handles the configuration of the target software to control the exposure of attack vectors to the system. It can also configure the fuzzer, connect it with the target software, start the test, and provide a report when the test is complete. In a best-case scenario, the CI system has a repeatable configuration as code covering tests for the whole attack surface.
 

Collaboration

Integrating the CI/CD pipeline with fuzz testing can also improve collaboration. Often, the security team is responsible for attack surface analysis and fuzz test planning, assisted by the development team. In the Managed level of FTMM (level 3), the security or QA team executes the tests, and the development team fixes discovered issues. When fuzzing is integrated into the CI pipeline, the tests are defined by the security team but developers also get direct access to results on a regular basis. If a test goes red, the development team checks for details of the fail, performs the analysis, and delivers a fix. As security teams have less trouble reporting issues, they can use more time on attack surface analysis and increase coverage over the attack surface.
 

Earlier issue detection in smaller increments of codebase changes

One of the benefits of integrating fuzzing with the CI/CD pipeline is earlier feedback. However, fuzzing is an open-ended pursuit—an infinite space problem. Can it keep up with the feedback rate expected and improve development speed?

Fuzzing takes time: Template fuzzers start with an example input, and the quality and feature coverage of the example affect how quickly the fuzzer starts producing meaningful tests. The speed is often great, but testing requires a large number of iterations. In generational fuzzing or model-based fuzzing tools such as Defensics®, all the test cases are “smart” but there are a lot of them, and the fuzzer takes time to encode and decode exchanged messages on each iteration. Fuzzing of a single attack vector easily takes hours. You could continue fuzzing infinitely, but at some point the likelihood of finding new issues is low.

The benefit of integrating fuzzing with the CI/CD pipeline might be less about testing speed and more about detecting issues earlier and getting feedback for smaller increments of codebase changes. Even though you would not trigger a fuzzer on every commit or even on every merge, there’s a chance to fuzz much more frequently than on major releases. Fuzzing on a nightly or weekly basis can bring benefits.

When a baseline of SUT robustness is achieved, fuzzing can be triggered automatically. Fails in fuzz testing results are often security issues that should break the build—notify development team to start a manual process to analyze and fix the issue, and optionally auto-block the software increment road to the production branch or production deployment. Once basic robustness is achieved via fuzz testing and penetration testing, QA and security teams might want to focus on adding depth and breadth of testing to cover more of the attack surface, and add closer instrumentation to detect issues better. All this is recommended before release, but it’s just as important to ensure that the software stays robust and secure over its whole life cycle and maintenance time. This is best achieved with fully automated fuzzing.

Use a dedicated test environment for fuzzing

Fuzzing, as well as other DAST techniques, requires that the SUT is running. This adds challenges to setting up the testing environment, but it also has advantages. Since there’s no source code required, DAST can be run against any runnable target, including third-party components and binaries for which source code may not be available.

Fuzzing shouldn’t be done in a production environment. Testing in production may sometimes be desirable, but fuzzing with anomalous messages is by nature less predictable than the more common testing techniques, so it can more easily cause issues in not just the main SUT but also in all other connected services. This is especially true for lower-layer networks and routing protocols such as DNS, DHCP, STP, BGP, and the like. Having a dedicated staging or preproduction environment that’s completely isolated for these sorts of tests is highly recommended.

A dedicated testing environment also helps maintain a high level of repeatability. Having a static, known state as the starting point for both the fuzzer and the SUT and its running environment is an important factor in having repeatable tests. Virtualization and containerization are good ways to provide an easily repeatable configuration for the testing environment. When designing a suitable test environment, it’s vital to know your SUT and what it needs. Fuzzing application layer protocols is usually a lot simpler than routing protocols, and it can usually be done in containers. The tests of lower-layer protocols may include spoofed MAC/IP addresses though, which often are blocked by virtualization environments by default and so require greater care in the environment's configuration.

The type of environment and how it’s set up may also require special consideration when it comes to instrumentation. For example, container orchestrators, like Kubernetes, often come with automated self-healing and scaling capabilities that may mask system failures and other issues from connection-style instrumentation methods. Make sure to tailor your instrumentation to catch any noteworthy failure events that happen in your environment. For example, Defensics recently added an instrumentation agent capable of detecting system failures in Kubernetes environments.

Start small and slowly add coverage and instrumentation

Some systems and software might have a large attack surface that is exposing lots of features and alternative communication channels. Fuzz testing doesn’t have to cover absolutely everything from the start. The configuration of the fuzzer can start small and grow, adding tests for one attack vector or protocol at a time.

Externally exposed interfaces should be prioritized in testing as they are the gateway to exploiting weaknesses. System hardening and reducing attack surfaces are good ways to reduce security risk. However, there’s a benefit to testing and fuzzing internal components as well. For example, fuzzing an internal component that’s responsible for subset of functionality instead of fuzzing a proxying public interfacing component can better expose unwanted behavior. And black box fuzzing can check that valid requests are continuing to be served as expected during fuzzing.

Load balancers and automated error recovery can make it difficult to detect issues that don't immediately freeze the observable behavior. For example, if you were developing a website where users post their own images and do simple editing such as adding funny text, the externally exposed interface of the service would include an HTTP API for posting an image file, and another HTTP endpoint for editing the image. A fuzzer could post an anomalous JPEG image and make another call to trigger the editing function. By observing the HTTP response codes, the fuzzer could detect, for example, "500," indicating that something unexpected happened. But it couldn’t really tell if it was a problem in the image parsing, some other business logic, or the database. If fuzzing had been done for the image processing library in isolation, it would have been straightforward to associate the problem with the image processor and develop a fix.

Another benefit of fuzzing internal components instead of full systems is simplified test environment setup. Even though internal components with issues may not expose an exploitable attack vector, you may still want to detect and fix the quality issue.

Fuzzing can cause unwanted behavior such as freezing, kernel panics, unhandled exceptions, assertion failures, busy loops, and resource consumption issues that will be detected by the instrumentation function of the fuzzer. By default, Defensics does black box instrumentation by observing the behavior of the software from the outside. It ensures that the software is still responsive, and valid requests are being processed by the software as expected. Failures and DoS typically get revealed this way, and some unexpected responses may reveal logic and parsing issues in the implementation. This kind of instrumentation comes with some transient failures, especially when testing networking software. Requests timing out due to an unrelated network glitch can be expected and should be allowed in results. If they start to become a hindrance in inspecting results though, it’s a good idea to see if something can be done.

It’s important to ensure there are no issues that can compromise the target software. Transient issues are are harder to reproduce, so analyzing for them is more difficult, but some may be real issues so you don’t want to ignore them. Be sure to minimize the amount of other traffic and changes in the network while testing and see if it helps to keep transient issues away.

If the fuzzer doesn’t discover any failures, check if closer instrumentation would reveal issues. By observing the software’s ability to process valid functions from the outside, you may miss problems often visible to system administrators and developers only. For example, a resource consumption peak may temporarily slow down the service externally but it may heal itself quickly and go unspotted by the fuzzer. We encountered these kinds of obstacles while working on the recent Jetty vulnerability. On the other hand, such problems may become exploitable with a simple loop. If you only have access to binaries of the software, you can often add tooling on the host machine to monitor memory and CPU consumption, watch for log files typically available to system admins, and subscribe to system events. In addition, if you have access to the source code you may have more options to address sanitization. An instrumented source code issue log can give very good hints for determining the root causes of issues.

Defensics can hook to many different instrumentation data sources, some of which can be enabled with a few configuration options, and others that will need the user to run a prebuild agent binary on the test target host machine. Each instrumentation agent does specific monitoring, and multiple agents can run simultaneously. Extending instrumentation with a custom agent is possible as well. A good way to start is to create a list of the various system health areas you would manually check and then automate instrumentation for them. The main limiting factors are access to the host machine running the software, access to the source code, and your imagination.