Access Official Technical PDF

Share Summary:LinkedInEmail

Greybox fuzzing can help identify software defects by repeatedly generating and testing inputs while using program execution feedback to guide the search. For industrial automation teams, the technique is relevant to the software that supports connected operations, from applications and embedded components to systems that communicate with control environments. A key design choice is the coverage metric: the rule that decides whether a test input reveals enough new behavior to be kept and explored further.

Research presented at the 22nd International Symposium on Research in Attacks, Intrusions and Defenses examined how different coverage metrics affect fuzzing results. Its central practical lesson is that a more detailed metric is not automatically the best choice. Metrics can expose different behaviors, and a combination of approaches may be more useful than relying on one measure alone.

How coverage-guided greybox fuzzing works

A greybox fuzzer begins with one or more seed inputs. It selects a seed, changes it through mutation, and runs the resulting test cases against the program. During execution, instrumentation records information about the program’s behavior. If a test case produces coverage that the fuzzer has not seen before, it may be retained as a new seed for later mutations.

This feedback loop distinguishes coverage-guided greybox fuzzing from testing that generates inputs without using information about program execution. The coverage metric shapes the loop: it influences which test cases count as interesting and therefore which parts of the input space receive further attention. If an intermediate test case is discarded, a sequence of mutations that might eventually reach a defect can be cut short.

Coverage metrics reveal different kinds of behavior

Basic block coverage records which blocks of code execute. Branch coverage adds information about transitions between blocks, distinguishing some executions that block coverage treats as equivalent. The research also evaluated metrics that extend branch coverage in different directions:

  • Context-sensitive branch coverage distinguishes execution of a branch in different calling contexts.
  • N-gram branch coverage uses sequences of recent block transitions to capture more execution history. Larger values of n can distinguish more sequences, but may also create more seeds for the fuzzer to manage.
  • Memory-access-aware coverage incorporates memory locations and whether they are read or written, alongside branch information.
  • Memory-write-aware coverage focuses on writes rather than recording both reads and writes.

These measures differ in what they consider a meaningful change. For example, two inputs might follow the same branch but access different memory locations. A memory-aware metric may keep one as a new seed, while ordinary branch coverage may not. Conversely, additional detail can increase the number of inputs judged interesting, placing more demands on limited fuzzing time and computing resources.

Why no single greybox fuzzing metric wins

The study implemented several metrics in AFL and evaluated them using the DARPA Cyber Grand Challenge dataset, the LAVA-M dataset, and 12 real-world applications. The full evaluation covered 221 binaries. The researchers measured outcomes including unique crashes, time to crash, and seed counts.

Across the evaluations, some more sensitive metrics found more bugs or found them sooner than baseline branch coverage. However, each metric had its own strengths, and different metrics often found different sets of bugs. The strongest performer could also vary over the course of a fuzzing run. The results therefore did not identify one coverage metric that consistently outperformed all others.

This is important because fuzzing is performed with finite time and computing capacity. A metric that distinguishes more executions may preserve useful test cases, but it can also produce a larger seed queue. More sensitivity can help the search, yet it does not guarantee better results if the fuzzer cannot effectively schedule and mutate the expanded set of seeds.

Combining metrics and applying the lesson to automation

The researchers also explored cross-seeding: using test cases found by fuzzers configured with different coverage metrics as seeds for one another. In their evaluation, this combined fuzzing approach crashed at least as many binaries in the CGC dataset as Driller, a prior approach combining fuzzing with concolic execution, while using fewer computing resources. This is a result for the evaluated dataset and setup, not a general guarantee for every program or industrial environment.

For automation engineers, system integrators, and security teams, the takeaway is to treat coverage configuration as a testing decision rather than a universal setting. When selecting or building a fuzzing workflow, teams can consider:

  • Whether the target’s available instrumentation supports useful branch, calling-context, or memory information.
  • How the chosen metric affects seed growth and the time available to explore each input.
  • Whether running complementary configurations and sharing their seeds could expose behaviors that one configuration misses.
  • How fuzzing fits alongside other testing and validation activities for software used in operational environments.

The proceedings also include work on timing patterns and correlations in spontaneous SCADA traffic for anomaly detection, as well as research on physics-based attestation of control systems. These are distinct security topics: the fuzzing study does not evaluate SCADA networks or establish that its results transfer directly to control-system testing. Together, however, the topics underscore the range of software and communication behaviors that security research examines in relation to connected systems.

Conclusion

Greybox fuzzing depends not only on generating inputs, but also on deciding which executions deserve further exploration. The evaluated coverage metrics captured different aspects of program behavior, and no single metric dominated across the board. For industrial software testing, the practical implication is to match metrics to the target and resource constraints, and to consider complementary approaches such as cross-seeding. That makes the testing strategy more deliberate without assuming that one coverage measure can reveal every defect.

Access Official Technical PDF

About The Author