Hipcheck

The Hipcheck 3.11.0 release features a variety of quality-of-life improvements for users, especially around reporting and diagnosing configuration errors. This includes an overhauled hc ready command, better configuration error types and printed error messages, updates to english-language analysis explanations, a more informative JSON output format, and more. This release also features multiple internal refactors and CI updates described in a separate subsection for those interested.

User-facing Improvements

🔍 Plugin Configuration Error Reporting

`hc ready` Upgrade

With Hipcheck 3.11.0, hc ready now expects a policy file and behaves like an hc check dry-run; it starts and configures plugins but does not proceed to analysis.

The hc ready subcommand was first written before Hipcheck transitioned to a plugin architecture. At that time, we knew every possible analysis and their requirements (including additional configuration files, expected binaries and environment variables, etc.). Thus, hc ready could be hard-coded to check those requirements. In the new plugin-based context, the set of analyses that Hipcheck will run depends on the input policy file, which may involve third-party analysis plugins with unknown requirements. Hipcheck's "readiness" is now directly dependent on the readiness of the plugins a user will employ in a given analysis; we are in turn defining a plugin's "readiness" as whether it successfully completes the configuration step. We encourage plugin developers to use this step to do all necessary file parsing and environment checking, and not to report configuration success unless your criteria are met.

Better Configuration Error Reporting

With the above changes, hc ready now informs us if the configuration of a plugin (based on the supplied policy file) failed. To help users better understand and address configuration errors that may arise, we made the following additional changes. Firstly, for transparency Hipcheck now prints the name and version of the plugins that failed configuration. We also expanded the range of configuration error types that plugins can report, so that plugin authors can be more clear to users about what caused the failure. This change is reflected in the ConfigurationStatus enum, part of our gRPC protocol definition at hipcheck-common/proto/hipcheck/v1/hipcheck.proto. The new errors are as follows:

// The plugin encountered an internal error, probably due to incorrect assumptions.
CONFIGURATION_STATUS_INTERNAL_ERROR = 5;

// A necessary plugin input file was not found.
CONFIGURATION_STATUS_FILE_NOT_FOUND = 6;

// The plugin's input data could not be parsed correctly.
CONFIGURATION_STATUS_PARSE_ERROR = 7;

// An environment variable needed by the plugin was not set.
CONFIGURATION_STATUS_ENV_VAR_NOT_SET = 8;

// The plugin could not find or run a needed program.
CONFIGURATION_STATUS_MISSING_PROGRAM = 9;

The suite of plugins maintained by the Hipcheck team has been updated to make use of these new error types where appropriate. Finally, using the miette crate we have updated our plugins that do file parsing to return high-fidelity parsing error messages that show where in the file the error arose and suggest fixes.

📝 JSON-formatted Analysis Reporting

Users can select between having Hipcheck emit its analysis report in a human-readable format or as a JSON file using the --format flag. Previously, the JSON format was not at information-parity with the shell output. With Hipcheck 3.11.0 we have addressed this deficiency and the two formats are mostly equal in expressiveness. The default JSON output (--format json) reports only an abridged version of each plugin's raw analysis output to prioritize report readability. If users are interested in a JSON output file with each plugin's raw output (i.e. before converted to a pass/fail determination with the policy expression), they can use the new --format debug option.

Internally, the ReportBuilder structure that Hipcheck previously used to build the shell output has been eliminated; now, shell output is produced using a mini-jinja template applied to the JSON output, which simplifies the internals for report generation while also making future reporting improvements easier.

✏️ English-language Policy Explanations Changes

In Hipcheck 3.10.0 we introduced code to attempt to "explain" why analysis output failed when a particular policy expression was applied to it (while policy expressions are powerful, they can be difficult to read and understand why an analysis reported a failed determination). In this release, we have expanded these generated explanations to cover successful analyses as well.

Furthermore, we improved our detection of expressions that calculate percentages and the printing of percent-like floats. We also improved our explanation of expressions using the (count $) function, and updated span types to be expressed in number of days.

Together, these improvements should make explanations of failed analyses clearer, and we will continue to work on more improvements going forward.

📦 Smaller User-facing Improvements

In general, we have found MacOS environments are more likely to need a higher backoff-interval for plugin startup. This can be achieved with a custom Exec.kdl config file, but we decided to simply make the in-memory default Exec.kdl be more OS-aware by starting at a higher backoff-interval in MacOS environments. We hope this will make it less likely that Mac users disproportionately need to reach for a custom Exec.kdl.
We added a line to the hc check output that displays where the policy for the current execution came from. Between the policy-file- vs. (deprecated) config-folder-based executions and the use of environment variables and system defaults to derive the policy source, we wanted to ensure Hipcheck is transparent with the user about where the policy came from.

📦 Internal Changes

➡️ Salsa v3 Migration

Hipcheck uses the function output memoization crate salsa to eliminate redundant queries to plugins. Prior to the plugin architecture transition, we used salsa to memoize many more extraneous things that are now obsolete. With Hipcheck 3.11.0 we have cleared out the last remnants of our non-query salsa usage, stripping it down to where it is most relevant and impactful.

Furthermore, since salsa stopped releasing on crates.io, it has undergone at least two major refactors; for this release, we have forked the salsa GitHub repo to get access to the latest refactor in a way we can control so if future upstream changes would break our code we can continue targeting a commit that works for us. We refactored our query memoization for the new API and are now up-to-date with Salsa v3!

📤 CI Pipeline Quality-of-Life

We made a variety of small-to-medium sized changes to the CI workflow for PRs to help the development team catch bugs and speed up release builds. These changes include:

Changing our set of CI runners to match the set of architectures that we officially support.
Using GitHub merge queues to catch bugs that would occur only once a PR is merged into main before they happen, and to simplify the workflow for updating PRs with requested changes.
Adjusting releases using dist to only build the target crate instead of the entire workspace, significantly cutting down release build times.
Expanding the set of paths to which changes cause the build and test workflow to include more supporting crates.
We now run full hc check analyses in CI instead of just unit tests.

📦 Miscellaneous Changes

With the query batching feature we introduced in Hipcheck 3.10.0, we have been able to strip out redundant code in the mitre/churn and mitre/entropy plugins and have them query mitre/linguist instead to find out which files in a repository qualify as source code.
We consolidated the kdl parsing code used by multiple plugins and Hipcheck core into an internal hipcheck-kdl crate.

🗺️ How to Get Involved

We're always looking for new contributors! If you'd like to learn more about Hipcheck and get involved in contributing, please checkout our Roadmap and feel free to get in touch with us through our Discussions board!

⭐️ Thank You to Our Contributors and Supporters

As always, we want to say a big "Thank you!" to everyone who supports the project at MITRE, to CISA for sponsoring our current work on it, to our prior government sponsors who have helped advance Hipcheck, and to everyone who has contributed, given feedback, or encouraged us in building it.

The following team members contributed to this release:

Hipcheck 3.11.0 Release