Hipcheck 3.12.0 Release

Hipcheck 3.12.0 is out! Featuring support for SemVer constraints for plugins, a new --arch flag for hc ready, support for managing the plugin cache with hc cache plugin, and more. With 3.12.0, Hipcheck now requires glibc 2.35 or newer on x86 Linux hosts.

For the unfamiliar, Hipcheck is a tool to help open source software users decide what dependencies to use by assessing project practices and by detecting possible malicious activity.

⚠️  Bumping our minimum glibc version, adopting mold for Linking on Linux

In 3.12.0, we've made two changes which impact our prebuilt Linux binaries and building on Linux hosts.

First, our GitHub Actions runner for building x86 Linux binaries for new releases has moved from Ubuntu 20.04 to Ubuntu 22.04. This is because GitHub is phasing out support for Ubuntu 20.04 in their free actions tier. GitHub's policy is to support the two most recent stable releases of Ubuntu for free, and with Ubuntu 24.04 released last year, GitHub has begun to sunset 20.04.

For users this means our prebuilt Linux binaries are now built with glibc version 2.35-0ubuntu3.9, which is an Ubuntu-patched variant of glibc 2.35. Since glibc versions are forward-compatible but not backward-compatible, our prebuilt binaries should link and run on any system with glibc 2.35 or newer, but not with any older versions.

To use our prebuilt binaries on Ubuntu 20.04, you'll need to install a new-enough version of glibc, which can be done by upgrading the libc6 package to its latest version.

You'll likely still be able to build Hipcheck yourself on Ubuntu 20.04 in the near-term. Though we won't be testing it in CI anymore and can't guarantee continued compatibility.

Our second change is to start using mold for linking on x86 Linux hosts. mold is a highly parallel linker, which in our testing has improved the performance of our whole-workspace Linux builds by around 30% in CI.

This change won't matter for anyone using prebuilt binaries from our releases. If you're building Hipcheck yourself, you'll either need to install mold or remove it as the configured linker by deleting the following block from .cargo/config.toml in Hipcheck's source repository:

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

In the future, we'd like to also offer prebuilt binaries compiled with musl libc instead of glibc, and to perform regular CI testing with musl. For more updates you can follow our open issue on musl support.

⛓️  SemVer Constraints for Plugins

When you specify plugins to use in your policy file, you include a version attribute which specifies what version to use for that plugin. Previously, this only accepted a fully-specified SemVer version, of the form X.Y.Z. This was overly restrictive, and meant that if a plugin had a new patch version you would not get it without manually updating the requested version in your policy file.

With 3.12.0, we now accept a set of "version constraints," specifically the set defined in the Rust semver crate, which implements the set of operators accepted by Cargo, Rust's standard build tool. Those operators are:

• Caret Requirements (like ^1.2.3): Permits any SemVer-compatible updates.
• Tilde Requirements (like ~1.2.3): Permits only minimal updates. Specifically:
  • ~X.Y.Z or ~X.Y: Only patch updates are allowed.
  • ~X: Minor or patch updates are allowed.
• Wildcard Requirements (like 1.2.*): Allows any version where the wildcard is placed. Only one wildcard allowed at a time, * is not acceptable as a version requirement.
• Comparison Requirements (like >= 1.2.3): Specifies a version range, which may be bounded or unbounded, using the <, <=, >, >= and = operators.

For any of the above, multiple bounds can be provided, separated by a comma. For example: >= 1.0.0, < 2.0.0, which would be equivalent to ~1.

Our one difference from Cargo is that, for backwards-compatibility with our prior version-solving approach, bare versions specified without an operator will be treated as an equality comparison requirement (prepended with an =) instead of being treated as a default requirement.

We've updated the default version constraints in our starter policy file to use caret requirements, so users automatically get the latest available patch version, instead of the prior equality comparison requirement.

This change also applies for plugins specifying their own dependencies in their plugin manifest file (plugin.kdl). Those dependencies may now also be set with SemVer constraints in addition to fully-specified SemVer versions.

💻  New --arch flag for hc ready

The hc ready command exists to let you check that your installation of Hipcheck and the plugins specified in your policy file are ready to run. It does this by resolving your plugins, downloading them if necessary, starting them up, and then attempting to configure them with any configuration items from your policy file and reporting relevant errors.

To get the right plugins, Hipcheck needs to know your host's "target triple." That's the string that explains what kind of host it is, usually including information like the Instruction Set Architecture (ISA) and host operating system, though they can have other information as well. By default, Hipcheck infers the target triple for the current host, but this inference can be wrong. In that case, Hipcheck might download plugins you're not able to run on your current platform.

The --arch flag, which already exists for the hc check subcommand, has been added to hc ready. This flag takes a target triple argument, which will be used for plugin matching instead of being inferred by Hipcheck. In general, we use the same target triples as those tracked by the Rust programming language project.

🏪  Manage the plugin cache with hc cache plugin

Hipcheck maintains two caches today: the "target cache," which contains copies of the source repositories of targets Hipcheck analyses, and the "plugin cache" which contains copies of plugins.

Since we don't currently do anything to actively reduce the size of these caches, they will grow over time as you analyze more targets or use more plugins. While we're likely to add some automated cache eviction in the future, for now we offer the hc cache subcommand to manage these caches and remove entries that are no longer needed.

In 3.12.0, we're introducing the hc cache plugin subcommand, which is specifically for managing cached plugins. It has two subcommands:

• hc cache plugin list: List cached plugins, their version, and when they were last modified.
• hc cache plugin delete: Remove plugins from the plugin cache.

The arguments for hc cache plugin delete are similar to the ones on hc cache target delete, and include:

• -s/--strategy: How to select plugins to delete, and optionally how many.
• -N/--name: Plugin name to filter by.
• -P/--publisher: Publisher name to filter by.

For example, to delete the three least-recently-used plugins from MITRE, you'd run:

$ hc cache plugin delete -s 'oldest 3' -P 'mitre'

Previously, the hc cache command implicitly manages the target cache. With the introduction of the hc cache plugin command to manage the plugin cache, all operations previously on hc cache have been moved to hc cache target.

🚚  Initial Support for Containerized Plugins

All of the plugins MITRE ships today for Hipcheck are written in Rust and ship as pre-built binaries for our supported platforms, and they run as standalone programs except for their dependency on a sufficiently-recent version of libc from the system, or on CLI tools whose presence is checked during plugin configuration. This has worked well-enough for us, but until now we haven't had a decent answer for how to bundle and distribute plugins where you don't want to rely on the presence of dependencies on the host system.

With 3.12.0, we've introduced new documentation for how to package and ship containerized plugins, including a new example found at tests/test-plugins/activity-container in the Hipcheck source repository, and we've updated the Rust Hipcheck SDK to make it easier to define a plugin which can run inside of a container.

The key challenge for packaging a container-based plugin is making the CLI invocation for starting the plugin comply with Hipcheck's CLI protocol by accepting a --port flag to indicate what port the program should listen on. In our new example plugin, we do this with a shell script that wraps starting the plugin container and passess along the flag appropriately.

Within the SDK, we also now expose an API for launching a plugin by having it listen on 0.0.0.0 instead of listening on the local loopback address 127.0.0.1.

We plan to continue to improve the story around shipping plugins with more complex dependencies, and to further improve documentation and tooling for this setup.

🐍  Experimental Python SDK for Plugins

So far we've only shipped a plugin SDK for Rust, because Hipcheck itself is written in Rust and so we needed an SDK to use ourselves when we split our existing analyses out into plugins last year.

However, the design of the plugin system is language-agnostic. So long as a program speaks the plugin protocol it can be used as a plugin.

We're currently working on a plugin SDK for Python. It's still unfinished and experimental, but you can find it at sdk/python in the Hipcheck source repository, and we've reserved the package name hipcheck-sdk on PyPI, which we intend to start publishing to soon.

If you're interested in developing a plugin in Python, feel free to try out the SDK. We'd love to hear feedback from people who do try it so we can address any deficiencies or awkwardness in the API or in the experience of shipping plugins written in a language that's not ahead-of-time compiled like Rust!

🪵  Plugin Logging Improvements

Part of the design of the plugin protocol is that plugins are intended to log relevant information during their execution, which can then be collected and managed by Hipcheck. Previously, this was done on an ad-hoc basis, and the plugin protocol defined no formal method for Hipcheck to control what level of information plugins should log, no clear format for logs from plugins, and no action taken by Hipcheck to actually gather logs that plugins produce.

With 3.12.0, we've introduced three key changes:

• Updated the plugin protocol to require support for a --log-level command line argument for all plugins, which accepts a string detailing the threshold log level to produce, and updated our own first-party plugins to respect this flag.
• Defined an expected format for log entries from plugins.
• Updated Hipcheck to begin gathering and passing along this log information based on its own configured log level.

Let's cover each of these in order:

The new --log-level flag takes one of the following levels, in order:

• Off: Do not produce logs.
• Error: Log unrecoverable error state information.
• Warn: Log recoverable warning information.
• Info: Log general information.
• Debug: Log debugging information.
• Trace: Log detailed tracing information.

Each log level includes entries from the levels above it.

Plugins are expected to accept and correctly handle these levels, only producing log entries consistent with the provided log level threshold.

The log format expected is a single-line JSON object with three fields:

• "level": the log level string indicating what log level the entry is.
• "target": a string indicating what module in the plugin produced the log entry, with segments in any module path separated by ::.
• "fields": an object with one field:
  • "message": a string with the actual content of the log entry.

We reserve the possibility of including more fields under the fields object in the future.

Finally, Hipcheck now gathers this log information and forwards it based on the user's requested log level and any user-provided filtering. All of this is taken from the HC_LOG environment variable, which is used to configure Hipcheck's logging generally, including logging collected from plugins.

Altogether, these changes should make it easier to debug issues in Hipcheck by incorporating any logging information produced by plugins.

🗺️  How to Get Involved

We're always looking for new contributors! If you'd like to learn more about Hipcheck and get involved in contributing, please checkout our Roadmap and feel free to get in touch with us through our Discussions board!

⭐️  Thank You to Our Contributors and Supporters

As always, we want to say a big "Thank you!" to everyone who supports the project at MITRE, to CISA for sponsoring our current work on it, to our prior government sponsors who have helped advance Hipcheck, and to everyone who has contributed, given feedback, or encouraged us in building it.

The following team members contributed to this release:

• Julian Lanson
• Michael Chernicoff
• Patrick Casey
• Aisha Mohammed
• Nino Agrawal
• Kirill Usubyan
• Devin Lake
• Cal Stepanian