Announcing tartufo v3.0!

As I’ve written about previously, tartufo is a tool vital to our organization. It helps us to ensure that our source code is clear of all forms of credentials, secrets, keys, and things we just generally don’t want to see potentially getting exposed. It is just one small, yet absolutely vital, link in our security toolchain. So it is with this in mind that I am now pleased to announce the release of tartufo v3.0!

Previously, in tartufo 2.0

With our previous major version bump, v2.0, the goal was to improve the developer experience. The project from which we originated our work, truffleHog, was and still is, an amazingly powerful tool. But it left some things to be desired from a developer perspective. So we set to work to improve this: we split the code into distinct modules, separating the CLI from the API; we moved the code into a hierarchy of classes to enable future growth; we moved the different “modes” of operation into sub-commands to further de-clutter the codebase. But all of this work did little to help out those who matter most: the users of this tool.

Fast Forward to Today

So, with that in mind, our major focus in v3.0 has been to vastly improve the user experience! We have taken countless bits of feedback from our users around what they liked, what they didn’t like, what was great, what was just flat broken, and we’ve done our best to improve on as many of those things as possible. Our goal with this new major release is to take a tool that can oftentimes be frustrating and confusing for a user, into a tool that developers truly don’t mind, and in fact want to have as part of their toolchain! To escape the shackles of the classic security conundrum, where the tools are often so cumbersome and difficult to use that developers would rather forego them, despite the cost.

Many developer hours have gone into this new release, and it would not be possible without all of the fine contributors both from GoDaddy engineers, as well as a number of new external contributors! I would like, however, to give a very special shout-out to our very own Jeremiah Gowdy. His initial work on porting the backend of tartufo to use pygit2 paved the way for much of the speed gains in this release, and without his efforts, this certainly would not have happened in the time frame that it did. Thank you, Jeremiah!

And that brings us to the big question: What’s so great about this new version?

The New Hotness

Here is a summary of the big-ticket items we’ve added, fixed, or improved for this new major release.

Performance

• First and foremost, as mentioned before, the backend switched from GitPython to pygit2. This means that, in this background, we are no longer just running git commands and interpreting the output. We are now actually interacting directly with GitDB! This provides us massive speedups, as well as makes it easier for us to disregard user-config. This means that git operations are now performed in the manner that tartufo intends them, regardless of the user’s own git config.

  • Initial work by Jeremiah Gowdy — #122
  • Conclusion of the transition by Sushant Mimani — #233
  • Further polishing to ensure parity of findings with v2.x by Esha Mayuri — #249

• Scott Bailey put in extensive work to allow issues to be displayed immediately as they are found, as opposed to buffering them for display in one big batch at the end. While this may not increase the overall runtime of a scan, it will eliminate the “…is it actually doing anything?” aspect of long runs — #227

Configuration Changes

• A number of our configuration options have made a transition from contrived / specially constructed strings, to a wonder of the TOML format: an array of tables. This has enabled more verbose and readable configuration, but has also removed them as command-line options, forcing them to be used only in the context of the configuration file. These include:
  • The --rules configuration option has changed. This was our final remaining piece of configuration that still referred to an external file. Now, this has changed to --rule-patterns, which are patterns stored directly in your configuration file, alongside all the rest of your config. This work was done by Joey Wilhelm (that’s me!) — #287
  • The “old” style of configuring exclude-entropy-patterns has been removed. Sushant Mimani did the work to make this new style the one and only way of configuring these. — #282
  • One of our most requested features has been “exclusion reasons”. Our users have found that they end up with an extensive list of excluded-signatures, and have a hard time knowing what each represents. Thanks to Esha Mayuri and this new format, this is now possible — #286
  • While we were at it, we decided that consistency was key, so we updated the configuration style for --include-path-patterns and --exclude-path-patterns to match this new style as well! Once again, thanks to Esha Mayuri for making this happen — #289
• The scan-folder sub-command now has a --recurse / --no-recurse flag, so that you can control whether all sub-folders are scanned, or only the current folder. This was added by Esha Mayuri — #283
• The recently added --b64-entropy-score and --hex-entropy-score options raised some great discussion and sparked in-depth investigation into what these values represented, and how they might be more usefully represented / handled. This led to the new --entropy-sensitivity option, with a range of 0-100, for capturing values that range from totally non-random all the way up to only wholly random strings. The discussion, research, and option were tackled by Scott Bailey — #272
• The --json and --compact options have been consolidated into the new --output-format / --of option. In addition to consolidating these two, this gives us a place to more easily add future output formats as we see fit. This work was done by Esha Mayuri — #266
• The --fetch and --no-fetch options are no more! We realized that this was never something that tartufo itself should have been doing, so we have completely removed the functionality in this new version. Thanks go to Sushant Mimani for this work — #244

Bug Fixes

• Previously, when tartufo would attempt to scan a shallow clone, such as what would be produced by default by the actions/checkout GitHub Action, it wouldn’t actually scan anything. It would see that there were no local branches to be scanned, and then report success. This could have led to a false sense of security, such as if tartufo was being run as part of a GitHub Actions workflow. Now, when tartufo detects that it is being run against such a clone, it will instead scan the entire checkout as a single commit. This was fixed by Joey Wilhelm — #270

Other Contributors

Additional contributions were made by Erik Owen, Mertay Dayanc, and Anatoliy Serputov. Without their work, this release would not have been possible!

Looking to the Future

As you can see, a great deal of work has gone into this release. And these are only the largest changes listed here! We hope that this gives you some idea of what we’ve been working on, and we hope that this new release truly does provide for a better experience for you. Please let us know in the issues or discussions what you think, what you like, or what problems you find! Who knows, maybe we’ll be adding your name to this list of contributors for the next major release!

But Wait, There’s More!

You didn’t think that was all we had for you, did you? Of course not!

In addition to all the work that has gone into this new release for tartufo, our very own Wayne Berry has been working on a VSCode extension that will help identify high entropy strings, and get exclusions added to your tartufo.toml configuration file, right from your editor!

You can check out the source code here, or head over to the Visual Studio Marketplace to install it here. And if you don’t believe yet how wonderful this is, check it out in action below!

Join Us!

Want to help us work on these tools and help empower the future of Everyday Entrepreneurs? GoDaddy is always looking for more talented individuals! Check out our careers page to find the role that is perfect for you!

---