Proactive Security with Trivy

In this blog post I want to share my knowledge on how we implemented Trivy for our repositories and as part of our CI/CD pipelines.

Whether it was our for our Bicep templates in a shared registry or finding vulnerabilities in containers before deploying to production, we where missing proactive security scanning for these resources. As I’ve worked with Trivy before, it was my immediate preference, and I was confident it would help us enhance our holistic security posture. By striking a balance between a multifacted approach that combines both reactive and proactive measures, Trivy not only streamlines our security assessment process but also ensures efficiency in safeguarding our systems against potential vulnerabilities.

What is Trivy

Trivy is a comprehensive and easy-to-use open-source vulnerability scanner. Trivy can detect vulnerabilites of OS packages and language specific packages. It can also scan IaC files like ARM templates and Kubernetes. The purpose of using a vulnerability scanner tool, is to identify known security vulnerabilities in the packages listed in your images or files. This gives the opportunity to find vulnerabilities and fix them before pushing the code to production. Trivy has scanners that look for security issues, and targets where it can find those issues.

Targets (what Trivy can scan):

  • Container Image
  • Filesystem
  • Git Repository (remote)
  • Virtual Machine Image
  • Kubernetes
  • AWS

Scanners (what Trivy can find there):

  • OS packages and software dependencies in use (SBOM)
  • Known vulnerabilities (CVEs)
  • IaC issues and misconfigurations
  • Sensitive information and secrets
  • Software licenses

Trivy supports most popular programming languages, operating systems, and platforms. For a complete list, see the Scanning Coverage page. More information about Trivy can be found here.

Scanners

The –scanners parameter can be configured depending on what you want Trivy to detect. Default is vuln and secret, but it can also be used for IaC misconfiguration (config) and licenses (license) bash

--scanners vuln,secret,config,license

Scanning a image

After developing and consolidating your application into an image (Docker or so), you have the option of finding out any security issue you may have overlooked.

trivy image nginx

In case you need the vulnerability test to be filtered per severity such as only HIGH and CRITICAL vulnerabilities, then this can be added with the severity parameter:

trivy image --severity HIGH,CRITICAL nginx:latest

By default, Trivy exits with code 0 even when vulnerabilities are detected. Use the –exit-code option if you want to exit with a non-zero exit code. With the code below, the test will fail only when a critical vulnerability is found.

trivy image --exit-code 0 --severity MEDIUM,HIGH ruby:2.4.0
trivy image --exit-code 1 --severity CRITICAL ruby:2.4.0

Scanning Infrastructure as Code

This example shows how to scan a ARM template for misconfigurations:

trivy conf ./main.json

To scan Bicep files, the bicep file needs to be converted to an ARM template first (using bicep build or az bicep build).

Scanning a FileSystem

Trivy can scan a filesystem (such as a host machine, a virtual machine image, or an unpacked container image filesystem).

trivy fs /home/vagrant

Scanning a GIT repository

Fortunately, you can scan your remote git repository with this simple yet powerful tool. And it should be noted that only public repositories are supported here.

trivy repo https://github.com/testrepository

Ignore certain vulnerabilities

It is possible to ignore certain vulnerabilities, which might be accepted risks or not having an impact on the particular application. This can be done with a .trivyignore file that looks like this:

# Accept the risk
CVE-2018-14618

# No impact in our settings
CVE-2019-1543

Scanning local images and files

Trivy can be used to scan images and files that are stored locally. Here you can find more information about the different ways to install Trivy https://aquasecurity.github.io/trivy/v0.44/getting-started/installation/

## Install Trivy
latest_release=$(curl -sL "https://api.github.com/repos/aquasecurity/trivy/releases/latest")
trivyVersion=$(echo "$latest_release" | jq -r '.name')
trivyVersion=${trivyVersion#?}
wget https://github.com/aquasecurity/trivy/releases/download/v${trivyVersion}/trivy_${trivyVersion}_Linux-64bit.deb
sudo dpkg -i trivy_${trivyVersion}_Linux-64bit.deb
trivy -v

## Build docker iamge
docker build -t ioImage:latest ioImage

## Trivy scan docker image
trivy image ioImage:latest

Azure DevOps

Trivy can be used in an Azure DevOps pipeline to scan container images, or it can be used to scan Azure DevOps repositories. This GitHub repository https://github.com/AshwinSarimin/teknologi-trivy shows a couple of templates for Trivy that can be used in an Azure DevOps pipeline.

Repository scanning

The pipeline template trivy-repositories-securityscan.yaml can be added to an existing pipelines to perform a security scan for one or multiple repositories. To scan remote repositories directly can only be done with public repositories. The pipeline performs a checkout of the repositories and scans the local repositories for vulnerabilities, secrets and IaC misconfigurations. The pipeline will have an error when High or Critical findings are found. The pipeline can be added to an existing pipeline with the following code:

- template: templates/repositories-securityscan.yaml
    parameters:
      repositoryNames:
        - self
        - repositoryName1
        - repositoryName2
        - repositoryName3

Container image scanning

Trivyscan extension

The pipeline template trivy-containerscan.yaml can be added to an existing pipelines to perform a security scan for a docker container. This pipeline uses the Trivyscan extension in Azure DevOps. The results for the scan can be found in the pipeline deployment run overview » Trivy tab:

Trivy extension

The following inputs can be configured for the extension:

  docker: false #Run Trivy using the aquasec/trivy docker image. Alternatively the Trivy binary will be run natively. Defaults to true.
  image: '${{ parameters.dockerRegistryBasePath }}/${{ parameters.buildName }}:${{ parameters.tag }}' #The image to scan 
  version: "$(trivyVersion)" #The version of Trivy to use.
  debug: true #Enable debug logging in the build output.
  ignoreUnfixed: true #When set to true all unfixed vulnerabilities will be skipped. Defaults to false.
  exitCode: 1 #The exit-code to use when Trivy detects issues. Set to 0 to prevent the build failing when Trivy finds issues. Defaults to 1.

More information can be found here

Install Trivy on agent

It is also possible to install Trivy on the build agent and perform a scan by adding the following code to an existing Build Image deployment pipeline.

- task: Bash@3
  displayName: 'Download and install Trivy'
  inputs:
    targetType: inline
    script: |
      sudo apt-get -y install rpm
      latest_release=$(curl -sL "https://api.github.com/repos/aquasecurity/trivy/releases/latest")
      trivyVersion=$(echo "$latest_release" | jq -r '.name')
      trivyVersion=${trivyVersion#?}
      wget https://github.com/aquasecurity/trivy/releases/download/v${trivyVersion}/trivy_${trivyVersion}_Linux-64bit.deb
      sudo dpkg -i trivy_${trivyVersion}_Linux-64bit.deb
      trivy -v
- task: Bash@3
  displayName: 'Scan container for vulnerabilities'
  inputs:
    targetType: inline
    script: |
      docker images
      trivy image --scanners vuln,secret --exit-code 0 --severity LOW,MEDIUM --format template --template '@/usr/local/share/trivy/templates/junit.tpl' -o junit-report-low-medium.xml "${{ parameters.dockerRegistryBasePath }}/${{ parameters.buildName }}:${{ parameters.tag }}"
      trivy image --scanners vuln,secret --exit-code 1 --severity HIGH,CRITICAL --format template --template '@/usr/local/share/trivy/templates/junit.tpl' -o junit-report-high-critical.xml "${{ parameters.dockerRegistryBasePath }}/${{ parameters.buildName }}:${{ parameters.tag }}"
- task : PublishTestResults@2
  inputs: 
    displayName: Publish test results - Low/Medium - ${{ parameters.buildName }}
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/junit-report-low-medium.xml'
    mergeTestResults: true
    failTaskOnFailedTests: false
    testRunTitle: 'Trivy - Low and Medium Vulnerabilities - ${{ parameters.buildName }}'
  condition: 'always()'
- task : PublishTestResults@2
  inputs: 
    displayName: Publish test results - High/Critical - ${{ parameters.buildName }}
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/junit-report-high-critical.xml'
    mergeTestResults: true
    failTaskOnFailedTests: false
    testRunTitle: 'Trivy - High and Critical Vulnerabilities - ${{ parameters.buildName }}'
  condition: 'always()'

With this code the pipeline will fail when High or Critical findings are found. It is best to run this code and remediate the findings before pushing the images to an Container Registry.

GitHub Action

Trivy can be used in an Github Action pipeline to scan container images, IaC or to use scan GitHub repositories.

There are 2 ways to use Trivy in a GitHub Action:

  • Installing the Trivy application on the build agent
  • Using the Trivy Action (aquasecurity/trivy-action@master) (preferred)

The following examples show how to scan Bicep templates in a GitHub Action workflow, with the first method by installing the Trivy application, and the second method using the Trivy GitHub Action:

Installing Trivy in a GitHub Action worklow

Trivy can be installed and used for scanning a Bicep template with the following code:

steps:
  - name: Install Trivy
    shell: bash
    run: |
      latest_release=$(curl -sL "https://api.github.com/repos/aquasecurity/trivy/releases/latest")
      trivyVersion=$(echo "$latest_release" | jq -r '.name')
      trivyVersion=${trivyVersion#?}
      wget https://github.com/aquasecurity/trivy/releases/download/v${trivyVersion}/trivy_${trivyVersion}_Linux-64bit.deb
      sudo dpkg -i trivy_${trivyVersion}_Linux-64bit.deb
      trivy -v
  - name: Convert Bicep to ARM
    shell: bash
    run: |
      az bicep build ${{ inputs.templateFolderPath }}/bicep/main.bicep
  - name: Scan module folder
    shell: bash
    run: |
      trivy conf --exit-code 0 --severity LOW,MEDIUM ${{ inputs.templateFolderPath }}
      trivy conf --exit-code 1 --severity HIGH,CRITICAL ${{ inputs.templateFolderPath }}

The code installs Trivy, converts to Bicep template to ARM and scans the ARM template twice. First for Low and Medium severities and then for High and Critical vulnerabilities. When High or Critical fails are found, the pipeline will break because of exit-code 1.

Trivy Action

The Trivy Action can be used for scanning a Bicep template with the following code:

steps:
  - name: Convert Bicep to ARM
    shell: bash
    run: |
      az bicep build --file bicepfolder/main.bicep 
  - name: Run Trivy config scanner - Low,Medium
    uses: aquasecurity/trivy-action@master
    with:
      scan-ref: bicepfolder
      scan-type: 'config'
      hide-progress: false
      exit-code: '0'
      ignore-unfixed: true
      severity: 'LOW,MEDIUM'
      format: template
      template: "@/github/workspace/software/trivy/templates/markdown.tpl"
      output: trivy-low-medium.md
  - name: Run Trivy config scanner - High,Critical
    uses: aquasecurity/trivy-action@master
    with:
      scan-ref: bicepfolder
      scan-type: 'config'
      hide-progress: false
      exit-code: '0'
      ignore-unfixed: true
      severity: 'CRITICAL,HIGH'
      format: template
      template: "@/github/workspace/software/trivy/templates/markdown.tpl"
      output: trivy-high-critical.md
  - name: Output Trivy Results to step summary
    shell: bash
    if: always()
    run: |
      cat trivy-low-medium.md >> $GITHUB_STEP_SUMMARY
      cat trivy-high-critical.md >> $GITHUB_STEP_SUMMARY

The code starts by converting the Bicep template to ARM. One important thing to note is when converting the main bicep file, all submodules that are invoked will also be converted into the same ARM template.

It then uses the Trivy Action to scan the ARM template with the following inputs:

  • scan-ref: bicepfolder = scan reference
  • scan-type: config
  • hide-progress: false = Suppress progress bar
  • exit-code: ‘0’ = Exit code when specified vulnerabilities are found. When High or Critical fails are found, the pipeline will break.
  • ignore-unfixed: true = Ignore unpatched/unfixed vulnerabilities
  • severity: ‘CRITICAL,HIGH’ = Severities of vulnerabilities to scanned for and displayed
  • format: template = Output format (table, json, sarif, github)
  • template: “@/github/workspace/software/trivy/templates/markdown.tpl” = Custom output template
  • output: trivy-high-critical.md = Save results to a file

More information can be found here https://github.com/aquasecurity/trivy-action

Export results to GitHub Action Job Summary

The format, template and output inputs in the GitHub Action step, are used to export the scan results to the GitHub Action Job Summary.

Out of the box the Trivy Action can export the results in SARIF format to GitHub Advanced Security. But because we don’t have Advanced Security, the template format is used to generate a markdown report to use in the job summary. By default, Trivy does not provide a template for markdown, so a custom template is used to convert the scan results to markdown: markdown.tpl

Because the Trivy scanner runs inside a docker instance, it mounts the checked out repository as a volume inside that container. The volume is mounted as:

-v "/home/runner/work/REPOSITORY":"/github/workspace"

To be able the retrieve the markdown template from the repository, ‘@/github/workspace/’ must be used with the template file location trivy/templates/markdown.tpl

cat trivy-low-medium.md >> $GITHUB_STEP_SUMMARY
cat trivy-high-critical.md >> $GITHUB_STEP_SUMMARY

The code above exports the markdown data to the Github environment file $GITHUB_STEP_SUMMARY. When a job finishes, the summaries for all steps in a job are grouped together into a single job summary and are shown on the workflow run summary page, like this:

job summary

I hope this post provides valuable insights in how to leverage Trivy for enhancing proactive security within CI/CD processes. By integrating Trivy seamlessly into your workflows, you not only strengthen your security against potential vulnerabilities but also instill a proactive mindset that safeguards your applications from threats.