Metadata

What is Metadata?

Metadata is a structured key/value storage of relevant information about a build. It can be updated or retrieved throughout the build by using the built-in meta CLI in the steps.

Default Metadata

By default, Screwdriver sets the following keys in metadata:

Manipulating Metadata

Screwdriver provides the shell command meta get to extract information from the meta store and meta set to save information to the meta store.

Same pipeline

Screwdriver builds can retrieve metadata set by itself or by previous builds within the same event.

Example: build1 -> build2 -> build3

build2’s metadata will consist of metadata set by itself and build1

build3’s metadata will consist of metadata from build2 (which also includes metadata from build1)

$ meta set example.coverage 99.95
$ meta get example.coverage
99.95
$ meta get example
{"coverage":99.95}

Example:

$ meta set foo[2].bar[1] baz
$ meta get foo
[null,null,{"bar":[null,"baz"]}]

Example repo: https://github.com/screwdriver-cd-test/workflow-metadata-example

Note: If foo is not set and you try to meta get foo, it will return a string with value null by default.

Restart case

The metadata from the parent event will be accessible for use in the restarted build as well as in any downstream builds.

Example: build1 -> build2 -> build3 are builds in event1.

Upon completion of build3, the metadata from build1, build2, and build3 becomes available in event1. In other words, the metadata from each completed build is merged into the event in which the build is currently running.

When build1 is restarted, it will have access to metadata from event1. This metadata from build1 will then be accessible to downstream builds, allowing build2 and build3 to also receive the metadata from event1.

Caveat

If a build is restarted from a parent event containing only one build that has never been run (e.g., a virtual job or frozen build), the restarted build will not inherit any metadata because the parent event’s metadata is empty.

External pipeline

Screwdriver build can also access metadata from an external triggering job by adding the --external flag followed by the triggering job.

Example: sd@123:publish -> build1. Then inside build1:

$ meta get example --external sd@123:publish
{"coverage":99.95}

Notes:

• meta set is not allowed for external builds.
• If the --external pipeline job did not trigger the build, then meta from the last successful build for the external job will be fetched.

Using the API

You can also prepopulate event meta by configuring the payload of the POST request to /v4/events.

See the API docs for more information on API endpoints. See the event meta trigger example repo and corresponding event meta example repo for reference.

Pull Request Comments

Note: This feature is only available in Github plugin at the moment.

You can write comments to pull requests in Git from a Screwdriver build through using metadata. The contents of the comments should be written to the meta summary object from your pipeline’s PR build.

To write out metadata to a pull request, you just need to set meta.summary with desired information. This data will show up as a comment in Git by a headless Git user.

For example, to add a coverage description, your screwdriver.yaml should look something like below:

jobs:
  main:
    steps:
      - comment: meta set meta.summary.coverage "Coverage increased by 15%"

You can also write things in markdown syntax as shown in the following example:

jobs:
  main:
    steps:
      - comment: meta set meta.summary.markdown "this markdown comment is **bold** and *italic*"

These settings will result in a Git comment that looks like:

Additionally, you can set meta.splitComments to write multiple comments instead of one

jobs:
  main:
    steps:
      - comment: meta set meta.splitComments "split"

This setting will result in Git comments that looks like:

Note: Screwdriver will try to edit the same comment in Git if multiple builds are run on it.

Additional Pull Request Checks

Note: This feature is only available in Github plugin at the moment.

You can also add additional status checks to pull requests to provide more granular information about the pull request build.

To additional checks to a pull request, you just need to set meta.status.<check> with desired information in JSON string format. This data will show up as a Git check on the pull request.

The fields you can set:

For example, to add two additional checks for findbugs and coverage, your screwdriver.yaml should look something like below:

jobs:
  main:
    steps:
      - status: |
          meta set meta.status.findbugs '{"status":"FAILURE","message":"923 issues found. Previous count: 914 issues.","url":"http://findbugs.com"}'
          meta set meta.status.coverage '{"status":"SUCCESS","message":"Coverage is above 80%."}'

These settings will result in Git checks that look like:

Coverage and Test Results

You can populate coverage results and test results, along with their url to build artifact on build page from a Screwdriver build through using metadata. Screwdriver UI will read from tests.coverage, tests.results, tests.coverageUrl and tests.resultsUrl in metadata and display/set them accordingly.

Example screwdriver.yaml should look something like below:

jobs:
  main:
    steps:
      - set-coverage-and-test-results: |
          meta set tests.coverage 100 # this should be the coverage percentage number
          meta set tests.results 10/10 # this should be `pass_tests_number/total_tests_number`
          meta set tests.coverageUrl /test/coverageReport.html # this should be a relative path to a build artifact
          meta set tests.resultsUrl /test/testReport.html # this should be a relative path to a build artifact

Note: metadata will override SonarQube results, and the precedence of meta overrides is: meta.tests.saucelabs > meta.tests.sonarqube > meta.tests

These settings will result in build page that looks like:

Event Labels

You can label your events using the label key from meta. This key can be useful when trying to determine which event to rollback.

Example screwdriver.yaml:

jobs:
  main:
    steps:
      - set-label: |
          meta set label VERSION_3.0 # this will show up in your pipeline events page

Result:

Slack Notifications

You can customize notification messages with meta. Meta keys are different for each notification plugin.

You will need special string formatting for mentions or channel links in Slack. You can read more about the available options in the Slack Documentation.

Basic

Example screwdriver.yaml notifying with Slack:

jobs:
  main:
    steps:
      - meta: |
          meta set notification.slack.message "<@yoshwata> Hello Meta!"

Result:

Job-based Slack message

Note: Job-based Slack notification meta data will overwrite the basic notification message.

Structure of meta variable is notification.slack.<jobname>.message, replacing <jobname> with the name of the Screwdriver job.

Example screwdriver.yaml notifying with specific Slack message for job slack-notification-test:

jobs:
  main:
    steps:
      - meta: |
          meta set notification.slack.slack-notification-test.message "<@yoshwata> Hello Meta!"

Result:

Job-based Slack Channel

Note: Job-based Slack channel meta will only overwrite the basic Slack notification channel. It is not a replacement for setting a notification channel.

Structure of meta variable is notification.slack.<jobName>.channels, replacing <jobname> with the name of the Screwdriver job.

The setting is a comma-separated string that allows setting multiple channels.

Example screwdriver.yaml notifying different Slack channels upon job failure for the component job:

shared:
  image: docker.ouroath.com:4443/x/y/z

  settings:
    slack:
      channels: [main_channel]
      statuses: [FAILURE]

jobs:
  component:
    steps:
      - meta: |
          meta set notification.slack.component.channels "fail_channel, prod_channel"

In the above example a Slack notification failure message will be send to channels fail_channel and prod_channel instead of main_channel. All other jobs in this pipeline would still post to main_channel.

Job-based minimized setting

Job-based Slack minimized meta setting will overwrite the default Slack minimized setting.

Structure of meta variable is notification.slack.<jobName>.minimized, replacing <jobName> with the name of the Screwdriver job.

Example screwdriver.yaml sending a minimized Slack message in case the component job was triggered by the scheduler:

shared:
  image: docker.ouroath.com:4443/x/y/z

  settings:
    slack:
      channels: [main_channel]
      statuses: [FAILURE]
      minimized: false

jobs:
  component:
    steps:
      - meta: |
          if [[ $SD_SCHEDULED_BUILD == true ]]; then
             meta set notification.slack.component.minimized true
          fi

In the above example a Slack notification message will be send in minimized format for the component job if it was triggered by the scheduler.

Mark Builds and Set Warnings

You can mark builds/events with warning status by setting build.warning.

Example screwdriver.yaml:

jobs:
  main:
    steps:
      - warning: meta set build.warning true

Result:

Additionally you can also set warning messages Example screwdriver.yaml:

jobs:
  main:
    steps:
      - setWarning: meta set build.warning.message "this is a warning message"
      - setAnotherWarning: meta set build.warning.anotherMessage "this is another warning message"

Result: Note: Only builds/events with successful status will be marked, warning message will still be displayed in build detail page regardless of build/event status

Using Lua for atomic updates

The meta tool creates a lock file, and holds a flock around each of its operations so that it may be executed by parallel operations in your build (for instance a Makefile invoked with make -j 4)

In addition to atomic get and set operations, should “update” be required, the meta commands may be invoked by an embedded Lua interpreter.

Example use cases:

atomically increment a number

meta lua -E 'meta.set("myNum", (tonumber(meta.get("myNum")) or 0) + 1)'

atomically insert some json into an array and return its index

One motivational use case is a job, which builds many things (like docker images) in parallel, and then in a later “publish” step, interrogates the array to do the pushes (to docker registry, e.g.).

meta lua insert.lua myArray '{"foo": "baz"}'

Explanation: Following the convention of the lua CLI, arguments are made available to Lua code in the global table arg. Without the -E flag, arg[0] will be run as a file.

File insert.lua:

-- Ensure args are passed as expected
assert(#arg >= 2, string.format("usage: %s key json_value", arg[0]))

-- https://github.com/vadv/gopher-lua-libs is preloaded, so any of its modules may be required
local json = require("json")
local key = arg[1]

-- This converts the json string argument to a Lua table, error
local toInsert, err = json.decode(arg[2])

-- Report errors, if any
assert(not err, tostring(err))

-- Get the current array from meta using the key arg or, when nil, an empty table
local array = meta.get(key) or {}

-- table.insert without index does "append"
table.insert(array, toInsert)

-- meta.set to save the value after insertion - Lua values are passed and converted to json for storage under the hood.
meta.set(key, array)

-- print the index
-- NOTE: index for meta.(get/set) purposes is 0-based, so subtract 1 from the array size
print(#array - 1)