Pipeline Templates

Pipeline Templates are snippets of predefined configuration that users can use to define a screwdriver.yaml. A pipeline template contains a series of predefined jobs with steps along with a selected container image.

Finding pipeline templates

To figure out which templates already exist, you can make a GET call to the /pipeline/templates API endpoint. You can also find Pipeline Templates through the UI by navigating to <YOUR_UI_URL>/templates/pipeline.

Example pipeline template

namespace: nodejs
name: test
version: '1.0.4'
description: An example pipeline template for nodejs
maintainer: foo@bar.com
config:
  shared:
    image: node:lts
  jobs:
    main:
      steps:
        - install: npm install
        - test: npm test
      secrets:
        - NPM_TOKEN

Using a pipeline template

To use a template, define a screwdriver.yaml with a top level template key. In this example, we are using a nodejs/test template.

Example screwdriver.yaml:

template: nodejs/test@1.0.4
shared:
  environment: 
    FOO: bar

Version is semver compatible. For example you can refer above template with nodejs/test@1 or nodejs/test@1.0

You can also refer to a template version with a tag name if the template has one. All the versions and tags of a template are displayed at the bottom of a template’s description, such as for the example template, which has the tags latest and stable.

Most templates will tag the most recent version as latest, and many templates use either automated testing or manual curation to identify some version as stable. These are floating tags, so using them in a job means its template-provided steps may suddenly change.

If no template version is specified, the most recently published will be used. This is usually synonymous with specifying the latest tag. It is generally better to explicitly state a template version than to implicitly use latest.

The most reliable way to avoid unexpected template changes is to refer to a specific version of the template. For instance, nodejs/test@1.0.4 is an immutable reference to a particular list of steps. Using a reference such as nodejs/test@1.0 means that a job will automatically use nodejs/test@1.0.5 when it becomes available, but that comes with risk of an unexpected change in behavior.

Customization

Many fields can be customized when using Pipeline Template: shared, jobs, cache, subscribe, parameters, annotations, and stages.

Shared

When shared is set in either the Pipeline Template or user yaml, priority will be (in decreasing order): user job > user shared > pipeline template job > pipeline template shared

Jobs

Some customization can be done within the jobs configuration when using a Pipeline Template.

For user-defined jobs with names that already exist in the pipeline template, customization is limited to a set of certain fields: image, settings, environment, and requires.

For user-defined jobs with names that do not already exist in the pipeline template, all jobs fields are allowed.

Example

Given a pipeline template:

shared:
  image: node:lts
  environment:
    VAR1: "one"
    VAR2: "two"
  steps:
    - init: npm install
    - test: npm test
  settings:
    email: [test@email.com, test2@email.com]
    slack: 'mychannel'
jobs:
  main:
    requires: [~pr, ~commit]
  second:
    requires: [main]

And user defined yaml:

template: sd-test/example-template@latest
jobs:
  main:
    requires: [~commit]
    image: node:20
    settings:
      email: [test@email.com, test3@email.com]
    environment:
      VAR3: "three"
      VAR1: "empty"
  third:
    requires: []
    image: node:lts
    steps:
      - echo: echo third job

Resulting config would be:

jobs:
  main:
    requires: [~commit]
    image: node:20
    steps:
      - init: npm install
      - test: npm test
    environment:
      VAR1: "empty"
      VAR2: "two"
      VAR3: "three"
    settings:
      email: [test@email.com, test3@email.com]
      slack: 'mychannel'
  second:
    requires: [main]
    image: node:lts
    steps:
      - init: npm install
      - test: npm test
    environment:
      VAR1: "one"
      VAR2: "two"
    settings:
      email: [test@email.com, test2@email.com]
      slack: 'mychannel'
  third:
    requires: []
    image: node:lts
    steps:
      - echo: echo third job

Version/Tag Semantics

Template versions can be referenced in a variety of ways that express users’ trade-off between an unchanging set of steps and automatically using improvements that a template’s maintainers have added. As above, these examples reference nodejs/test template.

Example

For this configuration:

template: nodejs/test@stable
shared: 
  environment:
    FOO: bar

Screwdriver takes the template configuration and plugs it in, so that the screwdriver.yaml becomes:

shared:
  environment:
    FOO: bar
jobs:
  main:
    image: node:lts
    requires: [~pr, ~commit]
    steps:
      - install: npm install
      - test: npm test
    secrets:
      - NPM_TOKEN

Creating a template

Publishing and running pipeline templates must be done from a Screwdriver pipeline.

Writing a template yaml

To create a template, create a new repo with a sd-template.yaml file. The file should contain a namespace, name, version, description, maintainer email, and a config with jobs. Basic example can be found in our screwdriver-cd-test/pipe;line-template-example repo.

Example sd-template.yaml:

namespace: myNamespace
name: template_name
version: '1.3'
description: pipeline template for testing
maintainer: foo@bar.com
config:
  jobs:
    main:
      requires: [~pr, ~commit]
      steps:
        - install: npm install
        - test: npm test
      environment:
          FOO: bar
          TEST: data
      secrets:
          - NPM_TOKEN

Writing a screwdriver yaml for your template repo

Validating templates

To validate your template, run the pipeline-template-validate script from the screwdriver-template-main npm module in your main job to validate your template. This means the build image must have NodeJS and NPM properly installed to use it. To publish your template, run the pipeline-template-publish script from the same module in a separate job.

By default, the file at ./sd-template.yaml will be read. However, a user can specify a custom path using the env variable: SD_TEMPLATE_PATH.

You can also validate your sd-template.yaml and screwdriver.yaml through the UI by copy pasting it at <YOUR_UI_URL>/validator.

Tagging templates

You can optionally tag a specific template version by running the pipeline-template-tag script from the screwdriver-template-main npm package. This must be done by the same pipeline that your template is created by. You will need to provide arguments to the script: template namespace, name and tag. You can optionally specify a version; the version needs to be an exact version (see tag step). If the version is omitted, the most recent version will be tagged.

To remove a template tag, run the pipeline-template-remove-tag script. You will need to provide the template name and tag as arguments.

Example screwdriver.yaml:

shared:
  image: node:lts
jobs:
  main:
    requires: [~pr, ~commit]
    steps:
      - validate: npx -y -p screwdriver-template-main pipeline-template-validate
    environment:
      SD_TEMPLATE_PATH: ./path/to/template.yaml
  publish:
    requires: [main]
    steps:
      - publish: npx -y -p screwdriver-template-main pipeline-template-publish
      - tag: npx -y -p screwdriver-template-main pipeline-template-tag --namespace myNamespace --name template_name --version 1.3.0 --tag stable
    environment:
      SD_TEMPLATE_PATH: ./path/to/template.yaml
  remove:
    steps:
      - remove: npx -y -p screwdriver-template-main pipeline-template-remove --namespace myNamespace --name template_name
  remove_tag:
    steps:
      - remove_tag: npx -y -p screwdriver-template-main pipeline-template-remove-tag --namespace myNamespace --name template_name --tag stable

Create a Screwdriver pipeline with your template repo and start the build to validate and publish it.

To update a Screwdriver template, make changes in your SCM repository and rerun the pipeline build.

Testing a template

In order to test your template, set up a remote test for your template by creating another pipeline which uses your template, as seen in the pipeline-template-test-example. This example pipeline runs after the publish_nodejs_template is done by using the remote triggers feature. Note: You cannot test your template in the same pipeline, as template step expansion is done at event creation time. Therefore, the pipeline would use an older version of your template if you try to use it in the pipeline where you create it.

Removing a template

Using screwdriver-template-main npm package

To remove your template, you can run the pipeline-template-remove script. You will need to provide the template namespace and name as an argument.

Using the UI

Or, you can remove your template and all its associated tags and versions by clicking on the trash icon in the UI on the template page.

Note: Do not delete your template pipeline beforehand, because it is required to determine who has permission to delete the template.