Job Templates

Templates are snippets of predefined code that people can use to replace a job definition in a screwdriver.yaml. A template contains a series of predefined steps along with a selected Docker image.

Finding templates

To figure out which templates already exist, you can make a GET call to the /templates API endpoint. You can also see templates in the UI at <YOUR_UI_URL>/templates. You can also find Job Templates through the UI by navigating to <YOUR_UI_URL>/templates/job.

Example templates page: Templates

Using a template

To use a template, define a screwdriver.yaml with a template key. In this example, we are using the nodejs/test template.

Example screwdriver.yaml:

jobs:
    main:
        requires: [~pr, ~commit]
        template: nodejs/test@1.0.4

Version is semver compatible. For example you can refer above template with nodejs/test@1 or nodejs/test@1.0

You can also refer to a template version with a tag name if the template has one. All the versions and tags of a template are displayed at the bottom of a template’s description, such as for the example template, which has the tags latest and stable.

Most templates will tag the most recent version as latest, and many templates use either automated testing or manual curation to identify some version as stable. These are floating tags, so using them in a job means its template-provided steps may suddenly change.

If no template version is specified, the most recently published will be used. This is usually synonymous with specifying the latest tag. It is generally better to explicitly state a template version than to implictly use latest.

The most reliable way to avoid unexpected template changes is to refer to a specific version of the template. For instance, nodejs/test@1.0.4 is an immutable reference to a particular list of steps. Using a reference such as nodejs/test@1.0 means that a job will automatically use nodejs/test@1.0.5 when it becomes available, but that comes with risk of an unexpected change in behavior.

Version/Tag Semantics

Template versions can be referenced in a variety of ways that express users’ trade-off between an unchanging set of steps and automatically using improvements that a template’s maintainers have added. As above, these examples reference nodejs/test template.

Example

For this configuration:

jobs:
    main:
        requires: [~pr, ~commit]
        template: nodejs/test@stable

Screwdriver takes the template configuration and plugs it in, so that the screwdriver.yaml becomes:

jobs:
    main:
        image: node:lts
        requires: [~pr, ~commit]
        steps:
          - install: npm install
          - test: npm test
        environment:
            FOO: bar
        secrets:
              - NPM_TOKEN

Overriding Template Steps

A job can override Template steps by wrapping or replacing existing steps.

Wrap

Wrapping is when you add commands to run before and/or after an existing step. To wrap a step from a template, add a pre or post in front of the step name.

Example:

jobs:
    main:
        requires: [~pr, ~commit]
        template: nodejs/test@1.0.3
        steps:
            - preinstall: echo pre-install
            - postinstall: echo post-install

This will run the command echo pre-install before the template’s install step, and echo post-install after the template’s install step.

Replace

To replace a step from a template, add your command with the same template’s step name.

Example:

jobs:
    main:
        requires: [~pr, ~commit]
        template: nodejs/test@1.0.3
        steps:
            - install: echo skip installing

This will run the command echo skip installing for the install step.

Note: You cannot replace a locked step.

You can also replace the image defined in the template. Some template steps might rely on commands or environment invariants that your image may not have, so be careful when replacing.

Example:

jobs:
    main:
        requires: [~pr, ~commit]
        image: node:latest
        template: nodejs/test@1.0.3

Merging with shared steps

When overriding Template steps, a job can get the step definitions from either shared.steps or job.steps with precedence for steps defined in job section. This follows the same order of precedence for step definitions without using a template. Users can change this behavior using annotation screwdriver.cd/mergeSharedSteps: true. When true steps in shared and job sections are merged when a Template is used.

Example

shared:
  annotations:
    screwdriver.cd/mergeSharedSteps: true
  steps:
     - premotd: echo build
jobs:
    main:
        template: python/package_rpm@latest
        requires: [~pr, ~commit]
        steps:
         - preinit_os: echo replace

Example

The following example defines a merged shared configuration for image and steps, which is used by the main and main2 jobs.

shared:
    image: node:lts
    steps:
        - init: npm install
        - pretest: npm lint
        - test: npm test

jobs:
    main:
        requires: [~pr, ~commit]
        image: node:lts
    main2:
        template: sd/noop@latest
        annotations:
            screwdriver.cd/mergeSharedSteps: true
        requires: [main]
        steps:
            - test: echo Skipping test

The above example would be equivalent to:

jobs:
    main:
        requires: [~pr, ~commit]
        image: node:lts
        steps:
             - init: npm install
             - pretest: npm lint
             - test: npm test
    main2:
        template: sd/noop@latest
        annotations:
             screwdriver.cd/mergeSharedSteps: true
        requires: [main]
        image: node:lts
        steps:
             - pretest: npm lint
             - test: echo Skipping test

Order

When using a template in your configuration, you can pick and choose steps defined by the template and your own configuration with the order field. This field is defined as an ordered array of step names.

Caveats when using order:

Example sd-template.yaml:

namespace: nodejs
name: publish
version: "2.0.1"
description: 'Publish an npm package'
maintainer: myname@foo.com
images:
  lts: node:lts
  latest: node:latest
config:
  image: stable
  steps:
    - install: npm install
    - publish: npm publish
    - coverage: coverage test.js

Example screwdriver.yaml:

jobs:
  main:
    requires: [~commit]
    image: stable
    template: nodejs/publish@2
    order: [clone, install, doesnotexist, test, publish, coverage]
    steps:
      - test: npm test
      - clone: git clone https://github.com/screwdriver-cd/toolbox.git ci
      - coverage: ./ci/coverage.sh

Result:

jobs:
  main:
    requires: [~commit]
    image: node:lts
    steps:
      - clone: git clone https://github.com/screwdriver-cd/toolbox.git ci
      - install: npm install
      - test: npm test
      - publish: npm publish
      - coverage: ./ci/coverage.sh  # This step was overwritten by the job

Creating a template

Publishing and running templates must be done from a Screwdriver pipeline.

Writing a template yaml

To create a template, create a new repo with a sd-template.yaml file. The file should contain a namespace, name, version, description, maintainer email, and a config with an image and steps. If no namespace is specified, a ‘default’ namespace will be applied. An optional images keyword can be defined to list supported images with a descriptive label. Basic example can be found in our screwdriver-cd-test/template-example repo.

Example sd-template.yaml:

namespace: myNamespace
name: template_name
version: '1.3'
description: template for testing
maintainer: foo@bar.com
images:
    lts-image: node:lts
    latest-image: node:latest
config:
    image: stable-image
    steps:
        - install: npm install
        - test: npm test
    environment:
        FOO: bar
    secrets:
        - NPM_TOKEN

Template images

We recommend using the images feature, which can be configured to list supported images with a descriptive label or alias.

For example:

namespace: myNamespace
name: template_name
version: '1.3'
description: template for testing
maintainer: foo@bar.com
images:
    lts-image: node:lts
    latest-image: node:latest

Users can pick an alias from the list and use it like so:

jobs:
    main:
        template: myNamespace/template_name@1.3.0
        image: stable-image

Example repo: https://github.com/screwdriver-cd-test/template-images-example

Template Steps

Avoid using any wrapping prefixes (pre or post) in your step names, as it can lead to problems when users try to modify or enhance your steps. For example, if a template has these steps:

config:
    image: node:lts
    steps:
        - preinstall: echo Installing
        - install: npm install
        - test: npm test

And a user consumes that template with some additional steps:

jobs:
    main:
        template: myNamespace/template_name@1.3.0
        steps:
          - preinstall: echo foo

It becomes unclear whether the user was trying to override preinstall or wrap install.

Template Locked Steps

You can specify steps that cannot be overwritten and must be included when using order by adding a locked key to your steps. It expects a boolean value (true/false; default is false). This flag applies to any template or job that uses this template. All templates using a template with a locked step will also have the same locked step.

config:
    image: node:lts
    steps:
        - preinstall: echo Installing
        - install: npm install
        - test:
            command: npm test
            locked: true

Template parameters

You can define parameters that can be used in the steps.

Example sd-template.yaml:

namespace: myNamespace
name: favorites
version: '2.0.1'
description: template for testing parameters
maintainer: foo@bar.com
config:
  image: node:lts
  parameters:
    music:
      value: [ "country", "hip hop" ]
      description: "favorite music"
    color: [ "black", "white" ]
    sports:
      value: [ "baseball", "basketball" ]
  steps:
    - step_print_template_parameters: |
        echo music = $(meta get parameters.music)
        echo color = $(meta get parameters.color)
        echo sports = $(meta get parameters.sports)

Example repo: https://github.com/screwdriver-cd-test/template-parameters-example

These parameters are inherited by all the jobs using the template.

Example screwdriver.yaml:

jobs:
  # Inherits parameters "music", "color" and "sports" from the template.
  main1:
    requires: [~pr, ~commit]
    template: favorites/myNamespace@2
  # Inherits parameters "music", "color" and "sports" from the template.
  main2:
    requires: [main1]
    template: favorites/myNamespace@2

is equivalent to

jobs:
  main1:
    requires: [~pr, ~commit]
    parameters:
      music:
        value: [ "country", "hip hop" ]
        description: "favorite music"
      color: [ "black", "white" ]
      sports:
        value: [ "baseball", "basketball" ]
    steps:
      - step_print_template_parameters: |
          echo music = $(meta get parameters.music)
          echo color = $(meta get parameters.color)
          echo sports = $(meta get parameters.sports)
  main2:
    requires: [main1]
    parameters:
      music:
        value: [ "country", "hip hop" ]
        description: "favorite music"
      color: [ "black", "white" ]
      sports:
        value: [ "baseball", "basketball" ]
    steps:
      - step_print_template_parameters: |
          echo music = $(meta get parameters.music)
          echo color = $(meta get parameters.color)
          echo sports = $(meta get parameters.sports)

Users can override the parameter definition at pipeline scope or/and at job scope with job scope taking precedence over pipeline scope.

Example screwdriver.yaml

# Overrides the parameter "music" from the template (favorites/myNamespace) used by the jobs at pipeline scope which get applied to all the jobs, unless the job overrides it.
parameters:
  music: [jazz, rock]

jobs:
  # Inherits parameters "color" and "sports" from the template.
  # Since the parameter "music" from the template is overridden at the pipeline scope, it is not is not inherited at job scope.
  default_template_params:
    requires: [~pr, ~commit]
    template: favorites/myNamespace@2

  # Inherits parameter "sports" from the template.
  # Since the parameter "music" from the template is overridden at the pipeline scope, it is not is not inherited at job scope.
  # Overrides the parameter "color" from the template.
  override_template_params:
    requires: default_template_params
    template: favorites/myNamespace@2
    parameters:
      color: [ red, blue ]

Example repo: https://github.com/screwdriver-cd-test/job-with-template-parameters-build-example

Caveats

Template Composition

You can also use a template in the config section of an sd-template.yaml file.

Caveats:

Example preexisting sd-template.yaml:

namespace: nodejs
name: publish
version: "2.0.1"
description: 'Publish an npm package'
maintainer: myname@foo.com
images:
  lts: node:lts
  latest: node:latest
config:
  image: stable
  steps:
    - install: npm install
    - publish: npm publish
    - coverage: coverage test.js

Example sd-template.yaml:

namespace: d2lam
name: personal
version: "1.0.2"
description: 'Do some stuff'
maintainer: d2lam@foo.com
images:
  test: node:lts
config:
  template: nodejs/publish@2
  image: stable
  order: [clone, install, doesnotexist, test, publish, coverage]
  steps:
    - test: npm test
    - clone: git clone https://github.com/screwdriver-cd/toolbox.git ci
    - coverage: ./ci/coverage.sh

Result:

namespace: d2lam
name: personal
version: "1.0.2"
description: 'Do some stuff'
maintainer: d2lam@foo.com
images:
  lts: node:lts
  latest: node:latest
  test: node:lts
config:
  image: stable
  steps:
    - clone: git clone https://github.com/screwdriver-cd/toolbox.git ci
    - install: npm install
    - test: npm test
    - publish: npm publish
    - coverage: ./ci/coverage.sh  # This step was overwritten by the d2lam/personal template

Writing a screwdriver yaml for your template repo

Validating templates

To validate your template, run the template-validate script from the screwdriver-template-main npm module in your main job to validate your template. This means the build image must have NodeJS and NPM properly installed to use it. To publish your template, run the template-publish script from the same module in a separate job.

By default, the file at ./sd-template.yaml will be read. However, a user can specify a custom path using the env variable: SD_TEMPLATE_PATH.

You can also validate your sd-template.yaml and screwdriver.yaml through the UI by copy pasting it at <YOUR_UI_URL>/validator.

Tagging templates

You can optionally tag a specific template version by running the template-tag script from the screwdriver-template-main npm package. This must be done by the same pipeline that your template is created by. You will need to provide arguments to the script: template name and tag. You can optionally specify a version; the version needs to be an exact version (see tag step). If the version is omitted, the most recent version will be tagged (see autotag step).

To remove a template tag, run the template-remove-tag script. You will need to provide the template name and tag as arguments.

Example screwdriver.yaml:

shared:
    image: node:lts
jobs:
    main:
        requires: [~pr, ~commit]
        steps:
            - install: npm install screwdriver-template-main
            - validate: ./node_modules/.bin/template-validate
        environment:
            SD_TEMPLATE_PATH: ./path/to/template.yaml
    publish:
        requires: [main]
        steps:
            - install: npm install screwdriver-template-main
            - publish: ./node_modules/.bin/template-publish
            - autotag: ./node_modules/.bin/template-tag --name myNamespace/template_name --tag latest
            - tag: ./node_modules/.bin/template-tag --name myNamespace/template_name --version 1.3.0 --tag stable
        environment:
            SD_TEMPLATE_PATH: ./path/to/template.yaml
    remove:
        steps:
            - install: npm install screwdriver-template-main
            - remove: ./node_modules/.bin/template-remove --name myNamespace/template_name
    remove_tag:
        steps:
            - install: npm install screwdriver-template-main
            - remove_tag: ./node_modules/.bin/template-remove-tag --name myNamespace/template_name --tag stable

Create a Screwdriver pipeline with your template repo and start the build to validate and publish it.

To update a Screwdriver template, make changes in your SCM repository and rerun the pipeline build.

Testing a template

In order to test your template, set up a remote test for your template by creating another pipeline which uses your template, as seen in the template-test-example. This example pipeline runs after the publish_nodejs_template is done by using the remote triggers feature. Note: You cannot test your template in the same pipeline, as template step expansion is done at event creation time. Therefore, the pipeline would use an older version of your template if you try to use it in the pipeline where you create it.

Using the build cache

To use the build cache feature, the store-cli command can be invoked in a step. For instance, if you are caching your node_modules/ folder, you can specify a step before the npm install command that downloads the cache and another step afterwards that uploads the cache. You can also move the uploading cache step to a teardown with the teardown- prefix.

config:
    image: node:lts
    steps:
        - getcache: store-cli get node_modules/ --type=cache --scope=event || echo "Failed to fetch Cache"
        - install: npm install
        - teardown-putcache: store-cli set node_modules/ --type=cache --scope=event || echo "Failed to publish Cache"

Removing a template

Using screwdriver-template-main npm package

To remove your template, you can run the template-remove script. You will need to provide the template name as an argument.

Using the UI

Or, you can remove your template and all its associated tags and versions by clicking on the trash icon in the UI on the template page.

Note: Do not delete your template pipeline beforehand, because it is required to determine who has permission to delete the template.

Removing