Suggest an Edit

Receiving artifacts from GCS

This guide explains how to configure Spinnaker to trigger pipelines based on changes in a Google Cloud Storage (GCS) bucket, and inject changed GCS objects as artifacts into a pipeline.

This functionality uses Google’s Pub/Sub system for delivering messages to Spinnaker, and must be configured to send messages to Spinnaker’s event bus as shown below.

Prerequisite configuration/setup

If you (or your Spinnaker admin) has already configured Spinnaker to listen to a Pub/Sub messages from the GCS bucket you plan to publish objects to, you can skip this section. One Pub/Sub subscription can be used to trigger as many independent Spinnaker pipelines as needed.

You need the following:

  • A billing enabled Google Cloud Platform (GCP) project.

    This will be referred to as $PROJECT from now on.

  • gcloud. Make sure to run gcloud auth login if you have installed gcloud for the first time.

  • A running Spinnaker instance. This guide shows you how to configure an existing one to accept GCS messages, and download the files referenced by the messages in your pipelines.

At this point, we will configure Pub/Sub, and a GCS artifact account. The intent is that the Pub/Sub messages will be received by Spinnaker whenever a file is uploaded or changed, and the artifact account will allow you to download these where necessary.

1. Configure Google Pub/Sub for GCS

Follow the Pub/Sub configuration, in particular, pay attention to the GCS section since this is where we’ll be publishing our files to.

2. Configure a GCS artifact account

Follow the GCS artifact configuration.

3. Apply your configuration changes

Once the Pub/Sub and artifact changes have been made using Halyard, run

hal deploy apply

to apply them in Spinnaker.

Using GCS artifacts in pipelines

We will need either an existing or a new pipeline that we want to be triggered on changes to GCS artifacts. If you do not have a pipeline, create one as shown below.

You can create and edit pipelines in the Pipelines tab of Spinnaker

Configure the GCS artifact

Once you have your pipeline ready, we need to declare that this pipeline expects to have a specific artifact matching some criteria is available before the pipeline starts executing. In doing so, you guarantee that an artifact matching your description is present in the pipeline’s execution context; if no artifact for this description is present, the pipeline won’t start.

Now to configure the artifact, change the “Custom” dropdown to “GCS”, and enter the fully qualifed GCS path in the Object path field. Note: this path can be a regex. You can, for example, set the object path to be gs://${BUCKET}/folder/.* to trigger on any change to an object inside folder in your ${BUCKET}.

${BUCKET} is a placeholder for the GCS bucket name that you have configured to receive Pub/Sub messages from above.

Configure the GCS trigger

Now that the expected artifact has been added, let’s add a Pub/Sub trigger to run our pipeline.

Next, we must configure the trigger:

  • Type is “Pub/Sub”.
  • Pub/Sub System Type is “Google”.
  • Subscription Name depends on what you’ve configured in your Pub/Sub configuration using Halyard.
  • Attribute Constraints must be configured to include the pair eventType: OBJECT_FINALIZE (see the docs for an explanation).
  • Expected Artifacts must reference the artifact defined previously.
By setting the Expected Artifacts field in the trigger config, you guarantee that this pubsub subscription will only trigger this pipeline when an artifact matching your requirements is present in the pubsub message.

Test the pipeline

If you upload a file to a path matching your configured Object path, the pipeline should execute. If it doesn’t, you can start by checking the logs in the Echo service.