Integrate OpenTelemetry tracing with Google Cloud Tasks

Cloud Tasks is a basic queuing system available in GCP. Tasks inside a queue are dispatched to an HTTP endpoint for processing, and the details of where the task gets sent to is configured on the task itself. Like so:

import { CloudTasksClient } from '@google-cloud/tasks'; const tasks = new CloudTasksClient(); client.createTask({  // Queue ID  parent: client.queuePath('gcp-project-id', 'australia-southeast1', 'my-queue'),  // HTTP request details  httpRequest: {    headers: {      // ...    },    httpMethod: 'POST',    body: Buffer.from(JSON.stringify({      some: 'payload',    })).toString('base64'),  },});

When you make an HTTP request normally, OpenTelemetry will propagate context across that HTTP request so that the trace for the callee's execution ends up linked to the caller's trace. This is unfortunately not the case for Cloud Tasks; you'll see a span for the network request to Google's servers for the createTask call and nothing further. Unlike the Pub/Sub client library, there is no enableOpenTelemetryTracing option you can set to opt-in to trace context propagation.

Fortunately it is pretty easy to fix this by using the propagation API from OpenTelemetry:

import { CloudTasksClient } from '@google-cloud/tasks';import { context, propagation } from '@opentelemetry/api'; const client = new CloudTasksClient(); const headers: Record<string, string> = {};propagation.inject(context.active(), headers); client.createTask({  // parent  httpRequest: {    headers,    // ...  },});

The propagation.inject method will inject propagation information according to whatever propagator(s) you have configured. If you are using the B3 context propagator, for instance, then the headers object will end up containing a b3 header value. So long as the worker on the other end of the queue is also set up with OpenTelemetry using the same context propagator, then everything will magically work after you make this change.

You should think about the lifecycle of your task handler when you propagate context like this. Cloud Tasks will automatically retry failed work according to the retry policy defined on the queue, and it is a legitimate pattern to implement polling of an external resource by having a task handler intentionally fail to trigger retries over an extended period of time. If you have such a task handler, it is probably not a good idea to propagate context. Doing so will make your trace much larger than it would otherwise be.