Why your OpenTelemetry build broke, and how to check the whole set

If you have ever seen this, and had no idea why, this post is for you:

# go.opentelemetry.io/contrib/bridges/otelslog
.../[email protected]/handler.go:347:37: undefined: log.KeyValue
.../[email protected]/convert.go:20:30: undefined: log.Value
.../[email protected]/convert.go:30:14: too many errors

You didn't touch otelslog. You may never have opened it. The symbols it is missing live in a different module you also didn't touch. And whatever you actually changed appears nowhere in the error.

OpenTelemetry for Go is a set of modules that are released together and only work together, they carry different version numbers, and the two repositories they live in do not release at the same moment. So you can end up with two halves of the set from different releases - by pinning something, or just by running go get -u on the wrong day.

First, why Go usually saves you from this

Go's dependency management is famously dull, and that is a compliment. You run go get -u, you run go mod tidy, and it works.

The reason is minimal version selection. When two modules in your build ask for different versions of the same dependency, Go takes the highest one and uses that everywhere. One version per dependency per build, chosen deterministically, no lockfile negotiation.

So drift is normal and mostly invisible. Across our estate - 75 modules, 217 third-party dependencies - 20 are required at more than one version. Nine of those are things like golang.org/x/net at v0.57.0 in one module and v0.58.0 in the rest. Nobody has noticed, because MVS quietly picks v0.58.0 and both modules are fine with it.

That works because those dependencies are independent. golang.org/x/net does not care what else you have installed.

OpenTelemetry is not one dependency

It is roughly a dozen modules, published across two repositories, released together, with interfaces that move between releases.

Worse, they do not share a version number. Here is a real go.mod, OTel lines only:

go.opentelemetry.io/otel                          v1.45.0
go.opentelemetry.io/otel/sdk                      v1.45.0
go.opentelemetry.io/otel/log                      v0.21.0
go.opentelemetry.io/otel/sdk/log                  v0.21.0
go.opentelemetry.io/contrib/instrumentation/.../otelhttp  v0.70.0
go.opentelemetry.io/contrib/bridges/otelslog      v0.20.0
go.opentelemetry.io/auto/sdk                      v1.2.1

Five different version numbers. Every one of those is the same release.

That is the crux, so before going further: how would you know that?

Read versions.yaml - it is already on your disk

Every OTel module ships a file called versions.yaml at its root, and it is in your module cache right now:

$ cat $(go env GOMODCACHE)/go.opentelemetry.io/[email protected]/versions.yaml
module-sets:
  stable-v1:
    version: v1.45.0
    modules:
      - go.opentelemetry.io/otel
      - go.opentelemetry.io/otel/sdk
      - go.opentelemetry.io/otel/metric
      - go.opentelemetry.io/otel/trace
      ...
  experimental-logs:
    version: v0.21.0
    modules:
      - go.opentelemetry.io/otel/log
      - go.opentelemetry.io/otel/sdk/log
      - go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp
      - go.opentelemetry.io/otel/exporters/stdout/stdoutlog
      ...

There it is. stable-v1 is at v1.45.0, experimental-logs is at v0.21.0, and those are one release. The contrib repository has its own:

$ cat $(go env GOMODCACHE)/go.opentelemetry.io/[email protected]/versions.yaml
  experimental-instrumentation:
    version: v0.70.0
    modules:
      - go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
      - go.opentelemetry.io/contrib/instrumentation/runtime
  experimental-bridge:
    version: v0.20.0
    modules:
      - go.opentelemetry.io/contrib/bridges/otelslog

So the release that ships otel v1.45.0 also ships log v0.21.0, otelhttp v0.70.0 and otelslog v0.20.0. Four numbers, one thing.

Could we just use the stable modules?

That was my first thought after reading VERSIONING.md, which is explicit that v0 modules are experimental and may break at any time. If pinning an experimental module is asking for trouble, use the stable ones.

You can't. Here is what is actually stable as of the v1.45.0 release:

And here is what is not:

If you want to emit logs, or instrument an HTTP server, there is no stable module to use. Those are not exotic requirements. So "prefer the stable modules" is not advice anyone can act on - the experimental modules are the product for most of what people actually do with OTel in Go.

So do the docs address the problem?

Not the one that bites.

VERSIONING.md says all stable modules of the same major version share the same entire version number. That is clear, useful, and about the half that was never going to hurt you. It says experimental modules may change at any time, which explains why a mismatch breaks but offers nothing on avoiding it.

What it does not say - what nothing consumer-facing says - is which experimental versions belong with which release. Count the table above: nine separate experimental version lines, plus two stable ones, all shipped in the same release, with no arithmetic relating any of them to any other. log v0.21.0 and otelslog v0.20.0 are one release apart in appearance and the same release in fact.

The failure we hit was otelslog from experimental-bridge against otel/log from experimental-logs. Two different sets, in two different repositories, with unrelated version numbers, and nothing anywhere telling you they move together.

So yes, pinning an experimental module was the wrong instinct, and I would not do it again. But "don't pin experimental modules" only helps if you can tell what the unpinned, mutually-consistent set is. That answer exists in exactly one place, and it is not written for you.

What actually causes it

Two things have to be true at once for this to bite, and it is worth separating them because only one of them is anybody's mistake.

One: the modules are a set. They are released together, their interfaces move between releases, and a member from release N generally will not compile against a member from release N+1. Go has no concept of this. MVS resolves each module independently, which is right for almost everything else you depend on.

Two: the set does not become available all at once. This is the part I had not appreciated, and it is in their own versioning policy:

Contrib modules will be kept up to date with this project's releases. Due to the dependency contrib modules will implicitly have on this project's modules the release of stable contrib modules to match the released version number will be staggered after this project's release. There is no explicit time guarantee for how long after this project's release the contrib release will be.

So the core repository releases, and the contrib repository releases some time afterwards. During that window the newest published otel/log and the newest published otelslog are from different releases, because the matching otelslog does not exist yet.

Which means go get -u during that window gives you a broken set, with no pin involved and no mistake on your part. You asked for the latest of everything and the latest of everything was not, at that moment, a coherent thing to ask for.

Our own break was self-inflicted - we pinned a module and left it. But the staggered release means the same failure is reachable by doing exactly what the tooling encourages. That is the difference between a footgun and a sharp edge, and it is why a check is worth having rather than just a rule about not pinning.

(I have not measured how long these windows typically are, only that the policy declines to bound them. If somebody has, I would like to see the numbers.)

I call these lockstep sets: groups of modules that are only ever tested together and whose interfaces move between releases. OTel is one. gRPC and protobuf behave similarly. Almost nothing else does.

Go cannot see any of this. MVS resolves each module independently, which is correct behaviour for 206 of our 217 dependencies and wrong for the eleven that are really four things.

(I have only checked the documentation shipping in the repository. If the wider OpenTelemetry site covers the stable-to-experimental mapping somewhere, I did not find it, and would be glad to be corrected.)

Watch it break, in about a minute

mkdir otel-split && cd otel-split
go mod init example.com/otelsplit

main.go:

package main

import (
	"go.opentelemetry.io/contrib/bridges/otelslog"
	sdklog "go.opentelemetry.io/otel/sdk/log"
)

func main() {
	lp := sdklog.NewLoggerProvider()
	_ = otelslog.NewHandler("x", otelslog.WithLoggerProvider(lp))
}

A matched pair from one release builds clean:

go get go.opentelemetry.io/contrib/bridges/[email protected]
go get go.opentelemetry.io/otel/sdk/[email protected]
go build ./...

Now hold one member back, which is exactly what a stale pin in a shared library does to every consumer:

go get go.opentelemetry.io/contrib/bridges/[email protected]
go build ./...

And you get the error at the top of this post. Note where it points: files inside otelslog, for symbols that live in otel/log. Your change is not mentioned. The error is accurate, and about four steps from the cause.

(Those versions will age. Take any two releases a few apart and it behaves the same way.)

If your build just broke, do this

Move the whole set to one release, in one command. Not the module that is erroring - that is the symptom:

go get \
  go.opentelemetry.io/[email protected] \
  go.opentelemetry.io/otel/[email protected] \
  go.opentelemetry.io/otel/[email protected] \
  go.opentelemetry.io/otel/sdk/[email protected] \
  go.opentelemetry.io/otel/exporters/otlp/otlplog/[email protected] \
  go.opentelemetry.io/contrib/instrumentation/net/http/[email protected] \
  go.opentelemetry.io/contrib/bridges/[email protected]
go mod tidy

Take the version for each module from the versions.yaml of the release you are moving to, rather than from whatever go get -u happened to resolve.

Do not use exclude for this

We did, for months, and it is the worst of the available options - on top of being the wrong instinct in the first place, given what the versioning policy says about experimental modules.

exclude does not mean "use this version". It means "never resolve this one", and leaves MVS to find something else. Which means:

• it is invisible where it hurts. The build error names an instrumentation package; the exclude is in a different file entirely.
• it fights go get -u silently rather than failing, so it has to be re-applied, so it becomes a ritual, so eventually someone forgets.
• it propagates. An exclude in a library applies to everything that depends on that library. Ours lived in shared code, so it came back every time that code moved.

If you must hold a version, pin it in require where a human will see it, and write down why. Better: move the set and delete the pin. Ours went estate-wide on 9 August and none have come back.

Checking it, including in CI

Two different failures, and only one of them is visible from a single repo.

A module holding half a set. One go.mod with stable-v1 at the current release and experimental-logs a release behind. This is what breaks a build, and everything you need to detect it is in that repo's module cache. It belongs in per-repo CI.

A repository that is wholly behind. Internally consistent, builds fine alone, and only bites when its libraries meet a newer set somewhere else. Nothing in one checkout can see that. It needs something that looks at every repository - a scheduled job, or a workspace-level command.

For the first one, the check is: for each OTel module in go.mod, look up which module-set it belongs to in versions.yaml, and confirm every module in that set is at the set's version. That is a script, not a project, and it needs no network and no hardcoded table.

I will admit I built ours the hard way first - inferring the sets by seeing which versions the rest of the workspace agreed on - because I had not noticed versions.yaml was sitting in the module cache the whole time. It works, and it is strictly worse than reading the file the maintainers already publish.

The wider point

Some dependencies are not dependencies. They are a distribution, and the version number on any one part of it does not describe the whole. A database driver and its migration tooling can be like this. Any library whose README contains a compatibility matrix almost certainly is - the matrix is there because there is nowhere else to put it.

OTel is unusual in that it does publish the answer in a machine-readable form. Most such sets do not, and you are left comparing README tables by hand.

What is missing is not the data, not the tooling, and not the policy - all three exist and all three are public. What is missing is the one line joining them up for a consumer: the experimental modules that ship with a given stable release are listed in versions.yaml, and here is how to read it.

That line would be worth more than the tool I wrote. Nine experimental version lines is not a thing anyone tracks by hand, and telling people the stable modules stay in step does not help when the stable modules are not the part they are using.

If you have a cleaner way of doing this in Go, I would like to hear it.