Copilot CLI toolcache cache-hit never creates /usr/local/bin/copilot → agent dies with spawn ENOENT ("engine terminated before producing output")

[Shazwazza]

Summary

When install_copilot_cli.sh hits the runner toolcache, it activates the cached CLI by appending the toolcache directory to GITHUB_PATH and returns early — it never creates /usr/local/bin/copilot. But the compiled lock file launches the agent inside the AWF sandbox using that hardcoded absolute path, so spawn() fails with ENOENT and the run dies before producing any output.

Surfaced to users as:

Engine Failure: The copilot engine terminated before producing output.

This makes it a latent, cache-dependent failure: the same unchanged workflow succeeds or fails depending purely on whether the runner's toolcache happens to hold a CLI version inside the compat window.

Reproduction

Any engine: copilot workflow with the firewall sandbox enabled, running on a runner whose toolcache already contains a Copilot CLI version satisfying the compat range, with no explicit engine.version pinned.

Real failing run: Shazwazza/Examine run 32063259575 (gh-aw v0.85.4, firewall 0.27.44).

Setup step — cache hit, no /usr/local/bin/copilot created

Installing GitHub Copilot CLI (os: Linux, arch: x86_64)...
No explicit Copilot CLI version requested. Attempting compat-driven version resolution...
Compatibility matrix matched row 0: gh-aw 0.72.0..*, copilot 1.0.21..1.0.80
Resolved Copilot CLI version from compatibility matrix: 1.0.80
Searching toolcache for GitHub Copilot CLI (requested: latest, arch: x64, range: 1.0.21..1.0.80)...
  Found candidate: /opt/hostedtoolcache/copilot-cli/1.0.78/x64/bin/copilot (version: 1.0.78, arch: x64)
  Cache age: 13 days (within TTL of 14 days)
  Selected best cached version: 1.0.78
Using cached GitHub Copilot CLI from /opt/hostedtoolcache/copilot-cli/1.0.78/x64/bin/copilot
Activating cached Copilot CLI from /opt/hostedtoolcache/copilot-cli/1.0.78/x64/bin/copilot...
  Prepended /opt/hostedtoolcache/copilot-cli/1.0.78/x64/bin to PATH
  Exporting ... to GITHUB_PATH
✓ Copilot CLI installation complete (cached)

Agent step — instant ENOENT

[copilot-harness] starting: command=/usr/local/bin/copilot ...
[copilot-harness] pre-flight: command not found: /usr/local/bin/copilot (F_OK check failed — binary does not exist at this path)
[copilot-harness] attempt 1: failed to start process '/usr/local/bin/copilot': spawn /usr/local/bin/copilot ENOENT
[copilot-harness] attempt 1: no output produced — not retrying
[copilot-harness] done: exitCode=1 totalDuration=0s

Contrast — a fresh install works

An earlier run on gh-aw v0.80.9 had an explicit version compiled in (engine_versions: {"copilot":"1.0.63"}), which didn't match the cached 1.0.78, so it fell through to a fresh download that untars into /usr/local/bin:

Installing GitHub Copilot CLI version 1.0.63 (os: Linux, arch: x86_64)...
[copilot-harness] pre-flight: command is accessible and executable: /usr/local/bin/copilot
[copilot-harness] success on attempt 1: totalDuration=5m 55s

Root cause

github/gh-aw-actions → setup/sh/install_copilot_cli.sh, function activate_cached_copilot_bin:

activate_cached_copilot_bin() {
  local cached_copilot_bin="$1"
  ...
  export PATH="${cached_copilot_dir}:$PATH"
  echo "  Prepended ${cached_copilot_dir} to PATH"

  if [ -n "${GITHUB_PATH:-}" ]; then
    echo "  Exporting ${cached_copilot_dir} to GITHUB_PATH (${GITHUB_PATH})"
    echo "$cached_copilot_dir" >> "${GITHUB_PATH}"
    return 0                      # <-- returns without creating ${INSTALL_DIR}/copilot
  fi

  # Outside GitHub Actions there is no GITHUB_PATH file, so install a small wrapper
  # instead of symlinking or copying the cached script and risking broken relative paths.
  echo "  GITHUB_PATH not set — installing wrapper at ${INSTALL_DIR}/copilot"
  ...
  maybe_sudo install -m 0755 "$wrapper_path" "${INSTALL_DIR}/copilot"
}

GITHUB_PATH is always set under GitHub Actions, so the wrapper-install branch is effectively dead code in exactly the environment that needs it. Meanwhile the fresh-install path does create the file:

maybe_sudo tar -xz -C "${INSTALL_DIR}" -f "${TEMP_DIR}/${TARBALL_NAME}"
maybe_sudo chmod +x "${INSTALL_DIR}/copilot"

Adding the directory to GITHUB_PATH can't help here: the harness receives /usr/local/bin/copilot as a literal argv value and Node's spawn does an F_OK existence check on that exact path rather than a PATH lookup.

Why pinning engine.version isn't a general fix

In find_cached_copilot_bin, an explicitly requested version still returns a toolcache hit when it matches exactly:

if [ -n "$requested_version_normalized" ]; then
  if [ "$candidate_version_normalized" = "$requested_version_normalized" ]; then
    echo "  Exact version match found: ${candidate}" >&2
    printf '%s\n' "$candidate"
    return 0          # -> activate_cached_copilot_bin -> same bug
  fi
  echo "  Skipping candidate (version mismatch: ...)" >&2
  continue
fi

So pinning only avoids the bug while the pinned version differs from whatever is cached. Once that version lands in the toolcache, pinned workflows break too.

Suggested fix

Have the GITHUB_PATH branch also install the wrapper, so the canonical path exists regardless of how the CLI was provisioned:

  if [ -n "${GITHUB_PATH:-}" ]; then
    echo "  Exporting ${cached_copilot_dir} to GITHUB_PATH (${GITHUB_PATH})"
    echo "$cached_copilot_dir" >> "${GITHUB_PATH}"
    # fall through so ${INSTALL_DIR}/copilot is created too
  fi

  wrapper_path="${TEMP_DIR}/copilot"
  cat > "$wrapper_path" <<EOF
#!/usr/bin/env bash
exec "$cached_copilot_bin" "\$@"
EOF
  maybe_sudo install -m 0755 "$wrapper_path" "${INSTALL_DIR}/copilot"
  echo "  Wrapper installed at ${INSTALL_DIR}/copilot"

Alternatively, have the compiler emit the resolved binary path into the lock file instead of hardcoding /usr/local/bin/copilot, or have copilot_harness.cjs fall back to a PATH lookup when the literal path fails its pre-flight check. The wrapper approach looks smallest and matches the script's own stated preference for a wrapper over a symlink.

Affected versions

activate_cached_copilot_bin appears unchanged across v0.85.4, v0.86.2, v0.86.3, v0.87.0 and main, so upgrading doesn't avoid it.

Workaround (for anyone hitting this)

A pre-agent-steps entry in the workflow frontmatter recreates the missing wrapper:

pre-agent-steps:
  - name: Ensure copilot CLI is at /usr/local/bin
    run: |
      set -euo pipefail
      if [ -x /usr/local/bin/copilot ]; then exit 0; fi
      target="$(command -v copilot || true)"
      if [ -z "$target" ]; then echo "::error::copilot CLI not found on PATH"; exit 1; fi
      printf '#!/usr/bin/env bash\nexec "%s" "$@"\n' "$target" | sudo tee /usr/local/bin/copilot >/dev/null
      sudo chmod 0755 /usr/local/bin/copilot

I couldn't find an existing issue for this; apologies if it's a duplicate.

[pelikhan]

@copilot ensure that we have tests in github/gh-aw that use the tool cache in dev mode.