Azure Pipeline auth broken since CLI 3.104.0: use a dedicated service principal

TL;DR

What was reported: Issue pulumi/pulumi#15786 (closed, 31 comments): Since the CLI version 3.104.0 (3.103.1 works fine), we are unable to run in our Azure pipeline.

Steps

1. What was reported:

Issue pulumi/pulumi#15786 (closed, 31 comments): Since the CLI version 3.104.0 (3.103.1 works fine), we are unable to run in our Azure pipeline. Error: secrets (code=Unknown): DefaultAzureCredential authentication failed. GET the cloud provider's instance metadata endpoint returned 400 Bad Request.

2. What works:

If pulumi fails in Azure Pipelines after upgrading past CLI 3.103.1, with DefaultAzureCredential errors against the instance metadata endpoint (the cloud provider's instance metadata endpoint), upstream go-cloud changed the credential behavior and will not revert it. The fix that worked in the thread: stop using the Azure DevOps ServiceConnection for Pulumi auth and create a dedicated service principal for Pulumi instead.

When to use

You are seeing this: Error: secrets (code=Unknown): DefaultAzureCredential authentication failed. Use this skill when you run into "Azure Pipeline auth broken since CLI 3.104.0: use a dedicated service principal".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 3.104.0, 3.103.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In the GitHub issue, the reporter asked: Azure Pipeline auth broken since CLI 3.104.0: use a dedicated service principal. The working answer that came out of the thread: If pulumi fails in Azure Pipelines after upgrading past CLI 3.103.1, with DefaultAzureCredential errors against the instance metadata endpoint (the cloud provider's instance metadata endpoint), upstream go-cloud changed the credential behavior and will not revert it. The fix that worked in the thread: stop using the Azure DevOps ServiceConnection for Pulumi auth and create a dedicated service principal for Pulumi instead. Source: https://github.com/pulumi/pulumi/issues/15786