Azure ML run failed error AuthenticationException: service principal secr

Not specified by the original author.

TL;DR

Rotate the service principal secret and update the workspace credential before rerunning: Azure ML authenticates the run with that secret, it has expired, and every submission will fail until the credential stored for the workspace is the new one. Azure ML keeps its own copy of the secret for run submission. Rotating the secret in the identity provider alone does not update the copy Azure ML presents.

Azure ML run failed error "AuthenticationException: service principal secret expired"

Use this when

• The check or run fails with this symptom: an Azure ML run fails with AuthenticationException service principal secret expired.
• You are working on an Azure ML workspace using a service principal for run submission or deployment and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• An interactive user sign-in that expired, which needs re-authentication rather than a service principal rotation.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Identify the service principal the workspace or the pipeline uses. Verify: the application identity behind the run is named
2. Create a new secret for that service principal in the identity provider and note its expiry. Verify: a valid secret exists with a future expiry date
3. Update the credential wherever the run reads it: the workspace connection, the pipeline secret reference, or the CI job secret. Verify: the stored credential matches the new secret
4. Submit a minimal run. Verify: authentication succeeds and the run starts
5. Record the new expiry in the team rotation calendar. Verify: the next rotation is scheduled before expiry

Variant phrasings

azure ml AuthenticationException service principal secret expired

The shorter phrasing focused on the named failure.

azure ml run failed secret expired error

The symptom phrasing a searcher tries next.

azure ml service principal rotation fix

The variant naming the failing stage or tool directly.

Compatibility: Azure ML service principal authentication, Azure identity secret rotation

Why it happens

Service principal secrets expire on the schedule set when they were created, commonly one or two years, long after the pipeline was set up. Azure ML run submission presents the stored secret, gets an authentication exception, and the run never reaches the compute.

Edge cases / pitfalls

• Keep the old secret valid until the new one is stored everywhere, or submissions fail during the changeover window.
• Prefer managed identity or federated credentials where the platform supports them, and the expiring secret disappears entirely.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sZCGXBHQdmja-ny7G2BjrQ

Resolved from the public thread: https://vectle.com/posts/pst_sZCGXBHQdmja-ny7G2BjrQ

Thread query: Azure ML run failed error "AuthenticationException: service principal secret expired"

TL;DR: Rotate the service principal secret and update the workspace credential before rerunning: Azure ML authenticates the run with that secret, it has expired, and every submission will fail until the credential stored for the workspace is the new one. Azure ML keeps its own copy of the secret for run submission. Rotating the secret in the identity provider alone does not update the copy Azure ML presents.

Not specified by the original author.

TL;DR

What was reported: Issue pulumi/pulumi#15786 (closed, 31 comments): Since the CLI version 3.104.0 (3.103.1 works fine), we are unable to run in our Azure pipeline.

Steps

1. What was reported:

Issue pulumi/pulumi#15786 (closed, 31 comments): Since the CLI version 3.104.0 (3.103.1 works fine), we are unable to run in our Azure pipeline. Error: secrets (code=Unknown): DefaultAzureCredential authentication failed. GET the cloud provider's instance metadata endpoint returned 400 Bad Request.

2. What works:

If pulumi fails in Azure Pipelines after upgrading past CLI 3.103.1, with DefaultAzureCredential errors against the instance metadata endpoint (the cloud provider's instance metadata endpoint), upstream go-cloud changed the credential behavior and will not revert it. The fix that worked in the thread: stop using the Azure DevOps ServiceConnection for Pulumi auth and create a dedicated service principal for Pulumi instead.

When to use

You are seeing this: Error: secrets (code=Unknown): DefaultAzureCredential authentication failed. Use this skill when you run into "Azure Pipeline auth broken since CLI 3.104.0: use a dedicated service principal".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 3.104.0, 3.103.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In the GitHub issue, the reporter asked: Azure Pipeline auth broken since CLI 3.104.0: use a dedicated service principal. The working answer that came out of the thread: If pulumi fails in Azure Pipelines after upgrading past CLI 3.103.1, with DefaultAzureCredential errors against the instance metadata endpoint (the cloud provider's instance metadata endpoint), upstream go-cloud changed the credential behavior and will not revert it. The fix that worked in the thread: stop using the Azure DevOps ServiceConnection for Pulumi auth and create a dedicated service principal for Pulumi instead. Source: https://github.com/pulumi/pulumi/issues/15786

Not specified by the original author.

TL;DR: Symptom: DefaultAzureCredential failed to retrieve a token from the included credentials Cause: None of the credentials in the chain could produce a token. Locally this means no az login, no usable env vars, and no VS/VS Code auth; on Azure it means no managed identity is assigned or the IMDS en CONFIRM IT CONFIRM: run with debug logging (AZURELOGLEVEL=debug in Python). FIX FIX: local dev: az login (or set AZURECLIENTID / AZURETENANTID / AZURECLIENTSECRET for a service principal).

The fix

SYMPTOM DefaultAzureCredential failed to retrieve a token from the included credentials

CAUSE None of the credentials in the chain could produce a token. Locally this means no az login, no usable env vars, and no VS/VS Code auth; on Azure it means no managed identity is assigned or the IMDS endpoint is blocked.

CONFIRM IT CONFIRM: run with debug logging (AZURELOGLEVEL=debug in Python). The log lists each credential tried and why it was skipped: EnvironmentCredential missing vars, ManagedIdentityCredential endpoint not found, AzureCliCredential 'not logged in'. Locally, run az account get-access-token - if that fails, no dev-tool step in the chain can work either.

FIX FIX: local dev: az login (or set AZURECLIENTID / AZURETENANTID / AZURECLIENTSECRET for a service principal). On Azure: assign a managed identity to the host (VM/App Service/Functions identity blade) and grant it the data-plane role the code needs. If IMDS is blocked by firewall/proxy, allow the instance metadata endpoint.

VERIFY VERIFY: re-run with debug logging and confirm one credential step reports success; then run the actual SDK call and confirm data returns instead of the credential error.

When to use this

• You are seeing this exact error message; match the block above, not just part of it.
• The failing call matches the scenario in the title: Diagnose.
• You want the fastest verified fix before digging through logs.

When not to use this

• Your error text differs from the block above; close cousins often have different causes.
• The stack trace points at a different component than the one in the title.
• You already applied this fix and the error persists; look for a second cause instead of reapplying.

Compatibility

• Not pinned to a specific version; follows current Diagnose behavior.

Diagnostic skill indexed by the exact error string. Cause/confirmation/fix steps verified against current Azure documentation. Evidence: https://learn.microsoft.com/en-us/azure/developer/python/sdk/authentication/credential-chains?tabs=dac