agent applied the wrong conditional access policy fix: policy id collision in search results
This skill prevents a helpdesk agent from editing the wrong conditional access policy when two policies share similar names. It shows how to resolve policies by their immutable policy id instead of display name before any change. For agents that diagnose or remediate Entra conditional access policies.
TL;DR
Display names are not unique, so the agent must resolve every policy to its immutable id before acting. Search by name, require an exact single match or ask for disambiguation, pin the id for all later calls, and verify the id on every write.
Steps
Change the policy lookup to return both display name and policy id for every match. Expected: The agent sees ids alongside names.
If the search returns more than one policy, stop and ask which one, showing id and last-modified date. Expected: Ambiguity is resolved before any change.
Carry the chosen policy id through every read and write call for the rest of the run. Expected: No later step re-resolves by name.
Log the policy id with every change the agent makes. Expected: The audit trail names the exact policy edited.
When to use
- The agent edits conditional access policies
- The tenant has similarly named policies across environments
- A past run changed the wrong policy
When not to use
- The agent only reads policies for reporting
- Policy names are guaranteed unique by your naming standard
- Changes go through a human approval step that catches the wrong target
Why this happens
Conditional access policy display names are free text and often duplicated across test and production or regions. Name-based lookups silently pick the first match, so the agent edits a policy it never meant to touch.
Edge cases
- Deleted policies can leave name ghosts in cached listings; always refresh before resolving.
- Graph search is substring-based, so prod and prod-pilot both match a search for prod.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstZJ6bZYsK4h1iUwrtrmTfw