agent mTLS origin test failed: presented the wrong client cert

agent mTLS origin test failed: presented the wrong client cert

TL;DR

Your mTLS test failed because the client certificate that actually crossed the wire is not the one you intended to send. Capture the presented certificate's fingerprint on the server side (or with openssl s_client -msg on the client side), compare it to the cert file you meant to use, verify the cert/key pair actually matches with an openssl modulus comparison, and check the origin's acceptable-CA list. In practice this is almost always one of three things: a stale or rotated client cert, a cert file paired with the wrong private key, or the client silently falling back to a default cert because its real cert's issuer is not in the server's acceptable-CA list.

The exact error

agent mTLS origin test failed: presented the wrong client cert

This error string is produced by test harnesses and agents that compare the certificate they configured for the test against the certificate the origin observed in the TLS handshake. The handshake itself may have succeeded at the TLS level (no fatal alert) while the identity check failed, or the server may have aborted with a bad-certificate alert. Either way, the debugging target is the same: find out exactly which certificate the client sent.

Steps

1. Capture what the client actually sent

Do this on whichever side you control.

Server side (if you have the origin's logs or config):

• nginx: log $ssl_client_s_dn and $ssl_client_fingerprint for the failing request. Example log format addition: log_format mtls '$ssl_client_s_dn $ssl_client_fingerprint';

• Apache: %{SSL_CLIENT_S_DN}x and %{SSL_CLIENT_I_DN}x in the access log.

• Cloudflare (API Shield / Workers mTLS): inspect request.cf.tlsClientAuth.certPresented and request.cf.tlsClientAuth.certVerified in a debug Worker. certPresented of 1 with certVerified FAILED means the client sent something the zone does not trust.

Client side (reproduce the test and watch the wire):

openssl s_client -connect YOUR_ORIGIN:443 -servername YOUR_ORIGIN \
  -cert client.crt -key client.key -CAfile ca-chain.pem -msg -state

Expected output: after the CertificateRequest from the server, you see the client's Certificate handshake message. Write down its subject; it must match the certificate you intended to send. If no Certificate message appears, the client sent nothing (skip to step 5).

2. Compare the presented fingerprint to the intended one

openssl x509 -in client.crt -noout -fingerprint -sha256 -subject

Expected output:

sha256 Fingerprint=AA:BB:CC:...
subject=CN = test-client, OU = agents, O = example

Compare the fingerprint against what step 1 showed. Success check: the two fingerprints are identical. If they differ, the test used a different file than you think — a stale copy, a rotated cert, or a bundle file where another cert sits first.

3. Verify the cert file matches the private key file

A cert paired with the wrong key produces a CertificateVerify failure (TLS alert 51), which servers and test harnesses often report as a generic wrong-cert failure.

openssl x509 -noout -modulus -in client.crt | openssl md5
openssl rsa  -noout -modulus -in client.key | openssl md5

Expected output: both commands print the same hash, for example (stdin)= 5f4dcc3b5aa765d61d8327deb882cf99. Success check: the two hashes match. If they differ, the cert and key are from different issuances — re-pair them from the same issuance (or re-issue the client cert).

4. Verify the chain and validity of the cert you sent

openssl x509 -in client.crt -noout -subject -issuer -dates
openssl verify -CAfile ca-chain.pem client.crt

Expected output: client.crt: OK, notBefore/notAfter covering today, and an issuer that is (or chains to) a CA your origin trusts. Success check: the issuer chain terminates at one of the CAs configured on the origin (nginx ssl_client_certificate, Apache SSLCACertificateFile, Envoy validation_context.trusted_ca, AWS ALB trust-store). A valid cert signed by the wrong CA is still "the wrong cert" from the origin's point of view.

5. Check the origin's acceptable-CA list

During the handshake the origin sends a CertificateRequest listing the distinguished names of the CAs it accepts. Clients use this list to choose which cert to send. If your cert's issuer is not on the list, clients pick a different cert from their store (curl, Go, and the JVM all do some version of this) or send nothing at all (JVM: "No X.509 certificate for client authentication, use empty Certificate message instead").

How to check:

openssl s_client -connect YOUR_ORIGIN:443 -servername YOUR_ORIGIN \
  -CAfile ca-chain.pem -msg

Look for the "Acceptable client certificate CA names" section in the CertificateRequest. Success check: your cert's issuer DN appears in that list. If it does not, fix the origin's CA bundle (not the client) — no client cert will ever be accepted until its issuer is listed.

6. Re-run the test with explicit files and confirm

curl -v --cert client.crt --key client.key --cacert ca-chain.pem \
  https://YOUR_ORIGIN/healthz

Expected output: curl shows * SSL connection using TLSv1.3 with SSL certificate verify ok, and your server-side log (step 1) now shows the intended fingerprint. Success check: the test harness no longer reports a mismatched cert.

Use this skill when

• An mTLS test or agent probe reports the client presented the wrong (or unexpected) client certificate.

• The handshake completes but identity checks on the presented cert fail.

• You see TLS alert 42/48 (server does not trust the client cert's issuer) or alert 51 (CertificateVerify failed) and the cert "looks fine".

Do NOT use this skill when

• The client certificate is expired — that is a renewal problem, not a wrong-cert problem.

• The server certificate is untrusted (unable to get local issuer certificate, curl error 60) — that is a server-chain/trust-store problem.

• The mTLS setup never worked at all — start from the basic mTLS configuration before debugging cert selection.

Variant phrasings

• "mTLS handshake failed: server got the wrong client certificate"
• "client presented a different certificate than configured"

• "tls: peer did not return a certificate" (the extreme case: the client sent nothing because no cert matched the acceptable-CA list)

• "SSL client certificate verification failed: wrong DN"
• "origin rejected the client cert: identity mismatch"

Why it happens

mTLS has two independent selection steps and both can pick "wrong". On the client, selection picks which cert to send — governed by the server's acceptable-CA list, the client's cert store contents, and which cert/key file the test actually pointed at. On the server, validation checks the presented cert against its trust store. "Presented the wrong client cert" means the selection step produced a cert that is valid TLS but the wrong identity: a stale copy from a rotation, a cert/key mismatch, a bundle file where the first cert wins (a real CVE class — one mTLS implementation acted on the last cert in the file instead of the first), or a fallback default because the intended cert's CA was missing from the server's list. The fix is always the same shape: observe what was actually sent, then walk back to why the client chose it.

Edge cases and pitfalls

• Bundle files with multiple certs. If the client cert file contains more than one certificate, some clients use the first and some the last. Split the file (csplit) or re-export a single-cert file to remove ambiguity.

• Rotated certs with the same filename. A rotation that wrote a new cert but left the old key (or vice versa) fails step 3. Always rotate cert and key as an atomic pair.

• TLS 1.3 post-handshake authentication. The client cert can arrive after the handshake; packet captures that stop at the handshake miss it. Use server-side logging instead.

• Test vs production identity. Test harnesses often run with a test-client cert while the origin expects the production DN. The cert is "wrong" relative to the environment, not broken.

• Clock skew. A cert that is not-yet-valid because the client clock is ahead produces verification failures that look like wrong-cert rejections. Check date -u against the cert's notBefore.

• Middlebox interception. A TLS-inspecting proxy between the test client and the origin can substitute its own certs. If the presented cert's issuer is your organization's proxy CA, the problem is the network path, not the cert files.