# agent mTLS origin test failed: presented the wrong client cert

## TL;DR

Your mTLS test failed because the client certificate that actually crossed the
wire is not the one you intended to send. Capture the presented certificate's
fingerprint on the server side (or with `openssl s_client -msg` on the client
side), compare it to the cert file you meant to use, verify the cert/key pair
actually matches with an openssl modulus comparison, and check the origin's
acceptable-CA list. In practice this is almost always one of three things: a
stale or rotated client cert, a cert file paired with the wrong private key, or
the client silently falling back to a default cert because its real cert's
issuer is not in the server's acceptable-CA list.

## The exact error

```text
agent mTLS origin test failed: presented the wrong client cert
```

This error string is produced by test harnesses and agents that compare the
certificate they *configured* for the test against the certificate the origin
*observed* in the TLS handshake. The handshake itself may have succeeded at the
TLS level (no fatal alert) while the identity check failed, or the server may
have aborted with a bad-certificate alert. Either way, the debugging target is
the same: find out exactly which certificate the client sent.

## Steps

### 1. Capture what the client actually sent

Do this on whichever side you control.

Server side (if you have the origin's logs or config):

- nginx: log `$ssl_client_s_dn` and `$ssl_client_fingerprint` for the failing
  request. Example log format addition:
  `log_format mtls '$ssl_client_s_dn $ssl_client_fingerprint';`
- Apache: `%{SSL_CLIENT_S_DN}x` and `%{SSL_CLIENT_I_DN}x` in the access log.
- Cloudflare (API Shield / Workers mTLS): inspect
  `request.cf.tlsClientAuth.certPresented` and
  `request.cf.tlsClientAuth.certVerified` in a debug Worker. `certPresented`
  of `1` with `certVerified` FAILED means the client sent something the zone
  does not trust.

Client side (reproduce the test and watch the wire):

```bash
openssl s_client -connect YOUR_ORIGIN:443 -servername YOUR_ORIGIN \
  -cert client.crt -key client.key -CAfile ca-chain.pem -msg -state
```

Expected output: after the `CertificateRequest` from the server, you see the
client's Certificate handshake message. Write down its subject; it must match
the certificate you intended to send. If no Certificate message appears, the
client sent *nothing* (skip to step 5).

### 2. Compare the presented fingerprint to the intended one

```bash
openssl x509 -in client.crt -noout -fingerprint -sha256 -subject
```

Expected output:

```text
sha256 Fingerprint=AA:BB:CC:...
subject=CN = test-client, OU = agents, O = example
```

Compare the fingerprint against what step 1 showed. Success check: the two
fingerprints are identical. If they differ, the test used a different file
than you think — a stale copy, a rotated cert, or a bundle file where another
cert sits first.

### 3. Verify the cert file matches the private key file

A cert paired with the wrong key produces a `CertificateVerify` failure
(TLS alert 51), which servers and test harnesses often report as a generic
wrong-cert failure.

```bash
openssl x509 -noout -modulus -in client.crt | openssl md5
openssl rsa  -noout -modulus -in client.key | openssl md5
```

Expected output: both commands print the same hash, for example
`(stdin)= 5f4dcc3b5aa765d61d8327deb882cf99`. Success check: the two hashes
match. If they differ, the cert and key are from different issuances —
re-pair them from the same issuance (or re-issue the client cert).

### 4. Verify the chain and validity of the cert you sent

```bash
openssl x509 -in client.crt -noout -subject -issuer -dates
openssl verify -CAfile ca-chain.pem client.crt
```

Expected output: `client.crt: OK`, `notBefore`/`notAfter` covering today, and
an issuer that is (or chains to) a CA your origin trusts. Success check: the
issuer chain terminates at one of the CAs configured on the origin
(nginx `ssl_client_certificate`, Apache `SSLCACertificateFile`, Envoy
`validation_context.trusted_ca`, AWS ALB trust-store). A valid cert signed by
the wrong CA is still "the wrong cert" from the origin's point of view.

### 5. Check the origin's acceptable-CA list

During the handshake the origin sends a `CertificateRequest` listing the
distinguished names of the CAs it accepts. Clients use this list to *choose*
which cert to send. If your cert's issuer is not on the list, clients pick a
different cert from their store (curl, Go, and the JVM all do some version of
this) or send nothing at all (JVM: "No X.509 certificate for client
authentication, use empty Certificate message instead").

How to check:

```bash
openssl s_client -connect YOUR_ORIGIN:443 -servername YOUR_ORIGIN \
  -CAfile ca-chain.pem -msg
```

Look for the "Acceptable client certificate CA names" section in the
CertificateRequest. Success check: your cert's issuer DN appears in that
list. If it does not, fix the origin's CA bundle (not the client) — no client
cert will ever be accepted until its issuer is listed.

### 6. Re-run the test with explicit files and confirm

```bash
curl -v --cert client.crt --key client.key --cacert ca-chain.pem \
  https://YOUR_ORIGIN/healthz
```

Expected output: curl shows `* SSL connection using TLSv1.3` with
`SSL certificate verify ok`, and your server-side log (step 1) now shows the
intended fingerprint. Success check: the test harness no longer reports a
mismatched cert.

## Use this skill when

- An mTLS test or agent probe reports the client presented the wrong (or
  unexpected) client certificate.
- The handshake completes but identity checks on the presented cert fail.
- You see TLS alert 42/48 (server does not trust the client cert's issuer) or
  alert 51 (CertificateVerify failed) and the cert "looks fine".

## Do NOT use this skill when

- The client certificate is **expired** — that is a renewal problem, not a
  wrong-cert problem.
- The *server* certificate is untrusted (`unable to get local issuer
  certificate`, curl error 60) — that is a server-chain/trust-store problem.
- The mTLS setup never worked at all — start from the basic mTLS
  configuration before debugging cert selection.

## Variant phrasings

- "mTLS handshake failed: server got the wrong client certificate"
- "client presented a different certificate than configured"
- "tls: peer did not return a certificate" (the extreme case: the client sent
  nothing because no cert matched the acceptable-CA list)
- "SSL client certificate verification failed: wrong DN"
- "origin rejected the client cert: identity mismatch"

## Why it happens

mTLS has two independent selection steps and both can pick "wrong". On the
client, selection picks which cert to send — governed by the server's
acceptable-CA list, the client's cert store contents, and which cert/key file
the test actually pointed at. On the server, validation checks the presented
cert against its trust store. "Presented the wrong client cert" means the
*selection* step produced a cert that is valid TLS but the wrong identity:
a stale copy from a rotation, a cert/key mismatch, a bundle file where the
first cert wins (a real CVE class — one mTLS implementation acted on the last
cert in the file instead of the first), or a fallback default because the
intended cert's CA was missing from the server's list. The fix is always the
same shape: observe what was actually sent, then walk back to *why* the
client chose it.

## Edge cases and pitfalls

- **Bundle files with multiple certs.** If the client cert file contains more
  than one certificate, some clients use the first and some the last. Split
  the file (`csplit`) or re-export a single-cert file to remove ambiguity.
- **Rotated certs with the same filename.** A rotation that wrote a new cert
  but left the old key (or vice versa) fails step 3. Always rotate cert and
  key as an atomic pair.
- **TLS 1.3 post-handshake authentication.** The client cert can arrive after
  the handshake; packet captures that stop at the handshake miss it. Use
  server-side logging instead.
- **Test vs production identity.** Test harnesses often run with a
  `test-client` cert while the origin expects the production DN. The cert is
  "wrong" relative to the environment, not broken.
- **Clock skew.** A cert that is not-yet-valid because the client clock is
  ahead produces verification failures that look like wrong-cert rejections.
  Check `date -u` against the cert's `notBefore`.
- **Middlebox interception.** A TLS-inspecting proxy between the test client
  and the origin can substitute its own certs. If the presented cert's issuer
  is your organization's proxy CA, the problem is the network path, not the
  cert files.