OCSP response is not stapled when using 'openssl s_client'

### Detailed Description of the Problem

HAProxy successfully gets the OCSP response from the OCSP responder server:

```
echo "show ssl ocsp-response 303b300906052b0e03021a05000414fb1f13ee9b62ff5ce28500bccb28584007d615240414200df0756fa74cafb9421a42a6c1c965532fc36902021001" | sudo sudo socat stdio unix-connect:/run/haproxy/admin.sock

OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, ST = Ohio, O = Example Corp, OU = IT Department, CN = ocsp.example.com
    Produced At: Dec 17 19:06:10 2024 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: FB1F13EE9B62FF5CE28500BCCB28584007D61524
      Issuer Key Hash: 200DF0756FA74CAFB9421A42A6C1C965532FC369
      Serial Number: 1001
    Cert Status: good
    This Update: Dec 17 19:06:10 2024 GMT
    Next Update: Dec 17 19:11:10 2024 GMT
```


But when I use `openssl s_client` to make a request, HAProxy isn't stapling it. It shows **OCSP response: no response sent**.

```
openssl s_client -connect www.example.com:443 -servername www.example.com -status </dev/null

CONNECTED(00000003)
depth=1 C = US, ST = Ohio, L = Columbus, O = Example Corp, OU = IT Department, CN = Root CA
verify return:1
depth=0 C = US, ST = Ohio, O = Example Corp, OU = IT Department, CN = www.example.com
verify return:1
OCSP response: no response sent
```


### Expected Behavior

HAProxy should return the OCSP response to the client and it should appear in the output of `openssl s_client`.

### Steps to Reproduce the Behavior

To validate your OCSP stapling setup, you can use `openssl` commands to run an OCSP Responder server in your QA environment. The OCSP Responder server simulates returning OCSP responses. It also functions as a Certificate Authority (CA), issuing TLS certificates that you would install onto your QA load balancer and later revoke to see OCSP stapling in action. Deploy a Linux server in the same network as your QA load balancer.

This procedure uses Debian 12 (Bookworm) as the operating system for the OCSP Responder server. 

After creating the server, perform these tasks on it:

1. We rely on the `openssl` command-line utility. To check that it's installed, run:

   ```nix
   openssl version
   ```

   ```text title="output"
   OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
   ```

1. This server will function as a CA that generates TLS certificates, private keys, CRLs, and CSRs. Create the directory structure for storing these different types of files:

   ```nix
   sudo mkdir -p /exampleCA/rootCA/{certs,newcerts,crl,private,csr}
   echo 1000 | sudo tee /exampleCA/rootCA/serial
   echo 0100 | sudo tee /exampleCA/rootCA/crlnumber 
   sudo touch /exampleCA/rootCA/index.txt
   sudo touch /exampleCA/openssl-root.cnf
   ```

1. Edit the file `/exampleCA/openssl-root.cnf` and add the following content, which defines the OpenSSL settings we need.

   Change the line `authorityInfoAccess = OCSP;URI:http://192.168.56.39:8888` to use your OCSP Responder server's IP address. This will be embedded into every TLS server certificate as a call address for making OCSP queries.

   ```text title="openssl-root.cnf"
   [ ca ]                                                   # The default CA section
   default_ca = CA_default                                  # The default CA name
   
   [ CA_default ]                                           # Default settings for the CA
   dir               = /exampleCA/rootCA                    # CA directory
   certs             = $dir/certs                           # Certificates directory
   crl_dir           = $dir/crl                             # CRL directory
   new_certs_dir     = $dir/newcerts                        # New certificates directory
   database          = $dir/index.txt                       # Certificate index file
   serial            = $dir/serial                          # Serial number file
   RANDFILE          = $dir/private/.rand                   # Random number file
   private_key       = $dir/private/rootCA.key              # Root CA private key
   certificate       = $dir/certs/rootCA.crt                # Root CA certificate
   crl               = $dir/crl/crl.pem                     # Root CA CRL
   crlnumber         = $dir/crlnumber                       # Root CA CRL number
   crl_extensions    = crl_ext                              # CRL extensions
   default_crl_days  = 30                                   # Default CRL validity days
   default_md        = sha256                               # Default message digest
   preserve          = no                                   # Preserve existing extensions
   email_in_dn       = no                                   # Exclude email from the DN
   name_opt          = ca_default                           # Formatting options for names
   cert_opt          = ca_default                           # Certificate output options
   policy            = policy_strict                        # Certificate policy
   unique_subject    = no                                   # Allow multiple certs with the same DN
   
   [ policy_strict ]                                        # Policy for stricter validation
   countryName             = match                          # Must match the issuer's country
   stateOrProvinceName     = match                          # Must match the issuer's state
   organizationName        = match                          # Must match the issuer's organization
   organizationalUnitName  = optional                       # Organizational unit is optional
   commonName              = supplied                       # Must provide a common name
   emailAddress            = optional                       # Email address is optional
   
   [ req ]                                                  # Request settings
   default_bits        = 2048                               # Default key size
   distinguished_name  = req_distinguished_name             # Default DN template
   string_mask         = utf8only                           # UTF-8 encoding
   default_md          = sha256                             # Default message digest
   prompt              = no                                 # Non-interactive mode
   
   [ req_distinguished_name ]                               # Template for the DN in the CSR
   countryName                     = Country Name (2 letter code)
   stateOrProvinceName             = State or Province Name (full name)
   localityName                    = Locality Name (city)
   0.organizationName              = Organization Name (company)
   organizationalUnitName          = Organizational Unit Name (section)
   commonName                      = Common Name (your domain)
   emailAddress                    = Email Address
   
   [ v3_ca ]                                           # Root CA certificate extensions
   subjectKeyIdentifier = hash                         # Subject key identifier
   authorityKeyIdentifier = keyid:always,issuer        # Authority key identifier
   basicConstraints = critical, CA:true                # Basic constraints for a CA
   keyUsage = critical, keyCertSign, cRLSign           # Key usage for a CA
   
   [ crl_ext ]                                         # CRL extensions
   authorityKeyIdentifier = keyid:always,issuer        # Authority key identifier
   
   [ v3_intermediate_ca ]
   subjectKeyIdentifier = hash
   authorityKeyIdentifier = keyid:always,issuer
   basicConstraints = critical, CA:true, pathlen:0
   keyUsage = critical, digitalSignature, cRLSign, keyCertSign
   
   [ v3_OCSP ]
   basicConstraints = CA:FALSE
   keyUsage = nonRepudiation, digitalSignature, keyEncipherment
   extendedKeyUsage = OCSPSigning
   
   [ server_cert ]
   basicConstraints = CA:false
   nsCertType = server
   subjectKeyIdentifier = hash
   authorityKeyIdentifier = keyid,issuer:always
   keyUsage = critical, digitalSignature, keyEncipherment
   extendedKeyUsage = serverAuth
   authorityInfoAccess = OCSP;URI:http://192.168.56.39:8888
   ```

1. Generate a root certificate and private key for the CA:

   ```nix
   sudo openssl genrsa -out /exampleCA/rootCA/private/rootCA.key 4096
   
   sudo openssl req \
     -config /exampleCA/openssl-root.cnf \
     -extensions v3_ca \
     -new \
     -x509 \
     -days 7300 \
     -sha256 \
     -subj "/C=US/ST=Ohio/L=Columbus/O=Example Corp/OU=IT Department/CN=Root CA" \
     -key /exampleCA/rootCA/private/rootCA.key \
     -out /exampleCA/rootCA/certs/rootCA.crt
   ```

1. Add this root certificate to the server's list of CA certificates so that it becomes trusted.

   ```nix
   sudo mkdir /usr/local/share/ca-certificates/extra
   sudo cp /exampleCA/rootCA/certs/rootCA.crt /usr/local/share/ca-certificates/extra/
   sudo update-ca-certificates
   ```

   ```text title="output"
   Updating certificates in /etc/ssl/certs...
   1 added, 0 removed; done.
   Running hooks in /etc/ca-certificates/update.d...
   done.
   ```

1. Create a private key and a CSR that you'll use to sign OCSP responses. This will prompt you to enter a passphrase.

   ```nix
   sudo openssl req \
     -config /exampleCA/openssl-root.cnf \
     -new \
     -sha256 \
     -subj "/C=US/ST=Ohio/L=Columbus/O=Example Corp/OU=IT Department/CN=ocsp.example.com" \
     -keyout /exampleCA/rootCA/private/ocsp.example.com.key \
     -out /exampleCA/rootCA/csr/ocsp.example.com.csr
   ```

1. Generate a certificate named `ocsp.example.com.crt` from the CSR. This will prompt you to sign the certificate and commit it to the certificates database.

   ```nix
   sudo openssl ca \
     -config /exampleCA/openssl-root.cnf \
     -extensions v3_OCSP \
     -days 375 \
     -notext \
     -md sha256 \
     -in /exampleCA/rootCA/csr/ocsp.example.com.csr \
     -out /exampleCA/rootCA/certs/ocsp.example.com.crt
      ```

   ```text title="output"
   Sign the certificate? [y/n]:y

   1 out of 1 certificate requests certified, commit? [y/n]y
   Write out database with 1 new entries
   Database updated
   ```

1. Start the OCSP server. This will prompt you to enter your passphrase.

   ```nix
   sudo openssl ocsp \
     -port 8888 \
     -index /exampleCA/rootCA/index.txt \
     -rsigner /exampleCA/rootCA/certs/ocsp.example.com.crt \
     -rkey /exampleCA/rootCA/private/ocsp.example.com.key \
     -CA /exampleCA/rootCA/certs/rootCA.crt \
     -text \
     -nmin 5
   ```

   ```text title="output"
   ACCEPT 0.0.0.0:8888 PID=3241
   Enter pass phrase for /exampleCA/rootCA/private/ocsp.example.com.key:
   ocsp: waiting for OCSP client connections...
   ```

1. In another terminal window, create a TLS certificate named `www.example.com.crt` for your load balancer. This will prompt you to sign the certificate and commit it to the certificates database.

   ```nix
   sudo openssl req \
     -config /exampleCA/openssl-root.cnf \
     -newkey rsa:2048 \
     -nodes \
     -subj "/C=US/ST=Ohio/L=Columbus/O=Example Corp/OU=IT Department/CN=www.example.com" \
     -keyout /exampleCA/rootCA/private/www.example.com.key \
     -out /exampleCA/rootCA/csr/www.example.com.csr

   sudo openssl ca \
     -config /exampleCA/openssl-root.cnf \
     -extensions server_cert \
     -days 3650 \
     -notext \
     -md sha256 \
     -in /exampleCA/rootCA/csr/www.example.com.csr \
     -out /exampleCA/rootCA/certs/www.example.com.crt
   ```

   ```text title="output"
   Sign the certificate? [y/n]:y

   1 out of 1 certificate requests certified, commit? [y/n]y
   Write out database with 1 new entries
   Database updated
   ```

1. At this point, you can query the OCSP Responder server with this certificate. It should return a good status.

   ```nix
   openssl ocsp \
     -issuer /exampleCA/rootCA/certs/rootCA.crt \
     -cert /exampleCA/rootCA/certs/www.example.com.crt \
     -url http://192.168.56.39:8888 \
     -resp_text
   ```

   ```text title="output" {12}
   OCSP Response Data:
     OCSP Response Status: successful (0x0)
     Response Type: Basic OCSP Response
     Version: 1 (0x0)
     Responder Id: C = US, ST = Ohio, O = Example Corp, OU = IT Department, CN = ocsp.example.com
     Produced At: Dec 17 15:07:20 2024 GMT
     Responses:
     Certificate ID:
       Hash Algorithm: sha1
       Issuer Name Hash: FB1F13EE9B62FF5CE28500BCCB28584007D61524
       Issuer Key Hash: 200DF0756FA74CAFB9421A42A6C1C965532FC369
       Serial Number: 1001
     Cert Status: good
     This Update: Dec 17 15:07:20 2024 GMT
     Next Update: Dec 17 15:12:20 2024 GMT
   ```

1. Copy the contents of the `www.example.com.crt` certificate, the `www.example.com.key` private key, and the CA root certificate into a single PEM file to be used on your load balancer:

   ```nix
   sudo cat /exampleCA/rootCA/certs/www.example.com.crt > ~/www.example.com.pem
   sudo cat /exampleCA/rootCA/private/www.example.com.key >> ~/www.example.com.pem
   sudo cat /exampleCA/rootCA/certs/rootCA.crt >> ~/www.example.com.pem
   ```

1. Copy these files to your home directory on your QA load balancer:

   * `~/www.example.com.pem`
   * `/exampleCA/rootCA/certs/rootCA.crt`

1. On your load balancer, add the root certificate to the server's list of CA certificates so that it becomes trusted.

   ```nix
   sudo mkdir /usr/local/share/ca-certificates/extra
   sudo cp ~/rootCA.crt /usr/local/share/ca-certificates/extra/
   sudo update-ca-certificates
   ```

   ```text title="output"
   Updating certificates in /etc/ssl/certs...
   1 added, 0 removed; done.
   Running hooks in /etc/ca-certificates/update.d...
   done.
   ```

1. Copy the `www.example.com.pem` file to your `certs` directory. For example:

   ```nix
   sudo cp ~/www.example.com.pem /etc/haproxy/certs/
   ```

1. Update your `/etc/haproxy/certs/crt-list` file to use this certificate and enable OCSP stapling.

   ```text title="crt-list.txt"
   /etc/haproxy/certs/www.example.com.pem [ocsp-update on]
   ```

1. Update your frontend to use the `crt-list`.

   ```haproxy
   frontend www
     bind :443 ssl crt-list /etc/haproxy/certs/crt-list.txt
     default_backend webservers
   ```

1. Reload the load balancer configuration.

   ```nix
   sudo systemctl reload haproxy-lb
   ```

1. At this point, your load balancer logs should show that the OCSP updates are completing successfully.

   ```nix
   sudo journalctl -u haproxy
   ```

   ```text title="output"
   <OCSP-UPDATE> /etc/haproxy/certs/www.example.com.pem 1 "Update successful" 0 1
   ```

1. To map the domain `www.example.com` to your load balancer's IP address, edit the server's `/etc/hosts` file:

   ```text title="hosts"
   127.0.0.1 www.example.com
   ```

1. Test that the load balancer is stapling the OCSP response:

   ```nix
   openssl s_client -connect www.example.com:443 -servername www.example.com -status </dev/null
   ```

### Do you have any idea what may have caused this?

No, but I've used this log format to ensure that the SNI is being sent to HAProxy:

```
mode tcp
  tcp-request inspect-delay 5s
  tcp-request content set-var(txn.ssl_sni) ssl_fc_sni
  tcp-request content set-var(txn.has_sni) str(true) if { ssl_fc_has_sni -m bool }
  tcp-request content set-var(txn.has_sni) str(false) if !{ ssl_fc_has_sni -m bool }
  log-format "HAS_SNI=%[var(txn.has_sni)]  SNI=%[var(txn.ssl_sni)]"
```

Log shows:

```
HAS_SNI=true SNI=www.example.com
```

### Do you have an idea how to solve the issue?

_No response_

### What is your configuration?

```haproxy
global
	log /dev/log	local0
	log /dev/log	local1 debug
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

defaults
	log	global
	mode	http
	option	httplog
    timeout connect 5000
    timeout client  50000
    timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

frontend example
  bind ipv4@:443 ssl crt-list /etc/haproxy/crt-list.txt
  default_backend servers

  mode tcp
  tcp-request inspect-delay 5s
  tcp-request content set-var(txn.ssl_sni) ssl_fc_sni
  tcp-request content set-var(txn.has_sni) str(true) if { ssl_fc_has_sni -m bool }
  tcp-request content set-var(txn.has_sni) str(false) if !{ ssl_fc_has_sni -m bool }
  log-format "HAS_SNI=%[var(txn.has_sni)]  SNI=%[var(txn.ssl_sni)]"


backend servers
  mode tcp
  server web1 192.168.56.41:80 check
```


### Output of `haproxy -vv`

```plain
HAProxy version 3.1.1-1~bpo12+1 2024/12/14 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-3.1.1.html
Running on: Linux 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64
Build options :
  TARGET  = linux-glibc
  CC      = x86_64-linux-gnu-gcc
  CFLAGS  = -O2 -g -fwrapv -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_OT=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 3.0.15 3 Sep 2024
Running on OpenSSL version : OpenSSL 3.0.11 19 Sep 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [  OT] opentracing
        [SPOE] spoe
        [TRACE] trace
```


### Last Outputs and Backtraces

_No response_

### Additional Information

_No response_

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCSP response is not stapled when using 'openssl s_client' #2822

Detailed Description of the Problem

Expected Behavior

Steps to Reproduce the Behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

What is your configuration?

Output of `haproxy -vv`

Last Outputs and Backtraces

Additional Information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development