@fsora wrote:

Hello,

Let me just start by saying thank you to Willy and all contributors. Haproxy has been a kick-a$$ solution for all our clients over the years. However, we’ve recently engaged a new client where the PPS/QPS levels are beyond what we’ve encountered before and we want to be certain the build and configuration is in the sweet spot.

We’ve deployed v1.7.9, compiled locally with Lua support. We’re observing that soft IRQ’s are ~8% while the system is under roughly 30% load overall. We believe this to be fairly high and we’re a bit concerned as the system is only experiencing about 50% of the traffic we expect to see in the coming months when my client hits their high season for business which just happens to be 4th quarter every year. So we’re a little pressed for time on making sure our setup is just right for their situation as they’ve opted for Ha-proxy as opposed to hardware based NLB’s in their new datacenter deployment.

Any insights or suggestions on how we might improve this is greatly appreciated.

I’ve provided all the build and configuration information below.

----total-cpu-usage---- -dsk/total- -net/total- —load-avg— ---system-- ------memory-usage----- ----system----
usr sys idl wai hiq siq| read writ| recv send| 1m 5m 15m | int csw | used buff cach free| time
1 2 96 0 0 1|2938B 14k| 0 0 |8.80 8.99 7.68| 29k 54k|2640M 90.1M 414M 28.3G|22-09 13:43:50
9 13 70 0 0 8| 0 0 | 71M 74M|8.80 8.99 7.68| 184k 289k|2641M 90.1M 414M 28.3G|22-09 13:43:51
8 14 69 0 0 9| 0 0 | 72M 75M|8.80 8.99 7.68| 185k 295k|2641M 90.1M 414M 28.3G|22-09 13:43:52
9 13 71 0 0 8| 0 0 | 72M 74M|8.80 8.99 7.68| 185k 289k|2639M 90.1M 414M 28.3G|22-09 13:43:53
9 12 71 0 0 8| 0 88k| 71M 74M|8.80 8.99 7.68| 184k 298k|2638M 90.1M 414M 28.3G|22-09 13:43:54
9 13 70 0 0 8| 0 0 | 71M 73M|8.65 8.96 7.67| 183k 286k|2639M 90.1M 414M 28.3G|22-09 13:43:55
9 13 70 0 0 8| 0 0 | 70M 73M|8.65 8.96 7.67| 183k 295k|2639M 90.1M 414M 28.3G|22-09 13:43:56
9 13 71 0 0 8| 0 0 | 70M 74M|8.65 8.96 7.67| 185k 295k|2637M 90.1M 414M 28.3G|22-09 13:43:57
8 13 71 0 0 8| 0 4096B| 71M 74M|8.65 8.96 7.67| 182k 286k|2639M 90.1M 414M 28.3G|22-09 13:43:58
9 13 70 0 0 8| 0 0 | 71M 74M|8.65 8.96 7.67| 187k 298k|2639M 90.1M 414M 28.3G|22-09 13:43:59
9 15 68 0 0 8| 0 28k| 73M 76M|8.52 8.93 7.67| 189k 310k|2637M 90.1M 414M 28.3G|22-09 13:44:00^C

Platform:

OS/Kernel: Debian GNU/Linux 8 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u3 (2017-08-15)

CPU:
model name : Intel® Xeon® CPU E5-2670 v3 @ 2.30GHz
Architecture: x86_64
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 1
BogoMIPS: 4594.12
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K

MEMORY (MB):
total used free shared buffers cached
Mem: 32132 2046 30085 28 86 377
Swap: 11583 0 11583

Build Parameters:

HA-Proxy version 1.7.9 2017/08/18
Copyright 2000-2017 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
OPTIONS = USE_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.2l 25 May 2017
Running on OpenSSL version : OpenSSL 1.0.2l 25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
Running on PCRE version : 8.35 2014-04-04
PCRE library supports JIT : yes
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe

TCP Stack tweaks:

net.core.rmem_max = 8738000
net.core.somaxconn = 4000
net.ipv4.tcp_max_syn_backlog = 10050
vm.min_free_kbytes = 131072
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.tcp_rmem = 8192 873800 8738000
net.ipv4.tcp_max_orphans = 1048576
net.ipv4.tcp_wmem = 4096 655360 6553600
net.core.wmem_max = 6553600
net.core.netdev_max_backlog = 4050
fs.nr_open = 10000000
net.ipv4.tcp_mem = 3093984 4125312 6187968
net.ipv4.netfilter.ip_conntrack_max = 20000000
net.ipv4.ip_local_port_range = 1024 65535

NIC Setup:

auto bond0
iface bond0 inet static
address xx.xxx.xx.81
netmask 255.255.255.0
gateway xx.xxx.xx.1
slaves eth0 eth1
bond_mode 802.3ad
bond_miimon 100
bond_downdelay 200
bond_updelay 200

auto bond1
iface bond1 inet static
address xxx.xx.x.81
netmask 255.255.248.0
slaves eth2 eth3
bond_mode 802.3ad
bond_miimon 100
bond_downdelay 200
bond_updelay 200

Configuration:
https://pastebin.ca/3876372

Posts: 1

Participants: 1

Read full topic

@Onichan wrote:

We put our site into maintenance mode (no access) to do major upgrades, but have to take it out of maintenance mode to do sanity testing. The problem is users can hit the site and post issues while we are sanity testing. Recently, we had to roll back a change which lost a few posts my users. This is considered a big deal by management and we’d like to avoid this in the future. With that in mind, I’m trying to come up with a change to our haproxy settings that would do the job. That is reject anyone not on a special ip address whitelist, sending them to our “we’re in maintenance” web page. The script that kicks off maintenance is executed from a special system that sends out commands to various systems via mcollective. I can easily have it touch and remove a file using that tool, but I can’t figure out a way to take advantage of this in haproxy. I was hoping to do something like this:

acl tester_ip src -f tester_ips.map
acl test_mode file-exists /etc/haproxy/TESTING
# if in test mode and not a tester's ip address, send to "we're in maintenance" page
use_backend sorry if test_mode !tester_ip

But how to I do the “file-exists” part (second line)? Is this possible. If not, is the an alternate way of achieving my goal?

Posts: 1

Participants: 1

Read full topic

@Onichan wrote:

I’ve created this acl:

acl in_maintenance always_false -u 1

And use it like this (yes, I know there are other ways to do it, there’s a reason for this):

use_backend sorry if in_maintenance

But can’t figure out how to change the acl to the value “always_true” using socat. I can see the acl when I run “show acl”:

1 () acl ‘always_false’ file ‘/etc/haproxy/haproxy.cfg’ line 74

But any attempt to use “add acl” or “del acl” throws an error or says Done, but doesn’t change the backend. For example:

echo “add acl #1 always_true” | socat tcp-connect:192.168.2.107:9999 stdio
Done.

Hitting the url doen’t take you to the “sorry” backend.

Likewise, attempting to delete:

echo “del acl #1 always_false” | socat tcp-connect:192.168.2.107:9999 stdio
Key not found.

Obviously, I don’t understand how to use this.

To put it simply, how do I change the acl from “always_false” to “always_true” and back?

Posts: 1

Participants: 1

Read full topic

@stefws wrote:

Where should one report possible bugs?

MaxSslRate doesn’t seem to reset when doing a ‘clear counters all’ other counters in ‘show info’

HAproxy 1.7.9 and older version as well

Posts: 1

Participants: 1

Read full topic

@amirhd wrote:

Hi,

I am trying to figure out a way to see the logs using HAProxy.

Every time I need to check the logs I need to get into specific folder using ssh and read those files.

Is it possible to somehow config HAProxy for example in some way which serve web content that is my desired folder of logs:

frontend https
acl logs-in-folder path_beg -i /logs/
use_backend view_log if logs-in-folder

backend view_log
# here I would like to serve the web content which point to the folders of logs

The application is using docker so the I have a tomcat in docker

the index.html

Index of /logs/

Index of /logs/

---

tomcat-logs-1/ tomcat-logs-2/

---

Posts: 1

Participants: 1

Read full topic

@cmendelson wrote:

I’ve been doing some load testing in anticipation of our busy season. I’m able to push through 800 connections per second to a single backend without much problem. When I add additional
servers to the backend pool, and increase the load on the haproxy machine to 3000-4000 connections per second, haproxy starts marking my backends as down, even though I’ve increased my backend capacity 10x, and only 4x-5x’ed my load.

Running: watch -n1 'curl --silent "http://localhost:9000/haproxy_stats;csv" | cut -d "," -f 2,5,37'

Shows backends going from L7OK -> *L7OK (what does the * mean?) -> L4TOUT -> Healthy. The downstream server is always healthy, and tcp connections from another machine succeed without any issue.

CPU on the haproxy machine I’m using is fine (an r4.large EC2 instance), as is memory usage. The machine is maxing out at 50MBps bandwidth usage, which is a 1/10th of what we push in production, so I’m not bandwidth limited.

Any ideas why haproxy isn’t able to successfully complete a healthcheck? Any suggestions on debugging this?

Here is my haproxy.cfg:

global
  maxconn 60000
  user haproxy
  group haproxy
  daemon
  tune.ssl.default-dh-param 2048

defaults
  mode http
  retries 3
  option redispatch
  timeout connect 5000
  timeout client 50000
  timeout server 50000

resolvers dns
    nameserver dns 169.254.169.253:53
backend server_pool
  balance hdr(x-mi-cbe)
  option httpchk GET /?health=true

  server a 10.2.2.183:9292 check inter 5000
  server b 10.2.1.184:9292 check inter 5000
  server c 10.2.2.174:9292 check inter 5000
  server d 10.2.0.41:9292 check inter 5000
  server e 10.2.0.16:9292 check inter 5000
  server f 10.2.2.216:9292 check inter 5000
  server g 10.2.1.135:9292 check inter 5000
  server h 10.2.0.162:9292 check inter 5000
  server i 10.2.1.232:9292 check inter 5000
  server j 10.2.1.253:9292 check inter 5000
  server k 10.2.2.141:9292 check inter 5000

backend stats
  balance roundrobin
  stats enable
frontend cors_proxy_http
  bind 0.0.0.0:80
  monitor-uri /haproxy?health
  default_backend server_pool
  option forwardfor
  option http-server-close
  maxconn 60000
  log /var/lib/haproxy/dev/log local2 debug
  option httplog
  option dontlognull

frontend stats
  bind 0.0.0.0:9000
  monitor-uri /haproxy?health
  default_backend stats
  stats uri /haproxy_stats

Posts: 4

Participants: 2

Read full topic

@Bas wrote:

Hello,

We are currently sending our HAProxy logs to a syslog server using the “log” option of HAProxy.

We use Logstash as our syslog server and it will send the logs to Elasticsearch which we can read from in Kibana.

I would like to be able to see which log level (emerg, alert, crit, err, warning, notice, info, debug) the log line is for so I can sort on that in Kibana.

Would it be possible to append the log level to the log line or is there any other way to make the Logstash server aware of the level?

Thank you for reading this.

Best regards,
Bas

Posts: 1

Participants: 1

Read full topic

@dalu wrote:

Hey guys,
I’m working on a project for my Bachelor thesis and I am currently trying to understand how HAProxy works. For my project I’d like to log more information from HAProxy. My defaults are

mode http
timeout client 120s
timeout server 120s
timeout connect 30s
option http-buffer-request
option httplog
option tcplog
option log-health-checks
log global

I would like to log where every single http-Package is sent to according to my ACL. My ACL, according to my .cnf SHOULD send a package according to information in the http-body.

acl name req.body -m reg < WhatImLookingFor >
use_backend bert if name

Am I on the right track? Or do I need to change something in the source code?

Sorry, I’m a newbie to communication protocols and C.

Thanks in advance,
DL

Posts: 1

Participants: 1

Read full topic

@Bas wrote:

Hello,

We would like to know how the stick table expire works.

If for example we set the expiry to 8 hours, what if a client connects again after 4 hours, would it still keep the record for 4 hours or another 8 hours?

Best regards,
Bas

Posts: 2

Participants: 2

Read full topic

@jared.dembrun wrote:

I have a basic haproxy running on my machine just trying to forward any traffic to a web server running on the same machine (port 9000). This works correctly on localhost, but not for any other devices. I have tried to connect via netcat and a browser on three different networks. The connection always times out (but is not refused like it is for other ports). Nmapping the port from the machine running haproxy shows it is open, shows it is ‘filtered’ in scans from other devices.

The configuration file is a trimmed version of the one on this page. Looks like this:

defaults
   mode http
   timeout connect 10000ms
   timeout client 50000ms
   timeout server 50000ms

frontend http-in
   bind *:9050
   default_backend websockets_support

backend websockets_support
   server ws1 127.0.0.1:9000

This is a work computer (mac) with certain security configurations. I am going to try running a similar proxy on my own linux machine when I get home tonight. In the meantime, if anyone else has dealt with this problem, I would appreciate some help. I am doubtful that it is security configurations since there is not a connection refused, just a timeout.

Posts: 1

Participants: 1

Read full topic

@dalu wrote:

Hey guys,

Another post. I’m working on HAProxy for my Bachelor thesis and would like to get to know some nifty http-client tools that I can use to test my HAProxy and its settings (viewing the Logs etc.). It should be able to send http-Requests locally to the listening port of my proxy and (of course) fully operational via console. I’m using SuSE Linux 9.1 if that helps.

Please be free to give me pros and cons about said tool.

I really appreciate any help I can get.

DL

Posts: 1

Participants: 1

Read full topic

@wilnzi wrote:

Hi,

I have problem whith backend redirection.
I’m want when user go to www.example.com, i give it the page located to www.example.com/new/new/

How can i make it ??

The current regirep instruction.
reqirep ^([^\ :])\ /(.) \1\ /new/new/\2

Posts: 2

Participants: 2

Read full topic

@Gurvan wrote:

Hi,

I intend to use haproxy as a protection for slow DDoS attacks when thousands of IPs request a single URL, so I am testing it on a few docker containers.

The setup is as follows : 2 apache backend, 1 haproxy and 2 clients.

I send a request every 2 seconds from the 2 clients, and I do see them in the Apache backends logs so they are transmitted properly, but however I try to update the tables, they have a very low count compared to the number they actually receive, and they typically don’t show the same number of requests for both clients.

The version is 1.7.5-2 on a Debian 9 docker container (clients and backends are Debian 9 containers too, there doesn’t appear to be any network problem between them).

Here is the configuration (please excuse the mess, I’m trying a bunch of things to make it work) :

global
log /dev/log local0
log /dev/log local1 notice
#chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 666
#stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

Default SSL material locations

ca-base /etc/ssl/certs
crt-base /etc/ssl/private

Default ciphers to use on SSL-enabled listening sockets.

For more information, see ciphers(1SSL). This list is from:

https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/

An alternative list with additional directives can be obtained from

https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy

ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend fe
bind *:80
mode http
stick-table type ip size 500 expire 60m store http_req_cnt,http_req_rate(60000)
#http-request track-sc0 src
tcp-request content track-sc0 src
http-request track-sc0 base table dummy
tcp-request content track-sc0 base table dummy2
#http-request track-sc0 src #sets only 1 request even if 5-6 were made
#http-request sc-inc-gpc0(“vla”)(2)
#http-request sc-inc-gpc0(base)
#http-request sc-inc-gpc0(“base”)
#stick on src not backend/listen

default_backend lamp

backend lamp
mode http
server lamp1 backend1:80 check

backend dummy
stick-table type string len 100 size 50 expire 30m store conn_rate(5m),http_req_rate(1m),conn_cur,gpc0
#stick-table type ip size 50 expire 30m store conn_rate(5m),http_req_rate(1m),conn_cur

backend dummy2
stick-table type string len 100 size 50 expire 30m store conn_rate(5m),http_req_rate(1m),conn_cur,gpc0
#stick-table type ip size 50 expire 30m store conn_rate(5m),http_req_rate(1m),conn_cur

#define stick table for ip

#define stick table for base

#define 3 acls

#drop if 3 acls matched

Let me know if you would like more information.

Any idea why http_req_cnt doen’t update properly?

Regards,
Gurvan

Posts: 1

Participants: 1

Read full topic

@gpmacarthur wrote:

I can’t seem to get my HAProxy to start, any ideas whats causing the problem?

root@haproxy-www:/# service haproxy restart
root@haproxy-www:/# service haproxy status
haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Mon 2017-10-02 17:21:11 AEDT; 4s ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 11014 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f ${CONFIG} -p /run/haproxy.pid $EXTRAOPTS (code=exited, status=0/SUCCESS)
  Process: 11011 ExecStartPre=/usr/sbin/haproxy -f ${CONFIG} -c -q (code=exited, status=0/SUCCESS)
 Main PID: 11014 (code=exited, status=0/SUCCESS)

Oct 02 17:21:11 haproxy-www systemd[1]: haproxy.service: Service hold-off time over, scheduling restart.
Oct 02 17:21:11 haproxy-www systemd[1]: Stopped HAProxy Load Balancer.
Oct 02 17:21:11 haproxy-www systemd[1]: haproxy.service: Start request repeated too quickly.
Oct 02 17:21:11 haproxy-www systemd[1]: Failed to start HAProxy Load Balancer.
root@haproxy-www:/#

Here is my haproxy.cfg file

global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    tcp
        option  tcplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend www
        bind ***.***.***.**:80
        default_backend site-backend

backend site-backend
        balance roundrobin
        mode tcp
        server webserver-1 10.240.0.178:80 check
        server webserver-2 10.240.0.42:80 check

Thank you, I really appreciate the help =)

Posts: 1

Participants: 1

Read full topic

@crow123 wrote:

Hello,
I am running haproxy 1.7.9-1~bpo9+1 on debian 9.1. And after running a while with production data haproxy stops working wiith segmentation fault:

haproxy[26291]: segfault at 5562af80e000 ip 00007f5985e48149 sp 00007ffe1d613488 error 4 in libc-2.24

Can you please help or have any ideas?
Thanks a lot!!!

Here is haproxy -vv:
HA-Proxy version 1.7.9-1~bpo9+1 2017/08/24
Copyright 2000-2017 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fdebug-prefix-map=/build/haproxy-1.7.9=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_NS=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.1.0f 25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f 25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with network namespace support

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe

Posts: 5

Participants: 2

Read full topic

@MrBasset wrote:

Hi,

We recently deployed a change to our HAProxy configuration for a client that needed to handle multiple different SSL certs and behaviours over the same IP/Port combination. For reasons I won’t go into, we couldn’t considered additional IPs, so instead we looked to utilise the SNI routing discussed in this blog to route the requests to different internal front-ends where we could terminate and handle the SSL as appropriate.

The configuration works and we have had no issues with the routing, however, the number of reported connections within the haproxy stats doubled for the Front Ends and Backends operating in TCP mode; whereas the Front Ends/Backends performing the SSL termination continue to report a similar number of connections to the figures we have prior to the change.

For example, using the names in the below, fe_TCP_SNI_Entry and be_Entry_B report 330 connections where as fe_Entry_Web_B reports 165 connections.

Has anyone experienced anything similar or have any suggestions as to why this difference (and exact doubling of the connections)? The increase in reported connections is playing havoc with our monitoring and scaling configuration.

HAProxy version:

$ haproxy -v
HA-Proxy version 1.6.4 2016/03/13

(yes, we know that we need to update)

Sanitised Configuration

global
    user                    haproxy
    group                   haproxy

    maxconn                 100000
    spread-checks           5

    pidfile                 /var/run/haproxy.pid
    chroot                  /var/lib/haproxy
    stats socket            /var/run/haproxy.sock mode 600 user haproxy level admin
    stats socket            /caci/haproxy/stats/haproxy.sock mode 600 user shinken level operator

    log                     127.0.0.1    local3     info

    #default SSL locations
    ca-base                 /etc/ssl/certs
    crt-base                /etc/ssl/private

    <snip ... removed SSL opts and ciphers />

defaults
    log                     global
    mode                    http

    #Don't log messages with no data exchange - relying on BrightSolid to protect from port scans
    option                  dontlognull
    option                  log-separate-errors

    option                  splice-auto
    option                  http-server-close
    option                  redispatch
    option                  contstats

    retries                 3

   <snip. ... removed timeouts and errors />

frontend fe_TCP_SNI_Entry
    mode tcp

    bind 0.0.0.0:443

    tcp-request inspect-delay 2s
    tcp-request content accept if { req_ssl_hello_type 1 }

    acl admin  req.ssl_sni -i admin."${HAPROXY_DOMAIN}"

    no log
    use_backend be_Entry_A if admin
    use_backend be_Entry_B

backend be_Entry_B
    ## Performs an internal proxy redirect to the conventional HTTP SSL termination.

    mode tcp
    server localhost localhost:46870 send-proxy

frontend fe_Web_Https_B
    mode http

    ## Perform SSL termination
    bind localhost:46870 accept-proxy ssl crt /etc/ssl/private/web.pem

   <snip ... removed server def />

backend be_Entry_A
    ## Performs an internal proxy redirect to the HTTP frontend where ACL rules are applied.

    mode tcp
    server localhost localhost:46869 send-proxy

frontend fe_Web_Https_A
    bind localhost:46869 accept-proxy ssl crt /etc/ssl/private/web2.pem ca-file /etc/ssl/private/client.crt verify optional crt-ignore-err all

   <snip ... remove SSL client certs and server defintions />

Thanks in advance,
MrBasset

Posts: 3

Participants: 2

Read full topic

@wilnzi wrote:

Hi,

I’m want to configure SSL Termination with version 1.6.
I have the error but i don’t know where.
I don’t know where can i see the log.

Thanks in advance.

My configuration file is :

frontend www-http
bind *:80
mode http
option httpclose
option forwardfor

acl localACL hdr_dom(host) -i localhost
use_backend mytravelBackend if localACL
default_backend defaultBackendServer

frontend www-https
bind *:21 ssl crt /etc/ssl/certs/certificate.pem
mode https
option httpclose
option forwardfor

acl localACL hdr_dom(host) -i localhost
use_backend localBackend if localACL
default_backend defaultBackendServer

backend defaultBackendServer
mode http
balance source
server SRV-WEB-1 192.168.2.13:80 check
server SRV-WEB-2 192.168.2.14:80 check

backend localBackend
mode http
balance source
acl no_redir url_beg /new/new
reqirep ^([^\ :])\ /(.) \1\ /new/new\2 if !no_redir
server SRV-WEB-1 192.168.2.13:80 check
server SRV-WEB-2 192.168.2.14:80 check

Posts: 2

Participants: 2

Read full topic

@mchesler wrote:

We’re looking to enable option abortonclose so that our application behind HAProxy can see when a client aborts a connection. Turning the option on worked fine, but we noticed that it resulted in a definite jump in hrsp_5xx counters. From the application’s perspective, the responses still seem to result in 200s. Why is HAProxy logging 5xx responses? Is there something else we can adjust to stop this from happening? We monitor the occurrence of 5xx response as an indicator of backend health, so we can’t simply ignore it. I’ve confirmed that removing the options eliminates the 5xx counter increments.

Posts: 1

Participants: 1

Read full topic

@skasch wrote:

Hi,

i am on haproxy 1.7.5 (debian) and try to setup what is mentioned here:
"how-to-set-ssl-verify-client-for-specific-domain-name"
my haproxy is located behind a firewall and requests are NATed

i’d like to have some users that are not in the networks_allowed list, to present a certificate.
others should be routed without certificate.

but on loading the page, firefox complains about SSL_ERROR_RX_RECORD_TOO_LONG
and the logs show:

x.x.x.x:50741 [04/Oct/2017:14:32:17.145] https_tcp_443 https_tcp_443/ -1/-1/0 188 PR 0/0/0/0/0 0/0
x.x.x.x:50740 [04/Oct/2017:14:32:17.146] https_tcp_443 https_tcp_443/ -1/-1/0 188 PR 0/0/0/0/0 0/0
x.x.x.x:50743 [04/Oct/2017:14:32:19.704] https_tcp_443 https_tcp_443/ -1/-1/0 188 PR 0/0/0/0/0 0/0

which looks like some backend is not found or a condition not met…

my config is as follows:

global
  log 127.0.0.1:514 local0 info
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  daemon
  maxconn 2048
  tune.ssl.default-dh-param 2048
  ssl-dh-param-file /etc/haproxy/dhparams.pem

  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private
  ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
  ssl-default-bind-options no-sslv3 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
  ssl-default-server-options no-sslv3 no-tls-tickets

defaults
  log global
  mode http
  balance roundrobin
  option httplog
  option dontlognull
  option forwardfor
  option http-server-close
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

frontend http_80
  mode http
  bind :80
  timeout http-request 5s

  acl acl_letsencrypt_http path_beg /.well-known/acme-challenge/
  use_backend backend_letsencrypt if acl_letsencrypt_http

  acl acl_dx hdr(host) -i dx.domain.com
  acl acl_manikin hdr(host) -i manikin.domain.com
  acl acl_publish hdr(host) -i publish.domain.com
  acl acl_publish-edeka hdr(host) -i publish-edeka.domain.com
  acl acl_zws hdr(host) -i zws.domain.com
  acl acl_easyjob hdr(host) -i easyjob.domain.com

  # Define a rule to detect SSL
  acl acl_hasSSL ssl_fc

  # Use rule acl_hasSSL to detect SSL and if not redirect to https
  redirect scheme https if !acl_hasSSL acl_zws !acl_letsencrypt_http
  redirect scheme https if !acl_hasSSL acl_easyjob !acl_letsencrypt_http
  redirect scheme https if !acl_hasSSL acl_dx !acl_letsencrypt_http
  redirect scheme https if !acl_hasSSL acl_manikin !acl_letsencrypt_http
  redirect scheme https if !acl_hasSSL acl_publish-edeka !acl_letsencrypt_http

  # Define non-SSL hosts that should be directed to their backend directly
  use_backend backend_publish if acl_publish

# intermediate frontend to handle client certificates when "verifiy required" externally
frontend https_tcp_443
  bind :443
  option tcplog
  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  acl networks_allowed src 192.168.124.0/24 192.168.126.0/24 192.168.127.0/24 192.168.223.0/24 192.168.224.0/24 172.20.3.0/24 172.20.4.0/22 172.20.8.0/22

  use_backend loop_clientcertenabled if { req_ssl_sni -i zws.domain.com or req_ssl_sni -i zws.domain.com } !networks_allowed
  default_backend loop_default

backend loop_clientcertenabled
  server loopback-for-tls abns@ssl_clientcert send-proxy-v2
backend loop_default
  server loopback-for-tls abns@ssl_default send-proxy-v2

frontend https_443_clientcert
  mode http
  bind abns@ssl_clientcert accept-proxy crt-list /etc/haproxy/certmappings.list ssl verify required ca-file /etc/haproxy/cacert.pem
  http-response set-header Strict-Transport-Security max-age=15768000

  # act definition to base check if user provided a certificate
  acl has_cert ssl_fc_has_crt

  # acl definition to check expiry of certificate
  acl valid_cert_expired ssl_c_verify 10

  use_backend backend_zws if { req_ssl_sni -i zws.domain.com } has_cert !valid_cert_expired
  use_backend backend_easyjob if { req_ssl_sni -i easyjob.domain.com } has_cert !valid_cert_expired

frontend https_443_default
  mode http
  bind abns@ssl_default accept-proxy crt-list /etc/haproxy/certmappings.list ssl
  http-response set-header Strict-Transport-Security max-age=15768000

  # define backends and conditionals
  use_backend backend_dx if { ssl_fc_sni dx.domain.com }
  use_backend backend_manikin if { ssl_fc_sni manikin.domain.com }
  use_backend backend_publish if { ssl_fc_sni publish.domain.com }
  use_backend backend_publish-edeka if { ssl_fc_sni publish-edeka.domain.com }
  use_backend backend_zws if { ssl_fc_sni zws.domain.com }
  use_backend backend_easyjob if { ssl_fc_sni easyjob.domain.com }

backend backend_letsencrypt
  mode http
  server localhost 127.0.0.1:8080

backend backend_dx
  mode http
  server dx 192.168.99.4:80

backend backend_manikin
  mode http
  server manikin-srv 172.20.4.255:80

backend backend_publish-edeka
  mode http
  server publish-edeka 192.168.99.32:80

backend backend_zws
  mode http
  server zws 192.168.99.9:80

backend backend_easyjob
  mode http
  server easyjob 172.20.3.26:80

any help is greatly appreciated

Posts: 1

Participants: 1

Read full topic

@bend66 wrote:

Hi,

I have an haproxy configuration where one frontend redirects to multiple backend depending on the SNI.

However I want to switch one of those backends to http2 and keep all the other on http1.1

My problem is I can’t find a way to tell haproxy that it should only accept http1 for a backend and accept http1 and http2 for the other.

Any idea if it feasible or not ?

Posts: 2

Participants: 2

Read full topic