Date: Thu, 27 Aug 2020 23:01:54 +0500 From: "Eugene M. Zheganin" <emz@norma.perm.ru> To: freebsd-stable@freebsd.org Subject: running out of ports: every client port is used only once in outgoing connection Message-ID: <a0607c68-30ed-5969-15eb-cbc67d8f90eb@norma.perm.ru>
next in thread | raw e-mail | index | archive | help
Hello,
I have a situation where I'm running out of client ports on a huge
reverse-proxy.
Say I have an nginx upstream like this:
upstream geoplatform {
hash $hashkey consistent;
server 127.0.0.1:4079 fail_timeout=10s;
server 127.0.0.1:4080 fail_timeout=10s;
server 10.100.34.5:4079 fail_timeout=10s;
server 10.100.34.5:4080 fail_timeout=10s;
server 10.100.34.7:4079 fail_timeout=10s;
server 10.100.34.7:4080 fail_timeout=10s;
server 10.100.34.8:4079 fail_timeout=10s;
server 10.100.34.8:4080 fail_timeout=10s;
}
And as soon as I'm switching to it from DNS RR I'm starting to get get
"Can't assign outgoing address when connecting to ...". The usual
approach would be to assign multiple IP aliases to the destination
backends, so I will get more of socket tuples. So I did this:
upstream geoplatform {
hash $hashkey consistent;
server 127.0.0.1:4079 fail_timeout=10s;
server 127.0.0.1:4080 fail_timeout=10s;
server 127.0.0.2:4079 fail_timeout=10s;
server 127.0.0.2:4080 fail_timeout=10s;
server 127.0.0.3:4079 fail_timeout=10s;
server 127.0.0.3:4080 fail_timeout=10s;
server 10.100.34.5:4079 fail_timeout=10s;
server 10.100.34.5:4080 fail_timeout=10s;
server 10.100.33.8:4079 fail_timeout=10s;
server 10.100.33.8:4080 fail_timeout=10s;
server 10.100.33.9:4079 fail_timeout=10s;
server 10.100.33.9:4080 fail_timeout=10s;
server 10.100.33.10:4079 fail_timeout=10s;
server 10.100.33.10:4080 fail_timeout=10s;
server 10.100.34.7:4079 fail_timeout=10s;
server 10.100.34.7:4080 fail_timeout=10s;
server 10.100.34.8:4079 fail_timeout=10s;
server 10.100.34.8:4080 fail_timeout=10s;
server 10.100.34.10:4079 fail_timeout=10s;
server 10.100.34.10:4080 fail_timeout=10s;
server 10.100.34.11:4079 fail_timeout=10s;
server 10.100.34.11:4080 fail_timeout=10s;
server 10.100.34.12:4079 fail_timeout=10s;
server 10.100.34.12:4080 fail_timeout=10s;
}
Surprisingly, this didn't work. So... I just checked if I really have
that much of connections. Seems like I'm starting to get troubles on
130K of connections, but even on the initial upstream configuration I
should be able to handle 65535 - 10K (since net.inet.ip.portrange.first
is 10K) = 55535, 55535 * 8 ~ 450K of connections. Looks like the client
port is not reused at all in socket tuples !
Indeed it does not: the below line is taken when there's no free ports,
since the nearby console window is flooded with "Can't assign requested
address", so I assume I should already have 10.100.34.6.57026 (local
IP-port pair) used in as many connection, as many servers I have. But it
occurs only once:
# netstat -an | grep 10.100.34.6.57026
tcp4 0 0 10.100.34.6.57026 10.100.34.5.4079 ESTABLISHED
[root@geo2ng:vhost.d/balancer]#
Second test: lets count how many times each port is used in netstat -an:
# netstat -an -p tcp | grep -v LISTEN | grep 10.100 | awk '{print $4}' |
sort | uniq -c | more | grep -v 1\
(none)
So, seems like FreeBSD isn't reusing client ports out-of-the-box.
Linux, on the other hand, does reuse ports for client connection, as
long as the socket tuple stays unique. How do I get the same behavior on
FreeBSD ?
Thanks.
Eugene.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a0607c68-30ed-5969-15eb-cbc67d8f90eb>