From owner-freebsd-stable@freebsd.org Thu Aug 27 18:02:06 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BFFA93BA775 for ; Thu, 27 Aug 2020 18:02:06 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [128.127.146.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.norma.perm.ru", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BcrCb5Bn4z43XJ for ; Thu, 27 Aug 2020 18:02:03 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [192.168.243.9] ([192.168.243.9]) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTP id 07RI2Jqj041236 for ; Thu, 27 Aug 2020 23:02:19 +0500 (+05) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1598551340; bh=cP5tfRnkGT6bcbhISie+V/6SR6F00ncgUANDOoqr1CU=; h=To:From:Subject:Date; b=j39rilgeY+0XWaWNNz7g43qMId8caEPBfrkzGOlP3QZD1KRDSUmFjwXzamRBHXQDa a0fzT6lg5ciYmwTrbiX2VB4IZ6V5ybiKNzjRitWpI+fFbJqnYM3Wy1SgzkcqM6U81q a36VyW9x/Nvj0uEOcyeWpMTz0jnwOUnNS1Sw3ndo= To: freebsd-stable@freebsd.org From: "Eugene M. Zheganin" Subject: running out of ports: every client port is used only once in outgoing connection Message-ID: Date: Thu, 27 Aug 2020 23:01:54 +0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-Rspamd-Queue-Id: 4BcrCb5Bn4z43XJ X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=norma.perm.ru header.s=key header.b=j39rilge; dmarc=pass (policy=none) header.from=perm.ru; spf=pass (mx1.freebsd.org: domain of emz@norma.perm.ru designates 128.127.146.8 as permitted sender) smtp.mailfrom=emz@norma.perm.ru X-Spamd-Result: default: False [5.99 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(0.00)[norma.perm.ru:s=key]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(0.00)[+a]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; NEURAL_SPAM_MEDIUM(0.99)[0.991]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; RBL_VIRUSFREE_BOTNET(2.00)[128.127.146.8:from]; NEURAL_SPAM_SHORT(0.74)[0.745]; DKIM_TRACE(0.00)[norma.perm.ru:+]; DMARC_POLICY_ALLOW(0.00)[perm.ru,none]; HFILTER_HELO_IP_A(1.00)[elf.hq.norma.perm.ru]; NEURAL_SPAM_LONG(0.96)[0.956]; HFILTER_HELO_NORES_A_OR_MX(0.30)[elf.hq.norma.perm.ru]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:57401, ipnet:128.127.146.0/24, country:RU]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2020 18:02:06 -0000 Hello, I have a situation where I'm running out of client ports on a huge reverse-proxy. Say I have an nginx upstream like this: upstream geoplatform {         hash $hashkey consistent;         server 127.0.0.1:4079 fail_timeout=10s;         server 127.0.0.1:4080 fail_timeout=10s;         server 10.100.34.5:4079 fail_timeout=10s;         server 10.100.34.5:4080 fail_timeout=10s;         server 10.100.34.7:4079 fail_timeout=10s;         server 10.100.34.7:4080 fail_timeout=10s;         server 10.100.34.8:4079 fail_timeout=10s;         server 10.100.34.8:4080 fail_timeout=10s; } And as soon as I'm switching to it from DNS RR I'm starting to get get "Can't assign outgoing address when connecting to ...". The usual approach would be to assign multiple IP aliases to the destination backends, so I will get more of socket tuples. So I did this: upstream geoplatform {         hash $hashkey consistent;         server 127.0.0.1:4079 fail_timeout=10s;         server 127.0.0.1:4080 fail_timeout=10s;         server 127.0.0.2:4079 fail_timeout=10s;         server 127.0.0.2:4080 fail_timeout=10s;         server 127.0.0.3:4079 fail_timeout=10s;         server 127.0.0.3:4080 fail_timeout=10s;         server 10.100.34.5:4079 fail_timeout=10s;         server 10.100.34.5:4080 fail_timeout=10s;         server 10.100.33.8:4079 fail_timeout=10s;         server 10.100.33.8:4080 fail_timeout=10s;         server 10.100.33.9:4079 fail_timeout=10s;         server 10.100.33.9:4080 fail_timeout=10s;         server 10.100.33.10:4079 fail_timeout=10s;         server 10.100.33.10:4080 fail_timeout=10s;         server 10.100.34.7:4079 fail_timeout=10s;         server 10.100.34.7:4080 fail_timeout=10s;         server 10.100.34.8:4079 fail_timeout=10s;         server 10.100.34.8:4080 fail_timeout=10s;         server 10.100.34.10:4079 fail_timeout=10s;         server 10.100.34.10:4080 fail_timeout=10s;         server 10.100.34.11:4079 fail_timeout=10s;         server 10.100.34.11:4080 fail_timeout=10s;         server 10.100.34.12:4079 fail_timeout=10s;         server 10.100.34.12:4080 fail_timeout=10s; } Surprisingly, this didn't work. So... I just checked if I really have that much of connections. Seems like I'm starting to get troubles on 130K of connections, but even on the initial upstream configuration I should be able to handle 65535 - 10K (since net.inet.ip.portrange.first is 10K) = 55535, 55535 * 8 ~ 450K of connections. Looks like the client port is not reused at all in socket tuples ! Indeed it does not: the below line is taken when there's no free ports, since the nearby console window is flooded with "Can't assign requested address", so I assume I should already have 10.100.34.6.57026 (local IP-port pair) used in as many connection, as many servers I have. But it occurs only once: # netstat -an | grep 10.100.34.6.57026 tcp4       0      0 10.100.34.6.57026      10.100.34.5.4079 ESTABLISHED [root@geo2ng:vhost.d/balancer]# Second test: lets count how many times each port is used in netstat -an: # netstat -an -p tcp | grep -v LISTEN | grep 10.100 | awk '{print $4}' | sort | uniq -c | more | grep -v 1\  (none) So, seems like FreeBSD isn't reusing client ports out-of-the-box. Linux, on the other hand, does reuse ports for client connection, as long as the socket tuple stays unique. How do I get the same behavior on FreeBSD ? Thanks. Eugene.