From owner-freebsd-stable@freebsd.org Thu Feb 4 16:08:20 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CB22A5478CF for ; Thu, 4 Feb 2021 16:08:20 +0000 (UTC) (envelope-from freebsd-stable@gomor.org) Received: from onyphe.fr (super1.onyphe.io [54.36.107.100]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DWk436PFRz4Y1N for ; Thu, 4 Feb 2021 16:08:19 +0000 (UTC) (envelope-from freebsd-stable@gomor.org) Received: (qmail 41359 invoked by uid 0); 4 Feb 2021 16:08:12 -0000 Received: from unknown (HELO www.onyphe.io) (172.16.6.254) by smtpout.jail with SMTP; 4 Feb 2021 16:08:12 -0000 MIME-Version: 1.0 Date: Thu, 04 Feb 2021 17:08:12 +0100 From: GomoR To: freebsd-stable@freebsd.org Cc: jhb@freebsd.org Subject: Suspected mbuf leak with Nginx + sendfile + TLS in 12.2-STABLE User-Agent: Roundcube Webmail/1.4.8 Message-ID: X-Sender: freebsd-stable@gomor.org Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DWk436PFRz4Y1N X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd-stable@gomor.org designates 54.36.107.100 as permitted sender) smtp.mailfrom=freebsd-stable@gomor.org X-Spamd-Result: default: False [-2.99 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[54.36.107.100:from]; R_SPF_ALLOW(-0.20)[+ip4:54.36.107.100]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[gomor.org]; SPAMHAUS_ZRD(0.00)[54.36.107.100:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.69)[-0.694]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:16276, ipnet:54.36.0.0/16, country:FR]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-stable]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2021 16:08:20 -0000 Dear FreeBSD community, we are encountering a DoS condition on our production machines. Our use case is an Nginx reverse proxy serving large files via HTTPS. This problem arose when switching kernel and userland from 12.1-RELEASE to 12.2-RELEASE. Ports were not upgraded (at first). Each time a user downloads a file, mbuf & mbuf_clusters are raising to reach the maximum limit in a matter of seconds. Those values are asserted by 'netstat -m' as follows: Normal situation: mbuf: 256, 26031105, 16767, 5974,428087938, 0, 0 mbuf_cluster: 2048, 8135232, 18408, 2704,101644203, 0, 0 Warning situtation: mbuf: 256, 26031105, 2981516, 151205,1109483561, 0, 0 mbuf_cluster: 2048, 8135232, 2983155, 4201,319714617, 0, 0 We have seen a patch related to sendfile + KTLS + mbuf at the below link and we updated to -STABLE to apply: Don't transmit mbufs that aren't yet ready on TOE sockets. This includes mbufs waiting for data from sendfile() I/O requests, or mbufs awaiting encryption for KTLS. https://github.com/freebsd/freebsd-src/commit/14c77f30b201bf76119d59678e72051c093333c2 Unfortunately for us, applying it didn't solve the issue. When we stop the download early, mbufs are freed. But past a threshold, we must reboot the server. The only remaining thing we can do is to ping the server, it is no more possible to connect with SSH, for instance. We also tried to set some loader.conf values which fixed nothing: hw.ix.enable_msix=0 hw.pci.enable_msix=0 hw.pci.enable_msi=0 net.inet.tcp.tso=0 hw.ix.flow_control=0 We also updated Nginx & OpenSSL to latest versions and tried Nginx to compile against FreeBSD shipped OpenSSL library. It did change nothing. Versions: openssl-1.1.1i,1 nginx-1.18.0_45,2 # ldd /usr/local/sbin/nginx /usr/local/sbin/nginx: libcrypt.so.5 => /lib/libcrypt.so.5 (0x800323000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x800344000) libssl.so.11 => /usr/local/lib/libssl.so.11 (0x8003e7000) libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (0x80047e000) libz.so.6 => /lib/libz.so.6 (0x800772000) libc.so.7 => /lib/libc.so.7 (0x80078e000) libthr.so.3 => /lib/libthr.so.3 (0x800b84000) NIC is: ix0: What can we do to help you find the root cause? Best regards, P.S.: adding jhb@ in Cc from bapt@ suggestion