From owner-freebsd-stable@freebsd.org Thu Feb 4 18:33:47 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C3C4952B7A0 for ; Thu, 4 Feb 2021 18:33:47 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DWnHv5DQwz4l1R; Thu, 4 Feb 2021 18:33:47 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro.local (unknown [IPv6:2601:648:8681:1cb0:2563:9ea7:c86c:ced1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 61935A254; Thu, 4 Feb 2021 18:33:47 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Subject: Re: Suspected mbuf leak with Nginx + sendfile + TLS in 12.2-STABLE To: GomoR , freebsd-stable@freebsd.org References: From: John Baldwin Message-ID: <9c56bfda-725c-9c2a-9db3-4599abcfeaa0@FreeBSD.org> Date: Thu, 4 Feb 2021 10:33:45 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2021 18:33:47 -0000 On 2/4/21 8:08 AM, GomoR wrote: > Dear FreeBSD community, > > we are encountering a DoS condition on our production machines. > Our use case is an Nginx reverse proxy serving large files via HTTPS. > This problem arose when switching kernel and userland from 12.1-RELEASE > to 12.2-RELEASE. Ports were not upgraded (at first). > > Each time a user downloads a file, mbuf & mbuf_clusters are raising to > reach the maximum limit in a matter of seconds. Those values are > asserted by 'netstat -m' as follows: > > Normal situation: > > mbuf: 256, 26031105, 16767, 5974,428087938, 0, > 0 > mbuf_cluster: 2048, 8135232, 18408, 2704,101644203, 0, > 0 > > Warning situtation: > > mbuf: 256, 26031105, 2981516, 151205,1109483561, 0, > 0 > mbuf_cluster: 2048, 8135232, 2983155, 4201,319714617, 0, > 0 > > We have seen a patch related to sendfile + KTLS + mbuf at the below link > and we updated to -STABLE to apply: None of the sendfile or KTLS changes from Netflix are in 12, they are only in 13 and later. > Don't transmit mbufs that aren't yet ready on TOE sockets. > This includes mbufs waiting for data from sendfile() I/O requests, or > mbufs awaiting encryption for KTLS. > https://github.com/freebsd/freebsd-src/commit/14c77f30b201bf76119d59678e72051c093333c2 This patch only applies to Chelsio T5/T6 NICs when using TOE (TCP offload) and doesn't affect freeing mbufs, it just fixes a race when the NIC could potentially send random garbage if it sends the mbuf before the scheduled disk I/O to populate it with data from disk has completed. > NIC is: > ix0: > > What can we do to help you find the root cause? The first step I would do if possible would be to bisect between the last known working version and the version that is known to be broken to determine which commit introduced the problem. One thing that could help here is to see if you can reproduce the problem using a 12.2 kernel on a 12.1 world + ports. If you can, then you can limit your bisecting to just building new kernels which will make that process quicker. You might also see if using a different NIC shows the same problem. If not, then it might point to a regression in the NIC driver (or perhaps in iflib as ix uses iflib I believe). -- John Baldwin