Date: Tue, 27 Jun 2017 00:40:10 +0900 From: YongHyeon PYUN <pyunyh@gmail.com> To: Julien Cigar <julien@perdition.city> Cc: "Andrey V. Elsukov" <bu7cher@yandex.ru>, FreeBSD Net <freebsd-net@freebsd.org>, Ryan Stone <rysto32@gmail.com>, Ben RUBSON <ben.rubson@gmail.com>, "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org> Subject: Re: mbuf_jumbo_9k & iSCSI failing Message-ID: <20170626154010.GA2488@michelle.fasterthan.co.kr> In-Reply-To: <20170626134458.GT43966@mordor.lan> References: <613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com> <2c9a9c2652a74d8eb4b34f5a32c7ad5c@AM5PR0502MB2916.eurprd05.prod.outlook.com> <DB3PR05MB089011A41EF87A40C7AC741C36E0@DB3PR05MB089.eurprd05.prod.outlook.com> <F19B51C7-7DDD-4FAB-9091-0B7C8A7CE649@gmail.com> <52A2608C-A57E-4E75-A952-F4776BA23CA4@gmail.com> <9B507AA6-40FE-4B8D-853F-2A9422A2DF67@gmail.com> <CAFMmRNzo=xB8XF6SFD%2BwksmBYjRZ_peYjiPBCXNVyqP%2BdxnujQ@mail.gmail.com> <CAFMmRNwbEwn=TmTAd56rViDV5nDXq_hPmTp-cDwmVqu1XYm=fA@mail.gmail.com> <64abec26-e310-d66d-93ae-3536914ddd84@yandex.ru> <20170626134458.GT43966@mordor.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jun 26, 2017 at 03:44:58PM +0200, Julien Cigar wrote: > On Mon, Jun 26, 2017 at 04:13:33PM +0300, Andrey V. Elsukov wrote: > > On 25.06.2017 18:32, Ryan Stone wrote: > > > Having looking at the original email more closely, I see that you showed an > > > mlxen interface with a 9020 MTU. Seeing allocation failures of 9k mbuf > > > clusters increase while you are far below the zone's limit means that > > > you're definitely running into the bug I'm describing, and this bug could > > > plausibly cause the iSCSI errors that you describe. > > > > > > The issue is that the newer version of the driver tries to allocate a > > > single buffer to accommodate an MTU-sized packet. Over time, however, > > > memory will become fragmented and eventually it can become impossible to > > > allocate a 9k physically contiguous buffer. When this happens the driver > > > is unable to allocate buffers to receive packets and is forced to drop > > > them. Presumably, if iSCSI suffers too many packet drops it will terminate > > > the connection. The older version of the driver limited itself to > > > page-sized buffers, so it was immune to issues with memory fragmentation. > > > > I think it is not mlxen specific problem, we have the same symptoms with > > ixgbe(4) driver too. To avoid the problem we have patches that are > > disable using of 9k mbufs, and instead only use 4k mbufs. > > I had the same issue on a lightly loaded HP DL20 machine (BCM5720 > chipsets), 8GB of RAM, running 10.3. Problem usually happens > within 30 days with 9k jumbo clusters allocation failure. > This looks strange to me. If I recall correctly bge(4) does not request physically contiguous 9k jumbo buffers for BCM5720 so it wouldn't suffer from memory fragmentation. (It uses m_cljget() and takes advantage of extended RX BDs to handle up to 4 DMA segments). If your controller is either BCM5714/BCM5715 or BCM5780, it requires physically contiguous 9k jumbo buffers to handle jumbo frames though.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170626154010.GA2488>