From owner-freebsd-scsi@freebsd.org Mon Jun 26 13:16:23 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CACBD86CDA; Mon, 26 Jun 2017 13:16:23 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward1m.cmail.yandex.net (forward1m.cmail.yandex.net [5.255.216.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A21C378EB1; Mon, 26 Jun 2017 13:16:22 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp1o.mail.yandex.net (smtp1o.mail.yandex.net [37.140.190.26]) by forward1m.cmail.yandex.net (Yandex) with ESMTP id 3B4B3215F8; Mon, 26 Jun 2017 16:16:13 +0300 (MSK) Received: from smtp1o.mail.yandex.net (localhost.localdomain [127.0.0.1]) by smtp1o.mail.yandex.net (Yandex) with ESMTP id 94CEC1300B9E; Mon, 26 Jun 2017 16:16:11 +0300 (MSK) Received: by smtp1o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id VSmE86ptau-GAcSd94F; Mon, 26 Jun 2017 16:16:10 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1498482970; bh=UwoC2WoPuBeLK+HvGWTll0M+Zmas0TUMuu8l5xHy7vY=; h=Subject:To:Cc:References:From:Message-ID:Date:In-Reply-To; b=MPpPn9vIZU0GezIh67TJPsoGiCPwKKFVMJnVHXHl8jaZm8rvEoU83F1u6bYJXu+d4 3ATLDqm/fwSkAnNaowxhdQGyieSSlRfqihokaKov36noijxH6iHd5C7yImLSvQelU/ BrFrFwD4FYY6eqPT5zutCCAWRn9EIEdu9PHiW68E= Authentication-Results: smtp1o.mail.yandex.net; dkim=pass header.i=@yandex.ru X-Yandex-Suid-Status: 1 0,1 0,1 0,1 0 Subject: Re: mbuf_jumbo_9k & iSCSI failing To: Ryan Stone , Ben RUBSON Cc: FreeBSD Net , "freebsd-scsi@freebsd.org" References: <486A6DA0-54C8-40DF-8437-F6E382DA01A8@gmail.com> <6a31ef00-5f7a-d36e-d5e6-0414e8b813c7@selasky.org> <613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com> <2c9a9c2652a74d8eb4b34f5a32c7ad5c@AM5PR0502MB2916.eurprd05.prod.outlook.com> <52A2608C-A57E-4E75-A952-F4776BA23CA4@gmail.com> <9B507AA6-40FE-4B8D-853F-2A9422A2DF67@gmail.com> From: "Andrey V. Elsukov" Openpgp: id=E6591E1B41DA1516F0C9BC0001C5EA0410C8A17A Message-ID: <64abec26-e310-d66d-93ae-3536914ddd84@yandex.ru> Date: Mon, 26 Jun 2017 16:13:33 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.0.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="smqF3QFS4WoFbpvK6WMrFmhjvPMtBWeiE" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jun 2017 13:16:23 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --smqF3QFS4WoFbpvK6WMrFmhjvPMtBWeiE Content-Type: multipart/mixed; boundary="LXCCDuFkvV5DOvwWRd8qkkXQ6ssJnmhDg"; protected-headers="v1" From: "Andrey V. Elsukov" To: Ryan Stone , Ben RUBSON Cc: FreeBSD Net , "freebsd-scsi@freebsd.org" Message-ID: <64abec26-e310-d66d-93ae-3536914ddd84@yandex.ru> Subject: Re: mbuf_jumbo_9k & iSCSI failing References: <486A6DA0-54C8-40DF-8437-F6E382DA01A8@gmail.com> <6a31ef00-5f7a-d36e-d5e6-0414e8b813c7@selasky.org> <613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com> <2c9a9c2652a74d8eb4b34f5a32c7ad5c@AM5PR0502MB2916.eurprd05.prod.outlook.com> <52A2608C-A57E-4E75-A952-F4776BA23CA4@gmail.com> <9B507AA6-40FE-4B8D-853F-2A9422A2DF67@gmail.com> In-Reply-To: --LXCCDuFkvV5DOvwWRd8qkkXQ6ssJnmhDg Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 25.06.2017 18:32, Ryan Stone wrote: > Having looking at the original email more closely, I see that you showe= d an > mlxen interface with a 9020 MTU. Seeing allocation failures of 9k mbuf= > clusters increase while you are far below the zone's limit means that > you're definitely running into the bug I'm describing, and this bug cou= ld > plausibly cause the iSCSI errors that you describe. >=20 > The issue is that the newer version of the driver tries to allocate a > single buffer to accommodate an MTU-sized packet. Over time, however, > memory will become fragmented and eventually it can become impossible t= o > allocate a 9k physically contiguous buffer. When this happens the driv= er > is unable to allocate buffers to receive packets and is forced to drop > them. Presumably, if iSCSI suffers too many packet drops it will termi= nate > the connection. The older version of the driver limited itself to > page-sized buffers, so it was immune to issues with memory fragmentatio= n. I think it is not mlxen specific problem, we have the same symptoms with ixgbe(4) driver too. To avoid the problem we have patches that are disable using of 9k mbufs, and instead only use 4k mbufs. --=20 WBR, Andrey V. Elsukov --LXCCDuFkvV5DOvwWRd8qkkXQ6ssJnmhDg-- --smqF3QFS4WoFbpvK6WMrFmhjvPMtBWeiE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAllRCH0ACgkQAcXqBBDI oXobwAf+Lxnto9w+KaTuAIkktZnoN42kjUNtY1bMNFn5amhEgFNlUeQxqFymZdvQ h7f2cAiNJ3SLVxiw/cxbeOK4r2Bk53JwNBEwI/VtyNCDLyikJ7Ov8yVOAgG4RydA llp+ZXsko19zxdlR5aBW140egiRCXTMvbNZ4IoqE3GiHwS6TDEFglbbEbUJK0r9l A9zlW+0EAxo3UELSQhfymIALfXHCPLzM0AYf/VvWdvNIio3Y1ZMeZK6Rkofgefux yddJnL54pVWKz1LvfuSEtE+wW9Tm/nF2MeRrdLKsiSWJGuUwTKlMq4porJ3K0Viq V6MHyi2U7818TQcDBbOkwrztEL9l3w== =S8WI -----END PGP SIGNATURE----- --smqF3QFS4WoFbpvK6WMrFmhjvPMtBWeiE--