From owner-freebsd-scsi@freebsd.org Sun Jun 25 15:32:01 2017 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8FDAD94000; Sun, 25 Jun 2017 15:32:01 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-it0-x229.google.com (mail-it0-x229.google.com [IPv6:2607:f8b0:4001:c0b::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ACBB273B81; Sun, 25 Jun 2017 15:32:01 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-it0-x229.google.com with SMTP id b205so28631718itg.1; Sun, 25 Jun 2017 08:32:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=MsAEoNurtT5IYa40c+cR50mA6+sUo55yg6jG04sP4Ik=; b=ZYDBYSw2SSa7lX/Aisw5huwcTxYk9wG5H2lSjTbX5edRxEUaSiSx9FJ6Fd7ZMMMlIe EwOh9cnjpPC6pYaHp8fiBbe2h87uO9C409kb86TSAlS5R1NvUwZZTX4mbD9g7w8r4yd/ HPQnRDVpLNVn67cCYIs8j7u4t/8MHsh01DosYi57tEOO3E3AXdooTWuyinRSGrrXcU9T Wm5FGbg8Ih+oyQ4TJVnswxE6rpUbrY1+geQ4VRNHeqBfnRZ7C+QX4i8ksiy6vL4ENsTP qHboF5RrEj+XqvC9gP3eyq/ti8VzZJntKPkR6y0LZRYKis44jUEJs/KBay+JuqmHr3ts 11nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=MsAEoNurtT5IYa40c+cR50mA6+sUo55yg6jG04sP4Ik=; b=tZzX5ya7qG2x3CoovlCTt7CLw/I+7EGfzB5IBh8ZJgh7V68Pypz2d+xYQDSR3l6iuK 6jRLb5xPUVoQj8IORBRadS+/fqswZ4bHR0x1Sf0lWrq03WlaPwaccUnw8NDtAxS5HmeU HOBL8SYYDDto7uPdlf/7mJBPScRncAjE+pC3COWZqcE1V4KvwLNmFCl7XAdB/nS377Ws /X0DpmEtS4G+0E2n07B28JvGdzWksSI4TO1P9m22SDId6z6DFRCulJbxmqpBiav+s22p 509UcXDmPj2BtWi63WFTnujEtHQDY8Tz7b+Q/KUmeNQbjro87KmN0DGftewh6isKXIay jtpQ== X-Gm-Message-State: AKS2vOzaagOaGSkhSHqrbEIyx+rgHxgkgzF0IW0NSeTipuONBwjwt1Ij woN7ZpcOZw11ajOg96CBa/H9hO9oDQ== X-Received: by 10.36.65.23 with SMTP id x23mr18161797ita.2.1498404721102; Sun, 25 Jun 2017 08:32:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.156.21 with HTTP; Sun, 25 Jun 2017 08:32:00 -0700 (PDT) In-Reply-To: References: <486A6DA0-54C8-40DF-8437-F6E382DA01A8@gmail.com> <6a31ef00-5f7a-d36e-d5e6-0414e8b813c7@selasky.org> <613AFD8E-72B2-4E3F-9C70-1D1E43109B8A@gmail.com> <2c9a9c2652a74d8eb4b34f5a32c7ad5c@AM5PR0502MB2916.eurprd05.prod.outlook.com> <52A2608C-A57E-4E75-A952-F4776BA23CA4@gmail.com> <9B507AA6-40FE-4B8D-853F-2A9422A2DF67@gmail.com> From: Ryan Stone Date: Sun, 25 Jun 2017 11:32:00 -0400 Message-ID: Subject: Re: mbuf_jumbo_9k & iSCSI failing To: Ben RUBSON Cc: FreeBSD Net , "freebsd-scsi@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jun 2017 15:32:02 -0000 Having looking at the original email more closely, I see that you showed an mlxen interface with a 9020 MTU. Seeing allocation failures of 9k mbuf clusters increase while you are far below the zone's limit means that you're definitely running into the bug I'm describing, and this bug could plausibly cause the iSCSI errors that you describe. The issue is that the newer version of the driver tries to allocate a single buffer to accommodate an MTU-sized packet. Over time, however, memory will become fragmented and eventually it can become impossible to allocate a 9k physically contiguous buffer. When this happens the driver is unable to allocate buffers to receive packets and is forced to drop them. Presumably, if iSCSI suffers too many packet drops it will terminate the connection. The older version of the driver limited itself to page-sized buffers, so it was immune to issues with memory fragmentation.