From owner-freebsd-fs@FreeBSD.ORG Tue Oct 22 19:42:08 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E3F7A636; Tue, 22 Oct 2013 19:42:08 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ve0-x22f.google.com (mail-ve0-x22f.google.com [IPv6:2607:f8b0:400c:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 932A322C6; Tue, 22 Oct 2013 19:42:08 +0000 (UTC) Received: by mail-ve0-f175.google.com with SMTP id pa12so5409986veb.6 for ; Tue, 22 Oct 2013 12:42:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Ot0BI8ZRT49L47kI9cMvxwUR/6atwDAxmJtjyfSH+dw=; b=rKLaMGttT/q/277VrNfFCJ08dURuMAevE/L5E3uVGD6Ny9qEmbJ18/d8qqd/7x/x/6 j0jAehxS2bMNTfVl6RCVHP5ViPsi4Bo6oaxeE0VCckkVs4xqSxXNSVZ3kT8u9STIRPT/ 0vnob+rKwq/xsU5qg8sXm7e9hFK8t8Py2lOG8u0zKv/CWiQwl8vhSihLcKE1JVJnZdOx cGOhOALCLlwbn+9nFisn5ux6GeZGZPq8sT0yrBMOv0z43HC9j16BpycbeqjOCnS27oD9 eSKeTCnNxvsdjTXzgSsL31AAxysY/WbV8vN2uh2/7aV0X8GqO3JjqdyQhJ7XU5nwRLzm 0fow== MIME-Version: 1.0 X-Received: by 10.58.207.15 with SMTP id ls15mr5917081vec.17.1382470927625; Tue, 22 Oct 2013 12:42:07 -0700 (PDT) Received: by 10.220.30.130 with HTTP; Tue, 22 Oct 2013 12:42:07 -0700 (PDT) Date: Tue, 22 Oct 2013 15:42:07 -0400 Message-ID: Subject: istgt causes massive jumbo nmbclusters loss From: Zaphod Beeblebrox To: FreeBSD Net , freebsd-fs Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Oct 2013 19:42:09 -0000 I have a server FreeBSD virtual.accountingreality.com 9.2-STABLE FreeBSD 9.2-STABLE #13 r256549M: Tue Oct 15 16:29:48 EDT 2013 root@virtual.accountingreality.com:/usr/obj/usr/src/sys/VRA amd64 That has an em0 with jumbo packets enabled: em0: flags=8843 metric 0 mtu 9014 It has (among other things): ZFS, NFS, iSCSI (via istgt) and Samba. Every day or two, it looses it's ability to talk to the network. ifconfig down/up on em0 gives the message about not being able to allocate the receive buffers... With everything running, but with specifically iSCSI not used, everything seems good. When I start hitting istgt, I see the denied stat for 9k mbufs rise very rapidly (this amount only took a few seconds): [1:47:347]root@virtual:/usr/local/etc/iet> netstat -m 1313/877/2190 mbufs in use (current/cache/total) 20/584/604/523514 mbuf clusters in use (current/cache/total/max) 20/364 mbuf+clusters out of packet secondary zone in use (current/cache) 239/359/598/261756 4k (page size) jumbo clusters in use (current/cache/total/max) 1023/376/1399/77557 9k jumbo clusters in use (current/cache/total/max) 0/0/0/43626 16k jumbo clusters in use (current/cache/total/max) 10531K/6207K/16738K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/50199/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines ... the denied number rises... and somewhere in the millions or more the machine stops --- but even with the large number of denied 9k clusters, the "9k jumbo clusters in use" line will always indicate some available. ... so is this a tuning or a bug issue? I've tried ietd --- basically it doesn't want to work with a zfs zvol, it seems (refuses to use it).