From owner-freebsd-questions@freebsd.org Thu Jul 12 18:49:57 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C3EF1033490 for ; Thu, 12 Jul 2018 18:49:57 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [IPv6:2607:f3e0:80:80::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost2.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F06F2790CC for ; Thu, 12 Jul 2018 18:49:55 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (lava.sentex.ca [IPv6:2607:f3e0:0:5:0:0:0:11]) by smarthost2.sentex.ca (8.15.2/8.15.2) with ESMTPS id w6CIntlX000394 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 12 Jul 2018 14:49:55 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [192.168.43.26] (saphire3.sentex.net [192.168.43.26]) by lava.sentex.ca (8.15.2/8.15.2) with ESMTP id w6CInqVD092072; Thu, 12 Jul 2018 14:49:52 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: Disk/ZFS activity crash on 11.2-STABLE To: Jim Long Cc: freebsd-questions@freebsd.org References: <20180711212959.GA81029@g5.umpquanet.com> <5ebd8573-1363-06c7-cbb2-8298b0894319@sentex.net> <20180712183512.GA75020@g5.umpquanet.com> From: Mike Tancsa Openpgp: preference=signencrypt Autocrypt: addr=mike@sentex.net; prefer-encrypt=mutual; keydata= xsBNBEzcA24BCACpwI/iqOrs0GfQSfhA1v6Z8AcXVeGsRyKEKUpxoOYxXWc2z3vndbYlIP6E YJeifzKhS/9E+VjhhICaepLHfw865TDTUPr5D0Ed+edSsKjlnDtb6hfNJC00P7eoiuvi85TW F/gAxRY269A5d856bYrzLbkWp2lKUR3Bg6NnORtflGzx9ZWAltZbjYjjRqegPv0EQNYcHqWo eRpXilEo1ahT6nmOU8V7yEvT2j4wlLcQ6qg7w+N/vcBvyd/weiwHU+vTQ9mT61x5/wUrQhdw 2gJHeQXeDGMJV49RT2EEz+QVxaf477eyWsdQzPVjAKRMT3BVdK8WvpYAEfBAbXmkboOxABEB AAHNHG1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5jYT7CwHgEEwECACIFAkzcA24CGwMGCwkI BwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEJXHwM2kc8rX+sMH/2V6pTBKsQ5mpWWLgs6wVP2k BC+6r/YKNXv9Rw/PrC6+9hTbgA+sSjJ+8gxsCbJsOQXZrxF0x3l9oYdYfuKcwdwXFX1/FS8p HfBeDkmlH+dI709xT9wgrR4dS5aMmKp0scPrXPIAKiYVOHjOlNItcLYTEEWEFBepheEVsgmk GrNbcrHwOx/u4igUQ8vcpyXPyUki+BsftPw8ZQvBU887igh0OxaCR8AurJppQ5UQd63r81cX E1ZjoFoWCaGK/SjPb/OhpYpu5swoZIhOxQbn7OtakYPsDd5t2A5KhvjI8BMTnd5Go+2xsCmr jlIEq8Bi29gCcfQUvNiClevi13ifmnnOwE0ETNwDbgEIALWGNJHRAhpd0A4vtd3G0oRqMBcM FGThQr3qORmEBTPPEomTdBaHcn+Xl+3YUvTBD/67/mutWBwgp2R5gQOSqcM7axvgMSHbKqBL 9sd1LsLw0UT2O5AYxv3EwzhG84pwRg3XcUqvWA4lA8tIj/1q4Jzi5qOkg1zxq4W9qr9oiYK5 bBR638JUvr3eHMaz/Nz+sDVFgwHmXZj3M6aE5Ce9reCGbvrae7H5D5PPvtT3r22X8SqfVAiO TFKedCf/6jbSOedPN931FJQYopj9P6b3m0nI3ZiCDVSqeyOAIBLzm+RBUIU3brzoxDhYR8pz CJc2sK8l6YjqivPakrD86bFDff8AEQEAAcLAXwQYAQIACQUCTNwDbgIbDAAKCRCVx8DNpHPK 1+iQB/99aqNtez9ZTBWELj269La8ntuRx6gCpzfPXfn6SDIfTItDxTh1hrdRVP5QNGGF5wus N4EMwXouskva1hbFX3Pv72csYSxxEJXjW16oV8WK4KjKXoskLg2RyRP4uXqL7Mp2ezNtVY5F 9nu3fj4ydpHCSaqKy5xd70A8D50PfZsFgkrsa5gdQhPiGGEdxhq/XSeAAnZ4uVLJKarH+mj5 MEhgZPEBWkGrbDZpezl9qbFcUem/uT9x8FYT/JIztMVh9qDcdP5tzANW5J7nvgXjska+VFGY ryZK4SPDczh74mn6GI/+RBi7OUzXXPgpPBrhS5FByjwCqjjsSpTjTds+NGIY Organization: Sentex Communications Message-ID: Date: Thu, 12 Jul 2018 14:49:53 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20180712183512.GA75020@g5.umpquanet.com> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.83 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2018 18:49:57 -0000 On 7/12/2018 2:35 PM, Jim Long wrote: > > Yes, ARC usage is high. top shows: > > last pid: 2215; load averages: 1.50, 1.69, 1.69 up 0+00:44:59 11:02:00 > 27 processes: 2 running, 25 sleeping > CPU: 0.0% user, 0.0% nice, 9.0% system, 0.1% interrupt, 90.9% idle > Mem: 1416K Active, 11M Inact, 2788K Laundry, 91G Wired, 1236M Free > ARC: 84G Total, 68M MFU, 83G MRU, 15M Anon, 877M Header, 4078K Other > 83G Compressed, 148G Uncompressed, 1.78:1 Ratio > Swap: 16G Total, 15M Used, 16G Free > > I'm re-running the test now. I had shut off hyperthreading to go down to > 24 cores, but it's back on now. This server will be a computational resource, > not just storage, so I'd like to find a way to stablize the configuration with > hyperthreading enabled, if possible. I would leave HT on. Based on the zfs stats CACHE HITS BY DATA TYPE: Demand Data: 51.38% 457.85k Prefetch Data: 0.00% 2 Demand Metadata: 44.71% 398.41k Prefetch Metadata: 3.91% 34.86k CACHE MISSES BY DATA TYPE: Demand Data: 3.22% 17.71k Prefetch Data: 81.77% 449.51k Demand Metadata: 8.57% 47.14k Prefetch Metadata: 6.43% 35.35k If I read this right, you are not hurting too much for meta data misses. I find things like zfs list -t snapshots REALLY slow down if there is not enough caching for meta data where there are a lot of files and directories. eg. on a box that has several million small files and many many directories, I increased the amount of RAM to vfs.zfs.arc_meta_limit and it helped a LOT. Even then I still see a lot of misses (it was worse before) CACHE HITS BY DATA TYPE: Demand Data: 10.82% 5.58b Prefetch Data: 0.66% 339.34m Demand Metadata: 50.83% 26.22b Prefetch Metadata: 37.69% 19.44b CACHE MISSES BY DATA TYPE: Demand Data: 6.79% 680.20m Prefetch Data: 8.43% 844.48m Demand Metadata: 79.52% 7.97b Prefetch Metadata: 5.26% 527.45m But I dont think thats your case. I would try and set a ceiling. On RELENG_11 you dont need to reboot Try sysctl -w vfs.zfs.arc_max=77946198016 which shaves off 20G from what ARC can gobble up. Not sure if thats your issue, but it is an issue for some users. If you are still hurting for caching, an SSD drive or NVME and make it a caching device for your pool. and what does zpool status show ? ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 x203 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada