From owner-freebsd-fs@freebsd.org Sun Nov 18 21:00:11 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 623F51121760 for ; Sun, 18 Nov 2018 21:00:11 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id EBD3170072 for ; Sun, 18 Nov 2018 21:00:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id AD4AC1121757; Sun, 18 Nov 2018 21:00:10 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CB621121755 for ; Sun, 18 Nov 2018 21:00:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 024A370065 for ; Sun, 18 Nov 2018 21:00:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 295C124F4C for ; Sun, 18 Nov 2018 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wAIL09um027744 for ; Sun, 18 Nov 2018 21:00:09 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wAIL095X027736 for fs@FreeBSD.org; Sun, 18 Nov 2018 21:00:09 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201811182100.wAIL095X027736@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 18 Nov 2018 21:00:09 +0000 MIME-Version: 1.0 X-Rspamd-Queue-Id: EBD3170072 X-Spamd-Result: default: False [0.41 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_MEDIUM(0.29)[0.292,0]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; NEURAL_SPAM_SHORT(0.12)[0.118,0] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 21:00:11 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off New | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 6 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Nov 19 18:57:51 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 464FE1123A1F for ; Mon, 19 Nov 2018 18:57:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CBADE86FD0 for ; Mon, 19 Nov 2018 18:57:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 862DB1123A1E; Mon, 19 Nov 2018 18:57:50 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7468C1123A1D for ; Mon, 19 Nov 2018 18:57:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 12DF086FCB for ; Mon, 19 Nov 2018 18:57:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 400658986 for ; Mon, 19 Nov 2018 18:57:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wAJIvnt7060761 for ; Mon, 19 Nov 2018 18:57:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wAJIvnJP060759 for fs@FreeBSD.org; Mon, 19 Nov 2018 18:57:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 233277] Panic when receiving incremental ZFS send (solaris assert: db->db_buf == ((void *)0)) Date: Mon, 19 Nov 2018 18:57:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: allanjude@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: CBADE86FD0 X-Spamd-Result: default: False [0.71 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(0.26)[0.260,0]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; NEURAL_SPAM_SHORT(0.45)[0.450,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 18:57:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233277 Allan Jude changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |allanjude@FreeBSD.org --- Comment #1 from Allan Jude --- I am hitting this same assert when receiving streams with embedded blocks (= zfs send -e) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Nov 19 19:50:30 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85ED61125468 for ; Mon, 19 Nov 2018 19:50:30 +0000 (UTC) (envelope-from coco@executive-computing.de) Received: from mail.moehre.org (mail.moehre.org [195.96.35.7]) by mx1.freebsd.org (Postfix) with ESMTP id CC27F89BBE for ; Mon, 19 Nov 2018 19:50:29 +0000 (UTC) (envelope-from coco@executive-computing.de) Received: from mail.moehre.org (unknown [195.96.35.7]) by mail.moehre.org (Postfix) with ESMTP id 130923A702 for ; Mon, 19 Nov 2018 20:50:29 +0100 (CET) X-Spam-Flag: NO X-Spam-Score: -100.818 X-Spam-Level: X-Spam-Status: No, score=-100.818 tagged_above=-999 required=5 tests=[ALL_TRUSTED=-1, AWL=0.105, TW_ZF=0.077, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mail.moehre.org ([195.96.35.7]) by mail.moehre.org (mail.moehre.org [195.96.35.7]) (amavisd-new, port 10024) with ESMTP id cr5qTLt8eRQK for ; Mon, 19 Nov 2018 20:50:28 +0100 (CET) Received: from bsdbuch.c0c0.intra (p54BEC3C8.dip0.t-ipconnect.de [84.190.195.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: coco@executive-computing.de) by mail.moehre.org (Postfix) with ESMTPSA id 9EC5E3A6FB for ; Mon, 19 Nov 2018 20:50:28 +0100 (CET) Date: Mon, 19 Nov 2018 20:50:40 +0100 From: Marco Steinbach To: freebsd-fs@freebsd.org Subject: zfsd: unable to shutdown after pulling drive, high CPU load and syslog 'spamming' after rebooting Message-ID: <20181119205040.2be0f039@bsdbuch.c0c0.intra> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: CC27F89BBE X-Spamd-Result: default: False [-0.76 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.76)[-0.763,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[executive-computing.de]; NEURAL_SPAM_SHORT(0.22)[0.217,0]; MX_GOOD(-0.01)[cached: mail.moehre.org]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-0.00)[country: DE(-0.01)]; ASN(0.00)[asn:8354, ipnet:195.96.32.0/19, country:DE]; RECEIVED_SPAMHAUS_PBL(0.00)[200.195.190.84.zen.spamhaus.org : 127.0.0.10] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 19:50:30 -0000 Hi. I'm using FreeBSD 11.2-STABLE #0 r339774 amd64. For getting a better grip on zfsd and its capabilities, I plugged in a USB drive (Samsung EVO in an external case driven by a JMicron), created a zpool, a dataset, and then pulled the drive in the middle of a cp(1) operation. The cp then hangs as expected, but I'm still able to issue zpool/zfs commands, and my zroot stays fully operational -- which is great, since without zfsd I am used to any zpool command simply hanging after loosing a device, forcing me to reboot. Here's my questions: 1. Trying to shutdown(8) the system after pulling the drive does not shutdown the machine. Instead, what I see on the console is the following (transcribed from a photo I took with my mobile): [...] Writing entropy file:. Writing early boot entropy file:. 90 econd watchdog timeout expired. Shutdown terminated. [Timestamp displayed] [Timestamp] bsdbuch init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode [Timestamp] bsdbuch: syslogd exiting on signal 15 [Timestamp] init: some processes would not die; ps axl advised I've tried this a few times, and waited for increasing amounts of time, tapping and occassionally, but after that last message, nothing happens and I had to power off manually. Also, in /var/log/syslog I see the following message right before the watchdog timeout: [Timestamp] console-kit-daemon[40332]: WARNING: Error waiting for native console 3 activation: Inappropriate ioctl for device (I am using i915kms, I tried switching consoles during shutdown to no avail after seeing this message for the first time -- as expected) Is the inability to shutdown intended behaviour or am I missing something ? 2. After the machine comes back up, zfsd seems to hog one of the CPU cores, busying itself in cooperation with devd, which seems to consume about half a CPU core, with spilling a boatload of messages to syslog like so: [Timestamp] bsdbuch ZFS: vdev state changed, pool_guid=3748130264323902517 vdev_guid=2431471203435552468 [Timestamp] bsdbuch syslogd: last message repeated 505 times Replugging the USB drive, and thus making the pool available again brings the load back to normal. Is this also intended behaviour ? MfG CoCo From owner-freebsd-fs@freebsd.org Tue Nov 20 12:42:36 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C766C1127E25; Tue, 20 Nov 2018 12:42:35 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 3C2CF6C308; Tue, 20 Nov 2018 12:42:35 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 8B29817C6; Tue, 20 Nov 2018 15:42:33 +0300 (MSK) To: FreeBSD stable , FreeBSD FS Reply-To: lev@FreeBSD.org From: Lev Serebryakov Subject: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: Date: Tue, 20 Nov 2018 15:42:24 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="2FzxMLVeuKZ3OUl5Xq0tbApZLtaLIJHCu" X-Rspamd-Queue-Id: 3C2CF6C308 X-Spamd-Result: default: False [0.35 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_SHORT(0.54)[0.535,0]; NEURAL_HAM_LONG(-0.22)[-0.218,0]; NEURAL_SPAM_MEDIUM(0.03)[0.028,0]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 12:42:36 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --2FzxMLVeuKZ3OUl5Xq0tbApZLtaLIJHCu Content-Type: multipart/mixed; boundary="3NXEgeGscIkTvx5g8diqcZbnH3iAg3lgP"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: FreeBSD stable , FreeBSD FS Message-ID: Subject: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. --3NXEgeGscIkTvx5g8diqcZbnH3iAg3lgP Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable I have server which is mostly torrent box. It uses ZFS and equipped with 16GiB of physical memory. It is running 11-STABLE (r339914 now). I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago. I was used to see 13-14GiB of memory in ZFS ARC and it was Ok. Sometimes it "locks" under heavy disk load due to ARC memory pressure, but it was bearable, and as ZFS is main reason this server exists, I didn't limit ARC. But new revision (r339914) shows very strange behaivor: ARC is no more than 4GiB, but kernel has 15GiB wired: Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other= 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio It is typical numbers for last week: 15G wired, 237M Free, but only 4252M ARC! Where is other 11G of memory?! I've checked USED and FREE in "vmstat -z" output and got this: $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n | tail -20 23001088 9171456 MAP ENTRY 29680800 8404320 VM OBJECT 34417408 10813952 256 36377964 2665656 S VFS Cache 50377392 53856 sa_cache 50593792 622985216 zio_buf_131072 68913152 976896 mbuf_cluster 73543680 7225344 mbuf_jumbo_page 92358552 67848 zfs_znode_cache 95731712 51761152 4096 126962880 159581760 dmu_buf_impl_t 150958080 233920512 mbuf_jumbo_9k 165164600 92040 VNODE 192701120 30350880 UMA Slabs 205520896 291504128 zio_data_buf_1048576 222822400 529530880 zio_data_buf_524288 259143168 293476864 zio_buf_512 352485376 377061376 zio_buf_16384 376109552 346474128 dnode_t 2943016960 5761941504 abd_chunk $ And total USED/FREE numbers is very strange for me: $ vmstat -z | tr : , | awk -F , '1{u+=3D$2*$4; f+=3D$2*$5} END{print u,f}= ' 5717965420 9328951088 $ So, only ~5.7G is used and 9.3G is free! But why this memory is not used by ARC anymore and why is it wired and not free? --=20 // Lev Serebryakov --3NXEgeGscIkTvx5g8diqcZbnH3iAg3lgP-- --2FzxMLVeuKZ3OUl5Xq0tbApZLtaLIJHCu Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlv0AThfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R48fbg//RdXbTqLsT62Wmp+HSbl8JMu2TYAyXJ10elr7nhm8wptE+xNoczvvXq9W 2iy8tFBB8vcpdUaMf4MKlioI3c/+IK8aFw7QhoPJi+C1TQ4OaDfMZ6nuwRS4uHF1 /j8x6HYxBEpO4TCacOM7tjzcKVkXdFnsKWyhbqJ2y6iycQTAGzqc+xbAiaHId62x wIo1s0Y4lGpLZigWGKz90Z4IytwmUQYlQeTx14PTH8v9xf6ftpERKvzptirTRsyd MeLL67VzOEKBPOg8RVpNAr9Ti/atc7ZmmbgVfo8MsisKgn9uJEAQSrS7GGebAx19 9Ss4aNbo4EYFlsPR3tULR/pO8mnN+3pMsVqUSEbML9Mu20QTsjHhQ2AP5MWzSDGA 471WHJIPJczOh7qwyx6K0ujQMrD+t/0PDWx8G5ulX0IJxRenNfFWRqfg4gU49jB5 NjyvfOzpcqRniZ+g/M0lJpHk6ZiDXTRSqT8gULPPx+ZJitFvhJHaD6pKr1Qcwu49 TMRTtcysnLxUuV3kXBhYCllGOc0Jxg3HkWkGj7HL7v7PJABGEUyD/FyF/Qjj0Bed 7DkVkGKCb/FqcYSLP3AI8St/KJgM06t7KiTXwIZZvfEdtjVuVWIDy6yU+a6ek99h j2hn9EhsED8y2or5dGhV/jQOdoiuQcSAe+F3Q7yunjOXm4zHDv4= =QkSG -----END PGP SIGNATURE----- --2FzxMLVeuKZ3OUl5Xq0tbApZLtaLIJHCu-- From owner-freebsd-fs@freebsd.org Tue Nov 20 19:30:40 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 517AD113B2BA; Tue, 20 Nov 2018 19:30:40 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D99097E42E; Tue, 20 Nov 2018 19:30:33 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [192.168.243.12] ([192.168.243.12]) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTP id wAKJUTL6064413; Wed, 21 Nov 2018 00:30:30 +0500 (+05) (envelope-from emz@norma.perm.ru) Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. To: freebsd-fs@freebsd.org References: From: "Eugene M. Zheganin" Cc: freebsd-stable@freebsd.org Message-ID: <29d0ab29-c967-ba5e-785e-5cb1ae670c8e@norma.perm.ru> Date: Tue, 20 Nov 2018 22:30:27 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-Rspamd-Queue-Id: D99097E42E X-Spamd-Result: default: False [-0.46 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.12)[-0.124,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[perm.ru]; IP_SCORE(0.00)[country: RU(0.01)]; MX_GOOD(-0.01)[mail-backup.norma.perm.ru,mail.norma.perm.ru]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.19)[0.186,0]; NEURAL_HAM_SHORT(-0.21)[-0.212,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:57401, ipnet:2a00:7540:1::/48, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 19:30:40 -0000 Hello, 20.11.2018 15:42, Lev Serebryakov пишет: > I have server which is mostly torrent box. It uses ZFS and equipped > with 16GiB of physical memory. It is running 11-STABLE (r339914 now). > > I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago. > > I was used to see 13-14GiB of memory in ZFS ARC and it was Ok. > Sometimes it "locks" under heavy disk load due to ARC memory pressure, > but it was bearable, and as ZFS is main reason this server exists, I > didn't limit ARC. > > But new revision (r339914) shows very strange behaivor: ARC is no more > than 4GiB, but kernel has 15GiB wired: > > Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free > ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other > 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio > > It is typical numbers for last week: 15G wired, 237M Free, but only > 4252M ARC! > > Where is other 11G of memory?! > > [...] > And total USED/FREE numbers is very strange for me: > > $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}' > 5717965420 9328951088 > $ > > So, only ~5.7G is used and 9.3G is free! But why this memory is not > used by ARC anymore and why is it wired and not free? I'm getting pretty much same story on recent 11-STABLE from 9th November. Previous versions didn't have that much questions about memory usage (and I run several 11-STABLEs). Eugene. From owner-freebsd-fs@freebsd.org Tue Nov 20 20:15:41 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B70A113C630 for ; Tue, 20 Nov 2018 20:15:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9E1AE80D98 for ; Tue, 20 Nov 2018 20:15:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6085C113C62F; Tue, 20 Nov 2018 20:15:40 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EDE3113C62E for ; Tue, 20 Nov 2018 20:15:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E202B80D96 for ; Tue, 20 Nov 2018 20:15:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id A032C15C7A for ; Tue, 20 Nov 2018 20:15:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wAKKFcwa034827 for ; Tue, 20 Nov 2018 20:15:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wAKKFccC034826 for fs@FreeBSD.org; Tue, 20 Nov 2018 20:15:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 233277] Panic when receiving incremental ZFS send (solaris assert: db->db_buf == ((void *)0)) Date: Tue, 20 Nov 2018 20:15:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: pcd@delphix.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: 9E1AE80D98 X-Spamd-Result: default: False [0.35 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.29)[-0.293,0]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; NEURAL_SPAM_SHORT(0.79)[0.787,0]; NEURAL_HAM_LONG(-0.14)[-0.141,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 20:15:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233277 Paul Dagnelie changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pcd@delphix.com --- Comment #2 from Paul Dagnelie --- Does anyone have a small reproducer for this issue, or a small send stream = they could upload? We haven't seen this on Illumos or Linux, but I'd be happy to= run a reproducer to see if the same issue crops up. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Nov 20 20:24:28 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4845C113CA9E for ; Tue, 20 Nov 2018 20:24:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D4A9F81419 for ; Tue, 20 Nov 2018 20:24:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 9709E113CA9D; Tue, 20 Nov 2018 20:24:27 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 718B7113CA9C for ; Tue, 20 Nov 2018 20:24:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0C8B581414 for ; Tue, 20 Nov 2018 20:24:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 3887D15DE0 for ; Tue, 20 Nov 2018 20:24:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wAKKOQAB052752 for ; Tue, 20 Nov 2018 20:24:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wAKKOQWp052751 for fs@FreeBSD.org; Tue, 20 Nov 2018 20:24:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 233277] Panic when receiving incremental ZFS send (solaris assert: db->db_buf == ((void *)0)) Date: Tue, 20 Nov 2018 20:24:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: D4A9F81419 X-Spamd-Result: default: False [0.37 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.28)[-0.279,0]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; NEURAL_SPAM_SHORT(0.79)[0.786,0]; NEURAL_HAM_LONG(-0.14)[-0.138,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 20:24:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233277 --- Comment #3 from Alexander Motin --- I've just reproduced it in clean environment. I suspect it is related to my recent patch fixing spill block loss on block size change (https://svnweb.freebsd.org/base?view=3Drevision&revision=3D340096). Here is the reproduction scenario: # zfs create ssd/zzz # dd if=3D/dev/zero of=3D/ssd/zzz/f bs=3D4k count=3D1 1+0 records in 1+0 records out 4096 bytes transferred in 0.001220 secs (3356604 bytes/sec) # dd if=3D/dev/random of=3D/ssd/zzz/f bs=3D4 count=3D1 conv=3Dnotrunc 1+0 records in 1+0 records out 4 bytes transferred in 0.002598 secs (1540 bytes/sec) # zfs snapshot ssd/zzz@a # zfs send -e ssd/zzz@a |zfs receive ssd/xxx # dd if=3D/dev/zero of=3D/ssd/zzz/f bs=3D16k count=3D1 conv=3Dnotrunc 1+0 records in 1+0 records out 16384 bytes transferred in 0.002887 secs (5674227 bytes/sec) # dd if=3D/dev/random of=3D/ssd/zzz/f bs=3D4 count=3D1 conv=3Dnotrunc 1+0 records in 1+0 records out 4 bytes transferred in 0.000199 secs (20141 bytes/sec) # zfs snapshot ssd/zzz@b # zfs send -e -i a ssd/zzz@b | zfs receive ssd/xxx panic: solaris assert: db->db_buf =3D=3D ((void *)0) (0xfffff80116233c00 = =3D=3D 0x0), file: /sources/FreeBSD/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c, line: 1097 cpuid =3D 44 time =3D 1542744219 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe010bc1f= 800 vpanic() at vpanic+0x1a3/frame 0xfffffe010bc1f860 panic() at panic+0x43/frame 0xfffffe010bc1f8c0 assfail3() at assfail3+0x2c/frame 0xfffffe010bc1f8e0 dmu_buf_will_fill() at dmu_buf_will_fill+0x24b/frame 0xfffffe010bc1f920 dmu_buf_write_embedded() at dmu_buf_write_embedded+0xe0/frame 0xfffffe010bc1f970 dmu_write_embedded() at dmu_write_embedded+0xea/frame 0xfffffe010bc1f9d0 receive_writer_thread() at receive_writer_thread+0x513/frame 0xfffffe010bc1= fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe010bc1fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe010bc1fab0 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Nov 20 20:41:06 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18423113D45C for ; Tue, 20 Nov 2018 20:41:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A492482A20 for ; Tue, 20 Nov 2018 20:41:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 675F9113D459; Tue, 20 Nov 2018 20:41:05 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55A51113D458 for ; Tue, 20 Nov 2018 20:41:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E9A6082A14 for ; Tue, 20 Nov 2018 20:41:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 0F78015F5C for ; Tue, 20 Nov 2018 20:41:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wAKKf3tu083403 for ; Tue, 20 Nov 2018 20:41:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wAKKf3To083402 for fs@FreeBSD.org; Tue, 20 Nov 2018 20:41:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 233277] Panic when receiving incremental ZFS send (solaris assert: db->db_buf == ((void *)0)) Date: Tue, 20 Nov 2018 20:41:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mav@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: A492482A20 X-Spamd-Result: default: False [0.35 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.29)[-0.293,0]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; NEURAL_SPAM_SHORT(0.79)[0.787,0]; NEURAL_HAM_LONG(-0.14)[-0.141,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 20:41:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233277 --- Comment #4 from Alexander Motin --- Yes, r340096 is what causing this. Now need to figure out why. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Tue Nov 20 20:45:42 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D3A6113D56C; Tue, 20 Nov 2018 20:45:42 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x135.google.com (mail-it1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6FE9E82D2A; Tue, 20 Nov 2018 20:45:41 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x135.google.com with SMTP id g85so5489123ita.3; Tue, 20 Nov 2018 12:45:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=eMbHUZxy8M2nGi8S2ZDf+v6rUFOgqm9t31QXsREIh1c=; b=r9jmaEsqh5w2t+TevMnsWRijLiWUIYicnqEs+KhvLMMs7ll+5BzAQz7DCMmOXRWs8f Yyate8KMkqpkd7jtKOndVMlY62JvrCL40Veuza+LFXJVO28lR5pzx6WMJAWRfO+QSEq4 qMAE7j6jWzmBN7WjNQQPMAIuxaUOLjLeV5DIz7XaWqwBahlklMuedldNw7X+lcx/F6kh Lyxv54phSIFc6OeuCSDKnx7RTkoTAlW1801Sx47bs6ImuSf0Ao6I4BHA8CiP33kX5gnD yFk5BJ/B01L3nplyJ/vLGOhXtRl/PK5BfmLsHGQqbsrCowQf/DAhxbsw573K/hWZsQIZ 3/lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=eMbHUZxy8M2nGi8S2ZDf+v6rUFOgqm9t31QXsREIh1c=; b=mkup9hgjesgi9YXveFGcnMBs228rT3PJuwL+zKRZLd/JyEQ15VA1lqbeO6nqOjfaAy QfSTPMjcgbTvLYQTv2TVynjn+RFsxo+KswCJLqilmLlRChvhMyb3HvMJ5LNBikFGO+W2 /xykFDEXW9NEhPEq3Alg2SBdvgZ6k2V8l7yFzF9gcfezs/dxNSRbQ//JeGzKzkflFXpt 2/+2BdFLObI0ck5HXhydFDe0aHKG1U9GtGL+rtkJkoiCWb0X3TCK/WGr05irUyZFGeAs syHKtEiTH4Yjwr9yhazHG5K/+qm8yya2uIyHDiQmalxsacj1XAdFVUsVXMdPTHR4IAoJ WCrg== X-Gm-Message-State: AA+aEWYClrO4dZGM69G95t6122Uh/tEjCx5XdhkXbJ/H9AOoF6JkfM6a twiLxXQx1rrI28pWQYsfkbs24nYhTcM= X-Google-Smtp-Source: AJdET5dyc5Fzqbdi6w8yYkly9lrgQsl+0gdYwvmYMWujVXZeTzMubVZGDGwg5FGU7b1pezw+sAkljA== X-Received: by 2002:a02:9911:: with SMTP id r17-v6mr3222502jaj.132.1542746740209; Tue, 20 Nov 2018 12:45:40 -0800 (PST) Received: from raichu (toroon0560w-lp130-07-64-229-95-98.dsl.bell.ca. [64.229.95.98]) by smtp.gmail.com with ESMTPSA id v74sm13885641ita.27.2018.11.20.12.45.38 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 20 Nov 2018 12:45:39 -0800 (PST) Sender: Mark Johnston Date: Tue, 20 Nov 2018 15:45:37 -0500 From: Mark Johnston To: Lev Serebryakov Cc: FreeBSD stable , FreeBSD FS Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. Message-ID: <20181120204537.GB2630@raichu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Queue-Id: 6FE9E82D2A X-Spamd-Result: default: False [-4.85 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-2.16)[ip: (-6.71), ipnet: 2607:f8b0::/32(-2.37), asn: 15169(-1.60), country: US(-0.09)]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_IN_DNSWL_NONE(0.00)[5.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; SUBJECT_HAS_QUESTION(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2018 20:45:42 -0000 On Tue, Nov 20, 2018 at 03:42:24PM +0300, Lev Serebryakov wrote: > > I have server which is mostly torrent box. It uses ZFS and equipped > with 16GiB of physical memory. It is running 11-STABLE (r339914 now). > > I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago. > > I was used to see 13-14GiB of memory in ZFS ARC and it was Ok. > Sometimes it "locks" under heavy disk load due to ARC memory pressure, > but it was bearable, and as ZFS is main reason this server exists, I > didn't limit ARC. > > But new revision (r339914) shows very strange behaivor: ARC is no more > than 4GiB, but kernel has 15GiB wired: > > Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free > ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other > 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio > > It is typical numbers for last week: 15G wired, 237M Free, but only > 4252M ARC! > > Where is other 11G of memory?! > > I've checked USED and FREE in "vmstat -z" output and got this: > > $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n | > tail -20 > 23001088 9171456 MAP ENTRY > 29680800 8404320 VM OBJECT > 34417408 10813952 256 > 36377964 2665656 S VFS Cache > 50377392 53856 sa_cache > 50593792 622985216 zio_buf_131072 > 68913152 976896 mbuf_cluster > 73543680 7225344 mbuf_jumbo_page > 92358552 67848 zfs_znode_cache > 95731712 51761152 4096 > 126962880 159581760 dmu_buf_impl_t > 150958080 233920512 mbuf_jumbo_9k > 165164600 92040 VNODE > 192701120 30350880 UMA Slabs > 205520896 291504128 zio_data_buf_1048576 > 222822400 529530880 zio_data_buf_524288 > 259143168 293476864 zio_buf_512 > 352485376 377061376 zio_buf_16384 > 376109552 346474128 dnode_t > 2943016960 5761941504 abd_chunk > $ > > And total USED/FREE numbers is very strange for me: > > $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}' > 5717965420 9328951088 > $ > > So, only ~5.7G is used and 9.3G is free! But why this memory is not > used by ARC anymore and why is it wired and not free? Could you show the output of "vmstat -s" when in this state? From owner-freebsd-fs@freebsd.org Wed Nov 21 10:56:21 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4AA641133C85; Wed, 21 Nov 2018 10:56:21 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id BDD687A004; Wed, 21 Nov 2018 10:56:20 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:310b:1d84:68df:8590]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 326A61A12; Wed, 21 Nov 2018 13:56:19 +0300 (MSK) Date: Wed, 21 Nov 2018 13:56:18 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD Message-ID: <1999292077.20181121135618@serebryakov.spb.ru> To: Mark Johnston CC: FreeBSD stable , FreeBSD FS Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. In-Reply-To: <20181120204537.GB2630@raichu> References: <20181120204537.GB2630@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: BDD687A004 X-Spamd-Result: default: False [0.52 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_SHORT(0.26)[0.259,0]; NEURAL_HAM_LONG(-0.19)[-0.194,0]; NEURAL_SPAM_MEDIUM(0.46)[0.458,0]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2018 10:56:21 -0000 Hello Mark, Tuesday, November 20, 2018, 11:45:37 PM, you wrote: > Could you show the output of "vmstat -s" when in this state? Now ARC is 1.9GB (!!!) and 15G is still Wired: % vmstat -s 1311134466 cpu context switches 2126950424 device interrupts 45414140 software interrupts 103215017 traps 2910067754 system calls 25 kernel threads created 108756 fork() calls 50914 vfork() calls 111 rfork() calls 0 swap pager pageins 0 swap pager pages paged in 0 swap pager pageouts 0 swap pager pages paged out 4201 vnode pager pageins 51719 vnode pager pages paged in 31597 vnode pager pageouts 63028 vnode pager pages paged out 7 page daemon wakeups 1212387275 pages examined by the page daemon 0 clean page reclamation shortfalls 0 pages reactivated by the page daemon 8281789 copy-on-write faults 7909 copy-on-write optimized faults 44440180 zero fill pages zeroed 24762 zero fill pages prezeroed 2 intransit blocking page faults 63838217 total VM faults taken 3816 page faults requiring I/O 0 pages affected by kernel thread creation 3996512 pages affected by fork() 1842784 pages affected by vfork() 5499 pages affected by rfork() 88771375 pages freed 0 pages freed by daemon 21591789 pages freed by exiting processes 4936 pages active 161234 pages inactive 16674 pages in the laundry queue 3848989 pages wired down 39951 pages free 4096 bytes per page 612681203 total name lookups cache hits (95% pos + 0% neg) system 0% per-directory deletions 0%, falsehits 0%, toolong 0% % -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Wed Nov 21 11:34:49 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E00531135142; Wed, 21 Nov 2018 11:34:48 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 6A5A87B88E; Wed, 21 Nov 2018 11:34:48 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:310b:1d84:68df:8590]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 58A901A33; Wed, 21 Nov 2018 14:34:46 +0300 (MSK) Date: Wed, 21 Nov 2018 14:34:45 +0300 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD Message-ID: <123493949.20181121143445@serebryakov.spb.ru> To: Mark Johnston CC: FreeBSD FS , FreeBSD stable Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. In-Reply-To: <1999292077.20181121135618@serebryakov.spb.ru> References: <20181120204537.GB2630@raichu> <1999292077.20181121135618@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 6A5A87B88E X-Spamd-Result: default: False [0.52 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_SHORT(0.26)[0.259,0]; NEURAL_HAM_LONG(-0.19)[-0.194,0]; NEURAL_SPAM_MEDIUM(0.46)[0.458,0]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2018 11:34:49 -0000 Hello Lev, Wednesday, November 21, 2018, 1:56:18 PM, you wrote: >> Could you show the output of "vmstat -s" when in this state? > Now ARC is 1.9GB (!!!) and 15G is still Wired: > % vmstat -s > 1311134466 cpu context switches > 2126950424 device interrupts > 45414140 software interrupts > 103215017 traps > 2910067754 system calls > 25 kernel threads created > 108756 fork() calls > 50914 vfork() calls > 111 rfork() calls > 0 swap pager pageins > 0 swap pager pages paged in > 0 swap pager pageouts > 0 swap pager pages paged out > 4201 vnode pager pageins > 51719 vnode pager pages paged in > 31597 vnode pager pageouts > 63028 vnode pager pages paged out > 7 page daemon wakeups > 1212387275 pages examined by the page daemon > 0 clean page reclamation shortfalls > 0 pages reactivated by the page daemon > 8281789 copy-on-write faults > 7909 copy-on-write optimized faults > 44440180 zero fill pages zeroed > 24762 zero fill pages prezeroed > 2 intransit blocking page faults > 63838217 total VM faults taken > 3816 page faults requiring I/O > 0 pages affected by kernel thread creation > 3996512 pages affected by fork() > 1842784 pages affected by vfork() > 5499 pages affected by rfork() > 88771375 pages freed > 0 pages freed by daemon > 21591789 pages freed by exiting processes > 4936 pages active > 161234 pages inactive > 16674 pages in the laundry queue > 3848989 pages wired down > 39951 pages free > 4096 bytes per page > 612681203 total name lookups > cache hits (95% pos + 0% neg) system 0% per-directory > deletions 0%, falsehits 0%, toolong 0% > % And here top consumers. I don't like that most of te memory is in FREE abd_chunks - 8GB! USED (b) FREE (b) NAME 28785840 9299280 VM OBJECT 33817600 11413760 256 36598392 2486808 S VFS Cache 38157856 246224 arc_buf_hdr_t_l2only 44302336 629276672 zio_buf_131072 50377824 53424 sa_cache 72323072 85626880 mbuf_jumbo_page 74094592 55296 mbuf_cluster 90320896 63102976 4096 92359344 67056 zfs_znode_cache 126912480 159632160 dmu_buf_impl_t 150967296 233911296 mbuf_jumbo_9k 165164600 92040 VNODE 199124480 23927520 UMA Slabs 209715200 8388608 zio_data_buf_1048576 218103808 48758784 zio_data_buf_524288 259242496 293377536 zio_buf_512 346030080 383516672 zio_buf_16384 375592672 346991008 dnode_t 531902464 8392536064 abd_chunk -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Wed Nov 21 13:23:56 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 093F41138D32; Wed, 21 Nov 2018 13:23:56 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D6D927FA62; Wed, 21 Nov 2018 13:23:44 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id wALDNaYb093490 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 21 Nov 2018 14:23:37 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: lev@FreeBSD.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id wALDNZVR050417 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 21 Nov 2018 20:23:35 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. To: lev@FreeBSD.org, Mark Johnston References: <20181120204537.GB2630@raichu> <1999292077.20181121135618@serebryakov.spb.ru> <123493949.20181121143445@serebryakov.spb.ru> Cc: FreeBSD FS , FreeBSD stable From: Eugene Grosbein Message-ID: Date: Wed, 21 Nov 2018 20:23:29 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <123493949.20181121143445@serebryakov.spb.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: D6D927FA62 X-Spamd-Result: default: False [-5.17 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MX_INVALID(0.50)[greylisted]; RCVD_COUNT_THREE(0.00)[3]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-2.59)[ip: (-6.73), ipnet: 2a01:4f8::/29(-3.34), asn: 24940(-2.88), country: DE(-0.01)]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2018 13:23:56 -0000 21.11.2018 18:34, Lev Serebryakov wrote: > And here top consumers. I don't like that most of te memory is in FREE > abd_chunks - 8GB! > > USED (b) FREE (b) NAME > 531902464 8392536064 abd_chunk This may be a leak. Can you monitor this value for longer run? From owner-freebsd-fs@freebsd.org Wed Nov 21 14:03:28 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 904501139E61; Wed, 21 Nov 2018 14:03:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 1A498816CA; Wed, 21 Nov 2018 14:03:28 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:310b:1d84:68df:8590]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 5E7D21A8C; Wed, 21 Nov 2018 17:03:26 +0300 (MSK) Date: Wed, 21 Nov 2018 17:03:25 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD Message-ID: <228548325.20181121170325@serebryakov.spb.ru> To: Eugene Grosbein , Mark Johnston CC: FreeBSD FS , FreeBSD stable Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. In-Reply-To: References: <20181120204537.GB2630@raichu> <1999292077.20181121135618@serebryakov.spb.ru> <123493949.20181121143445@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 1A498816CA X-Spamd-Result: default: False [0.43 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE]; NEURAL_SPAM_SHORT(0.23)[0.231,0]; NEURAL_HAM_LONG(-0.23)[-0.233,0]; NEURAL_SPAM_MEDIUM(0.43)[0.433,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2018 14:03:28 -0000 Hello Eugene, Wednesday, November 21, 2018, 4:23:29 PM, you wrote: >> And here top consumers. I don't like that most of te memory is in FREE >> abd_chunks - 8GB! >> >> USED (b) FREE (b) NAME >> 531902464 8392536064 abd_chunk > This may be a leak. Can you monitor this value for longer run? No problem. But I don't think it is "true" leak, when memory is NEVER freed, as uptime of this box is 11 days already, and sometimes ARC is much larger than 2GiB. For example, now it is inflated to 8GiB. Ok, and now it is: USED (b) FREE (b) NAME 35445924 3639276 S VFS Cache 37336376 1067704 arc_buf_hdr_t_l2only 47760912 2670336 sa_cache 51531120 55440 VM OBJECT 55574528 30408704 zio_data_buf_1048576 56183040 21365760 256 61734912 50331648 zio_buf_131072 70467584 2375680 mbuf_cluster 72028160 3895296 mbuf_jumbo_page 87561672 4856808 zfs_znode_cache 110448640 32768 4096 124137120 158133600 dmu_buf_impl_t 150976512 7188480 mbuf_jumbo_9k 156588360 8539896 VNODE 208290960 1040 UMA Slabs 247484416 280662016 zio_buf_512 275775488 96993280 zio_data_buf_524288 361406864 314169856 dnode_t 417366016 1015808 zio_buf_16384 7307493376 2383060992 abd_chunk It is more likely that there is a bug in "priority" for memory distribution: abd FREE memory has higher priority than ARC somehow. -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Wed Nov 21 14:19:47 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89992113A4AE; Wed, 21 Nov 2018 14:19:47 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BBED7825D1; Wed, 21 Nov 2018 14:19:46 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id wALEJcBN093894 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 21 Nov 2018 15:19:39 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: lev@FreeBSD.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id wALEJcja052043 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 21 Nov 2018 21:19:38 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. To: Lev Serebryakov , Mark Johnston References: <20181120204537.GB2630@raichu> <1999292077.20181121135618@serebryakov.spb.ru> <123493949.20181121143445@serebryakov.spb.ru> <228548325.20181121170325@serebryakov.spb.ru> Cc: FreeBSD FS , FreeBSD stable From: Eugene Grosbein Message-ID: <92e3ea2e-b7ed-ec26-c56b-2d2acbf41772@grosbein.net> Date: Wed, 21 Nov 2018 21:19:32 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <228548325.20181121170325@serebryakov.spb.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: BBED7825D1 X-Spamd-Result: default: False [-5.14 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; MX_INVALID(0.50)[cached]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.992,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-2.55)[ip: (-6.51), ipnet: 2a01:4f8::/29(-3.35), asn: 24940(-2.88), country: DE(-0.01)]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2018 14:19:47 -0000 21.11.2018 21:03, Lev Serebryakov wrote: >>> And here top consumers. I don't like that most of te memory is in FREE >>> abd_chunks - 8GB! >>> >>> USED (b) FREE (b) NAME >>> 531902464 8392536064 abd_chunk >> This may be a leak. Can you monitor this value for longer run? > No problem. > > But I don't think it is "true" leak, when memory is NEVER > freed, as uptime of this box is 11 days already, and sometimes ARC is much > larger than 2GiB. For example, now it is inflated to 8GiB. > > Ok, and now it is: > > USED (b) FREE (b) NAME > 7307493376 2383060992 abd_chunk > > It is more likely that there is a bug in "priority" for memory > distribution: abd FREE memory has higher priority than ARC somehow. OTOH, if "abd_chunks" part of ZFS ARC is not released to the UMA from which it was allocated (to be reused by other parts of the kernel) but is reused by ZFS itself, then top(1) should report it as part of ZFS ARC and not as other "Wired" memory. From owner-freebsd-fs@freebsd.org Wed Nov 21 14:53:15 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8F69113B4EC; Wed, 21 Nov 2018 14:53:14 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id 7241483BB0; Wed, 21 Nov 2018 14:53:14 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:310b:1d84:68df:8590]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 9E2821ABC; Wed, 21 Nov 2018 17:53:06 +0300 (MSK) Date: Wed, 21 Nov 2018 17:53:06 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD Message-ID: <807023120.20181121175306@serebryakov.spb.ru> To: Eugene Grosbein , Mark Johnston CC: FreeBSD FS , FreeBSD stable Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. In-Reply-To: <92e3ea2e-b7ed-ec26-c56b-2d2acbf41772@grosbein.net> References: <20181120204537.GB2630@raichu> <1999292077.20181121135618@serebryakov.spb.ru> <123493949.20181121143445@serebryakov.spb.ru> <228548325.20181121170325@serebryakov.spb.ru> <92e3ea2e-b7ed-ec26-c56b-2d2acbf41772@grosbein.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 7241483BB0 X-Spamd-Result: default: False [0.27 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_SHORT(0.08)[0.084,0]; NEURAL_HAM_LONG(-0.23)[-0.235,0]; NEURAL_SPAM_MEDIUM(0.42)[0.416,0]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2018 14:53:15 -0000 Hello Eugene, Wednesday, November 21, 2018, 5:19:32 PM, you wrote: >> It is more likely that there is a bug in "priority" for memory >> distribution: abd FREE memory has higher priority than ARC somehow. > OTOH, if "abd_chunks" part of ZFS ARC is not released to the UMA > from which it was allocated (to be reused by other parts of the kernel) > but is reused by ZFS itself, then top(1) should report it as part of ZFS ARC > and not as other "Wired" memory. It will be better to system use this memory for ARC and not as "Free" chunks for future usage :-) -- Best regards, Lev mailto:lev@FreeBSD.org From owner-freebsd-fs@freebsd.org Wed Nov 21 18:19:48 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9949E1140E20 for ; Wed, 21 Nov 2018 18:19:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2F9466ED50 for ; Wed, 21 Nov 2018 18:19:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id E6F031140E1F; Wed, 21 Nov 2018 18:19:47 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D57911140E1E for ; Wed, 21 Nov 2018 18:19:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 74EA76ED4B for ; Wed, 21 Nov 2018 18:19:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8A3D111B8 for ; Wed, 21 Nov 2018 18:19:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wALIJk53079141 for ; Wed, 21 Nov 2018 18:19:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wALIJk6p079133 for fs@FreeBSD.org; Wed, 21 Nov 2018 18:19:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 233277] Panic when receiving incremental ZFS send (solaris assert: db->db_buf == ((void *)0)) Date: Wed, 21 Nov 2018 18:19:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: 2F9466ED50 X-Spamd-Result: default: False [0.58 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.12)[-0.124,0]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; NEURAL_SPAM_SHORT(0.84)[0.838,0]; NEURAL_HAM_LONG(-0.14)[-0.138,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2018 18:19:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233277 --- Comment #5 from commit-hook@freebsd.org --- A commit references this bug: Author: mav Date: Wed Nov 21 18:18:57 UTC 2018 New revision: 340737 URL: https://svnweb.freebsd.org/changeset/base/340737 Log: Revert r340096: 9952 Block size change during zfs receive drops spill blo= ck It was reported, and I easily reproduced it, that this change triggers pa= nic when receiving replication stream with enabled embedded blocks, when short file compressing into one embedded block changes its block size. I am not sure that the problem is in this particuler patch, not just triggered by = it, but since investigation and fix will take some time, I've decided to reve= rt this for now. PR: 198457, 233277 Changes: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Nov 22 13:33:34 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 451781148E72; Thu, 22 Nov 2018 13:33:34 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [IPv6:2a01:4f8:201:6350::2]) by mx1.freebsd.org (Postfix) with ESMTP id 768FF76FD4; Thu, 22 Nov 2018 13:33:33 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 039851D06; Thu, 22 Nov 2018 16:33:31 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. To: Mark Johnston Cc: FreeBSD FS , FreeBSD stable References: <20181120204537.GB2630@raichu> From: Lev Serebryakov Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGCBBMBCAAsAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEFAlKbP8wFCQlmJwEACgkQ6rA8WL/cR4/6VBAAjRMyyX3PBFx/ HxyiIZ698EfwlWUua8Ft4crtrdK52m0qNkbBB9BH8xQgBHG32A1CwyzQnzxHgZuoOWMjh+Qq WJv7dmpM/q/c1GCJHhlPgewXrciTwpAamZILN071u+1GCPWwGRPzfQ/U+k63KJWx9ozf4doM WTTom6Cqcssi4J1u5kkt52a5ZRhsCK9pEVGilk36XTP9BakGrnMSIxF/NK4xeZVX2q+Nuqvf RchyofKXVgLEDLwb1cd/baLtBpDzy0PTN2Zl2lX4kOA6jwTKsqRya9A1Vui1KXwPh2XViTQ1 7Y3l5qg/M+sR73DohezP6bO6huOnLhty17jAqHPNlD6RonDo+j8uIlEg4iMSTN3MhzkBAu0Q pe3ucQ0o1767JiXN3fsNvRzSFhLVNDqPLce4uKlMogsbreXWvdgHGTN1ybOHGbybZnP77yHz uNBacbmG3vL/OLXMqwLdL2JXoiec4DmXjjCdhTBl5xLV9Hz/6VWKqElteg8QFVvHB3tHWzJ4 /rpiVEixytCIII6DS33BXZ0h2EOkK/6AYA2SJxy1vgOH4SZBtDBHoezmHV2nFnq5O0c7AuAB 7WPWgQG0sEwHQPZmg/baRGitRJnaxf/Gvf1DeD1x1VrcoVke2vwBcgDM3kugP8L9hsqic2D3 dI+gP76haeuvNNZr3y9L9zvOwU0EUpsaSwEQALRr3B+OjY/cnJPstz5CVsVWyEZtJtrNviZr tBgbkhlkPm98sEWR4+gbpyeufdYJengDjeGzMDKcLB7h5fICS/j6A8XdlJ40TlbPfNgb6OHa ebaIYKTJpXKR9sD7ZyGivYMofm0em40wGUX7BIkdkomaWj+wUiS0CdXU0FWDj9wv73+Eim+X zZyXeFgIPv97v+pET7DfwKkADOfrkW9s4OfvGVjd+wm35wc8EngQEz0qdPBxx74X7vZFAxlA SXu8gDBJGYt2Bkc3QwULnfeXrZJWgqNPR5o44gGu96yaiOFaN/C6CJtev5ZEX+0ZxbvsHHB7 Z5AtsRURKpZ4w5HFHGhzHtDtoAKgeZ/gbhTVXPHvNQR818eN+Nl5BV8BRF/8yhR6VlJb8GYw h8oKDeVGVYC34+raHZQAM9WoBnN7jlt4T9zzPwtmw5mIahGFgvw1KDr7OItN2ZgtZ20UYC5m Go602nmHq0aPbU6SwGi1xohrliNsKaaciYiMaVIGRQq8iGr9Fe2HlvaA3BpB275i/gCVlUdG y5XLAv+yQMUvn5Z7XVsMroxDk/O+ae1ElyBvKiKyfWGJXTg5XUukkkyQmfWPxWUGoNA1P/P4 GMHSu7/Rqe/7m4uPu/RyTTqsSjjKJdP9kBwEzvqPtXsVoZuShtrptRQJDYflhgE4qmKSMKen ABEBAAHCwWUEGAECAA8FAlKbGksCGwwFCRLMAwAACgkQ6rA8WL/cR48RkA//SNzeW3CI8KHx rA0aeHW6Nb5ieoqVRBGLyjBM06RX6vHB9v4dJL6Z+yV2jGN2s+XZX2HILbuTOwcTxGkI3xTT e0cDXVaF5K8R/liigUjtwuC2v/sWgoWyUmK1Cy9CPYdcXmFq6nESfkUe8DYiGOUULdHq5w63 F53yOZ72iXRBQBZgkhPtRFu4lPYIzOsMag9DIJ9CthR1r0ziqU/keb94Qt3l+aXK7CwGdY7X T4zUIMHNYsuAuyX+NJIXfsN68TT6m7QmlUwxPs13nxmoVQzm4ruV+hlQKh1MtbsjWRkNgPxF IPiqoAEhy8QoddlSvRTwL5Z7zFQiwMdiXU7toL8pfzj/zJR1jELXKMipijrt5MLrV8XX3OPN yZZvh95VIl8mv+iAqwSZUufd2EJnvj5TObB0eH+a+34NWf/XqA3fPjE6KHzmdnw9PZjPEjlx JCPECSs+6gse1+GaEfKYuXzB/ENe2ctlcfx5iQJXFc+/+zG/uU/JX/pXJHA12CUfB5g7lH6X BZIHvRo3VTCDjXgbF5xxDAe5V4exf8d4oSNjQIFLYxxN7zkvH89EN6RPfRgsWN7bYArCwfS9 MOgs9pFeCOewR6qieK150aoqNENGfKFXJup+5VVl6I0mU+j0rgVDZDht2/QgP/Tb4lGBe+ai pOGaK/GYNR+Ad6bUmokKsx4= Organization: FreeBSD Message-ID: <6c41605a-81b4-c436-6114-98f1c802fbd3@FreeBSD.org> Date: Thu, 22 Nov 2018 16:33:26 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: <20181120204537.GB2630@raichu> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="nqLE7tjI1KuD3G9rbxMAvS8uhOHDiUvcO" X-Rspamd-Queue-Id: 768FF76FD4 X-Spamd-Result: default: False [1.48 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_SPAM_LONG(0.48)[0.482,0]; NEURAL_SPAM_SHORT(0.34)[0.338,0]; NEURAL_SPAM_MEDIUM(0.66)[0.664,0]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2018 13:33:34 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --nqLE7tjI1KuD3G9rbxMAvS8uhOHDiUvcO Content-Type: multipart/mixed; boundary="sMcdeZmo5JPzFDejptSBbyacLa6uXWwKb"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: Mark Johnston Cc: FreeBSD FS , FreeBSD stable Message-ID: <6c41605a-81b4-c436-6114-98f1c802fbd3@FreeBSD.org> Subject: Re: Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore. References: <20181120204537.GB2630@raichu> In-Reply-To: <20181120204537.GB2630@raichu> --sMcdeZmo5JPzFDejptSBbyacLa6uXWwKb Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 20.11.2018 23:45, Mark Johnston wrote: >> Where is other 11G of memory?! >> >> I've checked USED and FREE in "vmstat -z" output and got this: >> >> $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n | >> tail -20 >> 23001088 9171456 MAP ENTRY >> 29680800 8404320 VM OBJECT >> 34417408 10813952 256 >> 36377964 2665656 S VFS Cache >> 50377392 53856 sa_cache >> 50593792 622985216 zio_buf_131072 >> 68913152 976896 mbuf_cluster >> 73543680 7225344 mbuf_jumbo_page >> 92358552 67848 zfs_znode_cache >> 95731712 51761152 4096 >> 126962880 159581760 dmu_buf_impl_t >> 150958080 233920512 mbuf_jumbo_9k >> 165164600 92040 VNODE >> 192701120 30350880 UMA Slabs >> 205520896 291504128 zio_data_buf_1048576 >> 222822400 529530880 zio_data_buf_524288 >> 259143168 293476864 zio_buf_512 >> 352485376 377061376 zio_buf_16384 >> 376109552 346474128 dnode_t >> 2943016960 5761941504 abd_chunk >> $ >> >> And total USED/FREE numbers is very strange for me: >> >> $ vmstat -z | tr : , | awk -F , '1{u+=3D$2*$4; f+=3D$2*$5} END{print u= ,f}' >> 5717965420 9328951088 >> $ One more datapoint: I needed to reboot system and right after reboot= ARC becomes 14GiB (what was expected) and now it slows defalate, though I didn't get more "Free" memory. So, ARC becomes smaller and smaller after first fill-up, but all memory is "Wired". --=20 // Lev Serebryakov --sMcdeZmo5JPzFDejptSBbyacLa6uXWwKb-- --nqLE7tjI1KuD3G9rbxMAvS8uhOHDiUvcO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlv2sCZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R49uzQ/+OaCSQrUCF1+3crHJJqygHkvy3vdAplEpZJG3QfXOniLnX/9n3J95gGGm CXS1n2zTLy+KzEXTR3slNRqZsuK6rXYHfN6+7J4N2xmsMaYrEDudieAc10uhu2/8 fsjw19I5fPCEzVhu3cL6td2yvY/KndRCrUrCp1llTZ8WSumWty2oxG+e4YBsWLKb GWNP4sTb7ut4PthPm5ZGRnuFBIOjp7dGfSPp6BrN8+k0OVtLJqHt3F88ZHcaQ8nz s7KR0F7UzKcF2bI39NCfOW6zH8dOcUQzTt+oJ6C3G8YCgHKxZt77rclfZzfpQQLp arOGbHAcWq85+7kgyeBNKLUUO2PFnSXKTsO2rwH3ibUbCjmFZF0Cv5DlrUW/BLCd MZgohUewgPYU9j5TCx7XsCS1DwRPY9dDIUpavBgZRXfMDYq7KEsVlFpZT6c7bt/C NR6M1LIRCcq81XnlM2ur3fUlAOh2SUgHAWXwwgR7iI/tN+ZHJL3SXVpGYgTiZzpB JBnYkKaXbktK617HwmZKV2/OXDX4bwIHqhjMcxM/oWpdHRxDwKYuMEuZnwxVAxqX OVk3mhdOeTFVEvMYn/S2mr5S+m8jLwilh2qFDbtHsnMPAoETM6q73jIiVDfpkPzb T9AEiUrc7CKnXT3oXJEFLZZdhmdfePArn69X6IRPSpcHZ2LdHG4= =C1fk -----END PGP SIGNATURE----- --nqLE7tjI1KuD3G9rbxMAvS8uhOHDiUvcO-- From owner-freebsd-fs@freebsd.org Thu Nov 22 16:07:04 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 854A6114B49D for ; Thu, 22 Nov 2018 16:07:04 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-yw1-xc33.google.com (mail-yw1-xc33.google.com [IPv6:2607:f8b0:4864:20::c33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D76107B77D for ; Thu, 22 Nov 2018 16:07:03 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-yw1-xc33.google.com with SMTP id x2so3787918ywc.9 for ; Thu, 22 Nov 2018 08:07:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:subject:message-id:date :to; bh=v/O1gjXAdCxfz0tnloImFqcP+yBEaPbW8+OeHftW+gg=; b=Y06GA/GnBPddMdWkoEf3Z5nKEwiaIAYY/G7Jtzx7w639PJzhCT912f0Rv3v8sclFnV Xd8MructJrryKeeYPx1D+KAQC/BfH6XuqsIwJnkxCeE+PDm35Hthqg7lYVH8FKFBBQJQ fs4VQgJr1F8TfSkdH/YX1XS+iPdH0M5RgtMaQKMgbVk2pI3npswYea3aQA5P5VOZgW/t EIifIHmCNGChZijwNAM23EBBkNT5ULodgqOKdgZMpVRc0aGaJO3p+9ROOMUH91JRxskv jo4xnFcGyQVtldUswT0jacZKp0MJorueCOu4t8gPqdY6TerU5bS5Gh42lWQHloYlg8/o dYrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=v/O1gjXAdCxfz0tnloImFqcP+yBEaPbW8+OeHftW+gg=; b=cn530TNrLN49sgvMLlEpDsZaG30a0csbji6TY6p7RBGBAUrefevlq9tsm3RD5IUBJX f9TTx1jJG/wJJ8m/OQTI579YWE9+YNtRhnr9iqz/nHESZT+7OQcljtjl1VBHZr4W4aOU 1hvCSCcNCQpQyGrWJ05IgLBvUuEJnbjNt2A0ekfbkfxNX/oF1SlnuxtuTzp3HHA7IJ9q 8xHaS+xDhYxdQp68yEb9f+AxJbjRzjZxLP9V6yTUL4EutZCzh7f2YVbOICU0DIrc2dm0 tWgKWcCYbSN+TgyIOqNy4vjg/XGIQtrgAYvOP3BAz7oQ0QktJg9HxMi5XRD/rKl5LGkL K9XQ== X-Gm-Message-State: AGRZ1gK6hHFVhMDnRLLIOQe0oTUgbixiDrhJVt/a2GvlYr0/A4Sckamo 2gfGNfR/jGhC4nFDGM8RM73nmIkm X-Google-Smtp-Source: AJdET5fDxvRhoYDieQL0LawostBhn1cMH2ATfKnpRpP1gI68lEMZYzjK6uj0Ln06s2pHWfafM8X/jg== X-Received: by 2002:a81:6c51:: with SMTP id h78mr11931368ywc.116.1542902823245; Thu, 22 Nov 2018 08:07:03 -0800 (PST) Received: from bens-mac.local (amontpellier-652-1-72-201.w109-210.abo.wanadoo.fr. [109.210.55.201]) by smtp.gmail.com with ESMTPSA id g70sm1625371ywe.105.2018.11.22.08.07.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 22 Nov 2018 08:07:02 -0800 (PST) From: Ben RUBSON Content-Type: text/plain; charset=us-ascii; delsp=yes; format=flowed Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: [ZFS] Metadata Allocation Classes Message-Id: <82E338C0-09E3-4648-B49B-C028A03B71D7@gmail.com> Date: Thu, 22 Nov 2018 17:06:59 +0100 To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: D76107B77D X-Spamd-Result: default: False [-4.16 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.91)[-0.910,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-0.74)[ipnet: 2607:f8b0::/32(-2.10), asn: 15169(-1.51), country: US(-0.09)]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[3.3.c.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2018 16:07:04 -0000 Hi, Intel has pushed the "Metadata Allocation Classes" feature to ZFS on Linux. This allows to store metadata on a so called "special" VDEV, made of SSD for instance. This looks really promising. I wrote a patch 2 years ago which pins metadata in L2ARC, at eviction time. This really helps with some specific workloads. But "Metadata Allocation Classes" should be one step further in terms of performance. Simple question, do you plan to backport this feature to FreeBSD ? Or do you first wait for it to reach Illumos (does not seem to be there yet) ? ZoL discussion : https://github.com/zfsonlinux/zfs/issues/3779 ZoL PR : https://github.com/zfsonlinux/zfs/pull/5182 Thank you very much ! Ben From owner-freebsd-fs@freebsd.org Fri Nov 23 01:20:34 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D01DF113685B; Fri, 23 Nov 2018 01:20:33 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "land.berklix.org", Issuer "land.berklix.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D92BB706A7; Fri, 23 Nov 2018 01:20:31 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from mart.js.berklix.net (p2E52CE62.dip0.t-ipconnect.de [46.82.206.98]) (authenticated bits=0) by land.berklix.org (8.15.2/8.15.2) with ESMTPSA id wAN1HgT1047482 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 23 Nov 2018 01:17:53 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id wAN1Hcks051877; Fri, 23 Nov 2018 02:17:38 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id wAN1HKAT037185; Fri, 23 Nov 2018 02:17:32 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201811230117.wAN1HKAT037185@fire.js.berklix.net> To: soralx@cydem.org cc: freebsd-hackers@freebsd.org, freebsd-fs@freebsd.org Subject: Re: [bug] fsck refuses to repair damaged UFS using backup superblock From: "Julian H. Stacey" Organization: http://berklix.eu BSD Unix Linux Consultants, Munich Germany User-agent: EXMH on FreeBSD http://berklix.eu/free/ X-From: http://www.berklix.eu/~jhs/ In-reply-to: Your message "Tue, 20 Nov 2018 05:30:00 -0800." <20181120053000.56fbee6b@mscad14> Date: Fri, 23 Nov 2018 02:17:20 +0100 X-Rspamd-Queue-Id: D92BB706A7 X-Spamd-Result: default: False [-2.39 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.978,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.95)[-0.953,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[berklix.com]; AUTH_NA(1.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[land.berklix.com,slim.berklix.com]; NEURAL_HAM_SHORT(-0.88)[-0.882,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[98.206.82.46.zen.spamhaus.org : 127.0.0.10]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:24940, ipnet:144.76.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-0.47)[ipnet: 144.76.0.0/16(0.56), asn: 24940(-2.91), country: DE(-0.01)] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 01:20:34 -0000 Hi soralx@cydem.org, Added cc: to ensure file system specialists see this. Reference: > From: > Date: Tue, 20 Nov 2018 05:30:00 -0800 soralx@cydem.org wrote: > > Howdy! > > Since send-pr(1) is now gone, I guess the next option is to send a > message directly to the developers... > > Yesterday, I ran into a bug in fsck_ffs that gave me a little scare. > > Short story: on -CURRENT, fsck refuses to check a FS with a corrupted > superblock, even when an alternate (backup) SB location is given. > > Long story. I've been testing a newly-built system based on an X399 > platform with a 2950X CPU and an Optane 905P 480GB U.2 drive. The > system ran a ~2-day old -CURRENT; when compiling newest world and > kernel, I found the machine in a locked-up state. After a hard reset, > boot failed because the root FS became corrupted & was not available: > kernel: Superblock check-hash failed: recorded check-hash XXX != computed check-hash YYY > > I have not yet figured out why the corruption happened... bad hardware? > bug in the NVMe driver? > > "OK", I thought, "No worries. We'll just boot using another disk, fsck > the corrupted FS with a backup superblock, and be up in a moment". > The machine was doing nothing but compiling, so no valuable data loss. > > So I did `dumpfs -m /dev/ada0p3` on the spare disk (which was the > source for the new disk image, thus had almost identical partitions > and filesystems) to get the FS details, then did `newfs -N [...] > /dev/ada0p3` to find locations of superblock backups, then finally > ran `fsck_ffs -b 192 /dev/nvd0p3` -- only to get the same "check- > -hash failed" message, plus another strange message: "Can't open > /dev/nvd0p3: [...]". Then fsck quits. > Note that `fsck_ffs -b ...` on a FS with good superblock works OK. > > After fiddling with a debugger for a bit, I commented out the line > "return (0);" in /usr/src/sbin/fsck_ffs/setup.c:136, recompiled fsck, > and the FS was recovered successfully. > > What was actually happening: fsck's setup.c calls ufs_disk_fillout() > from libufs' type.c, which in turn calls sbread() from the same > library, which then calls sbget(disk->d_fd, &fs, -1) [[where '-1' > is hard-coded to indicate the primary superblock]] that then simply > invokes ffs_sbget from ffs kernel driver -- and this returns ENOENT, > which eventually causes fsck to give up before even looking at the > specified backup superblock. > > I don't know what exactly ufs_disk_fillout() does, but fortunately > for me, fsck worked without the "sbread(disk)" part of that function > having much luck on a disk with corrupted superblock. Also, I have a > feeling that calling a kernel's ffs driver function when using fsck > to fix a broken filesystem is not the best thing to do... > > Please CC, as I am not subscribed. > > -- > [SorAlx] ridin' VN2000 Classic LT > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > Cheers, Julian -- Julian Stacey, Computer Consultant, Systems Engineer, BSD Linux Unix, Munich. Brexit referendum stole 3,700,000 votes from Brits abroad, inc. 700,000 in EU UK PM lied it's democratic in Article 50 http://exitbrexit.uk/brexit/#lie Campaign lies, criminal funded; Markets, jobs & pound down; New Referendum! From owner-freebsd-fs@freebsd.org Fri Nov 23 19:10:19 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1F38114F031; Fri, 23 Nov 2018 19:10:19 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "land.berklix.org", Issuer "land.berklix.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D69BA70A6D; Fri, 23 Nov 2018 19:10:17 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from mart.js.berklix.net (p2E52CE62.dip0.t-ipconnect.de [46.82.206.98]) (authenticated bits=0) by land.berklix.org (8.15.2/8.15.2) with ESMTPSA id wANJ7eri013764 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 23 Nov 2018 19:07:44 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id wANJ7ZKi058456; Fri, 23 Nov 2018 20:07:36 +0100 (CET) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.14.7/8.14.7) with ESMTP id wANJ7HYf058666; Fri, 23 Nov 2018 20:07:29 +0100 (CET) (envelope-from jhs@berklix.com) Message-Id: <201811231907.wANJ7HYf058666@fire.js.berklix.net> cc: soralx@cydem.org, freebsd-fs@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: [bug] fsck refuses to repair damaged UFS using backup superblock From: "Julian H. Stacey" Organization: http://berklix.eu BSD Unix Linux Consultants, Munich Germany User-agent: EXMH on FreeBSD http://berklix.eu/free/ X-From: http://www.berklix.eu/~jhs/ In-reply-to: Your message "Fri, 23 Nov 2018 02:17:20 +0100." <201811230117.wAN1HKAT037185@fire.js.berklix.net> Date: Fri, 23 Nov 2018 20:07:17 +0100 X-Rspamd-Queue-Id: D69BA70A6D X-Spamd-Result: default: False [3.05 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.40)[-0.397,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.76)[0.756,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[berklix.com]; AUTH_NA(1.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: land.berklix.com]; NEURAL_SPAM_LONG(0.27)[0.267,0]; MISSING_TO(2.00)[]; R_SPF_NA(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[98.206.82.46.zen.spamhaus.org : 127.0.0.10]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:144.76.0.0/16, country:DE]; IP_SCORE(-0.46)[ipnet: 144.76.0.0/16(0.54), asn: 24940(-2.84), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2018 19:10:19 -0000 "Julian H. Stacey" wrote: > Hi soralx@cydem.org, > Added cc: to ensure file system specialists see this. > > Please CC, as I am not subscribed. An over active spam filter bounced my CC to soralx@cydem.org so unless he/she thinks to subscribes, a chance any other CC may not be seen. Cheers, Julian -- Julian Stacey, Computer Consultant, Systems Engineer, BSD Linux Unix, Munich. Brexit referendum stole 3,700,000 votes from Brits abroad, inc. 700,000 in EU PM lied it's democratic in Article 50 http://exitbrexit.uk/brexit/#lie Campaign lies, criminal funded; Markets, jobs & pound down; New Referendum! From owner-freebsd-fs@freebsd.org Sat Nov 24 20:04:17 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 803481137A49 for ; Sat, 24 Nov 2018 20:04:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1598E8A2ED for ; Sat, 24 Nov 2018 20:04:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id CCDE41137A46; Sat, 24 Nov 2018 20:04:16 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9D021137A45 for ; Sat, 24 Nov 2018 20:04:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 520A68A2EA for ; Sat, 24 Nov 2018 20:04:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 94EC27959 for ; Sat, 24 Nov 2018 20:04:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wAOK4FSs042613 for ; Sat, 24 Nov 2018 20:04:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wAOK4FZq042612 for fs@FreeBSD.org; Sat, 24 Nov 2018 20:04:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 232738] ZFS panics on device removal (solaris assert: cvd->vdev_ashift == spa->spa_max_ashift) Date: Sat, 24 Nov 2018 20:04:15 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: allanjude@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-Rspamd-Queue-Id: 1598E8A2ED X-Spamd-Result: default: False [2.24 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_SPAM_LONG(0.78)[0.780,0]; NEURAL_SPAM_MEDIUM(0.73)[0.732,0]; ASN(0.00)[asn:10310, ipnet:2001:1900:2254::/48, country:US]; NEURAL_SPAM_SHORT(0.73)[0.733,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2018 20:04:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D232738 Allan Jude changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |allanjude@FreeBSD.org --- Comment #1 from Allan Jude --- In my testing I was only seeing this when I tried to remove a device that w= as recently added. If you export/import the pool before trying to remove it, it doesn't seem to have this issue. --=20 You are receiving this mail because: You are on the CC list for the bug.=