From owner-freebsd-stable@freebsd.org Sat Dec 3 01:13:27 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE9FCC637B6 for ; Sat, 3 Dec 2016 01:13:27 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.sub.de (uucp.dinoex.sub.de [IPv6:2001:1440:5001:1::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "uucp.dinoex.sub.de", Issuer "StartCom Class 1 DV Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D6881816 for ; Sat, 3 Dec 2016 01:13:26 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.sub.de (uucp.dinoex.sub.de [194.45.71.2]) by uucp.dinoex.sub.de (8.15.2/8.14.9) with ESMTPS id uB31D6KE082290 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 3 Dec 2016 02:13:07 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: from citylink.dinoex.sub.org (uucp@localhost) by uucp.dinoex.sub.de (8.15.2/8.14.9/Submit) with UUCP id uB31D6HX082289 for freebsd-stable@FreeBSD.ORG; Sat, 3 Dec 2016 02:13:06 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from gate.oper.dinoex.org (gate-e [192.168.98.2]) by citylink.dinoex.sub.de (8.15.2/8.15.2) with ESMTP id uB30a5L3047629 for ; Sat, 3 Dec 2016 01:36:06 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from gate.oper.dinoex.org (gate-e [192.168.98.2]) by gate.oper.dinoex.org (8.15.2/8.15.2) with ESMTP id uB30Y25K047182 for ; Sat, 3 Dec 2016 01:34:03 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: (from news@localhost) by gate.oper.dinoex.org (8.15.2/8.15.2/Submit) id uB30Y2UB047176 for freebsd-stable@FreeBSD.ORG; Sat, 3 Dec 2016 01:34:02 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-Authentication-Warning: gate.oper.dinoex.org: news set sender to li-fbsd@citylink.dinoex.sub.org using -f From: Peter Subject: Rel.10.3 zfs GEOM removal and memory leak Date: Sat, 3 Dec 2016 01:24:55 +0100 Organization: even some more stinky socks Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Date: Sat, 3 Dec 2016 00:25:32 +0000 (UTC) Injection-Info: oper.dinoex.de; logging-data="36162"; mail-complaints-to="usenet@citylink.dinoex.sub.org" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:42.0) Gecko/20100101 Firefox/42.0 SeaMonkey/2.39 X-Mozilla-News-Host: news://localhost:119 Sender: li-fbsd@citylink.dinoex.sub.org To: freebsd-stable@FreeBSD.ORG X-Milter: Spamilter (Reciever: uucp.dinoex.sub.de; Sender-ip: 194.45.71.2; Sender-helo: uucp.dinoex.sub.de; ) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (uucp.dinoex.sub.de [194.45.71.2]); Sat, 03 Dec 2016 02:13:08 +0100 (CET) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 01:13:27 -0000 Question: how to get ZFS l2arc working on FBSD 10.3 (RELENG or STABLE)? Problem using 10.3 RELENG: When ZFS is called the first time after boot, it will delete all device nodes of the drive carrying l2arc. ZFS itself will access it's slices by a "diskid/" string, but all other access is impossible - especially, a swapspace on the same drive (NOT under ZFS) will fail to activate: > NAME STATE READ WRITE CKSUM > gr ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > da0s2 ONLINE 0 0 0 > da1s2 ONLINE 0 0 0 > da2s2 ONLINE 0 0 0 > cache > diskid/DISK-162020405512s1e ONLINE 0 0 0 Here "diskid/DISK-162020405512s1e" equals to ada3s1e, and trying to open a swapspace on ada3s1b now fails, because that device is no longer present in /dev : > root@edge:~ # gpart show ada3 > gpart: No such geom: ada3. If we now remove the l2arc via "zfs remove gr diskid/DISK-162020405512s1e" then the device nodes magically reappear, and we can activate swapspace. Afterwards we can add the l2arc again, and it will be shown correctly as "ada3s1e" - but at the next boot the problem appears again. This problem does not exist in 10.3 STABLE, but instead there is: Problem using 10.3 STABLE: Here seems to be a memory leak: the ARC grows above its limits, while the space used is not accounted in one of [MFU MRU Anon Header Other L2Hdr]. After some time the MFU+MRU shrink to the bare minimum, and the system is all busy with arc_reclaim. The behaviour seems to be triggered by writing to l2arc.(*) Any advice on how to proceed (or which supported version might work better)? (*) Addendum: I tried to understand the phenomen, and found this on arcstats: (metadata_size + data_size) + hdr_size + l2_hdr_size + other_size = size and metadata_size + data_size = mfu_size + mru_size + anon_size + X The X is the memory leak, it does never shrink, does not disappear when all l2arc are removed, and while l2arc are written it does continually (but not linear) grow until the system is quite stuck and l2arc write ceases. Further investigations shows the growing of X being synchronous with the growing of kstat.zfs.misc.arcstats.l2_free_on_write figure.