From owner-freebsd-stable@freebsd.org Wed Aug 1 04:24:32 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A05E01048CFA for ; Wed, 1 Aug 2018 04:24:32 +0000 (UTC) (envelope-from gandalf@shopzeus.com) Received: from shopzeus.com (shopzeus.com [87.229.70.149]) by mx1.freebsd.org (Postfix) with ESMTP id EACD772F67 for ; Wed, 1 Aug 2018 04:24:31 +0000 (UTC) (envelope-from gandalf@shopzeus.com) Received: from [127.127.127.127] (localhost [127.127.127.127]) (Authenticated sender: gandalf) by shopzeus.com (Postfix) with ESMTPSA id 90E26889B842 for ; Tue, 31 Jul 2018 23:17:17 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopzeus.com; s=shopzeus_com; t=1533093437; bh=hX9d3jpL/tobQJdAfB3Iy7WQ6cXR3tfBj3LksCXAiy4=; h=Subject:To:References:From:Date:In-Reply-To:From; b=Qt1slwV2UquVwTpqp4Np5mRoylGOrpp/degU6D/yF72YjvMd0XRNuBM5I9iAcwu8N ZdG61hM4Ojr3S6teu4KIGShdJ5LSSPvF71KoOPZrhOmn3BsZ88tbzcyLBpbXtHsO8o 0iJsCT5V6Jv+YUQq8MNVWti/KOhj3iFyn94ITducvgLA2xj3pYd0qNp5iBYjFh+QWt LDRckp0KzbK7HhcgBQ9evpa+o/IMUcfwqj5tpTS/glQR9DEfqZE3svHzhVXh6lIvh8 mb4zDLg5vZrxDmI0ZISKJx8AVCL6VqkkMNiN5WBqW4Ixz4HcvynJQJcH3kRJCjSv6H wLVvGogR7P46w== Subject: Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64 To: freebsd-stable@freebsd.org References: <1a039af7758679ba1085934b4fb81b57@ijs.si> <3e56e4de076111c04c2595068ba71eec@ijs.si> <41a77fe4-8acf-76d5-c5f5-2f60418352a1@ShaneWare.Biz> From: =?UTF-8?Q?Nagy_L=c3=a1szl=c3=b3_Zsolt?= Openpgp: preference=signencrypt Autocrypt: addr=gandalf@shopzeus.com; prefer-encrypt=mutual; keydata= xsFNBFYDrhgBEADgy4VcUP5RL8bNoUL7iXjiapJ/T7Tr3IKEI6VGUwXD7AEX7qwCDEYKoQdA 55g9RqTyb0C2h07vDeiQWiiV1fkZznVMuBWt7sBez2FgVlw7B+VpSArYGDm2LelMIxMk3tiZ r3U7pGA8WLk2Yxk3x3vhVnmNz6h5aMeRv4cj9xRM/hEVlxk8gleamAzp9V0EIY8S5HU8R0Ii BGev96trV/FjStxhLmv41NUv8RMaq+z32G2FmTvuCTA+BVd3Yom4riUF29yW986uLcyBWKXz tnwX65GE8Wskzt84SiHOr3fwApbfq5QURnFTMI1yFjYxpyP6f4quyvJI68psnEaTxo4FRB1F lIPqsqKRlPb/PaNNnJ0A3ofDGklzMKPKols7srqWxggoojqJDzeBbd8GVKCTZjVFndqlu1WS 76w1XHUi00vAPbMHwaiheTT8frQD9nYPlz5D9TOzjwEy7LwsQ6nXr/8AHju23CgcKiZu7TTQ BVW0JkPW20/Ms7PgGUCLRTNzqZhdQyyXPZmW4LJVPLAnD0RhPyQ0+PpL8lpZITM3i6Q+YorS /MDo7Gcm3KJTx8AClt0KmWWJ368VCu2JaLNYYrahXcz7j2Pbj7tb5jMzlYHdO9zeqoelorrO 7kUeSfQyF69J0UPdCB1Sh5y1O1JQxj7rJvsxHMJ/LqbdU3nLmwARAQABzSpOYWd5IEzDoXN6 bMOzIFpzb2x0IDxnYW5kYWxmQHNob3B6ZXVzLmNvbT7CwYcEEwEIADECGyMCHgECF4ACGQEK CwkNCAwHCwoDAgcVCgkICwIDBRYCAwEABQJZ4wlLBQkFwI6zAAoJEKRezXy+8Hg85csQANzV Xn+j94eoKZPUKlPF/6SWwgEEY/OqcmFVadNsotpJJJlw7gN0jHXv5zo+Up+1nZUS/HmPt8k/ Js0XdpZD0ELD4Nd6KusMEWj+6oOEt2mbfXobQHu1neiFsNsjSed8W6fWZq5k1m39XmydSm81 Cd8yneh9lR2iIO9NsldlXOBkodkGcevDSfaC3nv0GiHE91YaT9vIiZqluG4hbAc7rZB9GZUi QPHDDQ2opZJ3P4yNksqlyAFjoWBOZ+jcaRE/emwAQl++SgWzUGhKqoGubfFHLjpvvu1BTVHh tLUuMd5/ZiKYhmeCT9FizmlGol4cpnc0tf0LSMN8JT0oZnqlCAnPrD3KJLjkJ771TLlslxao lq/RxeJDX09kNPf6iROxMgmnSnNbZ8/wS9kthb5s8nSd4PZniV40YN98+1+s78CeKzNAt99d dZ1n+jr/AQJvzvL41SksE83kjEAQ6SPoUmzu6HGYSJ1QvBFRqj/diHoncdQ/6BeixVLQiK97 EPNRNllH/0yTBN+AODsOXA5k3xpUSsdLQzj2bQGeLZC5SZEJLucU1lE5WyOj0CvckQPRi4yE nJqd0hAbBKtvsbXPJANscRLuFZiNqyCXjD+BII7i4o/HHxhX6NMSRTnAu0AZv4gZ28zc9Pt5 zBTyATz1AwcOET1NLq+OsTPlAOYMytxIzsFNBFYDrhgBEAC5ug5iSuivG78t6EEAZzRgwYTd LmgqiJQ+Jwk9BZukVxA0mdT2FZGyY68PxJSNydO93r5B8pAz8z6Uj9aeQQ6s11JNyULCXYsW jqLgLelkAyrPMctm55JI/Mu+/pAqdU9ouzRi2kPxjRz3ehn/AU6GB3aTRyvuLarqwMSAfdyS Ct+gG2fWTMMoWtsrQhJ0qRA+8IDyHB9fmxKa370Av5dkvhotWFTgfOEHm8XvP6knfxR0Gx7h BXfwnFqMQjL3+2pqMhysliTRFIYzJQoEAIcJRFc4a2gY6aldYmrWazgcUiZyugt2V5uR/6zG LFwMkPIYFUJWslV1AmtRGty7BQXLbPstBd+iEZUpVFK0QXHQaSLDOOWyryuzj+XPKLxCY/0F zwA+0N0nyqe/KRCQkqMwDAuqaAvi5o2+oIRDpakW53Sxdf8xoPiarH3U+qlvzaO9/p/phMXT Ii4aZWbu4KQoCdqZqkE7LFAXnsRQwoSK9FkTrUhZM9DT97THD5ZEnpf7F6AA8lr7f6V5AF6o fhSOmo/FEFdfpPjIp5azOn0Vmbq+Jc2NRKHHEfyau6/M5+XkeMpT293xVQx/SyiH7Iv9uiyA mudQ3qN7S+hzCeCRSB4hR8PXcXQ3l3zckI2jlxEYBavas7ZAFw4xquhcCPwCcqfw6ZAWPqmi sX+l1tPvcwARAQABwsFlBBgBCAAPAhsMBQJZ4wlMBQkFwI60AAoJEKRezXy+8Hg8ySEQAJcL 2FciESpePw4xOR8HdZCNf0V5yNLSLmV6wni/TvYIdAOm6gnPxyq8ukfkpXTCcgcLCT1TsUez +gnv4qPY+nvLFtCnTA0FENqmWKMLYXn1xry/M4fT+FkrYmY1MfegVFuPtxgDSKaae9p/DnLI kGS9DayG9sBcg3qQlhSp+G5hf4VJ8jwdTcWRgjXwO/sFVWdMWCYL4M3Ye832oa95GroQfeVE k5ircJnAfEUVNbkQcupUa3XzQ+yIG7n4KND8TYtyUjmY5blBZ6d4tF8dSny9L+6/lUmPasHX CPWr0SrblAbthRYp5b1Jkh7jqP6nWro/xgAoQSdA0qfgPf6DQzFHZYg6L7gfzsCD8++tqljt ddVVbhzBeS+wO+Pe5QcKYnY4xFAcs13uzaIbv/6IonMKU1im72o5aYZg//LCzrbGPyxfe+Ts j2pg5AjZFxX2pfuWIq1cDf1xu+o5aNdrVkgR6WfDubLTJ+oEwZyRHB2EVD/1wVX6WfcQB7b4 v2nebCYlca2FN6G0jp39s3OQ+rp8I7XhWPQm6i/HVJwnQbcUN26Vhz7on2lJpkIPLPr+QYMB C2EpSrMPqiYVM2FI47NVMAtVUOWWW97FRI1YQaKzxfFdewtTk0M+AQd0Zhsn1Q2CBNV5qkE+ wY/YTzrtMg/ELo5on6RCdxThVUJpzuK3 Message-ID: <35c3420a-085a-cb6d-506b-8c0887cd6dc3@shopzeus.com> Date: Wed, 1 Aug 2018 06:24:23 +0200 MIME-Version: 1.0 In-Reply-To: <41a77fe4-8acf-76d5-c5f5-2f60418352a1@ShaneWare.Biz> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Language: hu X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 04:24:32 -0000 I have experienced a very similar thing. After upgrading my machine from 11.1-R to 11.2-R, the swap space is filled up to about 66% in about every 2 days. First I tought that it was PostgreSQL, and lowered the shared_buffers setting, but it only postponed the problem for another day= =2E The only thing that has changed is the OS version 11.1-R -> 11.2-R. Here is the top of top: last pid: 50425;=C2=A0 load averages:=C2=A0 0.19,=C2=A0 0.16,=C2=A0 0.17=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 up 15+23:02:21=C2=A0 06:18:18 45 processes:=C2=A0 1 running, 43 sleeping, 1 zombie CPU:=C2=A0=C2=A0=C2=A0=C2=A0 % user,=C2=A0=C2=A0=C2=A0=C2=A0 % nice,=C2=A0= =C2=A0=C2=A0=C2=A0 % system,=C2=A0=C2=A0=C2=A0=C2=A0 % interrupt,=C2=A0=C2= =A0=C2=A0=C2=A0 % idle Mem: 81M Active, 91M Inact, 1577M Laundry, 14G Wired, 226M Free ARC: 9598M Total, 90M MFU, 8715M MRU, 105K Anon, 199M Header, 594M Other =C2=A0=C2=A0=C2=A0=C2=A0 8085M Compressed, 15G Uncompressed, 1.84:1 Ratio= Swap: 4096M Total, 3103M Used, 993M Free, 75% Inuse The ARC value seem to be growing for a while, then it starts to use the swap heavily. But this might be unrelated because the swap usage does not go above 80%. (E.g. it does not crash, but it is clearly using swap when it should not.) =C2=A0 Laszlo > On 01/08/2018 07:24, Mark Martinec wrote: >> I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE >> and the situation has not improved. Also turned off all services. >> ZFS is still leaking memory about 30 MB per hour, until the host >> runs out of memory and swap space and crashes, unless I reboot it >> first every four days. >> >> Any advise before I try to get rid of that faulted disk with a pool >> (or downgrade to 10.3, which was stable) ? >> >> =C2=A0 Mark >> >> >> 2018-07-23 17:12, myself wrote: >>> After upgrading an older AMD host from FreeBSD 10.3 to 11.1-RELEASE-p= 11 >>> (amd64), ZFS is gradually eating up all memory, so that it crashes ev= ery >>> few days when the memory is completely exhausted (after swapping heav= ily >>> for a couple of hours). >>> >>> This machine has only 4 GB of memory. After capping up the ZFS ARC >>> to 1.8 GB the machine can now stay up a bit longer, but in four days >>> all the memory is used up. The machine is lightly loaded, it runs >>> a bind resolver and a lightly used web server, the ps output >>> does not show any excessive memory use by any process. > When you say all used up - you mean the amount of wired ram goes higher= > than about 90% physical ram? You can watch the wired amount in top, or > calculate it as vm.stats.vm.v_wire_count * hw.pagesize > > ZFS ARC is marked as wired, there is also vm.max_wired which limits how= > much the kernel can wire, this defaults to 30% ram, so about 1.2G for > you. It seems these two wired values don't interact and can add up to > more than physical ram. I have reported this in bug 229764 > > Try the patch at > https://reviews.freebsd.org/D7538 > it has given me the best arc related memory improvements I have seen > since 10.1, I now see arc being released instead of swap being used. >