From owner-freebsd-stable@freebsd.org Fri Mar 29 09:52:47 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FF9315614D1 for ; Fri, 29 Mar 2019 09:52:47 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0735987F77 for ; Fri, 29 Mar 2019 09:52:45 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id 048CC4D195; Fri, 29 Mar 2019 10:52:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=subject:to:references:from:message-id:date:mime-version:in-reply-to:content-type:content-transfer-encoding; s=20140709; bh=CjM5jqs2Ghla6DP4KaEHHCYZ8Ys=; b=V+AXEHZVeSX98uphvC+/a8tn8Ngjr2fAkqm4GB5xcAgp6NttYhgDVhi6MCCCLfoDioOGFLBYZ6lUOtj3DME8lCVTSfk2BaRdiPJ5dYUeYD0UDYSXIp9nkRJAr/zlPZs5ckNH/K/b0Zse5KASBMqB/MhECRxA029ZUDiqxmNNh4c= Received: from mail.bytecamp.net (mail.bytecamp.net [212.204.60.9]) by mxout01.bytecamp.net (Postfix) with ESMTP id C05914D192 for ; Fri, 29 Mar 2019 10:52:37 +0100 (CET) Received: (qmail 87396 invoked from network); 29 Mar 2019 10:52:37 +0100 Received: from unknown (HELO ?192.168.3.32?) (rs%bytecamp.net@80.84.202.251) by mail.bytecamp.net with ESMTPS (AES128-SHA encrypted); 29 Mar 2019 10:52:37 +0100 Subject: Re: 11.2-STABLE kernel wired memory leak To: freebsd-stable@freebsd.org References: From: Robert Schulze Openpgp: preference=signencrypt Autocrypt: addr=rs@bytecamp.net; prefer-encrypt=mutual; keydata= mQENBFRpvvcBCACiXR5ENQUFzwmQCyyQmpG3ppKXS5GamZHZSl6dr+yeTRHW5XJ+46ltis64 U/i1V0DhBD9kHjAvwZvWO+BXVj1kf1GFf286WoUnFvRK6oLRtaqdZDDu+AA3EcvvryRzlkE9 uSBXkg+4ObhxhaLhjWowDBhmAyi3QOAtPpw9tme4c0ktoKMMlkryTAwPXw7sJi7TGJXv088T z2C2eMBKNKwyCWmGsP+0UoZRxO0A7qCU1kIGvsf7N9sYFD3OI9NaYZHvN8g7x7NTInz77a2o /u1EFYcQ6UbphP8AhUX9phvQLsZr0w0eCFQ+kJqtHWdesRckQcpDo8eXDQOvZH5vxErzABEB AAG0IFJvYmVydCBTY2h1bHplIDxyc0BieXRlY2FtcC5uZXQ+iQE+BBMBAgAoBQJUab73Ahsj BQkJZgGABgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDDYTHqanMMhMpTB/4xAqtfANpm GYyHTt3z2250jx62GwDfK569dASQpiz1U7AwVvZDfsQSMpsIT/GkaDB5meNZVLYIvfpk1jmi CZ11b0WdERShxQhQsZCBWSIJG4jDfbbmhr212hi1rYzBisLb4+sU/V2Gdd42VUitkPItiv8X ImStSrzWiUAOmeXMEgJA6728sh692oYjsVQVXxRMwcAJSwwnlo32HRuAWK037oJ/04UoZijC chjdp8RIU2ItEjjAWQRcS+KHtj0cJurb+PE6WA8vDU18eLNVUcFRwJToa3dlSLN39R7icGuc SlN+HlCrqgqoHVMjX043wPjf3XEXtLcJTFyYcptyWjCuuQENBFRpvvcBCADhu4eObbfXhhzG 4/onzvW+g5t5k7XJMsLjGl7FAmn26CpSMTkHZCFmUR74Bz7Pr/l4RTRvFt1vnHWX/cKhcwnR g3ZId2a+xljT4pe77gV7+VQwdw5T80aMGe+ASeS0VqdzIyauZl0tzzVqh0/zcNLNE2KTo3el CENru7bOAAJ9nJ1PEVBgpE5P789duErp71HrDGG6SFdk+pFGKK41gJWktt0Tf13gikEiwsL/ xhYzuPqHV3Rj3vsIewbMLkC4Ekrsjqr4gKpTlrLhv9xOG/+SFIpdWy4pKY1dCLsR6lZXtbzE 31jXxuKGRjNlhm9eDfRfrN6LfKq2Z4Y+OZBUMWrbABEBAAGJASUEGAECAA8FAlRpvvcCGwwF CQlmAYAACgkQw2Ex6mpzDIQ0sgf/UI9D2tti9zGNZMgr9dLuMrdi+/olRo0C/PQIsw4NaBCh seXjQgTOdg99WvynZrvtt2eE1YCCnUORMyk1h8dtR3ZmFTMy9DAf7hiY99/avadM+rzoczEV WpZND88rD8JZ1WQKZtTDzH+c2wklkARo4XIu3KDqLEVHhdysN1vACEdHmi1FB3WNSLHF05Mf BUTOsyF3dEP625d03KK/91Sp31F0GJjlSTANxIM05bTcJZsG18auIIevn/19ZXSRdFAC7vy9 ugBfv0HVEC1QsarOCWpqJxQOutZI8I/AywGJetGVkTpKkTRnrIfzoBaFEJFGW96gyLSsWuZs B1zN8/OwHQ== Organization: bytecamp GmbH Message-ID: Date: Fri, 29 Mar 2019 10:52:37 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: de-DE Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 0735987F77 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bytecamp.net header.s=20140709 header.b=V+AXEHZV; spf=pass (mx1.freebsd.org: domain of rs@bytecamp.net designates 212.204.60.217 as permitted sender) smtp.mailfrom=rs@bytecamp.net X-Spamd-Result: default: False [-2.83 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.94)[-0.936,0]; R_DKIM_ALLOW(-0.20)[bytecamp.net:s=20140709]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:212.204.60.0/24]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; DWL_DNSWL_LOW(-1.00)[bytecamp.net.dwl.dnswl.org : 127.0.5.1]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_SHORT(0.72)[0.721,0]; DKIM_TRACE(0.00)[bytecamp.net:+]; MX_GOOD(-0.01)[mail2.bytecamp.net]; DMARC_NA(0.00)[bytecamp.net]; IP_SCORE(-0.00)[country: DE(-0.01)]; RCVD_IN_DNSWL_LOW(-0.10)[217.60.204.212.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:12693, ipnet:212.204.32.0/19, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2019 09:52:47 -0000 Hi, I just want to report a similar issue here with 11.2-RELEASE-p8. The affected machine has 64 GB ram and does daily backups from several machines in the night, at daytime there a parallel runs of clamav on a specific dataset. One symtom is basic I/O-performance: After upgrading from 11.1 to 11.2 backup times have increased, and are even still increasing. After one week of operation, backup times have doubled - without having changed anything else. Then there is this wired memory and way too lazy reclaim of memory for user processes: The clamav scans start at 10:30 and get swapped out immediatly. Although vfs.zfs.arc_max=48G, wired is at 62 GB before the scans and it takes about 10 minutes for the scan processes to actually run on system ram, not swap. There is obviously something broken, as there are several threads with similar observations. with kind regards, Robert Schulze