From owner-freebsd-stable@freebsd.org Tue Feb 12 16:34:54 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6E3D14EC1AD for ; Tue, 12 Feb 2019 16:34:53 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x12e.google.com (mail-it1-x12e.google.com [IPv6:2607:f8b0:4864:20::12e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DDF2F6D45F for ; Tue, 12 Feb 2019 16:34:52 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x12e.google.com with SMTP id r11so8685811itc.2 for ; Tue, 12 Feb 2019 08:34:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=NSPpygq0DO9X63ZV9JVE4Y5cN8VOj/WHNq/g89YjciE=; b=ZzObx2B2DMRlNgWzO3V79uzQthEBwjx0xsTs9VqxSIrtCp47jry6f15rCa46Y6yCll Ai4DT4nFyDdmpA9qJ4uvxmSb6S/PXqGawzaBv8nWArAz2I/e0x1X6l88hAkxMM4cx/UD M+N1heilZE2mnpZC8oaINak+gpo4MXCZB4+8k2FHp65srzgO+EWBwZNwaq7KqyYFjxbD t5/pz21vFSSgVgrDODBkerHNGKIfR20xCUtRHTGXqH9sWUykKe/9tzE2t4dmH968mbgt ovE3dXkk/UAoiM02MQ3ayB/D/c5Gh6nJ+9vKyGXs8R6FtKonk5+VKksv9zVe+i/WhbV2 1B5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=NSPpygq0DO9X63ZV9JVE4Y5cN8VOj/WHNq/g89YjciE=; b=XiV+jboncfZZffijjdBf6tHWa/ilsr4Sz4279bqi3JI1sLR8elghqojPTfOL665o2F X8wNxIs/emRaz2BaagR1mK8oua8r+ot/XI3TcZ9u/evNM8Ph4YLZ92VLWyxTHsU/SElb 4iFnKBj5rj/LgKKXXLMD9WNzKhsUr6bDQFqkaKeHVQ5a0hJkmOTK+UDDPZBPD0j6SwA8 fJDRwNycWoXN+QOxFvFnA0x5RnoJeOiz+FQTeeFfN71ZyDlq9Luj15F3HXGkj0O2uZnH 514uhikWFLGeg6gnBsTmqkp208BU5kIndGnzIjXKrKtfuRf9VCwFbt3x5JDafKDyRTva oHKg== X-Gm-Message-State: AHQUAuYGWAwBh0B11iC4BtAKnXOhBRFpB5zeIRnv/CVDt6/Rm4Pp/56w aOnqDtoeMCZRkdPF5cwDefp3JeCN X-Google-Smtp-Source: AHgI3IZjHlW5iI/N/Epfw53WamJ8yq4vhPrA81hhqTLCUtp3VJOjGBdHQkdbez0fcpzwYKJ+MJXzcQ== X-Received: by 2002:a5d:9a98:: with SMTP id c24mr2650105iom.227.1549989292054; Tue, 12 Feb 2019 08:34:52 -0800 (PST) Received: from raichu (toroon0560w-lp140-01-69-159-36-102.dsl.bell.ca. [69.159.36.102]) by smtp.gmail.com with ESMTPSA id u3sm1607470ita.18.2019.02.12.08.34.50 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Feb 2019 08:34:51 -0800 (PST) Sender: Mark Johnston Date: Tue, 12 Feb 2019 11:34:46 -0500 From: Mark Johnston To: Eugene Grosbein Cc: FreeBSD stable Subject: Re: 11.2-STABLE kernel wired memory leak Message-ID: <20190212163446.GA29847@raichu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: DDF2F6D45F X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=ZzObx2B2; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::12e as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-5.34 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[e.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.65)[ip: (-8.71), ipnet: 2607:f8b0::/32(-2.51), asn: 15169(-1.97), country: US(-0.07)]; MID_RHS_NOT_FQDN(0.50)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Feb 2019 16:34:54 -0000 On Tue, Feb 12, 2019 at 11:14:31PM +0700, Eugene Grosbein wrote: > Hi! > > Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired memory over 81 days uptime > out of 8GB total RAM. > > Details follow. > > I have a workstation running Xorg, Firefox, Thunderbird, LibreOffice and occasionally VirtualBox for single VM. > > It has two identical 320GB HDDs combined with single graid-based array with "Intel" > on-disk format having 3 volumes: > - one "RAID1" volume /dev/raid/r0 occupies first 10GB or each HDD; > - two "SINGLE" volumes /dev/raid/r1 and /dev/raid/r2 that utilize "tails" of HDDs (310GB each). > > /dev/raid/r0 (10GB) has MBR partitioning and two slices: > - /dev/raid/r0s1 (8GB) is used for swap; > - /dev/raid/r0s2 (2GB) is used by non-redundant ZFS pool named "os" that contains only > root file system (177M used) and /usr file system (340M used). > > There is also second pool (ZMIRROR) named "z" built directly on top of /dev/raid/r[12] volumes, > this pool contains all other file systems including /var, /home, /usr/ports, /usr/local, /usr/{src|obj} etc. > > # zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > os 1,98G 520M 1,48G - - 55% 25% 1.00x ONLINE - > z 288G 79,5G 209G - - 34% 27% 1.00x ONLINE - > > This way I have swap outside of ZFS, boot blocks and partitioning mirrored by means of GEOM_RAID and > can use local console to break to single user mode to unmount all file system other than root and /usr > and can even export bigger ZFS pool "z". And I did that to see that ARC usage > (limited with vfs.zfs.arc_max="3G" in /boot/loader.conf) dropped from over 2500MB > down to 44MB but "Wired" stays high. Now after I imported "z" back and booted to multiuser mode > top(1) shows: > > last pid: 51242; load averages: 0.24, 0.16, 0.13 up 81+02:38:38 22:59:18 > 104 processes: 1 running, 103 sleeping > CPU: 0.0% user, 0.0% nice, 0.4% system, 0.2% interrupt, 99.4% idle > Mem: 84M Active, 550M Inact, 4K Laundry, 4689M Wired, 2595M Free > ARC: 273M Total, 86M MFU, 172M MRU, 64K Anon, 1817K Header, 12M Other > 117M Compressed, 333M Uncompressed, 2.83:1 Ratio > Swap: 8192M Total, 940K Used, 8191M Free > > I have KDB and DDB in my custom kernel also. How do I debug the leak further? > > I use nvidia-driver-340-340.107 driver for GK208 [GeForce GT 710B] video card. > Here are outputs of "vmstat -m": http://www.grosbein.net/freebsd/leak/vmstat-m.txt > and "vmstat -z": http://www.grosbein.net/freebsd/leak/vmstat-z.txt I suspect that the "leaked" memory is simply being used to cache UMA items. Note that the values in the FREE column of vmstat -z output are quite large. The cached items are reclaimed only when the page daemon wakes up to reclaim memory; if there are no memory shortages, large amounts of memory may accumulate in UMA caches. In this case, the sum of the product of columns 2 and 5 gives a total of roughly 4GB cached. > as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt > and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt