From owner-freebsd-current@freebsd.org Sat Mar 17 09:39:29 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0995CF570E0 for ; Sat, 17 Mar 2018 09:39:29 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7481E71ACA for ; Sat, 17 Mar 2018 09:39:28 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from thor.intern.walstatt.dynvpn.de ([77.179.222.246]) by mail.gmx.com (mrgmx102 [212.227.17.168]) with ESMTPSA (Nemesis) id 0LrIPo-1eTCvn3xCm-013AHI; Sat, 17 Mar 2018 10:39:24 +0100 Date: Sat, 17 Mar 2018 10:38:48 +0100 From: "O. Hartmann" To: Jeff Roberson Cc: Mark Millard , freebsd-current@freebsd.org, "O. Hartmann" Subject: Re: Strange ARC/Swap/CPU on yesterday's -CURRENT Message-ID: <20180317103915.081ca2dd@thor.intern.walstatt.dynvpn.de> In-Reply-To: References: Organization: WALSTATT User-Agent: OutScare 3.1415926 X-Operating-System: ImNotAnOperatingSystem 3.141592527 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/ltrqC+n=RO/WgeukIn_SvvE"; protocol="application/pgp-signature" X-Provags-ID: V03:K0:739Yxk2E2c0qcMXeotViyTWCoA1RGq/TOTuJHHM4oc6ovLpYdD/ CQ+6VddsTW6naeDHqc2s8v89qIogPaNPvI76m0r9dh6f8vwV9AC5/akgPqGp69ZA2ZnAVCY l3zzf6L85FfcYV2QJcYfDHa+sETNPMpfWwWkAZPY/5D2xDP7sDnAwIUgkgySHPqSTN8SXq8 RGVEFvzyNRDk7oe7ANiZw== X-UI-Out-Filterresults: notjunk:1;V01:K0:zd25C5N4mRA=:04OaUGIa94P+2GTBCrA+gE Bymi/HV2Y1fEhM/a7+xmYj0zn/uRQCeLOmL+EB8b2rcfbZ4B/HbCYJxWxvGyam81NEbWM9KIb XSwX5Jel47y/cnM6YC7+as2W8WuKGiyH1ugrPApjg/iUGWWOLpFsAIgKnYuaQV4inhD/yrMZG /jLLJiVy8bMJc4KIBbdrBZCsXngEUU06ykn8O8pKk+9xu8+tyHwOaJjtVZ3MplERRkbvoa5Q3 LVN9qOmORkKKBwabVY+Q83dSpnBR6pmVP2l2jzAru1dIYhs5oaGPfd+mi64i5lf0X3P1Dho9V w9m3OdSXgZV9vtbw4kol/HJRKVFbMzfsglkAOibjzrn220yBLCkBjSiInVv+ITZaRRNkNrCDs a8BRYZeiU+ABE2NzGz4qL5ivxuzyoHQommYeOfg4uPYoMsl4TE8UPkHl+4oclSqTjgDLS/od1 IyJtHkGdLhAEPOHKGKEQI+hMbaZrFshwFKkm2QaClT8SvgE18SAeFkFoH0OnYWl/U+wvT4kNq KhxfHQURWlmrbBaEOwXDFlJGZx+z65eINF5E/jFVcRqB2rXzuF3PIfLXTNAP4yIJLyEVIrXyA iwD515Gjbz9ELdCY96DVKj8AzIWECk8/lrSYAyx+x3P8HrdmIwftMq9nkT/MkeyZm9YCQJBJz l+DU4leDYfqn7e6PZUAU9NFDSdTDcj7RdOZYy5o3dRT821jCc3XESST+ewL6edMP9mZ1kKc9X P9m5s9iPtjZA23Lv+R9lJ6e9TLod+7NDCa3/yQjXCDj+ja9u11Ed+XS4QPvV+1HXM9Dl6ho4W RgWdAkDKZgedoXZeqYy4DLVTTFdlg== X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Mar 2018 09:39:29 -0000 --Sig_/ltrqC+n=RO/WgeukIn_SvvE Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Am Sun, 11 Mar 2018 18:00:35 -1000 (HST) Jeff Roberson schrieb: > On Sun, 11 Mar 2018, Mark Millard wrote: >=20 > > As I understand, O. Hartmann's report ( ohartmann at walstatt.org ) in: > > > > https://lists.freebsd.org/pipermail/freebsd-current/2018-March/068806.h= tml > > > > includes a system with a completely non-ZFS context: UFS only. Quoting = that part: > > =20 > >> This is from a APU, no ZFS, UFS on a small mSATA device, the APU (PCen= igine) works > >> as a firewall, router, PBX): > >> > >> last pid: 9665; load averages: 0.13, 0.13, 0.11 > >> up 3+06:53:55 00:26:26 19 processes: 1 running, 18 sleeping CPU: 0.= 3% user, 0.0% > >> nice, 0.2% system, 0.0% interrupt, 99.5% idle Mem: 27M Active, 6200K= Inact, 83M > >> Laundry, 185M Wired, 128K Buf, 675M Free Swap: 7808M Total, 2856K Used= , 7805M Free > >> [...] > >> > >> The APU is running CURRENT ( FreeBSD 12.0-CURRENT #42 r330608: Wed Mar= 7 16:55:59 > >> CET 2018 amd64). Usually, the APU never(!) uses swap, now it is starti= ng to swap > >> like hell for a couple of days and I have to reboot it failty often.=20 Yes, that is correct. The system in question (the PCengine APU4C, articially limited to 1 GB RAM = via boot loader option) does run an asterisk PBX and is our Firewall/Router applianc= e. The kernel/world is highly customized and "reduced" (via NanoBSD WITHOUT_ build= options). Since its existence mid of last year always running CURRENT since then, I r= ealized only that Asterisk might have a memory leak, but as some commits in the past sug= gests, it also could be triggered by a bug in syslog. That's the background for havin= g only 1 GB out of 4GB configured. Since a couple of weeks now, this APU starts swapping and keeping ~ 4GB of = allocated swap, right now, running CUURENT FreeBSD 12.0-CURRENT #50 r330750: Sun Mar 11 01:= 14:34 CET 2018 amd64: last pid: 16958; load averages: 0.10, 0.21, 0.16 up 6+08:57:07 10:34:01 19 processes: 1 running, 18 sleeping CPU: 0.3% us= er, 0.0% nice, 0.5% system, 0.0% interrupt, 99.2% idle Mem: 27M Active, 1504K Inac= t, 96M Laundry, 188M Wired, 1900K Buf, 664M Free Swap: 7808M Total, 4204K Used, 78= 04M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU CO= MMAND 997 asterisk 59 52 0 133M 61220K select 0 278:59 2.69% as= terisk 16958 root 1 20 0 13308K 3448K CPU2 2 0:00 0.14% top 579 root 1 20 0 15252K 3116K select 1 35:01 0.07% ppp 933 root 1 20 0 10892K 1688K select 0 1:38 0.02% po= werd 1038 root 1 20 0 11400K 764K nanslp 1 0:03 0.02% cr= on 930 root 1 20 0 18200K 18280K select 0 0:47 0.01% nt= pd 1005 root 1 20 0 14772K 4516K bpf 3 0:13 0.00% ar= pwatch 834 root 1 20 0 11364K 1992K select 1 4:46 0.00% sy= slogd 847 bind 7 52 0 59528K 30600K sigwai 2 1:30 0.00% na= med 989 root 1 20 0 32264K 1716K nanslp 2 0:20 0.00% pe= rl 863 daemon 1 20 0 11388K 1944K select 1 0:01 0.00% rp= cbind 872 root 1 20 0 11096K 1840K autofs 2 0:01 0.00% au= tomountd 975 root 1 20 0 14548K 0K nanslp 3 0:00 0.00% 968 dhcpd 1 20 0 22972K 7648K select 0 0:00 0.00% dh= cpd 878 root 1 20 0 10988K 0K kqread 1 0:00 0.00% 13271 root 1 20 0 12104K 3128K wait 1 0:00 0.00% lo= gin 16955 root 1 26 0 13204K 4080K pause 3 0:00 0.00% csh 1034 root 1 20 0 18340K 3976K select 0 0:00 0.00% ss= hd 1084 root 1 52 0 10928K 1724K ttyin 2 0:00 0.00% ge= tty This box doesn't have ZFS! There is a small mSATA device UFS2 formatted for= loggin and automountfs usage. I know this "repeated" report could annoy, but maybe it gives some more informations/confirmations according to have a time series of incidents. > > > > Unless this is unrelated, it would suggest that ZFS and its ARC need not > > be involved. > > > > Would what you are investigating relative to your "NUMA and concurrency > > related work" fit with such a non-ZFS (no-ARC) context? =20 >=20 > I think there are probably two different bugs. I believe the pid=20 > controller has caused the laundry thread to start being more aggressive=20 > causing more pageouts which would cause increased swap consumption. >=20 > The back-pressure mechanisms in arch should've resolved the other reports= .=20 > It's possible that I broke those. Although if the reports from 11.x are= =20 > to be believed I don't know that it was me. It is possible they have bee= n=20 > broken at different times for different reasons. So I will continue to=20 > look. >=20 > Thanks, > Jeff >=20 > > > > =3D=3D=3D > > Mark Millard > > marklmi at yahoo.com > > ( dsl-only.net went > > away in early 2018-Mar) > > =20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" --=20 O. Hartmann Ich widerspreche der Nutzung oder =C3=9Cbermittlung meiner Daten f=C3=BCr Werbezwecke oder f=C3=BCr die Markt- oder Meinungsforschung (=C2=A7 28 Abs.= 4 BDSG). --Sig_/ltrqC+n=RO/WgeukIn_SvvE Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iLUEARMKAB0WIQQZVZMzAtwC2T/86TrS528fyFhYlAUCWqziQwAKCRDS528fyFhY lIJUAf9L21mOcf9znvwBngGS3e64n2TyYi6cv3by++Ui0tjS7CjpMynSxmwOl+rO OyHrdzTCKOZnO/p4qKSsiO8OtCYyAf0V+bDOkxbYFsEAzmO6lJnje/T/iS5E/DmC nxSuQe/zJk2B5DcDdBSQkl6w13lD+RIyj3o8o3RGPXP2jaau9xxc =ifFN -----END PGP SIGNATURE----- --Sig_/ltrqC+n=RO/WgeukIn_SvvE--