From owner-freebsd-current@freebsd.org Sun Jun 24 10:03:49 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE9D710222A5 for ; Sun, 24 Jun 2018 10:03:48 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 72DE282A2C for ; Sun, 24 Jun 2018 10:03:48 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: by mailman.ysv.freebsd.org (Postfix) id 31B9F10222A4; Sun, 24 Jun 2018 10:03:48 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0CEBB10222A3 for ; Sun, 24 Jun 2018 10:03:48 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:375::1:5]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7360A82A2B for ; Sun, 24 Jun 2018 10:03:47 +0000 (UTC) (envelope-from Alexander@leidinger.net) Date: Sun, 24 Jun 2018 12:03:29 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=leidinger.net; s=outgoing-alex; t=1529834625; bh=J12K+ljpS13a2c3sc19LShDJma4CcW8zYiPZf0niG8k=; h=Date:From:To:Subject; b=df72j7WpasR9PXZ7m8uNTwq2hKw3El7idoixpcJbgrTv95inI1tzeTfhIwoRHBSOb tiG5sTq2CLbfaHn8+JxSJSSK/mu6xcVyfTXpaPrx/48wV8k8CYJdU1ttkr31TjMhCz j2oLDphXWhszwPD7vD0St45hSHwX15IhKQzm8PDemaVVbC9x9B5YRja9gWip2fMXM2 qgG7PBEpenlsn6ehtPieQ93rfIo5tyIynBIpE0uEX1vs4NBfWgUTrtEm45V+eIqRdz ZDuA32fOonrwb5ADFb4EcOP7N18gUiE1lyd67Wk8Jr6keWwvSWARxOeaDNNt8u9HZZ SDqEg9mJHY8jA== Message-ID: <20180624120329.Horde.HWORumQ7Ng1KAUeviJNtoc3@webmail.leidinger.net> From: Alexander Leidinger To: current@freebsd.org Subject: numa involved in instability and swap usage despite RAM free? User-Agent: Horde Application Framework 5 Content-Type: multipart/signed; boundary="=_-oBR2oHEOll7in8gWqzLcvA"; protocol="application/pgp-signature"; micalg=pgp-sha1 MIME-Version: 1.0 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jun 2018 10:03:49 -0000 This message is in MIME format and has been PGP signed. --=_-oBR2oHEOll7in8gWqzLcvA Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I don't have hard evidence, but there is enough "smell" to open up a=20=20 discussion... Short: Can=20it be that enabling numa in the kernel is the reason why some=20=20 people=20see instability with zfs and usage of swap while a lot of free=20= =20 RAM=20is available? Long: I have a dual-socket Xeon system (E5620 + L5630... yes, not the same,=20=20 but=20compatible enough to be able to run together) with 64 GB RAM. I=20=20 run=20-current on it (currently it's at r333966 and it was for all the=20= =20 tests=20below). What I see with numa enabled and no zfs patches is, that at some point=20= =20 I=20have half the RAM free, swap is in use, and after a lot of compiling=20= =20 ports=20in different jails ZFS comes to a halt (sometimes I can unblock=20= =20 by=20killing a compile, sometimes I can't even kill, only way out is=20=20 power-cycle).=20I've seen this around twice a week. When I keep numa enabled and have applied this ZFS patch=20=20 https://reviews.freebsd.org/D7538=20the bahavior changes. AFter a while=20= =20 half=20of the RAM is free, swap is in use, and after enough compiling=20=20 ports=20in jails I get a panic (unfortunately not enough debug info in=20= =20 the=20textdump to know exactly what he problem is). Since 2 weeks I have numa compiled out of the kernel (and still the=20=20 ZFS=20patch inside). The system is down to 17 GB free and NO swap in=20=20 use.=20I'm compiling ports in 16 jails (one of them with parts of KDE5 =3D= =20=20 currently=20about 700 ports compiled) and not a single issue like the=20=20 above. For=20everyone with swap issues or ZFS issues similar to the ones I=20=20 see...=20do you have numa enabled and can you please try without and=20=20 report=20back? Can it be that if memory request can not be fulfilled from one numa=20=20 domain,=20there is no fallback to another numa domain for all the=20=20 various=20kinds of memory allocation we have in the kernel=20=20 (contigmem/no-sleep/...)? Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_-oBR2oHEOll7in8gWqzLcvA Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAABAgAGBQJbL2xxAAoJEKrxQhqFIICEVcsP/jVTtyEtb92DPBitIacgUk1F 3xzRc+0SfWYqyU8PAHsIRkJQFdvQ7rvLqwNznqvkLdi3FvmT+6PKJcORhHrNBMEo KhJGByGFPAdpnAoA7ZVJY7r6RWJsNU61AvPdgn0ZORPX73cbuo5zeCfwpIiYPGEH oja29zNA7XLnh9AZ5NJMl3zmvCEUOOONkENoUE/tDi4YMVy1m8EVmtSFAsdtBYHU G9incSchNzRx3tkK8kojknLjZCFQn44yOKYhANKwgMSECR5sZuvAcQaSLUNUvdcr 6OpjCFcHWsTFIr5vJvEqml+rSSnvt89q5JZBsP9T8/6dXs1rUuBKKLkVWR2A/bT9 d7c5WX4xFlk6vRRfcZ6pQ6GQQOs06IaxxPvRXDAdJcFCMj1gKatKumg55+IXxbwQ ovxuHay+YfmQxi/DuO0HAeh5JNd1NL3u2Dpb9Xs+xey0yKQXorFTU4ADiqcWXp/v kAiz0Cyf3gnaYuRA6bRZdUdTThj/3NDKKWztcX6pe+VGepSBBRsskEcfIns6oNgG n+CnkUIp5sjos+kG0iXD7d+ZqVPiHsdiWdvLzOuJD1bjagDbMD23eoA+m3uYLcKn buf/DJoW8VthcvlKPsh9Lxh2ChfiT3C1MVS0BTGeN6tfOWsPpg9E5c0e1mouWV9+ nMgw+XE0WfA2UDrP0X8k =ckNr -----END PGP SIGNATURE----- --=_-oBR2oHEOll7in8gWqzLcvA--