From owner-freebsd-hackers@freebsd.org Wed Jun 14 22:04:25 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59D85BF4C26 for ; Wed, 14 Jun 2017 22:04:25 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 284BF7A5AA; Wed, 14 Jun 2017 22:04:24 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v5EM4JGb063455; Wed, 14 Jun 2017 15:04:19 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v5EM4IEu063454; Wed, 14 Jun 2017 15:04:18 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201706142204.v5EM4IEu063454@pdx.rh.CN85.dnsmgr.net> Subject: Re: [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD performance drop < 24 hours In-Reply-To: To: "Caza, Aaron" Date: Wed, 14 Jun 2017 15:04:18 -0700 (PDT) CC: Allan Jude , "freebsd-hackers@freebsd.org" X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=US-ASCII X-Mailman-Approved-At: Wed, 14 Jun 2017 23:37:04 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jun 2017 22:04:25 -0000 > > -----Original Message----- > > From: owner-freebsd-hackers@freebsd.org [mailto:owner-freebsd-hackers@f= reebsd.org] On Behalf Of Allan Jude > > Sent: Wednesday, June 14, 2017 11:20 AM > > To: freebsd-hackers@freebsd.org > > Subject: [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD perfo= rmance drop < 24 hours > > > > On 2017-06-14 13:11, Caza, Aaron wrote: > > > Further to this: > > > > > > I've now tested FreeBSD 10.3-RELEASE-p19 and FreeBSD 11.0-STABLE #0 r= 307264M, both of which suffer the same degraded performance. > > > > > > test$ uname -a > > > FreeBSD xyz.com 10.3-RELEASE-p19 FreeBSD 10.3-RELEASE-p19 #0 r319904M= : Tue Jun 13 12:38:29 MDT 2017 aaronc@WFT:XYZ amd64 > > > test$ uptime > > > 10:15AM up 21:09, 2 users, load averages: 1.00, 1.14, 1.30 test$ dd > > > if=3D/testdb/test of=3D/dev/null bs=3D1m > > > 16000+0 records in > > > 16000+0 records out > > > 16777216000 bytes transferred in 200.379127 secs (83727363 bytes/sec) > > > > > > After reboot: > > > test$ dd if=3D/testdb/test of=3D/dev/null bs=3D1m > > > 16000+0 records in > > > 16000+0 records out > > > 16777216000 bytes transferred in 23.213040 secs (722749623 bytes/sec) > > > > > > Same Intel Xeon E31240 with 8GB ram and 2x Samsung 850 Pro 256GB SSDs= as before. > > > > > > This message may contain confidential and privileged information. If = it has been sent to you in error, please reply to advise the sender of the = error and then immediately delete it. If you are not the intended recipient= , do not read, copy, disclose= or otherwise use this message. The sender disclaims any liability for such= unauthorized use. > > PLEASE NOTE that all incoming e-mails sent to Weathe= rford e-mail accounts will be archived and may be scanned by us and/or by e= xternal service providers to=20= detect and prevent threats to our systems, investigate illegal or inappropr= iate behavior, and/or eliminate unsolicited promotional e-mails (spam). Thi= s process could result in > > > > deletion of a legitimate e-mail before it= is read by its intended reci= pient at our organization. Moreover, based on the scanning results, the ful= l text of e-mails and attachments may be made available to Weatherford secu= rity and other personnel for review and appropriate action. If you have any= concerns abo > ut this process, > > > please contact us at dataprivacy@weatherford.com. > > > _______________________________________________ > > > freebsd-hackers@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd= .org" > > > > > > > Can you do the same test, but grab the memory lines from top(1) before = and after each of those two runs. > > > > I am guessing the ARC is being squeezed out by PostgreSQL, because you = have so little RAM. > > > > -- > > Allan Jude >=20 > Takes a while for the degradation to kick in now that I rebooted this mor= ning. >=20 > Regarding the ARC being squeezed - well, that doesn't explain why gstat s= hows on 95-100% busy on the drives on reboot but only ~15 %busy after the d= egradation hits. >=20 > In fact, ARC is being squeezed all the time because I've limited it to 50= M in /boot/loader.conf: > vfs.zfs.arc_min=3D"50M" > vfs.zfs.arc_max=3D"50M" Would you passify an old fart by at least having a delta of 1 between a min= and max please? Some code may oscilate when min=3Dmax. > Note that the FreeBSD 9.0 server that I tested on also hamstrings the ARC= to 50M but doesn't suffer a performance degradation hence why I hadn't bot= hered mentioning it before. >=20 > To remove Postgres entirely, I won't even start it and simply use dd on t= he 16GB file. The server is essentially doing nothing at all. >=20 > At this point, I'm looking at going back to FreeBSD 10.3-RELEASE-p7 as ye= sterday as 'trafdev' reported that he doesn't see any performance drop and = he's got 95 days uptime. He's also mentioned vfs.zfs.metaslab.lba_weightin= g_enabled=3D0 setting which I a= lso need to try. >=20 > -- > Aaron >=20 >=20 > This message may contain confidential and privileged information. If it h= as been sent to you in error, please reply to advise the sender of the erro= r and then immediately delete it. If you are not the intended recipient, do= not read, copy, disclose or=20= otherwise use this message. The sender disclaims any liability for such una= uthorized use. PLEASE NOTE that all incoming e-mails sent to Weatherford e-= mail accounts will be archived and may be scanned by us and/or by external = service providers to detect a= nd prevent threats to our systems, investigate illegal or inappropriate beh= avior, and/or eliminate unsolicited promotional e-mails (spam). This proces= s could result in deletion of a legitimate e-mail before it is read by its = intended recipient at our org= anization. Moreover, based on the scanning results, the full text of e-mail= s and attachments may be made available to Weatherford security and other p= ersonnel for review and appropriate action. If you have any concerns about = this process, > please contact us at dataprivacy@weatherford.com. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >=20 --=20 Rod Grimes rgrimes@freebsd.= org