Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 16 Jun 2017 05:06:26 +0000
From:      "Caza, Aaron" <Aaron.Caza@ca.weatherford.com>
To:        "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>
Subject:   RE: [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD performance drop < 24 hours
Message-ID:  <e3b007fa109643ebbe0a6418ce7b1aaf@DM2PR58MB013.032d.mgd.msft.net>
In-Reply-To: <201706142204.v5EM4IEu063454@pdx.rh.CN85.dnsmgr.net>
References:  <f2b7a8f619f844a9bd1549bbfbeafbfa@DM2PR58MB013.032d.mgd.msft.net> <201706142204.v5EM4IEu063454@pdx.rh.CN85.dnsmgr.net>

next in thread | previous in thread | raw e-mail | index | archive | help
> -----Original Message-----
> From: Rodney W. Grimes [mailto:freebsd-rwg@pdx.rh.CN85.dnsmgr.net]
> Sent: Wednesday, June 14, 2017 4:04 PM
> To: Caza, Aaron
> Cc: Allan Jude; freebsd-hackers@freebsd.org
> Subject: Re: [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD per=
formance drop < 24 hours
>
> > > -----Original Message-----
> > > From: owner-freebsd-hackers@freebsd.org
> > > [mailto:owner-freebsd-hackers@freebsd.org] On Behalf Of Allan Jude
> > > Sent: Wednesday, June 14, 2017 11:20 AM
> > > To: freebsd-hackers@freebsd.org
> > > Subject: [EXTERNAL] Re: FreeBSD10 Stable + ZFS + PostgreSQL + SSD
> > > performance drop < 24 hours
> > >
> > > On 2017-06-14 13:11, Caza, Aaron wrote:
> > > > Further to this:
> > > >
> > > > I've now tested FreeBSD 10.3-RELEASE-p19 and FreeBSD 11.0-STABLE #0=
 r307264M, both of which suffer the same degraded performance.
> > > >
> > > > test$ uname -a
> > > > FreeBSD xyz.com 10.3-RELEASE-p19 FreeBSD 10.3-RELEASE-p19 #0 r31990=
4M: Tue Jun 13 12:38:29 MDT 2017     aaronc@WFT:XYZ  amd64
> > > > test$ uptime
> > > > 10:15AM  up 21:09, 2 users, load averages: 1.00, 1.14, 1.30 test$
> > > > dd if=3D/testdb/test of=3D/dev/null bs=3D1m
> > > > 16000+0 records in
> > > > 16000+0 records out
> > > > 16777216000 bytes transferred in 200.379127 secs (83727363
> > > > bytes/sec)
> > > >
> > > > After reboot:
> > > > test$ dd if=3D/testdb/test of=3D/dev/null bs=3D1m
> > > > 16000+0 records in
> > > > 16000+0 records out
> > > > 16777216000 bytes transferred in 23.213040 secs (722749623
> > > > bytes/sec)
> > > >
> > > > Same Intel Xeon E31240 with 8GB ram and 2x Samsung 850 Pro 256GB SS=
Ds as before.
> > > >
> > >
> > > Can you do the same test, but grab the memory lines from top(1) befor=
e and after each of those two runs.
> > >
> > > I am guessing the ARC is being squeezed out by PostgreSQL, because yo=
u have so little RAM.
> > >
> > > --
> > > Allan Jude
> >
> > Takes a while for the degradation to kick in now that I rebooted this m=
orning.
> >
> > Regarding the ARC being squeezed - well, that doesn't explain why gstat=
 shows on 95-100% busy on the drives on reboot but only ~15 %busy after the=
 degradation hits.
> >
> > In fact, ARC is being squeezed all the time because I've limited it to =
50M in /boot/loader.conf:
> > vfs.zfs.arc_min=3D"50M"
> > vfs.zfs.arc_max=3D"50M"
>
> Would you passify an old fart by at least having a delta of 1 between a m=
in and max please?
> Some code may oscilate when min=3Dmax.

> > Note that the FreeBSD 9.0 server that I tested on also hamstrings the A=
RC to 50M but doesn't suffer a performance degradation hence why I hadn't b=
othered mentioning it before.
> >
> > To remove Postgres entirely, I won't even start it and simply use dd on=
 the 16GB file.  The server is essentially doing nothing at all.
> >
> > At this point, I'm looking at going back to FreeBSD 10.3-RELEASE-p7 as =
yesterday as 'trafdev' reported that he doesn't see any performance drop an=
d he's got 95 days uptime.  He's also mentioned vfs.zfs.metaslab.lba_weight=
ing_enabled=3D0 setting which I also need to try.
> >
> > --
> > Aaron
>
> --
> Rod Grimes                                                 rgrimes@freebs=
d.org

Allan, here's the top(1) output after degradation:

last pid:  9403;  load averages:  1.09,  1.48,  1.30    up 0+23:29:14  10:3=
0:06
21 processes:  1 running, 19 sleeping, 1 zombie
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 1528K Active, 46M Inact, 462M Wired, 4856K Cache, 624K Buf, 7376M Free
ARC: 53M Total, 5K MFU, 16M MRU, 16K Anon, 1137K Header, 36M Other

And, after reboot with performance back to normal:
last pid:   739;  load averages:  0.99,  0.95,  0.58    up 0+00:08:02  10:5=
3:21
19 processes:  1 running, 17 sleeping, 1 zombie
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 18M Active, 22M Inact, 224M Wired, 432K Buf, 7626M Free
ARC: 28M Total, 3385K MFU, 24M MRU, 16K Anon, 232K Header, 1005K Other

The above was done using FreeBSD 10.3-RELEASEp19 (amd64) on the aforementio=
ned Xeon server with 2x Samsung 850 Pro 256GB SSDs.  PostgreSQL was never s=
tarted or use - simply dd'ing the 16GB test file.  These results were still=
 using vfs.zfs.arc_min and vfs.zfs_arc_max of "50MB" though subsequent test=
s will utilize 50M and 51M, respectively as suggested by Rod.

Checking the FreeBSD 9.0 test server, it had vfs.zfs.arc_max=3D"50M" but di=
d not have vfs.zfs.arc_min set in /boot/loader.conf; consequently, it was a=
ctually higher that vfs.zfs.arc_max.  I've reset to 50M and 51M and reboote=
d.  Time for "select count(*)" in Postgres for ~21.5 million row test table=
, after 24hours, is still maintaining the ~82 seconds as it did after reboo=
t.

Additionally, on separate test servers I've checked the performance of both=
 FreeBSD 10.3-RELEASE-p6 r303605M and the FreeBSD 11.0-STABLE snapshot date=
d May 10, 2017 and both exhibited the same degradation in performance.

Suggestions?

--
Aaron
This message may contain confidential and privileged information. If it has=
 been sent to you in error, please reply to advise the sender of the error =
and then immediately delete it. If you are not the intended recipient, do n=
ot read, copy, disclose or otherwise use this message. The sender disclaims=
 any liability for such unauthorized use. PLEASE NOTE that all incoming e-m=
ails sent to Weatherford e-mail accounts will be archived and may be scanne=
d by us and/or by external service providers to detect and prevent threats =
to our systems, investigate illegal or inappropriate behavior, and/or elimi=
nate unsolicited promotional e-mails (spam). This process could result in d=
eletion of a legitimate e-mail before it is read by its intended recipient =
at our organization. Moreover, based on the scanning results, the full text=
 of e-mails and attachments may be made available to Weatherford security a=
nd other personnel for review and appropriate action. If you have any conce=
rns about this process, please contact us at dataprivacy@weatherford.com.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e3b007fa109643ebbe0a6418ce7b1aaf>