Date: Wed, 11 Oct 2017 14:56:30 +0000 (UTC) From: Paul Pathiakis <pathiaki2@yahoo.com> To: Adam Vande More <amvandemore@gmail.com>, Kate Dawson <k4t@3msg.es> Cc: FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: FreeBSD ZFS file server with SSD HDD Message-ID: <447544877.450092.1507733790531@mail.yahoo.com> In-Reply-To: <CA%2BtpaK3Cga3SKmbKnRts_SSp=D4qk9p%2BaTzNBZeEqDuvQGVd9A@mail.gmail.com> References: <20171011130512.GE24374@apple.rat.burntout.org> <CA%2BtpaK3Cga3SKmbKnRts_SSp=D4qk9p%2BaTzNBZeEqDuvQGVd9A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
This is sort of a complementary answer. Seems like a cool setup.=C2=A0=C2=
=A0
One thing as an SA is to do detailed operational awareness with history for=
things.=C2=A0 (aka a solid SNMP monitoring tool with graphing so that you =
can see what is happening at exactly that time regarding network status, I/=
O status, CPU + resources as well as other detail. )=C2=A0 There's a good a=
mount like Zabbix, Nagios, Munin, Ganglia, etc in ports/pkgs.=C2=A0 - My fa=
vorite is OpenNMS which I was going to do the port for but ran out of time.=
=C2=A0=C2=A0
They support point-in-time graphing.=C2=A0 Once you resolve this issue, get=
some tool up and running so you can 'see' status on everything at that tim=
e. (an you can give read-only access to management so they can ooo and ahhh=
at it.)=C2=A0 It is very useful to see where the bottleneck is... if there=
is one...=C2=A0 otherwise, tuning parameters for everything (ZFS, NFS, NIC=
, etc) will have to tried as you can't easily narrow down where the problem=
is.=C2=A0
Just trying to help in the long term.
P.
From: Adam Vande More <amvandemore@gmail.com>
To: Kate Dawson <k4t@3msg.es>=20
Cc: FreeBSD Questions <freebsd-questions@freebsd.org>
Sent: Wednesday, October 11, 2017 10:41 AM
Subject: Re: FreeBSD ZFS file server with SSD HDD
=20
On Wed, Oct 11, 2017 at 8:05 AM, Kate Dawson <k4t@3msg.es> wrote:
> Hi,
>
> Currently running a FreeBSD NFS server with a zpool comprising
>
> 12 x 1TB hard disk drives are arranged as pairs of mirrors in a strip set
> ( RAID 10 )
>
> An additional 2x 960GB SSD added. These two SSD are partitioned with a
> small partition begin used for a ZIL log, and larger partion arranged for
> L2ARC cache.
>
> Additionally the host has 64GB RAM and 16 CPU cores (AMD Opteron 2Ghz)
>
> A dataset from the pool is exported via NFS to a number of Debian
> Gnu/Linux hosts running a xen hypervisor. These run several disk image
> based virtual machines
>
> In general use, the FreeBSD NFS host sees very little read IO, which is t=
o
> expected
> as the RAM cache=C2=A0 and L2ARC are designed to minimise the amount of r=
ead
> load
> on the disks.
>
> However we're starting to see high load ( mostly IO WAIT ) on the Linux
> virtualisation hosts, and virtual machines - with kernel timeouts
> occurring resulting in crashes and instability.
>
> I believe this may be due to the limited number of random write IOPS
> available
> on the zpool NFS export.
>
> I can get sequential writes and reads to and from the NFS server at
> speeds that approach the maximum the network provides ( currently 1Gb/s
> + Jumbo Frames, and I could increase this by bonding multiple interfaces
> together. )
>
> However day to day usage does not show network utilisation anywhere near
> this maximum.
>
> If I look at the output of `zpool iostat -v tank 1 ` I see that every
> five seconds or so, the numner of write operation go to > 2k
>
> I think this shows that the I'm hitting the limit that the spinning disk
> can provide in this workload.
>
I doubt that is the cause.=C2=A0 It is more likely you have
vfs.zfs.txg.timeout
set to the default.=C2=A0 Have you tried any other zfs or nfs tuning?=C2=A0=
If so,
please share those details.
Does gstat reveal anything useful?
--=20
Adam
_______________________________________________
freebsd-questions@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org=
"
=20
From owner-freebsd-questions@freebsd.org Wed Oct 11 15:07:53 2017
Return-Path: <owner-freebsd-questions@freebsd.org>
Delivered-To: freebsd-questions@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id 835C3E2EBCB
for <freebsd-questions@mailman.ysv.freebsd.org>;
Wed, 11 Oct 2017 15:07:53 +0000 (UTC)
(envelope-from tech-lists@zyxst.net)
Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com
[66.111.4.25])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 525206B889
for <freebsd-questions@freebsd.org>; Wed, 11 Oct 2017 15:07:52 +0000 (UTC)
(envelope-from tech-lists@zyxst.net)
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
by mailout.nyi.internal (Postfix) with ESMTP id 6D91A213D5
for <freebsd-questions@freebsd.org>; Wed, 11 Oct 2017 11:07:46 -0400 (EDT)
Received: from frontend2 ([10.202.2.161])
by compute4.internal (MEProxy); Wed, 11 Oct 2017 11:07:46 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h=
content-type:date:from:in-reply-to:message-id:mime-version
:references:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=
fm1; bh=y9HO1wvtTfcKKuYmFC0rG0ab+YPr21ZTM6hVWpSOSgw=; b=YodnTMIx
qLwNf9DZlmWqLuLwzeSyOE24L2vWiDOl0NXiFc+jHIzIjyx6poYIgkqnpCM78/cQ
I9mph/Lm9fNCtTqKgssJhpx9uLNrgkQvOxT+V/b3YTnLxXdvQ7P/D3OlaXtCvGLp
nBeWu5l0L4FEHSWRnC+I0sP/I6TTtAniMwyL8/u4CWbmq/VfFlQg/iIEJspP6OlO
f/50mYnbjbyPZCTgxDEnMZxgHkfEuJLDLPF85XRBmrqzstewoLRXro1GOh/etPze
sdJWApxq+Vu2YsuGzzwLEAeTeicZo9zXMgK/2F4peg3nBMs24UB+4/HpE1x6E6gj
9WF/fUJMA1ikCg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
messagingengine.com; h=content-type:date:from:in-reply-to
:message-id:mime-version:references:subject:to:x-me-sender
:x-me-sender:x-sasl-enc; s=fm1; bh=y9HO1wvtTfcKKuYmFC0rG0ab+YPr2
1ZTM6hVWpSOSgw=; b=LO7xrg3627lAMzuZDoKiO4W7hsbnWao9Br1vGyCJr0RsW
KLZ0GB6SUcqQh8znM33XVMuilgWyHmwkoZG5RYTrgNePfXkvqtyPVb3m2T54of15
FHl+qX2TDrKZLT5d78Dj//1bM2w0Mb6fBYvDF4NkF2sAqTItOCZIRqDKUpt24TLA
3nx8DsKw+8GBwvhH8r7D1H6W+9B/mnKdTuht8Im6HM6o26nPaHE31/Dl5L2ea7GO
jr/mqUbxtaPD/KVZ5LMLq69ZMjQWunTJSKzrXZCMAO/xPwVD6NnXOo2IGZbliZ8j
UZ0pOZdtVMDZtsgKtt3Su7bA49Xv9LhUh14z9i6fw==
X-ME-Sender: <xms:wjPeWRVcaiTANmXKHBPIBxq6uH4EgTYq4qxD2yOlfB0wonCFBIeCaQ>
Received: from acer.zyxst.net (parsley.growveg.org [82.70.91.97])
by mail.messagingengine.com (Postfix) with ESMTPA id DE4A1247F0
for <freebsd-questions@freebsd.org>; Wed, 11 Oct 2017 11:07:45 -0400 (EDT)
Date: Wed, 11 Oct 2017 16:07:44 +0100
From: tech-lists <tech-lists@zyxst.net>
To: freebsd-questions@freebsd.org
Subject: Re: problems with pkg: Operation timed out
Message-ID: <20171011150743.GA23107@acer.zyxst.net>
Mail-Followup-To: freebsd-questions@freebsd.org
References: <CAJ5UdcPVYnF3GN5EyHZSbLjUiPq4icujvFSJSp7hwH0gLAykxQ@mail.gmail.com>
<VI1PR02MB1200A355A0975AB589DD6784F6740@VI1PR02MB1200.eurprd02.prod.outlook.com>
<CAJ5UdcMkRv_cd7n-fo9DL-H+_MVuzkFUH4eub1g_2AxYdsKVPg@mail.gmail.com>
<CAJ5UdcP+CU0=CMe953Mf+cW-N6wh24uMiz6cpu-MRsXftd6VkQ@mail.gmail.com>
<CA+4G5KZ+e=FXRjRYrLb+T0cd4bGg4fHV3MCBTR+06uH6dR0RUA@mail.gmail.com>
<CAJ5UdcP-DAE01jwOZ_eVm=oHJLc=4u_Lzykj+nQn3gR4BxxSJg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <CAJ5UdcP-DAE01jwOZ_eVm=oHJLc=4u_Lzykj+nQn3gR4BxxSJg@mail.gmail.com>
User-Agent: Mutt/1.9.1 (2017-09-22)
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-questions>,
<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>;
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>,
<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Oct 2017 15:07:53 -0000
Hi,
On Wed, Oct 11, 2017 at 08:07:39AM -0500, Antonio Olivares wrote:
>root@aceraspire:~ # traceroute pkg.freebsd.org
>traceroute to pkgmir.geo.freebsd.org (96.47.72.71), 64 hops max, 40 byte packets
> 1 10.155.142.1 (10.155.142.1) 0.914 ms 9.633 ms 0.952 ms
> 2 172.17.0.1 (172.17.0.1) 4.262 ms 2.012 ms 1.911 ms
> 3 * * *
> 4 * * *
> 5 * * *
> 6 * *^C
That indicates a problem with your internet connection (or whatever is after
172.17.0.1) rather than pkg.freebsd.org
--
J.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?447544877.450092.1507733790531>