From owner-freebsd-performance@FreeBSD.ORG Tue Jan 27 14:33:05 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C1FF106564A for ; Tue, 27 Jan 2009 14:33:05 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-ew0-f21.google.com (mail-ew0-f21.google.com [209.85.219.21]) by mx1.freebsd.org (Postfix) with ESMTP id F04758FC1C for ; Tue, 27 Jan 2009 14:33:04 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by ewy14 with SMTP id 14so2643990ewy.19 for ; Tue, 27 Jan 2009 06:33:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=jbt5ccL0hcASS9asTXFWMM2C9OOPS6LzoqSsFVCHIDM=; b=XaEMsFUnzQ7m+do3XP9DeiU1SVFsF7v2ZOKUg+RihwSXmcpVz4FKE7L9LJaLvhv/Oy WbLwWj6gkgrgSn14oaQpSWT+Vhksvh3MhluFeT9V7IyGXGnYcZPPd4e+XvGFqcYyhXVU Kky0Q7jXXK+RsOMqv0+guxWjrQwC3wUXfHk6E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=rNhUriG5jAQvNwRRTpYu2vnu1qrw58QBNOPO2Y7vbqZ9g4PESMg7guvweJwMXgAOgF 71XXH58XT1hZG7GJ6MW7/ZKb3R987ga+cq434zAbPL5H9wnGI0xVQPo58w0dylflpWQz bUHmEyUHJRiBXDFX9MF91/aAPG1PjrfLAzN70= MIME-Version: 1.0 Received: by 10.86.70.3 with SMTP id s3mr590182fga.78.1233065434760; Tue, 27 Jan 2009 06:10:34 -0800 (PST) Date: Tue, 27 Jan 2009 17:10:34 +0300 Message-ID: From: pluknet To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: mysql scalability: freebsd vs solaris X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jan 2009 14:33:05 -0000 Do anyone have MySQL scalability comparison numbers between FreeBSD7.x/8 and Solaris? Thanks. -- wbr, pluknet From owner-freebsd-performance@FreeBSD.ORG Tue Jan 27 16:47:57 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 211FE1065670 for ; Tue, 27 Jan 2009 16:47:57 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-ew0-f21.google.com (mail-ew0-f21.google.com [209.85.219.21]) by mx1.freebsd.org (Postfix) with ESMTP id AA7338FC18 for ; Tue, 27 Jan 2009 16:47:56 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by ewy14 with SMTP id 14so2886280ewy.19 for ; Tue, 27 Jan 2009 08:47:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=1utc7IKj1oo5ccYBPfCiIhjEdf+DPNVor91qLKailik=; b=prIC2qPcGZNrRz+FwFTCYPHh4/G+AbsIXLpb875GduoRe1FGOwmnGvberj27nliAZy JJWlvfeQeizMaBSlc+FvjVP99X+SdfBZRnry1TmqXnAOwIwLsV/1kkDy7A9Iru66X3XG ukYFa5o5tKLG4lJZ6u1NiI5GajmhzCmtgusls= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=gZoxRHVLtJ/Nuf0pxap+nQ4Ei2qUz6BUbdyHN5zcy3Z2IiYdNU6grXHAxgvBJqd1q7 UtdNI0LmB0bXUWy5rCnW9XHQkISwqosA4LP1zKJkIk1+wu5+PlG0z4o5rpP2V3uSh1V8 AYbhU8BaTGV9rTkefJVDUs5X8e7jDxmyQbPX4= MIME-Version: 1.0 Received: by 10.86.82.16 with SMTP id f16mr105832fgb.59.1233074875655; Tue, 27 Jan 2009 08:47:55 -0800 (PST) In-Reply-To: References: Date: Tue, 27 Jan 2009 19:47:55 +0300 Message-ID: From: pluknet To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: mysql scalability: freebsd vs solaris X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jan 2009 16:47:57 -0000 2009/1/27 pluknet : > Do anyone have MySQL scalability comparison numbers between > FreeBSD7.x/8 and Solaris? > > Thanks. I have quick'n'dirty scalability benchmark. HW: 2 x Intel(r) Xeon(r) CPU E5405 @ 2.00GHz 8 cpus summary. Values extracted from sysbench: read/write requests 1 thread / 8 threads FreeBSD 6.2 6091.80 5614.48 FreeBSD 7.1 6741.98 28091.60 SunOS 5.10 4286.61 6949.80 Too strange for Solaris.. -- wbr, pluknet From owner-freebsd-performance@FreeBSD.ORG Thu Jan 29 07:44:59 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B5701065670 for ; Thu, 29 Jan 2009 07:44:59 +0000 (UTC) (envelope-from brent@servuhome.net) Received: from mail-fx0-f15.google.com (mail-fx0-f15.google.com [209.85.220.15]) by mx1.freebsd.org (Postfix) with ESMTP id 7FC0F8FC17 for ; Thu, 29 Jan 2009 07:44:58 +0000 (UTC) (envelope-from brent@servuhome.net) Received: by fxm8 with SMTP id 8so752067fxm.19 for ; Wed, 28 Jan 2009 23:44:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.181.54.8 with SMTP id g8mr1907282bkk.114.1233213695968; Wed, 28 Jan 2009 23:21:35 -0800 (PST) Date: Wed, 28 Jan 2009 23:21:35 -0800 Message-ID: From: Brent Jones To: freebsd-performance@freebsd.org, pathiaki2@yahoo.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: ZFS, NFS and Network tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2009 07:44:59 -0000 I'm reviving this, as I too am seeing something eerily similar. I have made my own thread under freebsd-stable, so I will hopefully move that discussion to this list. I believe we are seeing performance problems when the FreeBSD NFS client issues FSYNC NFS instead of ASYNC, sending performance to a mere percentage of what disks and network links are capable of. Further testing tonight demonstrates that other NFSv3 and v4 clients do not issue FSYNC unless they modify attributed and close a file, or append and close a file. FreeBSD NFS client will issue FSYNCs anytime the write size (-w) is reached, instead of when just closing the file. This is not necessary, since NFSv3 and v4 TCP have provisions for safe async writes that 'guarantee' state of NFS writes. Here is the contents of what I wrote there verbatim: http://lists.freebsd.org/pipermail/freebsd-stable/2009-January/048063.html ------- Hello FreeBSD users, I am running into some performance problems with NFSv3/v4 mounts. I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram and a FreeBSD 7.1 32bit with same hardware. The issue I am seeing, is that for certain file types, the FreeBSD NFS client will either issue an ASYNC write, or an FSYNC. However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP versions of the protocol, so that should be the default. Issuing FSYNC's for every compete block transmitted adds substantial overhead and slows everything down. The two test files I have that can reproduce this data are a file created by 'dump' which is just binary data: $ file testbinery testbinery: data ASCII text file from a Maildir format: $ file ascittest ascittest: ASCII mail text My NFS mount command lines I have tried to get all data to ASYNC write: $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ Here is an excerpt from a snoop from the binary data file: $ snoop rpc nfs obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 (read,lookup,modify,extend,delete,execute) pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 testbinery pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3 obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 (read,lookup,modify,extend,delete,execute) pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) obsmtp02.local -> pdxfilu01 NFS C SETATTR3 FH=57D3 pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 582647808 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 592871424 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 605421568 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) And on and on.. it will acheive near full wire-speed, about 110MB/sec during the copy Here is the same snoop, only copying the ASCII mail file: $ snoop rpc nfs obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory obsmtp02.local -> pdxfilu01 NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3 obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) And so on. I've reproduced this with several files, and the only difference between tests is the file type. Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the file type/contents? If so, is there a tuneable setting to make all write ASYNC? Otherwise, FSYNC'ing for every block written over NFS will cause so many IOPS on the NFS server, that performance will degrade severely. Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for any file type, if mounted with NFSv3 of NFSv4 (TCP). Any ideas? Thanks in advance! -- Brent Jones brent@servuhome.net From owner-freebsd-performance@FreeBSD.ORG Thu Jan 29 08:43:08 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2AD971065673; Thu, 29 Jan 2009 08:43:08 +0000 (UTC) (envelope-from brent@servuhome.net) Received: from mail-fx0-f15.google.com (mail-fx0-f15.google.com [209.85.220.15]) by mx1.freebsd.org (Postfix) with ESMTP id 6AEF58FC12; Thu, 29 Jan 2009 08:43:07 +0000 (UTC) (envelope-from brent@servuhome.net) Received: by fxm8 with SMTP id 8so761424fxm.19 for ; Thu, 29 Jan 2009 00:43:06 -0800 (PST) MIME-Version: 1.0 Received: by 10.181.199.16 with SMTP id b16mr583902bkq.142.1233218586010; Thu, 29 Jan 2009 00:43:06 -0800 (PST) In-Reply-To: References: Date: Thu, 29 Jan 2009 00:43:05 -0800 Message-ID: From: Brent Jones To: freebsd-performance@freebsd.org, pathiaki2@yahoo.com, freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: ZFS, NFS and Network tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2009 08:43:08 -0000 On Wed, Jan 28, 2009 at 11:21 PM, Brent Jones wrote: > I'm reviving this, as I too am seeing something eerily similar. I have > made my own thread under freebsd-stable, so I will hopefully move that > discussion to this list. > > I believe we are seeing performance problems when the FreeBSD NFS > client issues FSYNC NFS instead of ASYNC, sending performance to a > mere percentage of what disks and network links are capable of. > Further testing tonight demonstrates that other NFSv3 and v4 clients > do not issue FSYNC unless they modify attributed and close a file, or > append and close a file. > FreeBSD NFS client will issue FSYNCs anytime the write size (-w) is > reached, instead of when just closing the file. > This is not necessary, since NFSv3 and v4 TCP have provisions for safe > async writes that 'guarantee' state of NFS writes. > > Here is the contents of what I wrote there verbatim: > > http://lists.freebsd.org/pipermail/freebsd-stable/2009-January/048063.html > > ------- > > > Hello FreeBSD users, > I am running into some performance problems with NFSv3/v4 mounts. > I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares > The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram > and a FreeBSD 7.1 32bit with same hardware. > > The issue I am seeing, is that for certain file types, the FreeBSD NFS > client will either issue an ASYNC write, or an FSYNC. > However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP > versions of the protocol, so that should be the default. > Issuing FSYNC's for every compete block transmitted adds substantial > overhead and slows everything down. > > The two test files I have that can reproduce this data are a file > created by 'dump' which is just binary data: > > $ file testbinery > testbinery: data > > ASCII text file from a Maildir format: > > $ file ascittest > ascittest: ASCII mail text > > My NFS mount command lines I have tried to get all data to ASYNC write: > > $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > > Here is an excerpt from a snoop from the binary data file: > > $ snoop rpc nfs > > obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 > (read,lookup,modify,extend,delete,execute) > pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 testbinery > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3 > obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 > (read,lookup,modify,extend,delete,execute) > pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) > obsmtp02.local -> pdxfilu01 NFS C SETATTR3 FH=57D3 > pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 582647808 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 592871424 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 605421568 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > > > And on and on.. it will acheive near full wire-speed, about 110MB/sec > during the copy > > > Here is the same snoop, only copying the ASCII mail file: > > $ snoop rpc nfs > > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory > obsmtp02.local -> pdxfilu01 NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest > pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3 > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > > > And so on. I've reproduced this with several files, and the only > difference between tests is the file type. > Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the > file type/contents? > If so, is there a tuneable setting to make all write ASYNC? > Otherwise, FSYNC'ing for every block written over NFS will cause so > many IOPS on the NFS server, that performance will degrade severely. > > Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for > any file type, if mounted with NFSv3 of NFSv4 (TCP). > > Any ideas? > > Thanks in advance! > > > > > -- > Brent Jones > brent@servuhome.net > I have found a 4 year old bug, which may be related to this. cp uses mmap for small files (and I imagine lots of things use mmap for file operations) and causes slowdowns via NFS, due to the fsync data provided above. http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/87792 That bugid accurately describes the issue, is there any way to attach more 'interested parties' or additional details to that bug? -- Brent Jones brent@servuhome.net From owner-freebsd-performance@FreeBSD.ORG Thu Jan 29 13:07:51 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 58D741065670 for ; Thu, 29 Jan 2009 13:07:51 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.175]) by mx1.freebsd.org (Postfix) with ESMTP id DFD828FC12 for ; Thu, 29 Jan 2009 13:07:50 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by ug-out-1314.google.com with SMTP id 30so349560ugs.39 for ; Thu, 29 Jan 2009 05:07:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=evngfUDLAh4oJ70O2VkSgrYoHX2VZ6tRC/cPPzuHfT4=; b=mjeHE+GhkAJUpQn6Jr/887dGcDy/WVEdItOwvjYQyN15dnOWNEBgPipNcDZisa2nPu 3VV6j+3BRBB84cwmnBz+9faDVvV5b5E3CZIBb63+jR//kFh1RiJoskAAd4KmLbSktHZ9 Lb3nvA7EIZT8aX1blpakUqRuhHw7FULGDWNiE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=lJQbhkeAVmHY+cs1HdGzyGJbZsFZrpWJWG+Os2l3zgviUqm2w6iqIoJWm79mJw5Spd UYtcBTlChFXmwbiqboA04GgjDdPLkqzJ6AETlzx7xIXeS7GGNLvtATlC+6A8Zznqn8kc keCQa1gfQaxwRaa7VPfVPpc9kRvVXSU0eq3VQ= MIME-Version: 1.0 Received: by 10.86.4.14 with SMTP id 14mr56816fgd.76.1233234469725; Thu, 29 Jan 2009 05:07:49 -0800 (PST) In-Reply-To: References: Date: Thu, 29 Jan 2009 16:07:49 +0300 Message-ID: From: pluknet To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: mysql scalability: freebsd vs solaris X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2009 13:07:51 -0000 2009/1/27 pluknet : > read/write requests 1 thread / 8 threads > > FreeBSD 6.2 6091.80 5614.48 > FreeBSD 7.1 6741.98 28091.60 > SunOS 5.10 4286.61 6949.80 > Err.. It seems we overoptimized our MySQL or whatever.. Official 5.0.67 build for SunOS gives us: SunOS 5.10 5676.93 25085.87 -- wbr, pluknet From owner-freebsd-performance@FreeBSD.ORG Thu Jan 29 20:19:30 2009 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C2DD01065672; Thu, 29 Jan 2009 20:19:30 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx09.syd.optusnet.com.au (fallbackmx09.syd.optusnet.com.au [211.29.132.242]) by mx1.freebsd.org (Postfix) with ESMTP id C92628FC13; Thu, 29 Jan 2009 20:19:29 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au [211.29.132.190]) by fallbackmx09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id n0TEBodT018443; Fri, 30 Jan 2009 01:11:50 +1100 Received: from c122-107-120-227.carlnfd1.nsw.optusnet.com.au (c122-107-120-227.carlnfd1.nsw.optusnet.com.au [122.107.120.227]) by mail09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id n0TEBfA8026452 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 30 Jan 2009 01:11:42 +1100 Date: Fri, 30 Jan 2009 01:11:41 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Brent Jones In-Reply-To: Message-ID: <20090129234158.B46285@delplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: pathiaki2@yahoo.com, freebsd-performance@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: ZFS, NFS and Network tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2009 20:19:31 -0000 On Thu, 29 Jan 2009, Brent Jones wrote: > On Wed, Jan 28, 2009 at 11:21 PM, Brent Jones wrote: >> ... >> The issue I am seeing, is that for certain file types, the FreeBSD NFS >> client will either issue an ASYNC write, or an FSYNC. >> However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP >> versions of the protocol, so that should be the default. >> Issuing FSYNC's for every compete block transmitted adds substantial >> overhead and slows everything down. I use some patches (mainly for nfs write clustering on the server) by Bjorn Gronwall and some local fixes (mainly for vfs write clustering on the server, and tuning off excessive nfs[io]d daemons which get in each other's way due to poor scheduling, and things that only help for lots of small files), and see reasonable performance in all cases (~90% of disk bandwidth with all-async mounts, and half that with the client mounted noasync on an old version of FreeBSD. The client in -current is faster.) Writing is actually faster than reading here. >> ... >> My NFS mount command lines I have tried to get all data to ASYNC write: >> >> $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ >> $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ >> $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ Also try -r16384 -w16384, and udp, and async on the server. I think block sizes default to 8K for udp and 32K for tcp. 8K is too small, and 32K may be too large (it increases latency for little benefit if the server fs block size is 16K). udp gives lower latency. async on the server makes little difference provided the server block size is not too small. > I have found a 4 year old bug, which may be related to this. cp uses > mmap for small files (and I imagine lots of things use mmap for file > operations) and causes slowdowns via NFS, due to the fsync data > provided above. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/87792 mmap apparently breaks the async mount preference in the following code: from vnode_pager.c: % /* % * pageouts are already clustered, use IO_ASYNC t o force a bawrite() % * rather then a bdwrite() to prevent paging I/O from saturating % * the buffer cache. Dummy-up the sequential heuristic to cause % * large ranges to cluster. If neither IO_SYNC or IO_ASYNC is set, % * the system decides how to cluster. % */ % ioflags = IO_VMIO; % if (flags & (VM_PAGER_PUT_SYNC | VM_PAGER_PUT_INVAL)) % ioflags |= IO_SYNC; This apparently gives lots of sync writes. (Sync writes are the default for nfs, but we mount with async to try to get async writes.) % else if ((flags & VM_PAGER_CLUSTER_OK) == 0) % ioflags |= IO_ASYNC; nfs doesn't even support this flag. In fact, ffs is the only file system that supports it, and here is the only place that sets it. This might explain some slowness. One of the bugs in vfs clustering that I don't have is related to this. IIRC, mounting the server with -o async doesn't work as well as it should because the buffer cache becomes congested with i/o that should have been sent to the disk. Some writes must be done async as explained above, but one place in vfs_cache.c is too agressive in delaying async writes for file systems that are mounted async. This problem is more noticeable for nfs, at least with networks not much faster than disks, since it results in the client and server taking turns waiting for each other. (The names here are very confusing -- the async mount flag normally delays both sync and async writes for as long as possible, except for nfs it doesn't affect delays but asks for async writes instead of sync writes on the server, while the IO_ASYNC flag asks for async writes and thus often has the opposite sense to the async mount flag.) % ioflags |= (flags & VM_PAGER_PUT_INVAL) ? IO_INVAL: 0; % ioflags |= IO_SEQMAX << IO_SEQSHIFT; Bruce From owner-freebsd-performance@FreeBSD.ORG Sat Jan 31 00:08:02 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61C4010656E4; Sat, 31 Jan 2009 00:08:02 +0000 (UTC) (envelope-from brent@servuhome.net) Received: from mail-bw0-f16.google.com (mail-bw0-f16.google.com [209.85.218.16]) by mx1.freebsd.org (Postfix) with ESMTP id 8FBA38FC18; Sat, 31 Jan 2009 00:08:01 +0000 (UTC) (envelope-from brent@servuhome.net) Received: by bwz9 with SMTP id 9so139942bwz.19 for ; Fri, 30 Jan 2009 16:08:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.180.232.16 with SMTP id e16mr621092bkh.77.1233360480034; Fri, 30 Jan 2009 16:08:00 -0800 (PST) In-Reply-To: <20090129230247.GF4375@elvis.mu.org> References: <20090129230247.GF4375@elvis.mu.org> Date: Fri, 30 Jan 2009 16:07:59 -0800 Message-ID: From: Brent Jones To: Alfred Perlstein Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org Subject: Re: NFS writes calling FSYNC and ASYNC not consistent X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Jan 2009 00:08:03 -0000 On Thu, Jan 29, 2009 at 3:02 PM, Alfred Perlstein wrote: > Apologies for being terse, in a hurry here. > > 1) -o async doesn't work with NFS, don't use that. > 2) how big are the text versus binary files? I tested with a 6MB text file, and a 2GB binary file. Text file would go ~1MB/sec, issuing FSYNCs for every block write size (32KB default) One thing I found that another FreeBSD user discovered exactly explains my situation: In bin/cp/utils.c (source) there is a check, if the file is less than 8MB or so, it uses mmap, if the file is larger, it will use write() I modified the source and recompiled to -never- use mmap, only to use write(), and my performance increased about 100 fold (from 1MB/sec over NFS, to over 100MB/sec). Changed line 143: original: fs->st_size <= 8 * 1048576) { New: fs->st_size <= 8 * 8) { It will use mmap still if the file is larger than 64bytes (if it uses bytes there, pretty sure it does). But is much faster for files ~1KB to 8MB now. Regards > 3) how are you copying them over nfs? > > I suspect, (could be wrong of course) that the ascii files > are a lot smaller than the binary files, so what's happening > is that for binary files, the client is issuing write-behind > async, however for ascii files its issuing the writes at > close time which will force the sync flag. > > -Alfred > -- Brent Jones brent@servuhome.net