From owner-freebsd-current@FreeBSD.ORG Wed Nov 9 05:59:56 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 936201065674 for ; Wed, 9 Nov 2011 05:59:56 +0000 (UTC) (envelope-from ktouet@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 15D438FC1B for ; Wed, 9 Nov 2011 05:59:55 +0000 (UTC) Received: by wyg36 with SMTP id 36so1737298wyg.13 for ; Tue, 08 Nov 2011 21:59:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=DHR7EedZMItYo77BVcuGQemeUFCqOxZAZI+Prse4Zgk=; b=r3iclLjXbLVNCh6HE3DqqsX7kijHOd6gPWQUqWsfl3bhztJKGHrgrjDfLLlSHoozNi SrmNatHJI939dtNNSVNBaAkZAKxjrLxwppYpHidE34voPST1133bs5BLil6dUTqr8ZpS tuS9KsBAmym/vgOI3qf6tCONkOfYhhmholv6o= MIME-Version: 1.0 Received: by 10.216.135.91 with SMTP id t69mr155112wei.63.1320818394810; Tue, 08 Nov 2011 21:59:54 -0800 (PST) Received: by 10.216.37.19 with HTTP; Tue, 8 Nov 2011 21:59:54 -0800 (PST) In-Reply-To: References: Date: Tue, 8 Nov 2011 23:59:54 -0600 Message-ID: From: Kurt Touet To: Dan The Man Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current@freebsd.org Subject: Re: samba+zfs X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Nov 2011 05:59:56 -0000 On Tue, Nov 8, 2011 at 8:16 PM, Dan The Man wrote: > > On Tue, 8 Nov 2011, Kurt Touet wrote: > >> On Tue, Nov 8, 2011 at 3:32 AM, Dan The Man wrote: >>> >>> >>> Sorry I meant it was running fine on beta3 and 8.2 stable, and NOT RC1: >>> asterisk:~# uname -a >>> FreeBSD asterisk.sunsaturn.com 9.0-RC1 FreeBSD 9.0-RC1 #0: Mon Oct 31 >>> 19:46:53 CDT 2011 >>> droot@asterisk.sunsaturn.com:/usr/obj/usr/src/sys/MYKERNEL >>> =A0amd64 >>> asterisk:~# >>> >>> >>> >>> Dan. >>> >>> >>> -- >>> Dan The Man >>> CTO/ Senior System Administrator >>> Websites, Domains and Everything else >>> http://www.SunSaturn.com >>> Email: Dan@SunSaturn.com >>> >>> On Tue, 8 Nov 2011, Dan The Man wrote: >>> >>>> >>>> Ok here is some specs: this been running fine on 8.2 stable and i was >>>> sure >>>> it was running fine on RC1 as well. I did some testing against samba 3= 4 >>>> 35 >>>> and 36 in the ports collection all with the same slow write problems. >>>> >>>> I did further testing mounting drive in question with NFS and it did n= ot >>>> suffer the same problem, so it seems just samba related here, where >>>> samba >>>> would actually outperform my NFS mount before, now its taking 10x as >>>> long >>>> to write anything. >>>> >>>> This is really most simplistic setup I have, all I want to do is map a >>>> network drive at the house read/write so my laptop, desktop etc all ha= ve >>>> access to it. I have played with all the smb.conf options, and can't >>>> seem >>>> to find where the issue is, further research suggests others are >>>> experiencing same problems with beta3 from following forum post: >>>> >>>> http://forums.freebsd.org/showthread.php?t=3D27300 >>>> >>>> Hardware this is running on: I beleive a 4 year old amd chip and board= , >>>> with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 S= SD >>>> as >>>> UFS boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap >>>> hitachi drives for my storage space,which is mirrored nightly with rsy= nc >>>> with another duplicate machine(cause I know someone is going to say wh= y >>>> not >>>> use raid5-raidz) >>>> >>>> Network specs: machine currently has dedicated IPV4 and gif0 tunneled >>>> IPV6 >>>> addresses to he.net. >>>> >>>> I've played with nearly every option in smb.conf disabling, enabling e= tc >>>> and can't seem to find the issue: here are my current config file >>>> settings >>>> on machine that could apply to samba: >>>> >>>> asterisk:~# cat /boot/loader.conf >>>> autoboot_delay=3D"5" >>>> accf_data_load=3D"YES" >>>> aio_load=3D"YES" >>>> zfs_load=3D"YES" >>>> kern.maxbcache=3D64M >>>> kern.ipc.maxpipekva=3D4M >>>> >>>> vfs.zfs.prefetch_disable=3D1 >>>> vm.kmem_size=3D"1844M" >>>> vfs.zfs.arc_min=3D"1024M" >>>> vfs.zfs.arc_max=3D"1536M" >>>> vfs.zfs.vdev.min_pending=3D2 >>>> vfs.zfs.vdev.max_pending=3D8 >>>> vfs.zfs.txg.timeout=3D5 >>>> vfs.zfs.zil_disable=3D"1" >>>> ahci_load=3D"YES" >>>> asterisk:~# >>>> >>>> asterisk:~# cat /usr/local/etc/smb.conf >>>> # Global parameters >>>> [global] >>>> =A0 =A0 =A0 workgroup =3D HOME >>>> =A0 =A0 =A0 netbios name =3D ASTERISK >>>> =A0 =A0 =A0 server string =3D "Primary backups" >>>> =A0 =A0 =A0 interfaces =3D sk0 >>>> =A0 =A0 =A0 #smb ports =3D 139 >>>> =A0 =A0 =A0 #security =3D USER >>>> =A0 =A0 =A0 security =3D SHARE >>>> =A0 =A0 =A0 encrypt passwords =3D Yes >>>> =A0 =A0 =A0 #socket options =3D TCP_NODELAY SO_RCVBUF=3D65536 SO_SNDBU= F=3D65536 >>>> =A0 =A0 =A0 domain master =3D no >>>> =A0 =A0 =A0 wins support =3D yes >>>> =A0 =A0 =A0 guest account =3D root >>>> =A0 =A0 =A0 socket options=3DSO_RCVBUF=3D131072 SO_SNDBUF=3D131072 TCP= _NODELAY >>>> =A0 =A0 =A0 use sendfile =3D no >>>> =A0 =A0 =A0 level2 oplocks =3D True >>>> =A0 =A0 =A0 read raw =3D no >>>> =A0 =A0 =A0 write cache size =3D 262144 >>>> =A0 =A0 =A0 min receivefile size =3D 16384 >>>> =A0 =A0 =A0 aio read size =3D 16384 >>>> =A0 =A0 =A0 aio write size =3D 16384 >>>> =A0 =A0 =A0 aio write behind =3D yes >>>> =A0 =A0 =A0 dns proxy =3D no >>>> =A0 =A0 =A0 max log size =3D 50 >>>> =A0 =A0 =A0 #log file =3D /dev/null >>>> =A0 =A0 =A0 log file =3D /var/log/samba.log >>>> =A0 =A0 =A0 debug level =3D 1 >>>> =A0 =A0 =A0 syslog =3D 0 >>>> >>>> [data] >>>> =A0 =A0 =A0 comment =3D "Primary backups" >>>> =A0 =A0 =A0 path =3D /data/public >>>> =A0 =A0 =A0 read only =3D No >>>> =A0 =A0 =A0 guest ok =3D Yes >>>> =A0 =A0 =A0 valid users =3D root >>>> asterisk:~# asterisk:~# cat /etc/sysctl.conf >>>> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmit= h >>>> Exp $ >>>> # >>>> # =A0This file is read when going to multi-user and its contents piped >>>> thru >>>> # =A0``sysctl'' to adjust kernel values. =A0``man 5 sysctl.conf'' for >>>> details. >>>> # >>>> >>>> # Uncomment this to prevent users from seeing information about >>>> processes >>>> that >>>> # are being run under another UID. >>>> #security.bsd.see_other_uids=3D0 >>>> >>>> #raise file descriptors on the system >>>> kern.maxfiles=3D204916 >>>> kern.maxfilesperproc=3D204916 >>>> >>>> #raise sockets we can accept >>>> kern.ipc.somaxconn=3D32768 >>>> >>>> #http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html >>>> kern.ipc.maxsockbuf=3D16777216 >>>> net.inet.tcp.rfc1323=3D1 >>>> net.inet.tcp.sendbuf_max=3D16777216 >>>> net.inet.tcp.recvbuf_max=3D16777216 >>>> net.inet.tcp.sendspace=3D65536 >>>> net.inet.tcp.recvspace=3D131072 >>>> >>>> #NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations >>>> net.inet.icmp.icmplim=3D500 >>>> kern.ipc.nmbjumbop=3D192000 >>>> kern.ipc.nmbclusters=3D229376 >>>> kern.ipc.maxsockets=3D204800 >>>> net.inet.tcp.maxtcptw=3D163840 >>>> #also add following to /boot/loader.conf >>>> #vm.kmem_size=3D1844M >>>> #kern.maxbcache=3D64M >>>> #kern.ipc.maxpipekva=3D4M >>>> >>>> #default setting of net.inet.ip.portrange.first is to low, causing us >>>> problems with bind >>>> net.inet.ip.portrange.last=3D65535 >>>> net.inet.ip.portrange.first=3D1024 >>>> >>>> #DOS protection >>>> net.inet.tcp.msl=3D7500 >>>> net.inet.tcp.blackhole=3D2 >>>> net.inet.udp.blackhole=3D1 >>>> net.inet.icmp.icmplim=3D50 >>>> net.inet.ip.accept_sourceroute=3D0 >>>> net.inet.ip.sourceroute=3D0 >>>> >>>> #some stuff for samba >>>> kern.ipc.nmbclusters=3D32768 >>>> kern.maxvnodes=3D800000 >>>> net.inet.tcp.delayed_ack=3D0 >>>> net.inet.tcp.inflight.enable=3D0 >>>> net.inet.tcp.path_mtu_discovery=3D0 >>>> net.inet.tcp.recvbuf_auto=3D1 >>>> net.inet.tcp.recvbuf_inc=3D524288 >>>> net.inet.tcp.sendbuf_auto=3D1 >>>> net.inet.tcp.sendbuf_inc=3D524288 >>>> net.inet.udp.maxdgram=3D57344 >>>> net.inet.udp.recvspace=3D65535 >>>> net.local.stream.recvspace=3D65535 >>>> net.local.stream.sendspace=3D65535 >>>> net.inet.tcp.mssdflt=3D1460 >>>> >>>> #IPSEC >>>> net.inet.ip.forwarding=3D1 >>>> net.inet6.ip6.forwarding=3D1 >>>> kern.module_path=3D/boot/kernel;/boot/modules;/usr/local/modules >>>> >>>> >>>> #NFS--not concerned with data integrity when playing mostly already >>>> stored >>>> movies >>>> vfs.nfsrv.async=3D1 >>>> >>>> #JAIL >>>> #i like to use ping etc inside jail! >>>> security.jail.allow_raw_sockets=3D1 >>>> asterisk:~# >>>> >>>> >>>> Here are logs of me trying to mux a DTS mkv file from samba.log on deb= ug >>>> level 10, I get the following over and over again: >>>> >>>> [2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data) >>>> [2011/11/08 03:24:00.067981, =A03] smbd/process.c:1466(switch_message) >>>> =A0switch message SMBwriteX (pid 64308) conn 0x805008450 >>>> [2011/11/08 03:24:00.067990, =A04] smbd/uid.c:345(change_to_user) >>>> =A0Skipping user change - already user >>>> [2011/11/08 03:24:00.068001, 10] >>>> locking/locking.c:120(strict_lock_default) >>>> =A0is_locked: optimisation - exclusive oplock on file >>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>>> [2011/11/08 03:24:00.068010, 10] >>>> locking/locking.c:162(strict_lock_default) >>>> =A0strict_lock_default: flavour =3D WINDOWS_LOCK brl start=3D83431665 >>>> len=3D65536 >>>> unlocked for fnum 49966 file >>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>>> [2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfil= e) >>>> =A0default_sys_recvfile: from =3D 33, to =3D 39, offset=3D83431665, co= unt =3D >>>> 65536 >>>> [2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file) >>>> =A0real_write_file (torrent_downloads_finished/Point.Break.1991.720p >>>> (1).mkv): pos =3D 83431665, size =3D 65536, returned 65536 >>>> [2011/11/08 03:24:00.069013, =A03] smbd/reply.c:4639(reply_write_and_X= ) >>>> =A0writeX fnum=3D49966 num=3D65536 wrote=3D65536 >>>> [2011/11/08 03:24:00.069038, 10] >>>> lib/util_sock.c:516(read_smb_length_return_keepalive) >>>> =A0got smb length of 65600 >>>> [2011/11/08 03:24:00.069052, 10] >>>> smbd/reply.c:4459(is_valid_writeX_buffer) >>>> =A0is_valid_writeX_buffer: true len =3D 65600, doff =3D 64, numtowrite= =3D 65536 >>>> [2011/11/08 03:24:00.069062, =A06] smbd/process.c:1659(process_smb) >>>> =A0got message type 0x0 of len 0x3f >>>> [2011/11/08 03:24:00.069072, =A03] smbd/process.c:1661(process_smb) >>>> =A0Transaction 15398 of length 67 (65536 toread) >>>> [2011/11/08 03:24:00.069081, =A05] lib/util.c:332(show_msg) >>>> [2011/11/08 03:24:00.069087, =A05] lib/util.c:342(show_msg) >>>> =A0size=3D63 >>>> =A0smb_com=3D0x2f >>>> =A0smb_rcls=3D0 >>>> =A0smb_reh=3D0 >>>> =A0smb_err=3D0 >>>> =A0smb_flg=3D24 >>>> =A0smb_flg2=3D51207 >>>> =A0smb_tid=3D1 >>>> =A0smb_pid=3D65279 >>>> =A0smb_uid=3D0 >>>> =A0smb_mid=3D36032 >>>> =A0smt_wct=3D14 >>>> =A0smb_vwv[ 0]=3D =A0255 (0xFF) >>>> =A0smb_vwv[ 1]=3D57054 (0xDEDE) >>>> =A0smb_vwv[ 2]=3D49966 (0xC32E) >>>> =A0smb_vwv[ 3]=3D 4337 (0x10F1) >>>> =A0smb_vwv[ 4]=3D 1274 (0x4FA) >>>> =A0smb_vwv[ 5]=3D65535 (0xFFFF) >>>> =A0smb_vwv[ 6]=3D65535 (0xFFFF) >>>> =A0smb_vwv[ 7]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[ 8]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[ 9]=3D =A0 =A01 (0x1) >>>> =A0smb_vwv[10]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[11]=3D =A0 64 (0x40) >>>> =A0smb_vwv[12]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[13]=3D =A0 =A00 (0x0) >>>> =A0smb_bcc=3D0 >>>> [2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data) >>>> [2011/11/08 03:24:00.069170, =A03] smbd/process.c:1466(switch_message) >>>> =A0switch message SMBwriteX (pid 64308) conn 0x805008450 >>>> [2011/11/08 03:24:00.069179, =A04] smbd/uid.c:345(change_to_user) >>>> =A0Skipping user change - already user >>>> [2011/11/08 03:24:00.069188, 10] >>>> locking/locking.c:120(strict_lock_default) >>>> =A0is_locked: optimisation - exclusive oplock on file >>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>>> [2011/11/08 03:24:00.069197, 10] >>>> locking/locking.c:162(strict_lock_default) >>>> =A0strict_lock_default: flavour =3D WINDOWS_LOCK brl start=3D83497201 >>>> len=3D65536 >>>> unlocked for fnum 49966 file >>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>>> [2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfil= e) >>>> =A0default_sys_recvfile: from =3D 33, to =3D 39, offset=3D83497201, co= unt =3D >>>> 65536 >>>> [2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file) >>>> =A0real_write_file (torrent_downloads_finished/Point.Break.1991.720p >>>> (1).mkv): pos =3D 83497201, size =3D 65536, returned 65536 >>>> [2011/11/08 03:24:00.070004, =A03] smbd/reply.c:4639(reply_write_and_X= ) >>>> =A0writeX fnum=3D49966 num=3D65536 wrote=3D65536 >>>> [2011/11/08 03:24:00.070030, 10] >>>> lib/util_sock.c:516(read_smb_length_return_keepalive) >>>> =A0got smb length of 65600 >>>> [2011/11/08 03:24:00.070044, 10] >>>> smbd/reply.c:4459(is_valid_writeX_buffer) >>>> =A0is_valid_writeX_buffer: true len =3D 65600, doff =3D 64, numtowrite= =3D 65536 >>>> [2011/11/08 03:24:00.070053, =A06] smbd/process.c:1659(process_smb) >>>> =A0got message type 0x0 of len 0x3f >>>> [2011/11/08 03:24:00.070063, =A03] smbd/process.c:1661(process_smb) >>>> =A0Transaction 15399 of length 67 (65536 toread) >>>> [2011/11/08 03:24:00.070072, =A05] lib/util.c:332(show_msg) >>>> [2011/11/08 03:24:00.070077, =A05] lib/util.c:342(show_msg) >>>> =A0size=3D63 >>>> =A0smb_com=3D0x2f >>>> =A0smb_rcls=3D0 >>>> =A0smb_reh=3D0 >>>> =A0smb_err=3D0 >>>> =A0smb_flg=3D24 >>>> =A0smb_flg2=3D51207 >>>> =A0smb_tid=3D1 >>>> =A0smb_pid=3D65279 >>>> =A0smb_uid=3D0 >>>> =A0smb_mid=3D36102 >>>> =A0smt_wct=3D14 >>>> =A0smb_vwv[ 0]=3D =A0255 (0xFF) >>>> =A0smb_vwv[ 1]=3D57054 (0xDEDE) >>>> =A0smb_vwv[ 2]=3D49966 (0xC32E) >>>> =A0smb_vwv[ 3]=3D 4337 (0x10F1) >>>> =A0smb_vwv[ 4]=3D 1275 (0x4FB) >>>> =A0smb_vwv[ 5]=3D65535 (0xFFFF) >>>> =A0smb_vwv[ 6]=3D65535 (0xFFFF) >>>> =A0smb_vwv[ 7]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[ 8]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[ 9]=3D =A0 =A01 (0x1) >>>> =A0smb_vwv[10]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[11]=3D =A0 64 (0x40) >>>> =A0smb_vwv[12]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[13]=3D =A0 =A00 (0x0) >>>> =A0smb_bcc=3D0 >>>> >>>> >>>> Hopefully maybe someone can shine some light on this.... >>>> >>>> >>>> Dan. >>>> >>>> >>>> -- >>>> Dan The Man >>>> CTO/ Senior System Administrator >>>> Websites, Domains and Everything else >>>> http://www.SunSaturn.com >>>> Email: Dan@SunSaturn.com >>>> >>>> On Fri, 28 Oct 2011, Garrett Cooper wrote: >>>> >>>>> On Thu, Oct 27, 2011 at 6:42 PM, Dan wrote: >>>>>> >>>>>> >>>>>> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs >>>>>> its taking over an hour to just mux in things like DTS english, wher= e >>>>>> it >>>>>> was >>>>>> 15 minutes on beta3. >>>>> >>>>> Hi Dan, >>>>> - Can you do more deterministic / scientific benchmarks? >>>>> - Did you upgrade Samba? >>>>> - What is your system's operating hardware profile? >>>>> Thanks! >>>>> -Garrett >>>>> >> >> I had noticed a similar problem with doing large writes from my >> windows workstation to my freebsd ZFS array. =A0Previously I had been >> able to write at around 30MB/s (limited by a slow SATA controller), >> and those speeds had dropped to 3-5MB/s with long pauses between >> writes to the array (monitoring with zpool iostat). =A0 However, I just >> updated to stable/9 r227357 and the issue seems to have gone away; I'm >> back up to 30MB/s writes. >> >> Hope you find the same sort of thing, >> -kurt >> > > Yep your right, on RC2 my samba ZFS write speeds are back. > Thanks for heads up Kurt. > > > Dan. > > > -- > Dan The Man > CTO/ Senior System Administrator > Websites, Domains and Everything else > http://www.SunSaturn.com > Email: Dan@SunSaturn.com > I may have spoken too soon. With the speeds back up to normal, I moved a bunch of content that I'd been storing temporarily on my workstation to the server. After a few hours and moving 40gb+ of content (not all at once), I noticed the same write problems cropping up again. I rebooted the server and speeds came back to normal. The oddity that I notice is that the array seems to be delaying writes to a single disc in my raidz1 vdev, and then writing out very slowly to it (1-2MB/s). The drive that it is doing it with has absolutely zero read/write/seek errors in SMART, and works just fine after the box is rebooted (so it shouldn't be a drive issue). The only thing interesting about this drive is that it replaced a failed drive a few weeks ago. Is anyone else seeing problems like this with samba/zfs ? Perhaps it's not exclusive to samba, either? -kurt