Date: Tue, 8 Nov 2011 16:28:56 -0600 From: Kurt Touet <ktouet@gmail.com> To: Dan The Man <dan@sunsaturn.com> Cc: Garrett Cooper <yanegomi@gmail.com>, freebsd-current@freebsd.org Subject: Re: samba+zfs Message-ID: <CALM%2B6aJkF=CFq8LA3FrSMYo8La-8txK4h2p4yZtdHshskBU6Vw@mail.gmail.com> In-Reply-To: <alpine.BSF.2.00.1111080328010.89703@sunsaturn.com> References: <alpine.BSF.2.00.1110272039500.50739@sunsaturn.com> <CAGH67wRZZx0hG9ug2k-5ohCOPJ9sZOU9iFVKg7hv9WM=R761GA@mail.gmail.com> <alpine.BSF.2.00.1111080259270.89703@sunsaturn.com> <alpine.BSF.2.00.1111080328010.89703@sunsaturn.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Nov 8, 2011 at 3:32 AM, Dan The Man <dan@sunsaturn.com> wrote: > > > Sorry I meant it was running fine on beta3 and 8.2 stable, and NOT RC1: > asterisk:~# uname -a > FreeBSD asterisk.sunsaturn.com 9.0-RC1 FreeBSD 9.0-RC1 #0: Mon Oct 31 > 19:46:53 CDT 2011 droot@asterisk.sunsaturn.com:/usr/obj/usr/src/sys/MYKER= NEL > =A0amd64 > asterisk:~# > > > > Dan. > > > -- > Dan The Man > CTO/ Senior System Administrator > Websites, Domains and Everything else > http://www.SunSaturn.com > Email: Dan@SunSaturn.com > > On Tue, 8 Nov 2011, Dan The Man wrote: > >> >> Ok here is some specs: this been running fine on 8.2 stable and i was su= re >> it was running fine on RC1 as well. I did some testing against samba 34 = 35 >> and 36 in the ports collection all with the same slow write problems. >> >> I did further testing mounting drive in question with NFS and it did not >> suffer the same problem, so it seems just samba related here, where samb= a >> would actually outperform my NFS mount before, now its taking 10x as lon= g >> to write anything. >> >> This is really most simplistic setup I have, all I want to do is map a >> network drive at the house read/write so my laptop, desktop etc all have >> access to it. I have played with all the smb.conf options, and can't see= m >> to find where the issue is, further research suggests others are >> experiencing same problems with beta3 from following forum post: >> >> http://forums.freebsd.org/showthread.php?t=3D27300 >> >> Hardware this is running on: I beleive a 4 year old amd chip and board, >> with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 SSD= as >> UFS boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap >> hitachi drives for my storage space,which is mirrored nightly with rsync >> with another duplicate machine(cause I know someone is going to say why = not >> use raid5-raidz) >> >> Network specs: machine currently has dedicated IPV4 and gif0 tunneled IP= V6 >> addresses to he.net. >> >> I've played with nearly every option in smb.conf disabling, enabling etc >> and can't seem to find the issue: here are my current config file settin= gs >> on machine that could apply to samba: >> >> asterisk:~# cat /boot/loader.conf >> autoboot_delay=3D"5" >> accf_data_load=3D"YES" >> aio_load=3D"YES" >> zfs_load=3D"YES" >> kern.maxbcache=3D64M >> kern.ipc.maxpipekva=3D4M >> >> vfs.zfs.prefetch_disable=3D1 >> vm.kmem_size=3D"1844M" >> vfs.zfs.arc_min=3D"1024M" >> vfs.zfs.arc_max=3D"1536M" >> vfs.zfs.vdev.min_pending=3D2 >> vfs.zfs.vdev.max_pending=3D8 >> vfs.zfs.txg.timeout=3D5 >> vfs.zfs.zil_disable=3D"1" >> ahci_load=3D"YES" >> asterisk:~# >> >> asterisk:~# cat /usr/local/etc/smb.conf >> # Global parameters >> [global] >> =A0 =A0 =A0 workgroup =3D HOME >> =A0 =A0 =A0 netbios name =3D ASTERISK >> =A0 =A0 =A0 server string =3D "Primary backups" >> =A0 =A0 =A0 interfaces =3D sk0 >> =A0 =A0 =A0 #smb ports =3D 139 >> =A0 =A0 =A0 #security =3D USER >> =A0 =A0 =A0 security =3D SHARE >> =A0 =A0 =A0 encrypt passwords =3D Yes >> =A0 =A0 =A0 #socket options =3D TCP_NODELAY SO_RCVBUF=3D65536 SO_SNDBUF= =3D65536 >> =A0 =A0 =A0 domain master =3D no >> =A0 =A0 =A0 wins support =3D yes >> =A0 =A0 =A0 guest account =3D root >> =A0 =A0 =A0 socket options=3DSO_RCVBUF=3D131072 SO_SNDBUF=3D131072 TCP_N= ODELAY >> =A0 =A0 =A0 use sendfile =3D no >> =A0 =A0 =A0 level2 oplocks =3D True >> =A0 =A0 =A0 read raw =3D no >> =A0 =A0 =A0 write cache size =3D 262144 >> =A0 =A0 =A0 min receivefile size =3D 16384 >> =A0 =A0 =A0 aio read size =3D 16384 >> =A0 =A0 =A0 aio write size =3D 16384 >> =A0 =A0 =A0 aio write behind =3D yes >> =A0 =A0 =A0 dns proxy =3D no >> =A0 =A0 =A0 max log size =3D 50 >> =A0 =A0 =A0 #log file =3D /dev/null >> =A0 =A0 =A0 log file =3D /var/log/samba.log >> =A0 =A0 =A0 debug level =3D 1 >> =A0 =A0 =A0 syslog =3D 0 >> >> [data] >> =A0 =A0 =A0 comment =3D "Primary backups" >> =A0 =A0 =A0 path =3D /data/public >> =A0 =A0 =A0 read only =3D No >> =A0 =A0 =A0 guest ok =3D Yes >> =A0 =A0 =A0 valid users =3D root >> asterisk:~# asterisk:~# cat /etc/sysctl.conf >> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith >> Exp $ >> # >> # =A0This file is read when going to multi-user and its contents piped t= hru >> # =A0``sysctl'' to adjust kernel values. =A0``man 5 sysctl.conf'' for de= tails. >> # >> >> # Uncomment this to prevent users from seeing information about processe= s >> that >> # are being run under another UID. >> #security.bsd.see_other_uids=3D0 >> >> #raise file descriptors on the system >> kern.maxfiles=3D204916 >> kern.maxfilesperproc=3D204916 >> >> #raise sockets we can accept >> kern.ipc.somaxconn=3D32768 >> >> #http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html >> kern.ipc.maxsockbuf=3D16777216 >> net.inet.tcp.rfc1323=3D1 >> net.inet.tcp.sendbuf_max=3D16777216 >> net.inet.tcp.recvbuf_max=3D16777216 >> net.inet.tcp.sendspace=3D65536 >> net.inet.tcp.recvspace=3D131072 >> >> #NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations >> net.inet.icmp.icmplim=3D500 >> kern.ipc.nmbjumbop=3D192000 >> kern.ipc.nmbclusters=3D229376 >> kern.ipc.maxsockets=3D204800 >> net.inet.tcp.maxtcptw=3D163840 >> #also add following to /boot/loader.conf >> #vm.kmem_size=3D1844M >> #kern.maxbcache=3D64M >> #kern.ipc.maxpipekva=3D4M >> >> #default setting of net.inet.ip.portrange.first is to low, causing us >> problems with bind >> net.inet.ip.portrange.last=3D65535 >> net.inet.ip.portrange.first=3D1024 >> >> #DOS protection >> net.inet.tcp.msl=3D7500 >> net.inet.tcp.blackhole=3D2 >> net.inet.udp.blackhole=3D1 >> net.inet.icmp.icmplim=3D50 >> net.inet.ip.accept_sourceroute=3D0 >> net.inet.ip.sourceroute=3D0 >> >> #some stuff for samba >> kern.ipc.nmbclusters=3D32768 >> kern.maxvnodes=3D800000 >> net.inet.tcp.delayed_ack=3D0 >> net.inet.tcp.inflight.enable=3D0 >> net.inet.tcp.path_mtu_discovery=3D0 >> net.inet.tcp.recvbuf_auto=3D1 >> net.inet.tcp.recvbuf_inc=3D524288 >> net.inet.tcp.sendbuf_auto=3D1 >> net.inet.tcp.sendbuf_inc=3D524288 >> net.inet.udp.maxdgram=3D57344 >> net.inet.udp.recvspace=3D65535 >> net.local.stream.recvspace=3D65535 >> net.local.stream.sendspace=3D65535 >> net.inet.tcp.mssdflt=3D1460 >> >> #IPSEC >> net.inet.ip.forwarding=3D1 >> net.inet6.ip6.forwarding=3D1 >> kern.module_path=3D/boot/kernel;/boot/modules;/usr/local/modules >> >> >> #NFS--not concerned with data integrity when playing mostly already stor= ed >> movies >> vfs.nfsrv.async=3D1 >> >> #JAIL >> #i like to use ping etc inside jail! >> security.jail.allow_raw_sockets=3D1 >> asterisk:~# >> >> >> Here are logs of me trying to mux a DTS mkv file from samba.log on debug >> level 10, I get the following over and over again: >> >> [2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data) >> [2011/11/08 03:24:00.067981, =A03] smbd/process.c:1466(switch_message) >> =A0switch message SMBwriteX (pid 64308) conn 0x805008450 >> [2011/11/08 03:24:00.067990, =A04] smbd/uid.c:345(change_to_user) >> =A0Skipping user change - already user >> [2011/11/08 03:24:00.068001, 10] >> locking/locking.c:120(strict_lock_default) >> =A0is_locked: optimisation - exclusive oplock on file >> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >> [2011/11/08 03:24:00.068010, 10] >> locking/locking.c:162(strict_lock_default) >> =A0strict_lock_default: flavour =3D WINDOWS_LOCK brl start=3D83431665 le= n=3D65536 >> unlocked for fnum 49966 file >> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >> [2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfile) >> =A0default_sys_recvfile: from =3D 33, to =3D 39, offset=3D83431665, coun= t =3D 65536 >> [2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file) >> =A0real_write_file (torrent_downloads_finished/Point.Break.1991.720p >> (1).mkv): pos =3D 83431665, size =3D 65536, returned 65536 >> [2011/11/08 03:24:00.069013, =A03] smbd/reply.c:4639(reply_write_and_X) >> =A0writeX fnum=3D49966 num=3D65536 wrote=3D65536 >> [2011/11/08 03:24:00.069038, 10] >> lib/util_sock.c:516(read_smb_length_return_keepalive) >> =A0got smb length of 65600 >> [2011/11/08 03:24:00.069052, 10] smbd/reply.c:4459(is_valid_writeX_buffe= r) >> =A0is_valid_writeX_buffer: true len =3D 65600, doff =3D 64, numtowrite = =3D 65536 >> [2011/11/08 03:24:00.069062, =A06] smbd/process.c:1659(process_smb) >> =A0got message type 0x0 of len 0x3f >> [2011/11/08 03:24:00.069072, =A03] smbd/process.c:1661(process_smb) >> =A0Transaction 15398 of length 67 (65536 toread) >> [2011/11/08 03:24:00.069081, =A05] lib/util.c:332(show_msg) >> [2011/11/08 03:24:00.069087, =A05] lib/util.c:342(show_msg) >> =A0size=3D63 >> =A0smb_com=3D0x2f >> =A0smb_rcls=3D0 >> =A0smb_reh=3D0 >> =A0smb_err=3D0 >> =A0smb_flg=3D24 >> =A0smb_flg2=3D51207 >> =A0smb_tid=3D1 >> =A0smb_pid=3D65279 >> =A0smb_uid=3D0 >> =A0smb_mid=3D36032 >> =A0smt_wct=3D14 >> =A0smb_vwv[ 0]=3D =A0255 (0xFF) >> =A0smb_vwv[ 1]=3D57054 (0xDEDE) >> =A0smb_vwv[ 2]=3D49966 (0xC32E) >> =A0smb_vwv[ 3]=3D 4337 (0x10F1) >> =A0smb_vwv[ 4]=3D 1274 (0x4FA) >> =A0smb_vwv[ 5]=3D65535 (0xFFFF) >> =A0smb_vwv[ 6]=3D65535 (0xFFFF) >> =A0smb_vwv[ 7]=3D =A0 =A00 (0x0) >> =A0smb_vwv[ 8]=3D =A0 =A00 (0x0) >> =A0smb_vwv[ 9]=3D =A0 =A01 (0x1) >> =A0smb_vwv[10]=3D =A0 =A00 (0x0) >> =A0smb_vwv[11]=3D =A0 64 (0x40) >> =A0smb_vwv[12]=3D =A0 =A00 (0x0) >> =A0smb_vwv[13]=3D =A0 =A00 (0x0) >> =A0smb_bcc=3D0 >> [2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data) >> [2011/11/08 03:24:00.069170, =A03] smbd/process.c:1466(switch_message) >> =A0switch message SMBwriteX (pid 64308) conn 0x805008450 >> [2011/11/08 03:24:00.069179, =A04] smbd/uid.c:345(change_to_user) >> =A0Skipping user change - already user >> [2011/11/08 03:24:00.069188, 10] >> locking/locking.c:120(strict_lock_default) >> =A0is_locked: optimisation - exclusive oplock on file >> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >> [2011/11/08 03:24:00.069197, 10] >> locking/locking.c:162(strict_lock_default) >> =A0strict_lock_default: flavour =3D WINDOWS_LOCK brl start=3D83497201 le= n=3D65536 >> unlocked for fnum 49966 file >> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >> [2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfile) >> =A0default_sys_recvfile: from =3D 33, to =3D 39, offset=3D83497201, coun= t =3D 65536 >> [2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file) >> =A0real_write_file (torrent_downloads_finished/Point.Break.1991.720p >> (1).mkv): pos =3D 83497201, size =3D 65536, returned 65536 >> [2011/11/08 03:24:00.070004, =A03] smbd/reply.c:4639(reply_write_and_X) >> =A0writeX fnum=3D49966 num=3D65536 wrote=3D65536 >> [2011/11/08 03:24:00.070030, 10] >> lib/util_sock.c:516(read_smb_length_return_keepalive) >> =A0got smb length of 65600 >> [2011/11/08 03:24:00.070044, 10] smbd/reply.c:4459(is_valid_writeX_buffe= r) >> =A0is_valid_writeX_buffer: true len =3D 65600, doff =3D 64, numtowrite = =3D 65536 >> [2011/11/08 03:24:00.070053, =A06] smbd/process.c:1659(process_smb) >> =A0got message type 0x0 of len 0x3f >> [2011/11/08 03:24:00.070063, =A03] smbd/process.c:1661(process_smb) >> =A0Transaction 15399 of length 67 (65536 toread) >> [2011/11/08 03:24:00.070072, =A05] lib/util.c:332(show_msg) >> [2011/11/08 03:24:00.070077, =A05] lib/util.c:342(show_msg) >> =A0size=3D63 >> =A0smb_com=3D0x2f >> =A0smb_rcls=3D0 >> =A0smb_reh=3D0 >> =A0smb_err=3D0 >> =A0smb_flg=3D24 >> =A0smb_flg2=3D51207 >> =A0smb_tid=3D1 >> =A0smb_pid=3D65279 >> =A0smb_uid=3D0 >> =A0smb_mid=3D36102 >> =A0smt_wct=3D14 >> =A0smb_vwv[ 0]=3D =A0255 (0xFF) >> =A0smb_vwv[ 1]=3D57054 (0xDEDE) >> =A0smb_vwv[ 2]=3D49966 (0xC32E) >> =A0smb_vwv[ 3]=3D 4337 (0x10F1) >> =A0smb_vwv[ 4]=3D 1275 (0x4FB) >> =A0smb_vwv[ 5]=3D65535 (0xFFFF) >> =A0smb_vwv[ 6]=3D65535 (0xFFFF) >> =A0smb_vwv[ 7]=3D =A0 =A00 (0x0) >> =A0smb_vwv[ 8]=3D =A0 =A00 (0x0) >> =A0smb_vwv[ 9]=3D =A0 =A01 (0x1) >> =A0smb_vwv[10]=3D =A0 =A00 (0x0) >> =A0smb_vwv[11]=3D =A0 64 (0x40) >> =A0smb_vwv[12]=3D =A0 =A00 (0x0) >> =A0smb_vwv[13]=3D =A0 =A00 (0x0) >> =A0smb_bcc=3D0 >> >> >> Hopefully maybe someone can shine some light on this.... >> >> >> Dan. >> >> >> -- >> Dan The Man >> CTO/ Senior System Administrator >> Websites, Domains and Everything else >> http://www.SunSaturn.com >> Email: Dan@SunSaturn.com >> >> On Fri, 28 Oct 2011, Garrett Cooper wrote: >> >>> On Thu, Oct 27, 2011 at 6:42 PM, Dan <dan@sunsaturn.com> wrote: >>>> >>>> >>>> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs >>>> its taking over an hour to just mux in things like DTS english, where = it >>>> was >>>> 15 minutes on beta3. >>> >>> Hi Dan, >>> - Can you do more deterministic / scientific benchmarks? >>> - Did you upgrade Samba? >>> - What is your system's operating hardware profile? >>> Thanks! >>> -Garrett >>> I had noticed a similar problem with doing large writes from my windows workstation to my freebsd ZFS array. Previously I had been able to write at around 30MB/s (limited by a slow SATA controller), and those speeds had dropped to 3-5MB/s with long pauses between writes to the array (monitoring with zpool iostat). However, I just updated to stable/9 r227357 and the issue seems to have gone away; I'm back up to 30MB/s writes. Hope you find the same sort of thing, -kurt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALM%2B6aJkF=CFq8LA3FrSMYo8La-8txK4h2p4yZtdHshskBU6Vw>