Date: Tue, 8 Nov 2011 03:26:44 -0600 (CST) From: Dan The Man <dan@sunsaturn.com> To: Garrett Cooper <yanegomi@gmail.com> Cc: freebsd-current@freebsd.org Subject: Re: samba+zfs Message-ID: <alpine.BSF.2.00.1111080259270.89703@sunsaturn.com> In-Reply-To: <CAGH67wRZZx0hG9ug2k-5ohCOPJ9sZOU9iFVKg7hv9WM=R761GA@mail.gmail.com> References: <alpine.BSF.2.00.1110272039500.50739@sunsaturn.com> <CAGH67wRZZx0hG9ug2k-5ohCOPJ9sZOU9iFVKg7hv9WM=R761GA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Ok here is some specs: this been running fine on 8.2 stable and i was sure it was running fine on RC1 as well. I did some testing against samba 34 35 and 36 in the ports collection all with the same slow write problems. I did further testing mounting drive in question with NFS and it did not suffer the same problem, so it seems just samba related here, where samba would actually outperform my NFS mount before, now its taking 10x as long to write anything. This is really most simplistic setup I have, all I want to do is map a network drive at the house read/write so my laptop, desktop etc all have access to it. I have played with all the smb.conf options, and can't seem to find where the issue is, further research suggests others are experiencing same problems with beta3 from following forum post: http://forums.freebsd.org/showthread.php?t=27300 Hardware this is running on: I beleive a 4 year old amd chip and board, with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 SSD as UFS boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap hitachi drives for my storage space,which is mirrored nightly with rsync with another duplicate machine(cause I know someone is going to say why not use raid5-raidz) Network specs: machine currently has dedicated IPV4 and gif0 tunneled IPV6 addresses to he.net. I've played with nearly every option in smb.conf disabling, enabling etc and can't seem to find the issue: here are my current config file settings on machine that could apply to samba: asterisk:~# cat /boot/loader.conf autoboot_delay="5" accf_data_load="YES" aio_load="YES" zfs_load="YES" kern.maxbcache=64M kern.ipc.maxpipekva=4M vfs.zfs.prefetch_disable=1 vm.kmem_size="1844M" vfs.zfs.arc_min="1024M" vfs.zfs.arc_max="1536M" vfs.zfs.vdev.min_pending=2 vfs.zfs.vdev.max_pending=8 vfs.zfs.txg.timeout=5 vfs.zfs.zil_disable="1" ahci_load="YES" asterisk:~# asterisk:~# cat /usr/local/etc/smb.conf # Global parameters [global] workgroup = HOME netbios name = ASTERISK server string = "Primary backups" interfaces = sk0 #smb ports = 139 #security = USER security = SHARE encrypt passwords = Yes #socket options = TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536 domain master = no wins support = yes guest account = root socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY use sendfile = no level2 oplocks = True read raw = no write cache size = 262144 min receivefile size = 16384 aio read size = 16384 aio write size = 16384 aio write behind = yes dns proxy = no max log size = 50 #log file = /dev/null log file = /var/log/samba.log debug level = 1 syslog = 0 [data] comment = "Primary backups" path = /data/public read only = No guest ok = Yes valid users = root asterisk:~# asterisk:~# cat /etc/sysctl.conf # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith Exp $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Uncomment this to prevent users from seeing information about processes that # are being run under another UID. #security.bsd.see_other_uids=0 #raise file descriptors on the system kern.maxfiles=204916 kern.maxfilesperproc=204916 #raise sockets we can accept kern.ipc.somaxconn=32768 #http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html kern.ipc.maxsockbuf=16777216 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=131072 #NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations net.inet.icmp.icmplim=500 kern.ipc.nmbjumbop=192000 kern.ipc.nmbclusters=229376 kern.ipc.maxsockets=204800 net.inet.tcp.maxtcptw=163840 #also add following to /boot/loader.conf #vm.kmem_size=1844M #kern.maxbcache=64M #kern.ipc.maxpipekva=4M #default setting of net.inet.ip.portrange.first is to low, causing us problems with bind net.inet.ip.portrange.last=65535 net.inet.ip.portrange.first=1024 #DOS protection net.inet.tcp.msl=7500 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.icmp.icmplim=50 net.inet.ip.accept_sourceroute=0 net.inet.ip.sourceroute=0 #some stuff for samba kern.ipc.nmbclusters=32768 kern.maxvnodes=800000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.sendbuf_inc=524288 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65535 net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 net.inet.tcp.mssdflt=1460 #IPSEC net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=1 kern.module_path=/boot/kernel;/boot/modules;/usr/local/modules #NFS--not concerned with data integrity when playing mostly already stored movies vfs.nfsrv.async=1 #JAIL #i like to use ping etc inside jail! security.jail.allow_raw_sockets=1 asterisk:~# Here are logs of me trying to mux a DTS mkv file from samba.log on debug level 10, I get the following over and over again: [2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data) [2011/11/08 03:24:00.067981, 3] smbd/process.c:1466(switch_message) switch message SMBwriteX (pid 64308) conn 0x805008450 [2011/11/08 03:24:00.067990, 4] smbd/uid.c:345(change_to_user) Skipping user change - already user [2011/11/08 03:24:00.068001, 10] locking/locking.c:120(strict_lock_default) is_locked: optimisation - exclusive oplock on file torrent_downloads_finished/Point.Break.1991.720p (1).mkv [2011/11/08 03:24:00.068010, 10] locking/locking.c:162(strict_lock_default) strict_lock_default: flavour = WINDOWS_LOCK brl start=83431665 len=65536 unlocked for fnum 49966 file torrent_downloads_finished/Point.Break.1991.720p (1).mkv [2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfile) default_sys_recvfile: from = 33, to = 39, offset=83431665, count = 65536 [2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file) real_write_file (torrent_downloads_finished/Point.Break.1991.720p (1).mkv): pos = 83431665, size = 65536, returned 65536 [2011/11/08 03:24:00.069013, 3] smbd/reply.c:4639(reply_write_and_X) writeX fnum=49966 num=65536 wrote=65536 [2011/11/08 03:24:00.069038, 10] lib/util_sock.c:516(read_smb_length_return_keepalive) got smb length of 65600 [2011/11/08 03:24:00.069052, 10] smbd/reply.c:4459(is_valid_writeX_buffer) is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536 [2011/11/08 03:24:00.069062, 6] smbd/process.c:1659(process_smb) got message type 0x0 of len 0x3f [2011/11/08 03:24:00.069072, 3] smbd/process.c:1661(process_smb) Transaction 15398 of length 67 (65536 toread) [2011/11/08 03:24:00.069081, 5] lib/util.c:332(show_msg) [2011/11/08 03:24:00.069087, 5] lib/util.c:342(show_msg) size=63 smb_com=0x2f smb_rcls=0 smb_reh=0 smb_err=0 smb_flg=24 smb_flg2=51207 smb_tid=1 smb_pid=65279 smb_uid=0 smb_mid=36032 smt_wct=14 smb_vwv[ 0]= 255 (0xFF) smb_vwv[ 1]=57054 (0xDEDE) smb_vwv[ 2]=49966 (0xC32E) smb_vwv[ 3]= 4337 (0x10F1) smb_vwv[ 4]= 1274 (0x4FA) smb_vwv[ 5]=65535 (0xFFFF) smb_vwv[ 6]=65535 (0xFFFF) smb_vwv[ 7]= 0 (0x0) smb_vwv[ 8]= 0 (0x0) smb_vwv[ 9]= 1 (0x1) smb_vwv[10]= 0 (0x0) smb_vwv[11]= 64 (0x40) smb_vwv[12]= 0 (0x0) smb_vwv[13]= 0 (0x0) smb_bcc=0 [2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data) [2011/11/08 03:24:00.069170, 3] smbd/process.c:1466(switch_message) switch message SMBwriteX (pid 64308) conn 0x805008450 [2011/11/08 03:24:00.069179, 4] smbd/uid.c:345(change_to_user) Skipping user change - already user [2011/11/08 03:24:00.069188, 10] locking/locking.c:120(strict_lock_default) is_locked: optimisation - exclusive oplock on file torrent_downloads_finished/Point.Break.1991.720p (1).mkv [2011/11/08 03:24:00.069197, 10] locking/locking.c:162(strict_lock_default) strict_lock_default: flavour = WINDOWS_LOCK brl start=83497201 len=65536 unlocked for fnum 49966 file torrent_downloads_finished/Point.Break.1991.720p (1).mkv [2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfile) default_sys_recvfile: from = 33, to = 39, offset=83497201, count = 65536 [2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file) real_write_file (torrent_downloads_finished/Point.Break.1991.720p (1).mkv): pos = 83497201, size = 65536, returned 65536 [2011/11/08 03:24:00.070004, 3] smbd/reply.c:4639(reply_write_and_X) writeX fnum=49966 num=65536 wrote=65536 [2011/11/08 03:24:00.070030, 10] lib/util_sock.c:516(read_smb_length_return_keepalive) got smb length of 65600 [2011/11/08 03:24:00.070044, 10] smbd/reply.c:4459(is_valid_writeX_buffer) is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536 [2011/11/08 03:24:00.070053, 6] smbd/process.c:1659(process_smb) got message type 0x0 of len 0x3f [2011/11/08 03:24:00.070063, 3] smbd/process.c:1661(process_smb) Transaction 15399 of length 67 (65536 toread) [2011/11/08 03:24:00.070072, 5] lib/util.c:332(show_msg) [2011/11/08 03:24:00.070077, 5] lib/util.c:342(show_msg) size=63 smb_com=0x2f smb_rcls=0 smb_reh=0 smb_err=0 smb_flg=24 smb_flg2=51207 smb_tid=1 smb_pid=65279 smb_uid=0 smb_mid=36102 smt_wct=14 smb_vwv[ 0]= 255 (0xFF) smb_vwv[ 1]=57054 (0xDEDE) smb_vwv[ 2]=49966 (0xC32E) smb_vwv[ 3]= 4337 (0x10F1) smb_vwv[ 4]= 1275 (0x4FB) smb_vwv[ 5]=65535 (0xFFFF) smb_vwv[ 6]=65535 (0xFFFF) smb_vwv[ 7]= 0 (0x0) smb_vwv[ 8]= 0 (0x0) smb_vwv[ 9]= 1 (0x1) smb_vwv[10]= 0 (0x0) smb_vwv[11]= 64 (0x40) smb_vwv[12]= 0 (0x0) smb_vwv[13]= 0 (0x0) smb_bcc=0 Hopefully maybe someone can shine some light on this.... Dan. -- Dan The Man CTO/ Senior System Administrator Websites, Domains and Everything else http://www.SunSaturn.com Email: Dan@SunSaturn.com On Fri, 28 Oct 2011, Garrett Cooper wrote: > On Thu, Oct 27, 2011 at 6:42 PM, Dan <dan@sunsaturn.com> wrote: >> >> >> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs >> its taking over an hour to just mux in things like DTS english, where it was >> 15 minutes on beta3. > > Hi Dan, > - Can you do more deterministic / scientific benchmarks? > - Did you upgrade Samba? > - What is your system's operating hardware profile? > Thanks! > -Garrett >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1111080259270.89703>