Date: Mon, 08 Sep 2008 16:23:30 +0400 From: Dmitriy <_pppp@mail.ru> To: Zin'kov Oleg <noc@bg.net.ua> Cc: freebsd-net@FreeBSD.org Subject: Re: Problem with process parallelization Message-ID: <E1KcfmE-0000KR-00._pppp-mail-ru@f106.mail.ru> In-Reply-To: <79dc33e3f3737f5beeadce88e96004bc.squirrel@webmail.bg.net.ua> References: <79dc33e3f3737f5beeadce88e96004bc.squirrel@webmail.bg.net.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
> Hello, freebsd-net mailing list. > > We have server such configurtion: > - 2 quadcore AMD Opteron processors; > - 4 GB RAM; > - NIC Intel Pro/1000 PT, Dual Port Server Adapter. > > ########################################################### > > Problem: > > in some moments of time, at the growth of the network activity, one of > the processors is fully loaded at 100%. > > ########################################################### > > Kernel configuration: > > FreeBSD atlantis.bg.net.ua 7.0-STABLE FreeBSD 7.0-STABLE #1: Tue Apr 1 > 15:06:30 EEST 2008 > root@atlantis.bg.net.ua:/usr/obj/usr/src/sys/ATLANTIS amd64 > > /etc/sysctl.conf: > > net.inet.tcp.blackhole=2 > net.inet.udp.blackhole=1 > kern.ipc.somaxconn=16384 > net.inet.ip.fastforwarding=1 > net.inet.ip.maxfragpackets=2000 > net.inet.ip.intr_queue_maxlen=1000 > net.inet.ip.dummynet.hash_size=2048 > net.inet.tcp.recvspace=65536 > net.inet.udp.recvspace=65536 > net.inet.raw.recvspace=32768 > net.local.stream.recvspace=32768 > net.local.dgram.recvspace=32768 > net.local.stream.sendspace=32768 > net.inet.tcp.sendspace=65536 > net.inet.icmp.icmplim=500 > dev.em.0.rx_int_delay=500 > dev.em.0.tx_int_delay=500 > dev.em.0.rx_abs_int_delay=800 > dev.em.0.tx_abs_int_delay=800 > dev.em.1.rx_int_delay=500 > dev.em.1.tx_int_delay=500 > dev.em.1.rx_abs_int_delay=800 > dev.em.1.tx_abs_int_delay=800 > net.link.ether.inet.max_age=600 > > /boot/loader.conf: > > hw.em.rxd=4096 > hw.em.txd=4096 > > /etc/rc.firewall: > > 82 pipes like theese: > > pipe 387 ip from any to 193.227.x.x in recv vlan10 > pipe 388 ip from 193.227.x.x to any out xmit vlan10 > > > ######################################### > Kernel: > > > cpu HAMMER > ident ATLANTIS > > # To statically compile in device wiring instead of /boot/device.hints > #hints "GENERIC.hints" # Default places to look for > devices. > > makeoptions DEBUG=-g # Build kernel with gdb(1) debug > symbols > > options SCHED_ULE # 4BSD scheduler > options PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > #options SCTP # Stream Control Transmission > Protocol > options FFS # Berkeley Fast Filesystem options > SOFTUPDATES # Enable FFS soft updates support options > UFS_ACL # Support for access control lists options > UFS_DIRHASH # Improve performance on big directories > options PROCFS # Process filesystem (requires > PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!] > options COMPAT_IA32 # Compatible with i386 binaries > options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options > COMPAT_FREEBSD5 # Compatible with FreeBSD5 options > COMPAT_FREEBSD6 # Compatible with FreeBSD6 options KTRACE > # ktrace(1) support > options SYSVSHM # SYSV-style shared memory options > SYSVMSG # SYSV-style message queues options > SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time > extensions > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options ADAPTIVE_GIANT # Giant mutex is adaptive. options > STOP_NMI # Stop CPUS using NMI instead of IPI > options AUDIT # Security event auditing > > # Make an SMP-capable kernel by default > options SMP # Symmetric MultiProcessor Kernel > > # Bus support. > device acpi > device pci > > # ATA and ATAPI devices > device ata > > device atadisk # ATA disk drives > options ATA_STATIC_ID # Static device numbering > > # RAID controllers > device twe # 3ware ATA RAID > > # atkbdc0 controls both the keyboard and the PS/2 mouse > device atkbdc # AT keyboard controller > device atkbd # AT keyboard > > device vga # VGA video card driver > > device splash # Splash screen and screen saver support > > # syscons is the default console driver, resembling an SCO console device > sc > > ### COM > device sio > > # PCI Ethernet NICs. > device em # Intel PRO/1000 adapter Gigabit Ethernet > Card > > # PCI Ethernet NICs that use the common MII bus controller code. > # NOTE: Be sure to keep the 'device miibus' line in order to use these > NICs! device miibus # MII bus support > device bge # Broadcom BCM570xx Gigabit Ethernet > device fxp # Intel EtherExpress PRO/100B (82557, > 82558) > > # Pseudo devices. > device loop # Network loopback > device random # Entropy device > device ether # Ethernet support > device pty # Pseudo-ttys (telnet etc) > device vlan > > # The `bpf' device enables the Berkeley Packet Filter. > # Be aware of the administrative consequences of enabling this! > # Note that 'bpf' is required for DHCP. > device bpf # Berkeley packet filter > > ## Custom options > # NetGraph > options NETGRAPH > options NETGRAPH_ONE2MANY > options NETGRAPH_NETFLOW > options NETGRAPH_CISCO > options NETGRAPH_ETHER > options NETGRAPH_KSOCKET > options NETGRAPH_SOCKET > options NETGRAPH_TEE > > options IPFIREWALL > options IPFIREWALL_VERBOSE > options IPFIREWALL_FORWARD > options IPFIREWALL_VERBOSE_LIMIT=1000 > options IPFIREWALL_DEFAULT_TO_ACCEPT > options DUMMYNET > options HZ=1000 > options DEVICE_POLLING > ##################################################### > > Interfaces: > - em0 > - em1 > - bge0 > - bge1 > - vlan (61 virtual interfaces) > > ##################################################### > top -S > > last pid: 9673; load averages: 1.94, 1.75, 1.57 > up 0+19:17:21 > 19:45:01 > 77 processes: 11 running, 49 sleeping, 17 waiting > CPU states: 0.0% user, 0.0% nice, 22.6% system, 0.3% interrupt, 77.0% > idle Mem: 198M Active, 410M Inact, 455M Wired, 228K Cache, 214M Buf, 2874M > Free Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 1 171 ki31 0K 16K CPU7 7 19.0H 100.00% idle: > cpu7 > 16 root 1 171 ki31 0K 16K CPU2 2 18.9H 100.00% idle: > cpu2 > 17 root 1 171 ki31 0K 16K RUN 1 18.8H 100.00% idle: > cpu1 > 13 root 1 171 ki31 0K 16K CPU5 5 18.8H 100.00% idle: > cpu5 > 18 root 1 171 ki31 0K 16K CPU0 0 916:13 100.00% idle: > cpu0 > 12 root 1 171 ki31 0K 16K CPU6 6 18.8H 99.85% idle: > cpu6 > 35 root 1 -68 - 0K 16K CPU4 4 466:17 96.00% em1 > taskq > 34 root 1 -68 - 0K 16K CPU3 3 482:01 90.38% em0 > taskq > 15 root 1 171 ki31 0K 16K RUN 3 655:20 13.38% idle: > cpu3 > 14 root 1 171 ki31 0K 16K RUN 4 671:52 3.08% idle: > cpu4 > > > ############################################## > 19:45[p0]root@atlantis#~>netstat -w 1 -I em0 > input (em0) output > packets errs bytes packets errs bytes colls > 57381 0 36442155 68726 0 69126050 0 > 56817 0 37480502 67656 0 66053093 0 > 57847 0 39532712 68603 0 67037042 0 > 56908 0 37197022 68924 0 68660108 0 > 57107 0 37643382 68398 0 68113937 0 > 56847 0 35944754 68394 0 67896267 0 > 58754 0 39763361 68966 0 70029090 0 > 58343 0 38301796 69635 0 69948678 0 > ^C > 19:46[p0]root@atlantis#~>netstat -w 1 -I em1 > input (em1) output > packets errs bytes packets errs bytes colls > 67944 0 68877031 55376 0 36252905 0 > 65943 0 66722222 54575 0 37710643 0 > 64639 0 67149621 53298 0 35423539 0 > 63988 0 65035759 51787 0 35402337 0 > 63849 0 65968513 50727 0 31683425 0 > 64301 0 66684912 50193 0 30917339 0 > > > > ################################################################### > > > How can we solve this problem and parallelize em1:taskq kernel processes > between all 8 processors? # sysctl net.isr.direct=0 would add one more kernel thread to handle your network traffic. Regards, Dmitriy. > > > -- > ISP BGNet > 288-03-53 > 246-68-98 > > Zin'kov Oleg > System administrator > > > > > > > > > > > > > -- > ISP BGNet > 288-03-53 > 246-68-98 > > Zin'kov Oleg > System administrator > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E1KcfmE-0000KR-00._pppp-mail-ru>