From owner-freebsd-current@FreeBSD.ORG Fri Aug 21 12:26:50 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15F6D106568E for ; Fri, 21 Aug 2009 12:26:50 +0000 (UTC) (envelope-from viktor@cisti.cz) Received: from mx0.estimese.net (super.estimese.net [194.149.99.140]) by mx1.freebsd.org (Postfix) with ESMTP id 877888FC51 for ; Fri, 21 Aug 2009 12:26:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) (uid 80) by mx0.estimese.net with local; Fri, 21 Aug 2009 14:26:51 +0200 id 000BDC50.000000004A8E928B.00003CEE To: 000.fbsd@quip.cz Received: from 194.149.99.140 (auth. user viktor@mx.estimese.net) by webmail.estimese.net with HTTP; Fri, 21 Aug 2009 12:26:51 +0000 X-IlohaMail-Blah: viktor@mx.estimese.net X-IlohaMail-Method: mail() [mem] X-IlohaMail-Dummy: moo X-Mailer: IlohaMail/0.8.13 (On: webmail.estimese.net) Message-ID: In-Reply-To: <4A8CBA58.3060809@quip.cz> From: "Viktor CISTICZ" Bounce-To: "Viktor CISTICZ" Errors-To: "Viktor CISTICZ" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Date: Fri, 21 Aug 2009 14:26:51 +0200 X-Mailman-Approved-At: Fri, 21 Aug 2009 12:32:51 +0000 Cc: "freebsd-current@freebsd.org" Subject: Re: GET BUF: dmamap load failure - 12 + kernel panic [was: NETIO UDP benchmark problem] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Aug 2009 12:26:50 -0000 Hello! Thanks for changing subject, this is better. Dne 20/8/2009, napsal "Miroslav Lachman" <000.fbsd@quip.cz>: >I changed subject to better describing one, because it is not problem in >netio itself. > >Viktor CISTICZ wrote: >> Hello, >> recently i have been testing two servers via crosslink. Both have >> Freebsd installed > >> >> twin1$ uname -a >> FreeBSD twin1 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Sun Aug 16 >> 22:57:29 CEST 2009 >> viktor@twin1:/usr/obj/usr/src/sys/GEN_NO_DBG amd64 >> >> twin2$ uname -a >> FreeBSD kitt.twin2 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Wed Jul 22 15:05:19 >> CEST 2009 viktor@kitt.twin2:/usr/obj/usr/src/sys/GEN_NO_DBG amd64 >> >> GEN_NO_DBG is GENERIC kernel without debugging options >> >> Twin2 is virtual machine on the same hardware as twin1, vmware used. >> >> I have this set installed on both machines > >> twin1$ pkg_info >> NetPIPE-3.7.1 A self-scaling network benchmark >> gettext-0.17_1 GNU gettext package >> gmake-3.81_3 GNU version of 'make' utility >> iperf-2.0.4 A tool to measure maximum TCP and UDP bandwidth >> libiconv-1.13.1 A character set conversion library >> libtool-2.2.6a Generic shared library support script >> netio-1.26 Network benchmark >> netperf-2.4.5 Network performance benchmarking package >> portaudit-0.5.13 Checks installed ports against a list of security >> vulnerabi >> portmaster-2.9 Manage your ports without external databases or >> languages >> screen-4.0.3_6 A multi-screen window manager >> ttcp-1.12 Benchmarking tool for analysing TCP and UDP >> performance >> unzip-5.52_5 List, test and extract compressed files in a ZIP >> archive >> >> Both machines are connected via crosslink > >> twin1# ifconfig >> igb0 : public interface >> igb1: flags=3D8843 metric 0 mtu 15= 00 >> options=3D13b >> ether 00:30:48:c8:f3:91 >> inet 10.10.10.10 netmask 0xffffff00 broadcast 10.10.10.255 >> media: Ethernet autoselect (1000baseT ) >> status: active >> >> twin2# ifconfig >> em0: public interface >> em1: flags=3D8843 metric 0 mtu 150= 0 >> options=3D9b >> ether 00:0c:29:4b:f1:76 >> inet 10.10.10.20 netmask 0xffffff00 broadcast 10.10.10.255 >> media: Ethernet autoselect (1000baseT ) >> status: active >> >> I have set up twin2 as a server for netio > >> kitt.twin2# netio -s -p 1122 >> >> And then tested from twin1 tcp test > >> twin1# netio -p 1122 -t 10.10.10.20 >> >> It was allright, i've got some results. >> >> But when I tried UDP, it failed. >> >> The server is still the same > >> kitt.twin2# netio -s -p 1122 >> >> Client >> twin1# netio -p 1122 -u 10.10.10.20 >> >> After around 1 minute, twin1 server stopped responding on public >> interface, i've been disconnected. Via remote console i could access >> the machine, it was acting normally. I could ping both interfaces. I >> could even ping from other side of crosslink, from twin2 the privat >> interface, but no reply on public interface. >> The interface was shown UP in ifconfig, no messages in /var/log/messages. >> >> Then i executed ifconfig igb1 down && ifconfig igb1 up and it worked >> again. > >Did you ifconfig down + up igb1 or igb0? As you said "server stopped >responding on public interface", but you have igb0 marked as public >interface and igb1 as private crosslink interface in ifconfig above. Server stopped responding on igb0(public interface) from outside of the box. Via igb1 there was ping response. >Are you really saying that heavy UDP load on private interface igb1 >caused freez of public interface igb0 and then kernel panic? Yeah, test was run via igb1, after it crashed, the server was not responding on public interface igb0 and crashed. > >> I have then executed netio udp test again with 2k udp packets >> >> The server is still the same > >> kitt.twin2# netio -s -p 1122 >> >> twin1# netio -p 1122 -u -b 2k 10.10.10.20 >> >> The same problem on twin1, but now it has crashed the computer > >> via remote console i could see this > >> twin1# GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> GET BUF: dmamap load failure - 12 >> >> then it executed core dump and restarted. >> >> I am lost, thanks for any advise. > >Did you run it as root or regular user? > >Can you add output of netstat -m right before interface freez / kernel >panic? > >And last - can you reproduce it with kernel with debugging options enabled? > >Miroslav Lachman > I have done another testing, this time i've booted twin1 server with GENERIC kernel from 8.0-BETA2-amd64 and it has happened again. this is what i've done on twin1 (client), i have done it as a normal user > twin1$ netio -u -p 1122 10.10.10.20 NETIO - Network Throughput Benchmark, Version 1.26 (C) 1997-2005 Kai Uwe Rommel UDP connection established. Packet size 1k bytes: 6494 KByte/s (32%) Tx, 114228 KByte/s (73%) Rx. Packet size 2k bytes: 115122 KByte/s (0%) Tx, on kitt.twin2 (server) kitt.twin2# netio -s -p 1122 this appeared on kitt.twin2 > sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available sendto(): No buffer space available Sending to client, packet size 1k ... Receiving from client, packet size 2k ... Then twin1 stopped working and kernel panic. On another terminal on kitt.twin2 I ran (while 1, while netstat -m, while sleep 1, while end) > 67272K/12510K/79782K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines 35585/8200/43785 mbufs in use (current/cache/total) 3892/4926/8818/25600 mbuf clusters in use (current/cache/total/max) 3892/4812 mbuf+clusters out of packet secondary zone in use (current/cache) 12648/152/12800/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 67272K/12510K/79782K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines I have captured via remote console what happened on twin1 (rewritten from the picture)> twin1# the same netstat -m statistic procedure as at kitt.twin2> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 60243K/5841K/66085K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines 34079/2791/36870 mbufs in use 514/2320/2834/25600 mbuf clusters in use 0/512 mbuf +clusters out of packet secondary 12674/126/12800/12800 4k (page size) jumbo clusters in use 0/0/0/6400 9k jumbo clusters in use 0/0/0/3200 16k jumbo clusters in use 60243K/5841K/66085K bytes alocated to network .. and during the crash > twin1# Memory modified after free 0xffffff0083eb800(256) val=3Da8 @ 0xffffff00083eb818 Memory modified after free 0xffffff0083ec600(256) val=3Da8 @ 0xffffff00083ec618 there was many lines of this similar text. VC