From owner-freebsd-stable@FreeBSD.ORG Tue Sep 7 21:34:54 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CCFC10656CE for ; Tue, 7 Sep 2010 21:34:54 +0000 (UTC) (envelope-from mahlon@martini.nu) Received: from martini.nu (martini.nu [198.145.180.83]) by mx1.freebsd.org (Postfix) with SMTP id 420A98FC12 for ; Tue, 7 Sep 2010 21:34:54 +0000 (UTC) Received: (qmail 91052 invoked by uid 1000); 7 Sep 2010 21:08:13 -0000 Date: Tue, 7 Sep 2010 14:08:13 -0700 From: "Mahlon E. Smith" To: freebsd-stable@freebsd.org Message-ID: <20100907210813.GI49065@martini.nu> Mail-Followup-To: "Mahlon E. Smith" , freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="yQbNiKLmgenwUfTN" Content-Disposition: inline X-GPG-Fingerprint: 19B8 DDB3 0156 3A03 FA80 8278 C0BE 6BFB 3606 B267 X-Sysinfo: FreeBSD 7.0-RELEASE-p1 i386 User-Agent: Mutt/1.5.17 (2007-11-01) Subject: Network memory allocation failures X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 21:34:54 -0000 --yQbNiKLmgenwUfTN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, all. I picked up a couple of Dell R810 monsters a couple of months ago. 96G of RAM, 24 core. With the aid of this list, got 8.1-RELEASE on there, and they are trucking along merrily as VirtualBox hosts. I'm seeing memory allocation errors when sending data over the network. It is random at best, however I can reproduce it pretty reliably. Sending 100M to a remote machine. Note the 2nd scp attempt worked. Most small files can make it through unmolested. obb# dd if=3D/dev/random of=3D100M-test bs=3D1M count=3D100 100+0 records in 100+0 records out 104857600 bytes transferred in 2.881689 secs (36387551 bytes/sec) obb# rsync -av 100M-test skin:/tmp/ sending incremental file list 100M-test Write failed: Cannot allocate memory rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: B= roken pipe (32) rsync: connection unexpectedly closed (28 bytes received so far) [sende= r] rsync error: unexplained error (code 255) at io.c(601) [sender=3D3.0.7] obb# scp 100M-test skin:/tmp/ 100M-test 52% 52MB 52.1MB/s 00:00 ETAWrite failed: Cannot a= llocate memory lost connection obb# scp 100M-test skin:/tmp/ 100M-test 100% 100MB 50.0MB/s 00:02 =20 obb# scp 100M-test skin:/tmp/ 100M-test 0% 0 0.0KB/s --:-- ETAWrite failed: Cannot a= llocate memory lost connection Fetching a file, however, works. obb# scp skin:/usr/local/tmp/100M-test . 100M-test 100% 100MB 20.0MB/s 00:05 =20 obb# scp skin:/usr/local/tmp/100M-test . 100M-test 100% 100MB 20.0MB/s 00:05 =20 obb# scp skin:/usr/local/tmp/100M-test . 100M-test 100% 100MB 20.0MB/s 00:05 =20 obb# scp skin:/usr/local/tmp/100M-test . 100M-test 100% 100MB 20.0MB/s 00:05 =20 ... I've ruled out bad hardware (mainly due to the behavior being *identical* on the sister machine, in a completely different data center.) It's a broadcom (bce) NIC. mbufs look fine to me. obb# netstat -m 511/6659/7170 mbufs in use (current/cache/total) 510/3678/4188/25600 mbuf clusters in use (current/cache/total/max) 510/3202 mbuf+clusters out of packet secondary zone in use (current/cache) 0/984/984/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 1147K/12956K/14104K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines Plenty of available mem (not surprising): obb# vmstat -hc 5 -w 5=20 procs memory page disks faults = cpu r b w avm fre flt re pi po fr sr mf0 mf1 in sy c= s us sy id 0 0 0 722M 92G 115 0 1 0 1067 0 0 0 429 32637 65= 20 0 1 99 0 0 0 722M 92G 1 0 0 0 0 0 0 0 9 31830 32= 79 0 0 100 0 0 0 722M 92G 0 0 0 0 3 0 0 0 8 33171 32= 23 0 0 100 0 0 0 761M 92G 2593 0 0 0 1712 0 5 4 121 35384 39= 07 0 0 99 1 0 0 761M 92G 0 0 0 0 0 0 0 0 10 30237 31= 56 0 0 100 Last bit of info, and here's where it gets really weird. Remember how I said this was a VirtualBox host? Guest machines running on it (mostly centos) don't exhibit the problem, which is also why it took me so long to notice it in the host. They can merrily copy data around at will, even though they are going out through the same host interface. I'm not sure what to check for or toggle at this point. There are all sorts of tunables I've been mucking around with to no avail, and so I've reverted them to defaults. Mostly concentrating on these: hw.intr_storm_threshold net.inet.tcp.rfc1323 kern.ipc.nmbclusters kern.ipc.nmbjumbop net.inet.tcp.sendspace net.inet.tcp.recvspace kern.ipc.somaxconn kern.ipc.maxsockbuf It was suggested to me to try limiting the RAM in loader.conf to under 32G and see what happens. When doing this, it does appear to be "okay". Not sure if that's coincidence, or directly related -- something with the large amount of RAM that is confusing a data structure somewhere? Or potentially a problem with the bce driver, specifically? I've kind of reached a limit here in what to dig for / try next. What else can I do to try and determine the root problem that would be helpful? Anyone ever have to deal with or seen something like this recently? (Or hell, not recently?) Ideas appreciated! -- Mahlon E. Smith =20 http://www.martini.nu/contact.html --yQbNiKLmgenwUfTN Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- iD8DBQFMhqm91bsjBDapbeMRAskpAJ9m0K/uBfhkShaHHjXkTGDbZZNJOQCePK5M E96Iyo2BCuaAhnkLNbfirkM= =1i1D -----END PGP SIGNATURE----- --yQbNiKLmgenwUfTN--