From owner-freebsd-net@FreeBSD.ORG Mon May 28 06:20:41 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7E96106566B for ; Mon, 28 May 2012 06:20:41 +0000 (UTC) (envelope-from mitya@yandex-team.ru) Received: from dinosaur.yandex.ru (dinosaur.yandex.ru [77.88.34.8]) by mx1.freebsd.org (Postfix) with ESMTP id 38B4F8FC0A for ; Mon, 28 May 2012 06:20:40 +0000 (UTC) Received: from Dmitrys-MacBook-Pro.local (v3-150-205.yandex.net [84.201.150.205]) by dinosaur.yandex.ru (Postfix) with ESMTP id 3W17Gn6gWmz2prv for ; Mon, 28 May 2012 10:20:33 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1338186033; bh=Xm/2KVunbv37MA4TrQQDjKmYp8CUwJ7s6L9fANlDUl4=; h=Date:From:To:Subject:References:In-Reply-To; b=ICtQTDFV4yRAlolynrbKomJ86VfpYw4DisV7eN8la/A7aLtDZjCZD4l7QQlaGn9s0 ZYtnCZZY6E187KhV40z15tdEe/xChQKFqST2K1M9ViA+bU0DCegPbKs8SDsEcGdYfy xbVF4WNWxSuXR2phEP+V53wcszwY5Gr93eju0XMQ= Message-ID: <4FC31931.6050502@yandex-team.ru> Date: Mon, 28 May 2012 10:20:33 +0400 From: Dmitry Sivachenko User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <4FB2BFD6.8040607@yandex-team.ru> In-Reply-To: <4FB2BFD6.8040607@yandex-team.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: network stops working X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 May 2012 06:20:41 -0000 Just for reference: I am almost sure that recent MFC of Intel drivers to stable/9 fixed that issue. I observed no network-related issues on my few test servers since that upgrade. On 5/16/12 12:43 AM, Dmitry Sivachenko wrote: > Hello! > > I am using FreeBSD-9-stable. > I use Intel network cards (em, igb) with mtu=9000 (it's important, the > problem disappears if I switch to mtu=1500). > > I have a number of servers running a few web-services written in our > company. > > After a few days of working network suddenly stops functioning. > There is nothing in log/messages. It just stops working. > > If I execute ifconfig down/up, I get the following error message: > > # ifconfig net0 down > # ifconfig net0 up > em1: Could not setup receive structures > # > > and network still does not work. > > If I stop mentioned programs serving web requests (even not all of them, > just random one), ifconfig net0 up command succeeds and network resumes > its operations... for more few days and then the same problems comes back. > > Consider netstat -m output: > > netstat -m > 1025/4945/5970 mbufs in use (current/cache/total) > 0/3446/3446/262144 mbuf clusters in use (current/cache/total/max) > 0/2090 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/1230/1230/65536 4k (page size) jumbo clusters in use > (current/cache/total/max) > 1023/4907/5930/65536 9k jumbo clusters in use (current/cache/total/max) > ^^^^^^^^^^^^^^^^^^^^^^^ > Note that max is much higher that total. > > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 9463K/57211K/66674K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/2542746/0 requests for jumbo clusters denied (4k/9k/16k) > ^^^^^^^^^^^^ > There are 9k jumbo clusters allocations denied. > > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 641 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > I am attaching vmstat -z output below for reference. > > What can be the cause of 9k jumbo closters allocation denies? > What additional information cat I provide to help trach this down? > > Thanks in advance! > > > vmstat -z > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP > > UMA Kegs: 208, 0, 92, 10, 92, 0, 0 > UMA Zones: 3456, 0, 92, 0, 92, 0, 0 > UMA Slabs: 568, 0, 6270, 702, 2222844, 0, 0 > UMA RCntSlabs: 568, 0, 8883, 581, 7364017, 0, 0 > UMA Hash: 256, 0, 1, 14, 4, 0, 0 > 16 Bucket: 152, 0, 39, 136, 185, 0, 0 > 32 Bucket: 280, 0, 130, 206, 401, 0, 0 > 64 Bucket: 536, 0, 138, 149, 470, 84, 0 > 128 Bucket: 1048, 0, 1812, 639, 1736739,16793, 0 > VM OBJECT: 232, 0, 52210, 7454,20006697, 0, 0 > MAP: 232, 0, 7, 25, 7, 0, 0 > KMAP ENTRY: 120, 1549690, 3102, 2602, 6160517, 0, 0 > MAP ENTRY: 120, 0, 7131, 3409,54473178, 0, 0 > fakepg: 120, 0, 0, 589, 53928, 0, 0 > mt_zone: 4112, 0, 276, 3, 276, 0, 0 > 16: 16, 0, 4047, 4353,3946273892, 0, 0 > 32: 32, 0, 3955, 2408,13600755, 0, 0 > 64: 64, 0, 11979, 7509,116304518, 0, 0 > 128: 128, 0, 20979, 36673,90194687, 0, 0 > 256: 256, 0, 5569, 16886,137956814, 0, 0 > 512: 512, 0, 4368, 5194, 8638401, 0, 0 > 1024: 1024, 0, 84, 1196, 4033660, 0, 0 > 2048: 2048, 0, 199, 1751, 1203566, 0, 0 > 4096: 4096, 0, 318, 2704, 1142726, 0, 0 > Files: 80, 0, 5613, 5097,2006566966, 0, 0 > TURNSTILE: 136, 0, 5605, 855, 6379, 0, 0 > umtx pi: 96, 0, 0, 0, 0, 0, 0 > PROC: 1160, 0, 63, 3657, 709134, 0, 0 > THREAD: 1112, 0, 3969, 1635, 524885, 0, 0 > SLEEPQUEUE: 80, 0, 5605, 1094, 6379, 0, 0 > VMSPACE: 392, 0, 43, 2387, 709117, 0, 0 > cpuset: 72, 0, 2, 98, 2, 0, 0 > mbuf_packet: 256, 0, 0, 1765,4356065566, 0, 0 > mbuf: 256, 0, 1024, 3181,26760389876, 0, 0 > mbuf_cluster: 2048, 262144, 1765, 1681,1318169467, 0, 0 > mbuf_jumbo_page: 4096, 65536, 0, 1230,61648445, 0, 0 > mbuf_jumbo_9k: 9216, 65536, 1023, 4907,8926430021,2542746, 0 > mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 > mbuf_ext_refcnt: 4, 0, 0, 2352, 5422000, 0, 0 > NetGraph items: 72, 4118, 0, 58, 5, 0, 0 > NetGraph data items: 72, 522, 0, 58, 2, 0, 0 > g_bio: 232, 0, 0, 4544,300858503, 0, 0 > ttyinq: 160, 0, 180, 252, 615, 0, 0 > ttyoutq: 256, 0, 95, 175, 327, 0, 0 > ata_request: 328, 0, 0, 0, 0, 0, 0 > ata_composite: 336, 0, 0, 0, 0, 0, 0 > VNODE: 480, 0, 78944, 3840, 952605, 0, 0 > VNODEPOLL: 112, 0, 0, 0, 1, 0, 0 > NAMEI: 1024, 0, 0, 1632,139785175, 0, 0 > S VFS Cache: 108, 0, 87356, 1579, 556273, 0, 0 > L VFS Cache: 328, 0, 2924, 1768, 15274, 0, 0 > DIRHASH: 1024, 0, 9, 1259, 20893, 0, 0 > NCLNODE: 560, 0, 0, 0, 0, 0, 0 > Mountpoints: 768, 0, 5, 10, 5, 0, 0 > AIO: 208, 0, 0, 0, 0, 0, 0 > AIOP: 32, 0, 0, 0, 0, 0, 0 > AIOCB: 480, 0, 0, 0, 0, 0, 0 > AIOL: 128, 0, 0, 0, 0, 0, 0 > AIOLIO: 272, 0, 0, 0, 0, 0, 0 > pipe: 728, 0, 7, 2928, 515298, 0, 0 > ksiginfo: 112, 0, 3680, 3250, 192170, 0, 0 > itimer: 344, 0, 1, 21, 6, 0, 0 > KNOTE: 128, 0, 31, 4406,11850035546, 0, 0 > socket: 680, 262140, 66, 4986,1949420563, 0, 0 > ipq: 56, 8253, 0, 0, 0, 0, 0 > udp_inpcb: 392, 262140, 20, 2170, 1133408, 0, 0 > udpcb: 16, 262248, 20, 3676, 1133408, 0, 0 > tcp_inpcb: 392, 262140, 3373, 24417,1947917187, 0, 0 > tcpcb: 976, 262140, 26, 5010,1947917187, 0, 0 > tcptw: 72, 41000, 3347, 24153,1352095629, 0, 0 > syncache: 152, 15375, 0, 2375,2029926070, 0, 0 > hostcache: 136, 153608, 5, 667, 4953, 0, 0 > tcpreass: 40, 16464, 0, 2436,27465901, 0, 0 > sackhole: 32, 0, 0, 909, 41329, 0, 0 > sctp_ep: 1368, 25600, 0, 0, 0, 0, 0 > sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 > sctp_laddr: 48, 80064, 0, 216, 13, 0, 0 > sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 > sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 > sctp_readq: 104, 400032, 0, 0, 0, 0, 0 > sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 > sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 > sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 > ripcb: 392, 262140, 0, 70, 138, 0, 0 > unpcb: 240, 262144, 14, 1986, 369808, 0, 0 > rtentry: 200, 0, 19, 38, 37, 0, 0 > IPFW dynamic rule: 120, 0, 0, 0, 0, 0, 0 > divcb: 392, 262140, 0, 0, 0, 0, 0 > g_stripe_zone: 131072, 100, 0, 0, 0, 0, 0 > selfd: 56, 0, 6286, 2534,62657688, 0, 0 > SWAPMETA: 288, 116519, 1109, 2817,14987719, 0, 0 > FFS inode: 168, 0, 78896, 5232, 952545, 0, 0 > FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 > FFS2 dinode: 256, 0, 78896, 5014, 952545, 0, 0 > > > > cat /boot/loader.conf > userconfig_script_load="YES" > vm.exec_map_entries="48" > hw.igb.rxd="256" > hw.igb.txd="256" > hw.usb.no_pf="1" >