From owner-freebsd-net@FreeBSD.ORG Tue May 15 20:50:11 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5BCA61065680 for ; Tue, 15 May 2012 20:50:11 +0000 (UTC) (envelope-from mitya@yandex-team.ru) Received: from elephant.yandex.ru (elephant.yandex.ru [77.88.34.7]) by mx1.freebsd.org (Postfix) with ESMTP id D763D8FC1E for ; Tue, 15 May 2012 20:50:10 +0000 (UTC) Received: from Dmitrys-MacBook-Pro.local (v3-151-66.yandex.net [84.201.151.66]) by elephant.yandex.ru (Postfix) with ESMTP id 3VsW1y6th3z45JpY for ; Wed, 16 May 2012 00:43:02 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1337114582; bh=qPlesAVQVeuR5XW3ou9b/LZT1140LrIhSZWh3WvxzPU=; h=Date:From:To:Subject; b=RfgZWqsU7TpfCd8XKRQ13IuTjUslKp/M4xqjJ54FlpvNj3m2rVIYDDTOt7GXEUW2F kGAhKZrmllWBYkHTfGsWRbamGd/vQQGue69iRIrCg5ElcDPdOmx3v5WA0VUwohCwRP JOpsqbBp5PrlPSXCES8qen/g3CWfwhw3RYEVJ480= Message-ID: <4FB2BFD6.8040607@yandex-team.ru> Date: Wed, 16 May 2012 00:43:02 +0400 From: Dmitry Sivachenko User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: network stops working X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 May 2012 20:50:11 -0000 Hello! I am using FreeBSD-9-stable. I use Intel network cards (em, igb) with mtu=9000 (it's important, the problem disappears if I switch to mtu=1500). I have a number of servers running a few web-services written in our company. After a few days of working network suddenly stops functioning. There is nothing in log/messages. It just stops working. If I execute ifconfig down/up, I get the following error message: # ifconfig net0 down # ifconfig net0 up em1: Could not setup receive structures # and network still does not work. If I stop mentioned programs serving web requests (even not all of them, just random one), ifconfig net0 up command succeeds and network resumes its operations... for more few days and then the same problems comes back. Consider netstat -m output: > netstat -m 1025/4945/5970 mbufs in use (current/cache/total) 0/3446/3446/262144 mbuf clusters in use (current/cache/total/max) 0/2090 mbuf+clusters out of packet secondary zone in use (current/cache) 0/1230/1230/65536 4k (page size) jumbo clusters in use (current/cache/total/max) 1023/4907/5930/65536 9k jumbo clusters in use (current/cache/total/max) ^^^^^^^^^^^^^^^^^^^^^^^ Note that max is much higher that total. 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 9463K/57211K/66674K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/2542746/0 requests for jumbo clusters denied (4k/9k/16k) ^^^^^^^^^^^^ There are 9k jumbo clusters allocations denied. 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 641 requests for I/O initiated by sendfile 0 calls to protocol drain routines I am attaching vmstat -z output below for reference. What can be the cause of 9k jumbo closters allocation denies? What additional information cat I provide to help trach this down? Thanks in advance! > vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 208, 0, 92, 10, 92, 0, 0 UMA Zones: 3456, 0, 92, 0, 92, 0, 0 UMA Slabs: 568, 0, 6270, 702, 2222844, 0, 0 UMA RCntSlabs: 568, 0, 8883, 581, 7364017, 0, 0 UMA Hash: 256, 0, 1, 14, 4, 0, 0 16 Bucket: 152, 0, 39, 136, 185, 0, 0 32 Bucket: 280, 0, 130, 206, 401, 0, 0 64 Bucket: 536, 0, 138, 149, 470, 84, 0 128 Bucket: 1048, 0, 1812, 639, 1736739,16793, 0 VM OBJECT: 232, 0, 52210, 7454,20006697, 0, 0 MAP: 232, 0, 7, 25, 7, 0, 0 KMAP ENTRY: 120, 1549690, 3102, 2602, 6160517, 0, 0 MAP ENTRY: 120, 0, 7131, 3409,54473178, 0, 0 fakepg: 120, 0, 0, 589, 53928, 0, 0 mt_zone: 4112, 0, 276, 3, 276, 0, 0 16: 16, 0, 4047, 4353,3946273892, 0, 0 32: 32, 0, 3955, 2408,13600755, 0, 0 64: 64, 0, 11979, 7509,116304518, 0, 0 128: 128, 0, 20979, 36673,90194687, 0, 0 256: 256, 0, 5569, 16886,137956814, 0, 0 512: 512, 0, 4368, 5194, 8638401, 0, 0 1024: 1024, 0, 84, 1196, 4033660, 0, 0 2048: 2048, 0, 199, 1751, 1203566, 0, 0 4096: 4096, 0, 318, 2704, 1142726, 0, 0 Files: 80, 0, 5613, 5097,2006566966, 0, 0 TURNSTILE: 136, 0, 5605, 855, 6379, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 PROC: 1160, 0, 63, 3657, 709134, 0, 0 THREAD: 1112, 0, 3969, 1635, 524885, 0, 0 SLEEPQUEUE: 80, 0, 5605, 1094, 6379, 0, 0 VMSPACE: 392, 0, 43, 2387, 709117, 0, 0 cpuset: 72, 0, 2, 98, 2, 0, 0 mbuf_packet: 256, 0, 0, 1765,4356065566, 0, 0 mbuf: 256, 0, 1024, 3181,26760389876, 0, 0 mbuf_cluster: 2048, 262144, 1765, 1681,1318169467, 0, 0 mbuf_jumbo_page: 4096, 65536, 0, 1230,61648445, 0, 0 mbuf_jumbo_9k: 9216, 65536, 1023, 4907,8926430021,2542746, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 2352, 5422000, 0, 0 NetGraph items: 72, 4118, 0, 58, 5, 0, 0 NetGraph data items: 72, 522, 0, 58, 2, 0, 0 g_bio: 232, 0, 0, 4544,300858503, 0, 0 ttyinq: 160, 0, 180, 252, 615, 0, 0 ttyoutq: 256, 0, 95, 175, 327, 0, 0 ata_request: 328, 0, 0, 0, 0, 0, 0 ata_composite: 336, 0, 0, 0, 0, 0, 0 VNODE: 480, 0, 78944, 3840, 952605, 0, 0 VNODEPOLL: 112, 0, 0, 0, 1, 0, 0 NAMEI: 1024, 0, 0, 1632,139785175, 0, 0 S VFS Cache: 108, 0, 87356, 1579, 556273, 0, 0 L VFS Cache: 328, 0, 2924, 1768, 15274, 0, 0 DIRHASH: 1024, 0, 9, 1259, 20893, 0, 0 NCLNODE: 560, 0, 0, 0, 0, 0, 0 Mountpoints: 768, 0, 5, 10, 5, 0, 0 AIO: 208, 0, 0, 0, 0, 0, 0 AIOP: 32, 0, 0, 0, 0, 0, 0 AIOCB: 480, 0, 0, 0, 0, 0, 0 AIOL: 128, 0, 0, 0, 0, 0, 0 AIOLIO: 272, 0, 0, 0, 0, 0, 0 pipe: 728, 0, 7, 2928, 515298, 0, 0 ksiginfo: 112, 0, 3680, 3250, 192170, 0, 0 itimer: 344, 0, 1, 21, 6, 0, 0 KNOTE: 128, 0, 31, 4406,11850035546, 0, 0 socket: 680, 262140, 66, 4986,1949420563, 0, 0 ipq: 56, 8253, 0, 0, 0, 0, 0 udp_inpcb: 392, 262140, 20, 2170, 1133408, 0, 0 udpcb: 16, 262248, 20, 3676, 1133408, 0, 0 tcp_inpcb: 392, 262140, 3373, 24417,1947917187, 0, 0 tcpcb: 976, 262140, 26, 5010,1947917187, 0, 0 tcptw: 72, 41000, 3347, 24153,1352095629, 0, 0 syncache: 152, 15375, 0, 2375,2029926070, 0, 0 hostcache: 136, 153608, 5, 667, 4953, 0, 0 tcpreass: 40, 16464, 0, 2436,27465901, 0, 0 sackhole: 32, 0, 0, 909, 41329, 0, 0 sctp_ep: 1368, 25600, 0, 0, 0, 0, 0 sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 216, 13, 0, 0 sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0, 0 sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 ripcb: 392, 262140, 0, 70, 138, 0, 0 unpcb: 240, 262144, 14, 1986, 369808, 0, 0 rtentry: 200, 0, 19, 38, 37, 0, 0 IPFW dynamic rule: 120, 0, 0, 0, 0, 0, 0 divcb: 392, 262140, 0, 0, 0, 0, 0 g_stripe_zone: 131072, 100, 0, 0, 0, 0, 0 selfd: 56, 0, 6286, 2534,62657688, 0, 0 SWAPMETA: 288, 116519, 1109, 2817,14987719, 0, 0 FFS inode: 168, 0, 78896, 5232, 952545, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 78896, 5014, 952545, 0, 0 > cat /boot/loader.conf userconfig_script_load="YES" vm.exec_map_entries="48" hw.igb.rxd="256" hw.igb.txd="256" hw.usb.no_pf="1"