Date: Mon, 28 May 2012 10:20:33 +0400 From: Dmitry Sivachenko <mitya@yandex-team.ru> To: freebsd-net@freebsd.org Subject: Re: network stops working Message-ID: <4FC31931.6050502@yandex-team.ru> In-Reply-To: <4FB2BFD6.8040607@yandex-team.ru> References: <4FB2BFD6.8040607@yandex-team.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Just for reference: I am almost sure that recent MFC of Intel drivers to stable/9 fixed that issue. I observed no network-related issues on my few test servers since that upgrade. On 5/16/12 12:43 AM, Dmitry Sivachenko wrote: > Hello! > > I am using FreeBSD-9-stable. > I use Intel network cards (em, igb) with mtu=9000 (it's important, the > problem disappears if I switch to mtu=1500). > > I have a number of servers running a few web-services written in our > company. > > After a few days of working network suddenly stops functioning. > There is nothing in log/messages. It just stops working. > > If I execute ifconfig down/up, I get the following error message: > > # ifconfig net0 down > # ifconfig net0 up > em1: Could not setup receive structures > # > > and network still does not work. > > If I stop mentioned programs serving web requests (even not all of them, > just random one), ifconfig net0 up command succeeds and network resumes > its operations... for more few days and then the same problems comes back. > > Consider netstat -m output: > > netstat -m > 1025/4945/5970 mbufs in use (current/cache/total) > 0/3446/3446/262144 mbuf clusters in use (current/cache/total/max) > 0/2090 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/1230/1230/65536 4k (page size) jumbo clusters in use > (current/cache/total/max) > 1023/4907/5930/65536 9k jumbo clusters in use (current/cache/total/max) > ^^^^^^^^^^^^^^^^^^^^^^^ > Note that max is much higher that total. > > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 9463K/57211K/66674K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/2542746/0 requests for jumbo clusters denied (4k/9k/16k) > ^^^^^^^^^^^^ > There are 9k jumbo clusters allocations denied. > > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 641 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > I am attaching vmstat -z output below for reference. > > What can be the cause of 9k jumbo closters allocation denies? > What additional information cat I provide to help trach this down? > > Thanks in advance! > > > vmstat -z > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP > > UMA Kegs: 208, 0, 92, 10, 92, 0, 0 > UMA Zones: 3456, 0, 92, 0, 92, 0, 0 > UMA Slabs: 568, 0, 6270, 702, 2222844, 0, 0 > UMA RCntSlabs: 568, 0, 8883, 581, 7364017, 0, 0 > UMA Hash: 256, 0, 1, 14, 4, 0, 0 > 16 Bucket: 152, 0, 39, 136, 185, 0, 0 > 32 Bucket: 280, 0, 130, 206, 401, 0, 0 > 64 Bucket: 536, 0, 138, 149, 470, 84, 0 > 128 Bucket: 1048, 0, 1812, 639, 1736739,16793, 0 > VM OBJECT: 232, 0, 52210, 7454,20006697, 0, 0 > MAP: 232, 0, 7, 25, 7, 0, 0 > KMAP ENTRY: 120, 1549690, 3102, 2602, 6160517, 0, 0 > MAP ENTRY: 120, 0, 7131, 3409,54473178, 0, 0 > fakepg: 120, 0, 0, 589, 53928, 0, 0 > mt_zone: 4112, 0, 276, 3, 276, 0, 0 > 16: 16, 0, 4047, 4353,3946273892, 0, 0 > 32: 32, 0, 3955, 2408,13600755, 0, 0 > 64: 64, 0, 11979, 7509,116304518, 0, 0 > 128: 128, 0, 20979, 36673,90194687, 0, 0 > 256: 256, 0, 5569, 16886,137956814, 0, 0 > 512: 512, 0, 4368, 5194, 8638401, 0, 0 > 1024: 1024, 0, 84, 1196, 4033660, 0, 0 > 2048: 2048, 0, 199, 1751, 1203566, 0, 0 > 4096: 4096, 0, 318, 2704, 1142726, 0, 0 > Files: 80, 0, 5613, 5097,2006566966, 0, 0 > TURNSTILE: 136, 0, 5605, 855, 6379, 0, 0 > umtx pi: 96, 0, 0, 0, 0, 0, 0 > PROC: 1160, 0, 63, 3657, 709134, 0, 0 > THREAD: 1112, 0, 3969, 1635, 524885, 0, 0 > SLEEPQUEUE: 80, 0, 5605, 1094, 6379, 0, 0 > VMSPACE: 392, 0, 43, 2387, 709117, 0, 0 > cpuset: 72, 0, 2, 98, 2, 0, 0 > mbuf_packet: 256, 0, 0, 1765,4356065566, 0, 0 > mbuf: 256, 0, 1024, 3181,26760389876, 0, 0 > mbuf_cluster: 2048, 262144, 1765, 1681,1318169467, 0, 0 > mbuf_jumbo_page: 4096, 65536, 0, 1230,61648445, 0, 0 > mbuf_jumbo_9k: 9216, 65536, 1023, 4907,8926430021,2542746, 0 > mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 > mbuf_ext_refcnt: 4, 0, 0, 2352, 5422000, 0, 0 > NetGraph items: 72, 4118, 0, 58, 5, 0, 0 > NetGraph data items: 72, 522, 0, 58, 2, 0, 0 > g_bio: 232, 0, 0, 4544,300858503, 0, 0 > ttyinq: 160, 0, 180, 252, 615, 0, 0 > ttyoutq: 256, 0, 95, 175, 327, 0, 0 > ata_request: 328, 0, 0, 0, 0, 0, 0 > ata_composite: 336, 0, 0, 0, 0, 0, 0 > VNODE: 480, 0, 78944, 3840, 952605, 0, 0 > VNODEPOLL: 112, 0, 0, 0, 1, 0, 0 > NAMEI: 1024, 0, 0, 1632,139785175, 0, 0 > S VFS Cache: 108, 0, 87356, 1579, 556273, 0, 0 > L VFS Cache: 328, 0, 2924, 1768, 15274, 0, 0 > DIRHASH: 1024, 0, 9, 1259, 20893, 0, 0 > NCLNODE: 560, 0, 0, 0, 0, 0, 0 > Mountpoints: 768, 0, 5, 10, 5, 0, 0 > AIO: 208, 0, 0, 0, 0, 0, 0 > AIOP: 32, 0, 0, 0, 0, 0, 0 > AIOCB: 480, 0, 0, 0, 0, 0, 0 > AIOL: 128, 0, 0, 0, 0, 0, 0 > AIOLIO: 272, 0, 0, 0, 0, 0, 0 > pipe: 728, 0, 7, 2928, 515298, 0, 0 > ksiginfo: 112, 0, 3680, 3250, 192170, 0, 0 > itimer: 344, 0, 1, 21, 6, 0, 0 > KNOTE: 128, 0, 31, 4406,11850035546, 0, 0 > socket: 680, 262140, 66, 4986,1949420563, 0, 0 > ipq: 56, 8253, 0, 0, 0, 0, 0 > udp_inpcb: 392, 262140, 20, 2170, 1133408, 0, 0 > udpcb: 16, 262248, 20, 3676, 1133408, 0, 0 > tcp_inpcb: 392, 262140, 3373, 24417,1947917187, 0, 0 > tcpcb: 976, 262140, 26, 5010,1947917187, 0, 0 > tcptw: 72, 41000, 3347, 24153,1352095629, 0, 0 > syncache: 152, 15375, 0, 2375,2029926070, 0, 0 > hostcache: 136, 153608, 5, 667, 4953, 0, 0 > tcpreass: 40, 16464, 0, 2436,27465901, 0, 0 > sackhole: 32, 0, 0, 909, 41329, 0, 0 > sctp_ep: 1368, 25600, 0, 0, 0, 0, 0 > sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 > sctp_laddr: 48, 80064, 0, 216, 13, 0, 0 > sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 > sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 > sctp_readq: 104, 400032, 0, 0, 0, 0, 0 > sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 > sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 > sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 > ripcb: 392, 262140, 0, 70, 138, 0, 0 > unpcb: 240, 262144, 14, 1986, 369808, 0, 0 > rtentry: 200, 0, 19, 38, 37, 0, 0 > IPFW dynamic rule: 120, 0, 0, 0, 0, 0, 0 > divcb: 392, 262140, 0, 0, 0, 0, 0 > g_stripe_zone: 131072, 100, 0, 0, 0, 0, 0 > selfd: 56, 0, 6286, 2534,62657688, 0, 0 > SWAPMETA: 288, 116519, 1109, 2817,14987719, 0, 0 > FFS inode: 168, 0, 78896, 5232, 952545, 0, 0 > FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 > FFS2 dinode: 256, 0, 78896, 5014, 952545, 0, 0 > > > > cat /boot/loader.conf > userconfig_script_load="YES" > vm.exec_map_entries="48" > hw.igb.rxd="256" > hw.igb.txd="256" > hw.usb.no_pf="1" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4FC31931.6050502>