From owner-freebsd-net@FreeBSD.ORG Wed May 14 02:56:06 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A685C266 for ; Wed, 14 May 2014 02:56:06 +0000 (UTC) Received: from mail-pb0-x234.google.com (mail-pb0-x234.google.com [IPv6:2607:f8b0:400e:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7996B2BE6 for ; Wed, 14 May 2014 02:56:06 +0000 (UTC) Received: by mail-pb0-f52.google.com with SMTP id rr13so1056419pbb.25 for ; Tue, 13 May 2014 19:56:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=5tn6xtGCVzTXbJ5CQUGtaA26yc2QDmsvGIoZJVczjBU=; b=NK3Y5n84C+VbYSMq/qy6Q5C9TxhZHm4vpz7bQGRydjy7Be+4LTqG4er0rkypu1kEzf MT0k3dvlBsSMeOOmObyIHNXScXcLXvkS4vTDqf04Me9CDiEfd+8DWsMzzhoPPuyVA0uP aPhuNHKRuzrJJogPhn6ZpjVOYv7oPIhCFELY6NOJov22Ycnca6RYs8H3GNNPO19+81ks SuHy+c8Z2BjmuPWRRKVHBfffkdf3791o+iOflBXhUcP2tWasl/SM1Go1yFkvmB2TqFtS vdjWV+sDuYI5sscCfFTDMEtvJrNyZrUw47V7h/67AOOt2sQOM2bPODwCItnvG3iIN9uc KQbA== X-Received: by 10.66.164.135 with SMTP id yq7mr1001436pab.126.1400036166013; Tue, 13 May 2014 19:56:06 -0700 (PDT) Received: from kmatoMacBook-Pro.local ([27.24.140.240]) by mx.google.com with ESMTPSA id g6sm1867910pat.2.2014.05.13.19.56.02 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 13 May 2014 19:56:05 -0700 (PDT) Message-ID: <5372DB3E.7000506@gmail.com> Date: Wed, 14 May 2014 10:55:58 +0800 From: k simon User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Christopher Forgeron Subject: Re: syslogd:sendto: no buffer available on 10-stable References: <53310C57.5070102@gmail.com> <533240B4.50809@gmail.com> In-Reply-To: <533240B4.50809@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 May 2014 02:56:06 -0000 The issue is still remained. Does it related to flowtable ? # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 109, 1, 109, 0, 0 UMA Zones: 1664, 0, 109, 1, 109, 0, 0 UMA Slabs: 80, 0, 48753, 1147,795180888, 0, 0 UMA RCntSlabs: 88, 0, 94781, 34, 94781, 0, 0 UMA Hash: 256, 0, 2, 13, 9, 0, 0 4 Bucket: 32, 0, 1, 1374, 16673, 0, 0 6 Bucket: 48, 0, 0, 1328, 68823, 0, 0 8 Bucket: 64, 0, 0, 1550, 111011, 0, 0 12 Bucket: 96, 0, 27, 1367, 635428, 0, 0 16 Bucket: 128, 0, 2985, 1882,63426594, 11, 0 32 Bucket: 256, 0, 7782, 1938,74452218, 46, 0 64 Bucket: 512, 0, 3027, 2917,23978224,4610, 0 128 Bucket: 1024, 0, 10672, 5868,47846405605, 742, 0 vmem btag: 56, 0, 26858, 36687,340295492, 895, 0 VM OBJECT: 256, 0, 210353, 17497,367582461, 0, 0 RADIX NODE: 144, 0, 347266, 41642,796163752, 0, 0 MAP: 240, 0, 3, 61, 3, 0, 0 KMAP ENTRY: 128, 0, 6, 273, 6, 0, 0 MAP ENTRY: 128, 0, 1638, 2113,51340019, 0, 0 VMSPACE: 448, 0, 43, 308, 1040861, 0, 0 fakepg: 104, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 325, 0, 325, 0, 0 16: 16, 0, 16452, 38517,35980070234, 0, 0 32: 32, 0, 40209, 27166,3888808650, 0, 0 64: 64, 0, 17213, 68719,68691410126, 0, 0 128: 128, 0, 10520, 157128,476747943, 0, 0 256: 256, 0, 7914, 87501,1264824107, 0, 0 512: 512, 0, 22137, 93663,492467476, 0, 0 1024: 1024, 0, 66, 262, 889383, 0, 0 2048: 2048, 0, 130, 796,791425021, 0, 0 4096: 4096, 0, 393, 939,207284522, 0, 0 SLEEPQUEUE: 80, 0, 289, 734, 289, 0, 0 64 pcpu: 8, 0, 132323, 669, 132323, 0, 0 Files: 80, 0, 5483, 21667,14367132928, 0, 0 rl_entry: 40, 0, 157, 843, 157, 0, 0 TURNSTILE: 136, 0, 289, 271, 289, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 ertt: 72, 0, 8469, 38831,9821340346, 0, 0 ertt_txseginfo: 40, 0, 16571, 159929,95003293642, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1208, 0, 68, 112, 1040885, 0, 0 THREAD: 1168, 0, 259, 29, 273, 0, 0 cpuset: 72, 0, 106, 169, 114, 0, 0 audit_record: 1248, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 6518985, 43732, 45393,199965865255, 0, 0 mbuf: 256, 6518985, 8890, 45745,383256720839, 0, 0 mbuf_cluster: 2048, 524288, 89125, 37, 89125, 0, 0 mbuf_jumbo_page: 4096, 509294, 8444, 41756,33590222643, 0, 0 mbuf_jumbo_9k: 9216, 150902, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 84882, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 g_bio: 248, 0, 7, 13113,1955534264, 0, 0 DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0 ttyinq: 160, 0, 180, 570, 2955, 0, 0 ttyoutq: 256, 0, 95, 640, 1531, 0, 0 ata_request: 336, 0, 0, 88, 148, 0, 0 cryptop: 88, 0, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0, 0 FPU_save_area: 512, 0, 0, 0, 0, 0, 0 VNODE: 472, 0, 209164, 41156,1494569676, 0, 0 VNODEPOLL: 112, 0, 2, 278, 2, 0, 0 BUF TRIE: 144, 0, 7226, 98155,399747594, 0, 0 NAMEI: 1024, 0, 0, 340,1778005738, 0, 0 S VFS Cache: 108, 0, 104334, 130446,1097507303, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 36, 63336, 8079613, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 NCLNODE: 528, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 57266, 934,53319431, 0, 0 procdesc: 128, 0, 0, 0, 0, 0, 0 Mountpoints: 816, 0, 7, 33, 7, 0, 0 pipe: 744, 0, 6, 164, 836922, 0, 0 AIO: 208, 0, 0, 0, 0, 0, 0 AIOP: 32, 0, 0, 0, 0, 0, 0 AIOCB: 480, 0, 0, 0, 0, 0, 0 AIOL: 128, 0, 0, 0, 0, 0, 0 AIOLIO: 272, 0, 0, 0, 0, 0, 0 ksiginfo: 112, 0, 168, 1477, 4629801, 0, 0 itimer: 352, 0, 1, 32, 1, 0, 0 KNOTE: 128, 0, 5245, 20020,122354662379, 0, 0 flows: 40, 696322, 696322, 0, 933075,93243583709, 0 socket: 696, 523485, 11682, 57448,9822208423, 0, 0 ipq: 56, 16401, 0, 0, 0, 0, 0 udp_inpcb: 392, 523490, 53, 327, 615958, 0, 0 udpcb: 16, 523586, 53, 1453, 615958, 0, 0 tcp_inpcb: 392, 523490, 17842, 60488,9821340337, 0, 0 tcpcb: 1024, 523488, 8463, 37937,9821340336, 0, 0 tcptw: 88, 32805, 9373, 23432,1622657168,320470, 0 syncache: 160, 409600, 120, 264980,6085298712, 0, 0 hostcache: 136, 0, 0, 0, 0, 0, 0 tcpreass: 40, 32800, 27, 1773,118471998, 0, 0 sackhole: 32, 0, 117, 1758,1200394757, 0, 0 sctp_ep: 1400, 523486, 0, 0, 0, 0, 0 sctp_asoc: 2344, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80012, 0, 830, 42, 0, 0 sctp_raddr: 720, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400026, 0, 0, 0, 0, 0 sctp_readq: 104, 400026, 0, 0, 0, 0, 0 sctp_stream_msg_out: 104, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400000, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 ripcb: 392, 523490, 0, 240, 237, 0, 0 unpcb: 240, 523488, 38, 698, 251845, 0, 0 rtentry: 200, 0, 768, 212, 832, 0, 0 IPFW dynamic rule: 120, 4125, 0, 0, 0, 0, 0 divcb: 392, 523490, 0, 0, 0, 0, 0 selfd: 56, 0, 245, 1388,4214189053, 0, 0 SWAPMETA: 288, 2037191, 0, 0, 0, 0, 0 FFS inode: 168, 0, 209130, 41478,1494565024, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 209130, 41490,1494564173, 0, 0 # uname -a FreeBSD sq-l1-n2 10.0-STABLE FreeBSD 10.0-STABLE #0 r264098: Fri Apr 4 11:14:53 CST 2014 root@sq-l1-n2:/usr/obj/usr/src/sys/10-stable-r264098 amd64 Regards Simon 于 14-3-26 10:51, k simon 写道: > Thanks, Christopher. But I think my problem may does not related to > TSO issue. I have tried disable tso with "ifconfig igb(x) -tso" and > ovserved with "netstat -ihw 1", and found "oErrs" does not disappeared. > > Regards > Simon > > 于 14-3-25 22:08, Christopher Forgeron 写道: >> Hi Simon, >> >> Try checking out the "9.2 ixgbe tx queue hang' thread here, and see if >> it applies to you. >> >> >> On Tue, Mar 25, 2014 at 1:55 AM, k simon > > wrote: >> >> Hi,Lists: >> I have got lots of "no buffer available" on 10-stable with igb >> nic. >> But em and bce works well. And I tried force igb to 4 or 8 queues and >> set hw.igb.rx_process_limit="-1", but have nothing helped. >> >> >> Regards >> Simon >> >> >> >> # uname -a >> FreeBSD sq-l1-n2 10.0-STABLE FreeBSD 10.0-STABLE #0 r262469: Tue >> Feb 25 >> 13:27:11 CST 2014 >> root@sq-l1-n2:/usr/obj/usr/src/sys/stable-10-262458 amd64 >> >> >> # netstat -mb >> 19126/73289/92415 mbufs in use >> (current/cache/total) >> 13289/46841/60130/524288 mbuf clusters in use >> (current/cache/total/max) >> 13289/46812 mbuf+clusters out of packet secondary zone in use >> (current/cache) >> 5638/22605/28243/262144 4k (page size) jumbo clusters in use >> (current/cache/total/max) >> 0/0/0/77672 9k jumbo clusters in use (current/cache/total/max) >> 0/0/0/43690 16k jumbo clusters in use (current/cache/total/max) >> 53914K/202424K/256338K bytes allocated to network >> (current/cache/total) >> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) >> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >> 0 requests for sfbufs denied >> 0 requests for sfbufs delayed >> 0 requests for I/O initiated by sendfile >> >> >> # netstat -di >> Name Mtu Network Address Ipkts Ierrs Idrop >> Opkts Oerrs Coll Drop >> igb0 1500 00:1b:21:70:5f:80 17212101113 1355809 0 >> 19612978862 0 0 >> igb1 1500 00:1b:21:70:5f:81 76601294282 >> 81162751 0 >> 74236432686 0 0 >> lo0 16384 20532742636 0 0 >> 20522475797 0 0 >> lo0 - your-net localhost 2736994243 - - >> 20520227166 - - >> >> >> # sysctl hw.igb >> hw.igb.rxd: 2048 >> hw.igb.txd: 2048 >> hw.igb.enable_aim: 1 >> hw.igb.enable_msix: 1 >> hw.igb.max_interrupt_rate: 12000 >> hw.igb.buf_ring_size: 4096 >> hw.igb.header_split: 0 >> hw.igb.num_queues: 1 >> hw.igb.rx_process_limit: 1000 >> >> >> # sysctl dev.igb.1 >> dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 >> dev.igb.1.%driver: igb >> dev.igb.1.%location: slot=0 function=1 >> dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x8086 >> subdevice=0xa04c class=0x020000 >> dev.igb.1.%parent: pci8 >> dev.igb.1.nvm: -1 >> dev.igb.1.enable_aim: 1 >> dev.igb.1.fc: 0 >> dev.igb.1.rx_processing_limit: 4096 >> dev.igb.1.link_irq: 3 >> dev.igb.1.dropped: 0 >> dev.igb.1.tx_dma_fail: 0 >> dev.igb.1.rx_overruns: 0 >> dev.igb.1.watchdog_timeouts: 0 >> dev.igb.1.device_control: 1086325313 >> dev.igb.1.rx_control: 67141634 >> dev.igb.1.interrupt_mask: 4 >> dev.igb.1.extended_int_mask: 2147483651 >> dev.igb.1.tx_buf_alloc: 0 >> dev.igb.1.rx_buf_alloc: 0 >> dev.igb.1.fc_high_water: 58976 >> dev.igb.1.fc_low_water: 58960 >> dev.igb.1.queue0.no_desc_avail: 10874 >> dev.igb.1.queue0.tx_packets: 74509997338 >> dev.igb.1.queue0.rx_packets: 76837720630 >> dev.igb.1.queue0.rx_bytes: 35589607860237 >> dev.igb.1.queue0.lro_queued: 0 >> dev.igb.1.queue0.lro_flushed: 0 >> dev.igb.1.mac_stats.excess_coll: 0 >> dev.igb.1.mac_stats.single_coll: 0 >> dev.igb.1.mac_stats.multiple_coll: 0 >> dev.igb.1.mac_stats.late_coll: 0 >> dev.igb.1.mac_stats.collision_count: 0 >> dev.igb.1.mac_stats.symbol_errors: 0 >> dev.igb.1.mac_stats.sequence_errors: 0 >> dev.igb.1.mac_stats.defer_count: 0 >> dev.igb.1.mac_stats.missed_packets: 81162751 >> dev.igb.1.mac_stats.recv_no_buff: 176691324 >> dev.igb.1.mac_stats.recv_undersize: 0 >> dev.igb.1.mac_stats.recv_fragmented: 0 >> dev.igb.1.mac_stats.recv_oversize: 0 >> dev.igb.1.mac_stats.recv_jabber: 0 >> dev.igb.1.mac_stats.recv_errs: 0 >> dev.igb.1.mac_stats.crc_errs: 0 >> dev.igb.1.mac_stats.alignment_errs: 0 >> dev.igb.1.mac_stats.coll_ext_errs: 0 >> dev.igb.1.mac_stats.xon_recvd: 0 >> dev.igb.1.mac_stats.xon_txd: 0 >> dev.igb.1.mac_stats.xoff_recvd: 0 >> dev.igb.1.mac_stats.xoff_txd: 0 >> dev.igb.1.mac_stats.total_pkts_recvd: 76925709917 >> dev.igb.1.mac_stats.good_pkts_recvd: 76837704301 >> dev.igb.1.mac_stats.bcast_pkts_recvd: 49174716 >> dev.igb.1.mac_stats.mcast_pkts_recvd: 282670 >> dev.igb.1.mac_stats.rx_frames_64: 31057121854 >> dev.igb.1.mac_stats.rx_frames_65_127: 19996324498 >> dev.igb.1.mac_stats.rx_frames_128_255: 1171960837 >> dev.igb.1.mac_stats.rx_frames_256_511: 2295894674 >> dev.igb.1.mac_stats.rx_frames_512_1023: 2026241811 >> dev.igb.1.mac_stats.rx_frames_1024_1522: 20290160627 >> dev.igb.1.mac_stats.good_octets_recvd: 36204302378783 >> dev.igb.1.mac_stats.good_octets_txd: 59038220741656 >> dev.igb.1.mac_stats.total_pkts_txd: 90973292365 >> dev.igb.1.mac_stats.good_pkts_txd: 90973292359 >> dev.igb.1.mac_stats.bcast_pkts_txd: 2408182 >> dev.igb.1.mac_stats.mcast_pkts_txd: 246782 >> dev.igb.1.mac_stats.tx_frames_64: 24604769631 >> dev.igb.1.mac_stats.tx_frames_65_127: 21373976133 >> dev.igb.1.mac_stats.tx_frames_128_255: 3047554762 >> dev.igb.1.mac_stats.tx_frames_256_511: 3751820879 >> dev.igb.1.mac_stats.tx_frames_512_1023: 3481550512 >> dev.igb.1.mac_stats.tx_frames_1024_1522: 34713620448 >> dev.igb.1.mac_stats.tso_txd: 9549335102 >> dev.igb.1.mac_stats.tso_ctx_fail: 0 >> dev.igb.1.interrupts.asserts: 16929843739 >> dev.igb.1.interrupts.rx_pkt_timer: 76834839132 >> dev.igb.1.interrupts.rx_abs_timer: 0 >> dev.igb.1.interrupts.tx_pkt_timer: 0 >> dev.igb.1.interrupts.tx_abs_timer: 76835042438 >> dev.igb.1.interrupts.tx_queue_empty: 90970561829 >> dev.igb.1.interrupts.tx_queue_min_thresh: 0 >> dev.igb.1.interrupts.rx_desc_min_thresh: 0 >> dev.igb.1.interrupts.rx_overrun: 0 >> dev.igb.1.host.breaker_tx_pkt: 0 >> dev.igb.1.host.host_tx_pkt_discard: 0 >> dev.igb.1.host.rx_pkt: 2865171 >> dev.igb.1.host.breaker_rx_pkts: 0 >> dev.igb.1.host.breaker_rx_pkt_drop: 0 >> dev.igb.1.host.tx_good_pkt: 2730536 >> dev.igb.1.host.breaker_tx_pkt_drop: 0 >> dev.igb.1.host.rx_good_bytes: 36204307257585 >> dev.igb.1.host.tx_good_bytes: 59038220743158 >> dev.igb.1.host.length_errors: 0 >> dev.igb.1.host.serdes_violation_pkt: 0 >> dev.igb.1.host.header_redir_missed: 0 >> >> >> >> # cat /var/run/dmesg.boot |grep igb >> igb0: mem >> 0xd5ee0000-0xd5efffff,0xd5800000-0xd5bfffff,0xd5edc000-0xd5edffff >> irq 18 >> at device 0.0 on pci8 >> igb0: Using MSIX interrupts with 2 vectors >> igb0: Ethernet address: 00:1b:21:70:5f:80 >> igb1: mem >> 0xd5ea0000-0xd5ebffff,0xd5400000-0xd57fffff,0xd5ed8000-0xd5edbfff >> irq 19 >> at device 0.1 on pci8 >> igb1: Using MSIX interrupts with 2 vectors >> igb1: Ethernet address: 00:1b:21:70:5f:81 >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> " >> >>