From owner-freebsd-net@FreeBSD.ORG Fri Feb 24 18:24:51 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD269106564A; Fri, 24 Feb 2012 18:24:51 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) by mx1.freebsd.org (Postfix) with ESMTP id 4276C8FC0C; Fri, 24 Feb 2012 18:24:50 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.5/8.14.5) with ESMTP id q1OIOldD084415; Sat, 25 Feb 2012 01:24:48 +0700 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4F47D5EF.7060908@rdtc.ru> Date: Sat, 25 Feb 2012 01:24:47 +0700 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: "net@freebsd.org" , Alexander Motin , Gleb Smirnoff References: <4F47C55C.8060006@rdtc.ru> In-Reply-To: <4F47C55C.8060006@rdtc.ru> Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 8bit Cc: Subject: Re: netisr+lagg+fragments=80% packet loss X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Feb 2012 18:24:52 -0000 25.02.2012 00:14, Eugene Grosbein пишет: > Hi! > > I've found that my PPPoE BRAS server gives 80% packet loss > for fragmented pings against itself while no packet loss for transit fragmented pings same time. > The problem vanishes if I do ONE of following: > > 1) disable indirect netisr mode, e.g. set back sysctl net.isr.direct=1 AND > net.isr.direct_force=1 (net.isr.direct=1 only is not enough to eliminate problem), OR > 2) bring down one of two lagg ports, OR > 3) decrease ping packets size so they stop get fragmented. > > More details. There is FreeBSD 8.2-STABLE/amd64 PPPoE server with mpd-5.5 > and 4 NICs: em0 and em1 combined to lagg0 (LACP mode) having IP address > that carries untagged IP packets; plus, igb0 and igb1 combined to lagg1 > (againg, LACP mode) having no IP addresses - instead, there are about 1000 > vlan interfaces built on top of lagg1. All vlan interfaces carry PPPoE traffic only. > > I use another FreeBSD 8.2-STABLE system as PPPoE client (again, mpd-5.5), > it connects to BRAS just fine and sends/receives transit traffic just fine. > It also pings BRAS with packets up to 1492 (MTU on ng0 interface). > But, "ping -s 1472" (1500 bytes at IP layer) gives 80% drops. > > tcpdump -i ng0 shows that EVERY fragmented outgoing ICMP echo-request > gets its echo-reply but only small part of replies get in order: > fragment 0 first, fragment 1 next. > > Most of replies get back out of order: fragment 1 first, fragment 0 second. > These produce no lines in ping's output and do increment > 'fragments dropped after timeout' counter in "netstat -ss" ouput. > However, tcpdump shows they are received at once, no significant delay. > > This problem occurs only when net.isr.direct=0/net.isr.direct_force=0. > And only when lagg1 has both ports up and running. And when I use oversized pings. > At the same time, transit oversized pings go through this BRAS just fine, > no packet loss at all. Running two copies of tcpdump for igb0 and igb1 simultaneously, I see that fragments of the same ICMP echo-reply packet encapsulated within PPPoE frame always go out through different ports of lagg1. Even when they arrive to client in order, it seems this depends of switching network in between PPPoE server and client. Eugene Grosbein