From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 00:36:56 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CC61B106564A; Sun, 6 Feb 2011 00:36:56 +0000 (UTC) (envelope-from ivo.vachkov@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 67A3C8FC14; Sun, 6 Feb 2011 00:36:56 +0000 (UTC) Received: by vxa40 with SMTP id 40so1157647vxa.13 for ; Sat, 05 Feb 2011 16:36:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=BvYHlbW7LpUzw48TYiptwg4B4YchsZ/z0I5neNGARSc=; b=jh28TpNQ5rifkmEk22zEG1rb6ijNbpOPthGlZpHW7hh+KyVvC4EbI3/NXPjXXC2QiS xLuR+XibY9EiEHrzJXQnn7FGFcSo7jWAwmxQpHyIELEtTYXvQH+ccvXom/r4nL1NBs8Y WWKXep2WZ1qEfibNNSBI0MKFeN7LJl3VXX7qQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; b=AxV9WElKOmXvB+TFtW8+13f6gQlzWWIctyS6aJG6/YgzYKswHZm3DU79fGOzBVifbn T+WTg20UFvQtiEcvHw0MlO7pYfYwI/gUnmLhuwG8jWX0CYHa0kZg5laI6+t2Cvv3kKBe m5LrWIzd0CtYYS65PmMqcNhNCZo3fNoGZtmE8= Received: by 10.220.172.9 with SMTP id j9mr3579340vcz.201.1296950983931; Sat, 05 Feb 2011 16:09:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.220.193.9 with HTTP; Sat, 5 Feb 2011 16:09:23 -0800 (PST) In-Reply-To: <4D4DCD1E.1050906@freebsd.org> References: <4D4DCD1E.1050906@freebsd.org> From: Ivo Vachkov Date: Sun, 6 Feb 2011 02:09:23 +0200 Message-ID: To: Julian Elischer Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Net Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 00:36:56 -0000 Hello, How can I help? /ipv On Sun, Feb 6, 2011 at 12:20 AM, Julian Elischer wrote= : > for some time now it has been apparent that the divert socket protocol wa= s a > little too heavily tied to IPv4. > > With IPv6 coming along now, it seems that we should look at how to extend > it. > > I see a couple of possible ways to do this: > > --- the first way: ---- > > One would be to add an IPV6 version of divert sockets, possibly from the > same base code. The ipfw code to call it would pass on whether it was an > ipv4 or ipv6 packet that is passed out (or it can just look) > and the divert packet would pass it to the correct socket if it was openn= ed. > > From an application point of view, this means you would have to open an i= pv4 > divert socket and an ipv6 divert socket. > > if you didn't have the right one open.. you would just never see the pack= et. > > Since applications that use divert would probably have to be rewritten to > cope with ipv6 anyhwo this seems to be an > ok solution/cost. > > Any app that was not updated would continue to run with ipv4 but would ne= ver > see IPV6 packets even if diverted. > > ------ another way ---- > > Another way to do this would be to recode divert to be its own protocol > family with its own sockaddr type. > > that socket addr would include the family as now, but would have enough r= oom > to support ipv4 and ipv6 addresses, as well as special fields that are > curently not available in divert or are just 'hacked' > (such as the fact that the name of the interface is hidden in the 'sa_zer= o' > bytes of the ipv4 socket address, and if you keep it and pass it back you > are effectively passing that information back too). > > In this scheme we would allow the socket address structure to have > enough fields to be able to encode some of the more intersting > packet layer information that is in the mbuf. > For example, the FIB, or somefo the other packet flags > or maybe even one or two of the common tags. > > I could see that some of these flags might be useful to a divert agent th= at > understood the protocol stack it was working with: > > #define M_PROTO1 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000010 /* protocol-specif= ic */ > #define M_PROTO2 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000020 /* protocol-specif= ic */ > #define M_PROTO3 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000040 /* protocol-specif= ic */ > #define M_PROTO4 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000080 /* protocol-specif= ic */ > #define M_PROTO5 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000100 /* protocol-specif= ic */ > #define M_BCAST =C2=A0 =C2=A0 =C2=A0 =C2=A0 0x00000200 /* send/received a= s link-level broadcast > */ > #define M_MCAST =C2=A0 =C2=A0 =C2=A0 =C2=A0 0x00000400 /* send/received a= s link-level multicast > */ > #define M_SKIP_FIREWALL 0x00004000 /* skip firewall processing */ > > #define M_VLANTAG =C2=A0 =C2=A0 =C2=A0 0x00010000 /* ether_vtag is valid = */ > #define M_PROMISC =C2=A0 =C2=A0 =C2=A0 0x00020000 /* packet was not for u= s */ > #define M_PROTO6 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00080000 /* protocol-specif= ic */ > #define M_PROTO7 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00100000 /* protocol-specif= ic */ > #define M_PROTO8 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00200000 /* protocol-specif= ic */ > #define M_FLOWID =C2=A0 =C2=A0 =C2=A0 =C2=A00x00400000 /* flowid is valid= */ > > > If we really wanted to do more, we could also define an OOB format > that could be used with recvmsg() and sendmsg() that would be > extensible enough to really give a lot of information. > > This would be the least compatible, and to tell the truth, I'd be tempted= to > leave the old ipv4 interface in place as an upgrade aid. > it could however handle all sorts of protocols, not just ipv4 and ipv6 > but possibly L2 packets etc. as well. > It may also be more work than I hope to do :-) > > ------ > > If anyone else has suggetions or man-power or would like to help.. > pipe up! > > > Julian > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > --=20 "UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity." Dennis Ritchie From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 01:42:48 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0CAA106564A for ; Sun, 6 Feb 2011 01:42:48 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 8A8A78FC08 for ; Sun, 6 Feb 2011 01:42:48 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p161ghiL038556 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 5 Feb 2011 17:42:47 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4D4DFC95.9010804@freebsd.org> Date: Sat, 05 Feb 2011 17:42:45 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Ivo Vachkov References: <4D4DCD1E.1050906@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 01:42:48 -0000 On 2/5/11 4:09 PM, Ivo Vachkov wrote: > Hello, > > How can I help? if you have ipv6 connectivity and experience, I have no experience or connectivity, with it so I'll be coding blind and will need a tester. If you have an application for IPV6 testing that would be even better. Divert is often used for NAT but that doesn't seem very useful for IPv6 and natd doesn't support it anyhow. > /ipv > > On Sun, Feb 6, 2011 at 12:20 AM, Julian Elischer wrote: >> for some time now it has been apparent that the divert socket protocol was a >> little too heavily tied to IPv4. >> >> With IPv6 coming along now, it seems that we should look at how to extend >> it. >> >> I see a couple of possible ways to do this: >> >> --- the first way: ---- >> >> One would be to add an IPV6 version of divert sockets, possibly from the >> same base code. The ipfw code to call it would pass on whether it was an >> ipv4 or ipv6 packet that is passed out (or it can just look) >> and the divert packet would pass it to the correct socket if it was openned. >> >> From an application point of view, this means you would have to open an ipv4 >> divert socket and an ipv6 divert socket. >> >> if you didn't have the right one open.. you would just never see the packet. >> >> Since applications that use divert would probably have to be rewritten to >> cope with ipv6 anyhwo this seems to be an >> ok solution/cost. >> >> Any app that was not updated would continue to run with ipv4 but would never >> see IPV6 packets even if diverted. >> >> ------ another way ---- >> >> Another way to do this would be to recode divert to be its own protocol >> family with its own sockaddr type. >> >> that socket addr would include the family as now, but would have enough room >> to support ipv4 and ipv6 addresses, as well as special fields that are >> curently not available in divert or are just 'hacked' >> (such as the fact that the name of the interface is hidden in the 'sa_zero' >> bytes of the ipv4 socket address, and if you keep it and pass it back you >> are effectively passing that information back too). >> >> In this scheme we would allow the socket address structure to have >> enough fields to be able to encode some of the more intersting >> packet layer information that is in the mbuf. >> For example, the FIB, or somefo the other packet flags >> or maybe even one or two of the common tags. >> >> I could see that some of these flags might be useful to a divert agent that >> understood the protocol stack it was working with: >> >> #define M_PROTO1 0x00000010 /* protocol-specific */ >> #define M_PROTO2 0x00000020 /* protocol-specific */ >> #define M_PROTO3 0x00000040 /* protocol-specific */ >> #define M_PROTO4 0x00000080 /* protocol-specific */ >> #define M_PROTO5 0x00000100 /* protocol-specific */ >> #define M_BCAST 0x00000200 /* send/received as link-level broadcast >> */ >> #define M_MCAST 0x00000400 /* send/received as link-level multicast >> */ >> #define M_SKIP_FIREWALL 0x00004000 /* skip firewall processing */ >> >> #define M_VLANTAG 0x00010000 /* ether_vtag is valid */ >> #define M_PROMISC 0x00020000 /* packet was not for us */ >> #define M_PROTO6 0x00080000 /* protocol-specific */ >> #define M_PROTO7 0x00100000 /* protocol-specific */ >> #define M_PROTO8 0x00200000 /* protocol-specific */ >> #define M_FLOWID 0x00400000 /* flowid is valid */ >> >> >> If we really wanted to do more, we could also define an OOB format >> that could be used with recvmsg() and sendmsg() that would be >> extensible enough to really give a lot of information. >> >> This would be the least compatible, and to tell the truth, I'd be tempted to >> leave the old ipv4 interface in place as an upgrade aid. >> it could however handle all sorts of protocols, not just ipv4 and ipv6 >> but possibly L2 packets etc. as well. >> It may also be more work than I hope to do :-) >> >> ------ >> >> If anyone else has suggetions or man-power or would like to help.. >> pipe up! >> >> >> Julian >> >> >> >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 03:25:16 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B19C106564A; Sun, 6 Feb 2011 03:25:16 +0000 (UTC) (envelope-from keramida@ceid.upatras.gr) Received: from igloo.linux.gr (igloo.linux.gr [62.1.205.36]) by mx1.freebsd.org (Postfix) with ESMTP id 730678FC08; Sun, 6 Feb 2011 03:25:14 +0000 (UTC) X-Spam-Status: No X-Hellug-MailScanner-From: keramida@ceid.upatras.gr X-Hellug-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-2.9, required 5, autolearn=not spam, ALL_TRUSTED -1.00, BAYES_00 -1.90) X-Hellug-MailScanner: Found to be clean X-Hellug-MailScanner-ID: p163E5dY014472 Received: from gkeramidas-glaptop.linux.gr (207.47.25.82.static.nextweb.net [207.47.25.82]) (authenticated bits=0) by igloo.linux.gr (8.14.3/8.14.3/Debian-9.4) with ESMTP id p163E5dY014472 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sun, 6 Feb 2011 05:14:12 +0200 From: Giorgos Keramidas To: Doug Barton References: <4D411CC6.1090202@gont.com.ar> <4D431258.8040704@FreeBSD.org> Date: Sat, 05 Feb 2011 19:14:01 -0800 In-Reply-To: <4D431258.8040704@FreeBSD.org> (Doug Barton's message of "Fri, 28 Jan 2011 11:00:40 -0800") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Cc: FreeBSD Net , Ivo Vachkov , bz@freebsd.org Subject: Re: Proposed patch for Port Randomization modifications according to RFC6056 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 03:25:16 -0000 On Fri, 28 Jan 2011 11:00:40 -0800, Doug Barton wrote: > I haven't reviewed the patch in detail yet but I wanted to first thank > you for taking on this work, and being so responsive to Fernando's > request (which I agreed with, and you updated before I even had a > chance to say so). :) Thanks from me too. > My one comment so far is on the name of the sysctl's. There are 2 > problems with sysctl/variable names that use an rfc title. The first is > that they are not very descriptive to the 99.9% of users who are not > familiar with that particular doc. The second is more esoteric, but if > the rfc is subsequently updated or obsoleted we're stuck with either an > anachronism or updating code (both of which have their potential areas > of confusion). > > So in order to avoid this issue, and make it more consistent with the > existing: > > net.inet.ip.portrange.randomtime > net.inet.ip.portrange.randomcps > net.inet.ip.portrange.randomized > > How does net.inet.ip.portrange.randomalg sound? I would also suggest > that the second sysctl be named > net.inet.ip.portrange.randomalg.alg5_tradeoff so that one could do > sysctl net.inet.ip.portrange.randomalg' and see both values. But I won't > quibble on that. :) It's a usability issue too, so I'd certainly support renaming the sysctls to something human-friendly. It's always bad enough to go through look at a search engine to find out what net.inet.rfc1234 means. It's worse when RFC 1234 has been obsoleted a few years ago and now it's called RFC 54321. From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 04:07:07 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6ACD1065672 for ; Sun, 6 Feb 2011 04:07:07 +0000 (UTC) (envelope-from jfesler@gigo.com) Received: from goat.gigo.com (ipv6.gigo.com [IPv6:2001:470:1:18::2]) by mx1.freebsd.org (Postfix) with ESMTP id 9C9A68FC19 for ; Sun, 6 Feb 2011 04:07:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by goat.gigo.com (Postfix) with ESMTP id 98F7EC1F1 for ; Sat, 5 Feb 2011 20:07:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gigo.com; h= content-type:mime-version:user-agent:message-id:subject:from :date:received:received; s=s1; t=1296965227; bh=xcyo97CKgT4k20lK ncjLq7xXaCEUkNz+yznsnmYGZHU=; b=E2cuEo/PbX5ZYLQ/nIqJI4dAKQ8QIYN9 jmUdEubZvDi+iegZ4l23/PpMF0EYKuWlz7mFQfnszjteD28Cr6ErCSLQ+gAMQfKD QdzB38GwKu8Xk+gYSZYdshmDT3dqV6pNeyqFlD81bLh2KKrdh9ViUTuUzsD9ZKaI RXkrd3LHpS4= Received: from goat.gigo.com ([127.0.0.1]) by localhost (vette.gigo.com [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id qv+istL9+wF4 for ; Sat, 5 Feb 2011 20:07:07 -0800 (PST) Received: from goat.gigo.com (unknown [IPv6:2001:470:1:18::114]) by goat.gigo.com (Postfix) with ESMTPS id 6E640BD12 for ; Sat, 5 Feb 2011 20:07:07 -0800 (PST) Date: Sat, 5 Feb 2011 20:07:07 -0800 (PST) From: Jason Fesler To: freebsd-net@freebsd.org Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: MSS rewrite / MSS clamping? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 04:07:07 -0000 I'm in search of MSS clamping for FreeBSD servers; in particular, for IPv6. I'm finding pretty much nothing (except iptables..) on the net. Am I chasing wild geese? -- Jason Fesler, email/jabber resume: http://jfesler.com "Give a man fire, and he'll be warm for a day; set a man on fire, and he'll be warm for the rest of his life." From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 04:12:58 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC8661065674 for ; Sun, 6 Feb 2011 04:12:58 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [128.238.9.200]) by mx1.freebsd.org (Postfix) with ESMTP id 5C0548FC0C for ; Sun, 6 Feb 2011 04:12:56 +0000 (UTC) Received: (qmail 22069 invoked from network); 6 Feb 2011 04:12:55 -0000 Received: from unknown (HELO ?192.168.0.2?) (spawk@96.224.221.101) by acm.poly.edu with CAMELLIA256-SHA encrypted SMTP; 6 Feb 2011 04:12:55 -0000 Message-ID: <4D4E1FAA.5010905@acm.poly.edu> Date: Sat, 05 Feb 2011 23:12:26 -0500 From: Boris Kochergin User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.12) Gecko/20101031 Thunderbird/3.1.6 MIME-Version: 1.0 To: Jason Fesler References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: MSS rewrite / MSS clamping? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 04:12:58 -0000 On 02/05/11 23:07, Jason Fesler wrote: > I'm in search of MSS clamping for FreeBSD servers; in particular, for > IPv6. I'm finding pretty much nothing (except iptables..) on the net. > > Am I chasing wild geese? > > pf.conf(5) mentions a "max-mss" option for traffic normalization. -Boris From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 05:35:46 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3766D106566B for ; Sun, 6 Feb 2011 05:35:46 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 0B2828FC08 for ; Sun, 6 Feb 2011 05:35:45 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p165ZhSb039338 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 5 Feb 2011 21:35:44 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4D4E3332.7000401@freebsd.org> Date: Sat, 05 Feb 2011 21:35:46 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jason Fesler References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: MSS rewrite / MSS clamping? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 05:35:46 -0000 On 2/5/11 8:07 PM, Jason Fesler wrote: > I'm in search of MSS clamping for FreeBSD servers; in particular, > for IPv6. Well, there is ng_tcpmss but I see that it only works for IPv4 It may however be relatively easy to add code to allow it to work for IPV6. there is also the code in ports net/tcpmss which also is only IPV4,but in addition relies on DIVERT which is currently IPV4 only (I hope to change that). > I'm finding pretty much nothing (except iptables..) on the net. I assume you don't include things like ppp links if you are talking about a server. > > Am I chasing wild geese? don't know about pf. it MAY be able to help. For what it is worth, I expect a lot of IPV6 stuff to get kicked into shape over the next few months :-) > > From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 06:43:08 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AEEE5106566C for ; Sun, 6 Feb 2011 06:43:08 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 882A68FC13 for ; Sun, 6 Feb 2011 06:43:08 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id B448E46B0C; Sun, 6 Feb 2011 01:43:07 -0500 (EST) Date: Sun, 6 Feb 2011 06:43:07 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Prabhu Hariharan In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: Connections not purged on address deletion X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 06:43:08 -0000 On Fri, 4 Feb 2011, Prabhu Hariharan wrote: > When I delete an IP-address from an interface, the TCP (and other) > connections using that local IP-address are not getting purged. The telnet > or ssh sessions on the other end just get hung, as FreeBSD address-deletion > doesn't handle this situation and fails to call pfctlinput() to notify > protocols on this event. The TCP connections simply linger in the system > and takes it due course on TCP timers to free those inpcbs. > > tcp4 0 0 30.30.30.31.22 30.30.30.30.58796 ESTABLISHED > > Is this by design? Or any significance on relying on applications > intelligently to do timeouts, without a notification from network layer? I don't know if it's by design per se, but it proves extremely handy in practice when 802.11 blips and DHCP goes funny. Or, perhaps more historically, when PPP was restarted, leading to addresses being removed and re-added. Robert From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 10:36:13 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D5501065673 for ; Sun, 6 Feb 2011 10:36:13 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id EBFFD8FC16 for ; Sun, 6 Feb 2011 10:36:12 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.4/8.14.4) with ESMTP id p16AaAQ7018702 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 6 Feb 2011 05:36:11 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4D4E799A.50902@sentex.net> Date: Sun, 06 Feb 2011 05:36:10 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jason Fesler References: In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on IPv6:2607:f3e0:0:1::12 Cc: freebsd-net@freebsd.org Subject: Re: MSS rewrite / MSS clamping? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 10:36:13 -0000 On 2/5/2011 11:07 PM, Jason Fesler wrote: > I'm in search of MSS clamping for FreeBSD servers; in particular, for > IPv6. I'm finding pretty much nothing (except iptables..) on the net. Hi, I am curious as to where you would be running into MTU issues on IPv6 where you would need to manually compensate ? Broken tunnel providers ? ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 12:40:28 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD2CD106566C; Sun, 6 Feb 2011 12:40:28 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from fep21.mx.upcmail.net (fep21.mx.upcmail.net [62.179.121.41]) by mx1.freebsd.org (Postfix) with ESMTP id 758FC8FC0A; Sun, 6 Feb 2011 12:40:23 +0000 (UTC) Received: from edge02.upcmail.net ([192.168.13.237]) by viefep14-int.chello.at (InterMail vM.8.01.02.02 201-2260-120-106-20100312) with ESMTP id <20110206122456.BZUU1458.viefep14-int.chello.at@edge02.upcmail.net>; Sun, 6 Feb 2011 13:24:56 +0100 Received: from pinky ([213.46.23.80]) by edge02.upcmail.net with edge id 4cQu1g0301jgp3H02cQvdy; Sun, 06 Feb 2011 13:24:56 +0100 X-SourceIP: 213.46.23.80 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Ivo Vachkov" , "Julian Elischer" References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> Date: Sun, 06 Feb 2011 13:24:54 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: <4D4DFC95.9010804@freebsd.org> User-Agent: Opera Mail/11.01 (Win32) X-Cloudmark-Analysis: v=1.1 cv=CqMFsqQC4gx7bBgpmnW/wKYuJF/a5pXPeCAfngFtYkU= c=1 sm=0 a=bgpUlknNv7MA:10 a=kj9zAlcOel0A:10 a=6I5d2MoRAAAA:8 a=YMp63OpqK5xru0OnG0kA:9 a=x8VsXwHAeZm_HNgDiqcA:7 a=5UnPsBkWlQmSoYTC6P4Hrr4IkoYA:4 a=CjuIK1q_8ugA:10 a=SV7veod9ZcQA:10 a=rpATFeWpJGt08AtD:21 a=utuwudBhSSpfQb5V:21 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 Cc: FreeBSD Net Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 12:40:28 -0000 On Sun, 06 Feb 2011 02:42:45 +0100, Julian Elischer wrote: > On 2/5/11 4:09 PM, Ivo Vachkov wrote: >> Hello, >> >> How can I help? > > if you have ipv6 connectivity and experience, I have no experience or > connectivity, with it so > I'll be coding blind and will need a tester. > If you have an application for IPV6 testing that would be even better. > Divert is often used for NAT but that doesn't seem very useful for IPv6 > and > natd doesn't support it anyhow. Aren't there a lot of IPv6 tunnels available? Ronald. > >> /ipv >> >> On Sun, Feb 6, 2011 at 12:20 AM, Julian Elischer >> wrote: >>> for some time now it has been apparent that the divert socket protocol >>> was a >>> little too heavily tied to IPv4. >>> >>> With IPv6 coming along now, it seems that we should look at how to >>> extend >>> it. >>> >>> I see a couple of possible ways to do this: >>> >>> --- the first way: ---- >>> >>> One would be to add an IPV6 version of divert sockets, possibly from >>> the >>> same base code. The ipfw code to call it would pass on whether it was >>> an >>> ipv4 or ipv6 packet that is passed out (or it can just look) >>> and the divert packet would pass it to the correct socket if it was >>> openned. >>> >>> From an application point of view, this means you would have to open >>> an ipv4 >>> divert socket and an ipv6 divert socket. >>> >>> if you didn't have the right one open.. you would just never see the >>> packet. >>> >>> Since applications that use divert would probably have to be rewritten >>> to >>> cope with ipv6 anyhwo this seems to be an >>> ok solution/cost. >>> >>> Any app that was not updated would continue to run with ipv4 but would >>> never >>> see IPV6 packets even if diverted. >>> >>> ------ another way ---- >>> >>> Another way to do this would be to recode divert to be its own protocol >>> family with its own sockaddr type. >>> >>> that socket addr would include the family as now, but would have >>> enough room >>> to support ipv4 and ipv6 addresses, as well as special fields that are >>> curently not available in divert or are just 'hacked' >>> (such as the fact that the name of the interface is hidden in the >>> 'sa_zero' >>> bytes of the ipv4 socket address, and if you keep it and pass it back >>> you >>> are effectively passing that information back too). >>> >>> In this scheme we would allow the socket address structure to have >>> enough fields to be able to encode some of the more intersting >>> packet layer information that is in the mbuf. >>> For example, the FIB, or somefo the other packet flags >>> or maybe even one or two of the common tags. >>> >>> I could see that some of these flags might be useful to a divert agent >>> that >>> understood the protocol stack it was working with: >>> >>> #define M_PROTO1 0x00000010 /* protocol-specific */ >>> #define M_PROTO2 0x00000020 /* protocol-specific */ >>> #define M_PROTO3 0x00000040 /* protocol-specific */ >>> #define M_PROTO4 0x00000080 /* protocol-specific */ >>> #define M_PROTO5 0x00000100 /* protocol-specific */ >>> #define M_BCAST 0x00000200 /* send/received as link-level >>> broadcast >>> */ >>> #define M_MCAST 0x00000400 /* send/received as link-level >>> multicast >>> */ >>> #define M_SKIP_FIREWALL 0x00004000 /* skip firewall processing */ >>> >>> #define M_VLANTAG 0x00010000 /* ether_vtag is valid */ >>> #define M_PROMISC 0x00020000 /* packet was not for us */ >>> #define M_PROTO6 0x00080000 /* protocol-specific */ >>> #define M_PROTO7 0x00100000 /* protocol-specific */ >>> #define M_PROTO8 0x00200000 /* protocol-specific */ >>> #define M_FLOWID 0x00400000 /* flowid is valid */ >>> >>> >>> If we really wanted to do more, we could also define an OOB format >>> that could be used with recvmsg() and sendmsg() that would be >>> extensible enough to really give a lot of information. >>> >>> This would be the least compatible, and to tell the truth, I'd be >>> tempted to >>> leave the old ipv4 interface in place as an upgrade aid. >>> it could however handle all sorts of protocols, not just ipv4 and ipv6 >>> but possibly L2 packets etc. as well. >>> It may also be more work than I hope to do :-) >>> >>> ------ >>> >>> If anyone else has suggetions or man-power or would like to help.. >>> pipe up! >>> >>> >>> Julian >>> >>> >>> >>> >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >> >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 12:40:36 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B6EC106572E; Sun, 6 Feb 2011 12:40:36 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from fep21.mx.upcmail.net (fep21.mx.upcmail.net [62.179.121.41]) by mx1.freebsd.org (Postfix) with ESMTP id C714A8FC08; Sun, 6 Feb 2011 12:40:32 +0000 (UTC) Received: from edge02.upcmail.net ([192.168.13.237]) by viefep14-int.chello.at (InterMail vM.8.01.02.02 201-2260-120-106-20100312) with ESMTP id <20110206122400.BZCX1458.viefep14-int.chello.at@edge02.upcmail.net>; Sun, 6 Feb 2011 13:24:00 +0100 Received: from pinky ([213.46.23.80]) by edge02.upcmail.net with edge id 4cPx1g01C1jgp3H02cPynF; Sun, 06 Feb 2011 13:24:00 +0100 X-SourceIP: 213.46.23.80 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "FreeBSD Net" , "Julian Elischer" References: <4D4DCD1E.1050906@freebsd.org> Date: Sun, 06 Feb 2011 13:23:56 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: <4D4DCD1E.1050906@freebsd.org> User-Agent: Opera Mail/11.01 (Win32) X-Cloudmark-Analysis: v=1.1 cv=CqMFsqQC4gx7bBgpmnW/wKYuJF/a5pXPeCAfngFtYkU= c=1 sm=0 a=bgpUlknNv7MA:10 a=kj9zAlcOel0A:10 a=6I5d2MoRAAAA:8 a=vLs56JVJYBiU_vO3d9QA:9 a=NFLMyS7m-mTggUmF7JgA:7 a=NF_2HRrXSTRYkzpDQDj8VC5jIIMA:4 a=CjuIK1q_8ugA:10 a=SV7veod9ZcQA:10 a=mQHzowD39I_5Kws-:21 a=rnmD05TK9CeI4HP6:21 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 Cc: Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 12:40:36 -0000 You could make it a google summer of code project? Ronald. On Sat, 05 Feb 2011 23:20:14 +0100, Julian Elischer wrote: > for some time now it has been apparent that the divert socket protocol > was a little too heavily tied to IPv4. > > With IPv6 coming along now, it seems that we should look at how to > extend it. > > I see a couple of possible ways to do this: > > --- the first way: ---- > > One would be to add an IPV6 version of divert sockets, possibly from the > same base code. The ipfw code to call it would pass on whether it was an > ipv4 or ipv6 packet that is passed out (or it can just look) > and the divert packet would pass it to the correct socket if it was > openned. > > From an application point of view, this means you would have to open an > ipv4 divert socket and an ipv6 divert socket. > > if you didn't have the right one open.. you would just never see the > packet. > > Since applications that use divert would probably have to be rewritten > to cope with ipv6 anyhwo this seems to be an > ok solution/cost. > > Any app that was not updated would continue to run with ipv4 but would > never see IPV6 packets even if diverted. > > ------ another way ---- > > Another way to do this would be to recode divert to be its own protocol > family with its own sockaddr type. > > that socket addr would include the family as now, but would have enough > room to support ipv4 and ipv6 addresses, as well as special fields that > are curently not available in divert or are just 'hacked' > (such as the fact that the name of the interface is hidden in the > 'sa_zero' bytes of the ipv4 socket address, and if you keep it and pass > it back you are effectively passing that information back too). > > In this scheme we would allow the socket address structure to have > enough fields to be able to encode some of the more intersting > packet layer information that is in the mbuf. > For example, the FIB, or somefo the other packet flags > or maybe even one or two of the common tags. > > I could see that some of these flags might be useful to a divert agent > that understood the protocol stack it was working with: > > #define M_PROTO1 0x00000010 /* protocol-specific */ > #define M_PROTO2 0x00000020 /* protocol-specific */ > #define M_PROTO3 0x00000040 /* protocol-specific */ > #define M_PROTO4 0x00000080 /* protocol-specific */ > #define M_PROTO5 0x00000100 /* protocol-specific */ > #define M_BCAST 0x00000200 /* send/received as link-level > broadcast */ > #define M_MCAST 0x00000400 /* send/received as link-level > multicast */ > #define M_SKIP_FIREWALL 0x00004000 /* skip firewall processing */ > > #define M_VLANTAG 0x00010000 /* ether_vtag is valid */ > #define M_PROMISC 0x00020000 /* packet was not for us */ > #define M_PROTO6 0x00080000 /* protocol-specific */ > #define M_PROTO7 0x00100000 /* protocol-specific */ > #define M_PROTO8 0x00200000 /* protocol-specific */ > #define M_FLOWID 0x00400000 /* flowid is valid */ > > > If we really wanted to do more, we could also define an OOB format > that could be used with recvmsg() and sendmsg() that would be > extensible enough to really give a lot of information. > > This would be the least compatible, and to tell the truth, I'd be > tempted to leave the old ipv4 interface in place as an upgrade aid. > it could however handle all sorts of protocols, not just ipv4 and ipv6 > but possibly L2 packets etc. as well. > It may also be more work than I hope to do :-) > > ------ > > If anyone else has suggetions or man-power or would like to help.. > pipe up! > > > Julian > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 13:05:08 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D8641065672 for ; Sun, 6 Feb 2011 13:05:08 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [IPv6:2001:4068:10::3]) by mx1.freebsd.org (Postfix) with ESMTP id 2C9728FC16 for ; Sun, 6 Feb 2011 13:05:08 +0000 (UTC) Received: from localhost (amavis.fra.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id 8721E41C64A; Sun, 6 Feb 2011 14:05:06 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([192.168.74.103]) by localhost (amavis.fra.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id xyATl7yQv8eU; Sun, 6 Feb 2011 14:05:05 +0100 (CET) Received: by mail.cksoft.de (Postfix, from userid 66) id A90D041C679; Sun, 6 Feb 2011 14:05:05 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id B8CF14448F3; Sun, 6 Feb 2011 13:03:40 +0000 (UTC) Date: Sun, 6 Feb 2011 13:03:40 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Mike Tancsa In-Reply-To: <4D4E799A.50902@sentex.net> Message-ID: <20110206130139.B80258@maildrop.int.zabbadoz.net> References: <4D4E799A.50902@sentex.net> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Jason Fesler Subject: Re: MSS rewrite / MSS clamping? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 13:05:08 -0000 On Sun, 6 Feb 2011, Mike Tancsa wrote: > On 2/5/2011 11:07 PM, Jason Fesler wrote: >> I'm in search of MSS clamping for FreeBSD servers; in particular, for >> IPv6. I'm finding pretty much nothing (except iptables..) on the net. > > Hi, > I am curious as to where you would be running into MTU issues on IPv6 > where you would need to manually compensate ? Broken tunnel providers ? ICMP is bad and so filtering ICMPv6 must be good? Right? Even vendors selling firewalls do it to their own webserver. MSS clamping is a bad workaround for broken PMTU, and the real answer really is, get the paths fixed! Ther alternative I occactionally do is a destination route with a lower MTU but you cannot go endlessly down with IPv6 as there's the minimum. -- Bjoern A. Zeeb You have to have visions! Stop bit received. Insert coin for new address family. From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 16:47:47 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40FBE106566C for ; Sun, 6 Feb 2011 16:47:47 +0000 (UTC) (envelope-from jfesler@gigo.com) Received: from goat.gigo.com (ipv6.gigo.com [IPv6:2001:470:1:18::2]) by mx1.freebsd.org (Postfix) with ESMTP id 241678FC0A for ; Sun, 6 Feb 2011 16:47:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by goat.gigo.com (Postfix) with ESMTP id F3D70B8DC; Sun, 6 Feb 2011 08:47:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gigo.com; h= content-type:mime-version:user-agent:references:message-id :in-reply-to:subject:from:date:received:received; s=s1; t= 1297010866; bh=u0jRmC2DC7mNnfusN2+diev01+iKbpmWxMM/eh68opo=; b=U jfTi35LnoE1/62YG/YscnYbO80kitX/r8KLOnlWdXG4FlGeNxPm538mkKhkuFqU7 qvOVmWTNeLSDeJSmJlQQk/3hD6FlIn228I8sIGaIH/bzn1vmbWwW1KkjKbQvAK3Q E9+00rUudOvjajG1gqYUvtka6jdyqsZDlkfCSOCSMQ= Received: from goat.gigo.com ([127.0.0.1]) by localhost (vette.gigo.com [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id KoBl8fNwTx6N; Sun, 6 Feb 2011 08:47:46 -0800 (PST) Received: from goat.gigo.com (unknown [IPv6:2001:470:1:18::114]) by goat.gigo.com (Postfix) with ESMTPS id C3A72B85B; Sun, 6 Feb 2011 08:47:46 -0800 (PST) Date: Sun, 6 Feb 2011 08:47:46 -0800 (PST) From: Jason Fesler To: Mike Tancsa In-Reply-To: <4D4E799A.50902@sentex.net> Message-ID: References: <4D4E799A.50902@sentex.net> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: freebsd-net@freebsd.org Subject: Re: MSS rewrite / MSS clamping? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 16:47:47 -0000 Thanks everyone. I'll summarize the questions I saw, in one message here: Boris Kochergin wrote: > pf.conf(5) mentions a "max-mss" option for traffic normalization. Bingo. That indeed solved what I was after, and had been overlooking. For the mailing list archives, my /etc/pf.conf : | scrub in on em0 inet6 proto tcp to XXX port 80 max-mss 1220 | scrub out on em0 inet6 proto tcp from XXX to any port 80 max-mss 1220 | pass all Mike Tancsa says: > I am curious as to where you would be running into MTU issues on IPv6 > where you would need to manually compensate ? Broken tunnel providers ? First the why: I do see broken PMTU cases on a site (test-ipv6.com). My hope is, as I have resources contributed, to find a way to effectively test different MTU's without having multiple NICs and without tricks like adding a router in the middle with multiple vlans. As to causes: It can be people who never learned from IPv4 that filtering *all* ICMP is bad, are in charge of the ICMPv6 filters. It can be the 6in4 tunnel, hits a smaller MTU - but the ICMPv4 message to the tunnel origin does not really help the IPv6 origin. There is the standard, then there is reality; I see a *ton* of people with broken PMTUD on IPv6. :-( Bjoern A. Zeeb says: > MSS clamping is a bad workaround for broken PMTU, and the real answer > really is, get the paths fixed! Agreed. But, like IPv4, fixing PMTU is death by a thousand paper cuts, especially when you're the content provider side. Via private email: > I do this from my dhcpd, it may be feasible in your environment. > option max-mtu IIRC In some environments, that may indeed be feasible. In my case, every server I touch has a static address, except during OS install. I also need different IPs to at least emulate different MTUs; and one wants to use the same MTU across a given broadcast domain. From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 16:48:46 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A5FE1065670 for ; Sun, 6 Feb 2011 16:48:46 +0000 (UTC) (envelope-from sthaug@nethelp.no) Received: from bizet.nethelp.no (bizet.nethelp.no [195.1.209.33]) by mx1.freebsd.org (Postfix) with SMTP id A5DF18FC15 for ; Sun, 6 Feb 2011 16:48:45 +0000 (UTC) Received: (qmail 13608 invoked from network); 6 Feb 2011 16:48:43 -0000 Received: from bizet.nethelp.no (HELO localhost) (195.1.209.33) by bizet.nethelp.no with SMTP; 6 Feb 2011 16:48:43 -0000 Date: Sun, 06 Feb 2011 17:48:43 +0100 (CET) Message-Id: <20110206.174843.85344468.sthaug@nethelp.no> To: freebsd-net@freebsd.org From: sthaug@nethelp.no X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: IPFW and IPv6 packets with fragment header and last frag only X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 16:48:46 -0000 IPFW incorrectly handles IPv6 packets with a fragment header followed by a last fragment only (i.e. the fragment header has fragment offset = 0 and M bit = 0). Such packets are allowed by RFC 2460. The problem is well described in kern/145733 from 16. April 2010, but nothing seems to have happened with this PR so far. I see the effects of this problem on several name servers which handle IPv6 traffic. One typical example is 15:49:26.408456 IP6 2001:1a68::d911:210a > 2001:8c0:2001::3:53: frag (0|50) 50017 > 53: 38139% [1au] AAAA? dns1.eunet.no. (42) 0x0000: 6008 f572 003a 2c36 2001 1a68 0000 0000 `..r.:,6...h.... 0x0010: 0000 0000 d911 210a 2001 08c0 2001 0000 ......!......... 0x0020: 0000 0000 0003 0053 1100 0000 a977 6460 .......S.....wd` 0x0030: c361 0035 0032 21f6 94fb 0010 0001 0000 .a.5.2!......... 0x0040: 0000 0001 0464 6e73 3105 6575 6e65 7402 .....dns1.eunet. 0x0050: 6e6f 0000 1c00 0100 0029 1000 0000 8000 no.......)...... 0x0060: 0000 .. which results in the following log entry: Feb 6 15:49:26 dns1 kernel: IPFW2: IPV6 - Invalid Fragment Header and then the packet is dropped, even though the packet is perfectly valid. The logs on my name servers are getting filled with these error messages... Does anybody have an idea of whether the patch in kern/145733 will be incorporated into ip_fw2.c any time soon? Steinar Haug, Nethelp consulting, sthaug@nethelp.no From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 17:20:11 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7FCCB106566C for ; Sun, 6 Feb 2011 17:20:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6F0FA8FC17 for ; Sun, 6 Feb 2011 17:20:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p16HKBQN044483 for ; Sun, 6 Feb 2011 17:20:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p16HKBCI044482; Sun, 6 Feb 2011 17:20:11 GMT (envelope-from gnats) Date: Sun, 6 Feb 2011 17:20:11 GMT Message-Id: <201102061720.p16HKBCI044482@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: sthaug@nethelp.no Cc: Subject: Re: kern/153951: Intel 10GBase-LR Ethernet card detected as 10GBase-SR X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: sthaug@nethelp.no List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 17:20:11 -0000 The following reply was made to PR kern/153951; it has been noted by GNATS. From: sthaug@nethelp.no To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/153951: Intel 10GBase-LR Ethernet card detected as 10GBase-SR Date: Sun, 06 Feb 2011 18:19:37 +0100 (CET) > Checking some more I find that the same Intel 82599ES controller based > card is also available in 10GBase-SR versions - it simply depends on > the type of SFP+ used in the card. See for instance > > http://ark.intel.com/Product.aspx?id=39773 > http://ark.intel.com/Product.aspx?id=39774 > > Thus the *correct* solution to my problem is probably to retrieve the > optics type from the SFP+, instead of setting it statically in the > driver. Until such a solution is available, my 1-line patch works for > me... It looks like dynamic dection of optics type has been incorporated into the newest version if ixgbe.c, 1.17.2.12.2.2, with description 217713 - Make phy detection dynamic so adapters that support multiple types of media can be properly identified by the interface layer Thus I believe kern/153951 should be closed. Steinar Haug, Nethelp consulting, sthaug@nethelp.no From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 19:35:33 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68BF81065670; Sun, 6 Feb 2011 19:35:33 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id CA9228FC12; Sun, 6 Feb 2011 19:35:32 +0000 (UTC) Received: by fxm16 with SMTP id 16so4337930fxm.13 for ; Sun, 06 Feb 2011 11:35:31 -0800 (PST) Received: by 10.223.101.135 with SMTP id c7mr8413707fao.76.1296988680013; Sun, 06 Feb 2011 02:38:00 -0800 (PST) Received: from julie.lab.techwires.net (dslb-088-065-049-167.pools.arcor-ip.net [88.65.49.167]) by mx.google.com with ESMTPS id 11sm805210faw.44.2011.02.06.02.37.54 (version=SSLv3 cipher=RC4-MD5); Sun, 06 Feb 2011 02:37:58 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: Alexander Zagrebin Date: Sun, 6 Feb 2011 11:34:39 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-RELEASE; KDE/4.5.5; amd64; ; ) References: <20110204060808.GA97298@gw.zagrebin.ru> <201102041314.17939.bschmidt@freebsd.org> <20110204230615.GB12555@gw.zagrebin.ru> In-Reply-To: <20110204230615.GB12555@gw.zagrebin.ru> MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201102061134.39622.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org, Adrian Chadd Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 19:35:33 -0000 On Saturday 05 February 2011 00:06:15 Alexander Zagrebin wrote: > Hi! > > On 04.02.2011 13:14:17 +0100, Bernhard Schmidt wrote: > > > Also I have to note, that it seems that other wlan drivers can > > > has this problem too: only ral's and ath's code uses > > > ieee80211_beacon_update(). > > > > Yeah, true. > > > > > Also, I've found a kern/124753. One of > > > > > > replies contains a log's fragment with many records, like following: > > > kernel: ath0: [00:18:41:c0:06:54] power save mode on, 1 sta's > > > in ps > > > > > > mode kernel: ath0: [00:18:41:c0:06:54] save frame with age 0, 1 > > > now queued kernel: ath0: [00:18:41:c0:06:54] save frame with age > > > 0, 2 now queued kernel: ath0: [00:18:41:c0:06:54] power save > > > mode off, 0 sta's in ps mode kernel: ath0: [00:18:41:c0:06:54] > > > flush ps queue, 2 packets queue But there are no records, which > > > have to be for a PSM enabled stations: When a beacon is properly > > > updated, then we have to see records about 1. TIM updating, like > > > > > > ieee80211_beacon_update: TIM updated, pending 1, off 0, len > > > 1 > > > > > > 2. poll messages from a stations, like > > > > > > wlan0: [18:86:ac:10:4b:88] recv ps-poll, send packet, queue > > > > > > empty > > > > > > Thanks for your cooperation! > > > > You mean there are missing debug messages in net80211/run? > > I meant, that when stations in the PSM are handled correctly, then > the log contains a records like this: > > kernel: wlan0: [18:86:ac:10:4b:88] power save mode on, 1 sta's in ps > mode kernel: wlan0: [18:86:ac:10:4b:88] save frame with age 8, 1 now > queued kernel: wlan0: ieee80211_beacon_update: TIM updated, pending > 1, off 0, len 1 kernel: wlan0: [18:86:ac:10:4b:88] recv ps-poll, > send packet, queue empty kernel: wlan0: ieee80211_beacon_update: TIM > updated, pending 0, off 0, len 1 > > But the log's fragment from the kern/124753 doesn't contain records > with the text "...TIM updated..." and "...recv ps-poll...". > I had the very similar records in the log with the unpatched > if_run.c. Ah I see, ok. Added Adrian to CC, he might be interested. -- Bernhard From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 19:48:56 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E61D106566C for ; Sun, 6 Feb 2011 19:48:56 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id DAE228FC13 for ; Sun, 6 Feb 2011 19:48:55 +0000 (UTC) Received: by fxm16 with SMTP id 16so4346301fxm.13 for ; Sun, 06 Feb 2011 11:48:55 -0800 (PST) Received: by 10.223.116.1 with SMTP id k1mr3152660faq.51.1296989158031; Sun, 06 Feb 2011 02:45:58 -0800 (PST) Received: from julie.lab.techwires.net (dslb-088-065-049-167.pools.arcor-ip.net [88.65.49.167]) by mx.google.com with ESMTPS id r24sm812154fax.3.2011.02.06.02.45.55 (version=SSLv3 cipher=RC4-MD5); Sun, 06 Feb 2011 02:45:56 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: PseudoCylon Date: Sun, 6 Feb 2011 11:42:43 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-RELEASE; KDE/4.5.5; amd64; ; ) References: <20110204060808.GA97298@gw.zagrebin.ru> <201102041314.17939.bschmidt@freebsd.org> <147585.89029.qm@web39304.mail.mud.yahoo.com> In-Reply-To: <147585.89029.qm@web39304.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201102061142.43865.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org, Alexander Zagrebin Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 19:48:56 -0000 On Saturday 05 February 2011 05:45:59 PseudoCylon wrote: > ----- Original Message ---- > > > From: Bernhard Schmidt > > To: Alexander Zagrebin > > Cc: freebsd-net@freebsd.org; PseudoCylon > > Sent: Fri, February 4, 2011 5:14:17 AM > > Subject: Re: if_run in hostap mode: issue with stations in the > > power save mode > > > > On Friday 04 February 2011 12:40:21 Alexander Zagrebin wrote: > > > Hi! > > > > > > On 04.02.2011 09:51:34 +0100, Bernhard Schmidt wrote: > > > > On Friday 04 February 2011 07:08:08 Alexander Zagrebin wrote: > > > > > I'm using an Ralink RT2870 based adapter (run(4) driver) in > > > > > the hostap mode. and I've noticed that if_run doesn't > > > > > support stations working in the power save mode (PSM). The > > > > > reason is in lack of the TIM in beacons. The attached patch > > > > > adds this functionality and completely fixes this issue. > > > > > Despite the fact that patch is working, it seems that it > > > > > needs an additional work. For example, now the result of > > > > > ieee80211_beacon_update is ignored with a corresponding > > > > > message, but may be necessary to process it... > > > > > > > > > > Can somebody review it? > > > > > > > > That looks about right, good catch! > > > > > > > > Handling ieee80211_beacon_update()'s return value doesn't seem > > > > to be necessary, the mbuf's length is handled in the next few > > > > lines of code anyways, doesn't matter if it changed or not. > > > > > > > > Though, I have a some doubts about just restoring bo_flags is > > > > enough (Can't prove that with some obvious code, still..). It > > > > > > > > feels saner to me if we just reuse the whole mbuf, similar to > > > > what ath(4) does. Can you look at attached patch? Completely > > > > untested, so I'm not sure what does happen on e.g. changing > > > > the SSID. > > > > > > I've thought about such solution, and it looks more right, but > > > I've decided just to add ieee80211_beacon_update() to make the > > > patch clear. I'll try your patch a bit later, but I already > > > have a question: on the first invocation of the > > > run_update_beacon_cb() only ieee80211_beacon_alloc() will be > > > called. So dynamic beacon contents will not updated. Is it a > > > problem? > > > > I don't think it is. The work beacon_update does is handling > > changes to bo_flags, which are only changed through calls to > > iv_update_beacon(), so this is safe, because the driver itself > > does change bo_flags which is immediately followed by the beacon > > update process. > > I like the way mwl(4) handles it. > http://fxr.watson.org/fxr/source/dev/mwl/if_mwl.c#L1927 > though I don't know why it uses ieee80211_beacon_alloc() instead of > _update() > > @@run_update_beacon(struct ieee80211vap *vap, int item) > struct run_vap *rvp = RUN_VAP(vap); > +int mcast = 0; > uint32_t i; > > +KASSERT(vap != NULL, ("no beacon")); > + > +switch (item) { > +case IEEE80211_BEACON_ERP: > +run_updateslot(ic->ic_ifp); > +break; > +case IEEE80211_BEACON_HTINFO: > +run_updateprot(ic); > +break; > +case IEEE80211_BEACON_TIM: > +mcast = 1;/*TODO*/ > +break; > +default: > + break; > } > > +setbit(rvp->bo.bo_flags, item); > +ieee80211_beacon_update(vap->iv_bss, &rvp->bo, rvp->bm, mcast); > + > i = RUN_CMDQ_GET(&sc->cmdq_store); > DPRINTF("cmdq_store=%d\n", i); > sc->cmdq[i].func = run_update_beacon_cb; That looks good, it handles the case about slottime changes i've mentioned. > It's been working fine updating protection mode in HT mode past a few > days. (Some how, iwn(4) works fine with run(4), I cannot tell it > works fin with power saving mode.) Afaik iwn(4) doesn't use PS, never got around implementing that. > I'd like to move ieee80211_beacon_alloc() into iv_vap_alloc(). Then > we don't need to test beacon_mbuf == NULL in run_update_beacon_cb(), > and there is already switch we can use for conditionally alloc mem. Sounds fine with we. > > There is one expection I'm not sure about how to handle though, > > slottime. This value might change based on nodes associating and > > leaving, resulting in a call to ic_updateslot() which is currently > > commentted out. > > That's only because of LOR. I'm adding another call back function > since run_updateprot() need to be deferred when it is called > from run_update_beacon(). Great, thanks. Can I talk you into integrating that into Alexander's patch? -- Bernhard From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 19:53:32 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B8891065673 for ; Sun, 6 Feb 2011 19:53:32 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id B98988FC08 for ; Sun, 6 Feb 2011 19:53:31 +0000 (UTC) Received: by fxm16 with SMTP id 16so4348802fxm.13 for ; Sun, 06 Feb 2011 11:53:30 -0800 (PST) Received: by 10.223.101.135 with SMTP id c7mr8412472fao.76.1296988556129; Sun, 06 Feb 2011 02:35:56 -0800 (PST) Received: from julie.lab.techwires.net (dslb-088-065-049-167.pools.arcor-ip.net [88.65.49.167]) by mx.google.com with ESMTPS id r24sm810241fax.3.2011.02.06.02.35.53 (version=SSLv3 cipher=RC4-MD5); Sun, 06 Feb 2011 02:35:54 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: Alexander Zagrebin Date: Sun, 6 Feb 2011 11:32:40 +0100 User-Agent: KMail/1.13.5 (FreeBSD/8.1-RELEASE; KDE/4.5.5; amd64; ; ) References: <20110204060808.GA97298@gw.zagrebin.ru> <201102040951.34201.bschmidt@freebsd.org> <20110204223339.GA12555@gw.zagrebin.ru> In-Reply-To: <20110204223339.GA12555@gw.zagrebin.ru> MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201102061132.41032.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 19:53:32 -0000 On Friday 04 February 2011 23:33:40 Alexander Zagrebin wrote: > Hi! > > On 04.02.2011 09:51:34 +0100, Bernhard Schmidt wrote: > > On Friday 04 February 2011 07:08:08 Alexander Zagrebin wrote: > > > I'm using an Ralink RT2870 based adapter (run(4) driver) in the > > > hostap mode. and I've noticed that if_run doesn't support > > > stations working in the power save mode (PSM). The reason is in > > > lack of the TIM in beacons. The attached patch adds this > > > functionality and completely fixes this issue. Despite the fact > > > that patch is working, it seems that it needs an additional > > > work. For example, now the result of ieee80211_beacon_update is > > > ignored with a corresponding message, but may be necessary to > > > process it... > > > > > > Can somebody review it? > > > > That looks about right, good catch! > > > > Handling ieee80211_beacon_update()'s return value doesn't seem to > > be necessary, the mbuf's length is handled in the next few lines > > of code anyways, doesn't matter if it changed or not. > > > > Though, I have a some doubts about just restoring bo_flags is > > enough (Can't prove that with some obvious code, still..). It > > feels saner to me if we just reuse the whole mbuf, similar to what > > ath(4) does. Can you look at attached patch? Completely untested, > > so I'm not sure what does happen on e.g. changing the SSID. > > I've tried the slightly modified version of your patch (see attached > files), and found that it is working too. Moreover, it looks more > safe. For example, suppose we need update the beacon due to a new > TIM. Immediately after updating, but before the beacon with a TIM > will be transmitted, we need update the beacon again for any other > reason. With the first version of the patch the beacon will > completely recreated, so a TIM will be lost. But if we are using the > second version of the patch, then TIM will be preserved. Yeah, correct, that was my intention. I see you also fixed the remaining RUN_VAP() calls I've missed, thanks for that. :) > I had the doubts about ability to change or hide/unhide the SSID, but > it works too. It seems that after `ifconfig wlan0 ssid ...` or > `ifconfig wlan0 (hidessid|-hidessid)` the following occurs: > 1. run_newstate() is called > 2. run_newstate() invokes run_update_beacon_cb() > 3. run_update_beacon_cb() invokes ieee80211_beacon_update() > But I couldn't understand where ieee80211_beacon_update() can change > a SSID... As I see it, it can't. Only a call to ieee80211_beacon_construct() would do that. Can I talk you into figuring out what's going on here? Getting a definitive statement about SSID changes are correctly handled (by means of a trace) would help alot. Thanks. -- Bernhard From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 20:38:25 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 50A2F106566B for ; Sun, 6 Feb 2011 20:38:25 +0000 (UTC) (envelope-from mjl@luckie.org.nz) Received: from mailfilter67.ihug.co.nz (mailfilter3.ihug.co.nz [203.109.136.3]) by mx1.freebsd.org (Postfix) with ESMTP id F04588FC26 for ; Sun, 6 Feb 2011 20:38:24 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmcGAKqOTk12XHWG/2dsb2JhbACXEI4fdLl6hVoElBg X-IronPort-AV: E=Sophos;i="4.60,434,1291546800"; d="scan'208";a="132854949" Received: from 118-92-117-134.dsl.dyn.ihug.co.nz (HELO spandex.luckie.org.nz) ([118.92.117.134]) by cust.filter3.content.vf.net.nz with ESMTP/TLS/DHE-RSA-AES256-SHA; 07 Feb 2011 09:08:51 +1300 Received: from mjl by spandex.luckie.org.nz with local (Exim 4.74 (FreeBSD)) (envelope-from ) id 1PmAuh-000F6V-J9; Mon, 07 Feb 2011 09:08:51 +1300 Date: Mon, 7 Feb 2011 09:08:51 +1300 From: Matthew Luckie To: sthaug@nethelp.no Message-ID: <20110206200851.GB58011@spandex.luckie.org.nz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110206.174843.85344468.sthaug@nethelp.no> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: IPFW and IPv6 packets with fragment header and last frag only X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 20:38:25 -0000 > Does anybody have an idea of whether the patch in kern/145733 will be > incorporated into ip_fw2.c any time soon? That PR is mine. I've emailed people off list several times (last on 25 Jan) but have not made progress. To say I'm frustrated is an understatement. From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 20:50:46 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27CC1106564A for ; Sun, 6 Feb 2011 20:50:46 +0000 (UTC) (envelope-from sthaug@nethelp.no) Received: from bizet.nethelp.no (bizet.nethelp.no [195.1.209.33]) by mx1.freebsd.org (Postfix) with SMTP id 548118FC1A for ; Sun, 6 Feb 2011 20:50:45 +0000 (UTC) Received: (qmail 50345 invoked from network); 6 Feb 2011 20:50:43 -0000 Received: from bizet.nethelp.no (HELO localhost) (195.1.209.33) by bizet.nethelp.no with SMTP; 6 Feb 2011 20:50:43 -0000 Date: Sun, 06 Feb 2011 21:50:43 +0100 (CET) Message-Id: <20110206.215043.115976544.sthaug@nethelp.no> To: freebsd-net@freebsd.org From: sthaug@nethelp.no In-Reply-To: <20110206200851.GB58011@spandex.luckie.org.nz> References: <20110206.174843.85344468.sthaug@nethelp.no> <20110206200851.GB58011@spandex.luckie.org.nz> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: IPFW and IPv6 packets with fragment header and last frag only X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 20:50:46 -0000 > > Does anybody have an idea of whether the patch in kern/145733 will be > > incorporated into ip_fw2.c any time soon? > > That PR is mine. I've emailed people off list several times (last on > 25 Jan) but have not made progress. To say I'm frustrated is an > understatement. Getting this fixed is necessary to have FreeBSD ipfw work properly with IPv6. As it is, I'm dropping perfectly valid IPv6 traffic on my name servers, e.g. % grep 'Feb 5.*Invalid Fragment Header' /var/log/messages | wc -l 16 Let's see if requests from more people will change things. And for the FreeBSD committers: What can be done to push a needed patch along (for those of us without commit privileges)? Steinar Haug, Nethelp consulting, sthaug@nethelp.no From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 21:56:47 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CBF91065675 for ; Sun, 6 Feb 2011 21:56:47 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from ftp.translate.ru (ftp.translate.ru [80.249.188.42]) by mx1.freebsd.org (Postfix) with ESMTP id 9D0398FC13 for ; Sun, 6 Feb 2011 21:56:46 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (89.112.15.178.pppoe.eltel.net [89.112.15.178]) (Authenticated sender: lev@serebryakov.spb.ru) by ftp.translate.ru (Postfix) with ESMTPA id BF44A13DF42; Mon, 7 Feb 2011 00:40:29 +0300 (MSK) Date: Mon, 7 Feb 2011 00:40:23 +0300 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <787079038.20110207004023@serebryakov.spb.ru> To: freebsd-stable@freebsd.org, freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----------6921DF2E50B683" Cc: "Vogel, Jack" Subject: em0 hangs without any messages like "Watchdog timeout", only down/up reset it. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 21:56:47 -0000 ------------6921DF2E50B683 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Hello, Freebsd-stable. My em0 (the same, copy'n'paste hardware info from previous mnessage): em0@pci0:0:25:0: class=3D0x020000 card=3D0x82681043 chip=3D0x10bd808= 6 rev=3D0x02 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'Intel 82566DM Gigabit Ethernet Adapter (82566DM)' class =3D network subclass =3D ethernet bar [10] =3D type Memory, range 32, base 0xfea40000, size 131072, ena= bled bar [14] =3D type Memory, range 32, base 0xfea79000, size 4096, enabl= ed bar [18] =3D type I/O Port, range 32, base 0xdc00, size 32, enabled cap 01[c8] =3D powerspec 2 supports D0 D3 current D0 cap 05[d0] =3D MSI supports 1 message, 64 bit cap 09[e0] =3D vendor (length 6) Intel cap 2 version 0 It is on-board LAN on Q35-based MoBo (ASUS P5E-VM DO) It hangs under load without any output. When it works with POLLING, it prints "Watchdog timeout" and reset automatically several times a day, but without POLLING it simply hangs with same frequency. It is 8.2-PRERELEASE (from RELENG_8), cvsupped AFTER 22 Jan (last commit to e1000 drivers family). Locally system works, but mbufs are overfilled. "ifconfig em0 down && ifconfig em0 up" solves problem. Output of different diagnostic tools (vmstat, netstat, ifconfig, sysctl of dev.em.0 tree, top -S) are attached in one file. Early (about half year ago) this sytem works without any problems with net. --=20 // Black Lion AKA Lev Serebryakov ------------6921DF2E50B683 Content-Type: application/octet-stream; name="net.hangup.log" Content-transfer-encoding: base64 Content-Disposition: attachment; filename="net.hangup.log" Pj4+IGlmY29uZmlnIGVtMAplbTA6IGZsYWdzPThjNDM8VVAsQlJPQURDQVNULFJVTk5JTkcs T0FDVElWRSxTSU1QTEVYLE1VTFRJQ0FTVD4gbWV0cmljIDAgbXR1IDE1MDAKCW9wdGlvbnM9 MjE5YjxSWENTVU0sVFhDU1VNLFZMQU5fTVRVLFZMQU5fSFdUQUdHSU5HLFZMQU5fSFdDU1VN LFRTTzQsV09MX01BR0lDPgoJZXRoZXIgMDA6MWU6OGM6NzU6MDM6MGQKCWluZXQgMTkyLjE2 OC4xMzQuMyBuZXRtYXNrIDB4ZmZmZmZmMDAgYnJvYWRjYXN0IDE5Mi4xNjguMTM0LjI1NQoJ bWVkaWE6IEV0aGVybmV0IDEwMDBiYXNlVCA8ZnVsbC1kdXBsZXg+CglzdGF0dXM6IGFjdGl2 ZQo8PDwgaWZjb25maWcgZW0wCj4+PiB2bXN0YXQgLW0KICAgICAgICAgVHlwZSBJblVzZSBN ZW1Vc2UgSGlnaFVzZSBSZXF1ZXN0cyAgU2l6ZShzKQogICAgICAgbW9kdWxlICAgMTUyICAg IDE5SyAgICAgICAtICAgICAgMTUyICAxMjgKICAgICAgICAgIFVTQiAgICA3NiAgICA2Nksg ICAgICAgLSAgICAgICA4MCAgMTYsMzIsNjQsMTI4LDI1NiwxMDI0LDIwNDgsNDA5NgogICAg IG10eF9wb29sICAgICAyICAgIDE2SyAgICAgICAtICAgICAgICAyICAKICAgQ0FNIHBlcmlw aCAgICAyMiAgICAgNksgICAgICAgLSAgICAgICA0NCAgMTYsMzIsNjQsMjU2CiAgICAgcGNp X2xpbmsgICAgMTYgICAgIDJLICAgICAgIC0gICAgICAgMTYgIDY0LDEyOAogICAgICBhY3Bp c2VtICAgIDE5ICAgICAzSyAgICAgICAtICAgICAgIDE5ICAxMjgKICAgICAgc3VicHJvYyAg IDQ4OSAgIDQxNksgICAgICAgLSAgICA3MDc4NCAgNTEyLDQwOTYKICAgICAgICAgcHJvYyAg ICAgMiAgICAxNksgICAgICAgLSAgICAgICAgMiAgCiAgICAgIHNlc3Npb24gICAgMjMgICAg IDNLICAgICAgIC0gICAgIDQ2OTcgIDEyOAogICAgICAgICBwZ3JwICAgIDI1ICAgICA0SyAg ICAgICAtICAgICA0OTU3ICAxMjgKICAgICAgICAgY3JlZCAgICA2MiAgICAxMEsgICAgICAg LSAgNjcxODI3MiAgNjQsMjU2CiAgICAgIHVpZGluZm8gICAgIDggICAgIDNLICAgICAgIC0g IDExMjA3MjAgIDEyOCwyMDQ4CiAgICAgICBwbGltaXQgICAgMTIgICAgIDNLICAgICAgIC0g ICAgNjAzNzkgIDI1NgogICAgYWNwaV9wZXJmICAgICAyICAgICAxSyAgICAgICAtICAgICAg ICAyICAxMjgKICAgICAgQ0FNIFhQVCAgIDI5MyAgIDQyNUsgICAgICAgLSAgICAgIDQxNCAg MTYsMzIsNjQsMTI4LDI1NiwxMDI0LDIwNDgKICAgICAgIERFVkZTMSAgIDE0NiAgICA3M0sg ICAgICAgLSAgICAgIDE1OCAgNTEyCiAgICBzeXNjdGx0bXAgICAgIDAgICAgIDBLICAgICAg IC0gICAgIDE2NjIgIDE2LDMyLDY0LDEyOCwyNTYKICAgIHN5c2N0bG9pZCAgMzUzNSAgIDE3 NUsgICAgICAgLSAgICAgMzYyOCAgMTYsMzIsNjQsMTI4CiAgICAgICBzeXNjdGwgICAgIDAg ICAgIDBLICAgICAgIC0gICAgMzcyOTMgIDE2LDMyLDY0CiAgICAgIGNhbGxvdXQgICAgIDEg ICA1MTJLICAgICAgIC0gICAgICAgIDEgIAogICAgICAgICB1bXR4ICAgNTIyICAgIDY2SyAg ICAgICAtICAgICAgNTIyICAxMjgKICAgICBwMTAwMy4xYiAgICAgMSAgICAgMUsgICAgICAg LSAgICAgICAgMSAgMTYKICAgICAgICAgU1dBUCAgICAgMiAgIDU0OUsgICAgICAgLSAgICAg ICAgMiAgNjQKICAgICAgIERFVkZTMyAgIDE3MiAgICA0M0sgICAgICAgLSAgICAgIDE4NSAg MjU2CiAgICAgICBidXMtc2MgICAgNzEgICA0MTNLICAgICAgIC0gICAgIDEyNTEgIDE2LDMy LDY0LDEyOCwyNTYsNTEyLDIwNDgsNDA5NgogICAgICAgICAgYnVzICAgNjQwICAgIDY1SyAg ICAgICAtICAgICA0NDQwICAxNiwzMiw2NCwxMjgsMjU2LDUxMiwxMDI0CiAgICAgIGRldnN0 YXQgICAgMTQgICAgMjlLICAgICAgIC0gICAgICAgMTQgIDMyLDQwOTYKIGV2ZW50aGFuZGxl ciAgICA2NyAgICAgNksgICAgICAgLSAgICAgICA2NyAgNjQsMTI4CiAgICAgICAgIGtvYmog ICAgOTMgICAzNzJLICAgICAgIC0gICAgICAxMTUgIDQwOTYKICAgICAgUGVyLWNwdSAgICAg MSAgICAgMUsgICAgICAgLSAgICAgICAgMSAgMzIKICAgICAgICBERVZGUyAgICAyMiAgICAg MUsgICAgICAgLSAgICAgICAyMyAgMTYsMTI4CiAgICAgICAgIHJtYW4gICAyMDIgICAgMjVL ICAgICAgIC0gICAgICA2MjMgIDE2LDMyLDEyOAogICAgICAgREVWRlNQICAgICAxICAgICAx SyAgICAgICAtICAgICAgICAzICA2NAogICAgICAgICBzYnVmICAgICAwICAgICAwSyAgICAg ICAtICAgICAyMTQ0ICAxNiwzMiw2NCwxMjgsMjU2LDUxMiwxMDI0LDIwNDgsNDA5NgogICAg cGZzX25vZGVzICAgIDIxICAgICA2SyAgICAgICAtICAgICAgIDIxICAyNTYKICAgICAgICBz dGFjayAgICAgMCAgICAgMEsgICAgICAgLSAgICAgICAgNiAgMjU2CiAgICB0YXNrcXVldWUg ICAgMTUgICAgIDJLICAgICAgIC0gICAgICAgMTUgIDE2LDMyLDEyOAogICAgICAgVW5pdG5v ICAgIDEwICAgICAxSyAgICAgICAtICAgICAgMTAwICAzMiw2NAogICAgICAgICAgaW92ICAg ICAwICAgICAwSyAgICAgICAtICAgOTQ5MzAxICAxNiwzMiw2NCwxMjgsMjU2LDUxMgogICAg ICAgc2VsZWN0ICAgMTkwICAgIDI0SyAgICAgICAtIDExNDAyMzk0ODIwICAxMjgsNTEyLDEw MjQKICAgICBpb2N0bG9wcyAgICAgMCAgICAgMEsgICAgICAgLSAyMzgyNzQxNzEgIDE2LDMy LDY0LDEyOCwyNTYsNTEyLDEwMjQsMjA0OCw0MDk2CiAgICAgICAgICBtc2cgICAgIDQgICAg MzBLICAgICAgIC0gICAgICAgIDQgIDIwNDgsNDA5NgogICAgICAgICAgc2VtICAgICA0ICAg IDExSyAgICAgICAtICAgICAgICA0ICA1MTIsMTAyNAogICAgICAgICAgc2htICAgICAxICAg IDIwSyAgICAgICAtICAgICAgICAxICAKICAgICAgICAgIHR0eSAgICAyMCAgICAyMEsgICAg ICAgLSAgICAgICAzMiAgMTAyNCwyMDQ4CiAgICAgICAgICBwdHMgICAgIDAgICAgIDBLICAg ICAgIC0gICAgICAgIDYgIDI1NgogICAgIG1idWZfdGFnICAgICAwICAgICAwSyAgICAgICAt ICAgICAgIDgzICAzMgogICAgICAgIHNobWZkICAgICAxICAgICA4SyAgICAgICAtICAgICAg ICAxICAKICAgICAgICAgR0VPTSAgIDE3NSAgICAzOEsgICAgICAgLSAgICAgIDc1NyAgMTYs MzIsNjQsMTI4LDI1Niw1MTIsMTAyNAogICAgICAgICAgcGNiICAgIDkyICAgIDE1SyAgICAg ICAtICA2ODU3OTk3ICAxNiwzMiwxMDI0LDIwNDgsNDA5NgogICAgICAgc29uYW1lICAgICA2 ICAgICAxSyAgICAgICAtIDI2OTY1MDIxICAxNiwzMiwxMjgKICAgICAgICAgIGFjbCAgICAg MCAgICAgMEsgICAgICAgLSAgICAyMDIyMCAgNDA5NgogICAgICAgYmlvYnVmICAgICAwICAg ICAwSyAgICAgICAtICAgICAgMTI3ICAyMDQ4CiAgICAgdmZzY2FjaGUgICAgIDEgIDEwMjRL ICAgICAgIC0gICAgICAgIDEgIAogICBjbF9zYXZlYnVmICAgICAwICAgICAwSyAgICAgICAt ICAgIDU5NzI0ICA2NCwxMjgKICBleHBvcnRfaG9zdCAgICAgMiAgICAgMUsgICAgICAgLSAg ICAgICAgMiAgMjU2CiAgICAgdmZzX2hhc2ggICAgIDEgICA1MTJLICAgICAgIC0gICAgICAg IDEgIAogICAgICAgdm5vZGVzICAgICAyICAgICAxSyAgICAgICAtICAgICAgICAyICAyNTYK ICB2bm9kZW1hcmtlciAgICAgMCAgICAgMEsgICAgICAgLSAgIDI5NDQ5MyAgNTEyCiAgICAg ICAgbW91bnQgICAxMDQgICAgIDZLICAgICAgIC0gICAgICAzMDQgIDE2LDMyLDY0LDEyOCwy NTYsNTEyCiAgICAgICAgICBCUEYgICAgIDcgICAgIDlLICAgICAgIC0gICAgICAgMjAgIDEy OCwyNTYsNTEyLDQwOTYKICBldGhlcl9tdWx0aSAgICAxMiAgICAgMUsgICAgICAgLSAgICAg ICAyNiAgMTYsNjQKICAgICAgIGlmYWRkciAgICAxNCAgICAgN0sgICAgICAgLSAgICAgICAx NyAgMzIsNTEyLDQwOTYKICAgICAgICBpZm5ldCAgICAgMyAgICAgNUsgICAgICAgLSAgICAg ICAgMyAgMTI4LDIwNDgKICAgICAgICBjbG9uZSAgICAgMiAgICAgOEsgICAgICAgLSAgICAg ICAgMiAgNDA5NgogICAgICAgYXJwY29tICAgICAxICAgICAxSyAgICAgICAtICAgICAgICAx ICAxNgogICAgICBsbHRhYmxlICAgICAzICAgICAySyAgICAgICAtICAgICAgIDU2ICAyNTYs NTEyCiAgICAgIGZ3X3hmZXIgICAgIDAgICAgIDBLICAgICAgIC0gICAgICAgIDEgIDI1Ngog ICAgIGZpcmV3aXJlICAgIDExICAgIDM1SyAgICAgICAtICAgICAgIDE0ICA2NCwxMjgsNTEy LDEwMjQsMjA0OCw0MDk2CiAgICAgIHNjc2lfZGEgICAgIDAgICAgIDBLICAgICAgIC0gICAg ICAgMTYgIDE2CiAgICAgICBrYmRtdXggICAgIDYgICAgMTBLICAgICAgIC0gICAgICAgIDYg IDE2LDUxMiwxMDI0LDIwNDgsNDA5NgogICAgICAgICAgTEVEICAgICAxICAgICAxSyAgICAg ICAtICAgICAgICAxICAxMjgKICAgICAgIGlzYWRldiAgICAgNSAgICAgMUsgICAgICAgLSAg ICAgICAgNSAgMTI4CiAgICAgcm91dGV0YmwgICAgMTQgICAgIDRLICAgICAgIC0gICAxNzAx ODEgIDMyLDY0LDEyOCwyNTYsNTEyCiAgICAgICAgIGlnbXAgICAgIDIgICAgIDFLICAgICAg IC0gICAgICAgIDIgIDI1NgpDQU0gZGV2IHF1ZXVlICAgICA4ICAgICAxSyAgICAgICAtICAg ICAgICA4ICAxMjgKICAgIENBTSBxdWV1ZSAgICA0MyAgICAgM0sgICAgICAgLSAgICAgIDE0 OCAgMTYsMzIsNjQsMjU2CiAgICAgIENBTSBTSU0gICAgIDggICAgIDJLICAgICAgIC0gICAg ICAgIDggIDI1NgogIGlwX21vcHRpb25zICAgICA0ICAgICAxSyAgICAgICAtICAgICAgICA4 ICA2NCwyNTYKICAgICBpbl9tdWx0aSAgICAgMyAgICAgMUsgICAgICAgLSAgICAgICAgNSAg MjU2CiAgIGluX21maWx0ZXIgICAgIDIgICAgIDJLICAgICAgIC0gICAgICAgIDQgIDEwMjQK ICAgIGhvc3RjYWNoZSAgICAgMSAgICAyOEsgICAgICAgLSAgICAgICAgMSAgCiAgICAgc3lu Y2FjaGUgICAgIDEgICAgOTZLICAgICAgIC0gICAgICAgIDEgIAogICAgICBORlMgRkhBICAg ICAxICAgICAySyAgICAgICAtICAgICAgMzM5ICA2NCwyMDQ4CiAgICAgICAgICBycGMgICAz NDggICAxNzlLICAgICAgIC0gICAgIDExMzcgIDMyLDY0LDEyOCwyNTYsNTEyLDIwNDgKYXVk aXRfZXZjbGFzcyAgIDE3MiAgICAgNksgICAgICAgLSAgICAgIDIxMSAgMzIKICAgICBzYXZl ZGlubyAgICAgMCAgICAgMEsgICAgICAgLSAgICAzNzMzNSAgMjU2CiAgICBuZXdkaXJibGsg ICAgIDAgICAgIDBLICAgICAgIC0gICAgICA1OTMgIDY0CiAgICAgICBkaXJyZW0gICAgIDAg ICAgIDBLICAgICAgIC0gICAyMTY1NTAgIDY0CiAgICAgICAgbWtkaXIgICAgIDAgICAgIDBL ICAgICAgIC0gICAgICAzMjggIDY0CiAgICAgICBkaXJhZGQgICAgIDIgICAgIDFLICAgICAg IC0gICAyMTY3MzMgIDY0CiAgICAgZnJlZWZpbGUgICAgIDIgICAgIDFLICAgICAgIC0gICAx MDQ0NjMgIDY0CiAgICAgZnJlZWJsa3MgICAgIDIgICAgIDFLICAgICAgIC0gICAxMDQ3NzIg IDI1NgogICAgIGZyZWVmcmFnICAgICAwICAgICAwSyAgICAgICAtICAgMTczNDgzICA2NAog ICBhbGxvY2luZGlyICAgICAwICAgICAwSyAgICAgICAtICAgNzE3ODEzICAxMjgKICAgICBp bmRpcmRlcCAgICAgMCAgICAgMEsgICAgICAgLSAgICAxMzgwOCAgNjQKICBhbGxvY2RpcmVj dCAgICAgMyAgICAgMUsgICAgICAgLSAgIDQ0Njk4MiAgMjU2CiAgICBibXNhZmVtYXAgICAg IDIgICAgIDFLICAgICAgIC0gICAxMDA1OTcgIDEyOAogICAgICAgbmV3YmxrICAgICAxICAg ICAxSyAgICAgICAtICAxMTY0Nzk2ICA2NCw1MTIKICAgICBpbm9kZWRlcCAgICAgNyAgIDUx NEsgICAgICAgLSAgIDIzODAxNSAgMjU2CiAgICAgIHBhZ2VkZXAgICAgIDMgICAxMjlLICAg ICAgIC0gICAgMzI0NzYgIDEyOAogIHVmc19kaXJoYXNoICAgMzI1ICAgMTAxSyAgICAgICAt ICAgNjIxNjQ0ICAxNiwzMiw2NCwxMjgsMjU2LDUxMiwxMDI0CiAgICB1ZnNfbW91bnQgICAg MTUgICAxMjdLICAgICAgIC0gICAgICAgMTUgIDUxMiwyMDQ4LDQwOTYKICAgICAgVU1BSGFz aCAgICAgMyAgICAxMUsgICAgICAgLSAgICAgICAxMCAgNTEyLDEwMjQsMjA0OCw0MDk2CiAg ZGRiX2NhcHR1cmUgICAgIDEgICAgNDhLICAgICAgIC0gICAgICAgIDEgIAogICAgICAgYWNw aWNhICAzODM3ICAgMzkzSyAgICAgICAtICAgIDg4NjE1ICAxNiwzMiw2NCwxMjgsMjU2LDUx MiwxMDI0LDIwNDgKICAgICAgICAgY2RldiAgICAgOCAgICAgMksgICAgICAgLSAgICAgICAg OCAgMjU2CiAgICB2bV9wZ2RhdGEgICAgIDIgICAxMjlLICAgICAgIC0gICAgICAgIDIgIDEy OAogICAgICBhY3BpZGV2ICAgIDc4ICAgICA1SyAgICAgICAtICAgICAgIDc4ICA2NAogICAg ICAgIHNpZ2lvICAgICAxICAgICAxSyAgICAgICAtICAgICAgICAxICA2NAogICAgIGZpbGVk ZXNjICAgIDYxICAgIDc3SyAgICAgICAtICAgMTEwMTQ5ICAxNiwzMiw2NCwxMjgsMjU2LDUx MiwxMDI0LDIwNDgsNDA5NgogICAgICAgICBrZW52ICAgIDc4ICAgIDExSyAgICAgICAtICAg ICAgIDgyICAxNiwzMiw2NCwxMjgKICAgICAgaW9fYXBpYyAgICAgMSAgICAgMksgICAgICAg LSAgICAgICAgMSAgMjA0OAogICAgICAga3F1ZXVlICAgICAyICAgIDEzSyAgICAgICAtICAg MjYzMTY2ICAyNTYsMjA0OCw0MDk2CiAgICAgIG1lbWRlc2MgICAgIDEgICAgIDRLICAgICAg IC0gICAgICAgIDEgIDQwOTYKICAgICBhY3BpdGFzayAgICAgMSAgICAgMksgICAgICAgLSAg ICAgICAgMSAgMjA0OAogICAgcHJvYy1hcmdzICAgIDI3ICAgICAySyAgICAgICAtICAgMjAw NDE1ICAxNiwzMiw2NCwxMjgsMjU2CiAgICAgYXRrYmRkZXYgICAgIDIgICAgIDFLICAgICAg IC0gICAgICAgIDIgIDY0CiAgICAgIGl0aHJlYWQgICAgNzYgICAgMTJLICAgICAgIC0gICAg ICAgNzYgIDMyLDEyOCwyNTYKICAgICAgZW50cm9weSAgMTAyNCAgICA2NEsgICAgICAgLSAg ICAgMTAyNCAgNjQKICAgICAgICAgVUFSVCAgICAgMyAgICAgMksgICAgICAgLSAgICAgICAg MyAgMTYsNTEyLDEwMjQKICAgICAgIEtUUkFDRSAgIDEwMCAgICAxM0sgICAgICAgLSAgICAg IDEwMCAgMTI4CiAgICAgICBsaW5rZXIgICAgNTcgICAgIDZLICAgICAgIC0gICAgICAgNjMg IDE2LDMyLDY0LDEyOCw1MTIKICAgICAgICBsb2NrZiAgICA1MyAgICAgNksgICAgICAgLSAg NjM1OTM0NCAgNjQsMTI4CiAgICAgICAgIHRlbXAgICAgMjIgICA0MDFLICAgICAgIC0gIDE0 MTAzMzkgIDE2LDMyLDY0LDEyOCwyNTYsNTEyLDEwMjQsMjA0OCw0MDk2CiAgICAgICBkZXZi dWYgMjAyMzAgMzU5MzNLICAgICAgIC0gICAgMjAzMjkgIDE2LDMyLDY0LDEyOCwyNTYsNTEy LDEwMjQsMjA0OCw0MDk2CiAgICAgICBVU0JkZXYgICAgNDcgICAgMTJLICAgICAgIC0gICAg ICAgNDcgIDY0LDEyOCwxMDI0CiAgICAgbmV4dXNkZXYgICAgIDMgICAgIDFLICAgICAgIC0g ICAgICAgIDMgIDE2CiAgIHJhaWQ1X2RhdGEgICAgIDYgIDUzODlLICAgICAgIC0gMjM1NDE2 NzU3ICAxNiw2NCw1MTIsNDA5Ngo8PDwgdm1zdGF0IC1tCj4+PiBuZXRzdGF0IC1tCjEyNjg2 LzYxODQvMTg4NzAgbWJ1ZnMgaW4gdXNlIChjdXJyZW50L2NhY2hlL3RvdGFsKQo0MjY3LzU4 NzcvMTAxNDQvMjA0ODAwIG1idWYgY2x1c3RlcnMgaW4gdXNlIChjdXJyZW50L2NhY2hlL3Rv dGFsL21heCkKNDIzOS81ODU4IG1idWYrY2x1c3RlcnMgb3V0IG9mIHBhY2tldCBzZWNvbmRh cnkgem9uZSBpbiB1c2UgKGN1cnJlbnQvY2FjaGUpCjAvMjUxLzI1MS8xOTIwMDAgNGsgKHBh Z2Ugc2l6ZSkganVtYm8gY2x1c3RlcnMgaW4gdXNlIChjdXJyZW50L2NhY2hlL3RvdGFsL21h eCkKMC8wLzAvNjQwMCA5ayBqdW1ibyBjbHVzdGVycyBpbiB1c2UgKGN1cnJlbnQvY2FjaGUv dG90YWwvbWF4KQowLzAvMC8zMjAwIDE2ayBqdW1ibyBjbHVzdGVycyBpbiB1c2UgKGN1cnJl bnQvY2FjaGUvdG90YWwvbWF4KQoxMTcwNUsvMTQzMDRLLzI2MDA5SyBieXRlcyBhbGxvY2F0 ZWQgdG8gbmV0d29yayAoY3VycmVudC9jYWNoZS90b3RhbCkKMC8wLzAgcmVxdWVzdHMgZm9y IG1idWZzIGRlbmllZCAobWJ1ZnMvY2x1c3RlcnMvbWJ1ZitjbHVzdGVycykKMC8wLzAgcmVx dWVzdHMgZm9yIGp1bWJvIGNsdXN0ZXJzIGRlbmllZCAoNGsvOWsvMTZrKQowLzAvMCBzZmJ1 ZnMgaW4gdXNlIChjdXJyZW50L3BlYWsvbWF4KQowIHJlcXVlc3RzIGZvciBzZmJ1ZnMgZGVu aWVkCjAgcmVxdWVzdHMgZm9yIHNmYnVmcyBkZWxheWVkCjAgcmVxdWVzdHMgZm9yIEkvTyBp bml0aWF0ZWQgYnkgc2VuZGZpbGUKMCBjYWxscyB0byBwcm90b2NvbCBkcmFpbiByb3V0aW5l cwo8PDwgbmV0c3RhdCAtbQo+Pj4gdG9wIC1TbmQgMQpsYXN0IHBpZDogNzAzNDQ7ICBsb2Fk IGF2ZXJhZ2VzOiAgMC4wMCwgIDAuMDAsICAwLjAwICB1cCA5KzE3OjM5OjMwICAgIDE2OjM5 OjU0CjExNCBwcm9jZXNzZXM6IDMgcnVubmluZywgOTIgc2xlZXBpbmcsIDE5IHdhaXRpbmcK Ck1lbTogOTNNIEFjdGl2ZSwgMTM3M00gSW5hY3QsIDM3OE0gV2lyZWQsIDc2TSBDYWNoZSwg MjEzTSBCdWYsIDUzTSBGcmVlClN3YXA6IDQwOTZNIFRvdGFsLCAzOTZLIFVzZWQsIDQwOTVN IEZyZWUKCgogIFBJRCBVU0VSTkFNRSAgICAgIFRIUiBQUkkgTklDRSAgIFNJWkUgICAgUkVT IFNUQVRFICAgQyAgIFRJTUUgICBXQ1BVIENPTU1BTkQKICAgMTEgcm9vdCAgICAgICAgICAg IDIgMTcxIGtpMzEgICAgIDBLICAgIDMySyBSVU4gICAgIDAgMzcxLjJIIDIwMC4wMCUgaWRs ZQogICAxMiByb290ICAgICAgICAgICAxOSAtNjAgICAgLSAgICAgMEsgICAzMDRLIFdBSVQg ICAgMCAzNzY6NDcgIDAuMTAlIGludHIKMzUyNDMgcnRvcnJlbnQgICAgICAgIDMgIDQ0ICAg IDAgOTgwNDRLIDgxNDI4SyBzZWxlY3QgIDEgIDM5LjFIICAwLjAwJSB0cmFuc21pc3Npb24t ZGFlbW9uCiAgICAwIHJvb3QgICAgICAgICAgICA5IC02OCAgICAwICAgICAwSyAgIDEyOEsg LSAgICAgICAxICA3ODozMiAgMC4wMCUga2VybmVsCiAgIDE0IHJvb3QgICAgICAgICAgIDMz IC02NCAgICAtICAgICAwSyAgIDUyOEsgLSAgICAgICAxICAzMzoyMSAgMC4wMCUgdXNiCiAg IDIwIHJvb3QgICAgICAgICAgICAyICAtOCAgICAtICAgICAwSyAgICAzMksgZ3I1ZG8gICAx ICAzMTowMSAgMC4wMCUgZ19yYWlkNQogICAgNCByb290ICAgICAgICAgICAgMSAgLTggICAg LSAgICAgMEsgICAgMTZLIC0gICAgICAgMCAgMjQ6NDIgIDAuMDAlIGdfZG93bgogICAgMyBy b290ICAgICAgICAgICAgMSAgLTggICAgLSAgICAgMEsgICAgMTZLIC0gICAgICAgMSAgMTg6 NTMgIDAuMDAlIGdfdXAKICAgMTcgcm9vdCAgICAgICAgICAgIDEgIDQ0ICAgIC0gICAgIDBL ICAgIDE2SyBzeW5jZXIgIDEgIDE2OjA3ICAwLjAwJSBzeW5jZXIKICAgMTMgcm9vdCAgICAg ICAgICAgIDEgLTE2ICAgIC0gICAgIDBLICAgIDE2SyAtICAgICAgIDEgIDEyOjAyICAwLjAw JSB5YXJyb3cKICAgIDcgcm9vdCAgICAgICAgICAgIDEgIDQ0ICAgIC0gICAgIDBLICAgIDE2 SyBwc2xlZXAgIDEgICA2OjE3ICAwLjAwJSBwYWdlZGFlbW9uCiAgNzk5IHJvb3QgICAgICAg ICAgICAxICA0NCAgICAwIDIxMDY0SyAgMjc2MEsgc2VsZWN0ICAxICAgMDozMyAgMC4wMCUg bm1iZAo0MTE3MCB1dWNwICAgICAgICAgICAgMSAgNDQgICAgMCAgNjkyMEsgIDEzMjhLIHNl bGVjdCAgMSAgIDA6MzMgIDAuMDAlIG1lZ2F0ZWMKICAgIDIgcm9vdCAgICAgICAgICAgIDEg IC04ICAgIC0gICAgIDBLICAgIDE2SyAtICAgICAgIDEgICAwOjI4ICAwLjAwJSBnX2V2ZW50 CiAgODcxIHJvb3QgICAgICAgICAgICAxICA0NCAgICAwIDExNzkySyAgMTk4OEsgc2VsZWN0 ICAxICAgMDoyOCAgMC4wMCUgbnRwZAogICAxOCByb290ICAgICAgICAgICAgMSAgNDQgICAg LSAgICAgMEsgICAgMTZLIHZscnV3dCAgMSAgIDA6MjYgIDAuMDAlIHZubHJ1CjQxMTcyIHV1 Y3AgICAgICAgICAgICAxICA0NCAgICAwIDEwOTMySyAgMjE3Nksgc2VsZWN0ICAxICAgMDox NCAgMC4wMCUgdXBzZAogMTA0MCByb290ICAgICAgICAgICAgMSAgNDQgICAgMCAxMjAyMEsg IDI5NTZLIHNlbGVjdCAgMSAgIDA6MTIgIDAuMDAlIHNlbmRtYWlsCgo8PDwgdG9wIC1TbmQg MQo+Pj4gc3lzY3RsIGRldi5lbS4wCmRldi5lbS4wLiVkZXNjOiBJbnRlbChSKSBQUk8vMTAw MCBOZXR3b3JrIENvbm5lY3Rpb24gNy4xLjkKZGV2LmVtLjAuJWRyaXZlcjogZW0KZGV2LmVt LjAuJWxvY2F0aW9uOiBzbG90PTI1IGZ1bmN0aW9uPTAgaGFuZGxlPVxfU0JfLlBDSTAuR0JF QwpkZXYuZW0uMC4lcG5waW5mbzogdmVuZG9yPTB4ODA4NiBkZXZpY2U9MHgxMGJkIHN1YnZl bmRvcj0weDEwNDMgc3ViZGV2aWNlPTB4ODI2OCBjbGFzcz0weDAyMDAwMApkZXYuZW0uMC4l cGFyZW50OiBwY2kwCmRldi5lbS4wLm52bTogLTEKZGV2LmVtLjAuZGVidWc6IC0xCmRldi5l bS4wLnJ4X2ludF9kZWxheTogMjAwCmRldi5lbS4wLnR4X2ludF9kZWxheTogMjAwCmRldi5l bS4wLnJ4X2Fic19pbnRfZGVsYXk6IDQwMDAKZGV2LmVtLjAudHhfYWJzX2ludF9kZWxheTog NDAwMApkZXYuZW0uMC5yeF9wcm9jZXNzaW5nX2xpbWl0OiA0MDk2CmRldi5lbS4wLmZsb3df Y29udHJvbDogMwpkZXYuZW0uMC5saW5rX2lycTogMApkZXYuZW0uMC5tYnVmX2FsbG9jX2Zh aWw6IDAKZGV2LmVtLjAuY2x1c3Rlcl9hbGxvY19mYWlsOiAwCmRldi5lbS4wLmRyb3BwZWQ6 IDAKZGV2LmVtLjAudHhfZG1hX2ZhaWw6IDEKZGV2LmVtLjAucnhfb3ZlcnJ1bnM6IDAKZGV2 LmVtLjAud2F0Y2hkb2dfdGltZW91dHM6IDQKZGV2LmVtLjAuZGV2aWNlX2NvbnRyb2w6IDEw NzQ3OTA5NzYKZGV2LmVtLjAucnhfY29udHJvbDogNjcxNDE2MzQKZGV2LmVtLjAuZmNfaGln aF93YXRlcjogODE5MgpkZXYuZW0uMC5mY19sb3dfd2F0ZXI6IDY2OTIKZGV2LmVtLjAucXVl dWUwLnR4ZF9oZWFkOiAzMzM5CmRldi5lbS4wLnF1ZXVlMC50eGRfdGFpbDogMzMwMgpkZXYu ZW0uMC5xdWV1ZTAudHhfaXJxOiAwCmRldi5lbS4wLnF1ZXVlMC5ub19kZXNjX2F2YWlsOiAw CmRldi5lbS4wLnF1ZXVlMC5yeGRfaGVhZDogMTg1NgpkZXYuZW0uMC5xdWV1ZTAucnhkX3Rh aWw6IDE4NTUKZGV2LmVtLjAucXVldWUwLnJ4X2lycTogMApkZXYuZW0uMC5tYWNfc3RhdHMu ZXhjZXNzX2NvbGw6IDAKZGV2LmVtLjAubWFjX3N0YXRzLnNpbmdsZV9jb2xsOiAwCmRldi5l bS4wLm1hY19zdGF0cy5tdWx0aXBsZV9jb2xsOiAwCmRldi5lbS4wLm1hY19zdGF0cy5sYXRl X2NvbGw6IDAKZGV2LmVtLjAubWFjX3N0YXRzLmNvbGxpc2lvbl9jb3VudDogMApkZXYuZW0u MC5tYWNfc3RhdHMuc3ltYm9sX2Vycm9yczogMApkZXYuZW0uMC5tYWNfc3RhdHMuc2VxdWVu Y2VfZXJyb3JzOiAwCmRldi5lbS4wLm1hY19zdGF0cy5kZWZlcl9jb3VudDogMTYxNwpkZXYu ZW0uMC5tYWNfc3RhdHMubWlzc2VkX3BhY2tldHM6IDE4MjE0CmRldi5lbS4wLm1hY19zdGF0 cy5yZWN2X25vX2J1ZmY6IDAKZGV2LmVtLjAubWFjX3N0YXRzLnJlY3ZfdW5kZXJzaXplOiAw CmRldi5lbS4wLm1hY19zdGF0cy5yZWN2X2ZyYWdtZW50ZWQ6IDUKZGV2LmVtLjAubWFjX3N0 YXRzLnJlY3Zfb3ZlcnNpemU6IDAKZGV2LmVtLjAubWFjX3N0YXRzLnJlY3ZfamFiYmVyOiAw CmRldi5lbS4wLm1hY19zdGF0cy5yZWN2X2VycnM6IDEzMQpkZXYuZW0uMC5tYWNfc3RhdHMu Y3JjX2VycnM6IDEyNgpkZXYuZW0uMC5tYWNfc3RhdHMuYWxpZ25tZW50X2VycnM6IDAKZGV2 LmVtLjAubWFjX3N0YXRzLmNvbGxfZXh0X2VycnM6IDEKZGV2LmVtLjAubWFjX3N0YXRzLnhv bl9yZWN2ZDogMTczOQpkZXYuZW0uMC5tYWNfc3RhdHMueG9uX3R4ZDogMApkZXYuZW0uMC5t YWNfc3RhdHMueG9mZl9yZWN2ZDogMjkzNApkZXYuZW0uMC5tYWNfc3RhdHMueG9mZl90eGQ6 IDAKZGV2LmVtLjAubWFjX3N0YXRzLnRvdGFsX3BrdHNfcmVjdmQ6IDU0MTU2NzMyMQpkZXYu ZW0uMC5tYWNfc3RhdHMuZ29vZF9wa3RzX3JlY3ZkOiA1NDE1NDQyOTAKZGV2LmVtLjAubWFj X3N0YXRzLmJjYXN0X3BrdHNfcmVjdmQ6IDIxMTMwCmRldi5lbS4wLm1hY19zdGF0cy5tY2Fz dF9wa3RzX3JlY3ZkOiA2MjcyCmRldi5lbS4wLm1hY19zdGF0cy5yeF9mcmFtZXNfNjQ6IDAK ZGV2LmVtLjAubWFjX3N0YXRzLnJ4X2ZyYW1lc182NV8xMjc6IDAKZGV2LmVtLjAubWFjX3N0 YXRzLnJ4X2ZyYW1lc18xMjhfMjU1OiAwCmRldi5lbS4wLm1hY19zdGF0cy5yeF9mcmFtZXNf MjU2XzUxMTogMApkZXYuZW0uMC5tYWNfc3RhdHMucnhfZnJhbWVzXzUxMl8xMDIzOiAwCmRl di5lbS4wLm1hY19zdGF0cy5yeF9mcmFtZXNfMTAyNF8xNTIyOiAwCmRldi5lbS4wLm1hY19z dGF0cy5nb29kX29jdGV0c19yZWN2ZDogNTg5NDMxMDIyMjUKZGV2LmVtLjAubWFjX3N0YXRz Lmdvb2Rfb2N0ZXRzX3R4ZDogMTM1NTExMDgxNDQyMgpkZXYuZW0uMC5tYWNfc3RhdHMudG90 YWxfcGt0c190eGQ6IDEwNDI0OTA1MTIKZGV2LmVtLjAubWFjX3N0YXRzLmdvb2RfcGt0c190 eGQ6IDEwNDI0OTA1MTIKZGV2LmVtLjAubWFjX3N0YXRzLmJjYXN0X3BrdHNfdHhkOiA0NjIz CmRldi5lbS4wLm1hY19zdGF0cy5tY2FzdF9wa3RzX3R4ZDogMjM2MzMKZGV2LmVtLjAubWFj X3N0YXRzLnR4X2ZyYW1lc182NDogMApkZXYuZW0uMC5tYWNfc3RhdHMudHhfZnJhbWVzXzY1 XzEyNzogMApkZXYuZW0uMC5tYWNfc3RhdHMudHhfZnJhbWVzXzEyOF8yNTU6IDAKZGV2LmVt LjAubWFjX3N0YXRzLnR4X2ZyYW1lc18yNTZfNTExOiAwCmRldi5lbS4wLm1hY19zdGF0cy50 eF9mcmFtZXNfNTEyXzEwMjM6IDAKZGV2LmVtLjAubWFjX3N0YXRzLnR4X2ZyYW1lc18xMDI0 XzE1MjI6IDAKZGV2LmVtLjAubWFjX3N0YXRzLnRzb190eGQ6IDMyNjM3MDYyMApkZXYuZW0u MC5tYWNfc3RhdHMudHNvX2N0eF9mYWlsOiAwCmRldi5lbS4wLmludGVycnVwdHMuYXNzZXJ0 czogMzUwMzEwNjg1CmRldi5lbS4wLmludGVycnVwdHMucnhfcGt0X3RpbWVyOiAwCmRldi5l bS4wLmludGVycnVwdHMucnhfYWJzX3RpbWVyOiAwCmRldi5lbS4wLmludGVycnVwdHMudHhf cGt0X3RpbWVyOiAwCmRldi5lbS4wLmludGVycnVwdHMudHhfYWJzX3RpbWVyOiAwCmRldi5l bS4wLmludGVycnVwdHMudHhfcXVldWVfZW1wdHk6IDAKZGV2LmVtLjAuaW50ZXJydXB0cy50 eF9xdWV1ZV9taW5fdGhyZXNoOiAwCmRldi5lbS4wLmludGVycnVwdHMucnhfZGVzY19taW5f dGhyZXNoOiAwCmRldi5lbS4wLmludGVycnVwdHMucnhfb3ZlcnJ1bjogMApkZXYuZW0uMC53 YWtlOiAwCjw8PCBzeXNjdGwgZGV2LmVtLjAKPj4+IHN5c2N0bCBuZXQuaXNyCm5ldC5pc3Iu bnVtdGhyZWFkczogMQpuZXQuaXNyLm1heHByb3Q6IDE2Cm5ldC5pc3IuZGVmYXVsdHFsaW1p dDogMjU2Cm5ldC5pc3IubWF4cWxpbWl0OiAxMDI0MApuZXQuaXNyLmJpbmR0aHJlYWRzOiAw Cm5ldC5pc3IubWF4dGhyZWFkczogMQpuZXQuaXNyLmRpcmVjdDogMQpuZXQuaXNyLmRpcmVj dF9mb3JjZTogMQo8PDwgc3lzY3RsIG5ldC5pc3IK ------------6921DF2E50B683-- From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 23:05:20 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24F51106564A for ; Sun, 6 Feb 2011 23:05:20 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smtp-out0.tiscali.nl (smtp-out0.tiscali.nl [195.241.79.175]) by mx1.freebsd.org (Postfix) with ESMTP id ABD7B8FC0A for ; Sun, 6 Feb 2011 23:05:19 +0000 (UTC) Received: from [212.123.145.58] (helo=sjakie.klop.ws) by smtp-out0.tiscali.nl with esmtp (Exim) (envelope-from ) id 1PmDVP-0003oE-2C; Sun, 06 Feb 2011 23:54:55 +0100 Received: from 212-123-145-58.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id 0FDD8D05C; Sun, 6 Feb 2011 23:54:50 +0100 (CET) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: pyunyh@gmail.com, sthaug@nethelp.no, "Ronald Klop" References: <708793006.292748.1295186099577.JavaMail.root@erie.cs.uoguelph.ca> <20110117005524.GA1305@michelle.cdnetworks.com> <20110118.093804.74673434.sthaug@nethelp.no> Date: Sun, 06 Feb 2011 23:54:49 +0100 MIME-Version: 1.0 From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/11.01 (FreeBSD) Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, rmacklem@uoguelph.ca Subject: Re: bogus 0 len IP packet, was: Hang in VOP_LOCK1_APV on 8-STABLE with NFS. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 23:05:20 -0000 On Sat, 22 Jan 2011 00:01:47 +0100, Ronald Klop =20 wrote: > On Tue, 18 Jan 2011 09:38:04 +0100, wrote: > >>> > So, does anyone have an idea why the IP length field would be set t= o =20 >>> 0 >>> > for these TCP/IP packets? >>> > >>> > Here's some info from Ronald w.r.t. his hardware. (All I can think = =20 >>> of is >>> > that he could try disabling TSO, etc?) >>> > >>> > Thanks in advance for any help with this, rick >>> > >>> >>> It seems that issue came from TSO. Driver will set ip_len and >>> ip_sum field to 0 before passing the TCP segment to controller. >>> The failed length were 4446, 5858, 3034 and 4310 and the total >>> number of such frames are more than 35k within 90 seconds. Since >>> failed length 4310 is continuously repeated I guess there is edge >>> case where em(4) didn't free failed TCP segment for TSO. >>> I remember there was commit to HEAD(r217295) which could be related >>> with this issue. >> >> I'm seeing the same problem with Broadcom NetXtreme (bce) cards: >> >> bce0@pci0:3:0:0: class=3D0x020000 card=3D0x03421014 chip=3D0x16= 4c14e4 =20 >> rev=3D0x12 hdr=3D0x00 >> vendor =3D 'Broadcom Corporation' >> device =3D 'Broadcom NetXtreme II Gigabit Ethernet Adapter =20 >> (BCM5708)' >> class =3D network >> subclass =3D ethernet >> >> This is with 8.2-PRERELEASE. Turning off TSO (ifconfig bce0 -tso) >> removes the problem. >> >> Steinar Haug, Nethelp consulting, sthaug@nethelp.no >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > I tried -tso and -txcsum in various combinations, but it didn't solve =20 > the problem. I wil look for another brand of network card to try. But =20 > this has to wait till monday when I'm at the office again. I also used another network card (rl0) and it has the same problem with =20 NFS. I'm going to change some network cables to see if that helps. I have= =20 some hints that there might be something wrong with that. Ronald. From owner-freebsd-net@FreeBSD.ORG Sun Feb 6 23:54:04 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C6AC01065670; Sun, 6 Feb 2011 23:54:04 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 637C18FC15; Sun, 6 Feb 2011 23:54:04 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.4/8.14.4) with ESMTP id p16Ns060002096 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 6 Feb 2011 18:54:01 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4D4F3497.6050505@sentex.net> Date: Sun, 06 Feb 2011 18:53:59 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jack Vogel References: <1290533941.3173.50.camel@home-yahoo> <4CEC0548.1080801@sentex.net> <4D2C636B.5040003@sentex.net> <4D3C4795.40205@sentex.net> <4D42EA74.4090807@sentex.net> <1296590190.2326.6.camel@hitfishpass-lx.corp.yahoo.com> <1296591565.2326.7.camel@hitfishpass-lx.corp.yahoo.com> <1296597827.2326.12.camel@hitfishpass-lx.corp.yahoo.com> <4D48C973.7080503@sentex.net> <4D49A26B.5050803@sentex.net> <1296842996.2233.0.camel@hitfishpass-lx.corp.yahoo.com> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on IPv6:2607:f3e0:0:1::12 Cc: "freebsd-net@freebsd.org" , "freebsd-hardware@freebsd.org" , Sean Bruno , Jan Koum , Ivan Voras Subject: Re: em driver, 82574L chip, and possibly ASPM X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 23:54:05 -0000 So far so good. I would often get a hang on the level zero dumps to my backup server Sunday AM, and it made it through! So a good sign, but not a definitive sign. I have a PCIe em card that has this chipset as well and was showing the same sort of problem in a customer's RELENG_7 box. I will see if I can get the customer to try the card in their box with the patch for RELENG_7 as it would show this issue at least once a day until I pulled the card for an older version ---Mike On 2/4/2011 1:12 PM, Jack Vogel wrote: > Was curious too, but being more patient than you :) > > Jack > > > On Fri, Feb 4, 2011 at 10:09 AM, Sean Bruno wrote: > >> Any more data on this problem or do we have to wait a while? >> >> Sean >> >> >> On Wed, 2011-02-02 at 10:28 -0800, Mike Tancsa wrote: >>> On 2/2/2011 12:37 PM, Jack Vogel wrote: >>>> So has everyone that wanted to get something testing been able to do >> so? >>> >>> I have been testing in the back and will deploy to my production box >>> this afternoon. As I am not able to reproduce it easily, it will be a >>> bit before I can say the issue is gone. Jan however, was able to >>> trigger it with greater ease ? >>> >>> ---Mike >>> >>>> >>>> Jack >>>> >>>> >>>> On Tue, Feb 1, 2011 at 7:03 PM, Mike Tancsa wrote: >>>> >>>>> On 2/1/2011 5:03 PM, Sean Bruno wrote: >>>>>> On Tue, 2011-02-01 at 13:43 -0800, Jack Vogel wrote: >>>>>>> To those who are going to test, here is the if_em.c, based on head, >>>>>>> with my >>>>>>> changes, I have to leave for the afternoon, and have not had a >> chance >>>>>>> to build >>>>>>> this, but it should work. I will check back in the later evening. >>>>>>> >>>>>>> Any blatant problems Sean, feel free to fix them :) >>>>>>> >>>>>>> Jack >>>>>>> >>>>>> >>>>>> >>>>>> I suspect that line 1490 should be: >>>>>> if (more_rx || (ifp->if_drv_flags & IFF_DRV_OACTIVE)) { >>>>>> >>>>> >>>>> >>>>> I have hacked up a RELENG_8 version which I think is correct including >>>>> the above change >>>>> >>>>> http://www.tancsa.com/if_em-8.c >>>>> >>>>> >>>>> >>>>> --- if_em.c.orig 2011-02-01 21:47:14.000000000 -0500 >>>>> +++ if_em.c 2011-02-01 21:47:19.000000000 -0500 >>>>> @@ -30,7 +30,7 @@ >>>>> POSSIBILITY OF SUCH DAMAGE. >>>>> >>>>> >>>>> >> ******************************************************************************/ >>>>> -/*$FreeBSD: src/sys/dev/e1000/if_em.c,v 1.21.2.20 2011/01/22 01:37:53 >>>>> jfv Exp $*/ >>>>> +/*$FreeBSD$*/ >>>>> >>>>> #ifdef HAVE_KERNEL_OPTION_HEADERS >>>>> #include "opt_device_polling.h" >>>>> @@ -93,7 +93,7 @@ >>>>> >> /********************************************************************* >>>>> * Driver version: >>>>> >> *********************************************************************/ >>>>> -char em_driver_version[] = "7.1.9"; >>>>> +char em_driver_version[] = "7.1.9-test"; >>>>> >>>>> >> /********************************************************************* >>>>> * PCI Device ID Table >>>>> @@ -927,11 +927,10 @@ >>>>> if (!adapter->link_active) >>>>> return; >>>>> >>>>> - /* Call cleanup if number of TX descriptors low */ >>>>> - if (txr->tx_avail <= EM_TX_CLEANUP_THRESHOLD) >>>>> - em_txeof(txr); >>>>> - >>>>> while (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) { >>>>> + /* First cleanup if TX descriptors low */ >>>>> + if (txr->tx_avail <= EM_TX_CLEANUP_THRESHOLD) >>>>> + em_txeof(txr); >>>>> if (txr->tx_avail < EM_MAX_SCATTER) { >>>>> ifp->if_drv_flags |= IFF_DRV_OACTIVE; >>>>> break; >>>>> @@ -1411,8 +1410,7 @@ >>>>> if (!drbr_empty(ifp, txr->br)) >>>>> em_mq_start_locked(ifp, txr, NULL); >>>>> #else >>>>> - if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) >>>>> - em_start_locked(ifp, txr); >>>>> + em_start_locked(ifp, txr); >>>>> #endif >>>>> EM_TX_UNLOCK(txr); >>>>> >>>>> @@ -1475,11 +1473,10 @@ >>>>> struct ifnet *ifp = adapter->ifp; >>>>> struct tx_ring *txr = adapter->tx_rings; >>>>> struct rx_ring *rxr = adapter->rx_rings; >>>>> - bool more; >>>>> - >>>>> >>>>> if (ifp->if_drv_flags & IFF_DRV_RUNNING) { >>>>> - more = em_rxeof(rxr, adapter->rx_process_limit, NULL); >>>>> + bool more_rx; >>>>> + more_rx = em_rxeof(rxr, adapter->rx_process_limit, >> NULL); >>>>> >>>>> EM_TX_LOCK(txr); >>>>> em_txeof(txr); >>>>> @@ -1487,12 +1484,10 @@ >>>>> if (!drbr_empty(ifp, txr->br)) >>>>> em_mq_start_locked(ifp, txr, NULL); >>>>> #else >>>>> - if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) >>>>> - em_start_locked(ifp, txr); >>>>> + em_start_locked(ifp, txr); >>>>> #endif >>>>> - em_txeof(txr); >>>>> EM_TX_UNLOCK(txr); >>>>> - if (more) { >>>>> + if (more_rx || (ifp->if_drv_flags & IFF_DRV_OACTIVE)) >> { >>>>> taskqueue_enqueue(adapter->tq, >> &adapter->que_task); >>>>> return; >>>>> } >>>>> @@ -1604,7 +1599,6 @@ >>>>> if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) >>>>> em_start_locked(ifp, txr); >>>>> #endif >>>>> - em_txeof(txr); >>>>> E1000_WRITE_REG(&adapter->hw, E1000_IMS, txr->ims); >>>>> EM_TX_UNLOCK(txr); >>>>> } >>>>> @@ -3730,17 +3724,17 @@ >>>>> txr->queue_status = EM_QUEUE_HUNG; >>>>> >>>>> /* >>>>> - * If we have enough room, clear IFF_DRV_OACTIVE >>>>> + * If we have a minimum free, clear IFF_DRV_OACTIVE >>>>> * to tell the stack that it is OK to send packets. >>>>> */ >>>>> - if (txr->tx_avail > EM_TX_CLEANUP_THRESHOLD) { >>>>> + if (txr->tx_avail > EM_MAX_SCATTER) >>>>> ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; >>>>> - /* Disable watchdog if all clean */ >>>>> - if (txr->tx_avail == adapter->num_tx_desc) { >>>>> - txr->queue_status = EM_QUEUE_IDLE; >>>>> - return (FALSE); >>>>> - } >>>>> - } >>>>> + >>>>> + /* Disable watchdog if all clean */ >>>>> + if (txr->tx_avail == adapter->num_tx_desc) { >>>>> + txr->queue_status = EM_QUEUE_IDLE; >>>>> + return (FALSE); >>>>> + } >>>>> >>>>> return (TRUE); >>>>> } >>>>> @@ -5064,8 +5058,8 @@ >>>>> char namebuf[QUEUE_NAME_LEN]; >>>>> >>>>> /* Driver Statistics */ >>>>> - SYSCTL_ADD_UINT(ctx, child, OID_AUTO, "link_irq", >>>>> - CTLFLAG_RD, &adapter->link_irq, 0, >>>>> + SYSCTL_ADD_UINT(ctx, child, OID_AUTO, "link_irq", >>>>> + CTLFLAG_RD, &adapter->link_irq,0, >>>>> "Link MSIX IRQ Handled"); >>>>> SYSCTL_ADD_ULONG(ctx, child, OID_AUTO, "mbuf_alloc_fail", >>>>> CTLFLAG_RD, &adapter->mbuf_alloc_failed, >>>>> @@ -5108,11 +5102,13 @@ >>>>> queue_list = SYSCTL_CHILDREN(queue_node); >>>>> >>>>> SYSCTL_ADD_PROC(ctx, queue_list, OID_AUTO, "txd_head", >>>>> - CTLFLAG_RD, adapter, >> E1000_TDH(txr->me), >>>>> + CTLFLAG_RD, adapter, >>>>> + E1000_TDH(txr->me), >>>>> em_sysctl_reg_handler, "IU", >>>>> "Transmit Descriptor Head"); >>>>> SYSCTL_ADD_PROC(ctx, queue_list, OID_AUTO, "txd_tail", >>>>> - CTLFLAG_RD, adapter, >> E1000_TDT(txr->me), >>>>> + CTLFLAG_RD, adapter, >>>>> + E1000_TDT(txr->me), >>>>> em_sysctl_reg_handler, "IU", >>>>> "Transmit Descriptor Tail"); >>>>> SYSCTL_ADD_ULONG(ctx, queue_list, OID_AUTO, "tx_irq", >>>>> @@ -5123,11 +5119,13 @@ >>>>> "Queue No Descriptor Available"); >>>>> >>>>> SYSCTL_ADD_PROC(ctx, queue_list, OID_AUTO, "rxd_head", >>>>> - CTLFLAG_RD, adapter, >> E1000_RDH(rxr->me), >>>>> + CTLFLAG_RD, adapter, >>>>> + E1000_RDH(rxr->me), >>>>> em_sysctl_reg_handler, "IU", >>>>> "Receive Descriptor Head"); >>>>> SYSCTL_ADD_PROC(ctx, queue_list, OID_AUTO, "rxd_tail", >>>>> - CTLFLAG_RD, adapter, >> E1000_RDT(rxr->me), >>>>> + CTLFLAG_RD, adapter, >>>>> + E1000_RDT(rxr->me), >>>>> em_sysctl_reg_handler, "IU", >>>>> "Receive Descriptor Tail"); >>>>> SYSCTL_ADD_ULONG(ctx, queue_list, OID_AUTO, "rx_irq", >>>>> @@ -5141,19 +5139,19 @@ >>>>> CTLFLAG_RD, NULL, "Statistics"); >>>>> stat_list = SYSCTL_CHILDREN(stat_node); >>>>> >>>>> - SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "excess_coll", >>>>> + SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "excess_coll", >>>>> CTLFLAG_RD, &stats->ecol, >>>>> "Excessive collisions"); >>>>> - SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "single_coll", >>>>> + SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "single_coll", >>>>> CTLFLAG_RD, &stats->scc, >>>>> "Single collisions"); >>>>> - SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "multiple_coll", >>>>> + SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "multiple_coll", >>>>> CTLFLAG_RD, &stats->mcc, >>>>> "Multiple collisions"); >>>>> - SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "late_coll", >>>>> + SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "late_coll", >>>>> CTLFLAG_RD, &stats->latecol, >>>>> "Late collisions"); >>>>> - SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "collision_count", >>>>> + SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "collision_count", >>>>> CTLFLAG_RD, &stats->colc, >>>>> "Collision Count"); >>>>> SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "symbol_errors", >>>>> @@ -5240,12 +5238,12 @@ >>>>> SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, >> "rx_frames_1024_1522", >>>>> CTLFLAG_RD, &adapter->stats.prc1522, >>>>> "1023-1522 byte frames received"); >>>>> - SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "good_octets_recvd", >>>>> + SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "good_octets_recvd", >>>>> CTLFLAG_RD, &adapter->stats.gorc, >>>>> "Good Octets Received"); >>>>> >>>>> /* Packet Transmission Stats */ >>>>> - SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "good_octets_txd", >>>>> + SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "good_octets_txd", >>>>> CTLFLAG_RD, &adapter->stats.gotc, >>>>> "Good Octets Transmitted"); >>>>> SYSCTL_ADD_QUAD(ctx, stat_list, OID_AUTO, "total_pkts_txd", >>>>> >>>>> -- >>>>> ------------------- >>>>> Mike Tancsa, tel +1 519 651 3400 >>>>> Sentex Communications, mike@sentex.net >>>>> Providing Internet services since 1994 www.sentex.net >>>>> Cambridge, Ontario Canada http://www.tancsa.com/ >>>>> >>>> >>> >>> >>> -- >>> ------------------- >>> Mike Tancsa, tel +1 519 651 3400 >>> Sentex Communications, mike@sentex.net >>> Providing Internet services since 1994 www.sentex.net >>> Cambridge, Ontario Canada http://www.tancsa.com/ >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> >> > -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 00:22:34 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A95F106564A for ; Mon, 7 Feb 2011 00:22:34 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id CC5038FC13 for ; Mon, 7 Feb 2011 00:22:33 +0000 (UTC) Received: by iwn39 with SMTP id 39so4106710iwn.13 for ; Sun, 06 Feb 2011 16:22:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=9TxMC8bv32U1GPnJcfZjEOeDc4t3RxMzYKrFnZS+wxM=; b=ElE+fg0R1w1l2nD8axTADesmk55ZuZFhcLCHbgSLpDQ0mjOsJ6C6MQQk2CrGp6rSsF TkYDGAH7JCXaoM11opwB93Rw5OuRHX9VaZd9huvMR8lniCJgcaU4pCIF3R2OLrcey2Bc KB7/8E6b8zT34x2w/KQth9qJBlmloRqUNz2fU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=mbPKjwUNvvQ4Zdr2KprKK1ENyQHC7wLdTBr92ZW20ZajrunrSvY6F/WSsNte2HozXP T62bvsRoC0XSY7s8Mn6qbObvOQI30ZvfitsnBzko6rwplmzrYRVuA0CiK+FKNXzzMc1w Mjz1zM1lQ9RqPreg08yiFqGta2n1S/PplPR6Q= Received: by 10.42.213.136 with SMTP id gw8mr17696725icb.359.1297038153231; Sun, 06 Feb 2011 16:22:33 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id d21sm3240369ibg.21.2011.02.06.16.22.30 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 06 Feb 2011 16:22:31 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Sun, 6 Feb 2011 16:22:36 -0800 From: Pyun YongHyeon Date: Sun, 6 Feb 2011 16:22:36 -0800 To: Ronald Klop Message-ID: <20110207002235.GA1244@michelle.cdnetworks.com> References: <708793006.292748.1295186099577.JavaMail.root@erie.cs.uoguelph.ca> <20110117005524.GA1305@michelle.cdnetworks.com> <20110118.093804.74673434.sthaug@nethelp.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, rmacklem@uoguelph.ca, sthaug@nethelp.no Subject: Re: bogus 0 len IP packet, was: Hang in VOP_LOCK1_APV on 8-STABLE with NFS. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 00:22:34 -0000 On Sun, Feb 06, 2011 at 11:54:49PM +0100, Ronald Klop wrote: > On Sat, 22 Jan 2011 00:01:47 +0100, Ronald Klop > wrote: > > >On Tue, 18 Jan 2011 09:38:04 +0100, wrote: > > > >>>> So, does anyone have an idea why the IP length field would be set to > >>>0 > >>>> for these TCP/IP packets? > >>>> > >>>> Here's some info from Ronald w.r.t. his hardware. (All I can think > >>>of is > >>>> that he could try disabling TSO, etc?) > >>>> > >>>> Thanks in advance for any help with this, rick > >>>> > >>> > >>>It seems that issue came from TSO. Driver will set ip_len and > >>>ip_sum field to 0 before passing the TCP segment to controller. > >>>The failed length were 4446, 5858, 3034 and 4310 and the total > >>>number of such frames are more than 35k within 90 seconds. Since > >>>failed length 4310 is continuously repeated I guess there is edge > >>>case where em(4) didn't free failed TCP segment for TSO. > >>>I remember there was commit to HEAD(r217295) which could be related > >>>with this issue. > >> > >>I'm seeing the same problem with Broadcom NetXtreme (bce) cards: > >> > >>bce0@pci0:3:0:0: class=0x020000 card=0x03421014 chip=0x164c14e4 > >>rev=0x12 hdr=0x00 > >> vendor = 'Broadcom Corporation' > >> device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter > >>(BCM5708)' > >> class = network > >> subclass = ethernet > >> > >>This is with 8.2-PRERELEASE. Turning off TSO (ifconfig bce0 -tso) > >>removes the problem. > >> > >>Steinar Haug, Nethelp consulting, sthaug@nethelp.no > >>_______________________________________________ > >>freebsd-net@freebsd.org mailing list > >>http://lists.freebsd.org/mailman/listinfo/freebsd-net > >>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > >I tried -tso and -txcsum in various combinations, but it didn't solve > >the problem. I wil look for another brand of network card to try. But > >this has to wait till monday when I'm at the office again. > > I also used another network card (rl0) and it has the same problem with > NFS. I'm going to change some network cables to see if that helps. I have > some hints that there might be something wrong with that. > Hmm, given that rl(4) also shows the issue it seems the issue could be in TCP/IP stack, not in driver side. rl(4) is dumb device so network stack should do segmentation and checksum computation. I highly doubt the issue came from faulty cable since other users also reported the same issue. Unfortunately I have no clue yet and I was not able to reproduce it on my box. I vaguely guess some code in kernel changed the ip_len to 0 in the middle of transmission. Rick's captured traffic looks normal except 0 ip_len given that controller is computing checksum on the fly. If mbuf chain was corrupted(e.g. m_len == 0) driver would have failed to send those frames. > Ronald. From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 03:36:13 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBBBC1065670; Mon, 7 Feb 2011 03:36:13 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C1F6B8FC12; Mon, 7 Feb 2011 03:36:13 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p173aDW9011736; Mon, 7 Feb 2011 03:36:13 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p173aDYO011732; Mon, 7 Feb 2011 03:36:13 GMT (envelope-from linimon) Date: Mon, 7 Feb 2011 03:36:13 GMT Message-Id: <201102070336.p173aDYO011732@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/154557: [tcp] Freeze tcp-session of the clients, if in the gateway is used quagga X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 03:36:14 -0000 Old Synopsis: Freeze tcp-session of the clients, if in the gateway is used quagga New Synopsis: [tcp] Freeze tcp-session of the clients, if in the gateway is used quagga Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Mon Feb 7 03:35:39 UTC 2011 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=154557 From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 06:10:05 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 197E1106564A; Mon, 7 Feb 2011 06:10:05 +0000 (UTC) (envelope-from alex@zagrebin.ru) Received: from mail.zagrebin.ru (gw.zagrebin.ru [91.215.205.128]) by mx1.freebsd.org (Postfix) with ESMTP id B31008FC16; Mon, 7 Feb 2011 06:10:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zagrebin.ru; s=mail; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=hGiyVh4R3Vpbp/sLMLnGsa94Oqbk9MPAFAT6QOEDDDY=; b=sglOjb6rLI3Q+B3RZVSAB0ufECEQko8ClC9KtdjbqDWgCHoOZ+gw8P3bUtnKYccW9xo0ll8+A9l0V0/+9f2zUmfXzF/DnQtX/j9+MUFfTepEby7Zn0TEIiQuQDV5wvMIJUA83QfiEsI8bKPQtPcny7tkJEISjn3ehW5n1PChKoM=; Received: from alex by mail.zagrebin.ru with local (Exim 4.74 (FreeBSD)) (envelope-from ) id 1PmKIU-000Fgt-0l; Mon, 07 Feb 2011 09:10:02 +0300 Date: Mon, 7 Feb 2011 09:10:01 +0300 From: Alexander Zagrebin To: Bernhard Schmidt Message-ID: <20110207061001.GB59910@gw.zagrebin.ru> References: <20110204060808.GA97298@gw.zagrebin.ru> <201102040951.34201.bschmidt@freebsd.org> <20110204223339.GA12555@gw.zagrebin.ru> <201102061132.41032.bschmidt@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201102061132.41032.bschmidt@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 06:10:05 -0000 Hi! On 06.02.2011 11:32:40 +0100, Bernhard Schmidt wrote: > > I had the doubts about ability to change or hide/unhide the SSID, but > > it works too. It seems that after `ifconfig wlan0 ssid ...` or > > `ifconfig wlan0 (hidessid|-hidessid)` the following occurs: > > 1. run_newstate() is called > > 2. run_newstate() invokes run_update_beacon_cb() > > 3. run_update_beacon_cb() invokes ieee80211_beacon_update() > > But I couldn't understand where ieee80211_beacon_update() can change > > a SSID... > > As I see it, it can't. Only a call to ieee80211_beacon_construct() would > do that. Can I talk you into figuring out what's going on here? Getting > a definitive statement about SSID changes are correctly handled (by > means of a trace) would help alot. Yes, sure! What kind of information you need? -- Alexander Zagrebin From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 07:14:32 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E254106564A; Mon, 7 Feb 2011 07:14:32 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) by mx1.freebsd.org (Postfix) with ESMTP id 8081A8FC15; Mon, 7 Feb 2011 07:14:30 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id p177ERFr031499; Mon, 7 Feb 2011 13:14:28 +0600 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4D4F9BCE.2070309@rdtc.ru> Date: Mon, 07 Feb 2011 13:14:22 +0600 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: lev@freebsd.org References: <787079038.20110207004023@serebryakov.spb.ru> In-Reply-To: <787079038.20110207004023@serebryakov.spb.ru> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: em0 hangs without any messages like "Watchdog timeout", only down/up reset it. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 07:14:32 -0000 On 07.02.2011 03:40, Lev Serebryakov wrote: > Hello, Freebsd-stable. > > > My em0 (the same, copy'n'paste hardware info from previous > mnessage): > > em0@pci0:0:25:0: class=0x020000 card=0x82681043 chip=0x10bd8086 rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82566DM Gigabit Ethernet Adapter (82566DM)' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xfea40000, size 131072, enabled > bar [14] = type Memory, range 32, base 0xfea79000, size 4096, enabled > bar [18] = type I/O Port, range 32, base 0xdc00, size 32, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 09[e0] = vendor (length 6) Intel cap 2 version 0 > > It is on-board LAN on Q35-based MoBo (ASUS P5E-VM DO) > > It hangs under load without any output. When it works with POLLING, it > prints "Watchdog timeout" and reset automatically several times a day, > but without POLLING it simply hangs with same frequency. It is > 8.2-PRERELEASE (from RELENG_8), cvsupped AFTER 22 Jan (last commit to > e1000 drivers family). > > Locally system works, but mbufs are overfilled. "ifconfig em0 down > && ifconfig em0 up" solves problem. > > Output of different diagnostic tools (vmstat, netstat, ifconfig, > sysctl of dev.em.0 tree, top -S) are attached in one file. > > Early (about half year ago) this sytem works without any problems > with net. Have you read thread "em driver, 82574L chip, and possibly ASPM" in net@ ? From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 08:11:17 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8EA31065672 for ; Mon, 7 Feb 2011 08:11:17 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7418D8FC14 for ; Mon, 7 Feb 2011 08:11:17 +0000 (UTC) Received: by fxm16 with SMTP id 16so4733254fxm.13 for ; Mon, 07 Feb 2011 00:11:16 -0800 (PST) Received: by 10.223.93.140 with SMTP id v12mr9834753fam.96.1297066276425; Mon, 07 Feb 2011 00:11:16 -0800 (PST) Received: from jessie.localnet (p5B2ECF7C.dip0.t-ipconnect.de [91.46.207.124]) by mx.google.com with ESMTPS id n3sm1039493fax.31.2011.02.07.00.11.14 (version=SSLv3 cipher=RC4-MD5); Mon, 07 Feb 2011 00:11:15 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: Alexander Zagrebin Date: Mon, 7 Feb 2011 09:11:02 +0100 User-Agent: KMail/1.13.5 (Linux/2.6.32-28-generic; KDE/4.4.5; i686; ; ) References: <20110204060808.GA97298@gw.zagrebin.ru> <201102061132.41032.bschmidt@freebsd.org> <20110207061001.GB59910@gw.zagrebin.ru> In-Reply-To: <20110207061001.GB59910@gw.zagrebin.ru> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201102070911.02791.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bschmidt@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 08:11:17 -0000 On Monday, February 07, 2011 07:10:01 Alexander Zagrebin wrote: > Hi! > > On 06.02.2011 11:32:40 +0100, Bernhard Schmidt wrote: > > > I had the doubts about ability to change or hide/unhide the SSID, > > > but it works too. It seems that after `ifconfig wlan0 ssid ...` > > > or `ifconfig wlan0 (hidessid|-hidessid)` the following occurs: > > > 1. run_newstate() is called > > > 2. run_newstate() invokes run_update_beacon_cb() > > > 3. run_update_beacon_cb() invokes ieee80211_beacon_update() > > > But I couldn't understand where ieee80211_beacon_update() can > > > change a SSID... > > > > As I see it, it can't. Only a call to ieee80211_beacon_construct() > > would do that. Can I talk you into figuring out what's going on > > here? Getting a definitive statement about SSID changes are > > correctly handled (by means of a trace) would help alot. > > Yes, sure! > What kind of information you need? For example, if you call 'ifconfig wlan0 ssid ' the new ssid is passed over using a IOCTL. It would be interesting to know what function in net80211 are called regarding beacon updates and which of those call into the run driver. Ultimately it's about figuring out if special handling for such cases are required and if so, how to do it. Thanks -- Bernhard From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 08:15:49 2011 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A759E106564A for ; Mon, 7 Feb 2011 08:15:49 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: from mail.sub.ru (mail.sub.ru [88.212.205.2]) by mx1.freebsd.org (Postfix) with SMTP id CD6E88FC12 for ; Mon, 7 Feb 2011 08:15:48 +0000 (UTC) Received: (qmail 14274 invoked from network); 7 Feb 2011 10:49:19 +0300 Received: from 83-69-208-138.in-addr.mastertelecom.ru (83-69-208-138.in-addr.mastertelecom.ru [83.69.208.138]) by mail.sub.ru ([88.212.205.2]) with ESMTP via TCP; 07 Feb 2011 07:49:19 -0000 Message-ID: <4D4FA3DA.7010004@webmail.sub.ru> Date: Mon, 07 Feb 2011 10:48:42 +0300 From: Alex Povolotsky User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.12) Gecko/20101103 Thunderbird/3.1.6 MIME-Version: 1.0 To: freebsd-net@FreeBSD.org Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: jail source address selection doesn't work? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 08:15:49 -0000 Hello! On a multihomed FreeBSD 8.1-RELEASE, in a multihomed jail, source IP selection suddenly refused to work. ifconfig on a box: bce0: flags=8943 metric 0 mtu 1500 options=c01bb ether 00:1a:64:c5:d0:c8 inet 192.168.80.40 netmask 0xffffff00 broadcast 192.168.80.255 media: Ethernet autoselect (100baseTX ) status: active bce1: flags=8943 metric 0 mtu 1500 options=c01bb ether 00:1a:64:c5:d0:ca media: Ethernet autoselect (100baseTX ) status: active lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 inet 127.0.0.2 netmask 0xff000000 nd6 options=3 vlan75: flags=8943 metric 0 mtu 1500 options=103 ether 00:1a:64:c5:d0:ca inet 192.168.75.4 netmask 0xffffff00 broadcast 192.168.75.255 media: Ethernet autoselect (100baseTX ) status: active vlan: 75 parent interface: bce1 vlan82: flags=8943 metric 0 mtu 1500 options=103 ether 00:1a:64:c5:d0:ca inet 192.168.82.2 netmask 0xffffff00 broadcast 192.168.82.255 media: Ethernet autoselect (100baseTX ) status: active vlan: 82 parent interface: bce1 vlan83: flags=8943 metric 0 mtu 1500 options=103 ether 00:1a:64:c5:d0:ca inet 83.69.203.3 netmask 0xfffffff0 broadcast 83.69.203.15 media: Ethernet autoselect (100baseTX ) status: active vlan: 83 parent interface: bce1 vlan63: flags=8943 metric 0 mtu 1500 options=103 ether 00:1a:64:c5:d0:ca inet 10.19.63.100 netmask 0xffffff00 broadcast 10.19.63.255 media: Ethernet autoselect (100baseTX ) status: active vlan: 63 parent interface: bce1 carp0: flags=49 metric 0 mtu 1500 inet 192.168.80.42 netmask 0xffffff00 carp: MASTER vhid 145 advbase 1 advskew 0 carp1: flags=49 metric 0 mtu 1500 inet 192.168.75.3 netmask 0xffffff00 carp: MASTER vhid 146 advbase 1 advskew 0 carp2: flags=49 metric 0 mtu 1500 inet 192.168.82.4 netmask 0xffffff00 carp: MASTER vhid 147 advbase 1 advskew 0 carp3: flags=49 metric 0 mtu 1500 inet 83.69.203.1 netmask 0xfffffff0 carp: MASTER vhid 148 advbase 1 advskew 0 carp4: flags=49 metric 0 mtu 1500 inet 10.19.63.67 netmask 0xffffff00 carp: MASTER vhid 149 advbase 1 advskew 0 ifconfig in a jail bce0: flags=8943 metric 0 mtu 1500 options=c01bb ether 00:1a:64:c5:d0:c8 media: Ethernet autoselect (100baseTX ) status: active bce1: flags=8943 metric 0 mtu 1500 options=c01bb ether 00:1a:64:c5:d0:ca media: Ethernet autoselect (100baseTX ) status: active lo0: flags=8049 metric 0 mtu 16384 options=3 vlan75: flags=8943 metric 0 mtu 1500 options=103 ether 00:1a:64:c5:d0:ca inet 192.168.75.4 netmask 0xffffff00 broadcast 192.168.75.255 media: Ethernet autoselect (100baseTX ) status: active vlan: 75 parent interface: bce1 vlan82: flags=8943 metric 0 mtu 1500 options=103 ether 00:1a:64:c5:d0:ca media: Ethernet autoselect (100baseTX ) status: active vlan: 82 parent interface: bce1 vlan83: flags=8943 metric 0 mtu 1500 options=103 ether 00:1a:64:c5:d0:ca media: Ethernet autoselect (100baseTX ) status: active vlan: 83 parent interface: bce1 vlan63: flags=8943 metric 0 mtu 1500 options=103 ether 00:1a:64:c5:d0:ca media: Ethernet autoselect (100baseTX ) status: active vlan: 63 parent interface: bce1 carp0: flags=49 metric 0 mtu 1500 inet 192.168.80.42 netmask 0xffffff00 carp: MASTER vhid 145 advbase 1 advskew 0 carp1: flags=49 metric 0 mtu 1500 carp: MASTER vhid 146 advbase 1 advskew 0 carp2: flags=49 metric 0 mtu 1500 carp: MASTER vhid 147 advbase 1 advskew 0 carp3: flags=49 metric 0 mtu 1500 inet 83.69.203.1 netmask 0xfffffff0 carp: MASTER vhid 148 advbase 1 advskew 0 carp4: flags=49 metric 0 mtu 1500 inet 10.19.63.67 netmask 0xffffff00 carp: MASTER vhid 149 advbase 1 advskew 0 routing table: Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 83.69.203.14 UGS 232 1221991 vlan83 10.0.0.0/8 10.19.63.126 UGS 0 8768 vlan63 10.19.63.0/24 link#7 U 185 613762 vlan63 10.19.63.67 link#12 UH 0 0 carp4 10.19.63.100 link#7 UHS 0 244 lo0 83.69.203.0/28 link#6 U 4 38198 vlan83 83.69.203.1 link#11 UH 0 1876305 carp3 83.69.203.3 link#6 UHS 0 154 lo0 127.0.0.1 link#3 UH 0 1078596 lo0 127.0.0.2 link#3 UH 0 18 lo0 172.16.0.0/12 10.19.63.126 UGS 0 0 vlan63 192.168.0.0/16 10.19.63.126 UGS 8 205694 vlan63 192.168.75.0/24 link#4 U 49 1222391 vlan75 192.168.75.3 link#9 UH 0 0 carp1 192.168.75.4 link#4 UHS 0 2 lo0 192.168.80.0/24 link#1 U 6 618586 bce0 192.168.80.40 link#1 UHS 0 130620 lo0 192.168.80.42 link#8 UH 0 95987 carp0 192.168.82.0/24 link#5 U 2 2361 vlan82 192.168.82.2 link#5 UHS 0 0 lo0 192.168.82.4 link#10 UH 0 0 carp2 Seems reasonable, yes? Pinging from the box # ping 192.168.75.59 PING 192.168.75.59 (192.168.75.59): 56 data bytes 64 bytes from 192.168.75.59: icmp_seq=0 ttl=64 time=0.993 ms 64 bytes from 192.168.75.59: icmp_seq=1 ttl=64 time=0.986 ms 64 bytes from 192.168.75.59: icmp_seq=2 ttl=64 time=0.988 ms ^C --- 192.168.75.59 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.986/0.989/0.993/0.003 ms 10:45:31.425232 IP 192.168.75.4 > 192.168.75.59: ICMP echo request, id 12430, seq 0, length 64 10:45:31.426283 IP 192.168.75.59 > 192.168.75.4: ICMP echo reply, id 12430, seq 0, length 64 10:45:32.425415 IP 192.168.75.4 > 192.168.75.59: ICMP echo request, id 12430, seq 1, length 64 10:45:32.426404 IP 192.168.75.59 > 192.168.75.4: ICMP echo reply, id 12430, seq 1, length 64 Okay, yes? From jail: # ping 192.168.75.59 PING 192.168.75.59 (192.168.75.59): 56 data bytes ^C --- 192.168.75.59 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss 10:45:52.146600 IP 83.69.203.1 > 192.168.75.59: ICMP echo request, id 14222, seq 0, length 64 10:45:53.146702 IP 83.69.203.1 > 192.168.75.59: ICMP echo request, id 14222, seq 1, length 64 Setting ip.saddrsel to 1 or 0 did not change anything. Kernel is GENERIC+ALTQ What could I miss?... Alex. From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 09:01:11 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8581106566C; Mon, 7 Feb 2011 09:01:11 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7F21A8FC08; Mon, 7 Feb 2011 09:01:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1791BdJ090949; Mon, 7 Feb 2011 09:01:11 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1791BFv090930; Mon, 7 Feb 2011 09:01:11 GMT (envelope-from linimon) Date: Mon, 7 Feb 2011 09:01:11 GMT Message-Id: <201102070901.p1791BFv090930@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/154567: [ath] ath(4) lot of bad series(0) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 09:01:11 -0000 Old Synopsis: ath(4) lot of bad series(0) New Synopsis: [ath] ath(4) lot of bad series(0) Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Mon Feb 7 09:00:56 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=154567 From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 09:16:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA344106564A for ; Mon, 7 Feb 2011 09:16:21 +0000 (UTC) (envelope-from lev@serebryakov.spb.ru) Received: from ftp.translate.ru (ftp.translate.ru [80.249.188.42]) by mx1.freebsd.org (Postfix) with ESMTP id A8D008FC1A for ; Mon, 7 Feb 2011 09:16:21 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (89.112.15.178.pppoe.eltel.net [89.112.15.178]) (Authenticated sender: lev@serebryakov.spb.ru) by ftp.translate.ru (Postfix) with ESMTPA id 487ED13DF42; Mon, 7 Feb 2011 12:16:20 +0300 (MSK) Date: Mon, 7 Feb 2011 12:16:19 +0300 From: Lev Serebryakov X-Priority: 3 (Normal) Message-ID: <242583358.20110207121619@serebryakov.spb.ru> To: Eugene Grosbein In-Reply-To: <4D4F9BCE.2070309@rdtc.ru> References: <787079038.20110207004023@serebryakov.spb.ru> <4D4F9BCE.2070309@rdtc.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: em0 hangs without any messages like "Watchdog timeout", only down/up reset it. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 09:16:22 -0000 Hello, Eugene. You wrote 7 =F4=E5=E2=F0=E0=EB=FF 2011 =E3., 10:14:22: > Have you read thread "em driver, 82574L chip, and possibly ASPM" in net@ ? Partially. Now I return to it and found some patch to if_em.c, I'll give it a try. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 10:27:06 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E25811065673; Mon, 7 Feb 2011 10:27:06 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) by mx1.freebsd.org (Postfix) with ESMTP id 52DC48FC1B; Mon, 7 Feb 2011 10:27:05 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id p17AR2rg033154; Mon, 7 Feb 2011 16:27:02 +0600 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4D4FC8F1.7060609@rdtc.ru> Date: Mon, 07 Feb 2011 16:26:57 +0600 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: Gleb Smirnoff References: <4D3011DB.9050900@frasunek.com> <4D30458D.30007@sentex.net> <4D309983.70709@rdtc.ru> <201101141437.55421.jhb@freebsd.org> <4D46575A.802@rdtc.ru> <4D4670C2.4050500@freebsd.org> <4D48513C.40503@rdtc.ru> <20110201185026.GB62007@glebius.int.ru> <4D4D554B.4050407@rdtc.ru> In-Reply-To: <4D4D554B.4050407@rdtc.ru> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Alexander Motin , John Baldwin Subject: Re: panic: bufwrite: buffer is not busy??? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 10:27:07 -0000 On 05.02.2011 19:48, Eugene Grosbein wrote: > Same box, more panics with KDB_TRACE, NETGRAPGH_DEBUG and your patch and Julian's. I've built new SuperMicro server (same hardware platform with ECC memory, same NanoBSD image and configuration but no lagg interfaces and only one uplink em0 and another downlink em1). After 3 days 20 hours it paniced with new kind of panic. Sadly, I forgot to enable crashdumps this time. Still, it had second panic (Sleeping thread owns a non-sleepable lock) before reboot. And it did not reboot itself, IPMI watchdog rebooted the box after 5 minutes. My servers run quagga/ospfd too, hence igmp in stack trace: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x2a0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff803f0888 stack pointer = 0x28:0xffffff80ec0057c0 frame pointer = 0x28:0xffffff80ec0057f0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 23426 (mpd5) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37 panic() at 0xffffffff8030b567 = panic+0x187 dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330 dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f trap() at 0xffffffff804c155f = trap+0x3df calltrap() at 0xffffffff804a8de4 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff803f0888, rsp = 0xffffff80ec0057c0, rbp = 0xffffff80ec0057f0 --- igmp_change_state() at 0xffffffff803f0888 = igmp_change_state+0x48 in_leavegroup_locked() at 0xffffffff803f8128 = in_leavegroup_locked+0x28 in_control() at 0xffffffff803f48c9 = in_control+0x13e9 ifioctl() at 0xffffffff803bee05 = ifioctl+0xca5 kern_ioctl() at 0xffffffff8034dea2 = kern_ioctl+0x102 ioctl() at 0xffffffff8034e0d0 = ioctl+0xf0 syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5 syscall() at 0xffffffff804c10fb = syscall+0x4b Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x8018e36ec, rsp = 0x7fffffbfe058, rbp = 0x8033ad518 --- Uptime: 3d20h15m12s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort Sleeping thread (tid 100234, pid 23426) owns a non-sleepable lock sched_switch() at 0xffffffff80330819 = sched_switch+0xf9 mi_switch() at 0xffffffff80313ac6 = mi_switch+0x176 sleepq_timedwait() at 0xffffffff803478a2 = sleepq_timedwait+0x42 _sleep() at 0xffffffff80314041 = _sleep+0x301 ucom_attach() at 0xffffffff8029756b = ucom_attach+0x129b ucom_attach() at 0xffffffff802976a4 = ucom_attach+0x13d4 ucom_attach() at 0xffffffff8029772b = ucom_attach+0x145b iicbus_release_bus() at 0xffffffff80230b88 = iicbus_release_bus+0x16f8 sc_puts() at 0xffffffff80263bec = sc_puts+0x24c sc_attach_unit() at 0xffffffff8026698c = sc_attach_unit+0x56c cncheckc() at 0xffffffff802cbd55 = cncheckc+0x65 dump_write() at 0xffffffff8030a8ba = dump_write+0xda kproc_shutdown() at 0xffffffff8030afc7 = kproc_shutdown+0x567 panic() at 0xffffffff8030b551 = panic+0x171 dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330 dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f trap() at 0xffffffff804c155f = trap+0x3df calltrap() at 0xffffffff804a8de4 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff803f0888, rsp = 0xffffff80ec0057c0, rbp = 0xffffff80ec0057f0 --- igmp_change_state() at 0xffffffff803f0888 = igmp_change_state+0x48 in_leavegroup_locked() at 0xffffffff803f8128 = in_leavegroup_locked+0x28 in_control() at 0xffffffff803f48c9 = in_control+0x13e9 ifioctl() at 0xffffffff803bee05 = ifioctl+0xca5 kern_ioctl() at 0xffffffff8034dea2 = kern_ioctl+0x102 ioctl() at 0xffffffff8034e0d0 = ioctl+0xf0 syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5 syscall() at 0xffffffff804c10fb = syscall+0x4b Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x8018e36ec, rsp = 0x7fffffbfe058, rbp = 0x8033ad518 --- panic: sleeping thread cpuid = 0 Uptime: 3d20h16m10s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 11:07:05 2011 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B3BC10656B0 for ; Mon, 7 Feb 2011 11:07:05 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 291858FC08 for ; Mon, 7 Feb 2011 11:07:05 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p17B755N027809 for ; Mon, 7 Feb 2011 11:07:05 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p17B74Ii027807 for freebsd-net@FreeBSD.org; Mon, 7 Feb 2011 11:07:04 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 7 Feb 2011 11:07:04 GMT Message-Id: <201102071107.p17B74Ii027807@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-net@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 11:07:05 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/154567 net [ath] ath(4) lot of bad series(0) o kern/154557 net [tcp] Freeze tcp-session of the clients, if in the gat o kern/154443 net [if_bridge] Kernel module bridgestp.ko missing after u o kern/154286 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/154284 net [ath] Modern ath wifi cards (such as AR9285) have miss o kern/154255 net [nfs] NFS not responding o kern/154214 net [stf] [panic] Panic when creating stf interface o kern/154169 net [multicast] [ip6] Node Information Query multicast add o kern/154134 net [ip6] stuck kernel state in LISTEN on ipv6 daemon whic o kern/154091 net [netgraph] [panic] netgraph, unaligned mbuf? o conf/154062 net [vlan] [patch] change to way of auto-generatation of v o kern/154006 net [tcp] [patch] tcp "window probe" bug on 64bit o kern/153951 net [ixgbe] Intel 10GBase-LR Ethernet card detected as 10G o kern/153938 net [run] [panic] [patch] Workaround for use-after-free pa o kern/153937 net [ral] ralink panics the system (amd64 freeBSDD 8.X) wh o kern/153936 net [ixgbe] [patch] MPRC workaround incorrectly applied to o kern/153816 net [ixgbe] ixgbe doesn't work properly with the Intel 10g o kern/153772 net [ixgbe] [patch] sysctls reference wrong XON/XOFF varia o kern/153671 net [em] [panic] 8.2-PRERELEASE repeatable kernel in if_em o kern/153497 net [netgraph] netgraph panic due to race conditions o kern/153454 net [patch] [wlan] [urtw] Support ad-hoc and hostap modes o kern/153308 net [em] em interface use 100% cpu o kern/153255 net [panic] 8.2-PRERELEASE repeatable kernel panic under h o kern/153244 net [em] em(4) fails to send UDP to port 0xffff o kern/152893 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/152853 net [em] tftpd (and likely other udp traffic) fails over e o kern/152828 net [em] poor performance on 8.1, 8.2-PRE o kern/152569 net [net]: Multiple ppp connections and routing table prob o kern/152360 net [dummynet] [panic] Crash related to dummynet. o kern/152235 net [arp] Permanent local ARP entries are not properly upd o kern/152141 net [vlan] encapsulate vlan in ng_ether before output to i o kern/151690 net [ep] network connectivity won't work until dhclient is o kern/151681 net [nfs] NFS mount via IPv6 leads to hang on client with o kern/151593 net [igb] [panic] Kernel panic when bringing up igb networ o kern/150920 net [ixgbe][igb] Panic when packets are dropped with heade o bin/150642 net netstat(1) doesn't print anything for SCTP sockets o kern/150557 net [igb] igb0: Watchdog timeout -- resetting o kern/150251 net [patch] [ixgbe] Late cable insertion broken o kern/150249 net [ixgbe] Media type detection broken o kern/150247 net [patch] [ixgbe] Version in -current won't build on 7.x o bin/150224 net ppp(8) does not reassign static IP after kill -KILL co f kern/149969 net [wlan] [ral] ralink rt2661 fails to maintain connectio o kern/149937 net [ipfilter] [patch] kernel panic in ipfilter IP fragmen o kern/149786 net [bwn] bwn on Dell Inspiron 1150: connections stall o kern/149643 net [rum] device not sending proper beacon frames in ap mo o kern/149609 net [panic] reboot after adding second default route o kern/149539 net [ath] atheros ar9287 is not supported by ath_hal o kern/149516 net [ath] ath(4) hostap with fake MAC/BSSID results in sta o kern/149373 net [realtek/atheros]: None of my network card working o kern/149307 net [ath] Doesn't work Atheros 9285 o kern/149306 net [alc] Doesn't work Atheros AR8131 PCIe Gigabit Etherne o kern/149117 net [inet] [patch] in_pcbbind: redundant test o kern/149086 net [multicast] Generic multicast join failure in 8.1 o kern/148322 net [ath] Triggering atheros wifi beacon misses in hostap o kern/148317 net [ath] FreeBSD 7.x hostap memory leak in net80211 or At o kern/148078 net [ath] wireless networking stops functioning o kern/147894 net [ipsec] IPv6-in-IPv4 does not work inside an ESP-only o kern/147155 net [ip6] setfb not work with ipv6 o kern/146845 net [libc] close(2) returns error 54 (connection reset by o kern/146792 net [flowtable] flowcleaner 100% cpu's core load o kern/146759 net [cxgb] [patch] cxgb panic calling cxgb_set_lro() witho o kern/146719 net [pf] [panic] PF or dumynet kernel panic o kern/146534 net [icmp6] wrong source address in echo reply o kern/146427 net [mwl] Additional virtual access points don't work on m o kern/146426 net [mwl] 802.11n rates not possible on mwl o kern/146425 net [mwl] mwl dropping all packets during and after high u f kern/146394 net [vlan] IP source address for outgoing connections o bin/146377 net [ppp] [tun] Interface doesn't clear addresses when PPP o kern/146358 net [vlan] wrong destination MAC address o kern/146165 net [wlan] [panic] Setting bssid in adhoc mode causes pani o kern/146082 net [ng_l2tp] a false invaliant check was performed in ng_ o kern/146037 net [panic] mpd + CoA = kernel panic o bin/145934 net [patch] add count option to netstat(1) o kern/145826 net [ath] Unable to configure adhoc mode on ath0/wlan0 o kern/145825 net [panic] panic: soabort: so_count o kern/145728 net [lagg] Stops working lagg between two servers. o kern/144987 net [wpi] [panic] injecting packets with wlaninject using o kern/144882 net MacBookPro =>4.1 does not connect to BSD in hostap wit o kern/144874 net [if_bridge] [patch] if_bridge frees mbuf after pfil ho o conf/144700 net [rc.d] async dhclient breaks stuff for too many people o kern/144642 net [rum] [panic] Enabling rum interface causes panic o kern/144616 net [nat] [panic] ip_nat panic FreeBSD 7.2 o kern/144572 net [carp] CARP preemption mode traffic partially goes to f kern/144315 net [ipfw] [panic] freebsd 8-stable reboot after add ipfw o kern/143939 net [ipfw] [em] ipfw nat and em interface rxcsum problem o kern/143874 net [wpi] Wireless 3945ABG error. wpi0 could not allocate o kern/143868 net [ath] [patch] [request] allow Atheros watchdog timeout o kern/143846 net [gif] bringing gif3 tunnel down causes gif0 tunnel to s kern/143673 net [stf] [request] there should be a way to support multi s kern/143666 net [ip6] [request] PMTU black hole detection not implemen o kern/143622 net [pfil] [patch] unlock pfil lock while calling firewall o kern/143593 net [ipsec] When using IPSec, tcpdump doesn't show outgoin o kern/143591 net [ral] RT2561C-based DLink card (DWL-510) fails to work o kern/143208 net [ipsec] [gif] IPSec over gif interface not working o conf/143079 net hostapd(8) startup missing multi wlan functionality o kern/143034 net [panic] system reboots itself in tcp code [regression] o kern/142877 net [hang] network-related repeatable 8.0-STABLE hard hang o kern/142774 net Problem with outgoing connections on interface with mu o kern/142772 net [libc] lla_lookup: new lle malloc failed o kern/142018 net [iwi] [patch] Possibly wrong interpretation of beacon- o kern/141861 net [wi] data garbled with WEP and wi(4) with Prism 2.5 f kern/141741 net Etherlink III NIC won't work after upgrade to FBSD 8, o kern/141023 net [carp] CARP arp replays with wrong src mac o kern/140796 net [ath] [panic] privileged instruction fault o kern/140742 net rum(4) Two asus-WL167G adapters cannot talk to each ot o kern/140682 net [netgraph] [panic] random panic in netgraph o kern/140634 net [vlan] destroying if_lagg interface with if_vlan membe o kern/140619 net [ifnet] [patch] refine obsolete if_var.h comments desc o kern/140346 net [wlan] High bandwidth use causes loss of wlan connecti o kern/140245 net [ath] [panic] Kernel panic during network activity on o kern/140142 net [ip6] [panic] FreeBSD 7.2-amd64 panic w/IPv6 o kern/140066 net [bwi] install report for 8.0 RC 2 (multiple problems) o kern/139565 net [ipfilter] ipfilter ioctl SIOCDELST broken o kern/139387 net [ipsec] Wrong lenth of PF_KEY messages in promiscuous o bin/139346 net [patch] arp(8) add option to remove static entries lis o kern/139268 net [if_bridge] [patch] allow if_bridge to forward just VL o kern/139204 net [arp] DHCP server replies rejected, ARP entry lost bef o kern/139117 net [lagg] + wlan boot timing (EBUSY) o kern/139058 net [ipfilter] mbuf cluster leak on FreeBSD 7.2 o kern/138850 net [dummynet] dummynet doesn't work correctly on a bridge o kern/138782 net [panic] sbflush_internal: cc 0 || mb 0xffffff004127b00 o kern/138688 net [rum] possibly broken on 8 Beta 4 amd64: able to wpa a o kern/138678 net [lo] FreeBSD does not assign linklocal address to loop o kern/138620 net [lagg] [patch] lagg port bpf-writes blocked o kern/138407 net [gre] gre(4) interface does not come up after reboot o kern/138332 net [tun] [lor] ifconfig tun0 destroy causes LOR if_adata/ o kern/138266 net [panic] kernel panic when udp benchmark test used as r o kern/138177 net [ipfilter] FreeBSD crashing repeatedly in ip_nat.c:257 o kern/137881 net [netgraph] [panic] ng_pppoe fatal trap 12 o bin/137841 net [patch] wpa_supplicant(8) cannot verify SHA256 signed p kern/137776 net [rum] panic in rum(4) driver on 8.0-BETA2 o kern/137775 net [netgraph] [patch] Add XMIT_FAILOVER to ng_one2many o bin/137641 net ifconfig(8): various problems with "vlan_device.vlan_i o kern/137592 net [ath] panic - 7-STABLE (Aug 7, 2009 UTC) crashes on ne o bin/137484 net [patch] Integer overflow in wpa_supplicant(8) base64 e o kern/137392 net [ip] [panic] crash in ip_nat.c line 2577 o kern/137372 net [ral] FreeBSD doesn't support wireless interface from o kern/137089 net [lagg] lagg falsely triggers IPv6 duplicate address de o bin/136994 net [patch] ifconfig(8) print carp mac address o kern/136943 net [wpi] [lor] wpi0_com_lock / wpi0 o kern/136911 net [netgraph] [panic] system panic on kldload ng_bpf.ko t o kern/136836 net [ath] atheros card stops functioning after about 12 ho o bin/136661 net [patch] ndp(8) ignores -f option o kern/136618 net [pf][stf] panic on cloning interface without unit numb o kern/136426 net [panic] spawning several dhclients in parallel panics o kern/135502 net [periodic] Warning message raised by rtfree function i o kern/134931 net [route] Route messages sent to all socket listeners re o kern/134583 net [hang] Machine with jail freezes after random amount o o kern/134531 net [route] [panic] kernel crash related to routes/zebra o kern/134168 net [ral] ral driver problem on RT2525 2.4GHz transceiver o kern/134157 net [dummynet] dummynet loads cpu for 100% and make a syst o kern/133969 net [dummynet] [panic] Fatal trap 12: page fault while in o kern/133968 net [dummynet] [panic] dummynet kernel panic o kern/133736 net [udp] ip_id not protected ... o kern/133595 net [panic] Kernel Panic at pcpu.h:195 o kern/133572 net [ppp] [hang] incoming PPTP connection hangs the system o kern/133490 net [bpf] [panic] 'kmem_map too small' panic on Dell r900 o kern/133235 net [netinet] [patch] Process SIOCDLIFADDR command incorre o kern/133218 net [carp] [hang] use of carp(4) causes system to freeze f kern/133213 net arp and sshd errors on 7.1-PRERELEASE o kern/133060 net [ipsec] [pfsync] [panic] Kernel panic with ipsec + pfs o kern/132889 net [ndis] [panic] NDIS kernel crash on load BCM4321 AGN d o conf/132851 net [patch] rc.conf(5): allow to setfib(1) for service run o kern/132734 net [ifmib] [panic] panic in net/if_mib.c o kern/132722 net [ath] Wifi ath0 associates fine with AP, but DHCP or I o kern/132705 net [libwrap] [patch] libwrap - infinite loop if hosts.all o kern/132672 net [ndis] [panic] ndis with rt2860.sys causes kernel pani o kern/132554 net [ipl] There is no ippool start script/ipfilter magic t o kern/132354 net [nat] Getting some packages to ipnat(8) causes crash o kern/132285 net [carp] alias gives incorrect hash in dmesg o kern/132277 net [crypto] [ipsec] poor performance using cryptodevice f o kern/132107 net [carp] carp(4) advskew setting ignored when carp IP us o kern/131781 net [ndis] ndis keeps dropping the link o kern/131776 net [wi] driver fails to init o kern/131753 net [altq] [panic] kernel panic in hfsc_dequeue o bin/131567 net [socket] [patch] Update for regression/sockets/unix_cm o kern/131549 net ifconfig(8) can't clear 'monitor' mode on the wireless o bin/131365 net route(8): route add changes interpretation of network f kern/130820 net [ndis] wpa_supplicant(8) returns 'no space on device' o kern/130628 net [nfs] NFS / rpc.lockd deadlock on 7.1-R o conf/130555 net [rc.d] [patch] No good way to set ipfilter variables a o kern/130525 net [ndis] [panic] 64 bit ar5008 ndisgen-erated driver cau o kern/130311 net [wlan_xauth] [panic] hostapd restart causing kernel pa o kern/130109 net [ipfw] Can not set fib for packets originated from loc f kern/130059 net [panic] Leaking 50k mbufs/hour f kern/129750 net [ath] Atheros AR5006 exits on "cannot map register spa f kern/129719 net [nfs] [panic] Panic during shutdown, tcp_ctloutput: in o kern/129517 net [ipsec] [panic] double fault / stack overflow o kern/129508 net [carp] [panic] Kernel panic with EtherIP (may be relat o kern/129219 net [ppp] Kernel panic when using kernel mode ppp o kern/129197 net [panic] 7.0 IP stack related panic o bin/128954 net ifconfig(8) deletes valid routes o bin/128602 net [an] wpa_supplicant(8) crashes with an(4) o kern/128448 net [nfs] 6.4-RC1 Boot Fails if NFS Hostname cannot be res o conf/128334 net [request] use wpa_cli in the "WPA DHCP" situation o bin/128295 net [patch] ifconfig(8) does not print TOE4 or TOE6 capabi o bin/128001 net wpa_supplicant(8), wlan(4), and wi(4) issues o kern/127826 net [iwi] iwi0 driver has reduced performance and connecti o kern/127815 net [gif] [patch] if_gif does not set vlan attributes from o kern/127724 net [rtalloc] rtfree: 0xc5a8f870 has 1 refs f bin/127719 net [arp] arp: Segmentation fault (core dumped) f kern/127528 net [icmp]: icmp socket receives icmp replies not owned by o bin/127192 net routed(8) removes the secondary alias IP of interface f kern/127145 net [wi]: prism (wi) driver crash at bigger traffic o kern/127057 net [udp] Unable to send UDP packet via IPv6 socket to IPv o kern/127050 net [carp] ipv6 does not work on carp interfaces [regressi o kern/126945 net [carp] CARP interface destruction with ifconfig destro o kern/126895 net [patch] [ral] Add antenna selection (marked as TBD) o kern/126874 net [vlan]: Zebra problem if ifconfig vlanX destroy o kern/126714 net [carp] CARP interface renaming makes system no longer o kern/126695 net rtfree messages and network disruption upon use of if_ o kern/126475 net [ath] [panic] ath pcmcia card inevitably panics under o kern/126339 net [ipw] ipw driver drops the connection o kern/126214 net [ath] txpower problem with Atheros wifi card o kern/126075 net [inet] [patch] internet control accesses beyond end of o bin/125922 net [patch] Deadlock in arp(8) o kern/125920 net [arp] Kernel Routing Table loses Ethernet Link status o kern/125845 net [netinet] [patch] tcp_lro_rx() should make use of hard o kern/125816 net [carp] [if_bridge] carp stuck in init when using bridg o kern/125721 net [ath] Terrible throughput/high ping latency with Ubiqu o kern/125617 net [ath] [panic] ath(4) related panic o kern/125501 net [ath] atheros cardbus driver hangs f kern/125442 net [carp] [lagg] CARP combined with LAGG causes system pa f kern/125332 net [ath] [panic] crash under any non-tiny networking unde o kern/125258 net [socket] socket's SO_REUSEADDR option does not work o kern/125239 net [gre] kernel crash when using gre o kern/124767 net [iwi] Wireless connection using iwi0 driver (Intel 220 o kern/124341 net [ral] promiscuous mode for wireless device ral0 looses o kern/124225 net [ndis] [patch] ndis network driver sometimes loses net o kern/124160 net [libc] connect(2) function loops indefinitely o kern/124021 net [ip6] [panic] page fault in nd6_output() o kern/123968 net [rum] [panic] rum driver causes kernel panic with WPA. o kern/123892 net [tap] [patch] No buffer space available o kern/123890 net [ppp] [panic] crash & reboot on work with PPP low-spee o kern/123858 net [stf] [patch] stf not usable behind a NAT o kern/123796 net [ipf] FreeBSD 6.1+VPN+ipnat+ipf: port mapping does not o kern/123758 net [panic] panic while restarting net/freenet6 o bin/123633 net ifconfig(8) doesn't set inet and ether address in one o kern/123559 net [iwi] iwi periodically disassociates/associates [regre o bin/123465 net [ip6] route(8): route add -inet6 -interfac o kern/123463 net [ipsec] [panic] repeatable crash related to ipsec-tool o conf/123330 net [nsswitch.conf] Enabling samba wins in nsswitch.conf c o kern/123160 net [ip] Panic and reboot at sysctl kern.polling.enable=0 f kern/123045 net [ng_mppc] ng_mppc_decompress - disabling node o kern/122989 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/122954 net [lagg] IPv6 EUI64 incorrectly chosen for lagg devices f kern/122780 net [lagg] tcpdump on lagg interface during high pps wedge o kern/122697 net [ath] Atheros card is not well supported o kern/122685 net It is not visible passing packets in tcpdump(1) o kern/122319 net [wi] imposible to enable ad-hoc demo mode with Orinoco o kern/122290 net [netgraph] [panic] Netgraph related "kmem_map too smal o kern/122033 net [ral] [lor] Lock order reversal in ral0 at bootup ieee o bin/121895 net [patch] rtsol(8)/rtsold(8) doesn't handle managed netw s kern/121774 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/121555 net [panic] Fatal trap 12: current process = 12 (swi1: net o kern/121443 net [gif] [lor] icmp6_input/nd6_lookup o kern/121437 net [vlan] Routing to layer-2 address does not work on VLA o bin/121359 net [patch] [security] ppp(8): fix local stack overflow in o kern/121257 net [tcp] TSO + natd -> slow outgoing tcp traffic o kern/121181 net [panic] Fatal trap 3: breakpoint instruction fault whi o kern/120966 net [rum] kernel panic with if_rum and WPA encryption p docs/120945 net [patch] ip6(4) man page lacks documentation for TCLASS o kern/120566 net [request]: ifconfig(8) make order of arguments more fr o kern/120304 net [netgraph] [patch] netgraph source assumes 32-bit time o kern/120266 net [udp] [panic] gnugk causes kernel panic when closing U o kern/120130 net [carp] [panic] carp causes kernel panics in any conste o bin/120060 net routed(8) deletes link-level routes in the presence of o kern/119945 net [rum] [panic] rum device in hostap mode, cause kernel o kern/119791 net [nfs] UDP NFS mount of aliased IP addresses from a Sol o kern/119617 net [nfs] nfs error on wpa network when reseting/shutdown f kern/119516 net [ip6] [panic] _mtx_lock_sleep: recursed on non-recursi o kern/119432 net [arp] route add -host -iface causes arp e o kern/119225 net [wi] 7.0-RC1 no carrier with Prism 2.5 wifi card [regr o kern/118727 net [netgraph] [patch] [request] add new ng_pf module s kern/117717 net [panic] Kernel panic with Bittorrent client. o kern/117448 net [carp] 6.2 kernel crash [regression] o kern/117423 net [vlan] Duplicate IP on different interfaces o bin/117339 net [patch] route(8): loading routing management commands o kern/117271 net [tap] OpenVPN TAP uses 99% CPU on releng_6 when if_tap o kern/116747 net [ndis] FreeBSD 7.0-CURRENT crash with Dell TrueMobile o bin/116643 net [patch] [request] fstat(1): add INET/INET6 socket deta o kern/116185 net [iwi] if_iwi driver leads system to reboot o kern/115239 net [ipnat] panic with 'kmem_map too small' using ipnat o kern/115019 net [netgraph] ng_ether upper hook packet flow stops on ad o kern/115002 net [wi] if_wi timeout. failed allocation (busy bit). ifco o kern/114915 net [patch] [pcn] pcn (sys/pci/if_pcn.c) ethernet driver f o kern/113432 net [ucom] WARNING: attempt to net_add_domain(netgraph) af o kern/112722 net [ipsec] [udp] IP v4 udp fragmented packet reject o kern/112686 net [patm] patm driver freezes System (FreeBSD 6.2-p4) i38 o bin/112557 net [patch] ppp(8) lock file should not use symlink name o kern/112528 net [nfs] NFS over TCP under load hangs with "impossible p o kern/111457 net [ral] ral(4) freeze o kern/109470 net [wi] Orinoco Classic Gold PC Card Can't Channel Hop o kern/109308 net [pppd] [panic] Multiple panics kernel ppp suspected [r o bin/108895 net pppd(8): PPPoE dead connections on 6.2 [regression] o kern/107944 net [wi] [patch] Forget to unlock mutex-locks f kern/107279 net [ath] [panic] ath_start: attempted use of a free mbuf! o conf/107035 net [patch] bridge(8): bridge interface given in rc.conf n o kern/106444 net [netgraph] [panic] Kernel Panic on Binding to an ip to o kern/106438 net [ipf] ipfilter: keep state does not seem to allow repl o kern/106316 net [dummynet] dummynet with multipass ipfw drops packets o kern/105945 net Address can disappear from network interface s kern/105943 net Network stack may modify read-only mbuf chain copies o bin/105925 net problems with ifconfig(8) and vlan(4) [regression] f kern/105348 net [ath] ath device stopps TX o kern/104851 net [inet6] [patch] On link routes not configured when usi o kern/104751 net [netgraph] kernel panic, when getting info about my tr o kern/103191 net Unpredictable reboot o kern/103135 net [ipsec] ipsec with ipfw divert (not NAT) encodes a pac o kern/102540 net [netgraph] [patch] supporting vlan(4) by ng_fec(4) o conf/102502 net [netgraph] [patch] ifconfig name does't rename netgrap o kern/102035 net [plip] plip networking disables parallel port printing o kern/101948 net [ipf] [panic] Kernel Panic Trap No 12 Page Fault - cau o kern/100709 net [libc] getaddrinfo(3) should return TTL info o kern/100519 net [netisr] suggestion to fix suboptimal network polling o kern/98978 net [ipf] [patch] ipfilter drops OOW packets under 6.1-Rel o kern/98597 net [inet6] Bug in FreeBSD 6.1 IPv6 link-local DAD procedu o bin/98218 net wpa_supplicant(8) blacklist not working o kern/97306 net [netgraph] NG_L2TP locks after connection with failed o conf/97014 net [gif] gifconfig_gif? in rc.conf does not recognize IPv f kern/96268 net [socket] TCP socket performance drops by 3000% if pack o kern/95519 net [ral] ral0 could not map mbuf o kern/95288 net [pppd] [tty] [panic] if_ppp panic in sys/kern/tty_subr o kern/95277 net [netinet] [patch] IP Encapsulation mask_match() return o kern/95267 net packet drops periodically appear f kern/93886 net [ath] Atheros/D-Link DWL-G650 long delay to associate f kern/93378 net [tcp] Slow data transfer in Postfix and Cyrus IMAP (wo o kern/93019 net [ppp] ppp and tunX problems: no traffic after restarti o kern/92880 net [libc] [patch] almost rewritten inet_network(3) functi s kern/92279 net [dc] Core faults everytime I reboot, possible NIC issu o kern/91859 net [ndis] if_ndis does not work with Asus WL-138 s kern/91777 net [ipf] [patch] wrong behaviour with skip rule inside an o kern/91364 net [ral] [wep] WF-511 RT2500 Card PCI and WEP o kern/91311 net [aue] aue interface hanging s kern/90086 net [hang] 5.4p8 on supermicro P8SCT hangs during boot if o kern/87521 net [ipf] [panic] using ipfilter "auth" keyword leads to k o kern/87421 net [netgraph] [panic]: ng_ether + ng_eiface + if_bridge s kern/86920 net [ndis] ifconfig: SIOCS80211: Invalid argument [regress o kern/86871 net [tcp] [patch] allocation logic for PCBs in TIME_WAIT s o kern/86427 net [lor] Deadlock with FASTIPSEC and nat o kern/86103 net [ipf] Illegal NAT Traversal in IPFilter o kern/85780 net 'panic: bogus refcnt 0' in routing/ipv6 o bin/85445 net ifconfig(8): deprecated keyword to ifconfig inoperativ p kern/85320 net [gre] [patch] possible depletion of kernel stack in ip o bin/82975 net route change does not parse classfull network as given o kern/82881 net [netgraph] [panic] ng_fec(4) causes kernel panic after o bin/82185 net [patch] ndp(8) can delete the incorrect entry o kern/81095 net IPsec connection stops working if associated network i o kern/79895 net [ipf] 5.4-RC2 breaks ipfilter NAT when using netgraph o bin/79228 net [patch] extend arp(8) to be able to create blackhole r o kern/78968 net FreeBSD freezes on mbufs exhaustion (network interface o kern/78090 net [ipf] ipf filtering on bridged packets doesn't work if o kern/77341 net [ip6] problems with IPV6 implementation o kern/77273 net [ipf] ipfilter breaks ipv6 statefull filtering on 5.3 s kern/77195 net [ipf] [patch] ipfilter ioctl SIOCGNATL does not match o kern/75873 net Usability problem with non-RFC-compliant IP spoof prot s kern/75407 net [an] an(4): no carrier after short time a kern/71474 net [route] route lookup does not skip interfaces marked d o kern/71469 net default route to internet magically disappears with mu o kern/70904 net [ipf] ipfilter ipnat problem with h323 proxy support o kern/66225 net [netgraph] [patch] extend ng_eiface(4) control message o kern/65616 net IPSEC can't detunnel GRE packets after real ESP encryp s kern/60293 net [patch] FreeBSD arp poison patch a kern/56233 net IPsec tunnel (ESP) over IPv6: MTU computation is wrong o kern/54383 net [nfs] [patch] NFS root configurations without dynamic s bin/41647 net ifconfig(8) doesn't accept lladdr along with inet addr s kern/39937 net ipstealth issue a kern/38554 net [patch] changing interface ipaddress doesn't seem to w o kern/34665 net [ipf] [hang] ipfilter rcmd proxy "hangs". o kern/31647 net [libc] socket calls can return undocumented EINVAL o kern/30186 net [libc] getaddrinfo(3) does not handle incorrect servna o kern/27474 net [ipf] [ppp] Interactive use of user PPP and ipfilter c o conf/23063 net [arp] [patch] for static ARP tables in rc.network 373 problems total. From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 11:43:55 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ABE16106564A; Mon, 7 Feb 2011 11:43:55 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) by mx1.freebsd.org (Postfix) with ESMTP id 9FFAA8FC0C; Mon, 7 Feb 2011 11:43:54 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id p17BhpZ6034103; Mon, 7 Feb 2011 17:43:52 +0600 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4D4FDAF2.8070709@rdtc.ru> Date: Mon, 07 Feb 2011 17:43:46 +0600 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: Gleb Smirnoff References: <4D3011DB.9050900@frasunek.com> <4D30458D.30007@sentex.net> <4D309983.70709@rdtc.ru> <201101141437.55421.jhb@freebsd.org> <4D46575A.802@rdtc.ru> <4D4670C2.4050500@freebsd.org> <4D48513C.40503@rdtc.ru> <20110201185026.GB62007@glebius.int.ru> In-Reply-To: <20110201185026.GB62007@glebius.int.ru> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Alexander Motin , John Baldwin Subject: Re: panic: bufwrite: buffer is not busy??? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 11:43:55 -0000 On 02.02.2011 00:50, Gleb Smirnoff wrote: > On Wed, Feb 02, 2011 at 12:30:20AM +0600, Eugene Grosbein wrote: > E> On 31.01.2011 14:20, Julian Elischer wrote: > E> > E> > replace with: > E> > > E> > 3504 if ((hook == NULL) || > E> > 3505 NG_HOOK_NOT_VALID(hook) || > E> > ((peer = NG_HOOK_PEER(hook)) == NULL) || > E> > 3506 NG_HOOK_NOT_VALID(peer) || > E> > ((peernode = NG_PEER_NODE(hook)) == NULL) || > E> > 3507 NG_NODE_NOT_VALID(peernode)) { > E> > if (peer) > E> > kassert((peernode != NULL), ("peer node NULL wile peer hook exists")); > E> > 3508 NG_FREE_ITEM(item); > E> > E> This day I have updated panicing router to RELENG_8 and combined changes supposed > E> by Julian and Gleb. After 8 hours it has just paniced again and could not finish > E> to write crashdump again: > E> > E> Fatal trap 12: page fault while in kernel mode > E> cpuid = 3; apic id = 06 > E> fault virtual address = 0x63 > E> fault code = supervisor read data, page not present > E> instruction pointer = 0x20:0xffffffff803d4ccd > E> stack pointer = 0x28:0xffffff80ebffc600 > E> frame pointer = 0x28:0xffffff80ebffc680 > E> code segment = base 0x0, limit 0xfffff, type 0x1b > E> = DPL 0, pres 1, long 1, def32 0, gran 1 > E> processor eflags = interrupt enabled, resume, IOPL = 0 > E> current process = 2390 (mpd5) > E> trap number = 12 > E> panic: page fault > E> cpuid = 3 > E> Uptime: 8h3m51s > E> Dumping 4087 MB (3 chunks) > E> chunk 0: 1MB (150 pages) ... ok > E> chunk 1: 3575MB (915088 pages) 3559 3543panic: bufwrite: buffer is not busy??? > E> cpuid = 3 > E> Uptime: 8h3m52s > E> Automatic reboot in 15 seconds - press a key on the console to abort > E> > E> # gdb kernel > E> GNU gdb 6.1.1 [FreeBSD] > E> Copyright 2004 Free Software Foundation, Inc. > E> GDB is free software, covered by the GNU General Public License, and you are > E> welcome to change it and/or distribute copies of it under certain conditions. > E> Type "show copying" to see the conditions. > E> There is absolutely no warranty for GDB. Type "show warranty" for details. > E> This GDB was configured as "amd64-marcel-freebsd"... > E> (gdb) l *0xffffffff803d4ccd > E> 0xffffffff803d4ccd is in ng_pppoe_disconnect (netgraph.h:191). > E> 186 int line); > E> 187 > E> 188 static __inline void > E> 189 _chkhook(hook_p hook, char *file, int line) > E> 190 { > E> 191 if (hook->hk_magic != HK_MAGIC) { > E> 192 printf("Accessing freed hook "); > E> 193 dumphook(hook, file, line); > E> 194 } > E> 195 hook->lastline = line; > E> (gdb) x/i 0xffffffff803d4ccd > E> 0xffffffff803d4ccd : cmpl $0x78573011,0x64(%rbx) > > This looks like ng_pppoe_disconnect() was called with NULL argument. > > Can you add KDB_TRACE option to kernel? Your boxes for some reason can't > dump core, but with this option we will have at least trace. Just paniced again: same box, same panic but this time fault virtual address = 0x64 instead of 0x63. And just locked while dumping core. Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 06 fault virtual address = 0x64 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff803e4f36 stack pointer = 0x28:0xffffff80ebff7400 frame pointer = 0x28:0xffffff80ebff7430 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2390 (mpd5) trap number = 12 panic: page fault cpuid = 1 KDB: stack backtrace: X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37 panic() at 0xffffffff8030b567 = panic+0x187 dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330 dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f trap() at 0xffffffff804c155f = trap+0x3df calltrap() at 0xffffffff804a8de4 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff803e4f36, rsp = 0xffffff80ebff7400, rbp = 0xffffff80ebff7430 --- ng_parse_get_token() at 0xffffffff803e4f36 = ng_parse_get_token+0x6596 ng_parse_get_token() at 0xffffffff803e5ccf = ng_parse_get_token+0x732f ng_destroy_hook() at 0xffffffff803d53b2 = ng_destroy_hook+0x222 ng_rmnode() at 0xffffffff803d6118 = ng_rmnode+0xa08 ng_snd_item() at 0xffffffff803d8520 = ng_snd_item+0x3f0 ng_destroy_hook() at 0xffffffff803d52ed = ng_destroy_hook+0x15d ng_rmnode() at 0xffffffff803d57b9 = ng_rmnode+0xa9 ng_rmnode() at 0xffffffff803d7664 = ng_rmnode+0x1f54 ng_snd_item() at 0xffffffff803d8520 = ng_snd_item+0x3f0 ng_parse_get_token() at 0xffffffff803e97fa = ng_parse_get_token+0xae5a sosend_generic() at 0xffffffff80373df6 = sosend_generic+0x436 kern_sendit() at 0xffffffff803776d5 = kern_sendit+0x1a5 kern_sendit() at 0xffffffff8037790c = kern_sendit+0x3dc sendto() at 0xffffffff803779fd = sendto+0x4d syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5 syscall() at 0xffffffff804c10fb = syscall+0x4b Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2 --- syscall (133, FreeBSD ELF64, sendto), rip = 0x8018c971c, rsp = 0x7fffffbfeab8, rbp = 0x80203dcc0 --- Uptime: 2d17h1m42s Dumping 4087 MB (3 chunks) chunk 0: 1MB (150 pages) ... ok chunk 1: 3575MB (915088 pages) 3559 3543 3527 3511 Rebooted by watchdog at this point. (gdb) l *0xffffffff803e4f36 0xffffffff803e4f36 is in pppoe_send_event (netgraph.h:191). 186 int line); 187 188 static __inline void 189 _chkhook(hook_p hook, char *file, int line) 190 { 191 if (hook->hk_magic != HK_MAGIC) { 192 printf("Accessing freed hook "); 193 dumphook(hook, file, line); 194 } 195 hook->lastline = line; (gdb) x/i 0xffffffff803e4f36 0xffffffff803e4f36 : cmpl $0x78573011,0x64(%r12) Eugene Grosbein From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 12:22:19 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D408D106564A; Mon, 7 Feb 2011 12:22:19 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 8DB318FC08; Mon, 7 Feb 2011 12:22:19 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.4/8.14.4) with ESMTP id p17CMD46070327 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 7 Feb 2011 07:22:14 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4D4FE3F3.6030604@sentex.net> Date: Mon, 07 Feb 2011 07:22:11 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: lev@freebsd.org References: <787079038.20110207004023@serebryakov.spb.ru> In-Reply-To: <787079038.20110207004023@serebryakov.spb.ru> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on IPv6:2607:f3e0:0:1::12 Cc: freebsd-net@freebsd.org, freebsd-stable@freebsd.org, "Vogel, Jack" Subject: Re: em0 hangs without any messages like "Watchdog timeout", only down/up reset it. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 12:22:19 -0000 On 2/6/2011 4:40 PM, Lev Serebryakov wrote: > Hello, Freebsd-stable. > It hangs under load without any output. When it works with POLLING, it > prints "Watchdog timeout" and reset automatically several times a day, > but without POLLING it simply hangs with same frequency. It is > 8.2-PRERELEASE (from RELENG_8), cvsupped AFTER 22 Jan (last commit to > e1000 drivers family). > > Locally system works, but mbufs are overfilled. "ifconfig em0 down > && ifconfig em0 up" solves problem. In non polling mode, do you have any special network or driver settings in /boot/loader.conf or /etc/sysctl.conf ? ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 12:52:06 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B3FA1065694; Mon, 7 Feb 2011 12:52:06 +0000 (UTC) (envelope-from lev@freebsd.org) Received: from ftp.translate.ru (ftp.translate.ru [80.249.188.42]) by mx1.freebsd.org (Postfix) with ESMTP id DB3148FC13; Mon, 7 Feb 2011 12:52:05 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (89.112.15.178.pppoe.eltel.net [89.112.15.178]) (Authenticated sender: lev@serebryakov.spb.ru) by ftp.translate.ru (Postfix) with ESMTPA id 49F5513DF5F; Mon, 7 Feb 2011 15:52:04 +0300 (MSK) Date: Mon, 7 Feb 2011 15:52:03 +0300 From: Lev Serebryakov X-Priority: 3 (Normal) Message-ID: <504473738.20110207155203@serebryakov.spb.ru> To: Mike Tancsa In-Reply-To: <4D4FE3F3.6030604@sentex.net> References: <787079038.20110207004023@serebryakov.spb.ru> <4D4FE3F3.6030604@sentex.net> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, freebsd-stable@freebsd.org, "Vogel, Jack" Subject: Re: em0 hangs without any messages like "Watchdog timeout", only down/up reset it. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 12:52:06 -0000 Hello, Mike. You wrote 7 =F4=E5=E2=F0=E0=EB=FF 2011 =E3., 15:22:11: >> It hangs under load without any output. When it works with POLLING, it >> prints "Watchdog timeout" and reset automatically several times a day, >> but without POLLING it simply hangs with same frequency. It is >> 8.2-PRERELEASE (from RELENG_8), cvsupped AFTER 22 Jan (last commit to >> e1000 drivers family). >>=20 >> Locally system works, but mbufs are overfilled. "ifconfig em0 down >> && ifconfig em0 up" solves problem. > In non polling mode, do you have any special network or driver settings > in /boot/loader.conf or /etc/sysctl.conf ? /boot/loader.conf: hw.em.rxd=3D4096 hw.em.txd=3D4096 net.link.ifqmaxlen=3D16384 /etc/sysctl.conf: dev.em.0.rx_int_delay=3D200 dev.em.0.tx_int_delay=3D200 dev.em.0.rx_abs_int_delay=3D4000 dev.em.0.tx_abs_int_delay=3D4000 dev.em.0.rx_processing_limit=3D4096 I'm trying to run with patch from "em driver, 82574L chip, and possibly ASPM" thread now under heavy network load. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 13:03:04 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1C77106566C; Mon, 7 Feb 2011 13:03:04 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id A9A9A8FC14; Mon, 7 Feb 2011 13:03:04 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.4/8.14.4) with ESMTP id p17D32Wh075560 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 7 Feb 2011 08:03:02 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4D4FED83.2010400@sentex.net> Date: Mon, 07 Feb 2011 08:02:59 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Lev Serebryakov References: <787079038.20110207004023@serebryakov.spb.ru> <4D4FE3F3.6030604@sentex.net> <504473738.20110207155203@serebryakov.spb.ru> In-Reply-To: <504473738.20110207155203@serebryakov.spb.ru> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on IPv6:2607:f3e0:0:1::12 Cc: freebsd-net@freebsd.org, freebsd-stable@freebsd.org, "Vogel, Jack" Subject: Re: em0 hangs without any messages like "Watchdog timeout", only down/up reset it. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 13:03:05 -0000 > > /boot/loader.conf: > hw.em.rxd=4096 > hw.em.txd=4096 > net.link.ifqmaxlen=16384 > > /etc/sysctl.conf: > dev.em.0.rx_int_delay=200 > dev.em.0.tx_int_delay=200 > dev.em.0.rx_abs_int_delay=4000 > dev.em.0.tx_abs_int_delay=4000 > dev.em.0.rx_processing_limit=4096 > > > I'm trying to run with patch from "em driver, 82574L chip, and > possibly ASPM" thread now under heavy network load. > Hi Lev, What if you try with just the defaults ? Not sure the patch posted by jack and sean will help as its meant for a specific type of em nic, but the RELENG_8 version I am testing with is here http://www.tancsa.com/if_em-8.c. There is a patch to jack's patch by Sean as well thats incorporated in that version ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 15:13:47 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 724081065670; Mon, 7 Feb 2011 15:13:47 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 48B028FC14; Mon, 7 Feb 2011 15:13:47 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p17FDlcE099134; Mon, 7 Feb 2011 15:13:47 GMT (envelope-from jhb@freefall.freebsd.org) Received: (from jhb@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p17FDk5m099130; Mon, 7 Feb 2011 15:13:46 GMT (envelope-from jhb) Date: Mon, 7 Feb 2011 15:13:46 GMT Message-Id: <201102071513.p17FDk5m099130@freefall.freebsd.org> To: sthaug@nethelp.no, jhb@FreeBSD.org, freebsd-net@FreeBSD.org From: jhb@FreeBSD.org Cc: Subject: Re: kern/153951: [ixgbe] Intel 10GBase-LR Ethernet card detected as 10GBase-SR X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 15:13:47 -0000 Synopsis: [ixgbe] Intel 10GBase-LR Ethernet card detected as 10GBase-SR State-Changed-From-To: open->closed State-Changed-By: jhb State-Changed-When: Mon Feb 7 15:13:30 UTC 2011 State-Changed-Why: Submitter reports issue resolved. http://www.freebsd.org/cgi/query-pr.cgi?pr=153951 From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 15:36:58 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0DAE1065670; Mon, 7 Feb 2011 15:36:58 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from sunner.semmy.ru (sunner.semmy.ru [IPv6:2a00:14d0:0:20::3]) by mx1.freebsd.org (Postfix) with ESMTP id 518818FC17; Mon, 7 Feb 2011 15:36:58 +0000 (UTC) Received: from dhcp170-205-red.yandex.net ([95.108.170.205]) by sunner.semmy.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.74 (FreeBSD)) (envelope-from ) id 1PmT95-000Oq3-Vj; Mon, 07 Feb 2011 18:36:55 +0300 Message-ID: <4D501198.6090901@FreeBSD.org> Date: Mon, 07 Feb 2011 18:36:56 +0300 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Julian Elischer References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> In-Reply-To: <4D4DFC95.9010804@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Ivo Vachkov , FreeBSD Net Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 15:36:58 -0000 06.02.2011 4:42, Julian Elischer wrote: > On 2/5/11 4:09 PM, Ivo Vachkov wrote: >> Hello, >> >> How can I help? > > if you have ipv6 connectivity and experience, I have no experience or > connectivity, with it so > I'll be coding blind and will need a tester. > If you have an application for IPV6 testing that would be even better. > Divert is often used for NAT but that doesn't seem very useful for IPv6 and > natd doesn't support it anyhow. Object :) Divert is really useful way to get packets from firewall to userspace, analyse or process them some way and put them back. Really I see no other way for this for IPv6. I've tried ng_socket+ng_nat but there is no easy way to put a packet back in firewall. I'm very interested in the process. And I'm ready to help in testing. From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 17:24:04 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F3E0106564A for ; Mon, 7 Feb 2011 17:24:04 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id EB8DE8FC15 for ; Mon, 7 Feb 2011 17:24:03 +0000 (UTC) Received: by pxi1 with SMTP id 1so1106137pxi.13 for ; Mon, 07 Feb 2011 09:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=FqS1EVS3oeKwS1D/xf8qUX3jZft0MwbMBoB8+a4sSKo=; b=RnwtbZEsUYtHEB1yJKlneMYgE7uXuimC3cfhaFVyGcVQZy+TV6HV8HQi/8Kh++ejJu j80MGCtBEwoHZGuqS/xUDy6ZXLvI0sLj0mIaJs4HDkLDstGSehq494i+cYJjUZ01IkG4 m+Z08VTGwEYsPa1Pedjj6wZuo2IgRvoM9vppQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Jw+7BDSbH8GBXmV8eLl+Eh9ftbwjmgdbZLWBL3Nm3B5TQDKXh1gTf2pEL9oyj9p0OC bLQccTR8GhsdfSfH+dX6j+gUbZgX09nuzjWDNGkvfX6V9tZIpeg9+Gue7CO6P1siCi/D u2lZxGNRnMpBafNKnr8RFISuDSHVI9Q06csRA= MIME-Version: 1.0 Received: by 10.142.241.16 with SMTP id o16mr15648508wfh.86.1297097884598; Mon, 07 Feb 2011 08:58:04 -0800 (PST) Received: by 10.142.246.3 with HTTP; Mon, 7 Feb 2011 08:58:04 -0800 (PST) Date: Mon, 7 Feb 2011 11:58:04 -0500 Message-ID: From: Karim Fodil-Lemelin To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 17:24:04 -0000 Hi List, The issue is with the igb driver from 7.4 RC3 r218406. If the driver runs out of mbuf clusters it simply stops receiving even after the clusters have been freed. igb0@pci0:7:0:0: class=0x020000 card=0x00008086 chip=0x10a78086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet Here is how to reproduce: 1) Set mbuf cluster count very low (4096) 2) do an intensive UDP or TCP transfer Some pointers: - After few seconds or minutes maximum, the traffic does not flow any more across the machine. - At this point we see the InErrors jump in a burst on network iterface igb0. Then, all packets in are counted as In Errors. - Their number is exactly equal to dev.igb.0.mac_stats.missed_packets, and not far from dev.igb.0.mac_stats.xoff_txd (Note that one is on receive the other on send). - Also we see this counter goes non-null: dev.igb.0.mac_stats.recv_no_buff - We tried to enable/disable TSO, Checksum on receive and send, flow control and aim. The problem still shows up, only the time to get there is different. - Forcing the driver to use one queue (static int igb_num_queues = 1;) did not help. - Using igb_header_split=TRUE does help a bit but not unlike disabling hardware acceleration it just takes more time (probably because its using mbufs more instead of clusters and there is 4times more mbufs then clusters). - Using larger values for mbufs cluster also delays the problem We think the code in "igb_refresh_mbufs()" does not handle well running out of mbufs but that is how far we can get at the moment. We are seeking your help and wisdom, and are willing to test patches or suggested settings. Thanks in advance, Karim. From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 17:28:50 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C10681065674; Mon, 7 Feb 2011 17:28:50 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from ns1temp.jnielsen.net (ns1temp.jnielsen.net [69.55.230.42]) by mx1.freebsd.org (Postfix) with ESMTP id 994468FC0A; Mon, 7 Feb 2011 17:28:50 +0000 (UTC) Received: from jnielsen.socialserve.com ([12.249.176.26]) (authenticated bits=0) by ns1temp.jnielsen.net (8.14.3/8.14.3) with ESMTP id p17GsuAi015248 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 7 Feb 2011 11:54:57 -0500 (EST) (envelope-from lists@jnielsen.net) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: John Nielsen In-Reply-To: <4D4DFC95.9010804@freebsd.org> Date: Mon, 7 Feb 2011 11:54:51 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <739B675A-A1EF-4578-B3AF-30674C567B97@jnielsen.net> References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> To: Julian Elischer X-Mailer: Apple Mail (2.1082) X-DCC--Metrics: ns1temp.jnielsen.net; whitelist X-Virus-Scanned: clamav-milter 0.96.5 at ns1temp.jnielsen.net X-Virus-Status: Clean Cc: FreeBSD Net Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 17:28:50 -0000 On Feb 5, 2011, at 8:42 PM, Julian Elischer wrote: > if you have ipv6 connectivity and experience, I have no experience or = connectivity, with it so > I'll be coding blind and will need a tester. > If you have an application for IPV6 testing that would be even better. > Divert is often used for NAT but that doesn't seem very useful for = IPv6 and > natd doesn't support it anyhow. The connectivity bit is easily remedied: I've used http://tunnelbroker.net/ with a FreeBSD router/firewall with = good success. Free IPv6 uplink over a gif IPv4 tunnel. JN From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 17:35:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32DF4106564A for ; Mon, 7 Feb 2011 17:35:10 +0000 (UTC) (envelope-from lev@serebryakov.spb.ru) Received: from ftp.translate.ru (ftp.translate.ru [80.249.188.42]) by mx1.freebsd.org (Postfix) with ESMTP id E607D8FC17 for ; Mon, 7 Feb 2011 17:35:09 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (89.112.15.178.pppoe.eltel.net [89.112.15.178]) (Authenticated sender: lev@serebryakov.spb.ru) by ftp.translate.ru (Postfix) with ESMTPA id 764D013DF42; Mon, 7 Feb 2011 20:35:08 +0300 (MSK) Date: Mon, 7 Feb 2011 20:35:07 +0300 From: Lev Serebryakov X-Priority: 3 (Normal) Message-ID: <10510673199.20110207203507@serebryakov.spb.ru> To: Karim Fodil-Lemelin In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 17:35:10 -0000 Hello, Karim. You wrote 7 =F4=E5=E2=F0=E0=EB=FF 2011 =E3., 19:58:04: > The issue is with the igb driver from 7.4 RC3 r218406. If the driver runs > out of mbuf clusters it simply stops receiving even after the clusters ha= ve > been freed. It looks like my problems with em0 (see thread "em0 hangs without any messages like "Watchdog timeout", only down/up reset it.")... Codebase for em and igb is somewhat common... --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 20:20:11 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5011A106566B for ; Mon, 7 Feb 2011 20:20:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 251C78FC15 for ; Mon, 7 Feb 2011 20:20:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p17KKA66019931 for ; Mon, 7 Feb 2011 20:20:10 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p17KKAWN019930; Mon, 7 Feb 2011 20:20:10 GMT (envelope-from gnats) Date: Mon, 7 Feb 2011 20:20:10 GMT Message-Id: <201102072020.p17KKAWN019930@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: "Vladislav V. Prodan" Cc: Subject: Re: kern/154557: [tcp] Freeze tcp-session of the clients, if in the gateway is used quagga X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "Vladislav V. Prodan" List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 20:20:11 -0000 The following reply was made to PR kern/154557; it has been noted by GNATS. From: "Vladislav V. Prodan" To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/154557: [tcp] Freeze tcp-session of the clients, if in the gateway is used quagga Date: Mon, 07 Feb 2011 22:13:58 +0200 #cat /etc/rc.conf fsck_y_enable="YES" background_fsck="YES" # Attempt to run fsck in the background where po background_fsck_delay="60" # Time to wait (seconds) before starting the fsc zfs_enable="YES" hostname="bgp.nextonmarket.com" ifconfig_re0_alias0="inet XXX.YYY.202.1/23" ifconfig_re0_alias1="inet XXX.YYY.202.2/23" ifconfig_re0_alias2="inet 192.168.1.100" sshd_enable="YES" gateway_enable="YES" ipv6_enable="YES" ipv6_gateway_enable="YES" ipv6_ifconfig_re0="2001::3/64 alias" ipv6_ifconfig_re0_alias0="2001::5" ipv6_ifconfig_re0_alias1="2001::200" ipv6_ifconfig_re0_alias2="2001::201" quagga_enable="YES" quagga_flags="-d -A 127.0.0.1" quagga_daemons="zebra bgpd" ntpdate_enable="YES" ntpdate_flags="ua.pool.ntp.org" ntpd_enable="YES" ifconfig_re0_ipv6="RTADV" rtadvd_enable="YES" rtadvd_interfaces="re0" ifconfig_re1_alias1="192.168.115.177/24" ifconfig_re2_alias0="ether 00:23:69:15:62:A8" ifconfig_re2_alias1="DHCP" bsdstats_enable="YES" bsnmpd_enable="YES" webmin_enable="YES" mpd_enable="YES" named_enable="YES" nginx_enable="YES" firewall_enable="YES" firewall_logging="YES" firewall_type="UNKNOWN" firewall_script="/etc/ipfw/ipfw.conf" Really need to do NAT for each real IP? -- Vladislav V. Prodan VVP24-UANIC +38[067]4584408 +38[099]4060508 vlad11@jabber.ru From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 20:43:26 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 526921065670 for ; Mon, 7 Feb 2011 20:43:26 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 24D538FC0C for ; Mon, 7 Feb 2011 20:43:26 +0000 (UTC) Received: by pzk32 with SMTP id 32so1149665pzk.13 for ; Mon, 07 Feb 2011 12:43:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=grkL+leM5fSBe03umsPCHkgcw9otH4D2hBnaoGTbqoI=; b=A7/CnnPlhv6swCMVO2cHDfuGVbE1vOeZlmhZ1PuZ2ab69M9Q61Z1uKwtGX7FwSMrA0 Yf3fwhKDH81M44vY6sW8/pcO9Oe1sAK+GSlOFB9cVS49GpZSd06oBwxxsvNNgsX/ktWn ZZfplpZG4GR0C3PWJahDu8a2Cc90X9HF1xWUY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=Aw3qIIQHymqOT2aIIrJu3oHbKALQYG2YzzVDR/rrY8MlR6EHWmykTTmfrZSgtEwUlZ OBg19J4WBPiloYs4OYSyD6k2DCSRkCUzNEtX0RztBopqJgcFXf6ux9GJkY62kok7A7u/ znxEE09xld3BYKkbiLIItwQ+ggLQ92LopItfk= MIME-Version: 1.0 Received: by 10.142.131.17 with SMTP id e17mr13231806wfd.371.1297111405664; Mon, 07 Feb 2011 12:43:25 -0800 (PST) Received: by 10.142.246.3 with HTTP; Mon, 7 Feb 2011 12:43:25 -0800 (PST) In-Reply-To: References: <10510673199.20110207203507@serebryakov.spb.ru> Date: Mon, 7 Feb 2011 15:43:25 -0500 Message-ID: From: Karim Fodil-Lemelin To: freebsd-net@freebsd.org Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Fwd: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 20:43:26 -0000 apologies if this gets in twice. ---------- Forwarded message ---------- From: Karim Fodil-Lemelin Date: 2011/2/7 Subject: Re: igb driver tx hangs when out of mbuf clusters To: Lev Serebryakov Cc: freebsd-net@freebsd.org 2011/2/7 Lev Serebryakov Hello, Karim. > You wrote 7 =C6=C5=D7=D2=C1=CC=D1 2011 =C7., 19:58:04: > > > > The issue is with the igb driver from 7.4 RC3 r218406. If the driver ru= ns > > out of mbuf clusters it simply stops receiving even after the clusters > have > > been freed. > It looks like my problems with em0 (see thread "em0 hangs without > any messages like "Watchdog timeout", only down/up reset it.")... > Codebase for em and igb is somewhat common... > > -- > // Black Lion AKA Lev Serebryakov > > I agree. Do you get missed packets in mac_stats (sysctl dev.em | grep missed)? I might not have mentioned but I can also 'fix' the problem by doing ifconfig igb0 down/up. I will try using POLLING to 'automatize' the reset as you mentioned in your thread. Karim. From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 22:33:48 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AF22B1065672 for ; Mon, 7 Feb 2011 22:33:48 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7B7748FC16 for ; Mon, 7 Feb 2011 22:33:48 +0000 (UTC) Received: by pwi10 with SMTP id 10so1210929pwi.13 for ; Mon, 07 Feb 2011 14:33:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=q9syjohT/nbpzreBfvNlZlCAxSWOfFomp6ScMizhlno=; b=E6+0eqZZgwa+OiS+MpmvEDl8z51oZbZn/JzEpmOgbvGF7jBMJr15PaLVKAa1+VGFWa fBTos5RosjSKS7FO2ZDnGPToy4EH0wPwesF7BJhJN3s4DQRucEQZmMy090CjPrRTG2EO yfouMyQ22VqxlsR7PzBOusnI7ak09EO7/azIY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=g9W8YWXRw41jTr6Ee5DaubFkyvFh/7N2xovlIslvLiMuuaMZd+XmA3q2p3hia1P+MG YqEPvo45tBYimdejRjLh7kkuazc39D1NFpB92DQe4Ijb6a3MLNSAc+3XajQCaCbk7wUd 9pTK0tSmDARI5u+xywJlQlmVpMyaZQEvm6K5M= MIME-Version: 1.0 Received: by 10.142.48.17 with SMTP id v17mr16049072wfv.314.1297118027261; Mon, 07 Feb 2011 14:33:47 -0800 (PST) Received: by 10.142.246.3 with HTTP; Mon, 7 Feb 2011 14:33:47 -0800 (PST) In-Reply-To: References: <10510673199.20110207203507@serebryakov.spb.ru> Date: Mon, 7 Feb 2011 17:33:47 -0500 Message-ID: From: Karim Fodil-Lemelin To: freebsd-net@freebsd.org Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Fwd: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 22:33:48 -0000 Subject: Re: igb driver tx hangs when out of mbuf clusters > To: Lev Serebryakov > Cc: freebsd-net@freebsd.org > > > 2011/2/7 Lev Serebryakov > > Hello, Karim. >> You wrote 7 =C6=C5=D7=D2=C1=CC=D1 2011 =C7., 19:58:04: >> >> >> > The issue is with the igb driver from 7.4 RC3 r218406. If the driver >> runs >> > out of mbuf clusters it simply stops receiving even after the clusters >> have >> > been freed. >> It looks like my problems with em0 (see thread "em0 hangs without >> any messages like "Watchdog timeout", only down/up reset it.")... >> Codebase for em and igb is somewhat common... >> >> -- >> // Black Lion AKA Lev Serebryakov >> >> I agree. > > Do you get missed packets in mac_stats (sysctl dev.em | grep missed)? > > I might not have mentioned but I can also 'fix' the problem by doing > ifconfig igb0 down/up. > > I will try using POLLING to 'automatize' the reset as you mentioned in yo= ur > thread. > > Karim. > > Follow up on tests with POLLING: The problem is still occurring although it takes more time ... Outputs of sysctl dev.igb0 and netstat -m will follow: 9219/99426/108645 mbufs in use (current/cache/total) 9217/90783/100000/100000 mbuf clusters in use (current/cache/total/max) 0/640 mbuf+clusters out of packet secondary zone in use (current/cache) 0/12800/12800/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 20738K/257622K/278361K bytes allocated to network (current/cache/total) 0/291/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/5/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7 dev.igb.0.%driver: igb dev.igb.0.%location: slot=3D0 function=3D0 dev.igb.0.%pnpinfo: vendor=3D0x8086 device=3D0x10a7 subvendor=3D0x8086 subdevice=3D0x0000 class=3D0x020000 dev.igb.0.%parent: pci7 dev.igb.0.nvm: -1 dev.igb.0.flow_control: 3 dev.igb.0.enable_aim: 1 dev.igb.0.rx_processing_limit: 100 dev.igb.0.link_irq: 4 dev.igb.0.dropped: 0 dev.igb.0.tx_dma_fail: 0 dev.igb.0.rx_overruns: 464 dev.igb.0.watchdog_timeouts: 0 dev.igb.0.device_control: 1490027073 dev.igb.0.rx_control: 67141658 dev.igb.0.interrupt_mask: 0 dev.igb.0.extended_int_mask: 0 dev.igb.0.tx_buf_alloc: 14 dev.igb.0.rx_buf_alloc: 34 dev.igb.0.fc_high_water: 29488 dev.igb.0.fc_low_water: 29480 dev.igb.0.queue0.interrupt_rate: 111111 dev.igb.0.queue0.txd_head: 877 dev.igb.0.queue0.txd_tail: 877 dev.igb.0.queue0.no_desc_avail: 0 dev.igb.0.queue0.tx_packets: 92013 dev.igb.0.queue0.rxd_head: 570 dev.igb.0.queue0.rxd_tail: 570 dev.igb.0.queue0.rx_packets: 163386 dev.igb.0.queue0.rx_bytes: 240260310 dev.igb.0.queue0.lro_queued: 0 dev.igb.0.queue0.lro_flushed: 0 dev.igb.0.mac_stats.excess_coll: 0 dev.igb.0.mac_stats.single_coll: 0 dev.igb.0.mac_stats.multiple_coll: 0 dev.igb.0.mac_stats.late_coll: 0 dev.igb.0.mac_stats.collision_count: 0 dev.igb.0.mac_stats.symbol_errors: 0 dev.igb.0.mac_stats.sequence_errors: 0 dev.igb.0.mac_stats.defer_count: 0 dev.igb.0.mac_stats.missed_packets: 3104 dev.igb.0.mac_stats.recv_no_buff: 4016 dev.igb.0.mac_stats.recv_undersize: 0 dev.igb.0.mac_stats.recv_fragmented: 0 dev.igb.0.mac_stats.recv_oversize: 0 dev.igb.0.mac_stats.recv_jabber: 0 dev.igb.0.mac_stats.recv_errs: 0 dev.igb.0.mac_stats.crc_errs: 0 dev.igb.0.mac_stats.alignment_errs: 0 dev.igb.0.mac_stats.coll_ext_errs: 0 dev.igb.0.mac_stats.xon_recvd: 0 dev.igb.0.mac_stats.xon_txd: 346 dev.igb.0.mac_stats.xoff_recvd: 0 dev.igb.0.mac_stats.xoff_txd: 3450 dev.igb.0.mac_stats.total_pkts_recvd: 166515 dev.igb.0.mac_stats.good_pkts_recvd: 163411 dev.igb.0.mac_stats.bcast_pkts_recvd: 0 dev.igb.0.mac_stats.mcast_pkts_recvd: 51 dev.igb.0.mac_stats.rx_frames_64: 10 dev.igb.0.mac_stats.rx_frames_65_127: 1601 dev.igb.0.mac_stats.rx_frames_128_255: 53 dev.igb.0.mac_stats.rx_frames_256_511: 42 dev.igb.0.mac_stats.rx_frames_512_1023: 18 dev.igb.0.mac_stats.rx_frames_1024_1522: 161687 dev.igb.0.mac_stats.good_octets_recvd: 240948229 dev.igb.0.mac_stats.good_octets_txd: 5947150 dev.igb.0.mac_stats.total_pkts_txd: 95809 dev.igb.0.mac_stats.good_pkts_txd: 92013 dev.igb.0.mac_stats.bcast_pkts_txd: 1516 dev.igb.0.mac_stats.mcast_pkts_txd: 1817 dev.igb.0.mac_stats.tx_frames_64: 90302 dev.igb.0.mac_stats.tx_frames_65_127: 1711 dev.igb.0.mac_stats.tx_frames_128_255: 0 dev.igb.0.mac_stats.tx_frames_256_511: 0 dev.igb.0.mac_stats.tx_frames_512_1023: 0 dev.igb.0.mac_stats.tx_frames_1024_1522: 0 dev.igb.0.mac_stats.tso_txd: 0 dev.igb.0.mac_stats.tso_ctx_fail: 0 dev.igb.0.interrupts.asserts: 5584 dev.igb.0.interrupts.rx_pkt_timer: 163411 dev.igb.0.interrupts.rx_abs_timer: 163386 dev.igb.0.interrupts.tx_pkt_timer: 92013 dev.igb.0.interrupts.tx_abs_timer: 0 dev.igb.0.interrupts.tx_queue_empty: 92013 dev.igb.0.interrupts.tx_queue_min_thresh: 0 dev.igb.0.interrupts.rx_desc_min_thresh: 19 dev.igb.0.interrupts.rx_overrun: 0 dev.igb.0.host.breaker_tx_pkt: 0 dev.igb.0.host.host_tx_pkt_discard: 0 dev.igb.0.host.rx_pkt: 0 dev.igb.0.host.breaker_rx_pkts: 0 dev.igb.0.host.breaker_rx_pkt_drop: 0 dev.igb.0.host.tx_good_pkt: 0 dev.igb.0.host.breaker_tx_pkt_drop: 0 dev.igb.0.host.rx_good_bytes: 240948229 dev.igb.0.host.tx_good_bytes: 5947150 dev.igb.0.host.length_errors: 0 dev.igb.0.host.serdes_violation_pkt: 0 dev.igb.0.host.header_redir_missed: 0 From owner-freebsd-net@FreeBSD.ORG Mon Feb 7 23:58:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CE52106564A; Mon, 7 Feb 2011 23:58:10 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id ABD8F8FC08; Mon, 7 Feb 2011 23:58:09 +0000 (UTC) Received: by ywl2 with SMTP id 2so391744ywl.13 for ; Mon, 07 Feb 2011 15:58:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=Z0esuCNU3IkUjJVQ9b1JoP5We2a/PEiUqWhE2TaMonE=; b=u8vwy/NIiHrX5kr63xUgrYBvgjzoAXsKLo8xD3H7+p/HPuPDq8OcrvTxZ9cH151l8m maqAbsnaABBGkW8tsIECP5cjYBJ+L8I+VyY4GYL6grVLOeRGB7z3Zlf7WPEnplnDZW4m 7tXUqWa19xIA+ed49W0ZUL0xt8ayP8Fh62qhg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent; b=SUeMsOhx9Gj4lXy7om/kPxEA7UfT7xhlWxBv6+uD/Vtv/Zuv9BZ2i9xValfp5ej0mF DVZtjT06QPjyiULc/D17+Kp0ZR7Sp0cBXIDQk8mlxeil9FnkOicOG6m+85i1TOyG7YYD ZJVfiHAnsc30biMfGHU+C1eAo5W+0BH5qYTeg= Received: by 10.100.165.20 with SMTP id n20mr9987419ane.114.1297123088626; Mon, 07 Feb 2011 15:58:08 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 17sm6255207anx.33.2011.02.07.15.58.06 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 07 Feb 2011 15:58:07 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 7 Feb 2011 15:58:12 -0800 From: Pyun YongHyeon Date: Mon, 7 Feb 2011 15:58:12 -0800 To: Karim Fodil-Lemelin Message-ID: <20110207235811.GA1306@michelle.cdnetworks.com> References: <10510673199.20110207203507@serebryakov.spb.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: jfv@FreeBSD.org, freebsd-net@freebsd.org Subject: Re: Fwd: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 23:58:10 -0000 On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin wrote: > Subject: Re: igb driver tx hangs when out of mbuf clusters > > > To: Lev Serebryakov > > Cc: freebsd-net@freebsd.org > > > > > > 2011/2/7 Lev Serebryakov > > > > Hello, Karim. > >> You wrote 7 Ñ„ÐµÐ²Ñ€Ð°Ð»Ñ 2011 г., 19:58:04: > >> > >> > >> > The issue is with the igb driver from 7.4 RC3 r218406. If the driver > >> runs > >> > out of mbuf clusters it simply stops receiving even after the clusters > >> have > >> > been freed. > >> It looks like my problems with em0 (see thread "em0 hangs without > >> any messages like "Watchdog timeout", only down/up reset it.")... > >> Codebase for em and igb is somewhat common... > >> > >> -- > >> // Black Lion AKA Lev Serebryakov > >> > >> I agree. > > > > Do you get missed packets in mac_stats (sysctl dev.em | grep missed)? > > > > I might not have mentioned but I can also 'fix' the problem by doing > > ifconfig igb0 down/up. > > > > I will try using POLLING to 'automatize' the reset as you mentioned in your > > thread. > > > > Karim. > > > > > Follow up on tests with POLLING: The problem is still occurring although it > takes more time ... Outputs of sysctl dev.igb0 and netstat -m will follow: > > 9219/99426/108645 mbufs in use (current/cache/total) > 9217/90783/100000/100000 mbuf clusters in use (current/cache/total/max) Do you see network processes are stuck in keglim state? If you see that I think that's not trivial to solve. You wouldn't even kill that process if it is under keglim state unless some more mbuf clusters are freed from other places. I think both igb(4) and em(4) pass received frame to upper stack before allocating new RX buffer. If driver fails to allocate new RX buffer driver will try to refill RX buffers in next run. Under extreme resource shortage case, this situation can produce no more RX buffers in RX descriptor ring and this will take the box out of network. Other drivers avoid that situation by allocating new RX buffer before passing received frame to upper stack. If RX buffer allocation fails driver will just reuse old RX buffer without passing received frame to upper stack. That does not completely solve the keglim issue though. I think you should have enough mbuf cluters to avoid keglim. However the output above indicates you have enough free mbuf clusters. So I guess igb(4) encountered zero available RX buffer situation in past but failed to refill the RX buffer again. I guess driver may be able to periodically check available RX buffers. Jack may have better idea if this was the case.(CCed) > 0/640 mbuf+clusters out of packet secondary zone in use (current/cache) > 0/12800/12800/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 20738K/257622K/278361K bytes allocated to network (current/cache/total) > 0/291/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/5/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=0 function=0 > dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086 > subdevice=0x0000 class=0x020000 > dev.igb.0.%parent: pci7 > dev.igb.0.nvm: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.rx_processing_limit: 100 > dev.igb.0.link_irq: 4 > dev.igb.0.dropped: 0 > dev.igb.0.tx_dma_fail: 0 > dev.igb.0.rx_overruns: 464 > dev.igb.0.watchdog_timeouts: 0 > dev.igb.0.device_control: 1490027073 > dev.igb.0.rx_control: 67141658 > dev.igb.0.interrupt_mask: 0 > dev.igb.0.extended_int_mask: 0 > dev.igb.0.tx_buf_alloc: 14 > dev.igb.0.rx_buf_alloc: 34 > dev.igb.0.fc_high_water: 29488 > dev.igb.0.fc_low_water: 29480 > dev.igb.0.queue0.interrupt_rate: 111111 > dev.igb.0.queue0.txd_head: 877 > dev.igb.0.queue0.txd_tail: 877 > dev.igb.0.queue0.no_desc_avail: 0 > dev.igb.0.queue0.tx_packets: 92013 > dev.igb.0.queue0.rxd_head: 570 > dev.igb.0.queue0.rxd_tail: 570 > dev.igb.0.queue0.rx_packets: 163386 > dev.igb.0.queue0.rx_bytes: 240260310 > dev.igb.0.queue0.lro_queued: 0 > dev.igb.0.queue0.lro_flushed: 0 > dev.igb.0.mac_stats.excess_coll: 0 > dev.igb.0.mac_stats.single_coll: 0 > dev.igb.0.mac_stats.multiple_coll: 0 > dev.igb.0.mac_stats.late_coll: 0 > dev.igb.0.mac_stats.collision_count: 0 > dev.igb.0.mac_stats.symbol_errors: 0 > dev.igb.0.mac_stats.sequence_errors: 0 > dev.igb.0.mac_stats.defer_count: 0 > dev.igb.0.mac_stats.missed_packets: 3104 > dev.igb.0.mac_stats.recv_no_buff: 4016 > dev.igb.0.mac_stats.recv_undersize: 0 > dev.igb.0.mac_stats.recv_fragmented: 0 > dev.igb.0.mac_stats.recv_oversize: 0 > dev.igb.0.mac_stats.recv_jabber: 0 > dev.igb.0.mac_stats.recv_errs: 0 > dev.igb.0.mac_stats.crc_errs: 0 > dev.igb.0.mac_stats.alignment_errs: 0 > dev.igb.0.mac_stats.coll_ext_errs: 0 > dev.igb.0.mac_stats.xon_recvd: 0 > dev.igb.0.mac_stats.xon_txd: 346 > dev.igb.0.mac_stats.xoff_recvd: 0 > dev.igb.0.mac_stats.xoff_txd: 3450 > dev.igb.0.mac_stats.total_pkts_recvd: 166515 > dev.igb.0.mac_stats.good_pkts_recvd: 163411 > dev.igb.0.mac_stats.bcast_pkts_recvd: 0 > dev.igb.0.mac_stats.mcast_pkts_recvd: 51 > dev.igb.0.mac_stats.rx_frames_64: 10 > dev.igb.0.mac_stats.rx_frames_65_127: 1601 > dev.igb.0.mac_stats.rx_frames_128_255: 53 > dev.igb.0.mac_stats.rx_frames_256_511: 42 > dev.igb.0.mac_stats.rx_frames_512_1023: 18 > dev.igb.0.mac_stats.rx_frames_1024_1522: 161687 > dev.igb.0.mac_stats.good_octets_recvd: 240948229 > dev.igb.0.mac_stats.good_octets_txd: 5947150 > dev.igb.0.mac_stats.total_pkts_txd: 95809 > dev.igb.0.mac_stats.good_pkts_txd: 92013 > dev.igb.0.mac_stats.bcast_pkts_txd: 1516 > dev.igb.0.mac_stats.mcast_pkts_txd: 1817 > dev.igb.0.mac_stats.tx_frames_64: 90302 > dev.igb.0.mac_stats.tx_frames_65_127: 1711 > dev.igb.0.mac_stats.tx_frames_128_255: 0 > dev.igb.0.mac_stats.tx_frames_256_511: 0 > dev.igb.0.mac_stats.tx_frames_512_1023: 0 > dev.igb.0.mac_stats.tx_frames_1024_1522: 0 > dev.igb.0.mac_stats.tso_txd: 0 > dev.igb.0.mac_stats.tso_ctx_fail: 0 > dev.igb.0.interrupts.asserts: 5584 > dev.igb.0.interrupts.rx_pkt_timer: 163411 > dev.igb.0.interrupts.rx_abs_timer: 163386 > dev.igb.0.interrupts.tx_pkt_timer: 92013 > dev.igb.0.interrupts.tx_abs_timer: 0 > dev.igb.0.interrupts.tx_queue_empty: 92013 > dev.igb.0.interrupts.tx_queue_min_thresh: 0 > dev.igb.0.interrupts.rx_desc_min_thresh: 19 > dev.igb.0.interrupts.rx_overrun: 0 > dev.igb.0.host.breaker_tx_pkt: 0 > dev.igb.0.host.host_tx_pkt_discard: 0 > dev.igb.0.host.rx_pkt: 0 > dev.igb.0.host.breaker_rx_pkts: 0 > dev.igb.0.host.breaker_rx_pkt_drop: 0 > dev.igb.0.host.tx_good_pkt: 0 > dev.igb.0.host.breaker_tx_pkt_drop: 0 > dev.igb.0.host.rx_good_bytes: 240948229 > dev.igb.0.host.tx_good_bytes: 5947150 > dev.igb.0.host.length_errors: 0 > dev.igb.0.host.serdes_violation_pkt: 0 > dev.igb.0.host.header_redir_missed: 0 From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 00:33:50 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CEF5C106564A for ; Tue, 8 Feb 2011 00:33:50 +0000 (UTC) (envelope-from cowbert@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 663288FC0A for ; Tue, 8 Feb 2011 00:33:50 +0000 (UTC) Received: by eyf6 with SMTP id 6so2696998eyf.13 for ; Mon, 07 Feb 2011 16:33:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=yaTulFgaxy92d71ai0hxG1d9Qd7SNUbOpbdcwjsUP5c=; b=N9tuMqhSx8ZHjhLib/qXRIBMyrnsAmKOUD6oxEqjHhDlsiM0MLMBv/qDM31WnHFqQC 2eo/VIEsYK/AQhdHV0SiMisff6TNjDexWuvNDJiuia5GlQi/E51q/Cs64Fdvmiqp4KRO Hic0lmVUDcfxFf1xp+i7KpUOFVhsGNLgrosvA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Q74rXRc5hjLIvDv4AwZlVpgHyrk/FLAFtawJmtANtQqGjCKkdLeTW/GTA/fnEQz7lM Qd/ORnrnxYMcLz5qXyY/L/4qEwOFCMsziPhJSoyV1p4dzCA+qDKfbHfDPnuxR/kCITSx cLReuY0Fc4Jp+fufSKz4Lr1BkVSzpEUDhULko= MIME-Version: 1.0 Received: by 10.213.28.141 with SMTP id m13mr8911458ebc.50.1297123756372; Mon, 07 Feb 2011 16:09:16 -0800 (PST) Received: by 10.213.31.65 with HTTP; Mon, 7 Feb 2011 16:09:16 -0800 (PST) Date: Mon, 7 Feb 2011 18:09:16 -0600 Message-ID: From: Peter Lai To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: bge wedging 8.2-RC1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 00:33:50 -0000 Hello I've got a new Dell Precision workstation here with a BCM5761 on intel mobo for westmere xeons that is wedging with interrupt storm and will lockup the system randomly. I have turned HTT and auto powermanagement off in bios (system cannot sleep), lowest cpu acpi state is C1. Here is dmesg: bge0: mem 0xf3be0000-0xf3beffff,0xf3bf0000-0xf3bfffff irq 17 at device 0.0 on pci6 bge0: CHIP ID 0x05761100; ASIC REV 0x5761; CHIP REV 0x57611; PCI-E miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow Here is pciconf -lv: bge0@pci0:6:0:0: class=0x020000 card=0x026d1028 chip=0x168114e4 rev=0x10 hdr=0x00 vendor = 'Broadcom Corporation' device = 'Broadcom 57XX Gigabit Integrated Controller (BCM5761)' class = network subclass = ethernet here is the setup in rc.conf: ifconfig_bge0="polling -tso -vlanhwtso -vlanhwtag -vlanmtu inet 192.168.123.124 netmask 255.255.255.0" I have the card plugged into a dlink DSS8 100mbps switch with one other 100mbps device on it (rich man's crossover cable). Before turning off TSO4 and VLAN tagging (because I don't use them), the card would do several things: 1. 1 out of 3 reboots: Fail to bring interface up. ifconfig would hang and systat/vmstat showed 800+ interrupts per second on IRQ256 2. After a few hours lock up the system, requiring hard reboot After disabling TSO4 and VLAN stuff: bge0: flags=8802 metric 0 mtu 1500 options=80083 media: Ethernet autoselect (100baseTX ) Everything seemed fine for about two weeks and then suddenly started acting up again, locked up, after hard reboot, soft reboot, link will not come up and I see interrupt storm again.... I am close to buying an intel card to replace the bcm, but then I noticed that the main intel desktop PCI-E card is 82574L-based and people are having em driver wedging on that too. So now I have broken ethernet on this box; my primary link is atheros 5212 pci card and I may be out of pci slots (or else I might try a pci intel card). From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 01:18:32 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27B97106566C for ; Tue, 8 Feb 2011 01:18:32 +0000 (UTC) (envelope-from moonlightakkiy@yahoo.ca) Received: from web39320.mail.mud.yahoo.com (web39320.mail.mud.yahoo.com [66.94.238.192]) by mx1.freebsd.org (Postfix) with SMTP id A45C58FC18 for ; Tue, 8 Feb 2011 01:18:31 +0000 (UTC) Received: (qmail 19359 invoked by uid 60001); 8 Feb 2011 01:18:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.ca; s=s1024; t=1297127911; bh=19u4Z4CsgjzxgA7ZISn0VT9ufzydM34BWiX+HeOH6Vs=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=MlLjkmJKr+WavEPJJSVnV7qcYUyFvB5o6x9KQLXG3JuiLEcihBhz7s6sw5MMhwlxDcLhcz4xk9WFt1qAoChnR3WqxI6iOSjqSDRkoO3+RSf9EU3Fs8j6CUzSYAsssR66ku2hjzcvcrAh3Zmy+nHNb4bqky7DJ875ocjwVb3RJLA= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.ca; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=eMXsoYXR94GnCpGqdBzr5OKv1oEqwtkN/JZNNvOcAkqsOitdivgUXhTq40hk2Hb+8X0oAmP+I/s4QhtTpYbaXu4bnUlF5KCnNCmr+zk4nwor3hZaYvLG1p1rUgHy4UgRTIaRgKtU8IcCX3NK/a8EahUnKJMhUOAmKP8OdUFbUj0=; Message-ID: <95679.17379.qm@web39320.mail.mud.yahoo.com> X-YMail-OSG: BAgeAdkVM1kbove23cV6hDQGpaLnXaCKSrGAQmFWNhHEekK RTdh5XWv1C4gxCpttuf0CK1IXEcyObJzxWPNNsU46RDor51I5mtNgD8g8T24 mrrXcP39W7g9WlQ2kozwnCHCJiH2fjG99BO4EDflw9oblNKOZOyqMhnnXQEV 01rRW3OxTNS2bK7YtaMErhjgpn.SQa7l0JSWP6C4ciQJ7nEDxiiN2tPCpfWE 4IEGXzDP4K298vrMG.3qxgqfzvk31RZsNMHCkFIEwc1M09sMAw2z28FldcKR nj7c0yf2OIhZy.8h2SYS4Sdg.pF_ViTiYBkvT9ziz6WbVACOrG592DyAnYVM Zyo49od.Ymw-- Received: from [206.75.146.55] by web39320.mail.mud.yahoo.com via HTTP; Mon, 07 Feb 2011 17:18:30 PST X-Mailer: YahooMailRC/555 YahooMailWebService/0.8.108.291010 References: <20110204060808.GA97298@gw.zagrebin.ru> <201102041314.17939.bschmidt@freebsd.org> <147585.89029.qm@web39304.mail.mud.yahoo.com> <201102061142.43865.bschmidt@freebsd.org> Date: Mon, 7 Feb 2011 17:18:30 -0800 (PST) From: PseudoCylon To: Bernhard Schmidt In-Reply-To: <201102061142.43865.bschmidt@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="0-1961750709-1297127910=:17379" Cc: freebsd-net@freebsd.org, Alexander Zagrebin Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 01:18:32 -0000 --0-1961750709-1297127910=:17379 Content-Type: text/plain; charset=us-ascii ----- Original Message ---- > From: Bernhard Schmidt > To: PseudoCylon > Cc: Alexander Zagrebin ; freebsd-net@freebsd.org > Sent: Sun, February 6, 2011 3:42:43 AM > Subject: Re: if_run in hostap mode: issue with stations in the power save mode Afaik iwn(4) doesn't use PS, never got around implementing that. > > > I'd like to move ieee80211_beacon_alloc() into iv_vap_alloc(). Then > > we don't need to test beacon_mbuf == NULL in run_update_beacon_cb(), > > and there is already switch we can use for conditionally alloc mem. > > Sounds fine with we. > Oops, there is switch before malloc vap. the test is still in run_update_beacon_cb() > Can I talk you into integrating that into Alexander's patch? > The patch is attached. (diff to HEAD) Bit long, just because there is a couple of new call back functions to avoid LOR. AK --0-1961750709-1297127910=:17379 Content-Type: application/octet-stream; name="run.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="run.patch" ZGlmZiAtLWdpdCBhL2Rldi91c2Ivd2xhbi9pZl9ydW4uYyBiL2Rldi91c2Iv d2xhbi9pZl9ydW4uYwppbmRleCBkZjcyZjcyLi4xY2QxNTkxIDEwMDY0NAot LS0gYS9kZXYvdXNiL3dsYW4vaWZfcnVuLmMKKysrIGIvZGV2L3VzYi93bGFu L2lmX3J1bi5jCkBAIC0zODgsNiArMzg4LDcgQEAgc3RhdGljIHZvaWQJcnVu X3NjYW5fZW5kKHN0cnVjdCBpZWVlODAyMTFjb20gKik7CiBzdGF0aWMgdm9p ZAlydW5fdXBkYXRlX2JlYWNvbihzdHJ1Y3QgaWVlZTgwMjExdmFwICosIGlu dCk7CiBzdGF0aWMgdm9pZAlydW5fdXBkYXRlX2JlYWNvbl9jYih2b2lkICop Owogc3RhdGljIHZvaWQJcnVuX3VwZGF0ZXByb3Qoc3RydWN0IGllZWU4MDIx MWNvbSAqKTsKK3N0YXRpYyB2b2lkCXJ1bl91cGRhdGVwcm90X2NiKHZvaWQg Kik7CiBzdGF0aWMgdm9pZAlydW5fdXNiX3RpbWVvdXRfY2Iodm9pZCAqKTsK IHN0YXRpYyB2b2lkCXJ1bl9yZXNldF9saXZlbG9jayhzdHJ1Y3QgcnVuX3Nv ZnRjICopOwogc3RhdGljIHZvaWQJcnVuX2VuYWJsZV90c2Zfc3luYyhzdHJ1 Y3QgcnVuX3NvZnRjICopOwpAQCAtMzk4LDYgKzM5OSw3IEBAIHN0YXRpYyB2 b2lkCXJ1bl9zZXRfbGVkcyhzdHJ1Y3QgcnVuX3NvZnRjICosIHVpbnQxNl90 KTsKIHN0YXRpYyB2b2lkCXJ1bl9zZXRfYnNzaWQoc3RydWN0IHJ1bl9zb2Z0 YyAqLCBjb25zdCB1aW50OF90ICopOwogc3RhdGljIHZvaWQJcnVuX3NldF9t YWNhZGRyKHN0cnVjdCBydW5fc29mdGMgKiwgY29uc3QgdWludDhfdCAqKTsK IHN0YXRpYyB2b2lkCXJ1bl91cGRhdGVzbG90KHN0cnVjdCBpZm5ldCAqKTsK K3N0YXRpYyB2b2lkCXJ1bl91cGRhdGVzbG90X2NiKHZvaWQgKik7CiBzdGF0 aWMgdm9pZAlydW5fdXBkYXRlX21jYXN0KHN0cnVjdCBpZm5ldCAqKTsKIHN0 YXRpYyBpbnQ4X3QJcnVuX3Jzc2kyZGJtKHN0cnVjdCBydW5fc29mdGMgKiwg dWludDhfdCwgdWludDhfdCk7CiBzdGF0aWMgdm9pZAlydW5fdXBkYXRlX3By b21pc2NfbG9ja2VkKHN0cnVjdCBpZm5ldCAqKTsKQEAgLTY3NCw3ICs2NzYs NyBAQCBydW5fYXR0YWNoKGRldmljZV90IHNlbGYpCiAJaWMtPmljX3NldF9j aGFubmVsID0gcnVuX3NldF9jaGFubmVsOwogCWljLT5pY19ub2RlX2FsbG9j ID0gcnVuX25vZGVfYWxsb2M7CiAJaWMtPmljX25ld2Fzc29jID0gcnVuX25l d2Fzc29jOwotCS8vaWMtPmljX3VwZGF0ZXNsb3QgPSBydW5fdXBkYXRlc2xv dDsKKwlpYy0+aWNfdXBkYXRlc2xvdCA9IHJ1bl91cGRhdGVzbG90OwogCWlj LT5pY191cGRhdGVfbWNhc3QgPSBydW5fdXBkYXRlX21jYXN0OwogCWljLT5p Y193bWUud21lX3VwZGF0ZSA9IHJ1bl93bWVfdXBkYXRlOwogCWljLT5pY19y YXdfeG1pdCA9IHJ1bl9yYXdfeG1pdDsKQEAgLTg1Niw2ICs4NTgsOCBAQCBy dW5fdmFwX2RlbGV0ZShzdHJ1Y3QgaWVlZTgwMjExdmFwICp2YXApCiAKIAlS VU5fTE9DSyhzYyk7CiAKKwltX2ZyZWVtKHJ2cC0+YmVhY29uX21idWYpOwor CiAJcnZwX2lkID0gcnZwLT5ydnBfaWQ7CiAJc2MtPnJhdGVjdGxfcnVuICY9 IH4oMSA8PCBydnBfaWQpOwogCXNjLT5ydnBfYm1hcCAmPSB+KDEgPDwgcnZw X2lkKTsKQEAgLTM5MDEsOCArMzkwNSwyOSBAQCBydW5fdXBkYXRlX2JlYWNv bihzdHJ1Y3QgaWVlZTgwMjExdmFwICp2YXAsIGludCBpdGVtKQogewogCXN0 cnVjdCBpZWVlODAyMTFjb20gKmljID0gdmFwLT5pdl9pYzsKIAlzdHJ1Y3Qg cnVuX3NvZnRjICpzYyA9IGljLT5pY19pZnAtPmlmX3NvZnRjOworCXN0cnVj dCBydW5fdmFwICpydnAgPSBSVU5fVkFQKHZhcCk7CisJaW50IG1jYXN0ID0g MDsKIAl1aW50MzJfdCBpOwogCisJS0FTU0VSVCh2YXAgIT0gTlVMTCwgKCJu byBiZWFjb24iKSk7CisKKwlzd2l0Y2ggKGl0ZW0pIHsKKwljYXNlIElFRUU4 MDIxMV9CRUFDT05fRVJQOgorCQlydW5fdXBkYXRlc2xvdChpYy0+aWNfaWZw KTsKKwkJYnJlYWs7CisJY2FzZSBJRUVFODAyMTFfQkVBQ09OX0hUSU5GTzoK KwkJcnVuX3VwZGF0ZXByb3QoaWMpOworCQlicmVhazsKKwljYXNlIElFRUU4 MDIxMV9CRUFDT05fVElNOgorCQltY2FzdCA9IDE7CS8qVE9ETyovCisJCWJy ZWFrOworCWRlZmF1bHQ6CisJCWJyZWFrOworCX0KKworCXNldGJpdChydnAt PmJvLmJvX2ZsYWdzLCBpdGVtKTsKKwlpZWVlODAyMTFfYmVhY29uX3VwZGF0 ZSh2YXAtPml2X2JzcywgJnJ2cC0+Ym8sIHJ2cC0+YmVhY29uX21idWYsIG1j YXN0KTsKKwogCWkgPSBSVU5fQ01EUV9HRVQoJnNjLT5jbWRxX3N0b3JlKTsK IAlEUFJJTlRGKCJjbWRxX3N0b3JlPSVkXG4iLCBpKTsKIAlzYy0+Y21kcVtp XS5mdW5jID0gcnVuX3VwZGF0ZV9iZWFjb25fY2I7CkBAIC0zOTE2LDE3ICsz OTQxLDI1IEBAIHN0YXRpYyB2b2lkCiBydW5fdXBkYXRlX2JlYWNvbl9jYih2 b2lkICphcmcpCiB7CiAJc3RydWN0IGllZWU4MDIxMXZhcCAqdmFwID0gYXJn OworCXN0cnVjdCBydW5fdmFwICpydnAgPSBSVU5fVkFQKHZhcCk7CiAJc3Ry dWN0IGllZWU4MDIxMWNvbSAqaWMgPSB2YXAtPml2X2ljOwogCXN0cnVjdCBy dW5fc29mdGMgKnNjID0gaWMtPmljX2lmcC0+aWZfc29mdGM7CiAJc3RydWN0 IHJ0Mjg2MF90eHdpIHR4d2k7Ci0Jc3RydWN0IG1idWYgKm07CisJc3RydWN0 IG1idWYgKm0gPSBydnAtPmJlYWNvbl9tYnVmOwogCXVpbnQ4X3QgcmlkeDsK IAogCWlmICh2YXAtPml2X2Jzcy0+bmlfY2hhbiA9PSBJRUVFODAyMTFfQ0hB Tl9BTllDKQogCQlyZXR1cm47CiAKLQlpZiAoKG0gPSBpZWVlODAyMTFfYmVh Y29uX2FsbG9jKHZhcC0+aXZfYnNzLCAmUlVOX1ZBUCh2YXApLT5ibykpID09 IE5VTEwpCi0JICAgICAgICByZXR1cm47CisJLyoKKwkgKiBpZWVlODAyMTFf YmVhY29uX3VwZGF0ZSgpIGlzIGNhbGxlZCBpbiBydW5fdXBkYXRlX2JlYWNv bigpCisJICogJ3J2cC0+YmVhY29uX21idWYgPT0gTlVMTCcgbWVhbnMgbm90 aGluZyB0byB1cGRhdGUgKGNhbGxlZCBmaXJzdCB0aW1lKQorCSAqLworCWlm IChydnAtPmJlYWNvbl9tYnVmID09IE5VTEwpIHsKKwkJaWYgKChydnAtPmJl YWNvbl9tYnVmID0gbSA9CisJCSAgICBpZWVlODAyMTFfYmVhY29uX2FsbG9j KHZhcC0+aXZfYnNzLCAmcnZwLT5ibykpID09IE5VTEwpCisJCQlyZXR1cm47 CisJfQogCiAJbWVtc2V0KCZ0eHdpLCAwLCBzaXplb2YgdHh3aSk7CiAJdHh3 aS53Y2lkID0gMHhmZjsKQEAgLTM5NDEsMTMgKzM5NzQsMTEgQEAgcnVuX3Vw ZGF0ZV9iZWFjb25fY2Iodm9pZCAqYXJnKQogCXR4d2kuZmxhZ3MgPSBSVDI4 NjBfVFhfVFM7CiAJdHh3aS54ZmxhZ3MgPSBSVDI4NjBfVFhfTlNFUTsKIAot CXJ1bl93cml0ZV9yZWdpb25fMShzYywgUlQyODYwX0JDTl9CQVNFKFJVTl9W QVAodmFwKS0+cnZwX2lkKSwKKwlydW5fd3JpdGVfcmVnaW9uXzEoc2MsIFJU Mjg2MF9CQ05fQkFTRShydnAtPnJ2cF9pZCksCiAJICAgICh1aW50OF90ICop JnR4d2ksIHNpemVvZiB0eHdpKTsKLQlydW5fd3JpdGVfcmVnaW9uXzEoc2Ms IFJUMjg2MF9CQ05fQkFTRShSVU5fVkFQKHZhcCktPnJ2cF9pZCkgKyBzaXpl b2YgdHh3aSwKKwlydW5fd3JpdGVfcmVnaW9uXzEoc2MsIFJUMjg2MF9CQ05f QkFTRShydnAtPnJ2cF9pZCkgKyBzaXplb2YgdHh3aSwKIAkgICAgbXRvZCht LCB1aW50OF90ICopLCAobS0+bV9wa3RoZHIubGVuICsgMSkgJiB+MSk7CS8q IHJvdW5kdXAgbGVuICovCiAKLQltX2ZyZWVtKG0pOwotCiAJcmV0dXJuOwog fQogCkBAIC0zOTU1LDYgKzM5ODYsMjAgQEAgc3RhdGljIHZvaWQKIHJ1bl91 cGRhdGVwcm90KHN0cnVjdCBpZWVlODAyMTFjb20gKmljKQogewogCXN0cnVj dCBydW5fc29mdGMgKnNjID0gaWMtPmljX2lmcC0+aWZfc29mdGM7CisJdWlu dDMyX3QgaTsKKworCWkgPSBSVU5fQ01EUV9HRVQoJnNjLT5jbWRxX3N0b3Jl KTsKKwlEUFJJTlRGKCJjbWRxX3N0b3JlPSVkXG4iLCBpKTsKKwlzYy0+Y21k cVtpXS5mdW5jID0gcnVuX3VwZGF0ZXByb3RfY2I7CisJc2MtPmNtZHFbaV0u YXJnMCA9IGljOworCWllZWU4MDIxMV9ydW50YXNrKGljLCAmc2MtPmNtZHFf dGFzayk7Cit9CisKK3N0YXRpYyB2b2lkCitydW5fdXBkYXRlcHJvdF9jYih2 b2lkICphcmcpCit7CisJc3RydWN0IGllZWU4MDIxMWNvbSAqaWMgPSBhcmc7 CisJc3RydWN0IHJ1bl9zb2Z0YyAqc2MgPSBpYy0+aWNfaWZwLT5pZl9zb2Z0 YzsKIAl1aW50MzJfdCB0bXA7CiAKIAl0bXAgPSBSVDI4NjBfUlRTVEhfRU4g fCBSVDI4NjBfUFJPVF9OQVZfU0hPUlQgfCBSVDI4NjBfVFhPUF9BTExPV19B TEw7CkBAIC00MTY0LDEyICs0MjA5LDI5IEBAIHJ1bl9zZXRfbWFjYWRkcihz dHJ1Y3QgcnVuX3NvZnRjICpzYywgY29uc3QgdWludDhfdCAqYWRkcikKIAkg ICAgYWRkcls0XSB8IGFkZHJbNV0gPDwgOCB8IDB4ZmYgPDwgMTYpOwogfQog Ci0vKiBBUkdTVVNFRCAqLwogc3RhdGljIHZvaWQKIHJ1bl91cGRhdGVzbG90 KHN0cnVjdCBpZm5ldCAqaWZwKQogewogCXN0cnVjdCBydW5fc29mdGMgKnNj ID0gaWZwLT5pZl9zb2Z0YzsKIAlzdHJ1Y3QgaWVlZTgwMjExY29tICppYyA9 IGlmcC0+aWZfbDJjb207CisJdWludDMyX3QgaTsKKworCWkgPSBSVU5fQ01E UV9HRVQoJnNjLT5jbWRxX3N0b3JlKTsKKwlEUFJJTlRGKCJjbWRxX3N0b3Jl PSVkXG4iLCBpKTsKKwlzYy0+Y21kcVtpXS5mdW5jID0gcnVuX3VwZGF0ZXNs b3RfY2I7CisJc2MtPmNtZHFbaV0uYXJnMCA9IGlmcDsKKwlpZWVlODAyMTFf cnVudGFzayhpYywgJnNjLT5jbWRxX3Rhc2spOworCisJcmV0dXJuOworfQor CisvKiBBUkdTVVNFRCAqLworc3RhdGljIHZvaWQKK3J1bl91cGRhdGVzbG90 X2NiKHZvaWQgKmFyZykKK3sKKwlzdHJ1Y3QgaWZuZXQgKmlmcCA9IGFyZzsK KwlzdHJ1Y3QgcnVuX3NvZnRjICpzYyA9IGlmcC0+aWZfc29mdGM7CisJc3Ry dWN0IGllZWU4MDIxMWNvbSAqaWMgPSBpZnAtPmlmX2wyY29tOwogCXVpbnQz Ml90IHRtcDsKIAogCXJ1bl9yZWFkKHNjLCBSVDI4NjBfQktPRkZfU0xPVF9D RkcsICZ0bXApOwpAQCAtNDc0Nyw3ICs0ODA5LDcgQEAgcnVuX2luaXRfbG9j a2VkKHN0cnVjdCBydW5fc29mdGMgKnNjKQogCXJ1bl9zZXRfY2hhbihzYywg aWMtPmljX2N1cmNoYW4pOwogCiAJLyogc2V0dXAgaW5pdGlhbCBwcm90ZWN0 aW9uIG1vZGUgKi8KLQlydW5fdXBkYXRlcHJvdChpYyk7CisJcnVuX3VwZGF0 ZXByb3RfY2IoaWMpOwogCiAJLyogdHVybiByYWRpbyBMRUQgb24gKi8KIAly dW5fc2V0X2xlZHMoc2MsIFJUMjg2MF9MRURfUkFESU8pOwpkaWZmIC0tZ2l0 IGEvZGV2L3VzYi93bGFuL2lmX3J1bnZhci5oIGIvZGV2L3VzYi93bGFuL2lm X3J1bnZhci5oCmluZGV4IDM5YWRkYmYuLmNjNjViMWYgMTAwNjQ0Ci0tLSBh L2Rldi91c2Ivd2xhbi9pZl9ydW52YXIuaAorKysgYi9kZXYvdXNiL3dsYW4v aWZfcnVudmFyLmgKQEAgLTEyMSw2ICsxMjEsNyBAQCBzdHJ1Y3QgcnVuX2Nt ZHEgewogc3RydWN0IHJ1bl92YXAgewogCXN0cnVjdCBpZWVlODAyMTF2YXAg ICAgICAgICAgICAgdmFwOwogCXN0cnVjdCBpZWVlODAyMTFfYmVhY29uX29m ZnNldHMgYm87CisJc3RydWN0IG1idWYJCQkqYmVhY29uX21idWY7CiAKIAlp bnQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICgqbmV3c3RhdGUpKHN0 cnVjdCBpZWVlODAyMTF2YXAgKiwKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgZW51bSBpZWVlODAyMTFfc3RhdGUsIGlu dCk7Cg== --0-1961750709-1297127910=:17379-- From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 01:38:39 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F5F21065674 for ; Tue, 8 Feb 2011 01:38:39 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 358398FC0A for ; Tue, 8 Feb 2011 01:38:38 +0000 (UTC) Received: by gxk8 with SMTP id 8so2098843gxk.13 for ; Mon, 07 Feb 2011 17:38:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=r4uowO65bsNInZT7pgMqWeW17C58wFXNgqcaDO5vaPQ=; b=WsE1olbaUbwIAkvxb6Ce8iWLH2HOHYzoEC6TZValSUiY/UNw1yVPVnxteSUw20Twte 08CdqlcxtVbfapJBw5TCNWOWor9nfcoCIEqpkCe1UaF+18Nbw4BY1tfCagbdVSpUhih8 9I+UA43zzG2p0o+DNeqTOaTs/sD+9Z6ueWsxE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=wCT11pyCOyO7SbWM4Nh16DiBJNb4RTmO5bec5BFRACyAuG+Mc6n01clUGXY5SPtmGA 3FhHueVJbr93nsYiSqWo6Yr+SYMcZCjjiCeSp1sNyR37faBqkY426G5c+8kADodev3p9 9HVLrbbQmBWkJa+AINlzMutn3YZc6/Xjx2o24= Received: by 10.151.18.19 with SMTP id v19mr8346450ybi.428.1297129118368; Mon, 07 Feb 2011 17:38:38 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id v39sm7640yba.7.2011.02.07.17.38.35 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 07 Feb 2011 17:38:37 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 7 Feb 2011 17:38:41 -0800 From: Pyun YongHyeon Date: Mon, 7 Feb 2011 17:38:41 -0800 To: Peter Lai Message-ID: <20110208013841.GB1306@michelle.cdnetworks.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: bge wedging 8.2-RC1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 01:38:39 -0000 On Mon, Feb 07, 2011 at 06:09:16PM -0600, Peter Lai wrote: > Hello > > I've got a new Dell Precision workstation here with a BCM5761 on intel > mobo for westmere xeons that is wedging with interrupt storm and will > lockup the system randomly. I have turned HTT and auto powermanagement > off in bios (system cannot sleep), lowest cpu acpi state is C1. > > Here is dmesg: > bge0: 0x5761100> mem 0xf3be0000-0xf3beffff,0xf3bf0000-0xf3bfffff irq 17 at > device 0.0 on pci6 > bge0: CHIP ID 0x05761100; ASIC REV 0x5761; CHIP REV 0x57611; PCI-E > miibus0: on bge0 > brgphy0: PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > > Here is pciconf -lv: > bge0@pci0:6:0:0: class=0x020000 card=0x026d1028 chip=0x168114e4 > rev=0x10 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'Broadcom 57XX Gigabit Integrated Controller (BCM5761)' > class = network > subclass = ethernet > > here is the setup in rc.conf: > > ifconfig_bge0="polling -tso -vlanhwtso -vlanhwtag -vlanmtu inet > 192.168.123.124 netmask 255.255.255.0" > > I have the card plugged into a dlink DSS8 100mbps switch with one > other 100mbps device on it (rich man's crossover cable). > > Before turning off TSO4 and VLAN tagging (because I don't use them), > the card would do several things: > 1. 1 out of 3 reboots: Fail to bring interface up. ifconfig would hang > and systat/vmstat showed 800+ interrupts per second on IRQ256 This is strange. bge(4) does not use MSI if you build bge(4) with DEVICE_POLLING so seeing IRQ256 interrupts looks odd to me. Are you sure bge(4) is using IRQ256? > 2. After a few hours lock up the system, requiring hard reboot > > After disabling TSO4 and VLAN stuff: > bge0: flags=8802 metric 0 mtu 1500 > options=80083 > media: Ethernet autoselect (100baseTX > ) > > Everything seemed fine for about two weeks and then suddenly started > acting up again, locked up, after hard reboot, soft reboot, link will > not come up and I see interrupt storm again.... > If you don't use DEVICE_POLLING, rebuild bge(4) with DEVICE_POLLING. For most cases, you don't need to enable polling on intelligent controllers like bge(4). I also have BCM5761 PCIe controller which shows no such issues. I know there is an edge case(send BD corruption) for BCM5761/BCM5784/ BCM57780 which needs to be investigated. I'm not sure you're seeing that edge case though. > I am close to buying an intel card to replace the bcm, but then I > noticed that the main intel desktop PCI-E card is 82574L-based and > people are having em driver wedging on that too. So now I have broken > ethernet on this box; my primary link is atheros 5212 pci card and I > may be out of pci slots (or else I might try a pci intel card). From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 02:21:46 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B4218106566C for ; Tue, 8 Feb 2011 02:21:46 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 80B678FC1B for ; Tue, 8 Feb 2011 02:21:46 +0000 (UTC) Received: by pzk32 with SMTP id 32so1209405pzk.13 for ; Mon, 07 Feb 2011 18:21:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=eXQyiEED7HSBP7xwnFVdHqoO9VgDPjlA8UKQKQyCVFE=; b=e6TYvpxs/2tqwSVXnt7XSlT5vzidzwCBOi7JeIR3gOyQN9D3Ykrua/6nrM4ZBMNVt8 Us2t7alfAHdy90DR9Q+UAyrw7mAazlK496lM8XkAeQO3eWetRUSsXB+vUC0LxfEBkdHO b264Z51Fmp1jGhOV7kM2nrOiAXz/fzoLYLCXc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=PGqRvo9WJ9fHjJEF6YEiebJIOEoRQTA3umsOzRcs0ey6LXxtUY5fojjHdB7bVXWC7V 1rGdMzOzVjQuf1Pmf1ydQjgkGIDoL/AGQEJ3V701j+b3MpRcVHeGfuul+/RoxMtttjZ8 X72TPGPxOnxArlTx7ZJwEfmUZVfWvhVTlmwAA= MIME-Version: 1.0 Received: by 10.142.48.17 with SMTP id v17mr16246789wfv.314.1297131705587; Mon, 07 Feb 2011 18:21:45 -0800 (PST) Received: by 10.142.246.3 with HTTP; Mon, 7 Feb 2011 18:21:45 -0800 (PST) In-Reply-To: <20110207235811.GA1306@michelle.cdnetworks.com> References: <10510673199.20110207203507@serebryakov.spb.ru> <20110207235811.GA1306@michelle.cdnetworks.com> Date: Mon, 7 Feb 2011 21:21:45 -0500 Message-ID: From: Karim Fodil-Lemelin To: pyunyh@gmail.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: jfv@freebsd.org, freebsd-net@freebsd.org Subject: Re: Fwd: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 02:21:46 -0000 2011/2/7 Pyun YongHyeon > On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin wrote: > > Subject: Re: igb driver tx hangs when out of mbuf clusters > > > > > To: Lev Serebryakov > > > Cc: freebsd-net@freebsd.org > > > > > > > > > 2011/2/7 Lev Serebryakov > > > > > > Hello, Karim. > > >> You wrote 7 =D1=84=D0=B5=D0=B2=D1=80=D0=B0=D0=BB=D1=8F 2011 =D0=B3.,= 19:58:04: > > >> > > >> > > >> > The issue is with the igb driver from 7.4 RC3 r218406. If the driv= er > > >> runs > > >> > out of mbuf clusters it simply stops receiving even after the > clusters > > >> have > > >> > been freed. > > >> It looks like my problems with em0 (see thread "em0 hangs without > > >> any messages like "Watchdog timeout", only down/up reset it.")... > > >> Codebase for em and igb is somewhat common... > > >> > > >> -- > > >> // Black Lion AKA Lev Serebryakov > > >> > > >> I agree. > > > > > > Do you get missed packets in mac_stats (sysctl dev.em | grep missed)? > > > > > > I might not have mentioned but I can also 'fix' the problem by doing > > > ifconfig igb0 down/up. > > > > > > I will try using POLLING to 'automatize' the reset as you mentioned i= n > your > > > thread. > > > > > > Karim. > > > > > > > > Follow up on tests with POLLING: The problem is still occurring althoug= h > it > > takes more time ... Outputs of sysctl dev.igb0 and netstat -m will > follow: > > > > 9219/99426/108645 mbufs in use (current/cache/total) > > 9217/90783/100000/100000 mbuf clusters in use (current/cache/total/max) > > Do you see network processes are stuck in keglim state? If you see > that I think that's not trivial to solve. You wouldn't even kill > that process if it is under keglim state unless some more mbuf > clusters are freed from other places. > No keglim state, here is a snapshot of top -SH while the problem is happening: 12 root 171 ki31 0K 8K CPU5 5 19:27 100.00% idle: cpu5 10 root 171 ki31 0K 8K CPU7 7 19:26 100.00% idle: cpu7 14 root 171 ki31 0K 8K CPU3 3 19:25 100.00% idle: cpu3 11 root 171 ki31 0K 8K CPU6 6 19:25 100.00% idle: cpu6 13 root 171 ki31 0K 8K CPU4 4 19:24 100.00% idle: cpu4 15 root 171 ki31 0K 8K CPU2 2 19:22 100.00% idle: cpu2 16 root 171 ki31 0K 8K CPU1 1 19:18 100.00% idle: cpu1 17 root 171 ki31 0K 8K RUN 0 19:12 100.00% idle: cpu0 18 root -32 - 0K 8K WAIT 6 0:04 0.10% swi4: clock s 20 root -44 - 0K 8K WAIT 4 0:08 0.00% swi1: net 29 root -68 - 0K 8K - 0 0:02 0.00% igb0 que 35 root -68 - 0K 8K - 2 0:02 0.00% em1 taskq 28 root -68 - 0K 8K WAIT 5 0:01 0.00% irq256: igb0 keep in mind that num_queues has been forced to 1. > > I think both igb(4) and em(4) pass received frame to upper stack > before allocating new RX buffer. If driver fails to allocate new RX > buffer driver will try to refill RX buffers in next run. Under > extreme resource shortage case, this situation can produce no more > RX buffers in RX descriptor ring and this will take the box out of > network. Other drivers avoid that situation by allocating new RX > buffer before passing received frame to upper stack. If RX buffer > allocation fails driver will just reuse old RX buffer without > passing received frame to upper stack. That does not completely > solve the keglim issue though. I think you should have enough mbuf > cluters to avoid keglim. > > However the output above indicates you have enough free mbuf > clusters. So I guess igb(4) encountered zero available RX buffer > situation in past but failed to refill the RX buffer again. I guess > driver may be able to periodically check available RX buffers. > Jack may have better idea if this was the case.(CCed) > That is exactly the pattern. The driver runs out of clusters but they eventually get consumed and freed although the driver refuses to process an= y new frames. It is, on the other hand, perfectly capable of sending out packets. > > 0/640 mbuf+clusters out of packet secondary zone in use (current/cache) > > 0/12800/12800/12800 4k (page size) jumbo clusters in use > > (current/cache/total/max) > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > > 20738K/257622K/278361K bytes allocated to network (current/cache/total) > > 0/291/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0/5/6656 sfbufs in use (current/peak/max) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > > > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7 > > dev.igb.0.%driver: igb > > dev.igb.0.%location: slot=3D0 function=3D0 > > dev.igb.0.%pnpinfo: vendor=3D0x8086 device=3D0x10a7 subvendor=3D0x8086 > > subdevice=3D0x0000 class=3D0x020000 > > dev.igb.0.%parent: pci7 > > dev.igb.0.nvm: -1 > > dev.igb.0.flow_control: 3 > > dev.igb.0.enable_aim: 1 > > dev.igb.0.rx_processing_limit: 100 > > dev.igb.0.link_irq: 4 > > dev.igb.0.dropped: 0 > > dev.igb.0.tx_dma_fail: 0 > > dev.igb.0.rx_overruns: 464 > > dev.igb.0.watchdog_timeouts: 0 > > dev.igb.0.device_control: 1490027073 > > dev.igb.0.rx_control: 67141658 > > dev.igb.0.interrupt_mask: 0 > > dev.igb.0.extended_int_mask: 0 > > dev.igb.0.tx_buf_alloc: 14 > > dev.igb.0.rx_buf_alloc: 34 > > dev.igb.0.fc_high_water: 29488 > > dev.igb.0.fc_low_water: 29480 > > dev.igb.0.queue0.interrupt_rate: 111111 > > dev.igb.0.queue0.txd_head: 877 > > dev.igb.0.queue0.txd_tail: 877 > > dev.igb.0.queue0.no_desc_avail: 0 > > dev.igb.0.queue0.tx_packets: 92013 > > dev.igb.0.queue0.rxd_head: 570 > > dev.igb.0.queue0.rxd_tail: 570 > > dev.igb.0.queue0.rx_packets: 163386 > > dev.igb.0.queue0.rx_bytes: 240260310 > > dev.igb.0.queue0.lro_queued: 0 > > dev.igb.0.queue0.lro_flushed: 0 > > dev.igb.0.mac_stats.excess_coll: 0 > > dev.igb.0.mac_stats.single_coll: 0 > > dev.igb.0.mac_stats.multiple_coll: 0 > > dev.igb.0.mac_stats.late_coll: 0 > > dev.igb.0.mac_stats.collision_count: 0 > > dev.igb.0.mac_stats.symbol_errors: 0 > > dev.igb.0.mac_stats.sequence_errors: 0 > > dev.igb.0.mac_stats.defer_count: 0 > > dev.igb.0.mac_stats.missed_packets: 3104 > > dev.igb.0.mac_stats.recv_no_buff: 4016 > > dev.igb.0.mac_stats.recv_undersize: 0 > > dev.igb.0.mac_stats.recv_fragmented: 0 > > dev.igb.0.mac_stats.recv_oversize: 0 > > dev.igb.0.mac_stats.recv_jabber: 0 > > dev.igb.0.mac_stats.recv_errs: 0 > > dev.igb.0.mac_stats.crc_errs: 0 > > dev.igb.0.mac_stats.alignment_errs: 0 > > dev.igb.0.mac_stats.coll_ext_errs: 0 > > dev.igb.0.mac_stats.xon_recvd: 0 > > dev.igb.0.mac_stats.xon_txd: 346 > > dev.igb.0.mac_stats.xoff_recvd: 0 > > dev.igb.0.mac_stats.xoff_txd: 3450 > > dev.igb.0.mac_stats.total_pkts_recvd: 166515 > > dev.igb.0.mac_stats.good_pkts_recvd: 163411 > > dev.igb.0.mac_stats.bcast_pkts_recvd: 0 > > dev.igb.0.mac_stats.mcast_pkts_recvd: 51 > > dev.igb.0.mac_stats.rx_frames_64: 10 > > dev.igb.0.mac_stats.rx_frames_65_127: 1601 > > dev.igb.0.mac_stats.rx_frames_128_255: 53 > > dev.igb.0.mac_stats.rx_frames_256_511: 42 > > dev.igb.0.mac_stats.rx_frames_512_1023: 18 > > dev.igb.0.mac_stats.rx_frames_1024_1522: 161687 > > dev.igb.0.mac_stats.good_octets_recvd: 240948229 > > dev.igb.0.mac_stats.good_octets_txd: 5947150 > > dev.igb.0.mac_stats.total_pkts_txd: 95809 > > dev.igb.0.mac_stats.good_pkts_txd: 92013 > > dev.igb.0.mac_stats.bcast_pkts_txd: 1516 > > dev.igb.0.mac_stats.mcast_pkts_txd: 1817 > > dev.igb.0.mac_stats.tx_frames_64: 90302 > > dev.igb.0.mac_stats.tx_frames_65_127: 1711 > > dev.igb.0.mac_stats.tx_frames_128_255: 0 > > dev.igb.0.mac_stats.tx_frames_256_511: 0 > > dev.igb.0.mac_stats.tx_frames_512_1023: 0 > > dev.igb.0.mac_stats.tx_frames_1024_1522: 0 > > dev.igb.0.mac_stats.tso_txd: 0 > > dev.igb.0.mac_stats.tso_ctx_fail: 0 > > dev.igb.0.interrupts.asserts: 5584 > > dev.igb.0.interrupts.rx_pkt_timer: 163411 > > dev.igb.0.interrupts.rx_abs_timer: 163386 > > dev.igb.0.interrupts.tx_pkt_timer: 92013 > > dev.igb.0.interrupts.tx_abs_timer: 0 > > dev.igb.0.interrupts.tx_queue_empty: 92013 > > dev.igb.0.interrupts.tx_queue_min_thresh: 0 > > dev.igb.0.interrupts.rx_desc_min_thresh: 19 > > dev.igb.0.interrupts.rx_overrun: 0 > > dev.igb.0.host.breaker_tx_pkt: 0 > > dev.igb.0.host.host_tx_pkt_discard: 0 > > dev.igb.0.host.rx_pkt: 0 > > dev.igb.0.host.breaker_rx_pkts: 0 > > dev.igb.0.host.breaker_rx_pkt_drop: 0 > > dev.igb.0.host.tx_good_pkt: 0 > > dev.igb.0.host.breaker_tx_pkt_drop: 0 > > dev.igb.0.host.rx_good_bytes: 240948229 > > dev.igb.0.host.tx_good_bytes: 5947150 > > dev.igb.0.host.length_errors: 0 > > dev.igb.0.host.serdes_violation_pkt: 0 > > dev.igb.0.host.header_redir_missed: 0 > From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 02:27:45 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78F31106566B for ; Tue, 8 Feb 2011 02:27:45 +0000 (UTC) (envelope-from cowbert@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 056608FC12 for ; Tue, 8 Feb 2011 02:27:44 +0000 (UTC) Received: by ewy24 with SMTP id 24so2696780ewy.13 for ; Mon, 07 Feb 2011 18:27:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=pXGpfBmHwLSoW5esEfpi2c9VtRP054+8Z5ESnQp+tWQ=; b=DXkzFdF4h1Swadl7j7DfqBxBi2L0fgfJxhowx3qm2Sy7spxPe2krQdiV6SAuws6S38 nEzcf1570e4W2TaaS5PVDAL7cXyB9z3kpfhkSnRGE5niiOLxWXzVWCkXU3Patbms5ZXV XiNfT7yGtx7harN+JVt+TOdYv32VUxOWP77OM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=A+hkyFRdPMDJwyTzeqQUxAikr93LiCV3I0alD9I8vU/9kveP8YF4Fd3+yPIb2ZQuYo khlIC6QW8DExas0ViGf8Po3UG8Glf86Ih4j/L7YpGeAZNwoZkeq+wWDIt3k9ZJVh6WXI EZlOjXKiYuqQJOWWVBZvZ3PhM/xKbEG2jCF9E= MIME-Version: 1.0 Received: by 10.213.108.69 with SMTP id e5mr380518ebp.76.1297132063701; Mon, 07 Feb 2011 18:27:43 -0800 (PST) Received: by 10.213.31.65 with HTTP; Mon, 7 Feb 2011 18:27:43 -0800 (PST) Received: by 10.213.31.65 with HTTP; Mon, 7 Feb 2011 18:27:43 -0800 (PST) In-Reply-To: <20110208013841.GB1306@michelle.cdnetworks.com> References: <20110208013841.GB1306@michelle.cdnetworks.com> Date: Mon, 7 Feb 2011 20:27:43 -0600 Message-ID: From: Peter Lai To: pyunyh@gmail.com Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: bge wedging 8.2-RC1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 02:27:45 -0000 On Feb 7, 2011 7:38 PM, "Pyun YongHyeon" wrote: > > On Mon, Feb 07, 2011 at 06:09:16PM -0600, Peter Lai wrote: > > Hello > > > > I've got a new Dell Precision workstation here with a BCM5761 on intel > > mobo for westmere xeons that is wedging with interrupt storm and will > > lockup the system randomly. I have turned HTT and auto powermanagement > > off in bios (system cannot sleep), lowest cpu acpi state is C1. > > > > Here is dmesg: > > bge0: > 0x5761100> mem 0xf3be0000-0xf3beffff,0xf3bf0000-0xf3bfffff irq 17 at > > device 0.0 on pci6 > > bge0: CHIP ID 0x05761100; ASIC REV 0x5761; CHIP REV 0x57611; PCI-E > > miibus0: on bge0 > > brgphy0: PHY 1 on miibus0 > > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > > > > Here is pciconf -lv: > > bge0@pci0:6:0:0: class=0x020000 card=0x026d1028 chip=0x168114e4 > > rev=0x10 hdr=0x00 > > vendor = 'Broadcom Corporation' > > device = 'Broadcom 57XX Gigabit Integrated Controller (BCM5761)' > > class = network > > subclass = ethernet > > > > here is the setup in rc.conf: > > > > ifconfig_bge0="polling -tso -vlanhwtso -vlanhwtag -vlanmtu inet > > 192.168.123.124 netmask 255.255.255.0" > > > > I have the card plugged into a dlink DSS8 100mbps switch with one > > other 100mbps device on it (rich man's crossover cable). > > > > Before turning off TSO4 and VLAN tagging (because I don't use them), > > the card would do several things: > > 1. 1 out of 3 reboots: Fail to bring interface up. ifconfig would hang > > and systat/vmstat showed 800+ interrupts per second on IRQ256 > > This is strange. bge(4) does not use MSI if you build bge(4) with > DEVICE_POLLING so seeing IRQ256 interrupts looks odd to me. > Are you sure bge(4) is using IRQ256? This is with GENERIC. I will rebuild with POLLING and try... > > > 2. After a few hours lock up the system, requiring hard reboot > > > > After disabling TSO4 and VLAN stuff: > > bge0: flags=8802 metric 0 mtu 1500 > > options=80083 > > media: Ethernet autoselect (100baseTX > > ) > > > > Everything seemed fine for about two weeks and then suddenly started > > acting up again, locked up, after hard reboot, soft reboot, link will > > not come up and I see interrupt storm again.... > > > > If you don't use DEVICE_POLLING, rebuild bge(4) with > DEVICE_POLLING. For most cases, you don't need to enable polling on > intelligent controllers like bge(4). > > I also have BCM5761 PCIe controller which shows no such issues. I > know there is an edge case(send BD corruption) for BCM5761/BCM5784/ > BCM57780 which needs to be investigated. I'm not sure you're seeing > that edge case though. > > > I am close to buying an intel card to replace the bcm, but then I > > noticed that the main intel desktop PCI-E card is 82574L-based and > > people are having em driver wedging on that too. So now I have broken > > ethernet on this box; my primary link is atheros 5212 pci card and I > > may be out of pci slots (or else I might try a pci intel card). From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 02:35:26 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F2F6A106564A for ; Tue, 8 Feb 2011 02:35:26 +0000 (UTC) (envelope-from kevlo@kevlo.org) Received: from ns.kevlo.org (kevlo.org [220.128.136.52]) by mx1.freebsd.org (Postfix) with ESMTP id 6F07D8FC0A for ; Tue, 8 Feb 2011 02:35:26 +0000 (UTC) Received: from [127.0.0.1] (kevlo@kevlo.org [220.128.136.52]) by ns.kevlo.org (8.14.3/8.14.3) with ESMTP id p182OmqP028544; Tue, 8 Feb 2011 10:24:48 +0800 (CST) From: Kevin Lo To: Peter Lai In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Date: Tue, 08 Feb 2011 10:24:48 +0800 Message-ID: <1297131888.2126.3.camel@nsl> Mime-Version: 1.0 X-Mailer: Evolution 2.30.3 Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: bge wedging 8.2-RC1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 02:35:27 -0000 Peter Lai wrote: > Hello > > I've got a new Dell Precision workstation here with a BCM5761 on intel > mobo for westmere xeons that is wedging with interrupt storm and will > lockup the system randomly. I have turned HTT and auto powermanagement > off in bios (system cannot sleep), lowest cpu acpi state is C1. > > Here is dmesg: > bge0: 0x5761100> mem 0xf3be0000-0xf3beffff,0xf3bf0000-0xf3bfffff irq 17 at > device 0.0 on pci6 > bge0: CHIP ID 0x05761100; ASIC REV 0x5761; CHIP REV 0x57611; PCI-E > miibus0: on bge0 > brgphy0: PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > > Here is pciconf -lv: > bge0@pci0:6:0:0: class=0x020000 card=0x026d1028 chip=0x168114e4 > rev=0x10 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'Broadcom 57XX Gigabit Integrated Controller (BCM5761)' > class = network > subclass = ethernet > > here is the setup in rc.conf: > > ifconfig_bge0="polling -tso -vlanhwtso -vlanhwtag -vlanmtu inet > 192.168.123.124 netmask 255.255.255.0" > > I have the card plugged into a dlink DSS8 100mbps switch with one > other 100mbps device on it (rich man's crossover cable). > > Before turning off TSO4 and VLAN tagging (because I don't use them), > the card would do several things: > 1. 1 out of 3 reboots: Fail to bring interface up. ifconfig would hang > and systat/vmstat showed 800+ interrupts per second on IRQ256 > 2. After a few hours lock up the system, requiring hard reboot > > After disabling TSO4 and VLAN stuff: > bge0: flags=8802 metric 0 mtu 1500 > options=80083 > media: Ethernet autoselect (100baseTX > ) > > Everything seemed fine for about two weeks and then suddenly started > acting up again, locked up, after hard reboot, soft reboot, link will > not come up and I see interrupt storm again.... > > I am close to buying an intel card to replace the bcm, but then I > noticed that the main intel desktop PCI-E card is 82574L-based and > people are having em driver wedging on that too. So now I have broken > ethernet on this box; my primary link is atheros 5212 pci card and I > may be out of pci slots (or else I might try a pci intel card). On -current, the em(4) driver works great for me on high load. em0@pci0:1:0:0: class=0x020000 card=0x6c801462 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 002421ffffdcd321 em1@pci0:2:0:0: class=0x020000 card=0x6c801462 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 002421ffffdcd322 Kevin From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 03:12:03 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB6DF1065672; Tue, 8 Feb 2011 03:12:03 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 89A198FC19; Tue, 8 Feb 2011 03:12:03 +0000 (UTC) Received: by iwn39 with SMTP id 39so5368542iwn.13 for ; Mon, 07 Feb 2011 19:12:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=XyU6ad1PPLYDSmglDYYTxthqRq2imsZP+q+Rvw1G8BY=; b=aqAgnXO4qOHUbU1XkkVgnsumYJ+AkCscbC1Gy206h29kiDJCin1YM9VKkUiq5YidBa NEk6F1etniKZAj8w5dDUSm63zcSrwehSgUElEgQe6H2XAYwX+HRSReFnfys+a+j8A+26 YcO65r3dVRK2RyzMvmJiXXMEj1KlbRj+Syehw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent; b=lEv/x7A2B3AcKxJEY9lf3+MTesv/DiD0TsWYVB7/aK+xpKWRvxDwN/4MQsRtNBGRBg 8GTn/G9Gf/cOd+e4WdfqMD10ofHPSdjeZHQggWesFvo661xBUa3qeiasqR8yh9NEh/bJ 1hSVKfYy2lfHzdZ64MMbfKggrVEwwc17/t+EU= Received: by 10.42.175.6 with SMTP id ay6mr5446957icb.185.1297134722847; Mon, 07 Feb 2011 19:12:02 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id jv9sm3880615icb.13.2011.02.07.19.11.59 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 07 Feb 2011 19:12:00 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 7 Feb 2011 19:12:06 -0800 From: Pyun YongHyeon Date: Mon, 7 Feb 2011 19:12:06 -0800 To: Karim Fodil-Lemelin Message-ID: <20110208031206.GC1306@michelle.cdnetworks.com> References: <10510673199.20110207203507@serebryakov.spb.ru> <20110207235811.GA1306@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: jfv@freebsd.org, freebsd-net@freebsd.org Subject: Re: Fwd: igb driver tx hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 03:12:03 -0000 On Mon, Feb 07, 2011 at 09:21:45PM -0500, Karim Fodil-Lemelin wrote: > 2011/2/7 Pyun YongHyeon > > > On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin wrote: > > > Subject: Re: igb driver tx hangs when out of mbuf clusters > > > > > > > To: Lev Serebryakov > > > > Cc: freebsd-net@freebsd.org > > > > > > > > > > > > 2011/2/7 Lev Serebryakov > > > > > > > > Hello, Karim. > > > >> You wrote 7 Ñ„ÐµÐ²Ñ€Ð°Ð»Ñ 2011 г., 19:58:04: > > > >> > > > >> > > > >> > The issue is with the igb driver from 7.4 RC3 r218406. If the driver > > > >> runs > > > >> > out of mbuf clusters it simply stops receiving even after the > > clusters > > > >> have > > > >> > been freed. > > > >> It looks like my problems with em0 (see thread "em0 hangs without > > > >> any messages like "Watchdog timeout", only down/up reset it.")... > > > >> Codebase for em and igb is somewhat common... > > > >> > > > >> -- > > > >> // Black Lion AKA Lev Serebryakov > > > >> > > > >> I agree. > > > > > > > > Do you get missed packets in mac_stats (sysctl dev.em | grep missed)? > > > > > > > > I might not have mentioned but I can also 'fix' the problem by doing > > > > ifconfig igb0 down/up. > > > > > > > > I will try using POLLING to 'automatize' the reset as you mentioned in > > your > > > > thread. > > > > > > > > Karim. > > > > > > > > > > > Follow up on tests with POLLING: The problem is still occurring although > > it > > > takes more time ... Outputs of sysctl dev.igb0 and netstat -m will > > follow: > > > > > > 9219/99426/108645 mbufs in use (current/cache/total) > > > 9217/90783/100000/100000 mbuf clusters in use (current/cache/total/max) > > > > Do you see network processes are stuck in keglim state? If you see > > that I think that's not trivial to solve. You wouldn't even kill > > that process if it is under keglim state unless some more mbuf > > clusters are freed from other places. > > > > No keglim state, here is a snapshot of top -SH while the problem is > happening: > > 12 root 171 ki31 0K 8K CPU5 5 19:27 100.00% idle: > cpu5 > 10 root 171 ki31 0K 8K CPU7 7 19:26 100.00% idle: > cpu7 > 14 root 171 ki31 0K 8K CPU3 3 19:25 100.00% idle: > cpu3 > 11 root 171 ki31 0K 8K CPU6 6 19:25 100.00% idle: > cpu6 > 13 root 171 ki31 0K 8K CPU4 4 19:24 100.00% idle: > cpu4 > 15 root 171 ki31 0K 8K CPU2 2 19:22 100.00% idle: > cpu2 > 16 root 171 ki31 0K 8K CPU1 1 19:18 100.00% idle: > cpu1 > 17 root 171 ki31 0K 8K RUN 0 19:12 100.00% idle: > cpu0 > 18 root -32 - 0K 8K WAIT 6 0:04 0.10% swi4: > clock s > 20 root -44 - 0K 8K WAIT 4 0:08 0.00% swi1: net > 29 root -68 - 0K 8K - 0 0:02 0.00% igb0 que > 35 root -68 - 0K 8K - 2 0:02 0.00% em1 taskq > 28 root -68 - 0K 8K WAIT 5 0:01 0.00% irq256: > igb0 > > keep in mind that num_queues has been forced to 1. > > > > > > I think both igb(4) and em(4) pass received frame to upper stack > > before allocating new RX buffer. If driver fails to allocate new RX > > buffer driver will try to refill RX buffers in next run. Under > > extreme resource shortage case, this situation can produce no more > > RX buffers in RX descriptor ring and this will take the box out of > > network. Other drivers avoid that situation by allocating new RX > > buffer before passing received frame to upper stack. If RX buffer > > allocation fails driver will just reuse old RX buffer without > > passing received frame to upper stack. That does not completely > > solve the keglim issue though. I think you should have enough mbuf > > cluters to avoid keglim. > > > > However the output above indicates you have enough free mbuf > > clusters. So I guess igb(4) encountered zero available RX buffer > > situation in past but failed to refill the RX buffer again. I guess > > driver may be able to periodically check available RX buffers. > > Jack may have better idea if this was the case.(CCed) > > > > That is exactly the pattern. The driver runs out of clusters but they > eventually get consumed and freed although the driver refuses to process any > new frames. It is, on the other hand, perfectly capable of sending out > packets. > Ok, this clearly indicates igb(4) failed to refill RX buffers since you can still send frames. I'm not sure whether igb(4) controllers could be configured to generate no RX buffer interrupts but that interrupt would be better suited to trigger RX refilling than timer based refilling. Since igb(4) keeps track of available RX buffers, igb(4) can selectively enable that interrupt once it see no RX buffers in the RX descriptor ring. However this does not work with polling. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 03:29:53 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0929106564A; Tue, 8 Feb 2011 03:29:53 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id A0C7D8FC1C; Tue, 8 Feb 2011 03:29:53 +0000 (UTC) Received: by pzk32 with SMTP id 32so1218292pzk.13 for ; Mon, 07 Feb 2011 19:29:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to:cc :content-type; bh=j6+EoQF08diNKLvxt/7vIdTkPpAMpa1t/qTuYFotU8E=; b=Ink+uXlHJ6st7Rc2Bngyj5EEWJ5GBGbYZco/YCynr+9P2zDXbUi69YHsbsHAFxNZgK bSHOBgT/GYH3s+WV7IIE3/pUTtU0srnQN2GWIU1n07bUlAnmdvmBYvUG/8j5v9HQphqw hBo3kfi1ExC0YJUN/w1Jt3FSPNZvUja7dGxdw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; b=Z4gcS3zMR6LtBQdMHWjHvuhC75dKRk26ocLQrM3pkAc+k/4YtTQ1wwyZ7kz/uHUzBf mGcWQejy1baU+lg5vZQ2/bGTIeRUWRhXrLRoFcKMGVRqqnBKaZ9HuO/IYZIMWjcX40Zf lJj66kpShIs1TRaSXRvUi1AY6CHXwKESsbR9g= MIME-Version: 1.0 Received: by 10.142.131.17 with SMTP id e17mr13615000wfd.371.1297135793069; Mon, 07 Feb 2011 19:29:53 -0800 (PST) Received: by 10.142.246.3 with HTTP; Mon, 7 Feb 2011 19:29:53 -0800 (PST) Date: Mon, 7 Feb 2011 22:29:53 -0500 Message-ID: From: Karim Fodil-Lemelin To: pyunyh@gmail.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: jfv@freebsd.org, freebsd-net@freebsd.org Subject: Re: Fwd: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 03:29:53 -0000 2011/2/7 Pyun YongHyeon > On Mon, Feb 07, 2011 at 09:21:45PM -0500, Karim Fodil-Lemelin wrote: > > 2011/2/7 Pyun YongHyeon > > > > > On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin wrote: > > > > Subject: Re: igb driver tx hangs when out of mbuf clusters > > > > > > > > > To: Lev Serebryakov > > > > > Cc: freebsd-net@freebsd.org > > > > > > > > > > > > > > > 2011/2/7 Lev Serebryakov > > > > > > > > > > Hello, Karim. > > > > >> You wrote 7 =D1=84=D0=B5=D0=B2=D1=80=D0=B0=D0=BB=D1=8F 2011 =D0= =B3., 19:58:04: > > > > >> > > > > >> > > > > >> > The issue is with the igb driver from 7.4 RC3 r218406. If the > driver > > > > >> runs > > > > >> > out of mbuf clusters it simply stops receiving even after the > > > clusters > > > > >> have > > > > >> > been freed. > > > > >> It looks like my problems with em0 (see thread "em0 hangs > without > > > > >> any messages like "Watchdog timeout", only down/up reset it.").= .. > > > > >> Codebase for em and igb is somewhat common... > > > > >> > > > > >> -- > > > > >> // Black Lion AKA Lev Serebryakov > > > > >> > > > > >> I agree. > > > > > > > > > > Do you get missed packets in mac_stats (sysctl dev.em | grep > missed)? > > > > > > > > > > I might not have mentioned but I can also 'fix' the problem by > doing > > > > > ifconfig igb0 down/up. > > > > > > > > > > I will try using POLLING to 'automatize' the reset as you mention= ed > in > > > your > > > > > thread. > > > > > > > > > > Karim. > > > > > > > > > > > > > > Follow up on tests with POLLING: The problem is still occurring > although > > > it > > > > takes more time ... Outputs of sysctl dev.igb0 and netstat -m will > > > follow: > > > > > > > > 9219/99426/108645 mbufs in use (current/cache/total) > > > > 9217/90783/100000/100000 mbuf clusters in use > (current/cache/total/max) > > > > > > Do you see network processes are stuck in keglim state? If you see > > > that I think that's not trivial to solve. You wouldn't even kill > > > that process if it is under keglim state unless some more mbuf > > > clusters are freed from other places. > > > > > > > No keglim state, here is a snapshot of top -SH while the problem is > > happening: > > > > 12 root 171 ki31 0K 8K CPU5 5 19:27 100.00% idle= : > > cpu5 > > 10 root 171 ki31 0K 8K CPU7 7 19:26 100.00% idle= : > > cpu7 > > 14 root 171 ki31 0K 8K CPU3 3 19:25 100.00% idle= : > > cpu3 > > 11 root 171 ki31 0K 8K CPU6 6 19:25 100.00% idle= : > > cpu6 > > 13 root 171 ki31 0K 8K CPU4 4 19:24 100.00% idle= : > > cpu4 > > 15 root 171 ki31 0K 8K CPU2 2 19:22 100.00% idle= : > > cpu2 > > 16 root 171 ki31 0K 8K CPU1 1 19:18 100.00% idle= : > > cpu1 > > 17 root 171 ki31 0K 8K RUN 0 19:12 100.00% idle= : > > cpu0 > > 18 root -32 - 0K 8K WAIT 6 0:04 0.10% swi4: > > clock s > > 20 root -44 - 0K 8K WAIT 4 0:08 0.00% swi1: > net > > 29 root -68 - 0K 8K - 0 0:02 0.00% igb0 > que > > 35 root -68 - 0K 8K - 2 0:02 0.00% em1 > taskq > > 28 root -68 - 0K 8K WAIT 5 0:01 0.00% irq25= 6: > > igb0 > > > > keep in mind that num_queues has been forced to 1. > > > > > > > > > > I think both igb(4) and em(4) pass received frame to upper stack > > > before allocating new RX buffer. If driver fails to allocate new RX > > > buffer driver will try to refill RX buffers in next run. Under > > > extreme resource shortage case, this situation can produce no more > > > RX buffers in RX descriptor ring and this will take the box out of > > > network. Other drivers avoid that situation by allocating new RX > > > buffer before passing received frame to upper stack. If RX buffer > > > allocation fails driver will just reuse old RX buffer without > > > passing received frame to upper stack. That does not completely > > > solve the keglim issue though. I think you should have enough mbuf > > > cluters to avoid keglim. > > > > > > However the output above indicates you have enough free mbuf > > > clusters. So I guess igb(4) encountered zero available RX buffer > > > situation in past but failed to refill the RX buffer again. I guess > > > driver may be able to periodically check available RX buffers. > > > Jack may have better idea if this was the case.(CCed) > > > > > > > That is exactly the pattern. The driver runs out of clusters but they > > eventually get consumed and freed although the driver refuses to proces= s > any > > new frames. It is, on the other hand, perfectly capable of sending out > > packets. > > > > Ok, this clearly indicates igb(4) failed to refill RX buffers since > you can still send frames. I'm not sure whether igb(4) controllers > could be configured to generate no RX buffer interrupts but that > interrupt would be better suited to trigger RX refilling than timer > based refilling. Since igb(4) keeps track of available RX buffers, > igb(4) can selectively enable that interrupt once it see no RX > buffers in the RX descriptor ring. However this does not work with > polling. > I think that your evaluation of the problem is correct although I do not understand the selective interrupt mechanism you described. Precisely, the exact same behavior happens (RX hang) if options DEVICE_POLLING is _not_ used in the kernel configuration file. I tried with POLLING since someone mentioned that it helped in a case mentioned earlier today. Unfortunately for igb with or without polling yields the same rx rin= g filing problem. By the way I fixed the subject where I erroneously said TX was hanging whil= e in fact RX is hanging and TX is just fine. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 08:24:34 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D61F106564A; Tue, 8 Feb 2011 08:24:34 +0000 (UTC) (envelope-from alex@zagrebin.ru) Received: from mail.zagrebin.ru (gw.zagrebin.ru [91.215.205.128]) by mx1.freebsd.org (Postfix) with ESMTP id B93418FC12; Tue, 8 Feb 2011 08:24:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zagrebin.ru; s=mail; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=pVqfatzhvy2ovgqAQ+cFEPgk06wjg2H1SabJHn1M5Mw=; b=s1EZieDeAnTlNYW3fID8N8Cr5vSRR/BtY41bq0N6sK440a+Da4UVlDCJX8z4Y0q+gpoxvq4Nle3FUj/cj5bZfb1k6EwmRCG98/Sa5zuY4zslzvycsR9wlpHQ+NhwXjY54ZDYvauvOvzGL21FZzBFkDzujGk3dk2nKFlorIE4qO4=; Received: from alex by mail.zagrebin.ru with local (Exim 4.74 (FreeBSD)) (envelope-from ) id 1Pmis9-000Lsg-JF; Tue, 08 Feb 2011 11:24:29 +0300 Date: Tue, 8 Feb 2011 11:24:29 +0300 From: Alexander Zagrebin To: Bernhard Schmidt Message-ID: <20110208082428.GA83490@gw.zagrebin.ru> References: <20110204060808.GA97298@gw.zagrebin.ru> <201102061132.41032.bschmidt@freebsd.org> <20110207061001.GB59910@gw.zagrebin.ru> <201102070911.02791.bschmidt@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="oyUTqETQ0mS9luUI" Content-Disposition: inline In-Reply-To: <201102070911.02791.bschmidt@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org, PseudoCylon Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 08:24:34 -0000 --oyUTqETQ0mS9luUI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi! On 07.02.2011 09:11:02 +0100, Bernhard Schmidt wrote: > For example, if you call 'ifconfig wlan0 ssid ' the new ssid is > passed over using a IOCTL. It would be interesting to know what function > in net80211 are called regarding beacon updates and which of those call > into the run driver. Ultimately it's about figuring out if special > handling for such cases are required and if so, how to do it. I've added a debug output on allocation, changing and deallocation of a beacon into if_run.c and tried to change SSID while the net.wlan.0.debug is -1. Here is the log contents: kernel: wlan0: ieee80211_init kernel: wlan0: start running, 1 vaps running kernel: wlan0: ieee80211_new_state_locked: RUN -> SCAN (nrunning 0 nscanning 0) kernel: wlan0: ieee80211_newstate_cb: RUN -> INIT arg 0 kernel: wlan0: hostap_newstate: RUN -> INIT (0) kernel: wlan0: node_reclaim: remove 0xffffff8003bd7000<00:14:d1:a8:66:1d> from station table, refcnt 1 kernel: wlan0: ieee80211_alloc_node 0xffffff8004eae000<00:14:d1:a8:66:1d> in station table kernel: wlan0: [00:14:d1:a8:66:1d] ieee80211_alloc_node: inact_reload 2 kernel: wlan0: ieee80211_newstate_cb: INIT -> SCAN arg 0 kernel: wlan0: hostap_newstate: INIT -> SCAN (0) kernel: wlan0: ieee80211_create_ibss: creating HOSTAP on channel 6 kernel: wlan0: ieee80211_alloc_node 0xffffff8003bd7000<00:14:d1:a8:66:1d> in station table kernel: kernel: wlan0: [00:14:d1:a8:66:1d] ieee80211_alloc_node: inact_reload 2 kernel: wlan0: set WME_AC_BE (chan) [acm 0 aifsn 3 logcwmin 4 logcwmax 6 txop 0] kernel: wlan0: set WME_AC_BE (bss ) [acm 0 aifsn 3 logcwmin 4 logcwmax 10 txop 0] kernel: wlan0: set WME_AC_BK (chan) [acm 0 aifsn 7 logcwmin 4 logcwmax 10 txop 0] kernel: wlan0: set WME_AC_BK (bss ) [acm 0 aifsn 7 logcwmin 4 logcwmax 10 txop 0] kernel: wlan0: set WME_AC_VI (chan) [acm 0 aifsn 1 logcwmin 3 logcwmax 4 txop 94] kernel: wlan0: set WME_AC_VI (bss ) [acm 0 aifsn 2 logcwmin 3 logcwmax 4 txop 94] kernel: wlan0: set WME_AC_VO (chan) [acm 0 aifsn 1 logcwmin 2 logcwmax 3 txop 47] kernel: wlan0: set WME_AC_VO (bss ) [acm 0 aifsn 2 logcwmin 2 logcwmax 3 txop 47] kernel: wlan0: ieee80211_wme_updateparams_locked: WME params updated, cap_info 0x6 kernel: wlan0: ieee80211_new_state_locked: SCAN -> RUN (nrunning 0 nscanning 0) kernel: wlan0: ieee80211_newstate_cb: SCAN -> RUN arg -1 kernel: run0: run_update_beacon_cb: updating beacon kernel: wlan0: ieee80211_beacon_update: traffic 0, enable aggressive mode kernel: wlan0: update WME_AC_BE (chan+bss) [acm 0 aifsn 2 logcwmin 4 logcwmax 10 txop 0] kernel: wlan0: update WME_AC_BE (chan+bss) logcwmin 3 kernel: wlan0: ieee80211_wme_updateparams_locked: WME params updated, cap_info 0x7 kernel: wlan0: hostap_newstate: SCAN -> RUN (-1) kernel: wlan0: synchronized with 00:14:d1:a8:66:1d ssid "test" channel 6 start 0Mb kernel: wlan0: [00:14:d1:a8:66:1d] ieee80211_node_authorize: inact_reload 20 As you can see, run_update_beacon_cb() is invoked, but at this time the beacon is already allocated. As the beacon is allocated, run_update_beacon_cb() invokes ieee80211_beacon_update(). As we know, the ieee80211_beacon_update() doesn't update the SSID, so the SSID remains untouched. Nevertheless the changing or hiding/unhiding a SSID seems to be working. It is possible to explain: the station uses an active scan. The ieee80211_send_proberesp()/ieee80211_alloc_proberesp() returns the frame, containing an updated SSID, but AP continues to broadcast beacon with the outdated data. The possible solution is to deallocate a beacon on a state change. I've decided to deallocate a beacon on 'to RUN' state transition. The additional patch is attached. I'll do an additional tests later today... -- Alexander Zagrebin --oyUTqETQ0mS9luUI Content-Type: text/x-patch; charset=us-ascii Content-Disposition: attachment; filename="patch-if_run.c" --- /sys/dev/usb/wlan/if_run.c.orig 2011-02-08 09:52:18.994743647 +0300 +++ /sys/dev/usb/wlan/if_run.c 2011-02-08 11:04:17.114484851 +0300 @@ -1793,6 +1793,12 @@ run_newstate(struct ieee80211vap *vap, e sc->runbmap |= bid; } + if (rvp->beacon_mbuf) { + m_freem(rvp->beacon_mbuf); + rvp->beacon_mbuf = NULL; + } + switch (vap->iv_opmode) { case IEEE80211_M_HOSTAP: case IEEE80211_M_MBSS: --oyUTqETQ0mS9luUI-- From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 09:10:56 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED76A106566B for ; Tue, 8 Feb 2011 09:10:56 +0000 (UTC) (envelope-from lev@serebryakov.spb.ru) Received: from ftp.translate.ru (ftp.translate.ru [80.249.188.42]) by mx1.freebsd.org (Postfix) with ESMTP id A6A788FC24 for ; Tue, 8 Feb 2011 09:10:56 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (89.112.15.178.pppoe.eltel.net [89.112.15.178]) (Authenticated sender: lev@serebryakov.spb.ru) by ftp.translate.ru (Postfix) with ESMTPA id 81AF513DF42; Tue, 8 Feb 2011 12:10:54 +0300 (MSK) Date: Tue, 8 Feb 2011 12:10:53 +0300 From: Lev Serebryakov X-Priority: 3 (Normal) Message-ID: <197599906.20110208121053@serebryakov.spb.ru> To: Karim Fodil-Lemelin In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: pyunyh@gmail.com, jfv@freebsd.org, freebsd-net@freebsd.org Subject: Re: Fwd: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 09:10:57 -0000 Hello, Karim. You wrote 8 =D1=84=D0=B5=D0=B2=D1=80=D0=B0=D0=BB=D1=8F 2011 =D0=B3., 6:29:5= 3: > Precisely, the exact same behavior happens (RX hang) if options > DEVICE_POLLING is _not_ used in the kernel configuration file. I tried wi= th > POLLING since someone mentioned that it helped in a case mentioned earlier > today. Unfortunately for igb with or without polling yields the same rx r= ing > filing problem. In my case (em(4), not igb(4) but symptoms are VERY similar) POLLING (both as kernel option AND "ifconfig em0 polling") options leads to resets (which drops all connections!) AFTER such kernel messages: em0: Watchdog timeout -- resetting em0: Queue(0) tdh =3D 1302, hw tdt =3D 1265 em0: TX(0) desc avail =3D 31,Next TX to Clean =3D 1296 --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 09:20:39 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 66D981065670 for ; Tue, 8 Feb 2011 09:20:39 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norman-vivat.ru [89.250.210.68]) by mx1.freebsd.org (Postfix) with ESMTP id B26298FC17 for ; Tue, 8 Feb 2011 09:20:36 +0000 (UTC) Received: from bsdrookie.norma.com. (bsdrookie.hq.norma.perm.ru [192.168.7.246]) by elf.hq.norma.perm.ru (8.14.3/8.14.3) with ESMTP id p1895cBk086297 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 8 Feb 2011 14:05:39 +0500 (YEKT) (envelope-from emz@norma.perm.ru) Message-ID: <4D510762.7090304@norma.perm.ru> Date: Tue, 08 Feb 2011 14:05:38 +0500 From: "Eugene M. Zheganin" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.8) Gecko/20100917 Thunderbird/3.1.2 MIME-Version: 1.0 To: FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.3 (elf.hq.norma.perm.ru [192.168.3.10]); Tue, 08 Feb 2011 14:05:39 +0500 (YEKT) X-Callback: Sender verified by milter-callback 1.5.12 at elf.hq.norma.perm.ru. X-Callback-Status: relay [192.168.7.246] found in white list. X-Callback-Envelope-From: emz@norma.perm.ru X-Spam-Status: No hits=-102.9 bayes=0.0000 testhits ALL_TRUSTED=-1, BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100 autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on elf.hq.norma.perm.ru Cc: Subject: ipfw, ipv6 and gif(4) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 09:20:39 -0000 Hi. I'm running FreeBSD 8.1-STABLE (I had major issues with em(4) on 8.1-RELEASE, so I had to upgrade this host to more recent STABLE). I'm using ipv6-over-ipv4 tunnel. gif0: flags=8051 metric 0 mtu 1280 tunnel inet 89.250.210.67 --> 216.66.80.26 inet6 2001:470:1f08:14c0::2 --> 2001:470:1f08:14c0::1 prefixlen 128 nd6 options=3 options=1 In order it to work I have to allow ipv4 packets between these two hosts: (and these are two first rules in the filter) 00005 14 1072 allow log ip4 from 89.250.210.67 to 216.66.80.26 out via vlan104 00006 14 1072 allow log ip4 from 216.66.80.26 to 89.250.210.67 in via vlan104 The thing is, normally (at least in ipv4 world) I would have to allow ipencap packets between these hosts (and that's what I did first thing), but this configuraion never worked. I've even added 'allow' strings for every type of encapsulation from /etc/protocols, just to see their counters never changed from zero. Those two rules above were made after 'ok, let's allow everything just to see in log what does it want' decision. I want to ask - why ip4 ? And the log looks even more weird: %ping6 2001:470:1f08:14c0::1 PING6(56=40+8+8 bytes) 2001:470:1f08:14c0::2 --> 2001:470:1f08:14c0::1 16 bytes from 2001:470:1f08:14c0::1, icmp_seq=0 hlim=64 time=93.917 ms 16 bytes from 2001:470:1f08:14c0::1, icmp_seq=1 hlim=64 time=93.307 ms Feb 8 13:56:48 ns kernel: ipfw: 5 Accept P:41 89.250.210.67 216.66.80.26 out via vlan104 Feb 8 13:56:48 ns kernel: ipfw: 6 Accept P:41 216.66.80.26 89.250.210.67 in via vlan104 Feb 8 13:56:49 ns kernel: ipfw: 5 Accept P:41 89.250.210.67 216.66.80.26 out via vlan104 Feb 8 13:56:49 ns kernel: ipfw: 6 Accept P:41 216.66.80.26 89.250.210.67 in via vlan104 As you can see, P:41 is IPv6: %grep 41 /etc/protocols ipv6 41 IPV6 # ipv6 And, of course, ipfw doesn't allow me to create the rules it is actually logging: %ipfw add 7 allow 41 from 216.66.80.26 to 89.250.210.67 in via vlan104 ipfw: bad address "216.66.80.26" Do I misunderstand the concept, or is it how it really should look ? Thanks. Eugene. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 09:51:32 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F15F1065670 for ; Tue, 8 Feb 2011 09:51:32 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id BC8D58FC13 for ; Tue, 8 Feb 2011 09:51:31 +0000 (UTC) Received: by fxm16 with SMTP id 16so6156472fxm.13 for ; Tue, 08 Feb 2011 01:51:30 -0800 (PST) Received: by 10.223.115.75 with SMTP id h11mr16089717faq.119.1297158690639; Tue, 08 Feb 2011 01:51:30 -0800 (PST) Received: from jessie.localnet (p5B2ECC47.dip0.t-ipconnect.de [91.46.204.71]) by mx.google.com with ESMTPS id n7sm1617090fam.11.2011.02.08.01.51.24 (version=SSLv3 cipher=RC4-MD5); Tue, 08 Feb 2011 01:51:26 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: Alexander Zagrebin Date: Tue, 8 Feb 2011 10:51:08 +0100 User-Agent: KMail/1.13.5 (Linux/2.6.32-28-generic; KDE/4.4.5; i686; ; ) References: <20110204060808.GA97298@gw.zagrebin.ru> <201102070911.02791.bschmidt@freebsd.org> <20110208082428.GA83490@gw.zagrebin.ru> In-Reply-To: <20110208082428.GA83490@gw.zagrebin.ru> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201102081051.08923.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org, PseudoCylon Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bschmidt@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 09:51:32 -0000 On Tuesday, February 08, 2011 09:24:29 Alexander Zagrebin wrote: > Hi! > > On 07.02.2011 09:11:02 +0100, Bernhard Schmidt wrote: > > For example, if you call 'ifconfig wlan0 ssid ' the new > > ssid is passed over using a IOCTL. It would be interesting to know > > what function in net80211 are called regarding beacon updates and > > which of those call into the run driver. Ultimately it's about > > figuring out if special handling for such cases are required and > > if so, how to do it. > > I've added a debug output on allocation, changing and deallocation of > a beacon into if_run.c and tried to change SSID while the > net.wlan.0.debug is -1. Here is the log contents: > > kernel: wlan0: ieee80211_init > kernel: wlan0: start running, 1 vaps running > kernel: wlan0: ieee80211_new_state_locked: RUN -> SCAN (nrunning 0 > nscanning 0) kernel: wlan0: ieee80211_newstate_cb: RUN -> INIT arg 0 > kernel: wlan0: hostap_newstate: RUN -> INIT (0) > kernel: wlan0: node_reclaim: remove > 0xffffff8003bd7000<00:14:d1:a8:66:1d> from station table, refcnt 1 > kernel: wlan0: ieee80211_alloc_node > 0xffffff8004eae000<00:14:d1:a8:66:1d> in station table kernel: > wlan0: [00:14:d1:a8:66:1d] ieee80211_alloc_node: inact_reload 2 > kernel: wlan0: ieee80211_newstate_cb: INIT -> SCAN arg 0 > kernel: wlan0: hostap_newstate: INIT -> SCAN (0) > kernel: wlan0: ieee80211_create_ibss: creating HOSTAP on channel 6 > kernel: wlan0: ieee80211_alloc_node > 0xffffff8003bd7000<00:14:d1:a8:66:1d> in station table kernel: > kernel: wlan0: [00:14:d1:a8:66:1d] ieee80211_alloc_node: inact_reload > 2 kernel: wlan0: set WME_AC_BE (chan) [acm 0 aifsn 3 logcwmin 4 > logcwmax 6 txop 0] kernel: wlan0: set WME_AC_BE (bss ) [acm 0 aifsn > 3 logcwmin 4 logcwmax 10 txop 0] kernel: wlan0: set WME_AC_BK (chan) > [acm 0 aifsn 7 logcwmin 4 logcwmax 10 txop 0] kernel: wlan0: set > WME_AC_BK (bss ) [acm 0 aifsn 7 logcwmin 4 logcwmax 10 txop 0] > kernel: wlan0: set WME_AC_VI (chan) [acm 0 aifsn 1 logcwmin 3 > logcwmax 4 txop 94] kernel: wlan0: set WME_AC_VI (bss ) [acm 0 aifsn > 2 logcwmin 3 logcwmax 4 txop 94] kernel: wlan0: set WME_AC_VO (chan) > [acm 0 aifsn 1 logcwmin 2 logcwmax 3 txop 47] kernel: wlan0: set > WME_AC_VO (bss ) [acm 0 aifsn 2 logcwmin 2 logcwmax 3 txop 47] > kernel: wlan0: ieee80211_wme_updateparams_locked: WME params > updated, cap_info 0x6 kernel: wlan0: ieee80211_new_state_locked: > SCAN -> RUN (nrunning 0 nscanning 0) kernel: wlan0: > ieee80211_newstate_cb: SCAN -> RUN arg -1 > kernel: run0: run_update_beacon_cb: updating beacon > kernel: wlan0: ieee80211_beacon_update: traffic 0, enable aggressive > mode kernel: wlan0: update WME_AC_BE (chan+bss) [acm 0 aifsn 2 > logcwmin 4 logcwmax 10 txop 0] kernel: wlan0: update WME_AC_BE > (chan+bss) logcwmin 3 > kernel: wlan0: ieee80211_wme_updateparams_locked: WME params updated, > cap_info 0x7 kernel: wlan0: hostap_newstate: SCAN -> RUN (-1) > kernel: wlan0: synchronized with 00:14:d1:a8:66:1d ssid "test" > channel 6 start 0Mb kernel: wlan0: [00:14:d1:a8:66:1d] > ieee80211_node_authorize: inact_reload 20 > > As you can see, run_update_beacon_cb() is invoked, but at this time > the beacon is already allocated. As the beacon is allocated, > run_update_beacon_cb() invokes ieee80211_beacon_update(). As we > know, the ieee80211_beacon_update() doesn't update the SSID, so the > SSID remains untouched. > Nevertheless the changing or hiding/unhiding a SSID seems to be > working. It is possible to explain: the station uses an active scan. > The ieee80211_send_proberesp()/ieee80211_alloc_proberesp() returns > the frame, containing an updated SSID, but AP continues to broadcast > beacon with the outdated data. > The possible solution is to deallocate a beacon on a state change. > I've decided to deallocate a beacon on 'to RUN' state transition. > The additional patch is attached. > I'll do an additional tests later today... Thank you. That's what I expected actually, when we are going through state changes (RUN -> ... -> RUN) net80211 expects us to throw most knowledge we have aways. This seems to be safest solution. When the beacon mbuf is completely thrown away and created from scratch we can be absolutely sure we handled all cases. -- Bernhard From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 09:53:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C26481065670 for ; Tue, 8 Feb 2011 09:53:10 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4D1668FC0C for ; Tue, 8 Feb 2011 09:53:10 +0000 (UTC) Received: by fxm16 with SMTP id 16so6157997fxm.13 for ; Tue, 08 Feb 2011 01:53:09 -0800 (PST) Received: by 10.223.96.6 with SMTP id f6mr2076787fan.22.1297158789304; Tue, 08 Feb 2011 01:53:09 -0800 (PST) Received: from jessie.localnet (p5B2ECC47.dip0.t-ipconnect.de [91.46.204.71]) by mx.google.com with ESMTPS id f24sm1616456fak.24.2011.02.08.01.53.07 (version=SSLv3 cipher=RC4-MD5); Tue, 08 Feb 2011 01:53:08 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: PseudoCylon , Alexander Zagrebin Date: Tue, 8 Feb 2011 10:52:53 +0100 User-Agent: KMail/1.13.5 (Linux/2.6.32-28-generic; KDE/4.4.5; i686; ; ) References: <20110204060808.GA97298@gw.zagrebin.ru> <201102061142.43865.bschmidt@freebsd.org> <95679.17379.qm@web39320.mail.mud.yahoo.com> In-Reply-To: <95679.17379.qm@web39320.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: Multipart/Mixed; boundary="Boundary-00=_1JRUN9SR6XzIG4g" Message-Id: <201102081052.54016.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bschmidt@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 09:53:10 -0000 --Boundary-00=_1JRUN9SR6XzIG4g Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit On Tuesday, February 08, 2011 02:18:30 PseudoCylon wrote: > ----- Original Message ---- > > > From: Bernhard Schmidt > > To: PseudoCylon > > Cc: Alexander Zagrebin ; freebsd-net@freebsd.org > > Sent: Sun, February 6, 2011 3:42:43 AM > > Subject: Re: if_run in hostap mode: issue with stations in the > > power save mode > > Afaik iwn(4) doesn't use PS, never got around implementing that. > > > > I'd like to move ieee80211_beacon_alloc() into iv_vap_alloc(). > > > Then we don't need to test beacon_mbuf == NULL in > > > run_update_beacon_cb(), and there is already switch we can use > > > for conditionally alloc mem. > > > > Sounds fine with we. > > Oops, there is switch before malloc vap. the test is still > in run_update_beacon_cb() > > > Can I talk you into integrating that into Alexander's patch? > > The patch is attached. (diff to HEAD) Bit long, just because there is > a couple of new call back functions to avoid LOR. Thank you! I've combined both patches (see attachment), if I get an ACK from both of you I'll try get this into the tree ASAP. -- Bernhard --Boundary-00=_1JRUN9SR6XzIG4g Content-Type: text/x-patch; charset="UTF-8"; name="run_beacon.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="run_beacon.diff" Index: sys/dev/usb/wlan/if_runvar.h =================================================================== --- sys/dev/usb/wlan/if_runvar.h (revision 218367) +++ sys/dev/usb/wlan/if_runvar.h (working copy) @@ -121,6 +121,7 @@ struct run_cmdq { struct run_vap { struct ieee80211vap vap; struct ieee80211_beacon_offsets bo; + struct mbuf *beacon_mbuf; int (*newstate)(struct ieee80211vap *, enum ieee80211_state, int); Index: sys/dev/usb/wlan/if_run.c =================================================================== --- sys/dev/usb/wlan/if_run.c (revision 218367) +++ sys/dev/usb/wlan/if_run.c (working copy) @@ -388,6 +388,7 @@ static void run_scan_end(struct ieee80211com *); static void run_update_beacon(struct ieee80211vap *, int); static void run_update_beacon_cb(void *); static void run_updateprot(struct ieee80211com *); +static void run_updateprot_cb(void *); static void run_usb_timeout_cb(void *); static void run_reset_livelock(struct run_softc *); static void run_enable_tsf_sync(struct run_softc *); @@ -398,6 +399,7 @@ static void run_set_leds(struct run_softc *, uint1 static void run_set_bssid(struct run_softc *, const uint8_t *); static void run_set_macaddr(struct run_softc *, const uint8_t *); static void run_updateslot(struct ifnet *); +static void run_updateslot_cb(void *); static void run_update_mcast(struct ifnet *); static int8_t run_rssi2dbm(struct run_softc *, uint8_t, uint8_t); static void run_update_promisc_locked(struct ifnet *); @@ -674,7 +676,7 @@ run_attach(device_t self) ic->ic_set_channel = run_set_channel; ic->ic_node_alloc = run_node_alloc; ic->ic_newassoc = run_newassoc; - //ic->ic_updateslot = run_updateslot; + ic->ic_updateslot = run_updateslot; ic->ic_update_mcast = run_update_mcast; ic->ic_wme.wme_update = run_wme_update; ic->ic_raw_xmit = run_raw_xmit; @@ -856,6 +858,9 @@ run_vap_delete(struct ieee80211vap *vap) RUN_LOCK(sc); + m_freem(rvp->beacon_mbuf); + rvp->beacon_mbuf = NULL; + rvp_id = rvp->rvp_id; sc->ratectl_run &= ~(1 << rvp_id); sc->rvp_bmap &= ~(1 << rvp_id); @@ -1790,6 +1795,9 @@ run_newstate(struct ieee80211vap *vap, enum ieee80 sc->runbmap |= bid; } + m_freem(rvp->beacon_mbuf); + rvp->beacon_mbuf = NULL; + switch (vap->iv_opmode) { case IEEE80211_M_HOSTAP: case IEEE80211_M_MBSS: @@ -3901,8 +3909,29 @@ run_update_beacon(struct ieee80211vap *vap, int it { struct ieee80211com *ic = vap->iv_ic; struct run_softc *sc = ic->ic_ifp->if_softc; + struct run_vap *rvp = RUN_VAP(vap); + int mcast = 0; uint32_t i; + KASSERT(vap != NULL, ("no beacon")); + + switch (item) { + case IEEE80211_BEACON_ERP: + run_updateslot(ic->ic_ifp); + break; + case IEEE80211_BEACON_HTINFO: + run_updateprot(ic); + break; + case IEEE80211_BEACON_TIM: + mcast = 1; /*TODO*/ + break; + default: + break; + } + + setbit(rvp->bo.bo_flags, item); + ieee80211_beacon_update(vap->iv_bss, &rvp->bo, rvp->beacon_mbuf, mcast); + i = RUN_CMDQ_GET(&sc->cmdq_store); DPRINTF("cmdq_store=%d\n", i); sc->cmdq[i].func = run_update_beacon_cb; @@ -3916,6 +3945,7 @@ static void run_update_beacon_cb(void *arg) { struct ieee80211vap *vap = arg; + struct run_vap *rvp = RUN_VAP(vap); struct ieee80211com *ic = vap->iv_ic; struct run_softc *sc = ic->ic_ifp->if_softc; struct rt2860_txwi txwi; @@ -3925,8 +3955,17 @@ run_update_beacon_cb(void *arg) if (vap->iv_bss->ni_chan == IEEE80211_CHAN_ANYC) return; - if ((m = ieee80211_beacon_alloc(vap->iv_bss, &RUN_VAP(vap)->bo)) == NULL) - return; + /* + * No need to call ieee80211_beacon_update(), run_update_beacon() + * is taking care of apropriate calls. + */ + if (rvp->beacon_mbuf == NULL) { + rvp->beacon_mbuf = ieee80211_beacon_alloc(vap->iv_bss, + &rvp->bo); + if (rvp->beacon_mbuf == NULL) + return; + } + m = rvp->beacon_mbuf; memset(&txwi, 0, sizeof txwi); txwi.wcid = 0xff; @@ -3941,13 +3980,11 @@ run_update_beacon_cb(void *arg) txwi.flags = RT2860_TX_TS; txwi.xflags = RT2860_TX_NSEQ; - run_write_region_1(sc, RT2860_BCN_BASE(RUN_VAP(vap)->rvp_id), + run_write_region_1(sc, RT2860_BCN_BASE(rvp->rvp_id), (uint8_t *)&txwi, sizeof txwi); - run_write_region_1(sc, RT2860_BCN_BASE(RUN_VAP(vap)->rvp_id) + sizeof txwi, + run_write_region_1(sc, RT2860_BCN_BASE(rvp->rvp_id) + sizeof txwi, mtod(m, uint8_t *), (m->m_pkthdr.len + 1) & ~1); /* roundup len */ - m_freem(m); - return; } @@ -3955,6 +3992,20 @@ static void run_updateprot(struct ieee80211com *ic) { struct run_softc *sc = ic->ic_ifp->if_softc; + uint32_t i; + + i = RUN_CMDQ_GET(&sc->cmdq_store); + DPRINTF("cmdq_store=%d\n", i); + sc->cmdq[i].func = run_updateprot_cb; + sc->cmdq[i].arg0 = ic; + ieee80211_runtask(ic, &sc->cmdq_task); +} + +static void +run_updateprot_cb(void *arg) +{ + struct ieee80211com *ic = arg; + struct run_softc *sc = ic->ic_ifp->if_softc; uint32_t tmp; tmp = RT2860_RTSTH_EN | RT2860_PROT_NAV_SHORT | RT2860_TXOP_ALLOW_ALL; @@ -4164,12 +4215,29 @@ run_set_macaddr(struct run_softc *sc, const uint8_ addr[4] | addr[5] << 8 | 0xff << 16); } -/* ARGSUSED */ static void run_updateslot(struct ifnet *ifp) { struct run_softc *sc = ifp->if_softc; struct ieee80211com *ic = ifp->if_l2com; + uint32_t i; + + i = RUN_CMDQ_GET(&sc->cmdq_store); + DPRINTF("cmdq_store=%d\n", i); + sc->cmdq[i].func = run_updateslot_cb; + sc->cmdq[i].arg0 = ifp; + ieee80211_runtask(ic, &sc->cmdq_task); + + return; +} + +/* ARGSUSED */ +static void +run_updateslot_cb(void *arg) +{ + struct ifnet *ifp = arg; + struct run_softc *sc = ifp->if_softc; + struct ieee80211com *ic = ifp->if_l2com; uint32_t tmp; run_read(sc, RT2860_BKOFF_SLOT_CFG, &tmp); @@ -4747,7 +4815,7 @@ run_init_locked(struct run_softc *sc) run_set_chan(sc, ic->ic_curchan); /* setup initial protection mode */ - run_updateprot(ic); + run_updateprot_cb(ic); /* turn radio LED on */ run_set_leds(sc, RT2860_LED_RADIO); --Boundary-00=_1JRUN9SR6XzIG4g-- From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 10:42:39 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57CC61065672; Tue, 8 Feb 2011 10:42:39 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from mail-n.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) by mx1.freebsd.org (Postfix) with ESMTP id A83718FC13; Tue, 8 Feb 2011 10:42:38 +0000 (UTC) Received: from [212.201.127.66] (unknown [212.201.127.66]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTP id A31C01C0B4610; Tue, 8 Feb 2011 11:42:36 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: multipart/mixed; boundary=Apple-Mail-3-1026976568 From: =?iso-8859-1?Q?Michael_T=FCxen?= X-Priority: 3 (Normal) In-Reply-To: <197599906.20110208121053@serebryakov.spb.ru> Date: Tue, 8 Feb 2011 11:42:36 +0100 Message-Id: References: <197599906.20110208121053@serebryakov.spb.ru> To: Lev Serebryakov X-Mailer: Apple Mail (2.1082) Cc: Karim Fodil-Lemelin , pyunyh@gmail.com, jfv@freebsd.org, freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 10:42:39 -0000 --Apple-Mail-3-1026976568 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On Feb 8, 2011, at 10:10 AM, Lev Serebryakov wrote: > Hello, Karim. > You wrote 8 =D1=84=D0=B5=D0=B2=D1=80=D0=B0=D0=BB=D1=8F 2011 =D0=B3., = 6:29:53: >=20 >> Precisely, the exact same behavior happens (RX hang) if options >> DEVICE_POLLING is _not_ used in the kernel configuration file. I = tried with >> POLLING since someone mentioned that it helped in a case mentioned = earlier >> today. Unfortunately for igb with or without polling yields the same = rx ring >> filing problem. > In my case (em(4), not igb(4) but symptoms are VERY similar) POLLING > (both as kernel option AND "ifconfig em0 polling") options leads to > resets (which drops all connections!) AFTER such kernel messages: >=20 > em0: Watchdog timeout -- resetting > em0: Queue(0) tdh =3D 1302, hw tdt =3D 1265 > em0: TX(0) desc avail =3D 31,Next TX to Clean =3D 1296 Can you apply the attached patch and report what the output for rx_nxt_refresh and rx_nxt_check is? Best regards Michael --Apple-Mail-3-1026976568 Content-Disposition: attachment; filename=patch Content-Type: application/octet-stream; x-unix-mode=0644; name="patch" Content-Transfer-Encoding: 7bit Index: if_igb.c =================================================================== --- if_igb.c (revision 218406) +++ if_igb.c (working copy) @@ -5158,6 +5158,12 @@ SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "rx_bytes", CTLFLAG_RD, &rxr->rx_bytes, "Queue Bytes Received"); + SYSCTL_ADD_UINT(ctx, queue_list, OID_AUTO, "rx_nxt_refresh", + CTLFLAG_RD, &rxr->next_to_refresh, 0, + "Next to refresh"); + SYSCTL_ADD_UINT(ctx, queue_list, OID_AUTO, "rx_nxt_check", + CTLFLAG_RD, &rxr->next_to_check, 0, + "Next to check"); SYSCTL_ADD_INT(ctx, queue_list, OID_AUTO, "lro_queued", CTLFLAG_RD, &lro->lro_queued, 0, "LRO Queued"); Index: if_em.c =================================================================== --- if_em.c (revision 218406) +++ if_em.c (working copy) @@ -5137,6 +5137,12 @@ SYSCTL_ADD_ULONG(ctx, queue_list, OID_AUTO, "rx_irq", CTLFLAG_RD, &rxr->rx_irq, "Queue MSI-X Receive Interrupts"); + SYSCTL_ADD_UINT(ctx, queue_list, OID_AUTO, "rx_nxt_refresh", + CTLFLAG_RD, &rxr->next_to_refresh, 0, + "Next to refresh"); + SYSCTL_ADD_UINT(ctx, queue_list, OID_AUTO, "rx_nxt_check", + CTLFLAG_RD, &rxr->next_to_check, 0, + "Next to check"); } /* MAC stats get their own sub node */ --Apple-Mail-3-1026976568 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii > > -- > // Black Lion AKA Lev Serebryakov > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > --Apple-Mail-3-1026976568-- From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 10:44:38 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74ED8106564A; Tue, 8 Feb 2011 10:44:38 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from mail-n.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) by mx1.freebsd.org (Postfix) with ESMTP id DB8F98FC15; Tue, 8 Feb 2011 10:44:37 +0000 (UTC) Received: from [212.201.127.66] (unknown [212.201.127.66]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTP id 004F01C0B4619; Tue, 8 Feb 2011 11:44:36 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: multipart/mixed; boundary=Apple-Mail-4-1027096974 From: =?iso-8859-1?Q?Michael_T=FCxen?= In-Reply-To: Date: Tue, 8 Feb 2011 11:44:36 +0100 Message-Id: References: To: Karim Fodil-Lemelin X-Mailer: Apple Mail (2.1082) Cc: pyunyh@gmail.com, jfv@freebsd.org, freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 10:44:38 -0000 --Apple-Mail-4-1027096974 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On Feb 8, 2011, at 4:29 AM, Karim Fodil-Lemelin wrote: > 2011/2/7 Pyun YongHyeon >=20 >> On Mon, Feb 07, 2011 at 09:21:45PM -0500, Karim Fodil-Lemelin wrote: >>> 2011/2/7 Pyun YongHyeon >>>=20 >>>> On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin = wrote: >>>>> Subject: Re: igb driver tx hangs when out of mbuf clusters >>>>>=20 >>>>>> To: Lev Serebryakov >>>>>> Cc: freebsd-net@freebsd.org >>>>>>=20 >>>>>>=20 >>>>>> 2011/2/7 Lev Serebryakov >>>>>>=20 >>>>>> Hello, Karim. >>>>>>> You wrote 7 =D1=84=D0=B5=D0=B2=D1=80=D0=B0=D0=BB=D1=8F 2011 =D0=B3= ., 19:58:04: >>>>>>>=20 >>>>>>>=20 >>>>>>>> The issue is with the igb driver from 7.4 RC3 r218406. If the >> driver >>>>>>> runs >>>>>>>> out of mbuf clusters it simply stops receiving even after the >>>> clusters >>>>>>> have >>>>>>>> been freed. >>>>>>> It looks like my problems with em0 (see thread "em0 hangs >> without >>>>>>> any messages like "Watchdog timeout", only down/up reset = it.")... >>>>>>> Codebase for em and igb is somewhat common... >>>>>>>=20 >>>>>>> -- >>>>>>> // Black Lion AKA Lev Serebryakov >>>>>>>=20 >>>>>>> I agree. >>>>>>=20 >>>>>> Do you get missed packets in mac_stats (sysctl dev.em | grep >> missed)? >>>>>>=20 >>>>>> I might not have mentioned but I can also 'fix' the problem by >> doing >>>>>> ifconfig igb0 down/up. >>>>>>=20 >>>>>> I will try using POLLING to 'automatize' the reset as you = mentioned >> in >>>> your >>>>>> thread. >>>>>>=20 >>>>>> Karim. >>>>>>=20 >>>>>>=20 >>>>> Follow up on tests with POLLING: The problem is still occurring >> although >>>> it >>>>> takes more time ... Outputs of sysctl dev.igb0 and netstat -m will >>>> follow: >>>>>=20 >>>>> 9219/99426/108645 mbufs in use (current/cache/total) >>>>> 9217/90783/100000/100000 mbuf clusters in use >> (current/cache/total/max) >>>>=20 >>>> Do you see network processes are stuck in keglim state? If you see >>>> that I think that's not trivial to solve. You wouldn't even kill >>>> that process if it is under keglim state unless some more mbuf >>>> clusters are freed from other places. >>>>=20 >>>=20 >>> No keglim state, here is a snapshot of top -SH while the problem is >>> happening: >>>=20 >>> 12 root 171 ki31 0K 8K CPU5 5 19:27 100.00% = idle: >>> cpu5 >>> 10 root 171 ki31 0K 8K CPU7 7 19:26 100.00% = idle: >>> cpu7 >>> 14 root 171 ki31 0K 8K CPU3 3 19:25 100.00% = idle: >>> cpu3 >>> 11 root 171 ki31 0K 8K CPU6 6 19:25 100.00% = idle: >>> cpu6 >>> 13 root 171 ki31 0K 8K CPU4 4 19:24 100.00% = idle: >>> cpu4 >>> 15 root 171 ki31 0K 8K CPU2 2 19:22 100.00% = idle: >>> cpu2 >>> 16 root 171 ki31 0K 8K CPU1 1 19:18 100.00% = idle: >>> cpu1 >>> 17 root 171 ki31 0K 8K RUN 0 19:12 100.00% = idle: >>> cpu0 >>> 18 root -32 - 0K 8K WAIT 6 0:04 0.10% = swi4: >>> clock s >>> 20 root -44 - 0K 8K WAIT 4 0:08 0.00% = swi1: >> net >>> 29 root -68 - 0K 8K - 0 0:02 0.00% = igb0 >> que >>> 35 root -68 - 0K 8K - 2 0:02 0.00% em1 >> taskq >>> 28 root -68 - 0K 8K WAIT 5 0:01 0.00% = irq256: >>> igb0 >>>=20 >>> keep in mind that num_queues has been forced to 1. >>>=20 >>>=20 >>>>=20 >>>> I think both igb(4) and em(4) pass received frame to upper stack >>>> before allocating new RX buffer. If driver fails to allocate new RX >>>> buffer driver will try to refill RX buffers in next run. Under >>>> extreme resource shortage case, this situation can produce no more >>>> RX buffers in RX descriptor ring and this will take the box out of >>>> network. Other drivers avoid that situation by allocating new RX >>>> buffer before passing received frame to upper stack. If RX buffer >>>> allocation fails driver will just reuse old RX buffer without >>>> passing received frame to upper stack. That does not completely >>>> solve the keglim issue though. I think you should have enough mbuf >>>> cluters to avoid keglim. >>>>=20 >>>> However the output above indicates you have enough free mbuf >>>> clusters. So I guess igb(4) encountered zero available RX buffer >>>> situation in past but failed to refill the RX buffer again. I guess >>>> driver may be able to periodically check available RX buffers. >>>> Jack may have better idea if this was the case.(CCed) >>>>=20 >>>=20 >>> That is exactly the pattern. The driver runs out of clusters but = they >>> eventually get consumed and freed although the driver refuses to = process >> any >>> new frames. It is, on the other hand, perfectly capable of sending = out >>> packets. >>>=20 >>=20 >> Ok, this clearly indicates igb(4) failed to refill RX buffers since >> you can still send frames. I'm not sure whether igb(4) controllers >> could be configured to generate no RX buffer interrupts but that >> interrupt would be better suited to trigger RX refilling than timer >> based refilling. Since igb(4) keeps track of available RX buffers, >> igb(4) can selectively enable that interrupt once it see no RX >> buffers in the RX descriptor ring. However this does not work with >> polling. >>=20 >=20 > I think that your evaluation of the problem is correct although I do = not > understand the selective interrupt mechanism you described. >=20 > Precisely, the exact same behavior happens (RX hang) if options > DEVICE_POLLING is _not_ used in the kernel configuration file. I tried = with > POLLING since someone mentioned that it helped in a case mentioned = earlier > today. Unfortunately for igb with or without polling yields the same = rx ring > filing problem. >=20 > By the way I fixed the subject where I erroneously said TX was hanging = while > in fact RX is hanging and TX is just fine. Katim, could you apply the attached patch and report what the value of rx_nxt_check and rx_nxt_refresh is when the interface hangs. You get the values using sysctl -a dev.igb Best regards Michael --Apple-Mail-4-1027096974 Content-Disposition: attachment; filename=patch Content-Type: application/octet-stream; x-unix-mode=0644; name="patch" Content-Transfer-Encoding: 7bit Index: if_igb.c =================================================================== --- if_igb.c (revision 218406) +++ if_igb.c (working copy) @@ -5158,6 +5158,12 @@ SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "rx_bytes", CTLFLAG_RD, &rxr->rx_bytes, "Queue Bytes Received"); + SYSCTL_ADD_UINT(ctx, queue_list, OID_AUTO, "rx_nxt_refresh", + CTLFLAG_RD, &rxr->next_to_refresh, 0, + "Next to refresh"); + SYSCTL_ADD_UINT(ctx, queue_list, OID_AUTO, "rx_nxt_check", + CTLFLAG_RD, &rxr->next_to_check, 0, + "Next to check"); SYSCTL_ADD_INT(ctx, queue_list, OID_AUTO, "lro_queued", CTLFLAG_RD, &lro->lro_queued, 0, "LRO Queued"); Index: if_em.c =================================================================== --- if_em.c (revision 218406) +++ if_em.c (working copy) @@ -5137,6 +5137,12 @@ SYSCTL_ADD_ULONG(ctx, queue_list, OID_AUTO, "rx_irq", CTLFLAG_RD, &rxr->rx_irq, "Queue MSI-X Receive Interrupts"); + SYSCTL_ADD_UINT(ctx, queue_list, OID_AUTO, "rx_nxt_refresh", + CTLFLAG_RD, &rxr->next_to_refresh, 0, + "Next to refresh"); + SYSCTL_ADD_UINT(ctx, queue_list, OID_AUTO, "rx_nxt_check", + CTLFLAG_RD, &rxr->next_to_check, 0, + "Next to check"); } /* MAC stats get their own sub node */ --Apple-Mail-4-1027096974 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > --Apple-Mail-4-1027096974-- From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 15:31:24 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CEC410656C8 for ; Tue, 8 Feb 2011 15:31:24 +0000 (UTC) (envelope-from ume@mahoroba.org) Received: from mail.mahoroba.org (ent.mahoroba.org [IPv6:2001:2f0:104:8010::1]) by mx1.freebsd.org (Postfix) with ESMTP id 27FDF8FC20 for ; Tue, 8 Feb 2011 15:31:23 +0000 (UTC) Received: from yuga.mahoroba.org (ume@yuga.mahoroba.org [IPv6:2001:2f0:104:8010:21b:d3ff:fe38:5381]) (user=ume mech=DIGEST-MD5 bits=0) by mail.mahoroba.org (8.14.4/8.14.4) with ESMTP/inet6 id p18FVBmk039572 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 9 Feb 2011 00:31:17 +0900 (JST) (envelope-from ume@mahoroba.org) Date: Wed, 09 Feb 2011 00:31:11 +0900 Message-ID: From: Hajimu UMEMOTO To: "Eugene M. Zheganin" In-Reply-To: <4D510762.7090304@norma.perm.ru> References: <4D510762.7090304@norma.perm.ru> User-Agent: xcite1.60> Wanderlust/2.15.9 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?ISO-2022-JP-2?B?R29qGyQoRCtXGyhC?=) APEL/10.8 Emacs/23.2 (i386-portbld-freebsd8.1) MULE/6.0 (HANACHIRUSATO) X-Operating-System: FreeBSD 8.2-PRERELEASE X-PGP-Key: http://www.imasy.or.jp/~ume/publickey.asc X-PGP-Fingerprint: 1F00 0B9E 2164 70FC 6DC5 BF5F 04E9 F086 BF90 71FE Organization: Internet Mutual Aid Society, YOKOHAMA MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (mail.mahoroba.org [IPv6:2001:2f0:104:8010::1]); Wed, 09 Feb 2011 00:31:17 +0900 (JST) X-Virus-Scanned: clamav-milter 0.96.5 at asuka.mahoroba.org X-Virus-Status: Clean X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on asuka.mahoroba.org Cc: FreeBSD Net Subject: Re: ipfw, ipv6 and gif(4) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 15:31:24 -0000 Hi, >>>>> On Tue, 08 Feb 2011 14:05:38 +0500 >>>>> "Eugene M. Zheganin" said: emz> As you can see, P:41 is IPv6: emz> %grep 41 /etc/protocols emz> ipv6 41 IPV6 # ipv6 emz> And, of course, ipfw doesn't allow me to create the rules it is emz> actually logging: emz> %ipfw add 7 allow 41 from 216.66.80.26 to 89.250.210.67 in via vlan104 emz> ipfw: bad address "216.66.80.26" emz> Do I misunderstand the concept, or is it how it really should look ? Something like `pass ip4 from any to any proto ipv6' should work for you. Sincerely, -- Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan ume@mahoroba.org ume@{,jp.}FreeBSD.org http://www.imasy.org/~ume/ From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 16:19:26 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5325F106564A for ; Tue, 8 Feb 2011 16:19:26 +0000 (UTC) (envelope-from cokeeffe@gmail.com) Received: from MARSHAL4.ul.ie (marshal4.ul.ie [193.1.100.137]) by mx1.freebsd.org (Postfix) with ESMTP id DC3B68FC0C for ; Tue, 8 Feb 2011 16:19:25 +0000 (UTC) Received: from staffexchange7.ul.campus (Not Verified[193.1.101.32]) by MARSHAL4.ul.ie with MailMarshal (v6, 8, 4, 9558) id ; Tue, 08 Feb 2011 16:02:34 +0000 Received: from [136.201.98.207] ([136.201.98.207]) by staffexchange7.ul.campus over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Tue, 8 Feb 2011 16:02:34 +0000 From: Colin O'Keeffe Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 8 Feb 2011 16:04:20 +0000 Message-Id: <9CF06DFA-117C-4509-BAAB-CFE30ECE9385@gmail.com> To: freebsd-net@freebsd.org Mime-Version: 1.0 (Apple Message framework v1082) X-Mailer: Apple Mail (2.1082) X-OriginalArrivalTime: 08 Feb 2011 16:02:34.0855 (UTC) FILETIME=[997ABF70:01CBC7A9] Subject: IPv6 Extension Headers X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 16:19:26 -0000 Hi, I'm looking for some guidance on implementing extension headers in the = kernel for outgoing packets and processing incoming packets. Is anybody = available to discuss it with me (on or off the mailing list) to help me = get the ball rolling. Thanks= From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 16:32:15 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 248801065693 for ; Tue, 8 Feb 2011 16:32:15 +0000 (UTC) (envelope-from rozhuk.im@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id A96068FC13 for ; Tue, 8 Feb 2011 16:32:14 +0000 (UTC) Received: by wyf19 with SMTP id 19so6061777wyf.13 for ; Tue, 08 Feb 2011 08:32:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:reply-to:from:to:references:in-reply-to:subject :date:message-id:mime-version:content-type:content-transfer-encoding :x-mailer:content-language:thread-index; bh=thau5mFw+HBqdjl48YnrARURguSyRoPR7YIO6Qre5zE=; b=f7TUbEP1Qis+qB1mJBAAiLUlImbCdma4rmLMbk8H2ync1lYxWNtrm0NNs6IukpB51y YZbLwWKufToCzm1kCgP8xnyZZLsNSS2MTq5gyvGfCjQ+rC3Q6eDpO1Zowo49keM5UBjJ Y5nnZylb7vAkAFJl35Kd8cOufPnOZ2jmvHfxc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=reply-to:from:to:references:in-reply-to:subject:date:message-id :mime-version:content-type:content-transfer-encoding:x-mailer :content-language:thread-index; b=qmXVVsVlGuiIAZxtqkWTILEhXz3CGzLKaboPxr3105ab2o9G7Aw1ZPw70VScmzGZzq 4jcrUzL6XuVyrQfnb7L1oOzI+EGg/GJs02ADk0VMNCqMg8nEcBDS3FTpQYaIdu3f9rQi yWErIAQbEKdCkYlQQvGOwQuahMnr0lq3ZkXeg= Received: by 10.227.132.69 with SMTP id a5mr17598606wbt.123.1297181291778; Tue, 08 Feb 2011 08:08:11 -0800 (PST) Received: from rimwks1x64 ([95.189.10.18]) by mx.google.com with ESMTPS id u9sm4541162wbg.18.2011.02.08.08.08.07 (version=SSLv3 cipher=RC4-MD5); Tue, 08 Feb 2011 08:08:10 -0800 (PST) From: rozhuk.im@gmail.com To: References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> <4D501198.6090901@FreeBSD.org> In-Reply-To: <4D501198.6090901@FreeBSD.org> Date: Wed, 9 Feb 2011 00:08:04 +0800 Message-ID: <4d516a6a.8937e30a.0996.2f26@mx.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Content-Language: ru Thread-Index: AcvG3OIukmf5op/ZQeWGpavfEOZxMAAzAq9w Subject: RE: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Rozhuk.IM@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 16:32:15 -0000 > -----Original Message----- > From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd- > net@freebsd.org] On Behalf Of Sergey Matveychuk > Sent: Monday, February 07, 2011 11:37 PM > To: Julian Elischer > Cc: Ivo Vachkov; FreeBSD Net > Subject: Re: divert rewrite > > 06.02.2011 4:42, Julian Elischer wrote: > > On 2/5/11 4:09 PM, Ivo Vachkov wrote: > >> Hello, > >> > >> How can I help? > > > > if you have ipv6 connectivity and experience, I have no experience or > > connectivity, with it so > > I'll be coding blind and will need a tester. > > If you have an application for IPV6 testing that would be even > better. > > Divert is often used for NAT but that doesn't seem very useful for > IPv6 and > > natd doesn't support it anyhow. > > Object :) > Divert is really useful way to get packets from firewall to userspace, > analyse or process them some way and put them back. Really I see no > other way for this for IPv6. I've tried ng_socket+ng_nat but there is > no > easy way to put a packet back in firewall. > > I'm very interested in the process. And I'm ready to help in testing. Did you try ng_ether + ng_ksocket? It can translate Ethernet frames incapsulated to udp to user space receiver. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 16:39:46 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0576106566B for ; Tue, 8 Feb 2011 16:39:46 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9EBB18FC0C for ; Tue, 8 Feb 2011 16:39:46 +0000 (UTC) Received: by pwi10 with SMTP id 10so1381233pwi.13 for ; Tue, 08 Feb 2011 08:39:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=/lQXGnpGI0VkVEKSW98uc7+rHZBAuhdptqT4+6aI690=; b=WnxCDWxn8zcW5Anjxpp9QHaF9j/AmHTtXxFf+1U+sBwvyGiqfBsnuLNPRV1z3IYFyP RSiNYO09lfcwjwUp/qbD6jVjn8PULSPjXUTwVP1W+KxcZlNM+buxGXoevaKVju9FxoXF HPskUE2xjT6QFVguH1/KhwHj//X77XBx/wPVc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=GpU3y5MuUJDCiEdt3q4gyKmzr5zzu4dKlkRx1C3/aSAsJrXydSQd1Vla4V5x6awPiu nRNv68ScaPrG2Fyi3a9/A5b5qfrKZ7nI2KVFcYLb9wga/2Oa9yxhCeTpvU4HQVVjAq/j QkBJ5h8UHXF0GRJaf8vtXROT55r8rTSAUk4bw= MIME-Version: 1.0 Received: by 10.142.216.3 with SMTP id o3mr17184241wfg.29.1297183185669; Tue, 08 Feb 2011 08:39:45 -0800 (PST) Received: by 10.142.246.3 with HTTP; Tue, 8 Feb 2011 08:39:45 -0800 (PST) In-Reply-To: References: Date: Tue, 8 Feb 2011 11:39:45 -0500 Message-ID: From: Karim Fodil-Lemelin To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 16:39:46 -0000 > 2011/2/8 Michael T=C3=BCxen > >> On Feb 8, 2011, at 4:29 AM, Karim Fodil-Lemelin wrote: >> >> > 2011/2/7 Pyun YongHyeon >> > >> >> On Mon, Feb 07, 2011 at 09:21:45PM -0500, Karim Fodil-Lemelin wrote: >> >>> 2011/2/7 Pyun YongHyeon >> >>> >> >>>> On Mon, Feb 07, 2011 at 05:33:47PM -0500, Karim Fodil-Lemelin wrote= : >> >>>>> Subject: Re: igb driver tx hangs when out of mbuf clusters >> >>>>> >> >>>>>> To: Lev Serebryakov >> >>>>>> Cc: freebsd-net@freebsd.org >> >>>>>> >> >>>>>> >> >>>>>> 2011/2/7 Lev Serebryakov >> >>>>>> >> >>>>>> Hello, Karim. >> >>>>>>> You wrote 7 =D1=84=D0=B5=D0=B2=D1=80=D0=B0=D0=BB=D1=8F 2011 =D0= =B3., 19:58:04: >> >>>>>>> >> >>>>>>> >> >>>>>>>> The issue is with the igb driver from 7.4 RC3 r218406. If the >> >> driver >> >>>>>>> runs >> >>>>>>>> out of mbuf clusters it simply stops receiving even after the >> >>>> clusters >> >>>>>>> have >> >>>>>>>> been freed. >> >>>>>>> It looks like my problems with em0 (see thread "em0 hangs >> >> without >> >>>>>>> any messages like "Watchdog timeout", only down/up reset it.")..= . >> >>>>>>> Codebase for em and igb is somewhat common... >> >>>>>>> >> >>>>>>> -- >> >>>>>>> // Black Lion AKA Lev Serebryakov >> >>>>>>> >> >>>>>>> I agree. >> >>>>>> >> >>>>>> Do you get missed packets in mac_stats (sysctl dev.em | grep >> >> missed)? >> >>>>>> >> >>>>>> I might not have mentioned but I can also 'fix' the problem by >> >> doing >> >>>>>> ifconfig igb0 down/up. >> >>>>>> >> >>>>>> I will try using POLLING to 'automatize' the reset as you mention= ed >> >> in >> >>>> your >> >>>>>> thread. >> >>>>>> >> >>>>>> Karim. >> >>>>>> >> >>>>>> >> >>>>> Follow up on tests with POLLING: The problem is still occurring >> >> although >> >>>> it >> >>>>> takes more time ... Outputs of sysctl dev.igb0 and netstat -m will >> >>>> follow: >> >>>>> >> >>>>> 9219/99426/108645 mbufs in use (current/cache/total) >> >>>>> 9217/90783/100000/100000 mbuf clusters in use >> >> (current/cache/total/max) >> >>>> >> >>>> Do you see network processes are stuck in keglim state? If you see >> >>>> that I think that's not trivial to solve. You wouldn't even kill >> >>>> that process if it is under keglim state unless some more mbuf >> >>>> clusters are freed from other places. >> >>>> >> >>> >> >>> No keglim state, here is a snapshot of top -SH while the problem is >> >>> happening: >> >>> >> >>> 12 root 171 ki31 0K 8K CPU5 5 19:27 100.00% >> idle: >> >>> cpu5 >> >>> 10 root 171 ki31 0K 8K CPU7 7 19:26 100.00% >> idle: >> >>> cpu7 >> >>> 14 root 171 ki31 0K 8K CPU3 3 19:25 100.00% >> idle: >> >>> cpu3 >> >>> 11 root 171 ki31 0K 8K CPU6 6 19:25 100.00% >> idle: >> >>> cpu6 >> >>> 13 root 171 ki31 0K 8K CPU4 4 19:24 100.00% >> idle: >> >>> cpu4 >> >>> 15 root 171 ki31 0K 8K CPU2 2 19:22 100.00% >> idle: >> >>> cpu2 >> >>> 16 root 171 ki31 0K 8K CPU1 1 19:18 100.00% >> idle: >> >>> cpu1 >> >>> 17 root 171 ki31 0K 8K RUN 0 19:12 100.00% >> idle: >> >>> cpu0 >> >>> 18 root -32 - 0K 8K WAIT 6 0:04 0.10% swi= 4: >> >>> clock s >> >>> 20 root -44 - 0K 8K WAIT 4 0:08 0.00% swi= 1: >> >> net >> >>> 29 root -68 - 0K 8K - 0 0:02 0.00% igb= 0 >> >> que >> >>> 35 root -68 - 0K 8K - 2 0:02 0.00% em1 >> >> taskq >> >>> 28 root -68 - 0K 8K WAIT 5 0:01 0.00% >> irq256: >> >>> igb0 >> >>> >> >>> keep in mind that num_queues has been forced to 1. >> >>> >> >>> >> >>>> >> >>>> I think both igb(4) and em(4) pass received frame to upper stack >> >>>> before allocating new RX buffer. If driver fails to allocate new RX >> >>>> buffer driver will try to refill RX buffers in next run. Under >> >>>> extreme resource shortage case, this situation can produce no more >> >>>> RX buffers in RX descriptor ring and this will take the box out of >> >>>> network. Other drivers avoid that situation by allocating new RX >> >>>> buffer before passing received frame to upper stack. If RX buffer >> >>>> allocation fails driver will just reuse old RX buffer without >> >>>> passing received frame to upper stack. That does not completely >> >>>> solve the keglim issue though. I think you should have enough mbuf >> >>>> cluters to avoid keglim. >> >>>> >> >>>> However the output above indicates you have enough free mbuf >> >>>> clusters. So I guess igb(4) encountered zero available RX buffer >> >>>> situation in past but failed to refill the RX buffer again. I guess >> >>>> driver may be able to periodically check available RX buffers. >> >>>> Jack may have better idea if this was the case.(CCed) >> >>>> >> >>> >> >>> That is exactly the pattern. The driver runs out of clusters but the= y >> >>> eventually get consumed and freed although the driver refuses to >> process >> >> any >> >>> new frames. It is, on the other hand, perfectly capable of sending o= ut >> >>> packets. >> >>> >> >> >> >> Ok, this clearly indicates igb(4) failed to refill RX buffers since >> >> you can still send frames. I'm not sure whether igb(4) controllers >> >> could be configured to generate no RX buffer interrupts but that >> >> interrupt would be better suited to trigger RX refilling than timer >> >> based refilling. Since igb(4) keeps track of available RX buffers, >> >> igb(4) can selectively enable that interrupt once it see no RX >> >> buffers in the RX descriptor ring. However this does not work with >> >> polling. >> >> >> > >> > I think that your evaluation of the problem is correct although I do n= ot >> > understand the selective interrupt mechanism you described. >> > >> > Precisely, the exact same behavior happens (RX hang) if options >> > DEVICE_POLLING is _not_ used in the kernel configuration file. I tried >> with >> > POLLING since someone mentioned that it helped in a case mentioned >> earlier >> > today. Unfortunately for igb with or without polling yields the same r= x >> ring >> > filing problem. >> > >> > By the way I fixed the subject where I erroneously said TX was hanging >> while >> > in fact RX is hanging and TX is just fine. >> >> could you apply the attached patch and report what the value of >> rx_nxt_check and rx_nxt_refresh is when the interface hangs. >> You get the values using sysctl -a dev.igb >> >> Thanks for the help, here is the complete dev.igb.0 output including the > two new sysctls. Some observations: > > 1) As always this only happens when we run out of mbufs clusters: > > # netstat -m > 0/222/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 2) Both rx_nxt_check and rx_nxt_refresh are essentially the same througho= ut > the test but as soon as RX hangs, rx_nxt_check lags behind by 1. > > # sysctl dev.igb.0 > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.0.7 > dev.igb.0.%driver: igb > dev.igb.0.%location: slot=3D0 function=3D0 > dev.igb.0.%pnpinfo: vendor=3D0x8086 device=3D0x10a7 subvendor=3D0x8086 > subdevice=3D0x0000 class=3D0x020000 > dev.igb.0.%parent: pci7 > dev.igb.0.nvm: -1 > dev.igb.0.flow_control: 3 > dev.igb.0.enable_aim: 1 > dev.igb.0.rx_processing_limit: 100 > dev.igb.0.link_irq: 4 > dev.igb.0.dropped: 0 > dev.igb.0.tx_dma_fail: 0 > dev.igb.0.rx_overruns: 0 > dev.igb.0.watchdog_timeouts: 0 > dev.igb.0.device_control: 1490027073 > dev.igb.0.rx_control: 67141658 > dev.igb.0.interrupt_mask: 4 > dev.igb.0.extended_int_mask: 2147483905 > dev.igb.0.tx_buf_alloc: 14 > dev.igb.0.rx_buf_alloc: 34 > dev.igb.0.fc_high_water: 29488 > dev.igb.0.fc_low_water: 29480 > dev.igb.0.queue0.interrupt_rate: 100000 > dev.igb.0.queue0.txd_head: 825 > dev.igb.0.queue0.txd_tail: 825 > dev.igb.0.queue0.no_desc_avail: 0 > dev.igb.0.queue0.tx_packets: 75577 > dev.igb.0.queue0.rxd_head: 714 > dev.igb.0.queue0.rxd_tail: 714 > dev.igb.0.queue0.rx_packets: 141002 > dev.igb.0.queue0.rx_bytes: 208279964 > dev.igb.0.queue0.rx_nxt_refresh: 715 > dev.igb.0.queue0.rx_nxt_check: 714 > dev.igb.0.queue0.lro_queued: 0 > dev.igb.0.queue0.lro_flushed: 0 > dev.igb.0.mac_stats.excess_coll: 0 > dev.igb.0.mac_stats.single_coll: 0 > dev.igb.0.mac_stats.multiple_coll: 0 > dev.igb.0.mac_stats.late_coll: 0 > dev.igb.0.mac_stats.collision_count: 0 > dev.igb.0.mac_stats.symbol_errors: 0 > dev.igb.0.mac_stats.sequence_errors: 0 > dev.igb.0.mac_stats.defer_count: 0 > dev.igb.0.mac_stats.missed_packets: 2844 > dev.igb.0.mac_stats.recv_no_buff: 495 > dev.igb.0.mac_stats.recv_undersize: 0 > dev.igb.0.mac_stats.recv_fragmented: 0 > dev.igb.0.mac_stats.recv_oversize: 0 > dev.igb.0.mac_stats.recv_jabber: 0 > dev.igb.0.mac_stats.recv_errs: 0 > dev.igb.0.mac_stats.crc_errs: 0 > dev.igb.0.mac_stats.alignment_errs: 0 > dev.igb.0.mac_stats.coll_ext_errs: 0 > dev.igb.0.mac_stats.xon_recvd: 0 > dev.igb.0.mac_stats.xon_txd: 127 > dev.igb.0.mac_stats.xoff_recvd: 0 > dev.igb.0.mac_stats.xoff_txd: 2967 > dev.igb.0.mac_stats.total_pkts_recvd: 143870 > dev.igb.0.mac_stats.good_pkts_recvd: 141026 > dev.igb.0.mac_stats.bcast_pkts_recvd: 0 > dev.igb.0.mac_stats.mcast_pkts_recvd: 11 > dev.igb.0.mac_stats.rx_frames_64: 70 > dev.igb.0.mac_stats.rx_frames_65_127: 693 > dev.igb.0.mac_stats.rx_frames_128_255: 11 > dev.igb.0.mac_stats.rx_frames_256_511: 12 > dev.igb.0.mac_stats.rx_frames_512_1023: 20 > dev.igb.0.mac_stats.rx_frames_1024_1522: 140220 > dev.igb.0.mac_stats.good_octets_recvd: 208878321 > dev.igb.0.mac_stats.good_octets_txd: 4864240 > dev.igb.0.mac_stats.total_pkts_txd: 78671 > dev.igb.0.mac_stats.good_pkts_txd: 75577 > dev.igb.0.mac_stats.bcast_pkts_txd: 693 > dev.igb.0.mac_stats.mcast_pkts_txd: 346 > dev.igb.0.mac_stats.tx_frames_64: 74175 > dev.igb.0.mac_stats.tx_frames_65_127: 1402 > dev.igb.0.mac_stats.tx_frames_128_255: 0 > dev.igb.0.mac_stats.tx_frames_256_511: 0 > dev.igb.0.mac_stats.tx_frames_512_1023: 0 > dev.igb.0.mac_stats.tx_frames_1024_1522: 0 > dev.igb.0.mac_stats.tso_txd: 0 > dev.igb.0.mac_stats.tso_ctx_fail: 0 > dev.igb.0.interrupts.asserts: 5716 > dev.igb.0.interrupts.rx_pkt_timer: 141023 > dev.igb.0.interrupts.rx_abs_timer: 140978 > dev.igb.0.interrupts.tx_pkt_timer: 75577 > dev.igb.0.interrupts.tx_abs_timer: 0 > dev.igb.0.interrupts.tx_queue_empty: 75577 > dev.igb.0.interrupts.tx_queue_min_thresh: 0 > dev.igb.0.interrupts.rx_desc_min_thresh: 16 > dev.igb.0.interrupts.rx_overrun: 0 > dev.igb.0.host.breaker_tx_pkt: 0 > dev.igb.0.host.host_tx_pkt_discard: 0 > dev.igb.0.host.rx_pkt: 3 > dev.igb.0.host.breaker_rx_pkts: 0 > dev.igb.0.host.breaker_rx_pkt_drop: 0 > dev.igb.0.host.tx_good_pkt: 0 > dev.igb.0.host.breaker_tx_pkt_drop: 0 > dev.igb.0.host.rx_good_bytes: 208878321 > dev.igb.0.host.tx_good_bytes: 4864304 > dev.igb.0.host.length_errors: 0 > dev.igb.0.host.serdes_violation_pkt: 0 > dev.igb.0.host.header_redir_missed: 0 > > From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 16:48:16 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16A1C1065673; Tue, 8 Feb 2011 16:48:16 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from sunner.semmy.ru (sunner.semmy.ru [IPv6:2a00:14d0:0:20::3]) by mx1.freebsd.org (Postfix) with ESMTP id 79CC68FC14; Tue, 8 Feb 2011 16:48:15 +0000 (UTC) Received: from dhcp170-205-red.yandex.net ([95.108.170.205]) by sunner.semmy.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.74 (FreeBSD)) (envelope-from ) id 1Pmqjd-00056l-BU; Tue, 08 Feb 2011 19:48:13 +0300 Message-ID: <4D5173CD.3040808@FreeBSD.org> Date: Tue, 08 Feb 2011 19:48:13 +0300 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Julian Elischer References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> <4D501198.6090901@FreeBSD.org> In-Reply-To: <4D501198.6090901@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Ivo Vachkov , FreeBSD Net Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 16:48:16 -0000 07.02.2011 18:36, Sergey Matveychuk wrote: > 06.02.2011 4:42, Julian Elischer wrote: >> On 2/5/11 4:09 PM, Ivo Vachkov wrote: >>> Hello, >>> >>> How can I help? >> >> if you have ipv6 connectivity and experience, I have no experience or >> connectivity, with it so >> I'll be coding blind and will need a tester. >> If you have an application for IPV6 testing that would be even better. >> Divert is often used for NAT but that doesn't seem very useful for >> IPv6 and >> natd doesn't support it anyhow. > > Object :) > Divert is really useful way to get packets from firewall to userspace, > analyse or process them some way and put them back. Really I see no > other way for this for IPv6. I've tried ng_socket+ng_nat but there is no > easy way to put a packet back in firewall. Oops, I meant ng_socket+ng_ipfw here. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 16:53:31 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6841A106566B for ; Tue, 8 Feb 2011 16:53:31 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from sunner.semmy.ru (sunner.semmy.ru [IPv6:2a00:14d0:0:20::3]) by mx1.freebsd.org (Postfix) with ESMTP id 15C118FC0A for ; Tue, 8 Feb 2011 16:53:31 +0000 (UTC) Received: from dhcp170-205-red.yandex.net ([95.108.170.205]) by sunner.semmy.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.74 (FreeBSD)) (envelope-from ) id 1Pmqoj-00057v-Jc; Tue, 08 Feb 2011 19:53:29 +0300 Message-ID: <4D51750A.3070303@FreeBSD.org> Date: Tue, 08 Feb 2011 19:53:30 +0300 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Rozhuk.IM@gmail.com References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> <4D501198.6090901@FreeBSD.org> <4d516a6a.8937e30a.0996.2f26@mx.google.com> In-Reply-To: <4d516a6a.8937e30a.0996.2f26@mx.google.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 16:53:31 -0000 08.02.2011 19:08, rozhuk.im@gmail.com wrote: > Did you try ng_ether + ng_ksocket? > It can translate Ethernet frames incapsulated to udp to user space receiver. The idea is catch packets from firewall (ng_ipfw, ng_nat was mentioned by mistake) and pass them to user space module that do some processing and puts back the packets into firewall (for rules with `diverted' keyword). It works now for IPv4 with `divert' and doesn't with IPv6. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 17:01:59 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1A0A106566C for ; Tue, 8 Feb 2011 17:01:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6EABC8FC17 for ; Tue, 8 Feb 2011 17:01:59 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 06D5E46B06 for ; Tue, 8 Feb 2011 12:01:59 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.10]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 101528A009 for ; Tue, 8 Feb 2011 12:01:58 -0500 (EST) From: John Baldwin To: net@freebsd.org Date: Tue, 8 Feb 2011 11:55:33 -0500 User-Agent: KMail/1.13.5 (FreeBSD/7.4-CBSD-20110107; KDE/4.4.5; amd64; ; ) MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201102081155.33268.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 08 Feb 2011 12:01:58 -0500 (EST) X-Virus-Scanned: clamav-milter 0.96.3 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Flag: YES X-Spam-Status: Yes, score=6.8 required=4.2 tests=BAYES_00,MAY_BE_FORGED, RDNS_DYNAMIC, TO_NO_BRKTS_DIRECT, TO_NO_BRKTS_DYNIP autolearn=no version=3.3.1 X-Spam-Report: * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.0 RDNS_DYNAMIC Delivered to internal network by host with * dynamic-looking rDNS * 1.4 MAY_BE_FORGED Relay IP's reverse DNS does not resolve to IP * 2.6 TO_NO_BRKTS_DIRECT To: misformatted and direct-to-MX * 3.7 TO_NO_BRKTS_DYNIP To: misformatted and dynamic rDNS X-Spam-Level: ****** X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bigwig.baldwin.cx Cc: Subject: TCP can advertise a really huge window X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 17:01:59 -0000 This is a very bizarre edge case, so bear with me. I was debugging an edge case at work recently that occurred when the socket buffer was filled up exactly (i.e. sbspace(&so->so_rcv) == 0). In TCP terms, this would be when rcv_nxt == rcv_adv. To simulate the real workload I had a very fast writer blasting over lo0 to a slow reader but had used a small buffer and turned off window scaling (as I had to fill the entire socket buffer, I was chasing an off-by-1 bug). However, I ended up with some bizarre behavior. I think it is less confusing to describe the sequence of events that I now know happened than how I figured this out, so here goes. - Assume we have advertised a window size of N which corresponds exactly to sbspace(&so->so_rcv). - The remote peer sends a packet of length N filling our window. We respond with a zero-window ACK. This advances rcv_nxt to == rcv_adv, but it does not grow rcv_adv because sbspace() is currently 0. - The userland app very slowly drains data from the socket buffer. However, the calls to tcp_usr_recvd() do not trigger a window update because in this case the link is over lo0 which has a relatively large t_maxseg (about 14k) and this condition in tcp_output() is not met: if (adv >= (long) (2 * tp->t_maxseg)) goto send; if (2 * adv >= (long) so->so_rcv.sb_hiwat) goto send; - A timer at the remote peer expires and it sends a window probe with one byte of data. Since userland has read some data (just not 2 * MSS), we accept this packet. However, receiving this packet moves rcv_nxt += 1, so rcv_nxt is now > rcv_adv. - We call tcp_output() to ACK the window probe and as part of this calculate the receive window to advertise here: if (recwin < (long)(so->so_rcv.sb_hiwat / 4) && recwin < (long)tp->t_maxseg) recwin = 0; if (recwin < (long)(tp->rcv_adv - tp->rcv_nxt)) recwin = (long)(tp->rcv_adv - tp->rcv_nxt); if (recwin > (long)TCP_MAXWIN << tp->rcv_scale) recwin = (long)TCP_MAXWIN << tp->rcv_scale; The "surprise" kicks in on the second conditional. The problem is that rcv_adv - rcv_nxt is now equal to (uint32_t)-1. On a 32-bit machine the cast to (long) effectively just makes this value signed and thus -1. On a 64-bit machine you actually end up with a ginormous value of 2^32 - 1, or a 4GB window (minus a byte). The third conditional truncates that to the maximum window we can advertise, but this value may be larger than the actual space in the socket buffer. The remote peer now has a huge window to throw data into. At work this proved disastrous. I'm not sure if there are any practical concerns. This is the patch I'm using as a fix: Index: tcp_output.c =================================================================== --- tcp_output.c (revision 215582) +++ tcp_output.c (working copy) @@ -928,7 +928,8 @@ if (recwin < (long)(so->so_rcv.sb_hiwat / 4) && recwin < (long)tp->t_maxseg) recwin = 0; - if (recwin < (long)(tp->rcv_adv - tp->rcv_nxt)) + if (SEQ_GT(tp->rcv_adv, tp->rcv_nxt) && + recwin < (long)(tp->rcv_adv - tp->rcv_nxt)) recwin = (long)(tp->rcv_adv - tp->rcv_nxt); if (recwin > (long)TCP_MAXWIN << tp->rcv_scale) recwin = (long)TCP_MAXWIN << tp->rcv_scale; -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 17:02:03 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73DEB10656C4 for ; Tue, 8 Feb 2011 17:02:03 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3E3AF8FC15 for ; Tue, 8 Feb 2011 17:02:03 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id DD70246B03 for ; Tue, 8 Feb 2011 12:02:02 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.10]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 13B278A01D for ; Tue, 8 Feb 2011 12:01:59 -0500 (EST) From: John Baldwin To: net@freebsd.org Date: Tue, 8 Feb 2011 12:01:54 -0500 User-Agent: KMail/1.13.5 (FreeBSD/7.4-CBSD-20110107; KDE/4.4.5; amd64; ; ) MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201102081201.54250.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 08 Feb 2011 12:01:59 -0500 (EST) X-Virus-Scanned: clamav-milter 0.96.3 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Flag: YES X-Spam-Status: Yes, score=6.8 required=4.2 tests=BAYES_00,MAY_BE_FORGED, RDNS_DYNAMIC, TO_NO_BRKTS_DIRECT, TO_NO_BRKTS_DYNIP autolearn=no version=3.3.1 X-Spam-Report: * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.0 RDNS_DYNAMIC Delivered to internal network by host with * dynamic-looking rDNS * 1.4 MAY_BE_FORGED Relay IP's reverse DNS does not resolve to IP * 2.6 TO_NO_BRKTS_DIRECT To: misformatted and direct-to-MX * 3.7 TO_NO_BRKTS_DYNIP To: misformatted and dynamic rDNS X-Spam-Level: ****** X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bigwig.baldwin.cx Cc: Subject: A small TCP bug: excessive duplicate ACKs X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 17:02:03 -0000 One thing I've noticed at work is that if a receiver's socket buffer fills and the receiver then drains the buffer all at once, we send a lot of duplicate ACKs. I narrowed this down to being due to the abnormally high window scaling factor we have. We set kern.ipc.maxsockbuf to 314572800 which results in a window scaling factor of 8k. This interacts poorly with the logic that decides whether or not to force a window update in tcp_output(): /* * Compare available window to amount of window * known to peer (as advertised window less * next expected input). If the difference is at least two * max size segments, or at least 50% of the maximum possible * window, then want to send a window update to peer. * Skip this if the connection is in T/TCP half-open state. * Don't send pure window updates when the peer has closed * the connection and won't ever send more data. */ if (recwin > 0 && !(tp->t_flags & TF_NEEDSYN) && !TCPS_HAVERCVDFIN(tp->t_state)) { /* * "adv" is the amount we can increase the window, * taking into account that we are limited by * TCP_MAXWIN << tp->rcv_scale. */ long adv = min(recwin, (long)TCP_MAXWIN << tp->rcv_scale) - (tp->rcv_adv - tp->rcv_nxt); if (adv >= (long) (2 * tp->t_maxseg)) goto send; if (2 * adv >= (long) so->so_rcv.sb_hiwat) goto send; } Specifically, we can send a duplicate ACK when (2 * tp->t_maxseg) or (so->so_rcv.sb_hiwat / 2) are less than the window scaling factor. I have a test app that you can run against a TCP chargen service from inetd to reproduce it. I also have two TCP dumps from before and after. The patch I'm using to fix this is below (I could rework it to not use the extra goto perhaps, but went with a simple hack to minimize reindenting for now): Index: tcp_output.c =================================================================== --- tcp_output.c (revision 217650) +++ tcp_output.c (working copy) @@ -560,11 +560,19 @@ long adv = min(recwin, (long)TCP_MAXWIN << tp->rcv_scale) - (tp->rcv_adv - tp->rcv_nxt); + /* + * If the new window size ends up being the same as the old + * size when it is scaled, then don't force a window update. + */ + if ((tp->rcv_adv - tp->rcv_nxt) >> tp->rcv_scale == + (adv + tp->rcv_adv - tp->rcv_nxt) >> tp->rcv_scale) + goto dontupdate; if (adv >= (long) (2 * tp->t_maxseg)) goto send; if (2 * adv >= (long) so->so_rcv.sb_hiwat) goto send; } +dontupdate: /* * Send if we owe the peer an ACK, RST, SYN, or urgent data. ACKNOW Note that if the ACK sequence number has moved then I think other checks in tcp_output() will still force an ACK packet out, so I don't think this will cause us to miss on sending ACKs to the peers. You can find the test app source (tcpslow.c) and the dumps at http://people.freebsd.org/~jhb/tcpslow/ If you look at tcp_bad.out, the receiver stops reading data the receiver's socket buffer fills up around packet 72 or so. The receiver wakes up at packet 88 and drains the buffer causing a small storm of window updates. However, due to the scaling factor, it actually sends duplicate ACKs in batches of threes (3 ACKs for 8k window, 3 ACKs for 16k window, etc.). This happens each time the receiver wakes up and drains a full socket buffer. The tcp_good.out dump shows the stream with the patch applied. A similar event of the receiver draining a full buffer starts at packet 83 and it sends a single ACK for each "real" window update. -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 17:03:51 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E4ED1065698 for ; Tue, 8 Feb 2011 17:03:51 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id F29BD8FC1A for ; Tue, 8 Feb 2011 17:03:50 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p18H3eFp053298 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 8 Feb 2011 09:03:44 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4D517775.6090107@freebsd.org> Date: Tue, 08 Feb 2011 09:03:49 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Sergey Matveychuk References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> <4D501198.6090901@FreeBSD.org> <4d516a6a.8937e30a.0996.2f26@mx.google.com> <4D51750A.3070303@FreeBSD.org> In-Reply-To: <4D51750A.3070303@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Rozhuk.IM@gmail.com Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 17:03:51 -0000 > 08.02.2011 19:08, rozhuk.im@gmail.com wrote: >> Did you try ng_ether + ng_ksocket? >> It can translate Ethernet frames incapsulated to udp to user space >> receiver. > > The idea is catch packets from firewall (ng_ipfw, ng_nat was > mentioned by mistake) and pass them to user space module that do > some processing and puts back the packets into firewall (for rules > with `diverted' keyword). yes, however did you try the ipfw netgraph keyword and the ng_ipfw node? I have also been wondering it it might not make sense to simpply replavce the diver code with a netgraph equivalent.. Using the ng_ipfw node one can almost do it with no changes as it is. > > It works now for IPv4 with `divert' and doesn't with IPv6. yes, I'm pondering the right fix for that.. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 18:26:24 2011 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8D1C106564A for ; Tue, 8 Feb 2011 18:26:23 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id B04308FC1D for ; Tue, 8 Feb 2011 18:26:23 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id EC2D746B0C for ; Tue, 8 Feb 2011 13:26:22 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.10]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A655B8A02A for ; Tue, 8 Feb 2011 13:26:21 -0500 (EST) From: John Baldwin To: net@freebsd.org Date: Tue, 8 Feb 2011 13:26:21 -0500 User-Agent: KMail/1.13.5 (FreeBSD/7.4-CBSD-20110107; KDE/4.4.5; amd64; ; ) MIME-Version: 1.0 Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <201102081326.21183.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 08 Feb 2011 13:26:21 -0500 (EST) X-Virus-Scanned: clamav-milter 0.96.3 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Flag: YES X-Spam-Status: Yes, score=6.8 required=4.2 tests=BAYES_00,MAY_BE_FORGED, RDNS_DYNAMIC, TO_NO_BRKTS_DIRECT, TO_NO_BRKTS_DYNIP autolearn=no version=3.3.1 X-Spam-Report: * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.0 RDNS_DYNAMIC Delivered to internal network by host with * dynamic-looking rDNS * 1.4 MAY_BE_FORGED Relay IP's reverse DNS does not resolve to IP * 2.6 TO_NO_BRKTS_DIRECT To: misformatted and direct-to-MX * 3.7 TO_NO_BRKTS_DYNIP To: misformatted and dynamic rDNS X-Spam-Level: ****** X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bigwig.baldwin.cx Cc: Subject: TCP connections stuck in persist state X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 18:26:24 -0000 I ran into a problem recently where a TCP socket seemed to never exit persist mode. What would happen is that the sender was blasting data faster than the receiver could receive it. When the receiver read some data, the sender would start sending again and everything would resume. However, after 3-4 instances of this, the sender would decide to not resume sending data when the receiver opened the window. Instead, it would slowly send a byte every few seconds via the persist timer even though the receiver was advertising a 64k window when it ACKd each of the window probes that was sent by the sender. I dug around in kgdb and found that both snd_cwnd and snd_ssthresh were set to 0 on the sender side. I think this means that the send window is effectively permamently stuck at zero as a result of this. (The tcpcb is also IN_FASTRECOVERY() on the sender side, probably from the storm of duplicate acks from the receiver when it sends a bunch of window updates (see my earlier e-mail to net@ for the source of duplicate ACKs.) Anyway, I think that this code in tcp_input() is what keeps the window at zero: /* * If the congestion window was inflated to account * for the other side's cached packets, retract it. */ if (tcp_do_newreno || (tp->t_flags & TF_SACK_PERMIT)) { if (IN_FASTRECOVERY(tp)) { if (SEQ_LT(th->th_ack, tp->snd_recover)) { if (tp->t_flags & TF_SACK_PERMIT) tcp_sack_partialack(tp, th); else tcp_newreno_partial_ack(tp, th); } else { /* * Out of fast recovery. * Window inflation should have left us * with approximately snd_ssthresh * outstanding data. * But in case we would be inclined to * send a burst, better to do it via * the slow start mechanism. */ KASSERT(tp->snd_ssthresh != 0, ("using bogus snd_ssthresh")); if (SEQ_GT(th->th_ack + tp->snd_ssthresh, tp->snd_max)) tp->snd_cwnd = tp->snd_max - th->th_ack + tp->t_maxseg; else tp->snd_cwnd = tp->snd_ssthresh; } } Specifically, since snd_recover and snd_una seem to keep advancing in lock-step with each window update, I think it ends up falling down to the last statement each time where snd_cwnd = snd_ssthresh thus keeping snd_cwnd at 0. This then causes a zero send window in tcp_output(): sendwin = min(tp->snd_wnd, tp->snd_cwnd); sendwin = min(sendwin, tp->snd_bwnd); Now, looking at the code, I can see no way that snd_ssthresh should ever be zero. It seems to always be calculated from some number of segments times t_maxseg. The one exception to this rule is when it is restored from snd_ssthresh_prev due to a bad retransmit (this example is from tcp_input()): /* * If we just performed our first retransmit, and the ACK * arrives within our recovery window, then it was a mistake * to do the retransmit in the first place. Recover our * original cwnd and ssthresh, and proceed to transmit where * we left off. */ if (tp->t_rxtshift == 1 && (int)(ticks - tp->t_badrxtwin) < 0) { ++tcpstat.tcps_sndrexmitbad; tp->snd_cwnd = tp->snd_cwnd_prev; tp->snd_ssthresh = tp->snd_ssthresh_prev; tp->snd_recover = tp->snd_recover_prev; if (tp->t_flags & TF_WASFRECOVERY) ENTER_FASTRECOVERY(tp); tp->snd_nxt = tp->snd_max; tp->t_badrxtwin = 0; /* XXX probably not required */ } So then my working theory is that somehow, snd_ssthresh_prev is being used when it hasn't been initialized. I then checked 'ticks' on my host and found that it had wrapped to a negative integer value. So that means that when t_badrxtwin is zero (which is it's default value), the second half of that expression is always true. This is fine, however, so long as t_rxtshift is always 0 so long as snd_ssthresh_prev and friends aren't initialized. That brings me to tcp_setpersist(): void tcp_setpersist(struct tcpcb *tp) { int t = ((tp->t_srtt >> 2) + tp->t_rttvar) >> 1; int tt; if (tcp_timer_active(tp, TT_REXMT)) panic("tcp_setpersist: retransmit pending"); /* * Start/restart persistance timer. */ TCPT_RANGESET(tt, t * tcp_backoff[tp->t_rxtshift], TCPTV_PERSMIN, TCPTV_PERSMAX); tcp_timer_activate(tp, TT_PERSIST, tt); if (tp->t_rxtshift < TCP_MAXRXTSHIFT) tp->t_rxtshift++; } Suppose that a connection has never had to retransmit a packet, so snd_ssthresh_prev and friends are all set to their initial values of zero. Suppose that the receiver's socket then fills up eventually triggering the persist timer. Now when tcp_setpersist() is called, it will set t_rxtshift to 1 with all those *_prev fields invalid. The next ACK that arrives in tcp_input() with the 'ticks' value negative will trash snd_cwnd and friends from the bogus *_prev fields. I believe this was exposed relatively recently when I fixed problems with 'ticks' rollover a while back as the tests used to be 'if (ticks < tp->t_badrxtwin)' which have a different set of bugs, but that would explain why this hasn't been seen before. I was never able to come up with a small test case for this. The only way I could reproduce it and test the fix was using mysql (unfortunately) though we believe we have also seen this in different production workloads as well. After some whiteboard discussion with gnn@ I came up with a firmer timeline of exactly how this state was triggered on the sender: 1) The receiver is "busy" for a while and does not service the socket causing the receive buffer to fill. Once the receiver does start reading the socket, it drains the entire socket resulting in a small storm of ACKs all for the same sequence number. The window is growing during the ACKs, however, due to a bug in our window update code and large scale values (we have a scale factor of 8k because we increase the maximum socket buffer size), several of the ACKs end up being duplicates (e.g. 3 ACKs with a window of 8k then 3 ACKs with a window of 16k, then 3 with 24k, etc.). I think these ACKs trigger fast recovery so that IN_FASTRECOVERY() is true for the sender. 2) This cycle happens a second time on receiver (getting busy, then a flood of ACKs). In this case, the receiver is busy long enough to cause the sender to enter the persist state which sets t_rxtshift to one. 3) At this point an ACK arrives after the receiver started reading again. Because ticks is -ve and badrxtwin is zero and t_rxtshift is one, the code thinks this ACK should trigger 'bad retransmit recovery'. However, snd_ssthresh_prev and snd_cwnd_prev are both zero, so this sets both snd_ssthresh and snd_cwnd to zero. 4) Because IN_FASTRECOVERY() is true, then when an ACK arrives, it triggers either NewReno or SACK partial ACK handling code which clamps snd_cwnd to be no longer than snd_ssthresh, so snd_cwnd is zero. It seems that once the socket is "stuck", then each ACK to the window probes is treated as a partial ACK perpetuating the zero congestion window forever. The solution I came up with was to add a new flag to indicate when the *_prev fields are valid and test that before triggering bad retransmit recovery. The patch: Index: netinet/tcp_input.c =================================================================== --- netinet/tcp_input.c (.../mirror/FreeBSD/stable/7/sys) (revision 219412) +++ netinet/tcp_input.c (.../stable/7/sys) (revision 219412) @@ -1030,6 +1031,7 @@ * "bad retransmit" recovery. */ if (tp->t_rxtshift == 1 && + tp->t_flags & TF_PREVVALID && (int)(ticks - tp->t_badrxtwin) < 0) { ++tcpstat.tcps_sndrexmitbad; tp->snd_cwnd = tp->snd_cwnd_prev; @@ -1039,6 +1041,7 @@ if (tp->t_flags & TF_WASFRECOVERY) ENTER_FASTRECOVERY(tp); tp->snd_nxt = tp->snd_max; + tp->t_flags &= ~TF_PREVVALID; tp->t_badrxtwin = 0; } @@ -1961,7 +1968,8 @@ * original cwnd and ssthresh, and proceed to transmit where * we left off. */ - if (tp->t_rxtshift == 1 && (int)(ticks - tp->t_badrxtwin) < 0) { + if (tp->t_rxtshift == 1 && tp->t_flags & TF_PREVVALID && + (int)(ticks - tp->t_badrxtwin) < 0) { ++tcpstat.tcps_sndrexmitbad; tp->snd_cwnd = tp->snd_cwnd_prev; tp->snd_ssthresh = tp->snd_ssthresh_prev; @@ -1969,6 +1977,7 @@ if (tp->t_flags & TF_WASFRECOVERY) ENTER_FASTRECOVERY(tp); tp->snd_nxt = tp->snd_max; + tp->t_flags &= ~TF_PREVVALID; tp->t_badrxtwin = 0; /* XXX probably not required */ } Index: netinet/tcp_timer.c =================================================================== --- netinet/tcp_timer.c (.../mirror/FreeBSD/stable/7/sys) (revision 219412) +++ netinet/tcp_timer.c (.../stable/7/sys) (revision 219412) @@ -476,7 +476,9 @@ else tp->t_flags &= ~TF_WASFRECOVERY; tp->t_badrxtwin = ticks + (tp->t_srtt >> (TCP_RTT_SHIFT + 1)); - } + tp->t_flags |= TF_PREVVALID; + } else + tp->t_flags &= ~TF_PREVVALID; tcpstat.tcps_rexmttimeo++; if (tp->t_state == TCPS_SYN_SENT) rexmt = TCP_REXMTVAL(tp) * tcp_syn_backoff[tp->t_rxtshift]; Index: netinet/tcp_var.h =================================================================== --- netinet/tcp_var.h (.../mirror/FreeBSD/stable/7/sys) (revision 219412) +++ netinet/tcp_var.h (.../stable/7/sys) (revision 219412) @@ -201,6 +221,7 @@ #define TF_NEEDSYN 0x000400 /* send SYN (implicit state) */ #define TF_NEEDFIN 0x000800 /* send FIN (implicit state) */ #define TF_NOPUSH 0x001000 /* don't push */ +#define TF_PREVVALID 0x002000 /* saved values for bad rxmit valid */ #define TF_MORETOCOME 0x010000 /* More data to be appended to sock */ #define TF_LQ_OVERFLOW 0x020000 /* listen queue overflow */ #define TF_LASTIDLE 0x040000 /* connection was previously idle */ Index: netinet/tcp_output.c =================================================================== --- netinet/tcp_output.c (.../mirror/FreeBSD/stable/7/sys) (revision 219412) +++ netinet/tcp_output.c (.../stable/7/sys) (revision 219412) @@ -1258,6 +1269,7 @@ int t = ((tp->t_srtt >> 2) + tp->t_rttvar) >> 1; int tt; + tp->t_flags &= ~TF_PREVVALID; if (tcp_timer_active(tp, TT_REXMT)) panic("tcp_setpersist: retransmit pending"); /* -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 18:46:23 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5CD2106566B; Tue, 8 Feb 2011 18:46:23 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from sunner.semmy.ru (sunner.semmy.ru [IPv6:2a00:14d0:0:20::3]) by mx1.freebsd.org (Postfix) with ESMTP id 5AA338FC08; Tue, 8 Feb 2011 18:46:23 +0000 (UTC) Received: from dhcp170-205-red.yandex.net ([95.108.170.205]) by sunner.semmy.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.74 (FreeBSD)) (envelope-from ) id 1PmsZy-0005XU-Gc; Tue, 08 Feb 2011 21:46:22 +0300 Message-ID: <4D518F7E.3020804@FreeBSD.org> Date: Tue, 08 Feb 2011 21:46:22 +0300 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Julian Elischer References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> <4D501198.6090901@FreeBSD.org> <4d516a6a.8937e30a.0996.2f26@mx.google.com> <4D51750A.3070303@FreeBSD.org> <4D517775.6090107@freebsd.org> In-Reply-To: <4D517775.6090107@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 18:46:24 -0000 08.02.2011 20:03, Julian Elischer wrote: >> 08.02.2011 19:08, rozhuk.im@gmail.com wrote: >>> Did you try ng_ether + ng_ksocket? >>> It can translate Ethernet frames incapsulated to udp to user space >>> receiver. >> >> The idea is catch packets from firewall (ng_ipfw, ng_nat was mentioned >> by mistake) and pass them to user space module that do some processing >> and puts back the packets into firewall (for rules with `diverted' >> keyword). > > yes, however did you try the ipfw netgraph keyword and the ng_ipfw node? > I have also been wondering it it might not make sense to simpply > replavce the diver code with > a netgraph equivalent.. Using the ng_ipfw node one can almost do it with > no changes as it is. I've tried ng_socket+ng_ipfw. It gets incoming packets, but outgoing packets drops because of a tag having lost after leaving kernel space. It looks like a magic can be done with ng_tag node, but really I could not tame it. > >> >> It works now for IPv4 with `divert' and doesn't with IPv6. > > yes, I'm pondering the right fix for that.. I'm first to test it please :) From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 18:47:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 155A5106564A for ; Tue, 8 Feb 2011 18:47:21 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx22.fluidhosting.com [204.14.89.5]) by mx1.freebsd.org (Postfix) with ESMTP id AE7CE8FC21 for ; Tue, 8 Feb 2011 18:47:20 +0000 (UTC) Received: (qmail 19275 invoked by uid 399); 8 Feb 2011 18:47:17 -0000 Received: from localhost (HELO doug-optiplex.ka9q.net) (dougb@dougbarton.us@127.0.0.1) by localhost with ESMTPAM; 8 Feb 2011 18:47:17 -0000 X-Originating-IP: 127.0.0.1 X-Sender: dougb@dougbarton.us Message-ID: <4D518FB3.3040503@FreeBSD.org> Date: Tue, 08 Feb 2011 10:47:15 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.13) Gecko/20110129 Thunderbird/3.1.7 MIME-Version: 1.0 To: Ivo Vachkov References: <4D411CC6.1090202@gont.com.ar> <4D431258.8040704@FreeBSD.org> <4D437B13.1070405@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.1.2 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , bz@freebsd.org Subject: Re: Proposed patch for Port Randomization modifications according to RFC6056 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 18:47:21 -0000 I've been up and running on this patch vs. r218391 for over 24 hours now, using algorithm 4 (as someone said is now the default in Linux) without any problems. I think Bjoern is better qualified than I to comment on the style of the patch, but it applies cleanly, and seems to run fine on both v4 and v6. hth, Doug On 01/31/2011 04:52, Ivo Vachkov wrote: > Hello, > > I attach the latest version of the port randomization code as a patch > against RELENG_8. > > Changelog: > 1) sysctl variable names are changed to: > - 'net.inet.ip.portrange.randomalg.version' - representing the > algorithm of choice. > - 'net.inet.ip.portrange.randomalg.alg5_tradeoff' - representing the > Algorithm 5 computational tradeoff value (the 'N' value in the > Algorithm 5 description in the RFC 6056). > 2) Code comments are synchronized with the current variable names. > > Ivo Vachkov > > On Sat, Jan 29, 2011 at 4:27 AM, Doug Barton wrote: >> On 01/28/2011 11:57, Ivo Vachkov wrote: >>> >>> On Fri, Jan 28, 2011 at 9:00 PM, Doug Barton wrote: >> >>>> How does net.inet.ip.portrange.randomalg sound? I would also suggest that >>>> the second sysctl be named net.inet.ip.portrange.randomalg.alg5_tradeoff >>>> so >>>> that one could do 'sysctl net.inet.ip.portrange.randomalg' and see both >>>> values. But I won't quibble on that. :) >>>> >>> >>> I have no objections with this. Since this is my first attempt to >>> contribute something back to the community I decided to see how it's >>> done before. So I found: >>> net.inet.tcp.rfc1323 >>> net.inet.tcp.rfc3465 >>> net.inet.tcp.rfc3390 >>> net.inet.tcp.rfc3042 >>> which probably led me in a wrong direction :) >> >> Yeah, I had actually intended to say something to the effect of "there are >> plenty of unfortunate examples in the tree already so your doing it that way >> is totally understandable" but I trimmed it. >> >>> I understand your point and agree with it. However, my somewhat >>> limited understanding of the sysctl internal organization is telling >>> me that tree node does not support values. Am I wrong? >> >> You are likely correct. :) It's an inconvenient fact that often forget >> because that's not the sandbox that I usually play in. >> >>> If my reasoning >>> is correct, maybe I can create the sysctl variables with the following >>> names: >>> - net.inet.ip.portrange.randomalg (Tree Node) >>> - net.inet.ip.portrange.randomalg.alg[orithm] (Leaf Node, to store the >>> selected algorithm) >> >> I would go with "version" to increase the visual distinctiveness. I searched >> the current tree and there doesn't seem to be a clear winner for how to >> portray "this is the current N/M that is in use" but "version" seems to have >> the most representatives. >> >>> - net.inet.ip.portrange.randomalg.alg5_tradeoff (Leaf Node, to store >>> the Algorithm 5 trade-off value) >> >> I'm assuming this is the "N" value mentioned in the RFC. If so, I commend >> you on your choice of "tradeoff" to represent it. :) From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 18:47:30 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 636A3106567A for ; Tue, 8 Feb 2011 18:47:30 +0000 (UTC) (envelope-from rozhuk.im@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id C1A198FC08 for ; Tue, 8 Feb 2011 18:47:29 +0000 (UTC) Received: by wwf26 with SMTP id 26so6086897wwf.31 for ; Tue, 08 Feb 2011 10:47:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:reply-to:from:to:references:in-reply-to:subject :date:message-id:mime-version:content-type:content-transfer-encoding :x-mailer:content-language:thread-index; bh=1gWB84YkZbT+AGlVBvFyHC6g6jZg+FCibE3gRJ1yZDM=; b=o05K2wn1anRFlVYXr9HjekdHn2zraWjYtpJro0/w8uxrGzr8RaCr02vPNAzbT9EIct rR8cTbFDXDqxj8XQ/2wznw3OlydZxQAoect43eoAskbvGN2VwV8U2L+NXxmg+1NAozmo 6c7xwqrmRtM/jPpgAHn58UhjxWErRgyDKK734= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=reply-to:from:to:references:in-reply-to:subject:date:message-id :mime-version:content-type:content-transfer-encoding:x-mailer :content-language:thread-index; b=hCxKDXryv8o2XE0aG5Iv2jdxzkAzjxuQKoVglvWOWIBNB4GCJ6PkrE17L2kycNweTS R5uywqL1KwoDEZp/iF/xXVqhWyhexIw5+FOgKiVDqmKhAiUnulo1dfQJBTxPtXv1P7Kq mkd/Xe/dkwCgd/8tvFsswbDYq7pL/yGfcGhvE= Received: by 10.216.89.5 with SMTP id b5mr9890873wef.63.1297190842590; Tue, 08 Feb 2011 10:47:22 -0800 (PST) Received: from rimwks1x64 ([95.189.10.18]) by mx.google.com with ESMTPS id r38sm3012129weq.23.2011.02.08.10.47.20 (version=SSLv3 cipher=RC4-MD5); Tue, 08 Feb 2011 10:47:22 -0800 (PST) From: rozhuk.im@gmail.com To: References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> <4D501198.6090901@FreeBSD.org> <4d516a6a.8937e30a.0996.2f26@mx.google.com> <4D51750A.3070303@FreeBSD.org> In-Reply-To: <4D51750A.3070303@FreeBSD.org> Date: Wed, 9 Feb 2011 02:47:18 +0800 Message-ID: <4d518fba.26ead80a.02ff.1058@mx.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Content-Language: ru Thread-Index: AcvHsLcV3k/Dt9dQSxWHGgSEstfyRQADqirA Subject: RE: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Rozhuk.IM@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 18:47:30 -0000 > -----Original Message----- > From: Sergey Matveychuk [mailto:sem@FreeBSD.org] > Sent: Wednesday, February 09, 2011 12:53 AM > To: Rozhuk.IM@gmail.com > Cc: freebsd-net@freebsd.org > Subject: Re: divert rewrite >=20 > 08.02.2011 19:08, rozhuk.im@gmail.com wrote: > > Did you try ng_ether + ng_ksocket? > > It can translate Ethernet frames incapsulated to udp to user space > receiver. >=20 > The idea is catch packets from firewall (ng_ipfw, ng_nat was mentioned > by mistake) and pass them to user space module that do some processing > and puts back the packets into firewall (for rules with `diverted' > keyword). >=20 > It works now for IPv4 with `divert' and doesn't with IPv6. I know how divert works, google: uTPControl ;) Its simple for developmet, stable, but uses many CPU. With ng_ether + ng_ksocket you can send custom Ethernet frames. There is some node that can filter traffic, for IPv6 you need allow 1 or = 2 ethernet types to pass. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 19:40:16 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F2ADE106566C for ; Tue, 8 Feb 2011 19:40:16 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from sunner.semmy.ru (sunner.semmy.ru [IPv6:2a00:14d0:0:20::3]) by mx1.freebsd.org (Postfix) with ESMTP id AA2EB8FC22 for ; Tue, 8 Feb 2011 19:40:16 +0000 (UTC) Received: from dhcp170-205-red.yandex.net ([95.108.170.205]) by sunner.semmy.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.74 (FreeBSD)) (envelope-from ) id 1PmtQ7-0005jt-BR; Tue, 08 Feb 2011 22:40:15 +0300 Message-ID: <4D519C1F.9020205@FreeBSD.org> Date: Tue, 08 Feb 2011 22:40:15 +0300 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Rozhuk.IM@gmail.com References: <4D4DCD1E.1050906@freebsd.org> <4D4DFC95.9010804@freebsd.org> <4D501198.6090901@FreeBSD.org> <4d516a6a.8937e30a.0996.2f26@mx.google.com> <4D51750A.3070303@FreeBSD.org> <4d518fba.26ead80a.02ff.1058@mx.google.com> In-Reply-To: <4d518fba.26ead80a.02ff.1058@mx.google.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-net@freebsd.org Subject: Re: divert rewrite X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 19:40:17 -0000 08.02.2011 21:47, rozhuk.im@gmail.com пишет: >> -----Original Message----- >> From: Sergey Matveychuk [mailto:sem@FreeBSD.org] >> Sent: Wednesday, February 09, 2011 12:53 AM >> To: Rozhuk.IM@gmail.com >> Cc: freebsd-net@freebsd.org >> Subject: Re: divert rewrite >> >> 08.02.2011 19:08, rozhuk.im@gmail.com wrote: >>> Did you try ng_ether + ng_ksocket? >>> It can translate Ethernet frames incapsulated to udp to user space >> receiver. >> >> The idea is catch packets from firewall (ng_ipfw, ng_nat was mentioned >> by mistake) and pass them to user space module that do some processing >> and puts back the packets into firewall (for rules with `diverted' >> keyword). >> >> It works now for IPv4 with `divert' and doesn't with IPv6. > > I know how divert works, google: uTPControl ;) > Its simple for developmet, stable, but uses many CPU. > > With ng_ether + ng_ksocket you can send custom Ethernet frames. > There is some node that can filter traffic, for IPv6 you need allow 1 or 2 ethernet types to pass. I know. But I've written a module for conjunction with ipfw. It makes a decision by some criteria to pass a traffic or to block it. Administrators in our nets decide what kind traffic to pass to my module (mostly TCP SYN and few UDP) in their firewalls. So a conjection with ipfw is the goal. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 20:35:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 331F9106564A for ; Tue, 8 Feb 2011 20:35:21 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from fep27.mx.upcmail.net (fep27.mx.upcmail.net [62.179.121.47]) by mx1.freebsd.org (Postfix) with ESMTP id 9CEB18FC26 for ; Tue, 8 Feb 2011 20:35:20 +0000 (UTC) Received: from edge05.upcmail.net ([192.168.13.212]) by viefep18-int.chello.at (InterMail vM.8.01.02.02 201-2260-120-106-20100312) with ESMTP id <20110208203422.OQSV1353.viefep18-int.chello.at@edge05.upcmail.net>; Tue, 8 Feb 2011 21:34:22 +0100 Received: from pinky ([213.46.23.80]) by edge05.upcmail.net with edge id 5YaG1g01P1jgp3H05YaHvV; Tue, 08 Feb 2011 21:34:22 +0100 X-SourceIP: 213.46.23.80 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Pyun YongHyeon" References: <708793006.292748.1295186099577.JavaMail.root@erie.cs.uoguelph.ca> <20110117005524.GA1305@michelle.cdnetworks.com> <20110118.093804.74673434.sthaug@nethelp.no> <20110207002235.GA1244@michelle.cdnetworks.com> Date: Tue, 08 Feb 2011 21:34:15 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: <20110207002235.GA1244@michelle.cdnetworks.com> User-Agent: Opera Mail/11.01 (Win32) X-Cloudmark-Analysis: v=1.1 cv=FXfX1j538S0toKVv4AeTokQJ/Dr7x98IW2CddS+R9NE= c=1 sm=0 a=trLqpcBp2roA:10 a=bgpUlknNv7MA:10 a=kj9zAlcOel0A:10 a=pGLkceISAAAA:8 a=jDIqnEEcAAAA:8 a=6I5d2MoRAAAA:8 a=CZ0vI5ODKe6_T9WnYkEA:9 a=kNvgDllnPDXFXOSbYdwA:7 a=fSqStiGloApSD3bJASq2lZ35NGoA:4 a=CjuIK1q_8ugA:10 a=MSl-tDqOz04A:10 a=SV7veod9ZcQA:10 a=WAC6n6Nrh-yNq3b5:21 a=AHIb1GCgqi-6RLmM:21 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 Cc: freebsd-net@freebsd.org, rmacklem@uoguelph.ca, sthaug@nethelp.no Subject: Re: bogus 0 len IP packet, was: Hang in VOP_LOCK1_APV on 8-STABLE with NFS. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 20:35:21 -0000 On Mon, 07 Feb 2011 01:22:36 +0100, Pyun YongHyeon wrote: > On Sun, Feb 06, 2011 at 11:54:49PM +0100, Ronald Klop wrote: >> On Sat, 22 Jan 2011 00:01:47 +0100, Ronald Klop >> wrote: >> >> >On Tue, 18 Jan 2011 09:38:04 +0100, wrote: >> > >> >>>> So, does anyone have an idea why the IP length field would be set >> to >> >>>0 >> >>>> for these TCP/IP packets? >> >>>> >> >>>> Here's some info from Ronald w.r.t. his hardware. (All I can think >> >>>of is >> >>>> that he could try disabling TSO, etc?) >> >>>> >> >>>> Thanks in advance for any help with this, rick >> >>>> >> >>> >> >>>It seems that issue came from TSO. Driver will set ip_len and >> >>>ip_sum field to 0 before passing the TCP segment to controller. >> >>>The failed length were 4446, 5858, 3034 and 4310 and the total >> >>>number of such frames are more than 35k within 90 seconds. Since >> >>>failed length 4310 is continuously repeated I guess there is edge >> >>>case where em(4) didn't free failed TCP segment for TSO. >> >>>I remember there was commit to HEAD(r217295) which could be related >> >>>with this issue. >> >> >> >>I'm seeing the same problem with Broadcom NetXtreme (bce) cards: >> >> >> >>bce0@pci0:3:0:0: class=0x020000 card=0x03421014 chip=0x164c14e4 >> >>rev=0x12 hdr=0x00 >> >> vendor = 'Broadcom Corporation' >> >> device = 'Broadcom NetXtreme II Gigabit Ethernet Adapter >> >>(BCM5708)' >> >> class = network >> >> subclass = ethernet >> >> >> >>This is with 8.2-PRERELEASE. Turning off TSO (ifconfig bce0 -tso) >> >>removes the problem. >> >> >> >>Steinar Haug, Nethelp consulting, sthaug@nethelp.no >> >>_______________________________________________ >> >>freebsd-net@freebsd.org mailing list >> >>http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > >> >I tried -tso and -txcsum in various combinations, but it didn't solve >> >the problem. I wil look for another brand of network card to try. But >> >this has to wait till monday when I'm at the office again. >> >> I also used another network card (rl0) and it has the same problem with >> NFS. I'm going to change some network cables to see if that helps. I >> have >> some hints that there might be something wrong with that. >> > > Hmm, given that rl(4) also shows the issue it seems the issue could > be in TCP/IP stack, not in driver side. rl(4) is dumb device so > network stack should do segmentation and checksum computation. > I highly doubt the issue came from faulty cable since other users > also reported the same issue. > Unfortunately I have no clue yet and I was not able to reproduce it > on my box. I vaguely guess some code in kernel changed the ip_len > to 0 in the middle of transmission. Rick's captured traffic looks > normal except 0 ip_len given that controller is computing checksum > on the fly. If mbuf chain was corrupted(e.g. m_len == 0) driver > would have failed to send those frames. Changing the cable didn't help indeed. I'm glad the issue is seen by others too. I will try to downgrade to an older version of FreeBSD to try to find the commit which broke it. But that can take a while, because it is time consuming and I have to do some real work also at work. :-) Thanks for taking the time for it and I hope we will find the cause someday, Ronald. From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 20:50:58 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7715C106566B for ; Tue, 8 Feb 2011 20:50:58 +0000 (UTC) (envelope-from rozhuk.im@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id C01098FC1C for ; Tue, 8 Feb 2011 20:50:57 +0000 (UTC) Received: by wyf19 with SMTP id 19so6322758wyf.13 for ; Tue, 08 Feb 2011 12:50:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:reply-to:from:to:subject:date:message-id :mime-version:content-type:content-transfer-encoding:x-mailer :content-language:thread-index; bh=Is3FabBKi2NVNd4C4blXJmsYOr29k14/MpiXAlipG00=; b=rC6UswpHvFH607iurjQRTdXY4Bl4WtuVSe6zIzAL+GXGWZzwR6wJF3KT5pRnW3SJ6x 7IzTRLBVd580olUgnBIgi6uozMG/jf86PLnrBBKT1k6juUhhiGABUJYf7fKFjlPU0qVx fw0QMnfts6gFPfK6HYyNRBaGRHKk/THl1GUEo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=reply-to:from:to:subject:date:message-id:mime-version:content-type :content-transfer-encoding:x-mailer:content-language:thread-index; b=geL59fqLXQlyZKxoDsj1WtN7UwT1xLnC2UpTPqsCvYP1PrrFum/FXgUVe5BXb+2t+R 05Nz+SjotL1jZj5FwJ58WCYsMUDduYPlB/NEGBux15LJ8nGl6ig69ugH0rZ008kHNowY XLYX2vqYUn4cbWih9oB9DuBp+y6OTsTqOK5kU= Received: by 10.216.141.75 with SMTP id f53mr16227000wej.16.1297198256786; Tue, 08 Feb 2011 12:50:56 -0800 (PST) Received: from rimwks1x64 ([95.189.10.18]) by mx.google.com with ESMTPS id r57sm3083041wes.25.2011.02.08.12.50.54 (version=SSLv3 cipher=RC4-MD5); Tue, 08 Feb 2011 12:50:55 -0800 (PST) From: rozhuk.im@gmail.com To: Date: Wed, 9 Feb 2011 04:50:52 +0800 Message-ID: <4d51acaf.cf03d90a.514d.1989@mx.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1251" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Content-Language: ru Thread-Index: AcvH0d+NUThG+uAaTAOckg7k55u11w== Subject: jumbo frames + geom_mirror = no net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Rozhuk.IM@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 20:50:58 -0000 Hi! I have 8.2 + latest updates, em + gigabit net, few HDDs in mirror. Samba for share HDDs to win hosts. (E5300, G33 + ICH9R, 2GB, PCI-E intel desktop GB adapter)=20 ifconfig_em0=3D"inet 172.16.0.254 netmask 255.255.255.0 mtu 9000" Then I start copy files to mirror (trough net or using cp from others = HDD on host) after some time host stop respond to net, no errors messages in logs. (top show free mem < 10mb just before it happen) (It started after first mirror was created) And don=92t respond until reboot or: ifconfig em0 mtu 1500 (from = console) Vmstat show my failures for mbuf_jumbo_9k: mbuf_jumbo_9k: 9216, 6400, 0, 0, 36202768, 8472897 =20 but no failures for simple 1,5k mbuf's :/ vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 128, 0, 97, 23, 97, 0 UMA Zones: 888, 0, 97, 3, 97, 0 UMA Slabs: 284, 0, 1068, 514, 90680, 0 UMA RCntSlabs: 544, 0, 2649, 529, 13469257, 0 UMA Hash: 128, 0, 1, 29, 4, 0 16 Bucket: 76, 0, 12, 88, 122, 0 32 Bucket: 140, 0, 16, 96, 308, 0 64 Bucket: 268, 0, 43, 69, 561, 13 128 Bucket: 524, 0, 323, 6, 599692, 5140 VM OBJECT: 136, 0, 19972, 36317, 1260502, 0 MAP: 136, 0, 7, 22, 7, 0 KMAP ENTRY: 72, 57505, 2188, 2423, 4613615, 0 MAP ENTRY: 72, 0, 2418, 762, 4043456, 0 DP fakepg: 72, 0, 0, 0, 0, 0 SG fakepg: 72, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 175, 237, 175, 0 16: 16, 0, 4239, 836, 16310509, 0 32: 32, 0, 2514, 1780, 124403442, 0 64: 64, 0, 5044, 3334, 402804800, 0 128: 128, 0, 792, 1428, 7002330, 0 256: 256, 0, 592, 593, 1093975, 0 512: 512, 0, 165, 1035, 671161, 0 1024: 1024, 0, 55, 181, 882578, 0 2048: 2048, 0, 150, 250, 73957, 0 4096: 4096, 0, 132, 243, 734964, 0 Files: 56, 0, 5737, 628, 8850040, 0 TURNSTILE: 72, 0, 416, 64, 451, 0 umtx pi: 52, 0, 0, 0, 0, 0 PROC: 680, 0, 81, 171, 38814, 0 THREAD: 720, 0, 277, 138, 11326, 0 SLEEPQUEUE: 44, 0, 416, 115, 451, 0 VMSPACE: 228, 0, 57, 164, 31952, 0 cpuset: 40, 0, 2, 182, 2, 0 mbuf_packet: 256, 0, 4100, 609, 76165127, 0 mbuf: 256, 0, 175, 1206, 398211372, 0 mbuf_cluster: 2048, 65536, 4845, 453, 30848491, 0 mbuf_jumbo_page: 4096, 12800, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 36202768, 8472897 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 4, 402, 25681942, 0 ttyoutq: 256, 0, 64, 41, 136, 0 g_bio: 140, 0, 0, 1568, 205564803, 0 ttyinq: 152, 0, 120, 62, 255, 0 ata_request: 208, 0, 0, 304, 67886293, 0 ata_composite: 180, 0, 0, 0, 0, 0 VNODE: 268, 0, 19632, 27674, 5052284, 0 VNODEPOLL: 60, 0, 16, 173, 21, 0 S VFS Cache: 72, 0, 12259, 51977, 1280216, 0 L VFS Cache: 292, 0, 2976, 729, 70042, 0 NAMEI: 1024, 0, 0, 260, 153323563, 0 DIRHASH: 1024, 0, 57, 343, 57340, 0 AIO: 120, 0, 3, 93, 19, 0 AIOP: 16, 0, 4, 402, 44, 0 AIOCB: 292, 0, 0, 390, 6897050, 0 AIOL: 64, 0, 0, 0, 0, 0 AIOLIO: 168, 0, 0, 0, 0, 0 pipe: 392, 0, 50, 170, 25245, 0 ksiginfo: 80, 0, 244, 332, 509687, 0 itimer: 220, 0, 1, 53, 15, 0 KNOTE: 72, 0, 1901, 643, 153128799, 0 socket: 412, 204804, 1936, 755, 4049131, 0 ipq: 32, 2147, 0, 0, 0, 0 udp_inpcb: 220, 204804, 18, 306, 105359, 0 udpcb: 8, 204827, 18, 388, 105359, 0 tcp_inpcb: 220, 204804, 1991, 943, 3034192, 0 tcpcb: 632, 204804, 1859, 829, 3034192, 0 tcptw: 52, 40968, 132, 588, 863691, 0 syncache: 112, 102410, 2, 383, 756808, 0 hostcache: 76, 15400, 577, 5623, 202354, 0 tcpreass: 20, 4225, 4, 503, 16547113, 0 sackhole: 20, 0, 2, 505, 816717, 0 ripcb: 220, 204804, 1, 35, 2, 0 unpcb: 172, 204815, 44, 324, 909334, 0 rtentry: 108, 0, 6, 102, 51, 0 selfd: 28, 0, 472, 417, 44350642, 0 SWAPMETA: 276, 121576, 0, 0, 0, 0 Mountpoints: 644, 0, 17, 19, 18, 0 FFS inode: 116, 0, 19560, 35385, 1771335, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 19560, 24255, 1771327, 0 md1: 512, 0, 121, 15, 121, 0 pfsrctrpl: 124, 10013, 0, 62, 6, 0 pfrulepl: 828, 0, 145, 347, 8368, 0 pfstatepl: 284, 16394, 5564, 1842, 4902363, 0 pfaltqpl: 224, 0, 0, 0, 0, 0 pfpooladdrpl: 68, 0, 26, 198, 1148, 0 pfrktable: 1240, 1002, 14, 46, 1086, 0 pfrkentry: 156, 200000, 46, 229, 4271, 0 pfrkentry2: 156, 0, 0, 0, 0, 0 pffrent: 16, 5075, 0, 406, 38707, 0 pffrag: 48, 0, 0, 312, 19255, 0 pffrcache: 48, 10062, 0, 0, 0, 0 pffrcent: 12, 50141, 0, 0, 0, 0 pfstatescrub: 28, 0, 56, 706, 99700, 0 pfiaddrpl: 100, 0, 73, 239, 2428, 0 pfospfen: 108, 0, 696, 240, 54288, 0 pfosfp: 28, 0, 407, 228, 31746, 0 NetGraph items: 36, 4130, 0, 354, 311914, 0 NetGraph data items: 36, 531, 0, 295, 217845511, 0 vmstat -m Type InUse MemUse HighUse Requests Size(s) session 38 3K - 8031 64 pgrp 44 3K - 8459 64 cred 218 21K - 6348824 64,128 uidinfo 8 2K - 645160 64,1024 plimit 19 5K - 68613 256 CAM SIM 10 2K - 10 128 sysctltmp 0 0K - 7474 16,32,64,128,2048 sysctloid 2712 83K - 2853 16,32,64 sysctl 0 0K - 2461365 16,32,64 callout 1 256K - 1 =20 umtx 415 26K - 450 64 p1003.1b 1 1K - 1 16 CAM periph 14 2K - 30 16,32,128 bus-sc 52 47K - 1208 16,32,64,128,256,512,1024,2048,4096 bus 575 31K - 3448 16,32,64,128,1024 devstat 20 41K - 20 16,4096 eventhandler 75 4K - 75 32,64,128 CAM XPT 95 37K - 13819 16,32,64,1024,2048 kobj 84 168K - 176 2048 Per-cpu 1 1K - 1 16 isadev 10 1K - 10 64 gprof 1 90840K - 3 =20 rman 162 10K - 572 16,32,64 entropy 1024 64K - 1024 64 sbuf 0 0K - 100620 16,32,64,128,256,512,1024,2048,4096 DEVFS1 90 23K - 134 256 DEVFS3 121 16K - 165 128,256 taskqueue 15 1K - 15 16,64 Unitno 12 1K - 22352 16,64 iov 0 0K - 20753624 16,64,128,256 select 232 15K - 232 64 ioctlops 0 0K - 1154325 16,32,64,128,256,512,1024,2048,4096 tty 23 12K - 30 512,1024 pts 6 1K - 11 128 accf 5 1K - 21 16,32 mbuf_tag 5 1K - 377663504 32,64 shmfd 1 4K - 1 4096 cdev 7 1K - 7 128 pcb 1344 152K - 2390492 16,32,512,1024 soname 29 3K - 9094379 16,32,128 acl 0 0K - 26680 4096 biobuf 1 2K - 4552 2048 vfscache 1 512K - 1 =20 cl_savebuf 0 0K - 162042 32,64 vfs_hash 1 256K - 1 =20 vnodes 4 1K - 4 32,128 vnodemarker 0 0K - 346290 512 mount 240 8K - 332 16,32,64,128,256 BPF 11 74K - 63 16,64,128,256,1024,4096 ether_multi 11 1K - 76 16,32,64 ifaddr 29 8K - 116 16,32,256,2048 ifnet 5 5K - 30 64,1024 clone 1 4K - 1 4096 arpcom 1 1K - 1 16 lltable 34 5K - 7502 128,256 DEVFS 7 1K - 8 16,64 sigio 3 1K - 5 32 filedesc 129 132K - 42677 16,32,64,128,256,512,1024,2048,4096 kenv 106 8K - 123 16,32,64,128,4096 kqueue 18 32K - 103730 32,128,256,1024,2048 DEVFSP 2 1K - 8 32 proc-args 51 3K - 45155 16,32,64,128,256 routetbl 54 10K - 44783 16,32,64,128,256 igmp 4 1K - 29 128 ipid 2 24K - 2 =20 in_multi 4 1K - 36 128 hostcache 1 512K - 1 =20 syncache 1 144K - 1 =20 savedino 0 0K - 93850 256 newdirblk 0 0K - 112 32 dirrem 1 1K - 839105 32 mkdir 0 0K - 9352 32 diradd 2 1K - 165136 64 freefile 1 1K - 775184 32 freeblks 1 1K - 784281 256 freefrag 0 0K - 392940 32 allocindir 345 22K - 59161846 64 indirdep 3 1K - 87473 32 allocdirect 2 1K - 1367113 128 bmsafemap 5 1K - 109568 64 newblk 1 1K - 60528960 64,256 inodedep 6 257K - 881018 128 pagedep 4 65K - 27393 64 ufs_dirhash 114 57K - 46296 16,32,64,128,256,512 ufs_mount 24 422K - 27 256,2048,4096 UMAHash 3 5K - 8 256,512,1024,2048,4096 ithread 65 6K - 65 16,64,128 GEOM 260 37K - 4447 = 16,32,64,128,256,512,1024,4096 vm_pgdata 1 64K - 1 =20 linker 196 611K - 538 = 16,32,64,128,256,1024,2048,4096 atkbddev 1 1K - 1 32 cpuctl 1 1K - 1 16 lockf 56 4K - 1001219 32,64 io_apic 1 1K - 1 1024 memdesc 1 4K - 1 4096 temp 31 592K - 2837830 16,32,64,128,256,512,1024,2048,4096 devbuf 1201 5184K - 1244 16,32,64,128,256,512,1024,2048,4096 module 142 9K - 142 64,128 LED 1 1K - 1 64 mtx_pool 2 8K - 2 4096 CAM dev queue 10 2K - 10 128 CAM queue 42 1K - 124 16 subproc 338 400K - 39116 256,4096 proc 2 8K - 2 4096 msi 3 1K - 3 64 nexusdev 4 1K - 4 16 mirror_data 6 2K - 288108 16,32,128,512 acpidev 83 3K - 83 32 acpica 2504 132K - 78527 16,32,64,128,256,512,1024 acpitask 1 1K - 1 1024 acpisem 15 2K - 15 64,128 apmdev 1 1K - 1 64 pci_link 16 2K - 16 64,128 md_sectors 3 12K - 3 4096 md_disk 5 5K - 5 16,2048 nullfs_node 13 1K - 3281277 16 nullfs_hash 1 1K - 1 64 nullfs_mount 8 1K - 8 16 netgraph_msg 0 0K - 99588 64,128,256,512,1024 netgraph_node 22 3K - 345 128 netgraph_hook 38 5K - 900 128 netgraph 8 9K - 142 32,64,256 netgraph_sock 7 1K - 101 64 netgraph_path 0 0K - 50601 16,32 netgraph_mppc 2 2K - 37 1024 netgraph_iface 2 1K - 27 64 netgraph_ppp 2 16K - 27 =20 netgraph_pppoe 2 9K - 15 64,512 netgraph_l2tp 2 2K - 48 64,1024 netgraph_ksock 1 1K - 22 64 =A0 -- Rozhuk Ivan =A0=20 From owner-freebsd-net@FreeBSD.ORG Tue Feb 8 22:49:00 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A97D1106564A; Tue, 8 Feb 2011 22:49:00 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 37A938FC0C; Tue, 8 Feb 2011 22:49:00 +0000 (UTC) Received: by gxk8 with SMTP id 8so2393344gxk.13 for ; Tue, 08 Feb 2011 14:48:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=+O7h9Tm3Utg8t3KLBqENCwVJIIxG17/P2q4aQElE7Hc=; b=H8Ntrh8vOeAuftbNjPb940vtAQoMcV7CpfcsvwJvY/A38XLUyclBEkKzuN2yafLwv3 z5UpvSM7GFAbME9eV0tWDP/96Fn1onoY9crtdxWfSUQ4eFFNUoTl30GWZsNwE7miNrNR DOIoq/wSaUSHj03Z9OC7XW/YaWLl0gVbkCuPg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=LPfmMynTN360c0PLIp0modC6WKxBNPcT5P9uclnehDQX+jKQdFV65xIogk/mbnDUWC zZ3S3iAR+FumgJ8aYcNgkEzBCL8TJ4yqvcxOdMwTIXqOSedz4MtBwf+JacQ9fButA9iG 5LBYl298Wq36GWKYLwrdS/fCSwUlYQu5Mrpvw= MIME-Version: 1.0 Received: by 10.91.54.29 with SMTP id g29mr630911agk.46.1297205339438; Tue, 08 Feb 2011 14:48:59 -0800 (PST) Received: by 10.147.171.17 with HTTP; Tue, 8 Feb 2011 14:48:59 -0800 (PST) In-Reply-To: References: Date: Tue, 8 Feb 2011 14:48:59 -0800 Message-ID: From: Jack Vogel To: Karim Fodil-Lemelin , Pyun YongHyeon , Michael Tuexen , beezarliu Content-Type: multipart/mixed; boundary=001485f92346754a72049bcd27d7 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 22:49:00 -0000 --001485f92346754a72049bcd27d7 Content-Type: text/plain; charset=ISO-8859-1 I have been following this, and thinking about it. I still am working from a theoretical standpoint, but based on a patch I got quite a long time back and never quite groked, I believe now that I might have a solution. The original PR and patch was kern/150516 from Beezar Liu, I was never quite comfortable with the code changes, nor convinced that it was a real issue and not a misunderstanding. However I think now that this very report might be behind what we are seeing today. I have a slightly different approach to solving it, of course it remains to be seen if it handles it properly. Please try the patch I've attached, I'm open to further correction or polishing of the changes. And thanks to Beezar for his original report and changes, this is not for em, but if this eliminates the problem its clearly needed in all drivers. Jack --001485f92346754a72049bcd27d7 Content-Type: text/x-patch; charset=US-ASCII; name="igb.patch" Content-Disposition: attachment; filename="igb.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gjwwm4jo0 UHJveHlDaGFpbnMtMy4xIChodHRwOi8vcHJveHljaGFpbnMuc2YubmV0KQpJbmRleDogaWZfaWdi LmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PQotLS0gaWZfaWdiLmMJKHJldmlzaW9uIDIxODQ2MykKKysrIGlmX2lnYi5j CSh3b3JraW5nIGNvcHkpCkBAIC00MzEyLDYgKzQzMTIsNyBAQAogCQlzdHJ1Y3QgbWJ1ZgkJKnNl bmRtcCwgKm1oLCAqbXA7CiAJCXN0cnVjdCBpZ2JfcnhfYnVmCSpyeGJ1ZjsKIAkJdTE2CQkJaGxl biwgcGxlbiwgaGRyLCB2dGFnOworCQlpbnQJCQljb21taXQ7CiAJCWJvb2wJCQllb3AgPSBGQUxT RTsKICAKIAkJY3VyID0gJnJ4ci0+cnhfYmFzZVtpXTsKQEAgLTQ0NDAsMTAgKzQ0NDEsMjIgQEAK IAkJYnVzX2RtYW1hcF9zeW5jKHJ4ci0+cnhkbWEuZG1hX3RhZywgcnhyLT5yeGRtYS5kbWFfbWFw LAogCQkgICAgQlVTX0RNQVNZTkNfUFJFUkVBRCB8IEJVU19ETUFTWU5DX1BSRVdSSVRFKTsKIAor CQljb21taXQgPSBpOwkvKiBjYXB0dXJlIHRoZSBvbGQgaW5kZXggKi8KKwogCQkvKiBBZHZhbmNl IG91ciBwb2ludGVycyB0byB0aGUgbmV4dCBkZXNjcmlwdG9yLiAqLwogCQlpZiAoKytpID09IGFk YXB0ZXItPm51bV9yeF9kZXNjKQogCQkJaSA9IDA7CiAJCS8qCisJCSoqIFNhbml0eSB0ZXN0IGZv ciByaW5nIGZ1bGwsIGlmIHRoaXMKKwkJKiogaGFwcGVucyB3ZSBuZWVkIHRvIHJlZnJlc2ggaW1t ZWRpYXRlbHkKKwkJKiogb3IgcmVmcmVzaCBtYXkgZGVhZGxvY2suCisJCSovCisJCWlmIChpID09 IHJ4ci0+bmV4dF90b19yZWZyZXNoKSB7CisJCQlpZ2JfcmVmcmVzaF9tYnVmcyhyeHIsIGNvbW1p dCk7CisJCQlwcm9jZXNzZWQgPSAwOworCQl9CisKKwkJLyoKIAkJKiogU2VuZCB0byB0aGUgc3Rh Y2sgb3IgTFJPCiAJCSovCiAJCWlmIChzZW5kbXAgIT0gTlVMTCkgewo= --001485f92346754a72049bcd27d7-- From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 00:06:40 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1252C106564A for ; Wed, 9 Feb 2011 00:06:40 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id CA74A8FC0C for ; Wed, 9 Feb 2011 00:06:39 +0000 (UTC) Received: by pxi1 with SMTP id 1so1429314pxi.13 for ; Tue, 08 Feb 2011 16:06:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=8yRWktruUSzbEXO9d+RKngwbTwTKYApx9YPbuiPhxcU=; b=PL9YUm7yaq7Ux7Za8o0qYk4GxoF9qhmytgoy2Ng54+zeztSGSZuF1lFkH61vvgNH32 8vCOOTffKbcUeeFZZ6cH0p341RMuvs6rHhPQFibcxasI28dgx9sSqEU47WuQFh5cEQBo 7c+xFl9jKgBvjcKJ35g8mU9/9vR7saWLsB/qw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=o1xcnlTQzRzRMujrQjWz4cPdFmURi25Wfa0CeJNsl5muNSNRU08XLE/ghMFZIYQ5lf iHADIYMrsRCRwtcJkpqxrvVO2oQea/M4iUMaKops/5PjZPCwMD5Dlev7ilnDXUZdISRi Dv66kI2/CaLiLFQdAbR/8AGhtCShIiN0jJRrA= MIME-Version: 1.0 Received: by 10.142.216.3 with SMTP id o3mr17667724wfg.29.1297209999308; Tue, 08 Feb 2011 16:06:39 -0800 (PST) Received: by 10.142.246.3 with HTTP; Tue, 8 Feb 2011 16:06:39 -0800 (PST) In-Reply-To: References: Date: Tue, 8 Feb 2011 19:06:39 -0500 Message-ID: From: Karim Fodil-Lemelin To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Pyun YongHyeon , beezarliu , Michael Tuexen , freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 00:06:40 -0000 2011/2/8 Jack Vogel > > I have been following this, and thinking about it. I still am working from > a theoretical > standpoint, but based on a patch I got quite a long time back and never > quite groked, > I believe now that I might have a solution. > > The original PR and patch was kern/150516 from Beezar Liu, I was never > quite comfortable > with the code changes, nor convinced that it was a real issue and not a > misunderstanding. > However I think now that this very report might be behind what we are > seeing today. I have > a slightly different approach to solving it, of course it remains to be > seen if it handles it > properly. > > Please try the patch I've attached, I'm open to further correction or > polishing of the > changes. And thanks to Beezar for his original report and changes, this is > not for em, > but if this eliminates the problem its clearly needed in all drivers. > > Jack > > > Hi Jack, Thanks for your help. I tried your patch and it didn't work so I added a couple of printf to see if the added code was getting hit: --- a/freebsd/sys/dev/e1000/if_igb.c --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c @@ -612,7 +612,7 @@ igb_attach(device_t dev) device_get_nameunit(dev)); INIT_DEBUGOUT("igb_attach: end"); - + printf("this driver has a patch from Jack Vogel\n"); return (0); err_late: @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, int *done) struct mbuf *sendmp, *mh, *mp; struct igb_rx_buf *rxbuf; u16 hlen, plen, hdr, vtag; + int commit; bool eop = FALSE; cur = &rxr->rx_base[i]; @@ -4255,10 +4256,23 @@ next_desc: bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); + commit = i; /* capture the old index */ + /* Advance our pointers to the next descriptor. */ if (++i == adapter->num_rx_desc) i = 0; /* + ** Sanity test for ring full, if this + ** happens we need to refresh immediately + ** or refresh may deadlock. + */ + if (i == rxr->next_to_refresh) { + igb_refresh_mbufs(rxr, commit); + printf("igb_refresh_mbufs called with commit %d\n", commit); + processed = 0; + } + + /* ** Send to the stack or LRO */ if (sendmp != NULL) { Here is the results: # dmesg | grep Vogel this driver has a patch from Jack Vogel this driver has a patch from Jack Vogel # netstat -m 60453/52707/113160 mbufs in use (current/cache/total) 48416/51584/100000/100000 mbuf clusters in use (current/cache/total/max) 2894/690 mbuf+clusters out of packet secondary zone in use (current/cache) 11946/854/12800/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 164834K/119760K/284595K bytes allocated to network (current/cache/total) 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/4/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines # dmesg | grep commit At this point RX has hung. Somehow the check (i == rxr->next_to_refresh) is never true in this case. Also, I did read kern/150516 and couldn't wrap my head around the patch for the em driver that Beezar Liu suggested. Regards, Karim. From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 05:24:29 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF1E7106566B; Wed, 9 Feb 2011 05:24:29 +0000 (UTC) (envelope-from alex@zagrebin.ru) Received: from mail.zagrebin.ru (gw.zagrebin.ru [91.215.205.128]) by mx1.freebsd.org (Postfix) with ESMTP id 7A4BA8FC1C; Wed, 9 Feb 2011 05:24:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zagrebin.ru; s=mail; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=shAURmZv/dlS1aBv1b/Xa/RvYVbRQRJqIzPdsGPnttk=; b=FOxWT7MFAWot1nIoS8t/dHZUc8YPQSdTJ8ubIN+FdJKCX1dvghjWlltbbhtAgc1krn3Afi+ctiu8EqzPUw7uh2wST9Cepm7QO5HNVgkpLdYyizLn4f9kbgqhO4XyQ74qb1sIsSXad6Vig3J8qE1liwG7X/oKSlDHb/txqo8sGJA=; Received: from alex by mail.zagrebin.ru with local (Exim 4.74 (FreeBSD)) (envelope-from ) id 1Pn2XQ-0001Jf-BT; Wed, 09 Feb 2011 08:24:24 +0300 Date: Wed, 9 Feb 2011 08:24:24 +0300 From: Alexander Zagrebin To: Bernhard Schmidt Message-ID: <20110209052423.GA4866@gw.zagrebin.ru> References: <20110204060808.GA97298@gw.zagrebin.ru> <201102061142.43865.bschmidt@freebsd.org> <95679.17379.qm@web39320.mail.mud.yahoo.com> <201102081052.54016.bschmidt@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201102081052.54016.bschmidt@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org, PseudoCylon Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 05:24:29 -0000 Hi! On 08.02.2011 10:52:53 +0100, Bernhard Schmidt wrote: > I've combined both patches (see attachment), if I get an ACK from both > of you I'll try get this into the tree ASAP. The resulted patch works fine for me. Big thanks for your help! Waiting for the 802.11n support... :) -- Alexander Zagrebin From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 05:25:07 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBCD81065697 for ; Wed, 9 Feb 2011 05:25:07 +0000 (UTC) (envelope-from moonlightakkiy@yahoo.ca) Received: from web39307.mail.mud.yahoo.com (web39307.mail.mud.yahoo.com [66.94.238.174]) by mx1.freebsd.org (Postfix) with SMTP id 0C7768FC24 for ; Wed, 9 Feb 2011 05:25:06 +0000 (UTC) Received: (qmail 92078 invoked by uid 60001); 9 Feb 2011 05:25:06 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.ca; s=s1024; t=1297229106; bh=1WxnUrmfUklNLkn0op0flu08KDoitMdPJfo3wUdwqCo=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=5Q6tVwPr2vhm0+ucabGO/A3fWEiLuFOM56gbvdNtbgRx/os8NY41/GuHRybWzxm1pdppM7sCgFcZJjElDfwRGFoaQWgw038T0LeeZmTz4zaQb3lY9qSgUfH5lRe4tn8YjcN5maIqWZI2lFYCaZ6gXy9jnR3ffWvbYuAaVwFdr3s= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.ca; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=EtPv0hHjEYnqrqCwLS4ogG3n5VEBpBgQvRsDv9CQ1vEM3OaVpXTo2rNnzrq/ehLOlOBXvNwnpXzdbZUj9qHEshaixYuE9lT+PGJ4vSse6OY/bKwfbdP7JMEoN1R1iMJ+slW+inoFX1rGsm8q9K1mtd3saASrCtAnPZFVnRupCDk=; Message-ID: <463483.30615.qm@web39307.mail.mud.yahoo.com> X-YMail-OSG: igs2F2cVM1mg5xYpBMk3NknFkwyLdPqbMpx9f_K_2B7dus1 vC1lKwyQ1vbYkLGdIM_aXI2px2X_Rkxfz2EbjB8pIV2C4tbu7GCx9KyyAsJQ k_8lF5Goxm9UPbs4emGIfSf3ukk0nscf2IFgQjybuh39CYBgdcwW9K8jXhCM nNNDQzqxAS9J7h0Cyb2KJjpGT91LyQ7fFtLZPO24z1MGuAamhxLNoed7QDkR WagTL89jfSDhAOiQd2b4T0YawOUqZqzmZ133HoRjNYG22W5mJJaYdCSgfI6. wPHKD81ksqfcvYAYj3kg8zk0JtQOs74MJgAYNFE8bsIoS9T4z6Iehlw1Oe74 gkSUnb10SkpfKYgZzT_7hhocAhQfIYLyn2DrWw_a3QrX8x8.7hw2o_Wb7HrM 0vNQfsdBK92PGhOo- Received: from [206.75.146.55] by web39307.mail.mud.yahoo.com via HTTP; Tue, 08 Feb 2011 21:25:06 PST X-Mailer: YahooMailRC/555 YahooMailWebService/0.8.108.291010 References: <20110204060808.GA97298@gw.zagrebin.ru> <201102061142.43865.bschmidt@freebsd.org> <95679.17379.qm@web39320.mail.mud.yahoo.com> <201102081052.54016.bschmidt@freebsd.org> Date: Tue, 8 Feb 2011 21:25:06 -0800 (PST) From: PseudoCylon To: bschmidt@freebsd.org, Alexander Zagrebin In-Reply-To: <201102081052.54016.bschmidt@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-net@freebsd.org Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 05:25:07 -0000 ----- Original Message ---- > From: Bernhard Schmidt > To: PseudoCylon ; Alexander Zagrebin > > Cc: freebsd-net@freebsd.org > Sent: Tue, February 8, 2011 2:52:53 AM > Subject: Re: if_run in hostap mode: issue with stations in the power save mode > > > > > The patch is attached. (diff to HEAD) Bit long, just because there is > > a couple of new call back functions to avoid LOR. > > Thank you! > > I've combined both patches (see attachment), if I get an ACK from both > of you I'll try get this into the tree ASAP. > > -- > Bernhard > No objection from me. Thanks AK From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 05:49:15 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E3921065694 for ; Wed, 9 Feb 2011 05:49:15 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from fallback5.mail.ru (fallback5.mail.ru [94.100.176.59]) by mx1.freebsd.org (Postfix) with ESMTP id D50628FC1B for ; Wed, 9 Feb 2011 05:49:14 +0000 (UTC) Received: from smtp2.mail.ru (smtp2.mail.ru [94.100.176.130]) by fallback5.mail.ru (mPOP.Fallback_MX) with ESMTP id 60E1A262554B for ; Wed, 9 Feb 2011 08:30:03 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail; h=Content-Transfer-Encoding:Content-Type:Subject:To:MIME-Version:Reply-To:From:Date:Message-ID; bh=6wiyVsDKdVkanID5HPURTd2EW5w5kNkVB8Sde7Qyo58=; b=ll5w7koifyUn/mHRIrqM33rivpMebVbEeBdcFv/06yEtxg9ZV70GoOuSzDSNTP60K82x/Y/bg1MTwNPvDpqktH9LT7b6q8CfaN8pqzCy2YuN2ZaAYTT5V8NaLwHaAfx6; Received: from [217.25.27.27] (port=64028 helo=[217.25.27.27]) by smtp2.mail.ru with asmtp id 1Pn2cr-0002nn-00 for freebsd-net@freebsd.org; Wed, 09 Feb 2011 08:30:01 +0300 Message-ID: <4D522657.10500@mail.ru> Date: Wed, 09 Feb 2011 09:29:59 +0400 From: rihad User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mras: Ok X-MR-Warn: 1 Subject: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rihad List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 05:49:15 -0000 Hi, we're a medium sized ISP that need to pass all incoming user traffic through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up until yesterday it had two 1 gb em cards, one for input, one for output. As we were approaching the bandwidth limitation we switched the cards for a two-port Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD 7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow, and at only about 300-400 mbps load (~30-50 IP kpps) the internet access was very slow. Also, there were many "IP fragmentation failed" errors (1-30 kpps in "systat -ip"). So I decided to source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). Late in the night yesterday I didn't have enough opportunity to test the newer FreeBSD under load, but from the time we did and I know, the same slowness started happening at about 300-400 mbps load. There are no more fragmentation failed errors. No evident drops as per "netstat -s | fgrep drop". Only the speed is slooow. Even the ssh console lags a bit. Both ix0 and ix1 are configured at their default settings. Then I read something about the number of ixgbe device descriptors (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up to 4096 permittable. But after some grepping on the source tree I saw that contrary to what the old docs say they are both set to an optimal value: /sys/dev/ixgbe/ixgbe.c: /* ** Number of TX descriptors per ring, ** setting higher than RX as this seems ** the better performing choice. */ static int ixgbe_txd = PERFORM_TXD; TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd); /* Number of RX descriptors per ring */ static int ixgbe_rxd = PERFORM_RXD; TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd) /sys/dev/ixgbe/ixgbe.h: /* * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the * number of transmit descriptors allocated by the driver. Increasing this * value allows the driver to queue more transmits. Each descriptor is 16 * bytes. Performance tests have show the 2K value to be optimal for top * performance. */ #define DEFAULT_TXD 1024 #define PERFORM_TXD 2048 #define MAX_TXD 4096 #define MIN_TXD 64 So, here's my kernel config for your viewing pleasure: include GENERIC ident SHAPER nomakeoptions DEBUG nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4 nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5 nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 nooptions COMPAT_FREEBSD32 # Compatible with i386 binaries nooptions INET6 # IPv6 communications protocols options ZERO_COPY_SOCKETS # XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts to work #options DEVICE_POLLING options IPFIREWALL #firewall options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default Here's /etc/sysctl.conf: net.inet.ip.fw.verbose=0 kern.ipc.shmall=65536 kern.ipc.shmmax=268435456 kern.ipc.semmap=1024 kern.ipc.nmbclusters=111111 net.inet.ip.fastforwarding=1 net.inet.ip.dummynet.io_fast=1 #XXX no longer used in 8.3?? net.isr.direct=0 net.inet.ip.intr_queue_maxlen=5000 hw.intr_storm_threshold=9000 #dev.em.0.rx_processing_limit=-1 # device not used any more Any tips? I'll be happy to try and add some more info upon request. Thanks. From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 07:11:19 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D57FB1065696 for ; Wed, 9 Feb 2011 07:11:19 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 644738FC19 for ; Wed, 9 Feb 2011 07:11:19 +0000 (UTC) Received: by eyf6 with SMTP id 6so3413711eyf.13 for ; Tue, 08 Feb 2011 23:11:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=vijqIIsYgSgjCGTBDhh3EVC+1szZYA8GfcV15ilhccM=; b=Rd/AtOsizinNZa+jr8KeBXWCPw2U23kU9V3B7pH+M1vn+KV2L4eeaSU+3GlhfYuBcW jD3sxDR+09nCd0qYIEt/ekdQWMw70EsmHyDOrd3uuPoP/xoMyfJDBpvGMkdG1DxvnrHq JCVWW+23n+ib4AYbQwBR2L5l5KuxTj4jZW2jU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=USvaWdj0z4rCHZp/X9NmmGBiG+xlHU+0AWm8kRlEhDTL0chJwSb84AbVjpDLpDsW5n ybR9JpB3AfUdrGF7WQX3BCLwJCGn184rPI8ACqEIlTwAoZ8b5HeyMNvyAAbJWXHiSjXR lNMSqDaZaoJm3pVtV7spMy2ANAglncrydWK/E= Received: by 10.213.3.20 with SMTP id 20mr1343062ebl.5.1297233788537; Tue, 08 Feb 2011 22:43:08 -0800 (PST) Received: from [10.0.0.49] (93-152-151-19.ddns.onlinedirect.bg [93.152.151.19]) by mx.google.com with ESMTPS id u1sm223066eeh.16.2011.02.08.22.43.06 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 08 Feb 2011 22:43:07 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: <4D522657.10500@mail.ru> Date: Wed, 9 Feb 2011 08:43:04 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4D522657.10500@mail.ru> To: rihad X-Mailer: Apple Mail (2.1082) Cc: freebsd-net@freebsd.org Subject: Re: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 07:11:19 -0000 On 9 Feb, 2011, at 07:29 , rihad wrote: > Hi, we're a medium sized ISP that need to pass all incoming user = traffic through a Intel Server Systems FreeBSD PC and its dummynet = pipes. Up until yesterday it had two 1 gb em cards, one for input, one = for output. As we were approaching the bandwidth limitation we switched = the cards for a two-port Intel 10GbE CX4 PCI-E adapter. With the then = used FreeBSD 7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it = was very slow, and at only about 300-400 mbps load (~30-50 IP kpps) the = internet access was very slow. Also, there were many "IP fragmentation = failed" errors (1-30 kpps in "systat -ip"). So I decided to = source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). Late in the night = yesterday I didn't have enough opportunity to test the newer FreeBSD = under load, but from the time we did and I know, the same slowness = started happening at about 300-400 mbps load. There are no more = fragmentation failed errors. No evident drops as per "netstat -s | fgrep = drop". Only the speed is slooow. Even the ssh console lags a bit. Both = ix0 and ix1 are configured at their default settings. >=20 > Then I read something about the number of ixgbe device descriptors = (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up = to 4096 permittable. But after some grepping on the source tree I saw = that contrary to what the old docs say they are both set to an optimal = value: >=20 > /sys/dev/ixgbe/ixgbe.c: > /* > ** Number of TX descriptors per ring, > ** setting higher than RX as this seems > ** the better performing choice. > */ > static int ixgbe_txd =3D PERFORM_TXD; > TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd); >=20 > /* Number of RX descriptors per ring */ > static int ixgbe_rxd =3D PERFORM_RXD; > TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd) >=20 >=20 > /sys/dev/ixgbe/ixgbe.h: > /* > * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is = the > * number of transmit descriptors allocated by the driver. Increasing = this > * value allows the driver to queue more transmits. Each descriptor is = 16 > * bytes. Performance tests have show the 2K value to be optimal for = top > * performance. > */ > #define DEFAULT_TXD 1024 > #define PERFORM_TXD 2048 > #define MAX_TXD 4096 > #define MIN_TXD 64 >=20 >=20 >=20 > So, here's my kernel config for your viewing pleasure: > include GENERIC >=20 > ident SHAPER >=20 > nomakeoptions DEBUG >=20 > nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4 > nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5 > nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6 > options COMPAT_FREEBSD7 # Compatible with FreeBSD7 > nooptions COMPAT_FREEBSD32 # Compatible with i386 = binaries >=20 > nooptions INET6 # IPv6 communications = protocols > options ZERO_COPY_SOCKETS > # XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts = to work > #options DEVICE_POLLING > options IPFIREWALL #firewall > options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by = default >=20 >=20 > Here's /etc/sysctl.conf: >=20 > net.inet.ip.fw.verbose=3D0 >=20 > kern.ipc.shmall=3D65536 > kern.ipc.shmmax=3D268435456 > kern.ipc.semmap=3D1024 > kern.ipc.nmbclusters=3D111111 >=20 > net.inet.ip.fastforwarding=3D1 > net.inet.ip.dummynet.io_fast=3D1 #XXX no longer used in 8.3?? > net.isr.direct=3D0 > net.inet.ip.intr_queue_maxlen=3D5000 >=20 > hw.intr_storm_threshold=3D9000 > #dev.em.0.rx_processing_limit=3D-1 # device not used any more >=20 >=20 >=20 >=20 > Any tips? I'll be happy to try and add some more info upon request. >=20 >=20 > Thanks. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" I don't know if it's the same issue, but I had severe performance issues=20= with ixgbe cards until I disable LRO (ifconfig ix0 -lro). That was on = 7.2 too. Regards, Nikolay= From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 07:32:27 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F984106564A; Wed, 9 Feb 2011 07:32:27 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id DA66B8FC0C; Wed, 9 Feb 2011 07:32:26 +0000 (UTC) Received: by gyf3 with SMTP id 3so2756878gyf.13 for ; Tue, 08 Feb 2011 23:32:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=EXf4gLDmSS8e0pHwwLjDOE6oDUDiAbxuttVqInZTNvk=; b=dgeUbUAdxkgrlODcYfWVG6NnGfZ+lqZeNqDw1SvfF3HycLZon6PdUUdaaJUFDwlanY 4H9U5sKnf685tvsbN1xsniUuN03tTlKrf8FswlQyQcZ4hjQYN5nTdTJAHHLp0bfQ6Gg3 VL1MEvl2tE7yK5/NRnniCA47+x4GXM2BskodE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=dDBazXiZAaQv05K7laDMV/eUajr+VPQbfcsmbnmZDn8NKBFy3Ee/EcnMtaffelytB3 +803/YP3g6atWoUsI0xNATPnh1u4jTHtWIYjom4h2CbBC0LDFYVzNFdlSx5dd70V5j9s 8RtNbGA+fI0l9SAYm6ik1xMVRn4h9cKAMlEEk= MIME-Version: 1.0 Received: by 10.236.103.41 with SMTP id e29mr3876311yhg.11.1297236745789; Tue, 08 Feb 2011 23:32:25 -0800 (PST) Received: by 10.147.171.17 with HTTP; Tue, 8 Feb 2011 23:32:25 -0800 (PST) In-Reply-To: References: Date: Tue, 8 Feb 2011 23:32:25 -0800 Message-ID: From: Jack Vogel To: Karim Fodil-Lemelin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Pyun YongHyeon , beezarliu , Michael Tuexen , freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 07:32:27 -0000 Hmmm, well so much for that theory :) Jack On Tue, Feb 8, 2011 at 4:06 PM, Karim Fodil-Lemelin < fodillemlinkarim@gmail.com> wrote: > > > 2011/2/8 Jack Vogel > > >> I have been following this, and thinking about it. I still am working from >> a theoretical >> standpoint, but based on a patch I got quite a long time back and never >> quite groked, >> I believe now that I might have a solution. >> >> The original PR and patch was kern/150516 from Beezar Liu, I was never >> quite comfortable >> with the code changes, nor convinced that it was a real issue and not a >> misunderstanding. >> However I think now that this very report might be behind what we are >> seeing today. I have >> a slightly different approach to solving it, of course it remains to be >> seen if it handles it >> properly. >> >> Please try the patch I've attached, I'm open to further correction or >> polishing of the >> changes. And thanks to Beezar for his original report and changes, this is >> not for em, >> but if this eliminates the problem its clearly needed in all drivers. >> >> Jack >> >> >> Hi Jack, > > Thanks for your help. I tried your patch and it didn't work so I added a > couple of printf to see if the added code was getting hit: > > --- a/freebsd/sys/dev/e1000/if_igb.c > --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c > @@ -612,7 +612,7 @@ igb_attach(device_t dev) > device_get_nameunit(dev)); > > INIT_DEBUGOUT("igb_attach: end"); > - > + printf("this driver has a patch from Jack Vogel\n"); > return (0); > > err_late: > @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, int > *done) > struct mbuf *sendmp, *mh, *mp; > struct igb_rx_buf *rxbuf; > u16 hlen, plen, hdr, vtag; > + int commit; > bool eop = FALSE; > > cur = &rxr->rx_base[i]; > @@ -4255,10 +4256,23 @@ next_desc: > bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, > BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); > > + commit = i; /* capture the old index */ > + > /* Advance our pointers to the next descriptor. */ > if (++i == adapter->num_rx_desc) > i = 0; > /* > + ** Sanity test for ring full, if this > + ** happens we need to refresh immediately > + ** or refresh may deadlock. > + */ > + if (i == rxr->next_to_refresh) { > + igb_refresh_mbufs(rxr, commit); > + printf("igb_refresh_mbufs called with commit %d\n", > commit); > + processed = 0; > + } > + > + /* > ** Send to the stack or LRO > */ > if (sendmp != NULL) { > > Here is the results: > > # dmesg | grep Vogel > this driver has a patch from Jack Vogel > this driver has a patch from Jack Vogel > > # netstat -m > 60453/52707/113160 mbufs in use (current/cache/total) > 48416/51584/100000/100000 mbuf clusters in use (current/cache/total/max) > 2894/690 mbuf+clusters out of packet secondary zone in use (current/cache) > 11946/854/12800/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 164834K/119760K/284595K bytes allocated to network (current/cache/total) > 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/4/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > # dmesg | grep commit > > At this point RX has hung. > > Somehow the check (i == rxr->next_to_refresh) is never true in this case. > Also, I did read kern/150516 and couldn't wrap my head around the patch for > the em driver that Beezar Liu suggested. > > Regards, > > Karim. > > From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 08:02:09 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92CB7106564A for ; Wed, 9 Feb 2011 08:02:09 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from smtp8.mail.ru (smtp8.mail.ru [94.100.176.53]) by mx1.freebsd.org (Postfix) with ESMTP id 49D448FC1C for ; Wed, 9 Feb 2011 08:02:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=Q6tVTNeLijx+7E8HnAQICIr3ogj4jAvrinZRM6ZaX/4=; b=UBuCz88ZW/xN9gk/j5lvJMGir7AA4xlxgeOkTSz9WRDZxtMFXVq216PVntg3qDnPQWxBhehT8wPmyeVJN8/L245nQtbEbs+YAhhl54Up3Lpxma+WWfGwAL27YqfRgOgj; Received: from [217.25.27.27] (port=39954 helo=[217.25.27.27]) by smtp8.mail.ru with asmtp id 1Pn503-0006DC-00; Wed, 09 Feb 2011 11:02:07 +0300 Message-ID: <4D5249FD.4040103@mail.ru> Date: Wed, 09 Feb 2011 12:02:05 +0400 From: rihad User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: Nikolay Denev References: <4D522657.10500@mail.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mras: Ok Cc: freebsd-net@freebsd.org Subject: Re: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 08:02:09 -0000 On 02/09/2011 10:43 AM, Nikolay Denev wrote: > I don't know if it's the same issue, but I had severe performance issues > with ixgbe cards until I disable LRO (ifconfig ix0 -lro). That was on 7.2 too. > So did I, with LR-based cards. But CX4 cards don't mention LRO in their ifconfig enabled flags. Besides, ixgbe.c doesn't enable it by default: /* Don't enable LRO by default */ ifp->if_capabilities |= IFCAP_LRO; From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 09:37:06 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A357106566C for ; Wed, 9 Feb 2011 09:37:06 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from smtp3.mail.ru (smtp3.mail.ru [94.100.176.131]) by mx1.freebsd.org (Postfix) with ESMTP id B55BC8FC1D for ; Wed, 9 Feb 2011 09:37:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:To:MIME-Version:From:Date:Message-ID; bh=GtkNKIkDBROJ/JN2Jrlsa6DDnuWmeRmO5dPlcQVYfnE=; b=1yMlwnhiQDI9V0YbrVEIegvFg2wUSWaSLkIOMl038z1B3rFbYu29vPQvfz4gCMVikLOFH+WGt0e/Z1gq9S89pYziLH40wPucA9cZwfwKntswOBx6Pr6wD+3IiBuR2o9B; Received: from [217.25.27.27] (port=48092 helo=[217.25.27.27]) by smtp3.mail.ru with asmtp id 1Pn6Tv-000135-00 for freebsd-net@freebsd.org; Wed, 09 Feb 2011 12:37:04 +0300 Message-ID: <4D52603D.4070604@mail.ru> Date: Wed, 09 Feb 2011 13:37:01 +0400 From: rihad User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <4D522657.10500@mail.ru> In-Reply-To: <4D522657.10500@mail.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mras: Ok X-MR-Warn: 1 Subject: Re: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 09:37:06 -0000 Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above with ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 was because dummynet wasn't being loaded at all... so no traffic could pass on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it "kldload dummynet" in order to do without a reboot. Works like a charm so far. Thanks to all! From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 10:36:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C2A26106566C for ; Wed, 9 Feb 2011 10:36:21 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from mail-n.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) by mx1.freebsd.org (Postfix) with ESMTP id CF65D8FC1D for ; Wed, 9 Feb 2011 10:36:20 +0000 (UTC) Received: from [212.201.127.66] (unknown [212.201.127.66]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTP id D54C31C0C0BCE; Wed, 9 Feb 2011 11:36:18 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Michael Tuexen In-Reply-To: Date: Wed, 9 Feb 2011 11:36:18 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Jack Vogel X-Mailer: Apple Mail (2.1082) Cc: Karim Fodil-Lemelin , Pyun YongHyeon , freebsd-net@freebsd.org, beezarliu Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 10:36:21 -0000 Hi Jack, I could recreate the problem. When the problem occurs, we see rx_nxt_check =3D n rx_nxt_refresh =3D n + 1 (This was also reported in a mail from Karim) This means that the *whole* receive ring has no buffers anymore. This = can occur if, for some amount of time, no clusters are available. Now outside of the driver, at some point of time, clusters are freed. I don't think that igb_refresh_mbufs() gets called, since it only gets called from igb_rxeof(), which gets called when a packet has been = received, which can not happen since the receive ring is empty. So how can the = driver know? I have no idea. Maybe we can periodically check for such an event and call igb_refresh_mbufs(). Does this make sense to you? Best regards Michael On Feb 9, 2011, at 8:32 AM, Jack Vogel wrote: > Hmmm, well so much for that theory :) >=20 > Jack >=20 >=20 > On Tue, Feb 8, 2011 at 4:06 PM, Karim Fodil-Lemelin = wrote: >=20 >=20 > 2011/2/8 Jack Vogel >=20 >=20 > I have been following this, and thinking about it. I still am working = from a theoretical > standpoint, but based on a patch I got quite a long time back and = never quite groked, > I believe now that I might have a solution. >=20 > The original PR and patch was kern/150516 from Beezar Liu, I was = never quite comfortable > with the code changes, nor convinced that it was a real issue and not = a misunderstanding. > However I think now that this very report might be behind what we are = seeing today. I have > a slightly different approach to solving it, of course it remains to = be seen if it handles it=20 > properly.=20 >=20 > Please try the patch I've attached, I'm open to further correction or = polishing of the=20 > changes. And thanks to Beezar for his original report and changes, = this is not for em, > but if this eliminates the problem its clearly needed in all drivers.=20= >=20 > Jack >=20 >=20 > Hi Jack, >=20 > Thanks for your help. I tried your patch and it didn't work so I added = a couple of printf to see if the added code was getting hit: >=20 > --- a/freebsd/sys/dev/e1000/if_igb.c > --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c > @@ -612,7 +612,7 @@ igb_attach(device_t dev) > device_get_nameunit(dev)); > =20 > INIT_DEBUGOUT("igb_attach: end"); > - > + printf("this driver has a patch from Jack Vogel\n"); > return (0); > =20 > err_late: > @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, int = *done) > struct mbuf *sendmp, *mh, *mp; > struct igb_rx_buf *rxbuf; > u16 hlen, plen, hdr, vtag; > + int commit; > bool eop =3D FALSE; > =20 > cur =3D &rxr->rx_base[i]; > @@ -4255,10 +4256,23 @@ next_desc: > bus_dmamap_sync(rxr->rxdma.dma_tag, = rxr->rxdma.dma_map, > BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); > =20 > + commit =3D i; /* capture the old index */ > + > /* Advance our pointers to the next descriptor. */ > if (++i =3D=3D adapter->num_rx_desc) > i =3D 0; > /* > + ** Sanity test for ring full, if this > + ** happens we need to refresh immediately > + ** or refresh may deadlock. > + */ > + if (i =3D=3D rxr->next_to_refresh) { > + igb_refresh_mbufs(rxr, commit); > + printf("igb_refresh_mbufs called with commit = %d\n", commit); > + processed =3D 0; > + } > + > + /* > ** Send to the stack or LRO > */ > if (sendmp !=3D NULL) { >=20 > Here is the results: >=20 > # dmesg | grep Vogel > this driver has a patch from Jack Vogel > this driver has a patch from Jack Vogel >=20 > # netstat -m > 60453/52707/113160 mbufs in use (current/cache/total) > 48416/51584/100000/100000 mbuf clusters in use = (current/cache/total/max) > 2894/690 mbuf+clusters out of packet secondary zone in use = (current/cache) > 11946/854/12800/12800 4k (page size) jumbo clusters in use = (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 164834K/119760K/284595K bytes allocated to network = (current/cache/total) > 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/4/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > # dmesg | grep commit >=20 > At this point RX has hung. >=20 > Somehow the check (i =3D=3D rxr->next_to_refresh) is never true in = this case. Also, I did read kern/150516 and couldn't wrap my head around = the patch for the em driver that Beezar Liu suggested. >=20 > Regards, >=20 > Karim. >=20 >=20 From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 13:47:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3ECFC106566C for ; Wed, 9 Feb 2011 13:47:10 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id F0F8C8FC15 for ; Wed, 9 Feb 2011 13:47:09 +0000 (UTC) Received: by qyk36 with SMTP id 36so111148qyk.13 for ; Wed, 09 Feb 2011 05:47:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=5qk1fQl+wmMSfTXTIIjFzOsQtH5KCtEnl2oRGkELMrw=; b=H5LlNbFVHA0aAFdlYRXxFTYG/mQM5Ji+DmY14iZDA/VwGoZa32TmwAoMpseDdQdReG r26gOH71L0rmQati1yctDVn0ZwuO9qpNNAl6fTWkY27N+sY/MpT1TOQ6ft//Saf+Sr7F ve/Viuilu970sflwzIEiuZZZ0Sfu1j970bUn0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=kgqPV4/DdFwh48e3s8XJ/0aIjYZfiNjCXjD1fbRqmOuODxHYRwKGyK366zwyYuRY+i 94l8o6GXPbCVu1Nc3z9ceuk+Cn6p2UXhN4Jf426OCNaXl9XkgfkL4TGgHFaMmDdmrKRU tzt7byBiykipdD0bXJIAwTXEKcPU9crkangro= MIME-Version: 1.0 Received: by 10.229.248.198 with SMTP id mh6mr15146657qcb.5.1297259229058; Wed, 09 Feb 2011 05:47:09 -0800 (PST) Received: by 10.229.102.87 with HTTP; Wed, 9 Feb 2011 05:47:09 -0800 (PST) In-Reply-To: <4D52603D.4070604@mail.ru> References: <4D522657.10500@mail.ru> <4D52603D.4070604@mail.ru> Date: Wed, 9 Feb 2011 16:47:09 +0300 Message-ID: From: Sergey Kandaurov To: rihad Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org Subject: Re: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 13:47:10 -0000 On 9 February 2011 12:37, rihad wrote: > Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above with > ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 > was because dummynet wasn't being loaded at all... so no traffic could pass > on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I > turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it "kldload > dummynet" in order to do without a reboot. Works like a charm so far. Thanks > to all! Looks like loading dummynet.ko via /boot/loader.conf doesn't work because dummynet.ko depends on dummynet.ko but of the different version. There are even more strange things: 1) dummynet.ko declares itself as version 1: /sys/netinet/ipfw/ip_dummynet.c: MODULE_VERSION(dummynet, 1); 2) dummynet.ko compiles into itself the various schedulers: fifo, prio, rr, etc; 3) these schedulers presumably think they are compiled standalone, so they are explicitly and strongly depend on dummynet of version 3 (why?): /sys/netinet/ipfw/dn_sched.h: MODULE_DEPEND(name, dummynet, 3, 3, 3); * That makes loader to error like "dummynet: loading required module 'dummynet'". and, if loading dummynet.ko in loader prompt manually, then "module 'dummynet' exists but with wrong version"] This shall fix the problem: rebuilding only dummynet should be enough. %%% Index: /sys/netinet/ipfw/ip_dummynet.c =================================================================== --- /sys/netinet/ipfw/ip_dummynet.c (revision 218026) +++ /sys/netinet/ipfw/ip_dummynet.c (working copy) @@ -2294,7 +2294,7 @@ #define DN_MODEV_ORD (SI_ORDER_ANY - 128) /* after ipfw */ DECLARE_MODULE(dummynet, dummynet_mod, DN_SI_SUB, DN_MODEV_ORD); MODULE_DEPEND(dummynet, ipfw, 2, 2, 2); -MODULE_VERSION(dummynet, 1); +MODULE_VERSION(dummynet, 3); /* * Starting up. Done in order after dummynet_modevent() has been called. %%% -- wbr, pluknet From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 15:16:00 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDCC2106566B for ; Wed, 9 Feb 2011 15:16:00 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from smtp1.mail.ru (smtp1.mail.ru [94.100.176.129]) by mx1.freebsd.org (Postfix) with ESMTP id 52D378FC08 for ; Wed, 9 Feb 2011 15:16:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=Ejv0ZkVwadIKOFDcK8+/RDZ7Ydm8OcIZyUbzfOkNoks=; b=eQapWyeqBUVV+0M19LQoO1iWooTV1QWgEiyfwFSG49cMCiVzsffQ1owZwuN3Vu1zqNKzp17U8ZVNWfah7pal0/fmQI2tDiEb+VvkLxplH0w/+46jJUZQhk+/PZ69L706; Received: from [217.25.27.27] (port=39381 helo=[217.25.27.27]) by smtp1.mail.ru with asmtp id 1PnBlu-0007BI-00; Wed, 09 Feb 2011 18:15:58 +0300 Message-ID: <4D52AFAB.2030103@mail.ru> Date: Wed, 09 Feb 2011 19:15:55 +0400 From: rihad User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: Sergey Kandaurov References: <4D522657.10500@mail.ru> <4D52603D.4070604@mail.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mras: Ok X-MR-Warn: 1 Cc: freebsd-net@freebsd.org Subject: Re: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 15:16:00 -0000 On 02/09/2011 05:47 PM, Sergey Kandaurov wrote: > On 9 February 2011 12:37, rihad wrote: >> Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above with >> ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 >> was because dummynet wasn't being loaded at all... so no traffic could pass >> on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I >> turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it "kldload >> dummynet" in order to do without a reboot. Works like a charm so far. Thanks >> to all! > > Looks like loading dummynet.ko via /boot/loader.conf doesn't work because > dummynet.ko depends on dummynet.ko but of the different version. > Would dummynet_enable="YES" in rc.conf still work? We haven't yet had a chance to reboot to test that. From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 15:27:15 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 52BB1106566C for ; Wed, 9 Feb 2011 15:27:15 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0A2718FC08 for ; Wed, 9 Feb 2011 15:27:14 +0000 (UTC) Received: by qwj9 with SMTP id 9so200269qwj.13 for ; Wed, 09 Feb 2011 07:27:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=eZ52vtD/5zXY3UNuVZ+NGI0EpXhXL/osX+iAxjyMNig=; b=mBxi8+HhIKQOWJOswOBnCTuPguPkw6yoHccJepGsel+hrgWf3JsjVuoq0ctaCTNi1t VUOlSNgkAU7wmZ58vnCb9DFjj9VX88Ypn3rA1youWNGQGJgS0tQPmPLoNLb1zuKV8wLt /ZtltkLbGtiNUsRcJFjQGfMFAOJFwgdZ9qxA0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=sEKyNXRoD4cJ01RFbw3FNi0TxjhUYoHU7wEjxuZ7I/dSWNrJGWqQ0hov6BWYjt9V7j yGjsJfsfvST764Blp8GlbifCfUQCHMoX1o/Jk6u02kmkNpT63VFtWghZmmxOFle+Hdev bA7FHVxtYOmn4uVc4mfMPLPgYWy4CqTWr9uKQ= MIME-Version: 1.0 Received: by 10.229.248.198 with SMTP id mh6mr15249884qcb.5.1297265233885; Wed, 09 Feb 2011 07:27:13 -0800 (PST) Received: by 10.229.102.87 with HTTP; Wed, 9 Feb 2011 07:27:13 -0800 (PST) In-Reply-To: <4D52AFAB.2030103@mail.ru> References: <4D522657.10500@mail.ru> <4D52603D.4070604@mail.ru> <4D52AFAB.2030103@mail.ru> Date: Wed, 9 Feb 2011 18:27:13 +0300 Message-ID: From: Sergey Kandaurov To: rihad Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 15:27:15 -0000 On 9 February 2011 18:15, rihad wrote: > On 02/09/2011 05:47 PM, Sergey Kandaurov wrote: >> >> On 9 February 2011 12:37, rihad =A0wrote: >>> >>> Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above >>> with >>> ixgbe (tons of "fragmentation failed" errors) was real. The issue in >>> 8.3-RC3 >>> was because dummynet wasn't being loaded at all... so no traffic could >>> pass >>> on it, despite dummynet_load=3D"YES" being set in /boot/loader.conf. So= I >>> turned it on in /etc/rc.conf : dummynet_enable=3D"YES" and loaded it >>> "kldload >>> dummynet" in order to do without a reboot. Works like a charm so far. >>> Thanks >>> to all! >> >> Looks like loading dummynet.ko via /boot/loader.conf doesn't work becaus= e >> dummynet.ko depends on dummynet.ko but of the different version. >> > Would dummynet_enable=3D"YES" in rc.conf still work? We haven't yet had a > chance to reboot to test that. > Yes, it would. Note that it depends on firewall_enable=3D"YES" also present in rc.conf. --=20 wbr, pluknet From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 15:28:26 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E6AB106566B for ; Wed, 9 Feb 2011 15:28:26 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from sunner.semmy.ru (sunner.semmy.ru [195.54.209.159]) by mx1.freebsd.org (Postfix) with ESMTP id 25D6A8FC17 for ; Wed, 9 Feb 2011 15:28:25 +0000 (UTC) Received: from dhcp170-205-red.yandex.net ([95.108.170.205]) by sunner.semmy.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.74 (FreeBSD)) (envelope-from ) id 1PnBSs-000Akv-D8 for freebsd-net@freebsd.org; Wed, 09 Feb 2011 17:56:18 +0300 Message-ID: <4D52AB12.6050506@FreeBSD.org> Date: Wed, 09 Feb 2011 17:56:18 +0300 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: route messages from NDP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 15:28:26 -0000 Hello. In my routing table I see entries after Neighbor Discovery Protocol processed: ... 2a02:6b8:0:401:51:4809:8158:1dcd 00:22:fb:3d:82:fe UHLW vlan438 ... I'd like to catch them via a routing socket when they appear. First, try to add a static entry: ndp -s 2a02:6b8:0:403::1:1 00:0e:0c:09:2e:7b and look at route -n monitor output: got message of size 240 on Wed Feb 9 17:26:50 2011 RTM_ADD: Add Route: len 240, pid: 82741, seq 2, errno 0, flags: locks: inits: sockaddrs: 2a02:6b8:0:403::1:1 0.e.c.9.2e.7b We have two sections here - DST and GATEWAY. DST is a IPv6 address (sa_family == AF_INET6) and GATEWAY is a MAC (sa_family == AF_LINK). Just for info sockaddr_dl looks like this: $1 = {sdl_len = 54 '6', sdl_family = 18 '\022', sdl_index = 24, sdl_type = 135 '\207', sdl_nlen = 0 '\0', sdl_alen = 6 '\006', sdl_slen = 0 '\0', sdl_data = "\000\016\f\t.{", '\0' } Looks good. Lets wait for NDP entry... Here is it: got message of size 328 on Wed Feb 9 17:27:11 2011 RTM_ADD: Add Route: len 328, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: 2a02:6b8:0:40c:daa2:5eff:fe8c:139 vlan438:0.30.48.33.4.92 fe80::230:48ff:fe33:492%vlan438 We have four section here DST, GATEWAY, IFP, IFA. DST is IPv6 address, IFP and IFA I don't care and GATEWAY section is empty. Let's see why: $1 = {sdl_len = 54 '6', sdl_family = 18 '\022', sdl_index = 8, sdl_type = 135 '\207', sdl_nlen = 0 '\0', sdl_alen = 0 '\0', sdl_slen = 0 '\0', sdl_data = '\0' } family is AF_LINK (18), it's a correct one. But sdl_alen, sdl_data are zeros. I see this for all routing messages from NDP. All created routing table entries are good (no problems here). Why sockaddr_dl in GATEWAY section has a zero address? Is it a bug? -- Sem. From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 17:35:27 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64178106566B; Wed, 9 Feb 2011 17:35:27 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 00D4F8FC17; Wed, 9 Feb 2011 17:35:26 +0000 (UTC) Received: by gxk8 with SMTP id 8so189450gxk.13 for ; Wed, 09 Feb 2011 09:35:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=C6XW/8R565x7lUnaU7UA8U7JLeAnFsMJQUBRVisDJDA=; b=JhvJ74p9WpkdhRjeO+DmhNM3alPG4cXXOGmiE9u5XEJRgZ67PV7z1uPs+tmFZa6mrp 8aKMZkcaYLeLyh7zPIGn3+kWj7/VAqG7LrdAfZx63XWTU4MndAIttefJsfYv5dXVD8sn OigcMfKeUa5nB/cmXnqKGJ5FCeldoKk2brTtQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=etUen1qUVASoEM+34H5aiiAb/eZ7mlm4OQWdvVr30bl81bRcr7VpdDFyMB1adubh+k Z8dGkGDXfNC7d5dmljeSXdffrbl8tqKYUTGSaQ5+m6TCJdbkZIVY0fx7XPL39pRlruVI XGyzSP4WXdhyWgKshjPNU8ZOa4bSlKmFGWAOc= MIME-Version: 1.0 Received: by 10.150.148.18 with SMTP id v18mr2013849ybd.54.1297272926023; Wed, 09 Feb 2011 09:35:26 -0800 (PST) Received: by 10.147.167.5 with HTTP; Wed, 9 Feb 2011 09:35:25 -0800 (PST) In-Reply-To: References: Date: Wed, 9 Feb 2011 09:35:25 -0800 Message-ID: From: Jack Vogel To: Michael Tuexen Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Karim Fodil-Lemelin , Pyun YongHyeon , freebsd-net@freebsd.org, beezarliu Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 17:35:27 -0000 OK, but the question is why does the ring get totally consumed this way, the ring has 1024 descriptors, it seems unintuitive that that whole quantity can be used without some being recharged. Do you see the system mbuf pool being depleted at the same time? Since you can reproduce it, do me a favor, in rxeof, change the processed value from 8 to 4 and then 1, effectively call refresh every descriptor, see if that eliminates the issue. Thanks for your help, Jack On Wed, Feb 9, 2011 at 2:36 AM, Michael Tuexen wrote: > Hi Jack, > > I could recreate the problem. When the problem occurs, we see > > rx_nxt_check = n > rx_nxt_refresh = n + 1 > > (This was also reported in a mail from Karim) > > This means that the *whole* receive ring has no buffers anymore. This can > occur if, for some amount of time, no clusters are available. > > Now outside of the driver, at some point of time, clusters are freed. > I don't think that igb_refresh_mbufs() gets called, since it only gets > called from igb_rxeof(), which gets called when a packet has been received, > which can not happen since the receive ring is empty. So how can the driver > know? I have no idea. Maybe we can periodically check for such an event > and call igb_refresh_mbufs(). > > Does this make sense to you? > > Best regards > Michael > > > On Feb 9, 2011, at 8:32 AM, Jack Vogel wrote: > > > Hmmm, well so much for that theory :) > > > > Jack > > > > > > On Tue, Feb 8, 2011 at 4:06 PM, Karim Fodil-Lemelin < > fodillemlinkarim@gmail.com> wrote: > > > > > > 2011/2/8 Jack Vogel > > > > > > I have been following this, and thinking about it. I still am working > from a theoretical > > standpoint, but based on a patch I got quite a long time back and never > quite groked, > > I believe now that I might have a solution. > > > > The original PR and patch was kern/150516 from Beezar Liu, I was never > quite comfortable > > with the code changes, nor convinced that it was a real issue and not a > misunderstanding. > > However I think now that this very report might be behind what we are > seeing today. I have > > a slightly different approach to solving it, of course it remains to be > seen if it handles it > > properly. > > > > Please try the patch I've attached, I'm open to further correction or > polishing of the > > changes. And thanks to Beezar for his original report and changes, this > is not for em, > > but if this eliminates the problem its clearly needed in all drivers. > > > > Jack > > > > > > Hi Jack, > > > > Thanks for your help. I tried your patch and it didn't work so I added a > couple of printf to see if the added code was getting hit: > > > > --- a/freebsd/sys/dev/e1000/if_igb.c > > --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c > > @@ -612,7 +612,7 @@ igb_attach(device_t dev) > > device_get_nameunit(dev)); > > > > INIT_DEBUGOUT("igb_attach: end"); > > - > > + printf("this driver has a patch from Jack Vogel\n"); > > return (0); > > > > err_late: > > @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, int > *done) > > struct mbuf *sendmp, *mh, *mp; > > struct igb_rx_buf *rxbuf; > > u16 hlen, plen, hdr, vtag; > > + int commit; > > bool eop = FALSE; > > > > cur = &rxr->rx_base[i]; > > @@ -4255,10 +4256,23 @@ next_desc: > > bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, > > BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); > > > > + commit = i; /* capture the old index */ > > + > > /* Advance our pointers to the next descriptor. */ > > if (++i == adapter->num_rx_desc) > > i = 0; > > /* > > + ** Sanity test for ring full, if this > > + ** happens we need to refresh immediately > > + ** or refresh may deadlock. > > + */ > > + if (i == rxr->next_to_refresh) { > > + igb_refresh_mbufs(rxr, commit); > > + printf("igb_refresh_mbufs called with commit > %d\n", commit); > > + processed = 0; > > + } > > + > > + /* > > ** Send to the stack or LRO > > */ > > if (sendmp != NULL) { > > > > Here is the results: > > > > # dmesg | grep Vogel > > this driver has a patch from Jack Vogel > > this driver has a patch from Jack Vogel > > > > # netstat -m > > 60453/52707/113160 mbufs in use (current/cache/total) > > 48416/51584/100000/100000 mbuf clusters in use (current/cache/total/max) > > 2894/690 mbuf+clusters out of packet secondary zone in use > (current/cache) > > 11946/854/12800/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > > 164834K/119760K/284595K bytes allocated to network (current/cache/total) > > 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0/4/6656 sfbufs in use (current/peak/max) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > # dmesg | grep commit > > > > At this point RX has hung. > > > > Somehow the check (i == rxr->next_to_refresh) is never true in this case. > Also, I did read kern/150516 and couldn't wrap my head around the patch for > the em driver that Beezar Liu suggested. > > > > Regards, > > > > Karim. > > > > > > From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 18:15:28 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20A7A1065670 for ; Wed, 9 Feb 2011 18:15:28 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id A92758FC1E for ; Wed, 9 Feb 2011 18:15:27 +0000 (UTC) Received: by bwz12 with SMTP id 12so1189064bwz.13 for ; Wed, 09 Feb 2011 10:15:26 -0800 (PST) Received: by 10.204.60.14 with SMTP id n14mr19686428bkh.118.1297275326050; Wed, 09 Feb 2011 10:15:26 -0800 (PST) Received: from maja.lab.techwires.net (p54B4D78C.dip.t-dialin.net [84.180.215.140]) by mx.google.com with ESMTPS id j11sm398682bka.0.2011.02.09.10.15.23 (version=SSLv3 cipher=RC4-MD5); Wed, 09 Feb 2011 10:15:24 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: PseudoCylon , Alexander Zagrebin Date: Wed, 9 Feb 2011 19:15:35 +0100 User-Agent: KMail/1.12.4 (FreeBSD/9.0-CURRENT; KDE/4.3.5; amd64; ; ) References: <20110204060808.GA97298@gw.zagrebin.ru> <95679.17379.qm@web39320.mail.mud.yahoo.com> <201102081052.54016.bschmidt@freebsd.org> In-Reply-To: <201102081052.54016.bschmidt@freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201102091915.35504.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: if_run in hostap mode: issue with stations in the power save mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 18:15:28 -0000 On Tuesday 08 February 2011 10:52:53 Bernhard Schmidt wrote: > I've combined both patches (see attachment), if I get an ACK from both > of you I'll try get this into the tree ASAP. Committed, thanks! -- Bernhard From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 18:45:57 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 110081065673 for ; Wed, 9 Feb 2011 18:45:57 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from mail-n.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) by mx1.freebsd.org (Postfix) with ESMTP id 1D6188FC0A for ; Wed, 9 Feb 2011 18:45:56 +0000 (UTC) Received: from [192.168.1.113] (p508FA862.dip.t-dialin.net [80.143.168.98]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTP id 884611C0C0BD8; Wed, 9 Feb 2011 19:45:53 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Michael Tuexen In-Reply-To: Date: Wed, 9 Feb 2011 19:45:52 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <12838373-FE96-443E-8979-AF5408705BF0@freebsd.org> References: To: Jack Vogel X-Mailer: Apple Mail (2.1082) Cc: Karim Fodil-Lemelin , Pyun YongHyeon , freebsd-net@freebsd.org, beezarliu Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 18:45:57 -0000 On Feb 9, 2011, at 6:35 PM, Jack Vogel wrote: > OK, but the question is why does the ring get totally consumed this = way, the > ring has 1024 descriptors, it seems unintuitive that that whole = quantity can be > used without some being recharged. Do you see the system mbuf pool = being > depleted at the same time? That was the test case I created: I set up a server accepting = connections but not reading anything. So the driver passes the mbufs to the = transport stack and they are not consumed. Then the problem occurs. Then I kill = the server. Now there are mbufs available again, but the driver doesn't = know. I had the impression that these were the circumstances in which the = problem showed up (mbuf allocations failing). >=20 > Since you can reproduce it, do me a favor, in rxeof, change the = processed > value from 8 to 4 and then 1, effectively call refresh every = descriptor, see if > that eliminates the issue. I will do. Need to see if I can do it remotely, since I'm not in my lab right now. Can do it tomorrow for sure. But I do not think that this solves the problem, since I did the things very slowly and you call it at least when you are leaving rxeof. Best regards Michael >=20 > Thanks for your help, >=20 > Jack >=20 >=20 > On Wed, Feb 9, 2011 at 2:36 AM, Michael Tuexen = wrote: > Hi Jack, >=20 > I could recreate the problem. When the problem occurs, we see >=20 > rx_nxt_check =3D n > rx_nxt_refresh =3D n + 1 >=20 > (This was also reported in a mail from Karim) >=20 > This means that the *whole* receive ring has no buffers anymore. This = can > occur if, for some amount of time, no clusters are available. >=20 > Now outside of the driver, at some point of time, clusters are freed. > I don't think that igb_refresh_mbufs() gets called, since it only gets > called from igb_rxeof(), which gets called when a packet has been = received, > which can not happen since the receive ring is empty. So how can the = driver > know? I have no idea. Maybe we can periodically check for such an = event > and call igb_refresh_mbufs(). >=20 > Does this make sense to you? >=20 > Best regards > Michael >=20 >=20 > On Feb 9, 2011, at 8:32 AM, Jack Vogel wrote: >=20 > > Hmmm, well so much for that theory :) > > > > Jack > > > > > > On Tue, Feb 8, 2011 at 4:06 PM, Karim Fodil-Lemelin = wrote: > > > > > > 2011/2/8 Jack Vogel > > > > > > I have been following this, and thinking about it. I still am = working from a theoretical > > standpoint, but based on a patch I got quite a long time back and = never quite groked, > > I believe now that I might have a solution. > > > > The original PR and patch was kern/150516 from Beezar Liu, I was = never quite comfortable > > with the code changes, nor convinced that it was a real issue and = not a misunderstanding. > > However I think now that this very report might be behind what we = are seeing today. I have > > a slightly different approach to solving it, of course it remains to = be seen if it handles it > > properly. > > > > Please try the patch I've attached, I'm open to further correction = or polishing of the > > changes. And thanks to Beezar for his original report and changes, = this is not for em, > > but if this eliminates the problem its clearly needed in all = drivers. > > > > Jack > > > > > > Hi Jack, > > > > Thanks for your help. I tried your patch and it didn't work so I = added a couple of printf to see if the added code was getting hit: > > > > --- a/freebsd/sys/dev/e1000/if_igb.c > > --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c > > @@ -612,7 +612,7 @@ igb_attach(device_t dev) > > device_get_nameunit(dev)); > > > > INIT_DEBUGOUT("igb_attach: end"); > > - > > + printf("this driver has a patch from Jack Vogel\n"); > > return (0); > > > > err_late: > > @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, = int *done) > > struct mbuf *sendmp, *mh, *mp; > > struct igb_rx_buf *rxbuf; > > u16 hlen, plen, hdr, vtag; > > + int commit; > > bool eop =3D FALSE; > > > > cur =3D &rxr->rx_base[i]; > > @@ -4255,10 +4256,23 @@ next_desc: > > bus_dmamap_sync(rxr->rxdma.dma_tag, = rxr->rxdma.dma_map, > > BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); > > > > + commit =3D i; /* capture the old index */ > > + > > /* Advance our pointers to the next descriptor. */ > > if (++i =3D=3D adapter->num_rx_desc) > > i =3D 0; > > /* > > + ** Sanity test for ring full, if this > > + ** happens we need to refresh immediately > > + ** or refresh may deadlock. > > + */ > > + if (i =3D=3D rxr->next_to_refresh) { > > + igb_refresh_mbufs(rxr, commit); > > + printf("igb_refresh_mbufs called with commit = %d\n", commit); > > + processed =3D 0; > > + } > > + > > + /* > > ** Send to the stack or LRO > > */ > > if (sendmp !=3D NULL) { > > > > Here is the results: > > > > # dmesg | grep Vogel > > this driver has a patch from Jack Vogel > > this driver has a patch from Jack Vogel > > > > # netstat -m > > 60453/52707/113160 mbufs in use (current/cache/total) > > 48416/51584/100000/100000 mbuf clusters in use = (current/cache/total/max) > > 2894/690 mbuf+clusters out of packet secondary zone in use = (current/cache) > > 11946/854/12800/12800 4k (page size) jumbo clusters in use = (current/cache/total/max) > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > > 164834K/119760K/284595K bytes allocated to network = (current/cache/total) > > 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > 0/4/6656 sfbufs in use (current/peak/max) > > 0 requests for sfbufs denied > > 0 requests for sfbufs delayed > > 0 requests for I/O initiated by sendfile > > 0 calls to protocol drain routines > > # dmesg | grep commit > > > > At this point RX has hung. > > > > Somehow the check (i =3D=3D rxr->next_to_refresh) is never true in = this case. Also, I did read kern/150516 and couldn't wrap my head around = the patch for the em driver that Beezar Liu suggested. > > > > Regards, > > > > Karim. > > > > >=20 >=20 From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 20:11:07 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 08278106567A for ; Wed, 9 Feb 2011 20:11:07 +0000 (UTC) (envelope-from gabor.radnai@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id B4ADC8FC1A for ; Wed, 9 Feb 2011 20:11:06 +0000 (UTC) Received: by vxa40 with SMTP id 40so275124vxa.13 for ; Wed, 09 Feb 2011 12:11:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:from:date:message-id:subject:to :content-type; bh=iciTs429lsc0Anvdwa/7qqeiZ7Hjj4dvJMBO0ksFQR4=; b=a6lrQnMdOBkBP9quaWU3lkQ+oRUH7wj6E61hRVpyONhSVKn3rIAFbMz2U11cNQo2rm mr2h6Rqp983VEYEbMMRTEaM0cM0/RU3XmAHlZUj8mE99erQZRPrcUgn6xt51Bj5dhb6d d++uLi3tHWVz2vJ6gxiWyfEVHh/Ppfswa2/EQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; b=WpcYfYMVAztPFRHxdr0kKuwB5El/x2cs9TrXDnAzhl4CJxxYtAVrMiWDpkPs8ED1GP 5qFmnc/bXUwNXwCKvlyd4tDzlD1kZi0scWRGhXCqF7TeohPFabS2ywq+IlxhrEWh+Zph hgXoNmlRXCzZcfosMIJuJUyibtnHH/ECEXVYU= Received: by 10.220.191.2 with SMTP id dk2mr5197005vcb.251.1297280439740; Wed, 09 Feb 2011 11:40:39 -0800 (PST) MIME-Version: 1.0 Received: by 10.220.117.148 with HTTP; Wed, 9 Feb 2011 11:40:24 -0800 (PST) From: Gabor Radnai Date: Wed, 9 Feb 2011 20:40:24 +0100 Message-ID: To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: Problem with re0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 20:11:07 -0000 > Both the em and re drivers have had a lot of work done recently. Are > you trying with 8.2RC1 ? Tried with 8.2RC2 (via fixit shell with em): the same symptoms sadly. Card recognized, driver loaded as a result ifconfig reports it as available interface. Though neither static IP addressing nor DHCP makes it accessible on network. Interface cannot ping even the default gateway and neither this machine can be pinged. I am very sad :-( From owner-freebsd-net@FreeBSD.ORG Wed Feb 9 21:41:27 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43D791065670 for ; Wed, 9 Feb 2011 21:41:27 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0E87A8FC2E for ; Wed, 9 Feb 2011 21:41:26 +0000 (UTC) Received: by pxi1 with SMTP id 1so122404pxi.13 for ; Wed, 09 Feb 2011 13:41:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=HLNVvetybiEEl1PFOwTczbKfP0iDGGRY/IHebus/ZtU=; b=lHpJ/YdHAUrl9kasazqkOvF+FrwtrrkJOxNliGKVrTEPj/xmPMIlNnpKbmZj/H0/KR rt8q+OTf1CjcLom2MAmd33qMiU2EPw9DL/xWceB2oK009SdE//fq4pt3k5U7v5o5pTK7 PH9qQXBQyyIroI1cwGpnacbG+4ihPGQUaUVZw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=F+zsxNiwtk/hg1GSp4ApgwNP6j8B9k/oVU/pb8NIxocpQlfDL3L1JOBGFCv3H6v2TZ ZLceuGsl4M0MpiEdzgDrLDSkuE6T4lxMoSeURuDpBMr9JaDw9g0A4DpypHT39ilvK87F kEJ3exsUvJmg19uxqsrseiaZAVvdnBKwIX+6w= Received: by 10.142.187.14 with SMTP id k14mr19138399wff.117.1297287686345; Wed, 09 Feb 2011 13:41:26 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id c3sm940902wfa.2.2011.02.09.13.41.22 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 09 Feb 2011 13:41:23 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 9 Feb 2011 13:41:30 -0800 From: Pyun YongHyeon Date: Wed, 9 Feb 2011 13:41:30 -0800 To: Peter Lai Message-ID: <20110209214130.GB10080@michelle.cdnetworks.com> References: <20110208013841.GB1306@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="X1bOJ3K7DJ5YkBrT" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: bge wedging 8.2-RC1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 21:41:27 -0000 --X1bOJ3K7DJ5YkBrT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Feb 07, 2011 at 08:27:43PM -0600, Peter Lai wrote: > On Feb 7, 2011 7:38 PM, "Pyun YongHyeon" wrote: > > > > On Mon, Feb 07, 2011 at 06:09:16PM -0600, Peter Lai wrote: > > > Hello > > > > > > I've got a new Dell Precision workstation here with a BCM5761 on intel > > > mobo for westmere xeons that is wedging with interrupt storm and will > > > lockup the system randomly. I have turned HTT and auto powermanagement > > > off in bios (system cannot sleep), lowest cpu acpi state is C1. > > > > > > Here is dmesg: > > > bge0: > > 0x5761100> mem 0xf3be0000-0xf3beffff,0xf3bf0000-0xf3bfffff irq 17 at > > > device 0.0 on pci6 > > > bge0: CHIP ID 0x05761100; ASIC REV 0x5761; CHIP REV 0x57611; PCI-E > > > miibus0: on bge0 > > > brgphy0: PHY 1 on miibus0 > > > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > > > 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > > > > > > Here is pciconf -lv: > > > bge0@pci0:6:0:0: class=0x020000 card=0x026d1028 chip=0x168114e4 > > > rev=0x10 hdr=0x00 > > > vendor = 'Broadcom Corporation' > > > device = 'Broadcom 57XX Gigabit Integrated Controller > (BCM5761)' > > > class = network > > > subclass = ethernet > > > > > > here is the setup in rc.conf: > > > > > > ifconfig_bge0="polling -tso -vlanhwtso -vlanhwtag -vlanmtu inet > > > 192.168.123.124 netmask 255.255.255.0" > > > > > > I have the card plugged into a dlink DSS8 100mbps switch with one > > > other 100mbps device on it (rich man's crossover cable). > > > > > > Before turning off TSO4 and VLAN tagging (because I don't use them), > > > the card would do several things: > > > 1. 1 out of 3 reboots: Fail to bring interface up. ifconfig would hang > > > and systat/vmstat showed 800+ interrupts per second on IRQ256 > > > > This is strange. bge(4) does not use MSI if you build bge(4) with > > DEVICE_POLLING so seeing IRQ256 interrupts looks odd to me. > > Are you sure bge(4) is using IRQ256? > > This is with GENERIC. I will rebuild with POLLING and try... > Let me know attached patch makes any difference on your box. The patch contains some other changes but that wouldn't affect your BCM5761 controller. If you see "CLKREQ enabled" message after applying the patch also let me know that too. > > > > > 2. After a few hours lock up the system, requiring hard reboot > > > > > > After disabling TSO4 and VLAN stuff: > > > bge0: flags=8802 metric 0 mtu 1500 > > > options=80083 > > > media: Ethernet autoselect (100baseTX > > > ) > > > > > > Everything seemed fine for about two weeks and then suddenly started > > > acting up again, locked up, after hard reboot, soft reboot, link will > > > not come up and I see interrupt storm again.... > > > > > > > If you don't use DEVICE_POLLING, rebuild bge(4) with > > DEVICE_POLLING. For most cases, you don't need to enable polling on > > intelligent controllers like bge(4). > > > > I also have BCM5761 PCIe controller which shows no such issues. I > > know there is an edge case(send BD corruption) for BCM5761/BCM5784/ > > BCM57780 which needs to be investigated. I'm not sure you're seeing > > that edge case though. > > > > > I am close to buying an intel card to replace the bcm, but then I > > > noticed that the main intel desktop PCI-E card is 82574L-based and > > > people are having em driver wedging on that too. So now I have broken > > > ethernet on this box; my primary link is atheros 5212 pci card and I > > > may be out of pci slots (or else I might try a pci intel card). --X1bOJ3K7DJ5YkBrT Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="bge.clkreq.diff" Index: sys/dev/bge/if_bgereg.h =================================================================== --- sys/dev/bge/if_bgereg.h (revision 218409) +++ sys/dev/bge/if_bgereg.h (working copy) @@ -2004,6 +2004,11 @@ #define BGE_EECTL_DATAOUT 0x00000010 #define BGE_EECTL_DATAIN 0x00000020 +/* PCIe Link control register */ +#define BGE_PCIE_LNKCTL 0x7D54 +#define BGE_PCIE_LNKCTL_L1_PLL_PD_ENB 0x00000008 +#define BGE_PCIE_LNKCTL_L1_PLL_PD_DIS 0x00000080 + /* MDI (MII/GMII) access register */ #define BGE_MDI_DATA 0x00000001 #define BGE_MDI_DIR 0x00000002 @@ -2769,6 +2774,7 @@ #define BGE_FLAG_4G_BNDRY_BUG 0x02000000 #define BGE_FLAG_RX_ALIGNBUG 0x04000000 #define BGE_FLAG_SHORT_DMA_BUG 0x08000000 +#define BGE_FLAG_CLKREQ_BUG 0x10000000 uint32_t bge_phy_flags; #define BGE_PHY_WIRESPEED 0x00000001 #define BGE_PHY_ADC_BUG 0x00000002 Index: sys/dev/bge/if_bge.c =================================================================== --- sys/dev/bge/if_bge.c (revision 218409) +++ sys/dev/bge/if_bge.c (working copy) @@ -879,6 +879,8 @@ { struct bge_softc *sc; struct mii_data *mii; + uint16_t lnkctl; + sc = device_get_softc(dev); mii = device_get_softc(sc->bge_miibus); @@ -905,6 +907,18 @@ sc->bge_link = 0; if (sc->bge_link == 0) return; + /* Disable CLKREQ when controller is running at 10/100Mbps. */ + if (sc->bge_flags & BGE_FLAG_CLKREQ_BUG) { + lnkctl = pci_read_config(sc->bge_dev, sc->bge_expcap + + PCIR_EXPRESS_LINK_CTL, 2); + if (IFM_SUBTYPE(mii->mii_media_active) == IFM_10_T || + IFM_SUBTYPE(mii->mii_media_active) == IFM_100_TX) + lnkctl &= ~0x0100; + else + lnkctl |= 0x0100; + pci_write_config(sc->bge_dev, sc->bge_expcap + + PCIR_EXPRESS_LINK_CTL, lnkctl, 2); + } BGE_CLRBIT(sc, BGE_MAC_MODE, BGE_MACMODE_PORTMODE); if (IFM_SUBTYPE(mii->mii_media_active) == IFM_1000_T || IFM_SUBTYPE(mii->mii_media_active) == IFM_1000_SX) @@ -1383,6 +1397,13 @@ i < BGE_STATUS_BLOCK_END + 1; i += sizeof(uint32_t)) BGE_MEMWIN_WRITE(sc, i, 0); + /* Disable PCIe PLL power down after reset. */ + if (sc->bge_asicrev == BGE_ASICREV_BCM57780) { + val = CSR_READ_4(sc, BGE_PCIE_LNKCTL); + val &= ~BGE_PCIE_LNKCTL_L1_PLL_PD_ENB; + val |= BGE_PCIE_LNKCTL_L1_PLL_PD_DIS; + CSR_WRITE_4(sc, BGE_PCIE_LNKCTL, val); + } if (sc->bge_chiprev == BGE_CHIPREV_5704_BX) { /* * Fix data corruption caused by non-qword write with WB. @@ -2911,7 +2932,28 @@ */ if (BGE_IS_5714_FAMILY(sc) && (sc->bge_flags & BGE_FLAG_PCIX)) sc->bge_flags |= BGE_FLAG_40BIT_BUG; + /* + * Controllers that are known to corrupt send BDs when CLKREQ is + * enabled. Diasbling CLKREQ at 10/100Mbps workaround it. + */ + if (sc->bge_flags & BGE_FLAG_PCIE) { + if (sc->bge_asicrev == BGE_ASICREV_BCM5761 || + sc->bge_asicrev == BGE_ASICREV_BCM5784 || + sc->bge_chipid == BGE_CHIPID_BCM57780_A0 || + sc->bge_chipid == BGE_CHIPID_BCM57780_A1) { + if (pci_read_config(sc->bge_dev, + sc->bge_expcap + PCIR_EXPRESS_LINK_CTL, 2) & + 0x0100) { +#if 1 + device_printf(dev, "CLKREQ enabled\n"); +#endif + sc->bge_flags |= BGE_FLAG_CLKREQ_BUG; + } + } + } + + /* * Allocate the interrupt, using MSI if possible. These devices * support 8 MSI messages, but only the first one is used in * normal operation. @@ -3322,6 +3364,17 @@ */ bge_writemem_ind(sc, BGE_SOFTWARE_GENCOMM, BGE_MAGIC_NUMBER); + /* + * Disable PCIe PLL power down otherwise 57780 may not respond + * to reset. + */ + if (sc->bge_asicrev == BGE_ASICREV_BCM57780) { + val = CSR_READ_4(sc, BGE_PCIE_LNKCTL); + val &= ~BGE_PCIE_LNKCTL_L1_PLL_PD_ENB; + val |= BGE_PCIE_LNKCTL_L1_PLL_PD_DIS; + CSR_WRITE_4(sc, BGE_PCIE_LNKCTL, val); + } + reset = BGE_MISCCFG_RESET_CORE_CLOCKS | BGE_32BITTIME_66MHZ; /* XXX: Broadcom Linux driver. */ @@ -3342,9 +3395,6 @@ if (BGE_IS_5705_PLUS(sc)) reset |= BGE_MISCCFG_GPHY_PD_OVERRIDE; - /* Issue global reset */ - write_op(sc, BGE_MISC_CFG, reset); - if (sc->bge_asicrev == BGE_ASICREV_BCM5906) { val = CSR_READ_4(sc, BGE_VCPU_STATUS); CSR_WRITE_4(sc, BGE_VCPU_STATUS, @@ -3354,6 +3404,9 @@ val & ~BGE_VCPU_EXT_CTRL_HALT_CPU); } + /* Issue global reset */ + write_op(sc, BGE_MISC_CFG, reset); + DELAY(1000); /* XXX: Broadcom Linux driver. */ --X1bOJ3K7DJ5YkBrT-- From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 00:28:33 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1AC7E1065674 for ; Thu, 10 Feb 2011 00:28:33 +0000 (UTC) (envelope-from cowbert@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id A00CF8FC08 for ; Thu, 10 Feb 2011 00:28:32 +0000 (UTC) Received: by ewy24 with SMTP id 24so523948ewy.13 for ; Wed, 09 Feb 2011 16:28:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vTho/TIhYBhSr2OXZlHURgdXLULv6wLmupYntp7Ym5g=; b=w6KFaeTNBmqerMD3OKthH3Fnn5HRP7MEe6IAp0nyiS7XMMVed71IzGlN81H6iXup/i otpuVM2iqyVzndYJOE3SOJ3Xcx+MJeelEA8uCcAeif7LBkGOqXDZf9RHCvCjkjSfxKlT Yh4G7qzTDvHyH/EWapbRcoktDpOfuoyslQ67A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=t2cy+J++f7zvQy3kYhHV9uLhatsOH6fG4QnZsHk662kDagi9ZUWTum8qpWQ2PdfnFU uLeDUsZRxC1cgAar6KgDT0ic1DdT/uJL5Zr0J3c/rEPbiRonXyB+2+bgMMATlxwFGWqt 5O1XMJaJnY9+knKEEwDTuWnVdANQW947bdwA8= MIME-Version: 1.0 Received: by 10.213.17.147 with SMTP id s19mr2733159eba.89.1297297711368; Wed, 09 Feb 2011 16:28:31 -0800 (PST) Received: by 10.213.31.65 with HTTP; Wed, 9 Feb 2011 16:28:31 -0800 (PST) In-Reply-To: <20110209214130.GB10080@michelle.cdnetworks.com> References: <20110208013841.GB1306@michelle.cdnetworks.com> <20110209214130.GB10080@michelle.cdnetworks.com> Date: Wed, 9 Feb 2011 18:28:31 -0600 Message-ID: From: Peter Lai To: pyunyh@gmail.com Content-Type: text/plain; charset=UTF-8 Cc: freebsd-net@freebsd.org Subject: Re: bge wedging 8.2-RC1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 00:28:33 -0000 > > Let me know attached patch makes any difference on your box. > The patch contains some other changes but that wouldn't affect your > BCM5761 controller. If you see "CLKREQ enabled" message after > applying the patch also let me know that too. > Can I apply this to 8.2-RC1 or should I update it to -RC3? From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 01:06:00 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A63E41065672 for ; Thu, 10 Feb 2011 01:06:00 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 59ABF8FC0A for ; Thu, 10 Feb 2011 01:06:00 +0000 (UTC) Received: by ywl2 with SMTP id 2so366224ywl.13 for ; Wed, 09 Feb 2011 17:05:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:date:to:cc:subject:message-id:reply-to :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=GcI7Kz05OJRNs8jwIWdjxYEiXzoa2iipzftcRwlI1bw=; b=cv2poxhLG5Qf8fEVhAtN0s69FG7P5QjMO/VRyfsa2Kt93XlI7P1snKoAXxxJI7RXU9 QvOy8mQuIoKo++yc+mtqnyuM3xZI4SlnbbbOpmK7wxg0Sqevn9yzXM/r/0ulGrdqn6Wr DbilM/VLcbxZf6GJBTUDNC3hjeupulMJvgtmM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=Z5wfu4jVSlWpKHgcjHkKQJQXCwA5asgNYaB7yhRWmzNgkNxW7bQIwfjtyr306Do9tL UMkdmumA2t3jBxNuZEMiCRGilw6+Swr0Af6xM02LnROsjTJj5FFDvo/Z0el+Q4LEbLD+ h95YRBaUiWG72n/Wn21SRFeOgdrJ3Hsyqv/Sg= Received: by 10.151.106.19 with SMTP id i19mr2626042ybm.324.1297299959620; Wed, 09 Feb 2011 17:05:59 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id v39sm1307047yba.19.2011.02.09.17.05.57 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 09 Feb 2011 17:05:58 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 9 Feb 2011 17:06:04 -0800 From: Pyun YongHyeon Date: Wed, 9 Feb 2011 17:06:04 -0800 To: Peter Lai Message-ID: <20110210010604.GC10080@michelle.cdnetworks.com> References: <20110208013841.GB1306@michelle.cdnetworks.com> <20110209214130.GB10080@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: bge wedging 8.2-RC1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 01:06:00 -0000 On Wed, Feb 09, 2011 at 06:28:31PM -0600, Peter Lai wrote: > > > > Let me know attached patch makes any difference on your box. > > The patch contains some other changes but that wouldn't affect your > > BCM5761 controller. If you see "CLKREQ enabled" message after > > applying the patch also let me know that too. > > > > Can I apply this to 8.2-RC1 or should I update it to -RC3? I guess you can apply it to 8.2-RC1 without a problem. From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 05:30:01 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A97D1065670 for ; Thu, 10 Feb 2011 05:30:01 +0000 (UTC) (envelope-from rihad@mail.ru) Received: from smtp7.mail.ru (smtp7.mail.ru [94.100.176.52]) by mx1.freebsd.org (Postfix) with ESMTP id BB6128FC0A for ; Thu, 10 Feb 2011 05:30:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=itUvHLkp35kQsxGgGuvYbmUtg6BXcz5CRF9pVJnVEI4=; b=sGvPIM1LiDEt38wbF0SiJphBvD/XBCrSYD6I5UiDV0eBPFiZRLLHyHoxyKLM4zgv/UUDXlcrCdBmfW4yk/aR0Xwxy0uOPvoTzshzfPK+SPAWpU6TMbl+1iJwb0Wzb2jt; Received: from [217.25.27.27] (port=55091 helo=[217.25.27.27]) by smtp7.mail.ru with asmtp id 1PnP6M-00016F-00; Thu, 10 Feb 2011 08:29:58 +0300 Message-ID: <4D5377D5.80305@mail.ru> Date: Thu, 10 Feb 2011 09:29:57 +0400 From: rihad User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: Sergey Kandaurov References: <4D522657.10500@mail.ru> <4D52603D.4070604@mail.ru> <4D52AFAB.2030103@mail.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mras: Ok X-MR-Warn: 1 Cc: freebsd-net@freebsd.org Subject: Re: Slow Intel 10GbE CX4 adapter behaviour X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 05:30:01 -0000 On 02/09/2011 07:27 PM, Sergey Kandaurov wrote: > On 9 February 2011 18:15, rihad wrote: >> On 02/09/2011 05:47 PM, Sergey Kandaurov wrote: >>> >>> On 9 February 2011 12:37, rihad wrote: >>>> >>>> Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above >>>> with >>>> ixgbe (tons of "fragmentation failed" errors) was real. The issue in >>>> 8.3-RC3 >>>> was because dummynet wasn't being loaded at all... so no traffic could >>>> pass >>>> on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I >>>> turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it >>>> "kldload >>>> dummynet" in order to do without a reboot. Works like a charm so far. >>>> Thanks >>>> to all! >>> >>> Looks like loading dummynet.ko via /boot/loader.conf doesn't work because >>> dummynet.ko depends on dummynet.ko but of the different version. >>> >> Would dummynet_enable="YES" in rc.conf still work? We haven't yet had a >> chance to reboot to test that. >> > > Yes, it would. > Note that it depends on firewall_enable="YES" also present in rc.conf. > Thanks, I see. Now I think that changing through rc.conf is the "official", or "supported", way of enabling dummynet upon reboot, but loader.conf is a little way under the hood. I always asked myself why it was settable in two places, and not one. But now I know. The fact that dummynet can be set to load in loader.conf is more like an undesired effect of generality. From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 05:42:34 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 360C7106566B; Thu, 10 Feb 2011 05:42:34 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0D1DD8FC16; Thu, 10 Feb 2011 05:42:34 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1A5gXhI090056; Thu, 10 Feb 2011 05:42:33 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1A5gXrN090052; Thu, 10 Feb 2011 05:42:33 GMT (envelope-from linimon) Date: Thu, 10 Feb 2011 05:42:33 GMT Message-Id: <201102100542.p1A5gXrN090052@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/154600: [tcp] [panic] Random kernel panics on tcp_output X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 05:42:34 -0000 Old Synopsis: Random kernel panics on tcp_output New Synopsis: [tcp] [panic] Random kernel panics on tcp_output Responsible-Changed-From-To: freebsd-amd64->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Thu Feb 10 05:41:32 UTC 2011 Responsible-Changed-Why: reclassify and assign. http://www.freebsd.org/cgi/query-pr.cgi?pr=154600 From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 05:44:02 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F83F1065672; Thu, 10 Feb 2011 05:44:02 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7318C8FC0C; Thu, 10 Feb 2011 05:44:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1A5i2vl090175; Thu, 10 Feb 2011 05:44:02 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1A5i25m090171; Thu, 10 Feb 2011 05:44:02 GMT (envelope-from linimon) Date: Thu, 10 Feb 2011 05:44:02 GMT Message-Id: <201102100544.p1A5i25m090171@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/154591: [msk] [panic] if_msk driver causes kernel panic (fatal trap while in kernel mode) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 05:44:02 -0000 Old Synopsis: if_msk driver causes kernel panic (fatal trap while in kernel mode) New Synopsis: [msk] [panic] if_msk driver causes kernel panic (fatal trap while in kernel mode) Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Thu Feb 10 05:43:45 UTC 2011 Responsible-Changed-Why: reassign. http://www.freebsd.org/cgi/query-pr.cgi?pr=154591 From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 10:06:42 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C144106564A for ; Thu, 10 Feb 2011 10:06:42 +0000 (UTC) (envelope-from monthadar@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 34C518FC18 for ; Thu, 10 Feb 2011 10:06:41 +0000 (UTC) Received: by wwf26 with SMTP id 26so1143099wwf.31 for ; Thu, 10 Feb 2011 02:06:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=xlN1USY9F6GRSAwH50SlAKnktVChXMDCpK+3vs0y1YE=; b=kbvazqB5OwR5dNGbDYwcUTmtXg5BtHbTQ+mBRJiyHJVXH1JI4E7PinU9Gcft/0Zexi n6gdUJuR0krzLG/bxE+ftNuZCrv6VDZ3w0PCtdW7Yy5UPLgKBpym/crsw/l9zLeqwf6k szAwjM1B0oQfqRj4ztFRdycEW/M9SSJZ0LK/s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=I9NcYypOtVAEUyWb2539/8UQU/KounopIH1Q9veDUQswkqY5cvmWuaaQaM8ETkkWem y6LIQeHhSkitx+jbXCgeNhWCKsDMbhfL+7f78KF8ddqoLZuYwOBhXNZY4aJFkGhQH54z wcCpSi5cJ+n4gXR2aG14+vXxe3X5V+1Uactdw= MIME-Version: 1.0 Received: by 10.227.156.10 with SMTP id u10mr1282395wbw.224.1297332401164; Thu, 10 Feb 2011 02:06:41 -0800 (PST) Received: by 10.227.134.137 with HTTP; Thu, 10 Feb 2011 02:06:41 -0800 (PST) Date: Thu, 10 Feb 2011 11:06:41 +0100 Message-ID: From: Monthadar Al Jaberi To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: 80211s HWMP problem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 10:06:42 -0000 Hej, wanna check if anyone encountered this problem: I setup 4 mesh nodes in a link topology ( 0 <-> 1 <-> 2 <-> 3) But I cant ping from 3 to 0, or 0 to 3, without first ping between the nodes to fill the hwmp route tables. This is hwmp+mesh debug output from one of the edge nodes when running "ping -c 1" wlan0: [00:15:6d:67:21:67] (fmt null) wlan0: [00:15:6d:67:21:67] (fmt null) wlan0: [00:15:6d:67:21:67] received PREQ, source 00:15:6d:65:fe:b2 wlan0: [00:15:6d:67:21:67] reply to 00:15:6d:65:fe:b2 wlan0: [00:15:6d:67:21:67] (fmt null) wlan0: [00:15:6d:65:fe:b2] start path discovery (src ) wlan0: [ff:ff:ff:ff:ff:ff] (fmt null) wlan0: [00:15:6d:67:21:67] (fmt null) wlan0: [00:15:6d:67:21:67] received PREP from 00:15:6d:67:21:67 wlan0: [00:15:6d:67:21:67] propagate PREP from 00:15:6d:67:21:67 wlan0: [ff:ff:ff:ff:ff:ff] (fmt null) wlan0: [00:15:6d:67:21:67] discard PREP for 00:15:6d:67:21:67, wrong seqno 6 != 22 br -- //Monthadar Al Jaberi From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 10:19:29 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71D811065673 for ; Thu, 10 Feb 2011 10:19:29 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0C66B8FC08 for ; Thu, 10 Feb 2011 10:19:28 +0000 (UTC) Received: by fxm16 with SMTP id 16so1285217fxm.13 for ; Thu, 10 Feb 2011 02:19:27 -0800 (PST) Received: by 10.223.83.194 with SMTP id g2mr14049574fal.75.1297333167892; Thu, 10 Feb 2011 02:19:27 -0800 (PST) Received: from jessie.localnet (p5B2ECC3D.dip0.t-ipconnect.de [91.46.204.61]) by mx.google.com with ESMTPS id n7sm609662fam.35.2011.02.10.02.19.25 (version=SSLv3 cipher=RC4-MD5); Thu, 10 Feb 2011 02:19:26 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: Monthadar Al Jaberi Date: Thu, 10 Feb 2011 11:19:06 +0100 User-Agent: KMail/1.13.5 (Linux/2.6.32-28-generic; KDE/4.4.5; i686; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201102101119.07350.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: 80211s HWMP problem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bschmidt@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 10:19:29 -0000 On Thursday, February 10, 2011 11:06:41 Monthadar Al Jaberi wrote: > Hej, wanna check if anyone encountered this problem: > I setup 4 mesh nodes in a link topology ( 0 <-> 1 <-> 2 <-> 3) > > But I cant ping from 3 to 0, or 0 to 3, without first ping between > the nodes to fill the hwmp route tables. Pinging 0 -> 2, 1 -> 3, 2 -> 0 and 3 -> 1 on freshly started nodes works though? -- Bernhard From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 10:42:07 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A9F31065726 for ; Thu, 10 Feb 2011 10:42:07 +0000 (UTC) (envelope-from monthadar@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id F27D08FC0C for ; Thu, 10 Feb 2011 10:42:06 +0000 (UTC) Received: by wyf19 with SMTP id 19so1198425wyf.13 for ; Thu, 10 Feb 2011 02:42:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=1yUsDKvtPH5eBadjL6X+aPoTa9gPOP8/9Xz1csGZncc=; b=Ika+Oyqg6QlNB7/N9gI8UZlX5bPo1hsL8rOAnwdhvmCelzHyINNYukFHVH8mcPd0Pi DZ3kHGcD3ei4CQ0TJJqcAtoLyT5pmSSifuP4q4RmqbNSY5eq1BCCeUz7WUoFDUMJX2dI cbw5TE4n/p1dB//1O6+NHy7lu8SmKgj6qURgM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=PwyV4QyMp4kg3TnzBv35vhdsKTH0Yj6yfE3ffXWhVXJPFiMUYQ6Mha6NtQIVzqRMrz TbnhXoYGR0zsfv3ZcCDvgb+cskYFxu3VA5Kbjl69yath+UeRwikoLEn5r8F3X3alZanx HYOwx1keJQFh8yE36j7xfYjwoSDA2Q+mFOhJg= MIME-Version: 1.0 Received: by 10.227.156.137 with SMTP id x9mr7429045wbw.108.1297334525246; Thu, 10 Feb 2011 02:42:05 -0800 (PST) Received: by 10.227.134.137 with HTTP; Thu, 10 Feb 2011 02:42:05 -0800 (PST) In-Reply-To: <201102101119.07350.bschmidt@freebsd.org> References: <201102101119.07350.bschmidt@freebsd.org> Date: Thu, 10 Feb 2011 11:42:05 +0100 Message-ID: From: Monthadar Al Jaberi To: bschmidt@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org Subject: Re: 80211s HWMP problem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 10:42:07 -0000 Okej, I run all four scenarios: 0->2 OK, but the first "ping -c 1" got lost 1->3 OK, 2->0 OK, 3->1 FAIL, works after "ping -c 1" 1->2 looks like path is built correct in one way only? :S On Thu, Feb 10, 2011 at 11:19 AM, Bernhard Schmidt wrote: > On Thursday, February 10, 2011 11:06:41 Monthadar Al Jaberi wrote: >> Hej, wanna check if anyone encountered this problem: >> I setup 4 mesh nodes in a link topology ( 0 <-> 1 <-> 2 <-> 3) >> >> But I cant ping from 3 to 0, or 0 to 3, without first ping between >> the nodes to fill the hwmp route tables. > > Pinging 0 -> 2, 1 -> 3, 2 -> 0 and 3 -> 1 on freshly started nodes works > though? > > -- > Bernhard > -- //Monthadar Al Jaberi From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 11:09:27 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03611106564A for ; Thu, 10 Feb 2011 11:09:27 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 90D378FC14 for ; Thu, 10 Feb 2011 11:09:26 +0000 (UTC) Received: by fxm16 with SMTP id 16so1335043fxm.13 for ; Thu, 10 Feb 2011 03:09:25 -0800 (PST) Received: by 10.223.101.140 with SMTP id c12mr12891638fao.16.1297336165179; Thu, 10 Feb 2011 03:09:25 -0800 (PST) Received: from jessie.localnet (p5B2ECC3D.dip0.t-ipconnect.de [91.46.204.61]) by mx.google.com with ESMTPS id a6sm978876fak.1.2011.02.10.03.09.22 (version=SSLv3 cipher=OTHER); Thu, 10 Feb 2011 03:09:22 -0800 (PST) Sender: Bernhard Schmidt From: Bernhard Schmidt To: Monthadar Al Jaberi Date: Thu, 10 Feb 2011 12:09:19 +0100 User-Agent: KMail/1.13.5 (Linux/2.6.32-28-generic; KDE/4.4.5; i686; ; ) References: <201102101119.07350.bschmidt@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201102101209.19983.bschmidt@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: 80211s HWMP problem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bschmidt@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 11:09:27 -0000 On Thursday, February 10, 2011 11:42:05 Monthadar Al Jaberi wrote: > Okej, I run all four scenarios: > > 0->2 OK, but the first "ping -c 1" got lost > 1->3 OK, > 2->0 OK, > 3->1 FAIL, works after "ping -c 1" 1->2 > > looks like path is built correct in one way only? :S I have no clue about the mesh code.. but, that smells like there are a few frames discard which are supposed to fill the ARP table. Can I talk you into dumping frames on the interfaces to figure out where those discards happen? > On Thu, Feb 10, 2011 at 11:19 AM, Bernhard Schmidt wrote: > > On Thursday, February 10, 2011 11:06:41 Monthadar Al Jaberi wrote: > >> Hej, wanna check if anyone encountered this problem: > >> I setup 4 mesh nodes in a link topology ( 0 <-> 1 <-> 2 <-> 3) > >> > >> But I cant ping from 3 to 0, or 0 to 3, without first ping between > >> the nodes to fill the hwmp route tables. > > > > Pinging 0 -> 2, 1 -> 3, 2 -> 0 and 3 -> 1 on freshly started nodes > > works though? > > > > -- > > Bernhard -- Bernhard From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 11:19:42 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 262C5106566B; Thu, 10 Feb 2011 11:19:42 +0000 (UTC) (envelope-from monthadar@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 873098FC14; Thu, 10 Feb 2011 11:19:41 +0000 (UTC) Received: by wyf19 with SMTP id 19so1228303wyf.13 for ; Thu, 10 Feb 2011 03:19:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=RbjwaemzGPnWQhOtabCanra8d8al3iZNFKHN/smK3+Y=; b=xAi2A/NxOsBiOWEJFcRmWmDVOQjPZTNh6nhGPZbpKFaIyGNlp5GfzS9dHBLxxwF21z WemocE7iVM+num5GQf5a+Pk+Ina5gtz3K05z+1ANe4Wu3iY+JaxAuM8Ac1BBmJoERSz3 x2ZUmRt1oWOheVlcIBdinEWD3sFJKQy7fEWM4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=uNRLMZ8hVwtT+fJ5MJmBoErwPgcKKZtme1kgN9TeHC/KnWJA6psTWoCMja4Hq870Iu z72IQlvP4tOOI/Vw8FAZC6C8CfpKc36oBc9Va4E9dLQx0p65sVM7qSrSnq/CuPv4ykAH pmxi1MtZ3e+dkNYr8fQUNvb/mGwv5uJ3dTTaI= MIME-Version: 1.0 Received: by 10.227.154.130 with SMTP id o2mr6203607wbw.137.1297336660615; Thu, 10 Feb 2011 03:17:40 -0800 (PST) Received: by 10.227.134.137 with HTTP; Thu, 10 Feb 2011 03:17:40 -0800 (PST) In-Reply-To: <201102101209.19983.bschmidt@freebsd.org> References: <201102101119.07350.bschmidt@freebsd.org> <201102101209.19983.bschmidt@freebsd.org> Date: Thu, 10 Feb 2011 12:17:40 +0100 Message-ID: From: Monthadar Al Jaberi To: bschmidt@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org Subject: Re: 80211s HWMP problem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 11:19:42 -0000 On Thu, Feb 10, 2011 at 12:09 PM, Bernhard Schmidt wrote: > On Thursday, February 10, 2011 11:42:05 Monthadar Al Jaberi wrote: >> Okej, I run all four scenarios: >> >> 0->2 OK, but the first "ping -c 1" got lost >> 1->3 OK, >> 2->0 OK, >> 3->1 FAIL, works after "ping -c 1" 1->2 >> >> looks like path is built correct in one way only? :S > > I have no clue about the mesh code.. but, that smells like there are a > few frames discard which are supposed to fill the ARP table. Can I talk > you into dumping frames on the interfaces to figure out where those > discards happen? please :) > >> On Thu, Feb 10, 2011 at 11:19 AM, Bernhard Schmidt > wrote: >> > On Thursday, February 10, 2011 11:06:41 Monthadar Al Jaberi wrote: >> >> Hej, wanna check if anyone encountered this problem: >> >> I setup 4 mesh nodes in a link topology ( 0 <-> 1 <-> 2 <-> 3) >> >> >> >> But I cant ping from 3 to 0, or 0 to 3, without first ping between >> >> the nodes to fill the hwmp route tables. >> > >> > Pinging 0 -> 2, 1 -> 3, 2 -> 0 and 3 -> 1 on freshly started nodes >> > works though? >> > >> > -- >> > Bernhard > > -- > Bernhard > -- //Monthadar Al Jaberi From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 15:32:00 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7F2F106566B for ; Thu, 10 Feb 2011 15:32:00 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from sunner.semmy.ru (sunner.semmy.ru [IPv6:2a00:14d0:0:20::3]) by mx1.freebsd.org (Postfix) with ESMTP id 269958FC17 for ; Thu, 10 Feb 2011 15:32:00 +0000 (UTC) Received: from dhcp170-205-red.yandex.net ([95.108.170.205]) by sunner.semmy.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.74 (FreeBSD)) (envelope-from ) id 1PnYUx-000O4q-B7 for freebsd-net@freebsd.org; Thu, 10 Feb 2011 18:31:59 +0300 Message-ID: <4D5404F0.9040605@FreeBSD.org> Date: Thu, 10 Feb 2011 18:32:00 +0300 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <4D52AB12.6050506@FreeBSD.org> In-Reply-To: <4D52AB12.6050506@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: route messages from NDP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 15:32:00 -0000 09.02.2011 17:56, Sergey Matveychuk wrote: [skipped] > DST is IPv6 address, IFP and IFA I don't care and GATEWAY section is empty. > Let's see why: > $1 = {sdl_len = 54 '6', sdl_family = 18 '\022', sdl_index = 8, sdl_type > = 135 '\207', sdl_nlen = 0 '\0', > sdl_alen = 0 '\0', sdl_slen = 0 '\0', sdl_data = '\0' } > > family is AF_LINK (18), it's a correct one. But sdl_alen, sdl_data are > zeros. I forget to mention it's for 7.3-STABLE. Looks like L2 (ARP/NDP) entries don't generate routing message at all for 8.x+. It's seems a feature ARP-rewritting project. Am I right? From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 16:09:34 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93C2A106566B for ; Thu, 10 Feb 2011 16:09:34 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta15.emeryville.ca.mail.comcast.net (qmta15.emeryville.ca.mail.comcast.net [76.96.27.228]) by mx1.freebsd.org (Postfix) with ESMTP id 785508FC19 for ; Thu, 10 Feb 2011 16:09:34 +0000 (UTC) Received: from omta23.emeryville.ca.mail.comcast.net ([76.96.30.90]) by qmta15.emeryville.ca.mail.comcast.net with comcast id 6FCG1g0021wfjNsAFFwQMu; Thu, 10 Feb 2011 15:56:24 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta23.emeryville.ca.mail.comcast.net with comcast id 6FwN1g01K2tehsa8jFwPeK; Thu, 10 Feb 2011 15:56:23 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4204F9B422; Thu, 10 Feb 2011 07:56:22 -0800 (PST) Date: Thu, 10 Feb 2011 07:56:22 -0800 From: Jeremy Chadwick To: freebsd-net@freebsd.org Message-ID: <20110210155622.GA60117@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Reliable PCI wifi cards, and layer 7 filtering X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 16:09:34 -0000 (I was considering cross-posting this to freebsd-pf but decided against it, instead starting here first. Please keep me CC'd as I'm not subscribed to freebsd-net) I'm looking into the possibility of using my home FreeBSD box as my home firewall/NAT box, to replace my Linksys E2000 router (which runs Linux, specifically the TomatoUSB firmware). I plan on using pf for the NAT and firewall layer. ipfw will not be used (I have long since moved away from it). I've got solutions for everything except two items: 1) Wireless hardware support - What consumer PCI cards are known to be reliable and have good support on FreeBSD? It looks like anything that relies on ath(4) might be a good choice, but I'm not sure what specific chipset is considered decent/worthwhile, or if there's a specific model of card from Vendor X(tm) which works great. - The card and driver need to support both 802.11b and 802.11g simultaneously. 802.11n (for the future) would also be good. - Driver or OS needs 128-bit WEP -- this is not a joke, I really do have devices which do not do WPA or WPA2. - MAC address filtering is needed too, but it looks like that's already available (looking at ifconfig(8) man page). 2) Layer 7 filtering - Specifically, the ability to block outbound packets in real-time which contain certain data in the TCP data portion of the packet. - More details: there are some HTTP-based requests which some software I use on XP submits to a server pool to return some ads. Filtering by IP address isn't possible since the A records of the FQDN often change. The software in question does not honour system proxy settings, so use of a proxy (Apache, squid, etc.) as a solution will not work. - I filter based on GET parameters or the HTTP: Host header. Thus, the matching mechanism doesn't need regex; simple substring matches (e.g. strcasestr()) would work fine. - Linux has kernel modules called ipt_web and xt_web which can do exactly this. They return TCP RST to the client which submit the packet, and never forwarding the original packet out the WAN. Item #2 above seems to be the kicker. Is there anything in the works regarding such a capability? I'd be more than happy to test out code or whatever. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 16:38:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 176D010656CB for ; Thu, 10 Feb 2011 16:38:21 +0000 (UTC) (envelope-from batcilla@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id C41FF8FC1C for ; Thu, 10 Feb 2011 16:38:20 +0000 (UTC) Received: by ywl2 with SMTP id 2so684063ywl.13 for ; Thu, 10 Feb 2011 08:38:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=+Si6mLTSwNQdMHf823Zy2bKKZVx9zf/+p4IP5Y2d1dE=; b=leMkI9V8O4b5JvqOKu6ni1Hta1xuLGpAcI8yS0gqwpRNDrPMBWmPF27Dny92Tj11lZ ybJM0YU2AVWAEeioivBI7Us8A6Dg5vApoGlRYNAFzl9biHe4cjMRY5FHj7MKWNPipkRS Ypsz8BJQgP/5qok+uHmXapkh5Qb1EVgmL1chA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=o/L9IZioaSNNun+IlMu8QA7t6FRPH2LZIkKDUgz9gGrNuXSUXSzjo3hJS44EJncIup W1YDiy9GgYUdAcK21kgCDrYRVDISWri+fZ7O+m3/Q4uTDj2ByaRIpzg1CejEFMmZe8Ko L9Hz5HCMDI3erKpaSm2NRR6lsUvhC7YKeWgg8= MIME-Version: 1.0 Received: by 10.229.224.212 with SMTP id ip20mr16072600qcb.237.1297355898802; Thu, 10 Feb 2011 08:38:18 -0800 (PST) Sender: batcilla@gmail.com Received: by 10.229.220.198 with HTTP; Thu, 10 Feb 2011 08:38:18 -0800 (PST) In-Reply-To: <20110210155622.GA60117@icarus.home.lan> References: <20110210155622.GA60117@icarus.home.lan> Date: Thu, 10 Feb 2011 18:38:18 +0200 X-Google-Sender-Auth: dRswkdsTsOcx3ZQoDOQWVdLYY5s Message-ID: From: Peter Lavee To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: Reliable PCI wifi cards, and layer 7 filtering X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 16:38:21 -0000 2011/2/10 Jeremy Chadwick [skip...] > I plan on using pf for the NAT and firewall layer. ipfw will not be > used (I have long since moved away from it). I've got solutions for > everything except two items: > > 1) Wireless hardware support > - What consumer PCI cards are known to be reliable and have good > support on FreeBSD? > [skip..] I' suggest to use PCI-mPCI adaptor and Wistron Neweb Atheros based cards. It is good for an AP. Use together with FreeBSD 8.1 or later. 2) Layer 7 filtering > > [skip...] I suppose you can use pf+squid as transparent filtering proxy ? I believe not much changed since that post: http://lists.freebsd.org/pipermail/freebsd-pf/2005-August/001450.html Item #2 above seems to be the kicker. Is there anything in the works > regarding such a capability? I'd be more than happy to test out code or > whatever. > > //batcilla From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 17:06:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB8EF106566B for ; Thu, 10 Feb 2011 17:06:21 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [128.238.9.200]) by mx1.freebsd.org (Postfix) with ESMTP id 4C74F8FC08 for ; Thu, 10 Feb 2011 17:06:19 +0000 (UTC) Received: (qmail 81392 invoked from network); 10 Feb 2011 17:06:18 -0000 Received: from unknown (HELO ?10.0.0.129?) (spawk@128.238.64.31) by acm.poly.edu with CAMELLIA256-SHA encrypted SMTP; 10 Feb 2011 17:06:18 -0000 Message-ID: <4D541B0C.6000909@acm.poly.edu> Date: Thu, 10 Feb 2011 12:06:20 -0500 From: Boris Kochergin User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.12) Gecko/20101106 Thunderbird/3.1.6 MIME-Version: 1.0 To: Jeremy Chadwick References: <20110210155622.GA60117@icarus.home.lan> In-Reply-To: <20110210155622.GA60117@icarus.home.lan> Content-Type: multipart/mixed; boundary="------------050803030603010300020209" Cc: freebsd-net@freebsd.org Subject: Re: Reliable PCI wifi cards, and layer 7 filtering X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 17:06:21 -0000 This is a multi-part message in MIME format. --------------050803030603010300020209 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 02/10/11 10:56, Jeremy Chadwick wrote: > (I was considering cross-posting this to freebsd-pf but decided against > it, instead starting here first. Please keep me CC'd as I'm not > subscribed to freebsd-net) > > I'm looking into the possibility of using my home FreeBSD box as my home > firewall/NAT box, to replace my Linksys E2000 router (which runs Linux, > specifically the TomatoUSB firmware). > > I plan on using pf for the NAT and firewall layer. ipfw will not be > used (I have long since moved away from it). I've got solutions for > everything except two items: > > 1) Wireless hardware support > - What consumer PCI cards are known to be reliable and have good > support on FreeBSD? It looks like anything that relies on ath(4) > might be a good choice, but I'm not sure what specific chipset is > considered decent/worthwhile, or if there's a specific model of > card from Vendor X(tm) which works great. I have a ton of Atheros 5212s deployed as access points. They are solid. Some field-tested implementations: - Netgear WPN311NAR - D-Link WDA-2320 Plenty of these to be found on eBay. > - The card and driver need to support both 802.11b and 802.11g > simultaneously. 802.11n (for the future) would also be good. Simultaneous 802.11b and 802.11g works, but the 5212 chipset does not support 802.11n. > - Driver or OS needs 128-bit WEP -- this is not a joke, I really do > have devices which do not do WPA or WPA2. This works. > - MAC address filtering is needed too, but it looks like that's > already available (looking at ifconfig(8) man page). Indeed. > 2) Layer 7 filtering > - Specifically, the ability to block outbound packets in real-time > which contain certain data in the TCP data portion of the packet. > - More details: there are some HTTP-based requests which some > software I use on XP submits to a server pool to return some ads. > Filtering by IP address isn't possible since the A records of > the FQDN often change. The software in question does not honour > system proxy settings, so use of a proxy (Apache, squid, etc.) > as a solution will not work. > - I filter based on GET parameters or the HTTP: Host header. Thus, > the matching mechanism doesn't need regex; simple substring matches > (e.g. strcasestr()) would work fine. > - Linux has kernel modules called ipt_web and xt_web which can do > exactly this. They return TCP RST to the client which submit the > packet, and never forwarding the original packet out the WAN. > > Item #2 above seems to be the kicker. Is there anything in the works > regarding such a capability? I'd be more than happy to test out code or > whatever. I don't have anything ideal for this, but will mention a series of horrible hacks to at least get the conversation started. I am working on a general-purpose, modular network-analysis framework in userspace (C/C++). One of the things that already works is an HTTP-parsing module. The code is totally suitable for taking some kind of action upon seeing a certain GET request or Host header. Of course, any action would be a race condition with the reply to the GET request. Though it leaves a bad taste in everyone's mouth, I think it would be fine in practice, as the web server's response would arrive a few milliseconds later due to distance, network hops, etc. The final issue is one of where to get the TCP state to send a valid RST packet to kill the client's TCP connection. The pf "flush" command looks almost like what you want, but a cursory glance at it suggests that it drops *all* state associated with an IP address. If that is the case, it's too much. The only other thing that comes to mind is running something like dsniff on the machine and calling tcpkill from the HTTP-parsing module. I may well implement TCP sequence number tracking and such in my code someday, but not in the timeline you're working with. Development tarball attached for completeness. -Boris --------------050803030603010300020209 Content-Type: application/octet-stream; name="sensor-2011-02-10.tbz" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="sensor-2011-02-10.tbz" QlpoOTFBWSZTWbg5D4sC7sd/////////////////////////////////////////////4Yd+ Pvtj6+uXgPjr5AAAPoGYAwigioJKiigDoALNVsFANPvGIG3uqfObYy8bY98z4uza+UfQNHX2 s97ull48AAAAAD0AAANvbNAPafUaeilZh3aO5pq6USlXu7c93jlJHnIph1smgAUtvgBkAp6s DAPgACGo+uTgDa0AADQACgAADBsAAb5DgQaAiBkIigxvc3lgfTcOPoxEPfcL3uFrXWPu++99 BUPPrxzu17U74TYAaHswhfD3DlD7ennDeKADWgCnYPuANBrO7K5ZG1e+MejdNW0bXXW+8uZK 9sZ7btmb3G9edEzbANNl3F2NaZDHQoGkgkKaHToAHQDRqELNIUEhrx1e+oa3rvb12+NPT6oP tntQHz1x93KPrbZvHxipuY2wDp08Xu22AemtNANDd6fPg9uXudvfPLvccBnXTWu9urjrWhd3 POePbbdHrtqzxje3nb1nKRq83Pth5Bt24dqYSCR9vneN3OPvtzXXo7Hlx7e4aZAGshbXE89c 1VW5ihj06CS7Po6RPbQD1sLa0vcziSkD3d12M+57vFbWyPvbqnOip6xXYX3nvHtQYJVaZvr6 d5obPSgPthFy6N7B99j6NU3mmnXz1aeve9uFUKgVV6yfbG2rd3QUKqVu4Opd3ODTNgPrqqNq 1tfZ64rz7d57OILcOt73ug0pTbC7QbDr2zTIFtpXrdjOwyqh0aHvZT1SHso49XcLbg6Blt5t IC83qalKIpFtiwDgudh0V3t094MK7e29PO8W6aU7Z0bVlOnr06aPr49vDbm7jplJha60tmuu tUwEuyjuc6lPvalO2WY9Vqld7DvO72sHtNYF7cWtpzsHF3WxNCtlZc7dh13OQYnIa3TA69u5 kHuAWtuRm64cTQdp2e9zSFIrLzGmXzcT7W9ttfeausUekud3Md77d482N2+2O9bdMToevXkH Lp7uePA5E2b319Xrzfc1Sq+u+853cbsKtpan23dnerbeiu9W51dfT6++us3xkUrQAus8exvq 9728+3A9dt7PeV009PodD6AB88sA9j2d073nr2pvuAGXRXoHV2OhoCj6EJxYvI053eb3ty7e t7VVaFmnRoE+hx7xjyoPcAXHQAAAANTRAgAAATTIAAARhANDQAAAAACYmTEYAEwjTEwTJgRi YEYTTBGTE00DExGEZMTCAZBk0TQSmiAgICaAmgAQAjJoJphGJpT02mSeqeTAJjSaap+0mlPZ DI0xTyAyZTyKY8k1P0J5U8amJgiep5Mo/Qp5T2lPU9T02knoyntSPNU9I9TammjTJp6gkEiI hGk0ZNBMRo0nkGpNobU0mGQp+mSnlNqZPQGifoppiY1NMmn6p6NR5TZQ9JmppoNqAZNBoGgA AA2KAMTZQBoGjagaNA0aASeqUkkENDQ1FNmRNPTUeqek9TTR6hoZ6hGmho0aBtR6jNIaBoaG gDIAGQABoANAAAAAGg0Bo0ADQBo0DQGgAiSRBMgBMIE0001MyDTIGiYA0TAE0wTI0T0aaCeg 1PQ01MyT1NGnoNAmKenpDIniYBDTIDRo0BNiaNMiejJkGlPaJp6R5JmlBEkQQCAEZDTQNCaA Amm0g01NlMh6DUnpT9TwEap+phRP1TT/VT9U9T1HpHjaCp6gP1Ieo/TRQDGoHqaaNGgGgA9I AAAaaAYgaeo0H/RgA/wK2S8Mp+aXjgDkzdQnMcfaEZaSpJIkJeO8M0zInjCb2jpjnik1D8Mx ls42SwZ5BuGMIJtBT7O0uutFP7C8H40EQ74iNw/vBNWA6xJ0xB1wRsEDrY+yQQxgK4Mdfg6N teZZb4P93zXjIky0lXym6UIPTF9J4WKgA4hLXh6K2bmFAH2irbW5zpnxF6YhtjIiqxVggjwL kKFOIF98O8X4YFogMfonWFBLJYthXB+i/K/cf1j9o0GR5jUOo/yHdjDdXMQFhEHkgjxMwDBw LAa5zZAOT8o+0KjoloOZewYBiYxjaYwBqSMCDIASLwkePWXbWhyRDIwoPZLFB1IYC5uP6B3D xT9E9ydh8E7T2iZ5ip7k75Ic4Et5uIeq1ztq6y9rFSvuljEfeIGYAsQgqITKYinmiAguUmpJ CMjkrmTecoYDiG4PPn756M1DQNA6kzUMwijvjdHtzLdzxq7kljl2KcYcoF50Z+0fKfkPbN4z O8fXOP0U2Q9g9QsFiqOOi0ABG0Suu3VWs3d7VeFRJoUwZglCNPwfVdKkINMJYNJX3Dc0CFou 7omMm87oJKjR7d0VCZGIondXMQ0aK9dy87lNBl3boRpICnnavXrvUxJFmQkxZZJcrdKasqar 6M2MZMjWkTBkwaxBIewII/gIAC5Q4ngBKCc41ZyT1/i/BxUJfZyv+5vz1PxdDrfXR9uj6P12 X8v4Tl/77vhedKdH2dHp9TKi7a8KixW8R5PqfZ1/t6v2Tg7FUPv7eSFhH3/d91jHc7eUHAGU C4X4GHnCvGwIm70EfoPkEUY0JWQZwZg40Ow2poGU9Oc6XqIP7J/0YCxnB5Y6460vM825R4Rh wsMsMYFR7I+4LDSDLbIasZ4MBnRLjchuMcOmGIvAvw2AzQ8JwbXwvZ+pvZ7PfdP0W2m48bZg ysrFc1QjunG22441JaTVms0xlrIroZTDNFcCYTOlHxmhar1a3IY1JcGMd8s6jv01zJciZwzR oplXTWTWq6Nqj0idPVPTPi1W4cwm7tpiqVeZR5klMtIBOdyWJZLmHms8jDrSjmvPX7aSO0io kEqONlnD1dGk1ob1Wxy7v08s3jBVuHj+SYt5eSfR1nLF1GJtLRH6cE8ODJvoUQJxlzqOeK3b pLlkM5GHYHUos+jysNcq4DKWbxnhpmXmGoZxkO5aWdwpBjLQUO4biUgshKvmC47ttCYw1ZZ4 /iN2kowmIDxtAfTsY6JSTZPpF/Gck+TTXSb+KrBfx3Lhy4mlG4KRkE5GOzbwptm8gulIEjLk 8Xx/SloaJiIUd9JwZRN8JaalHckvZs4T1UOnriNLpS7iG+mSaBN3GeMw5SjSj6YCwev5MLNp aRm4rl2JKUlERCYMIrRzIwDy0lNO3cnEWKZcCSby8SPFl3cBNrGXroMhGUYv4DCaaBm3hB7L tYjyanHgkTy0eViRA4gzDkN0njpy4fsEpJuqyZM5ZmoowhSKH0yqkZDpxMFDZtoLmXdpxFzy Wk5aOxXSMo/cPG8inJPFZiUfvIE3EYQIMV7OR5dw5cwA3m5mTazT2WbvDQJtZ82hBcJUPJuI 6jlGdSLaEI0Objqza5N0wm02UjMzbh8+esWxHcrpqPNJh+IK9eeSeRAkk0m1IbhUQQ/KWlok FZ7JBDGbdMG0eWeSMFvLsn8BERhIQVl7KWhxJN3JTEdeyaPHa5yk+isXiHrCQWDiaUQ6kGsv AiTcmzaMHcjEVYLQZeVdxZdVrLP2jCbgSCsJ89Tgu3MVZuIJ2Ug3O8aRpSKJuSZqzEs0ezca Vmw6fOISFYcJwxjTUq0lZpsIsWNGewl76I7YzMF7EBQIQYtUPpdwxiqOXMaTSQuYPEgzbzE0 yaSrp8iBJNWkV6JAmEk/O7jSLKWlYS+GG00wSYwmzSFBEFZ64dPZOadRJuA0Wl3ADhmygv4E yh3ItUlIsFWNHiJBkzYwUMTxmrh8+ZM3z6DLQ4Eozgy8uvhzTKDCO9hxUjxISjqJEDkNwqIb qG/iR5CAmyZwZmMo8jNmcRvDhw3cJu7JqiWjsIasJEovDURQ1ZvHsgIYdL4ceJJSDeQXpQ0J SEhISDOK8eMImG5+L9vb29t+jswVvD29uq9Cnbrh0mHZvzpy6eFqeWvXu5ca+bor3+fFs1J2 JpNzFmJUOZFxCTZQpWHDlVYjiGyTmobWG1hsIb+HEiOYacm8lGEQRIUFsk3DB49ewl0uvZrM m8NxAewX0rDThvnTBdJwnzuDCar3MKK6moLpjIsYj+JFWiRE2Lxu1jykqvWXSK5ms7PJLpMP 181MwY8JeZ0rAmFXzyRCirqAuYTD+UmnUNtBaxjspJE03Tkppu5dxk1IMlDl10xMPJeK4iBc FnAfQIkdo5mY0GMuZPXxPHEtHfKFKRQ+gwpAJx5Jc6TQiLCQquJC6PEcy7VZ66lYK567byzi ReiaizEqGcih9Kyr2IeaXJMokCTXyjt64cv4DqXSVesX8rDaa+F6vh9T3/i7ENxsbbaUI5FI lNexKY1jsZVZ7ThnoWt8uDm/Zzy6mvpIWyu+h6fO811Zmrw2O6e1o4UqRWkSXifGcr6+iVrn KIpijwQkkl3RRhgZ4m3s2llI8vkgMsX/F8ilp3dc65GnN3NsWetu1aYYWwwgiIvGE7SwfBSw we2GCwmTacqyphOsowte2CekrJxSlEItTbGCjpyrKbzpGNyZMe1sccCaxrji970Z0YZVxrD4 PlLIkWsq4YzxK0dGEYOp81sYqd9PLTornXa/LCt63lS2FJ1wtSmFrZY459r9Pe2FjPeY7bdn Xtrtvsd7Y7Ja57N+e/Bcc67dY4dkraFKxq5rTLlv2zzV99MMq47sdu2WmeeVtmOPM+2k+a0a 7czFZ7NyjW9b3z1rTG+2sttM8tuyVB88JKIup6qlZPlSdNaU7ldO7SePez9DwdXN3vH6W/Xs 9DhzdPX5vK7cuW7d7MvHDlq9btZRvMupdhLTMJ48eTCodO3j+WTVjwFGMgulmEWA/YwYsVR7 Chx27l1CZRJGBAYP37968kZGMs9fyTV27UlnEsT6TlXspBlJSWlZSG+htIclKpw4jmI6evpu XkITNVpIxmTmXatpRvGjyzyWaSTuVjL20JWE7dOnDiDKuHMs2fuCbP5RzJuJKSjxHr17HhPH qyI7GFKIWeOoBO1zF4tKHjpxpdB12NeinVgRhrK2N7X8uHhzvOU4fowrD8NPT9XSnXHi809x eUsvNXzwjr48evsfXbraKb9PNKluXKOfW2DxXn6yPRjXmnGPV1TwlwlKI6Ov0aS1lOmM5FNt KqXVjWW6tKTek6Oo3aylGi3bXekd4yau3UVwh+1cN3TxetNNXCqSj87l8zYNWL5i8exXUBoG rp01bLicx3JvphaDLjwjDk+dZaXfPqxz2Ppnyzfk/LPn5+c5uW22mO7hhvpflv3GPTrOXCxy 1z5b9mWuj482O7m5tvNw5s9mnDXLTlxfauhROdHpKcQcZcOh5qO7I2qabrnHVe0npOb6p+Z5 dnZrxvZU6I59Y3yxr2dc7wpUVM+uIvxrjLZXZaK4cLzw12UmlTHWyrWKkzhBfjBXQvjrnfdx fLOlTbjLsvfuV0107/e9B8+NqcZWWaKwHMWddTslLtI7KLBlnTCMwXP10w1YS8kvcr3UFjCg sJRdCet2D+RYsXkVpIzErKocKhpKTKcVcswhx5l1CcRIDZlAlXshJR2cvKrpqUQyhHiRm6Jp q6atGbR20jRl8VyUB/KyjN1AcwVz11AWZOlzl05fr4LSBAbQHjaA2as2kBr1dWHDq6u3tlKV se3LLqpv6+jh18OvXi/Pnrs68erqrry116dmuenDXp6WEYRY8dchN+s0VVWYwlzNq3YsoC5Q 7mC1ZQ5FEEmzWQgRGj2O6cw2EeRfTmlCitpeSWS8ZKxrf0ZenD+V9tkPnFnrhOlSKLJ1KUrY w5FVhjLBSi3m1jm8cubZsm7d/f5IvFPNEp7YnMUAlfv7DZnaB9arfZdzzdMs6djwln25E59z on1SiHHfp1fp7rQsH4uknd3esTVOEdFadc/S9R3iII6o1x5+XT27H7HHJ9dOzOa2TjivUzuc ofq7PDT0q4uLiyfTfrr3oiO9vbcgcRJaDKsX8vGeyKsqootJPlykBkwmGC9JovXrlztfOSkV q7giAacfFJB2QjLpM4XuUb0KGzukZaLSkUTYMdJJaksxPUrMGsBafinAcBgumpebRKH9X1fc q+GFsMJyovFOdoi0q1x9WcWrW2GFa5CnWV7VesvQph6soy9f0J8dfd3qXzJ+L18J6cMPZvwv rvllj0Xv4t1dKebL2vZ4cvP7cw6dPZ183cOJqtUDly3by083cNIzZpTNq25o6Gmgu3bujiOn bx4yftoDqA6Er3fJ39LS83m8GMtHXk7JaT2R5rqR4ZP4qzscsce57Gk+5r2vWz+ba0Zt5ekO yVcRIiTuSYUzF3BXwZmI8buYbJqyjs2cm1hwXcVCHKSR5aK8YSrVm2eS8nW4U1Gfv2cBmyYs o7OBWl7aLBmIseO0dNo7htLQ3kjKLmrFfIEJSDHmIaFnK9EvHhom3D2Zasq23lg5eB5DTC93 LKrpg5HPKoSfEDppJprjK1qWo3SdWqSlEvR7IjJf4fD/ycaTR0fwjPvyM+I7arP3kJkAEVDt VUSE6rEmo1j3DC2fd8/WB6gff3zDHjOfhpYcV7WTcyetQBz7eumgwFmUiPTyUIWUIkACAniY TRzqx2tiTC4hQazkXZc7UDFcG1ZLNlp5ZAMQaC0WieVh2iepSd0TH4yZaivnkrOma7yNA2SV HsWNDiuMKGhnx/0lrYXXlu56CPKvi8P85fwow9A+ePf+fCh2v4pyzp9ZEqDh4jo64lkXtQII NxY4NKxbWxj6wjMnrFfzZBMBSEDmD45GBA1yOZYVbEVKVyC3vEBEhsbgkwWZDCg6oTfMVBcR 84ePm7GZHZsszi9y2hLNlWlRSJKzB4IjHNHuNrvEIRZ1UU5BsAQhslE0JpkQ2zCQaiRtFH7y TabO9YC5gvbZnTrDv2cCmVnRUQ1hPpniHBy3k4hnQ0wYOCkWLMJrMBHsMwlfkzNb0lYkjVHQ SpuYcP7OJAcYW47x8TGfRZLId8xdwlp2wNXiUGpfMQ1G9CjKKmR6rahfNWWdrTqjvriAyn2Y TEv96LqzcOBPrIehAVtdTNVxJWMTp64bOBBDVUbYVr5O23ZwLDRLxvIQnSwjNcc57qTMekJ2 kSUI1A0H4C1zKgs8dS5DdC4OrVIi7r0ollPGb14jeMJiSFqXU7bBhZjYDtbwncGadrWYFY28 0IcqgLAiJ6sFXdlUUY23eBKfHQxxGQB0hGEJ9w47ljot6Jg9FxQmz5iIeoo+bAnjfTtY/ALF eoPsHCaSFUvGsF2bR9WLXsMfqNsFvD/okgbPkLKxx7oekhkx8tbFwu0iIWYvg9fl8YTOibZ8 XAOh9XVdfTbMt2x+ux1m9Hhlxjhi6DmjRvvoGr5PRH1fXnaL9veFN7Xo7IePyy4uja/D1m/4 cDRD+fmnBTh0frHlSdPNzvbP2l6dZ+ytp+jHMLYFdPYvVVIpTWb5LUhJJJVJMga7WoeYcdXD g+xpPHcvrmaT1it57x2tgeRhfpoEvy975cjwu11/QtVLq/Ys8Ts+OcwVzpmf7fjsMNua5/HQ 8dotW/tPBxfQq8Je/fy/k5WPm+l8PlrTDB9rBbaNq4MX/PTUvvVl7LfN9YvhyHdoHDxHGvXz TNo/zvxP/S986uf+TNzXveh+3dsL+w3OU9rK/V7eYvvtxN5pPSYfnWpL8vFSUnImGesxHvNx 6vo9Oot285Vw60LTdOxXfOtdtI3impqDO4ddvF+dy0WvYaYmrT9u4l6yPaaysixetJzFqPaV 2JwGGY4W5S2Pm8puhPToHK29H/ON33FwFkpa/mfDJUYB/78H/dV317ZrqZIQBb8D6yQf+7wj AOG7e6/lz2N2sjWo6+Vkfn9y9wWsOHAw9KDBYcbXNsvgJX54t26Xu4JefBu/o+brL9c6k7yn iHjD+FFHd+mwBN3U6gEhNSqQY/GLP64ARwiTET3Di4hJd0HtYvaKDmLmVTOdeYD3XhUYe8P2 bW7X7vUXQFLsn+NruivSvQFHpZ0YCnP9Zl7k2sv+owiEf58o/SFh/XpWX+Aq6/ZOG/v8Ky+B 6zzZCsH8dY1U/8Zdzy/+yYx38J5o/+Kr/fJxV/7f6R6E2NQpOP80RB+ibc6Qh7s8wCgkLbxb 9W7cm0IMDmgmIiIGEFBtEEnlzSEsQ8vMOHfv/jykHvncP2JlA3Fcn5P7iUh3NKWb/lJeGGP9 76Z5f9neQIsDwSyqYTnA5HJS3eiF1w2UTC9e0Rb0H6BDE8UWhbbOnU4tsBk3lyOOOSNp7qXk /w81pWd3/Nu3pp2S9B9q/1I/GI+0Xv681Dpj3kfELUB2HU1pww8fW/g44Hpe194alwmhDY5C vP+RoN5/9Kz5QYE68gYqw40oI34pHvBzgF0QfKEi8nchfeYkg5RSog15cGeQyoYkGid7Kt4A /wBwcxFGIA8Q4/gsGQakM+RslXzyBCVinDAH5C5JBi7oOcB4WZ5oNbEV8pZWHIpWHAnCcE4A ORhCiDEr3Bk+mO0cD+wX2u+DQpCYXqXTIJipzTdAaEDkxLygjaUkyA/Apa8nMkRTpH4Q8Bwg i809xvFirdb+9QwxWMOALMwObSghWeo4o72qoA/qYSG6Hd+kp4MMye0+8VhJ3kL4aMfD4/kz o7i3cfyWXMx0cxau+cnNsAzOE9xM15IbiRPBNrB6cyGSZvwIwZTpzMyMNJYUPkgLog/NBADD EinODOSBlUjgxFYb+cWyiqxObNMRvMXSKRtPqs1s2iduIxpNAOEERZAkTM+iHxWvMCT4hQEa aJ7OnFTVs5TZZZQJaVCAsRhqi8F0OYR3pd49ozlr2Idgw14j6iIwdAwSMYxgDMFgA91GmO29 IrrbIN+9VdTrOGRgKAAP2OFp8NyWF9Vd+p7NY3aU/co4cKIQhaN/MN4MVTZpd+WDty7b5PCN bHgFIZSHCiKnk2UlLNw3bOcBmNLt4UtMERjMYjeBHChOSMRiAjHkmsP093XnH32CTi6lg2cu gYOZWBMpGTjpQpFfq/ZQ0gzr2ShUTWUOTmSCSUi7c0ksKFo5aMllzpCTZqnPLlJI4gIPrFoF EuaRIY5HrnQe64L0F0on6vq+lSXDs4QWECSQmVaRDI7iiB5I5SWJevKF9DVYBevImEEYel/j KYBDG1yE29x3t42oBhxImDdgwt8G7vEYCzQqfIX200I0by30lav+EQleRhdKLblHAZWLAZWW axcI1UcqrhNgCCRdS21//efj+nmD/m54z3/dA+G1n3PIJ9W430aTcQbreSqMjzb7N8CLN7bj 4FrWjTLhnL3ou8dYhzLdPpxm3xqcXo5TXM5uPhvchowcKTeRu5S618w6YuOODcnGrvm7zdbJ uko202x7+yvxjWdFCTUV4dHBTrzlcskGyLkV2Yc7h7ho2d3Bco4aQWG4XNS1uqf7NfAVZLcp 6kEJH6V9Q5LhEFym10qk4omv4CAhBzmcnM0AtJE/OayXOMtKBqlbLi7EZI9tvFdVWlbUi5Xa nwbytTN5aOWQKuNEBoQgkA+gkEkGByIMD2oWjY2GVRIafmGZBa3rQFkRWrQgaaRt3TUAyEQV gyNq1sG9WJYwc01UaabWswzEGZlFYZW9g2NoTBgE3dGtwhRTKqkq3IkWESbS264VboUes21Q 2xst01Q20t0iW5BVyNTGLGi44hMyIyS7yKpHsCIbEuJR0LqjA5pAO6HckNwX56JUOydmdqpo JyGl5NIXYXmRkKEqYXddBskcXoQxTyd/EpcSrYwYpQeZl75bpuubdObG/35KHy/lBRDprA2j vuZoMAkS4gXRqHs4pH2jIv/t/YVBJkUef4K/iZjltHliDUmbSRV9hjp3II3vuTOvzMOBXBho czMtmeNeiKAY7giEkn2xCEEGgQ2MZGSQOsLRFQ6gFdNW+wl0JCipSrRAkEFHlIiVERG0Qapd 0ohLBOGUBdYoVPqLbxbXi6RUXLs21bWKta5aqjUXIptuayqnW39H99+V/l/4z9RhYJ7flMp2 5IbWYH1l0CA9sohBfq3wMFzYgkwVakw6yv91US40lfsdgjWKxZTr70Oh25IAVyqN69Vwa5sv 2/E3ExOLzmUN+/adb1uAYAA39GIxjA4o8qJxFWKqHD7V3Pkd12nneg6yvl+s31s9FAEDOL8s gyEY0PuvywYJifPrVGBkwYGMDGBr+PXx49LIYPT4zWXoWzLX6fvwpqNXLZjFMszkJxFiXZQU FAbL3AardedKjND1rtnLxxZ/LERiIChxk1ovQtQNFIIn6cBNNGSIgkQPh0Z4ISqPPOM9msok zYrKkU1skDCpEzp6P9Z8EUs+OO8g+FIWbIH+Bfd4nwb8PpvmZyKKHSicR++8hvLA+DYpVR43 p8FhAcKgGLx0LKQ6jrO5x9zgEV8PNx5PR9f2GZwOM0+N9Tx/l+BnY1Vm0+9egwszvFZhRs6f KZu6Tmbdp5x28OF1pOycrwveUWEg0hVqFAXwEP9vThq+zXZ2dzq6fzb3Z8RxCW3+X7ujp8Pi Twae32Kuxd7TpshdRkVSJ169WdwcBQ54gqnuigFgIgGyAXiqLZdOrVt3tWlDIN23bZS4iRMO PDYujgx5LNyAKzbaVPpfx/Xf/OEf6dV8j1n0/6uo081A0ASDhy4QCIJAxjGMTwiM/doAhxKO ytvg8ft1uow1i8OpsHK2Pt7+Gy1+27m9zhz9g9PudyyYEMIev27tj2gSu6FokiyB1fz5Dpww NWrMMwbwSSSDooakgSCEgTEUQ9ejKC3hAkkZGWhhlkhbBZEkRhuBBobBtBwob3iKNtts+R8e 8rGDu4JDu7odVZj5/5L/IrUPMjcj3GeljKaYEt27d7JOYkzBlWMfYxx22FRMOkhOhJqs/uV/ 1/pef/Fe1CENOkQstoIaSqBkRhhhgi3ULxUkEwE3xLUlgqAJ7t98vu+HTkheABJUY1FSfAzz 4e21vMFjIUZNEYN6W4Wi3d3zflXXpsVorG1gxa6u93u9a1eTW3ruqijsY30fJnnfLMSoJIOv Xx3wmBIJIsaNbaDYnXG2MIeo+mLGkDdaCNInbBEdn/dvN4wJiMbJplFgoMkSaYAkWk97a+7f 017v8Hze/9r8/5dalf8bUq6gZCRBJEJodlhFo0DQXgJsExjYHf34nreltpI0mIyZUARgpDno BvYpXTQOYlhScnRw21VnkMyBoCs2zlgSWuGUItKVSU3koQiifw7wr909v+6cI+kXmWDlVDMR HLZVShUjBU2my90ZGQMBwkgyemf+a+GCmEUJE7l21gSUdQX7N98C8OPiHv4K5sE5iqcqc0cX 5y38d4XHyrsGDBb5n7mIC3UYSx/ye1/I5/5V4JigN8UHvYg60HroB92IG84/PseT2ut/R394 LabZmvB9loV/1Pvc9yH2fr/LolTCUK53449eEeoIGHAIKX2XQYOyBMz3bsHSFR8X+/f9UOo/ RdVen/zZ8P9rlft/v6OA5j53N2kPXTdQJpUsgLuYNQ+XE/3qlNpF4kBwQ5GBx0t0FD/9j/1/ vpWfixP8jMx9Hx1b7AHgzm+5+fZHjiAfxwchD8oiGf0VL1EXV58u8n1f/jvdG/8HizUMircG bonmTVPDPwzM6bss00POd3nICLtiZ0PtddR9mKH+5x1Iv6cOMg9fB+lMkHIERGoblIn/aH93 sPvdXcPlOloW6Kh1M5KdTBuino44z+gp+lOeipoRH3pJvj11QESuHqvI9X5MCfzMgXDRX6sA 8MF3081DBEMofl0NQMmLc/ZvLxZEb9rxVi6B/VH+sgd9Dgw9uR9Z/vT94i/+IG+iHU6gGOyG SIdvHSiGjHRgPSYmmIIRdMoKIPo6NwvouXrc4Ehnr+wGMYwDhdKnzS4xNtxxuEAxmju9r1uA MBgcmZ+BCy3hEnuimjaYbNlEIxiDOmc2aQXtrsk3zl3C4mEIHQOJtorBVM4iQ553s24RDaZx a0NMdIKw7KKLCucMSATMxSYsFBDNGUoBMmKKt8cZBRfzf6P2u0uvkIKGXNDIGOGQpwXAe98l 6Q7Q+rxf2P28X4XP77tS4UX03b/bOyPK+W+t7bFxsPa8T9U4HO3qohtXj6TdWppIr/ZICCMc 5AwBgbI5GQtSRvBxrLgtNfJH+M73YOJELxWjF2D6aO3JLad1AEnP7Ps+zgYHvz24WVlqaMMf eykME+SYn/UMT3yZ187OK1FZOmvWkMAx+WfXDkSc+YIbNWXnFmYVtCMF5nwOagHxI2inRxQw YJjhYMzT+L2SVhbyMOxfGBIEloa4KA/UMj6VMwWE1CmaOSOU+Crscdal8oCRTnIHRMmrw1E2 h/4iIPoShnlObOZlL764lCYz6tGfFQRvwBR7qKWiAj+bGt+qgKLnEEkECRSQVJB44q1BZAVk BNmzfJvn9YwDX98oOkgekaMtGuWwbRzY5TilRGjbEAA3B8f33w5Nct8Ny3l5fdfPiYMFo8lu NZ+5d1cLlxa0b43w28mjLWnilzNx6zWm2XVdgy3eXjNb1xkzetR4Ya1Jmk+M2bi1shLvM3rM NaW3pXMTeY8sTunmpvFEklEBCDkoodIgoSaiAzanYADd7JL3Pt6PHceEh62S/Ln7VyqqN5lj hMi+M/Qe5xcp1VOCwX+19Se/5nD3dBG9P5pKSJTtwGKWjOqVaE5I9KCk/rqqsJn4UpNgypeD 1Yv3FLLvxIIaNz5yQb9Hpfv0l//EOfOHcRGj9mn8dWe2fZRxUlxJFQJdVnzvqUW4T86P4Va9 CkUUomcBacTYzZ4pdgpmlmESLyVnf6TkCsTGcX0CK2pXJI/zUZ6ScaUytXA+x/tJOVjr6ega JFDK2LutBYVMswWEImV3VdyKZriQSYQ17B5Ne0wZr1F9D+56Pwy0dJrAXqL3WkoXKFKgl/et gXdZOyyxxHtDCRlbHMTFGuxLOGg744l64de8raBOU9uQ4stHOJMiEg0EN1LWaZqVElmFtPR2 EU87aYC+ZdbqRX1pmy/KZZN69FUe/m2rSG7ZSXl5ZNKIQb05K2qRbKV8mbxH6vINjeqSzZeI KcvNJSTV/dGMBfIq/i7/BhSApyur3zNjRLR11vJUjVpg1j2Gaj0zYmK8NZONTTkS1PFxLor1 ZNNfPKKyRdsg5rK6oL7LKSpB0mytKVROSHZrcC2uKhnbOuuaMiB6OkZpmhIOPsr5686ORXeh YwaWdj/iV0gnX0zihsC4E0usJYLNXiIxTBWx/V15R53YVhXUJWRpXoVwZ+8qxLE1S7h1px/q s4T9Q6+mfWKqJ1ZDpkrTrDvoC+QdfvfE4hQpaZn/0TpqG11pzVWu7Ol1koj41+oFJZe3erCd IJk7Ipq8oGBcMEHt3ScpS9ZXLek2kTgqiinU3TRWylMrHER7eJZVRknVP6WgTKosTpw5uJ6R vMr4eNtz28ExWEBDp1SuklG5SxCDXZedYMntoWkCFO5mpZJq2PaqmZlINbf35fEoLRtFbx4p V7F/5w5pKGmjJyLZ2SKyUOI8wtJJUVqrHL+cUa1LlF0epoqE3KcmnSSM9KVt11YdkXMylSEr LH8KKqgbKMSbVi5+TuKwWrMKpdSCs6o6lYAXuYC58CIo6bhCECXfWejYq99odbE/vfUJEK7I ok6OssT+pmHy0hUPl7NMV4oU0eSu8RWfKpcv0URLgTY7SYOZsQatZFs2dVczMrlbpTMn1I9v MN/YH66UevmL2QO/fosjo8Z45eLq3MsWd/Vu694mk7YGelVMqs6STZw7boaYONCCz2WRXLS/ UgtrghEvJu0Onp6Khdnbrw3psOhqQako9YIJ6TPB2uvrKhrLyp5ZI0o4YS0ym2noFXOwlKGi bsqtPAlXqCDNRYyghYtERla3h4JUoeVP60i5m0kGt6IR3ClFIRGqQVJgpDlcFIi5FUt2CjNi H7wTDpabe1LZcA3WOeYOILxnU1ufnFnZ4ay+oosOySZvnt3Wl0724fWNfCUFghHjzU6RKpyJ zO1TkQrxIRIiIlTorESFtPUfocVBCapLtQJN2QxONpJdQN5I9IUrY4CBZCkCvcyeytj36CcT SwZw4iENVMABKnpZuOzXqSUHAu4adWGJpSgPNuj6CYOKBrMt3kyoHpPqN4GZB+F9iVjXCTuy QbybCefzjmgWCVxTTw//Wd/alzZV2hq9PJXa0LJR2B2hxIIBpWIyGNkzqCSOeuy8lCTsbG+E k0OyUlaR2mc8tZSqpJSfWo3CViu0VE9Qvr/60+ozs18X3lONAYAkP8NYUQQUKWuk1EtR1KBW FDpEzU0OTksLUzcWPPr48CfQ3pTifKGWDYLIg1o5nDKvNklCfNZ5FYkdczOqRQ6UippNJM7N XGrxcFEmFTIKJvDurJXGbe+3hcGN3oX04mD1VbZpMSD1FO1uKijhih+kEKVyC2mmKS6bXHEO 4WJynFxCq+QnV94ZrmkzNi1V+qaMZNpBnZtklBwz+9L1LAU5T0zV8mazVyyUUvLRH6zt8cUE uiagrHhwIFg/uim5u3wV0mxrU8rJSFqjNnky4YBhIBsxl165o4OGJBs8UermCqouhP355pJV tX5GeXVeAbMGVkWJ8hg4OrRz9rqUxg5mZmmlal1YyUgi7TNqu9/6/P6+Hjw8vreaOjyf9v2X o4Yd6EX0yjGpy9856SO8p6OZKwon0pC0bB3GeX5Bis6tSJC9zClscSCrlZ0q2RUW46T23ngE 1X2UpAJCrIHqalat2GG7WhIPUMUpu8WRwpJLMyWsFkn5GvHas5LA1UR5DqHLXHR0qhK63A60 A7op5dKorhBOrW8xKmrq5EaUomKQDOdjLJKkyqq7LL6bO4JZfKsTyDV+4gNqB1KsQeUdpsbl bVcHc06XHyKgVhRZkQkxMqrpCKnHdHRfDzBCzSsu0evk1AGpOIrEVpiz18dZVwyxj9N0/ZNo aahT5UhClIJW+ASAurksg1aivF0VY55Sw0a4U2OwiFGBwwJlgm6G79qrmD32AqMY9VYpYOLc kKLqdAWJ8VbQtSpkhOJWzt1J6USw9qXSaV0ctJern59WQkq0vhqIb2yho1l2GZsaEl5NMHX2 TooLS5wJtepe5vBVac2Tx1AQRES6bqK/Kuk11jjMWFSxfwpCmSWnpGRSjYeNBb0agOSUkeVk w9QlBVQCKBJOXD9cyVXkFT0FarlRPtqKVmmBhUOZCfPAkVKQqFFKckSViUvq6GFLfICZZUaa ai+BXGS0pJrlEVqdQrcmEBZ5GfHmo6trfOqxSLIrQ6embtZqErMytjxKsnZmeCSir4lupqWv wRAl14dyj9i6TdUFbWlbdWLlgzIidoPQKuq/FuqqobFMnYWSwxqJJWoxLR9TTys1LVyoO5uK ajRqdoCYSNlcvWDmaXRVlUGoHhzLEVDDFA9YTlkhJWCIdrRTdaTaCepT1AJWYXoVl7o+SNU2 hwgWCOcSNGwschJpzEnHp15ODihm3a5jYIC8lqBa9v5ZeITGJINDyFMyCy1DTRk7mWBvBxL1 byYjx1DBjFsp8A3hTpUTpfQLMI1GFZFamnoUsJeqUgWW7N1JlQsZTHScz7kWl9W0kpoqJjQ1 CmEUklIVLDTSann5SzIeVxggLFVT1hk5KUcVCX8qrrO/mIiFya7GUSSQsKxwiorUZJyQkpKW mll6klMs14rqYm7XGWUOmfHTh0KTZzBhHu717Asi9SSOunIkadoVZxcsqShGra98VmIVb9R/ ZHEmuk1JeosDSKmdhQkh4shcoqrsGklLKJ1ENS+ZGy1dndMHCzY8KZpYtiiPFcGg9RN1+PSU 8tO0EwvZQWLptGs8+eLYWCi5gK4qxpmtLU1a11l2UCLITKt1sFO1SFUvOvIHlGaGhXIhdJ48 xYl4mV6cgkeaISrs8dw2OhUHPYMpnAREPeBxmAgKkd/xv1UzFn2+Cz8KG3+A+GuGLbKyvQ/w +zr03f4t/fcf8/Lg34XzXLW2/uL60Ni2qmb3gd9YpOeW2ivjqvp5FWG2rgmubZr0yjLgqr0q gvngoNQBFVGxGwDvcfLcgOWMkceDcMxnfPOe802bJOlspRbYW2W2Fk+7FmGCyMDS0TUo2XkS m5lN7qH3wYYyjC74rXfrOetd9jDe8luVqbolg84nCV0RTe37qIo83/cGkQuRQQVhp4s919nF p22mjR1YfZccdnDR9MVlzIyvv2SythwlhiqUtPDZnTPC9J7NhhemWOcY00+hpNlfbaux7PWj i23vSuNbTjbEcn5owwtdUJYSihZYZYxt2vfY9c8YyURLHLXF6Klvj75YbqG2Wx8Yoq0KypEt dM553txfDHGFXHOdpG7XPdg+62FrrLPTS+Tvtyvw245QS3SiHpdQoprOclrssS2YvhZyeMZ2 nTDG2Sk+3bjIxrWz4YRrNhhmNwrKaYVnZ2wlECIq5unOKRTfKtpqIUGj2h1teGoiaDHFwrjf GJbMtm6Vcawiyeim74SpXLKkyjmEPaMayfMBqKGBAAJmEUAkFQtBENp85aW9NWA0hggqZb+b uML6tvv5ougTMjDMeB0BoDi4DY63lNkz9jaVquwXaKqOhEAc6AJIKiEIi1CQFflbYFKBfwBI gOoEAMEFRjXKvoCvo+dFW2vfCSSbbWvlQFsRUTrIBUUUMMWrypCgglEbgaravc1bltq5Xp68 5e9O6a3s21MuFdmN1ZXE260eeebybO9XlXrMWvWWKUbxtdvOq7UAuwtYKBiWYAhIDrIqWYjZ jtRUkJbmk23ddZXdrem3IoxmGrxW5qxWNczIsoMGDNsSm4KDWmBBgWwINMsJu6dxvZGrpvot tcqvFT2u9nqXhPSp671Eq2qvGuZ3VRbFtd3eN42CDEWm89ea1vPXXqiRkIYxEdsrEhVlGGSC warKxJLm8l5Nt6beLeK29zba8RJV4KrpuZIi6aro7tVGSMtoaaSoxUgUsGNoKxRpEaE3aNRW 3NVvZta/LtXIttfIq1cJhUoFXZBU7HkAaAR/JFkAQxSflI6CEIGs8pWEUS8KghUSjVs2wthU ogw9Nw3GLlku5TJBvwWFnESjDeCQlpmNGPcNN48rOZEbZGPT1UCSDgaBsrQKMAQtVTTjsjMi rqbbaG2MbbNDjS1IXUQfTEh9qMh7cEj0bEDIznygqJCA9CQMfIlXY6UEqASAFw3NZKk1dhfF fHd5Lb4lLS17EDGCbTZ7Z82FZprQwTYr+IQxpfs8gaRIvrXXavj7sfQmubfhZtyTaYlX6r3J Ua3r4erVbDTHwTkzWtZGW9vbsVvTVXpFo1barP1ygEC565S3sSjrKCkgQLDFDSRLEFRRGECS QiyAQzzfZPankZz2A32MCUdXCLEkmEShBC89bNfDiyGADHACGvU0aNIFba6rX1v+CtdtVv/5 lQe4EgPpoBd8xRoADAQFBxkEQFW3BRoUWcctKCM+OgUoTAAUqO+cgFAjQkBARzIoIi1ahFgH K9H1V4F4qEIiih74+obT658L+APbXbEqZoGhEAPkTUJTuICDYIoBadNDRgQFgMqBo0MQIVBn qYiCQ7CIYtOzB4uDBbI+eCOMRTCOJEwGAGZAARqJhEkSBBQSEUFbkWwQPFEiWBAYl/R/Kn0W gRw9sNg18jI2YNYmOgzoEV+KAaD3D/6WRBcvMBpsNiAKBIIcJo8lgb62imAX+GhVbwbkBAKI 0QC/pUZkMI4RHxwAcYtyHHgUgGIMMZ8rw+9vBIOeeaAkHLNDADzhmMxmDQBbCtJKuu6cBDaE HkZt7TYxbIGDA1RRMZ78wIYEEciKVlReZwUuNLo/5U+D9a186/R+H+od3Z+gtu9zeGb0b4y1 5xm9azWW5w+Jq6g28jV4JxN4+3p2NIS7nGCETm5a5FVN/4ihMZaSSBIgNTDAoVLT7tgJZ46U FLSgID7tpCR0YcPW9lEIOOAlmF4XNFMkjGKuMwgLIil+GDsDSSR9z9/+7WzZtTvAaEt/2nNz faU5TwwdSelySo2FWw422PMhjpxY7H4ImITxfOCX8M6l6n0BlslS1FeJoGHDpwN3wpEP0Jz1 b6ybZvqsWnhHjNB00ZtwUD/VjFSknmhTFSUlKgMchjsXb1OAN4Dj3GBLB2odvOkbQuCiGaiF Pd0UCjcJFIOhQih+YIiXgIPfzokmI6cLqO8rXrvre34LeGCNEimq3p+RvPdauMBL6ehGAMF4 YTiEYB03UBQQxBACqBTirFEAII9Zgqz7SaNgzYoB+9MEUM+Bgg2gME39VFtBF0iKWgr2/U0o H0J8SV7Y4Nv2/5KzJPj+SsoHFhAAc2dpBVM8Iqm/Gn50PCWQHGfNgtQuP83xdcnXD+jLM7UL zblmI/ziFHqVqgiW7dCNogYwET8LAHkYxiEZx8QMcU/YIeNAkBX05BFfuwQA+KaIOZAPLE1E SRQO1OrdQFXoSEAhE/PIApaxQmEX2fzFOyCDeAaYpz0CQRboJhiKAndkUE3EHnYoobSIc+Qa ikhIIa4RByaVEYA6RG0BNMGRVDVBAMUiY8spROCChciHDBXzEXUQfXhgsE4/yyrSQEgsQYQq GESQGxlTkRW8B0wdlikXCNjv93u+/7wnc7/47wu/KRKOiR4U1uUovaKToXDrAmBaZ/tyj8Ol JupOGV5E1YzKHjiQQIQ5pe6kCBlYSvDMo2mYTtpr6zm00RdXfqOaLsCB32VkL9bVRhVnOIZh xoMExztyiwYAtjuhmjDQcApbKLxrBtQSuZTtFKOSK1UfkRDzHmV+AJ0bJOJlYzluznSSal8U pwfiUPZWNKtVWcsjL8pKRZUVWpscvcoTIk+jkiCdXYcbZJG95TQUfheMMFh+LkXJx2Xedd9c 6vnerw3SLIIETNYgbLdmRK7IV4IldEpkzIcfDGdoxUyAqgdkIPWAQF4SW/Ae7TqRNopgxjPm 1YgcZOujmhbe2dIAdCIkkQSTsc9FTVhVpgo+33K7fJ3cSpsgqYMUOIgAlCRDZHg1823ENx9i kHIFMebddLhyx4IIWQ+l1UFiLARM6bQRtt46QtvwaN8gLrjldKgRSGWAUa+ZNKwbaWiOWgKF 24ETbjnSuDSBUXUUCpQMEOKdE45smiputuhj8PHaSWGsQpw4wBeoT3b5hYNHPuI+leGYGNEw 0hlikIEW+HBlWOkihZIGhQ7mXwOYKKkGb+uUg89BxCKQIM3Z3+EsdIVTWb2NnyMXJ9j7XH2Z 4Jklp8Ax71qMzB9KpJIZuY1czd8EFrK0EIYGAJCO6e6KLCczm9Km+QG9UQHvKnTHGjk5DjEA pXIjJwcHDnhgSCiBIS9E3+xZ1ycxq5VJTm4T+unDZ64SkAfCwNxBKblfpv1xWTUEMhMdE+g6 Ns5ow+XsgL7ZZy5ZuU591cZ7eM9u69Bcw+y2st89EDPRzNFkqYbCTHRo+7PBRC34PPthY7M8 9thpcnZhg120jHi7FqPE+GvlNk782lYudBsfDkaQNKW+sg5sTU5pMFAvwi0n0UNN2jfnfhNm sisbIlzDmqIyiueUqbZuzbSeNpAcUxw22gY5ICEyTVExuR2puhDBvQbUVWq5cz0TDKTtgit3 QmynWCeRKORaZBx0maYZU6IdmYstlR58tsDNzr8D7j7WfTzZ89mY3oY1W+XRBdTXBct7h2CO VcFXCbByyywxmU6XzOrjWs256xvzm11lR6O5cm659a0ndA6269Ir5l65brhn10nPOjN1XplK Wxw2z0jBkzVznG7e7Y7R50c28LQVNpt4S+aY7+UhsUbEGzhpsr5KDWbdKfYLDmN7JtyzC+Vr baFAUxnHArKh5ooicEBYb4SYZmLq5tfSrjAwXQUx4WiQ5m6zcN3Q/OU7QYGJIYC080WTchjU 6+jPjQpZw50DGhA4IXyw13TAi58tJxc/PxYGCojvQNHFx2FWtWFrB6AgqDmx0c3BYDdn26UF RTMNjX182U9Dm5RosL88M1EzdPB+tYU2QY7eaCnJzuRIlDz58K0UfOklsYRzQibstQPN8I3w rwtY54nw3GgH4BwMIK+P5HpjADxhzN2r5Vq3794ufO1J1IOupTlaPM9bXzlujeyr4OfuY6zK /PSj+5zmrzzL2P+Xzl97Pg+SSUAsliGl1mvczg4mf+UZ3rf593ZHxZ3H18Xj2x3ePol3k6/d 3uLGri9GrvPUvTjEZe6brt/pXr3l9t0o381VqrtTr0MGtm3zX74m5r7C3Ve8s9wa2Zf8uD+Y 5zsNdeOfbMHbPa0m5yfp57x3m4bfx97z+fLe9R8V9Vl79b8/6cD6c/gD5vU7HHKUGWUxFT9l PTYn6dxla96ifvs8nw5dbzPa+zoW/ctN/5W3YzMw55v01crT4nJT09O4+1/hzamRVaaGh9mT yN0k7hk6Tj4srL256WV8Pv82NGZ8WZ2anE03TY+j0eDn5L0YrqbTu3Rr2EIuD9PuZHGrQj2C 6bueY23farGI1fuPfrX/V8v3ZcJUKJiRXx/IGGL6sfXvlf81G8FPP9rmsn/6PrxM/R8Hn10L X51gE+W02Rg2+mDbhve3Fa1FKK85yzxaN4xYV+2FixkrJ3rRXD7Uu7fMDixWTng2o9nQ7utW TmTNdbjOXBzb8PNe7Rd0dmVw3pgD0aXiAxqQA2y8Bciy9mDbpeUKFhf2/k/p9l5f/xwNXaEk SAyagrnlhKRLCQU78SIoV8y6zYoPhywvkfxfGP+Aor6d/8vDvDDjDFCGJUGIhyRrAuF9TORq 2LFdNq/+eVj9PaLvq3HC9q7ztblq7MChk5qyzlJmqTcd+ezfQ1+M2WyoG+oXWSo5959L7PDm cHrIs7+P+a/4fJT1ry/58n93D26j/Nv81Z0a1/sfnZnh12+SPozPx/7Vs9jBf6em2OXylJx+ 1wLIYC54mHtG+uwNL+f0N8r8nM0F+zjb85yp70L8hn6OLReL3APz2tg9kXPG9gPhnFYdyyXd 045u8g+aRIxLb43ZZPJrJHVXef4bLmok+0+ih2PS2nNt3P13i5FHz+vn2Ph4WagDHxOTkh5m eiUVElwXFd5HGqbJqYFng1XLaTPuvqoW5ZPZxL1dnXZ3XZg/5F7neCrn/pyhlNaO59ON/vMD 0sZkLBYoEE7PYmK1Vadz+P9vc7ojB7ffV7i1TIWT4jcU0cbUfX4B3s0y0tdoORm/QGR8fYW+ fwPqzuWnGYbc+xjZi2pDX/kN4lZcN4n+FktuGI5A5Q9MIMDGAuAMCBjD39ccGqtcDq/jRf2D +n2lk6bkP5Try71U/5ifDh7Qh6vxqPxfaz55Z9RNCNytIlQcGGTJIJC3Ix6g6ag6P8PflYWd DeQH8SFp+IOBOCrZ9Rg1lB+dgH0/dhY/JuOQUgOsJoP7vIXq0E7IttmL5uOAdy8MFRLkBMgw OgxgbF4gtAUR970/W/r0tBsNkJB/e2sbLEfkkvzyCQIwIxG9EhuhohSt971hhnM2Jn+sNzHY Zmtmcm3FQQTH3gAAZPLQSUaAp5qonjwP3P8fR+OM0Z57psZ9+V2tjAEF7CDRDvjpT6yFvMH8 5j43P8j/Z8k/ht3xqU7WvB+3ktJ5v+k+lj0T/5i0shtPgmnbT9X7wozwyDijId3A2baQFiqn wm1dvQFxHOIog/2Gn6vhnH/xV0rjxAHkeOEozSE6Dzm/NDgUWk6CSV3uIdt/eUmD+qhQ82Za M2AeNYo2x7o3pYPTfazf9OVvGmD/XtS+x05+fR1MPY+PM6w8MhkIGjDtYB2M8gk+Mdp5Of+C iki8Ax5pvzTMnyz4Igi1i0BzJzMiv8+ij3LT2VKbShjTCY8zQjSTQei/z34T27aXfiDa2sfO PmG7vI8OZBxkML9dFlcv/lqtDo3WHI44+n4gz/HDY3Yj2PeGqxOgzdm6uQ5de49o6Gytheax 0+fwdJerNWWL7v10yd8ZYat2PSrY9k1tHVj53weBvj/Z38qlH97rf++zI/TZv15iaafs3DIL LKJ+K20fU29RpKizVqgYJ1Oi89p4vg9rp2OHuoa7I6KRjdvyd6ZmvXlVuxur5fr9g+V3Pg8e 39bjbj6/qHi2NJ6X2Zz68E59vNHvW1zzF0xYr8bS9bdfpztrTXLb6E+e/a65p9Sd/x+bibJg dvwkarfoIEPEcHKwJBCCLKSCDQMy6ZpK3IXmyXP0ORzvy7WAF+gfrLPPppMoOh1/VbQpCYS0 qPkmImkxAY4J66TjUD66acRP2l2DTbWWR4Az/tGIqYuhvGRjD2iNEuCgMCMlti+HOZVrre+t q3Qy97w+ZwPs72uSgrphPAjfJc6Lm5c5hmZkd1cyO4u+zy19ptf2sLRsbEExRQYkgtJGKjVJ FRqg0gZKktiJmhNYttiNaIo2Stsba0ltGxtJUWLW0kWoZqLTNgzNUZK1G2jbUmi1qNUatM2y bVzVa6bYxpKg1krGi0lEbbRttFYtaSKjbFtJqNsWtktJSQWrGrG2koqNV12u7tXW7tutdV3L LblJY1tzV0vxcq6ljWixYtNeda503DA66oXOH6p14vDdJz+oV6V6RXwr9M/ybK/Hzb2RFa9p XK3qUIYIygbA5gYDZULDdUD4kirIAkLFZHw48y57nE6vxWeQ37iLOturYuZ3v7Fqsx/a/Du5 rf+9ffc1m4xP5fPI2P5PPkbVi+XJdzUb7RNc/9ug/LuW7rdyLTZqBqPUtlmnabB9H3Pq7iyw IxEBLj1ryPJkB60s29A9upthZZnzFymP7Mx2q/p18YUWa1twEO3+ozdUW6xvEj3UGH8kAI4r mKPfSBwc5yABEYYHmboTE+LJXscfl/XHyOdxfondHFeb6s1fJCQvf0ChQUH+ux52yooBqocR zoRzsp/nf5ftj5RC+1mzr/y7T7Jz9cVm6a53duPpsApFu59Q9nUYLJt3/F5lFze6zNb/zy3l /0v4eKQ35xdCtoiKlVwLfCtSSEjcwLy1CFogBNkpLQtBX+UUQgJXq6Heficx9Y7LvPE4s6Q/ 9fy/n5UDTGMIaB8Ct3cVVGnZDTg2ECqAKCLiNQKIwJFvAqKfrxfJYpL5rEUqAFx/WRaOlqk9 xYqwkkVkWubavbWWlFrWU0pVn+t1r8/5rtpS2lpY1RqSrM1rIgRCRCIgwMRATzd3S2RMR36F AD1UUVwQQcyKHbwPtxAfJHuDtj3hY7M/QPwj3nHt/m+uZ0nU5fOvoxKsqM3G+jqC71c/3R8N DktfY/n5mnG36WyGjxkDNaIebOW32fOH3DNjLceNIDLBn5KDC/dtRXBzORj2djGv39wy+QGE +XVdX1szb9tuepbJbadBn1a9wLtZPt83E2a1aindbK2uenlL3d+heZi+9XL+9fNduslRudjv 3egd6v93bp03k+I7yCmHwXofl8aH/c855mbpR82QkK/h5XlWCp58OJJX3y9j6bj503mI+wmN 3MsfMvOutqX69eFearr03r8HtcjbcmryHu4qL7/s5xd4dJoxH7tc40vq/8TuzxXVS3P8yxOd LhPn7Ww5fqRrjfen6+nkz37zr004vnLbnSb/FW7iB9+eQ9YuF+bGiYfcJpJjAX15di90+T6n sj2BeABBlLnFXNlAYxjJjAUuqAXhMAwAMDPofO9yF40mMEPhUOm0bKzlsyszgv9qfQ3RMb1r dlje5d6XWZD+JX4utdrTrtPn+1+HIue/29s+Hj/9Wu1wLxo5ksVufUwk02rPsbTPqXAXBuKO YI4KyF7qm79VZBBUVlKVNVjle7Xa0thOh6ctvbOMyxYAwIZkaLX4CLNotcrfxTXKKuXMboBa KWiBIhmm66XM/Y1vU+z2OxPbG08D4nKf+TM4RzP7nebIv/J/2ur7PSWWvzr8srbW8+m1cQNG GBgYG+Q4AOQBiBjGmhIYP4uGoDBTYTv4cb11sxnteiF2Ort/a9sfN34b5/MYsyiLZqhUttuc loglb7PrkGNzqyikxOdGNMjaWWk0Wvwd12jXjsqNpSjaZtFFKrKxmmkrJBu66KA113S0VWkC GiYWMaxV3XVGyaopTYylFaIkjZSxpI2ZYLTU0U/+vxG+d+f/QfhvtPr/N+7/Rfp/i+ZfFGjB P0Xyeby0NsbF+0M1OIWCZsmxsXamV26oiixClJo0UaNo2GbRms1G6LVXcbF2q0W/9O3g/StX +PNvFRdZXW0o6pu2W3WXMmu1/G+eQQlvHSKNE+/puxWMaJVlBUaPenUpNB3dIGpJKkSijRFS SaS08urs67qxY7LqbXTXNf7DcUjWUisp95lVR5L+Xm/2zXO8ewMBxzyBgMwmIKccRzIRC1DU jEYoxUyK9d0V/IThoorGYpbEWjRVMsV81XV82l0SVoikN13Y2LQWiVia7dcmjJqNixtBkiK5 a5WLUSy/v2txLaKxtBqI3qbdkERko2hLRs01kqvU1uJqI39m3S1CqbYpM0ypsXnbqLIymWTG sYMlRX1W3k28UUUURBQUbdSuag0bJCaoqQplRvunbNtI/w+q6ySNGU2kn67dw2TKYIybfyXd FQfitpVc2aESa/F3brb8FbVlyzLJY22wt7BtZSbVAoqJxqJPqktJEOQjAAW3JJmxGNko1Pvb ubQG/5pW6Fk3xLibGCRKNFZDYKjRorFYsakxsSViZX6mW7GmqKMJRGpMFRUb4lc2ZjL7nq9l eCkybQUWLFjaKSNGsUViLG3zmrn4RzEYsRoSpKxksbP1DimWKNGospQY20VKbUmN/uNbhJig xUZ+bcUFJG0msYtFqKKNjBmJrEkUkVX8Td2DZmMbJrSaLaNRsVFIzYjFYIksRFSVjbRoNGxa IQsJgjaMkm0WxkMbEURsZmtksliLUFi0VEjDRoMbSVM0bJMoqNpI2NoskBYTaS0R8LdIrGtJ aII0lMCjZApMkai2KTFoooiv0fdtdHBiAIwBjGx+fHaHEGBH9UgvIvQ5lPJ6vgenhp8U8DH5 G+iwXWqf8vW+DxjbZfwsdsMRa+iNRx8/rc//PbF9oxmMFgb1V8CbF5zdw/nnSejS+G3WjT9f F4zJZvp4vHbNaU9Xzf3m9t63/M2NNhmPt4XEXLqabD7j1tYxwVzz923fqey+zX4cC34yi8mo 0/KbN4c7UfTdYuP6mIu+9+Px/XKbHo5W9d5ky1vmnT2vlkpHM5tT+vr5aqt3/vR/lipbN8lz Yx08fucefSPjhrMjeIFEFhtBc10j83D+DTfWMuOXwrLk9KNYasBjGBFSnOYFBCFVyNUBYr0f zt1/Rq4gPaOYn88kIUGRRT18N1ttWbtG1FFsbFFFRsUmSiA1FoqCGJGsVTTbFGsRitiNYo2N FjRFYska0WxoxrQyK0atFaKxWsaoqLbJoxajaDGizIsWIi2ya2iqjUYrRY1FBtg0asRGjRot RBEUQYojFJRWILZKisWSqCTJMktGDSVGpLbGNsWwG0lGsaMbajZNVk0ao0ZNRpDSG0WT8j+h 7wBo1fjJck0aqIooqwQYsbUUY2to22ry9ttq32X9t0+3nM/Djh5LSqTnkf98tojyn72t8/xs rl2fk1+YuXq7Hsyva1++uH56jP2mc6LyvS9259V8A9n/I3Tc+GflfMPcj0iRKUUilMPgY/u8 L2/e5F/jJqe61mTabW+NKRJpQ0EHt9mJgVNxuLX3FZTjbumGT4UrpjKntekvUSC9qhiEgcJM Tg+DLeM0x+tjYBKDZq9q2IYETtg8e411+XqYB0HZjBtAYFIAkwkgSCboP0llTNgButT+m0t4 /LjfcKazrunmZrmWkctqDCW2+Yq0jbyWMRA5hxGJwYdYJJkEJBCDI0lyJLX/YMXoOW41/wT7 1Fcp0mfj8vuf3ZbPtPDn8fnbb56UpL1HHzcxK5P9HzjwspOUkzck5j3DQD6UkiUNVftfErVM KeKj1BGFNI4K/OBOLP4N+eUMef1PMZT9zluLx6n6H5xpOr5FOiijrEU6dXs/sftbbajMATR4 f2/yP2s34PO/ifoptsqjliEgokgMI/UgoNQ+3VCASAsgBuaoFT6+i+x5M8BHACiZD+BWx6JE woBox/86bR6344ZitQEsReN44D7Ae8OTQKIKoPwzz9KoPsh99+ifTP1vN+g+TuT/mbg/o67O 99xZr/F7c8E3es8yK3GWH+8sUnP5HIs9k7+e7mSyqWjk5nJbHKYbGS69dcPW5zBzX7Jm6TH2 aPcMG4r/m8jdt27fc9x7cLlOudsX5W+99z4Po4/b12X2v7UHQybriZigs2Mqc5r/m3s3yO69 1l1eN1Mag7jb4f+uyrwXbuJKWf/OD+2RHFGQsqlzRYhmL1lgDDuSBjGMAbDGCBwyLjEmkDAj pAAiQa1aUZUf4qMQMrqv/NH9iW0+AsWDvOksdJi+/Sdzk72e4VpkvpFBB4f27DO8jkitbp9Z /esVqsOZ+LqdMeGbp+pwvQgUg7S9XsQU5qsygp2aosHdxMk/frEe9Lfd2kYU+3wk08P9oeBH 9v9B80XH2UCGBzUaLpTWIJzFSoEMz/hW2CYnR0uqMD6ug0gtg2Q36fDt+rzY8EYdi0Y07mw8 E8uYAouY1bP1eV3uHuq48hcH4b6MvxrtscR7vveelt3wAD4gAQmjvyFoBAD+Z5RAOT1B60Rg D+a3HozHd5fOzMlgBxuxnwzZuDeUjABCIYOEEXSOO2KFI21yG4U1hBMhzftOZBAbMgN6Iltt KQKxmihH3XBXyu0+xYDGMvC8KpjcIDHSMF6SPXMDBdcWjngGAA0SzMM5zGBY67we22bEp2Gg SJCrdc3F1u4VaEp+Z2y5O+AmDcvVB/YSbEsqGuNQl+DZd5jVkKjR/96DfXfWFQ3hkDpr2/24 RNuMbiEkeb5HQ/lu9ZMMX4ugpi35q+HPHjww65dWn9CB6HP/z7YxpT+fpHTunWZONE0ina+F Z+V3rHTLAl6xbGG/TvH87be7tWKZeWY9mUz7yUq+Scv3A59/hpA5VW/g4xnOufiYY+VXm/kd fS5fdTTls0jSKVc1n+3odExt2AZMAvZhVcNoGrJwi6th8AcjpDTdVz7G9tQ3A/Aai7j6dMKj giw2cVGb5HwC/C++f/Y5G6Gm84eHKDp6oWPEjLL7dnPh3nk2ddrmRwNZodUu/P29zju37MTE 7Jq0p2rRozqZnbyXEd2mlcf306LTP85xNEyrnj1nAodHoMX8FFg7T8MW+8O0UNNvfWlUkfB6 0y/8xtxcNQYi+5DeoneLbxv7/LfdtJETrLBAe8QBjAYIIPxdVOtdhzVU+VpLv0OR5950PguH XvD6ufbodLvF83/Py/i2cw/bwY7Yq/56Eh0B7GDwd9heP9/dr6g3nWqXm4tImgMR2bFjMV5R vu/NOrnVd7Y7uLff233XvWBs0e2vvUIEQMzKxj0K/ENf3/9GU1PN8edBjyH3wME1KEr175Au qHt5ekPJ18uCgvZfApB//uIiY000qKP73u+3wZcPLngHVHRENI4gug21fzW7z2cZXFphMFQ8 TdeXOsnk3J8rqWPY8HM3vU7fz95ltt4OX8t4y7Wm6HA13l59u6GS7zKDRRep9crOvaA8/0mu 47HHd3HavJP1q9Ofv+v7YlZX0fY59t9TJ/rCw338H4ay/Pei/svs7rE+9iT/JvWfzF5g2nYv R2AwIwPFI2R/Y4SXj0UAxu0K5wGSg+7fQKyxfB8nwV3yV6633UeHxugd3G8Xdpcxmx6HFpbX nLgG6WFZj0TsOviZzdSw5k0ML1gk+DtRLyrc38HP9dqcf4f9Xh9Jp9nl0jPaDQeeEpKhFZD2 EPpq2+X4vt9Pzns7uPN39Yx5AZ0Qk7e1hgsgeoDUQkKKh1/uab+BSGqQPxgAj/UYyk/GQg3k 9dAZgRgY5AZ/hlxwNz+YNgBtRJjU6W40mb53ufv/6xdTtV+36H9L/Sdn84fdutJnLjj+z3bV osZRXS4S+PeQbbjLdfLAUr2s7ROafw+Dx37O8vbt9F+vzWuk+bH37I7SF1elM439ulf8Y/+L u8Wv+78nv7z5rmMsKC0A1wABjC8mwRACaOREc4MEA1kC767AhIRklqqUra+QhJQ5TW+dj+h8 f8zz9Xreqtv9hjveY8aly38bq7Z/+IW1osVk+YXxW0a788z8H8y910G7uLKXqhMzLqfpNjzl VTBZgNoIy/UguxjVsdzPovsKXkLhx6MfVf8/Htkt5sLacRfoeE2hvCFlzuOhaZQiK9+dITPq SbrGereMLKCq0BjG+LAX3jUSZjGM9WuSXRvYfYc2uXflgkkzJmNn1422cCZmCYMNCt5pBgPM YYcLnQICbNJ/uSGBDrt/E2SDFmGyDGO9OAr/45lEl7gAaF7Cty3qfoDyoXCAxS4g1WoQbUII HMuEjgR1AK4f0+6G3gccTj99foXQP+Io7xT/BynGOvfskNH/01hglW2nPOl/0/73jGdq0wbf I/7148DY9cDzNi0dCbgYt5oYEfzeJwJfgYn8wyvUp+3e/zMjaf6u83Jjrw7VB67tsFuf70o1 7aOa0ds8FmK/6xWy9AXjswMIiv+MVDjoAGREeag29oCjyAT7gkNAO+ZZa83EkD04u4ZwD/2D gA8WXAGySbW5A6woxqrZ6fFzLp1x41js34YayYi5LXDCWKwXLAWzO9TRR6P+NLNZ2Qttfx7H cZu6vvMl9Pu89+M5tm2vp/S/9vvXxdx5eT4O/p77ztgugjrcj6up+5nzvdz8I0jvgfPiek29 qZIFGIiUJYogVCo0WK0ajG2LRRrRtaS0Y0zaNjWxqiCsiEirJGA/4+NYsLIins7/s9JnZ+fn /9/Oexy/6/d8d6HuvL+tZ8baXE1bBF9ntdrRSdP/6l0CekwfxWz3NDgrrEvrk5l2IGlQJoH7 nG3GC11BW6/nGS5ZZrOuGhmwN+hgQOAOORkIGDMYw7ng2A2Vs1t7wjtjkTGd65kAK9l8voiO kZUEAMPsC9oA1nXArdoROD9xu7PpvWeaqXacLycb7f167G3+WytNTya/+UvsP8tUvF9TiS+g o+JdfxgWLuZjYa7G0FX2u1UWqaf9237qzZTx4z59T8XBxvv+78f6Xj0c91pH96zF7jyscRJV 65vODzPTtd57V61EnfWUH4br9G01Wd13sQfk79h3Wl728q8fcLZ6uru/6UN5862Sma8jnHfZ bM/V6T+5zl1vxiu8sck+RGOIxEChkAPiI0uOCcX7n/w/V+t71je1GfxX+LVH+Q457btaGNnB I+I7QgDItAZiYXshjwWuMcEhAODENAnWO8Dx57b6LvfF0dLD5P3xAGxGMRAzom5MLL2rMlxV rqolo/5P+B9WjobnRIrIgYmAMXLhnCAYI/VKWpb2vmPRf0n0u15zEJi4HBgh/Ew+7IcoVwix 7aJhOS/Wf+Pj9G88i/W3i7zvhV69cImmM/0uq7SoUM0ppEmaX5J0abDYmUaCStu0vl/7638e 9Xt7RS037le+128kFBu/D97a3wrXgRjAbjDsaK1/Z/WOAYwbwCwby2DqYElYPiS+fOTpqRNg 25vmhlG450aG2MdcI1G2lJENqeP53zvrymkBrbOgMaajHXkQCENigmhjkhnEDFidmoCJxNlN n+s3gGLF8umF3DgkeCYwG6428iwqUYNjaWDEZMzNseNmUCtGN3Cyz0V+wfpgD96GeluNJsbI 4SOSKAxkGxepMVI+DuFGybMt3rptJDD0ucu1yvQ2r0uar13XJlORa7VV6dCGfI7zybjzVE2i GMjE2i+aG3B0eaZI3/49utaeoNNkka+vhCkwGrEOMrrb3mVQYLBlpBmjzqljBYm4xuIOy9Ks GxjbbY+IsV8qbmPb29rSvPWXOy7Kiy0jDNpvfu1deuSi3NOurpr3SxqLYjR4ub29XwvNi9vb slG3vertrxa9edqvc1kqCNJsWMARAxBecAJuzFYeeHfiMDB8P8QlsPVuMBZMKQ7Ev/SQTHQY qo6p3zez4BZDf148n2eMohxyiDIw5akObsoVcB4aKURLEeq1jBta2xJjSY2wNNEbGliDEnic K5mKlExDQ442bFFZD0njemtOFWstxt466QiILLFW02lFZQY6CkE0JjbM9aIqbbLAjNZXasXl lXLh4J2cFAyYDIsgYTSJ2H+3idnp38P0keEjA4MYvu3apOOJRuJjMSZ0JmIJCfuO3RQ7XsCo Kr13ZjRo2LKMbYyDoZ571c3sjbCy3AYMbKw0QZLUQLFFjhIKRicQOmIymFTdTrgz8EwjYrCA xhqTdhB7cNhgXzmcOuEhEzlpHLG0uGBhQgx+3aXnnvvZGtiZVijTundXwb49v6tu7/Jzy3Pv SbPvEuBGaNNriEQgNidNtcLCYvcqNXqe3dXsuhJJU91da86R2jxSvMTBl1YN2IMB63mzAxup u68O5jd1cUWVfRdfQ66vZ2IVKjAhI4xsY3FOB27eibnQ7ncZmo3O4tRW7XCG2+Zenbp8318U rxyd3K5eyyINtGozdGFsRCJStqiGNGDjGsiiYMERHEAyxGMbbEm2xVihydS7PP4OOOiGbIxZ rKVIkQxD1FAXJ2YjIYqaItj/8Y8JBP8S45LGGEHwCYXI4UshcMQDsDCRGEFd3EEnY4ubq8HP 1b+hefeXzHF2AQcgwDgZz/NLzL9DCfooIf3/p5LHEdX36Vds/tlZaJEQhCBMpTZznXd9H2LH hWtISwf1/Ylhi+zcTIVXafhyfDO9zHAJGZib9QXvvY9dacZYDce5BjbZZBqM6+nv0tQ9nf13 i3Xq6RSvfUY6z6XnUvKdsERGWxgWKDaK1HGj4GQYwaHu5RGh1kNEbUTRG9UrppFkjdrroNoG 1RitN0hYY1QZKxQabaZXCKu6wjrIMeKaZWwxmqiMoxjaYxNNOiikCRBpwwY4pYVpe5rVxNEa pK6dqvZlTJdLV5nNm3ENDE65RA2lI5A7SwoiAyPT+rMlXroWTM2915diMxTJqGTEyaS5ueZu SD7zrl7m3sCg9r13nd5P7D4/pPh89XsrBkI2xRkDBsJYoI1EmNpEZGRIETWVU/pbevLf4t8B M/cPKAJYh0qh6QqF7aKnpdr2+dUPEqF1QyVCbCKhx4AUAHGrQAZD3hL0N3P7jGXLbGXGiqGC +VUIuAKRYgn84AIlgAnAdZw5vXI4z5R2Fn6I9CGZgt2weVt+Q5TEtLz4RfgMMWTHSm1TB4Ly 8dPJKXmbyNcdedRDXRHszU3JM4h1aZEuo5TeNiZgjzenn4wr5RwcmVjSTtGawZME1ta9MVkp RrMhGzED7q3bcIhML4PbyvGjTMzJiupCgGQJA+5/X6X8D/X5R03rea6PJkILow0jdiuCwnFg VaNE/wbHwlhqN8yk/dj+9j/TOt77ljnAoYbSvLbdr4FfgAb1lp7d6qleKNpES5GiCSBaVFbR D3Ai1Alyk8XURuiTD2utEquWMfD6SLixRmrw5VGvhZ9x9f3a9+93vU5uWuurnBxJjm4+ARFY sYLBgLoYuHvI2hkbbbGiCe9hjZTbzzq9a77dvoslGTSW0VorYtUmi25wi0WiahUVxDm1SZwY rIneCen0N+Cidqf2Hj/0PoQ+8OMMtif3qk7JeONhevL/dmdDwLRUcCVrfded5r6mA6H0NLzd bfMcfjV2T7C3we3lEqTIX+r650bLSOvTzvcLKNtB5uGZX/feHOfl9WU/tpbtX7Ofu/TlajJK f37th1fx3Tl+vi794m26w/+57ajg+EfraDckAYXQkANOQCRgEEcwGy2ZjmN6w4Xe4rBl5WKM 5fMmwFg/BjedV8N75LT3PxHduY7L8NDAwGyQD/kEgcwBhsucDgwQbne2/16j/D3ZcYYISSBC CMkJMNJUBMHqvqf9dh+30PlMQpjiSCeCRAiRU9MOr6uzhsUrejM1JUyt+7Przf6623NbfpFq vzioiMYegREBTCgGE3OC9XO1zU2fr6az8a0Wms1DHQVHtWO1QsVZtt93M9ul3nwYmyYjt3mV 8Mpmtj8W7vfu5zgdSgwmy798zbTe+ext13p/Q9L06ub1//cX37Xy/Fafb4mtW4H9+hLXrQ+x 9voaKxtePaLNJ6Hy6P+vp5mT/f9bxQ77+sDlcJqdHq/Nz2Lc+gNzp8NwAfw+JyYAxjA1dyBw N4YgDn+zhaaaWwW8wn1sQY1X0Yvv7qRGA2jnRNr59kBfS2pkuyuitGfSP4hXXEx6W/+YK78G BDMZoMAX2gJIIkDPAazvbV2O09XDQgLwQDA5qc3bfekQQcKGMZoj278HrcAeUXy6i0C+f7cv cmMU50sShsexprznuRzrDZp6uUmNkLJIr/kvjHqYrkuP40Pi02O5W95DvT4XUMb5fqvc+rNM 65Re7jun0PXntHx8TL/Tv8VtpzktspyFLx5tU9R67ixfv4PXx3o20Yj+kJIuANi7KYYAhXSM jFkcJkbMAgBj+LbIHmyflc5ec/nKV7nWnk7DfQvm0+fd9jvb4nXH/hS0N4ZsNfA8xGjnfhtM YMQvIxiIi80h3wgHICYFZ3+c0yt+5ywxy8T4vxgEA2j3rUvLBX+xeJ8bCidsBhAP8SPoIPku +7o/XP6/o1HNtvw/Hqvow+yyG2b43p0XhaTrXU5DlZHdUs3C9GZsDLm/fC8uf+e7brZ/1lrx hYuf9rh6q+8LbfdltT6GrZ/TeNa4n7d+k39HF/e/U3uyGCmrZM3R/kPJyOVo12hcepr7/Ctt 53m0GAGAFvxl5OWROYRiAOgxzHOW09Xs7PG/d/HXx/xM+P0u3fPY2fRu+WsW64P84e8Up6O3 73WHt0ifi8y37JdjpRYfXkJVJIdgHk/4r6DZeQANoTA2ZMRGIEYEqBFL0rTCCQUYLVrIllYg EWhgHgF3YfU4n6/0fl+B83d5oAZUSRXO8LOsgOerEQ/UoXonXT8IkCpRRmlynsCI4hjsUhwb cEYJEDgD0C0PmpjV/h/XwZv7/Tsdx6nWnXXgwF60vktXZ5151vJwOrnNN4M3R1/zsNdtGpz8 bW/Hv8LVcD3f2/id/uV2ut9v7PLXPruHPn7rsfv6f78X4lazh5/fXKz8fYaNj6eMyv0ehgZz pfv3RfroAHYIEcuAEAXnPymz7l4/P+tZQeGX4jI2CDmo6qJ7Bjtd/uK1Xwox6mn9SRwO837P 4IrnDR8WgHBAxDz7cgZLzm2D9dZQxjFdnQsMCxRIpIDIEh2n7H0bO4ge6J7JUjEcNAf4HWF5 dgvV72CGEgeqBnSFikMBDwOt1xq3nIlkgJokGCaRRQEGDVUTwIkAiBIjIMCAnXe++aRsUQkc 7RK+9auy0/mW5EKN6MghbTkdPmxUGW16c3wRi2q8BbVnYJfehooY/6W1AUYyDs+lCzgPWKx4 oi47rr8ipVZfd9NaKgjz7YNd7z3PxHc6X5T9tTQT2pCJ2lE/6c3lcQ+n3NMJ4q5nEpW25cZD Ss4OiaY5P4NTtn3Kul+IiuclM55m3IpjVYlAk/CAg63errEdnoJyUuJwkx2TcBH7D4fhPH+h 9yU7JfP9L96R3hXd8r3NGksW30KX2POumtXNtSCEjaU1IqrF7IsDVQoCHfg2KUT4pD7u/Db7 3ursYG9YnfUGJu8y4fd9Td58wJyxHhvC1eU2Ab0OVVRzRSmgqJGCyAQjIEgSCQhSgnpyIP7o dsdb3Pr0xQ/uDPo/ZKPUhH/DvfX9Xckdh9Fmxczkx9fFcRPwyuCmt9nfU7XoYyt7H3PcaaTT YeY+LQzOe8XJ0/1UV65sx4POy9r+thwdLjZL8NNAnWXJ+/v9/N27xbr17zS9fGYqB0vh+L3d p0MBebdqY+0zs1kvdv+y9s5HpDKJ3sO28ACmuhQ4IEJEsooHdZ+GzgniF7XXCBrUjA2v8uUp uDbvJZPv42KzTE3w4N2bCG227qJZER9UZSz6YrRqjhgrzsEmrqcj8Drm5lp7l2aIWa4xA4HE UAQkR1++0SX4edPCvmgRDINPij88UiSrwGHoIhjFDZSo3w2V6Vna2DC+JGSDWhFzvhmA4op1 icgga2YCtUstYF6pmi868QTgGANM57iBYZm6fRNpJkrmb0KRIGGSUBHBwP437Gm/04/b+HDw hHNHuGzGwMcxdf4v+aUTJCDvOhIVuP7zEXXV/SVolFH9xPPEH3KkcJhpO+iWhiI/AT/kfHrs Jn90TcS5oRx/rjjf1l/FFUn2Ec4hunA9qZxQbnb4Nozte93RbMn+ZtaHMi50JecVb8chCQHe rR+CbEpPPHP3u+QudbXIb21WCoVRRQxqnrK7nXLsnd3Zcm5S7p2ScO41y5Bjdzuzrupm667M qSQZJBsJCEbaTCDUG13/fNNfwNFPojMBp/77xlbGDbHD4cMpBmI0UI3mVfk5aoNsxkAomL7m mMYVGPZYVoMijBjbabhHhSJnLCjMSf/VAEewgfN71ACkQOW8dfynuK2+tViBzlFJIVBN0dlw DCgHM0a3zVDC/9zxReNE4s9afe3tz4L6b9S0c+59PKa7u37Q9/EfbN4/NMbdWb7k0vP/jTfV 83jxl9/e+7nW/I1xftfBocxldy/1mk79d2EhmKKw4DBfC4FfNXzHIw8zFgwI4IJZ5ANwYM7A 6RVFJh1trXrVycbrhRVwuq2/PTpEQCXVZa3IMfkarMenwj6j6Pytzi19y1HPBV9vfZ7mcXEr 2lHvf29/8XAfON++ZIWJxeU+0SfUbDYhXCZ4a0bLaZ/V/f6UnML7/6VdxHe7uYz/cvtm623s cxqZPccndcf2MPcNpS1zczlbrjvLZK6ZLsQNdgbTx8T8eny2zqPqdfP8e0uHFxb3xwZW2tl9 5js8xrMnpPX8nt3veZHZwfcyVKXCP8foC8gXABdczmBVBJIBHIq4EAahv3tqt/6ybXjvORf9 d6VViujJNwYGMa6XouPg+ytypmsjovK+UymRTZerjegPNqZQzkDld+VqE8fvDgU3j6Pc+/Rk KjIZEMwHoo2BvCFNmQWaDzNiPbu/lSxrD9OTR4H3ZXd0jM/U72w4OZY3G7aHH7Py2+FWuzfs W0x+Xx268zg/H4tHv/40/z9e9VHJ+zfZmXibqu27o01J5uhe4KZ2sPwXr0P8yeLzzj6t9EjC 1mNIAgCFvMNaQxKDHMVwOYfBG7tXP8SuVXa4Gxs1zZTHfsd+1KnJwn/av++ToWDLQ5mKpxWK /dcPF5/GiUAgkDYgjGyOL6X6pD0AVbKQNigi5oD0Ea8qiZwRhPYcbzge78jLrZWtd7H03Lb2 BxgPVv3vYKe9Ht5yDvtH9Vg53q2H/cnV3vNsrHdsbyvJo/8rea+S8WrB7XHsu7jdRn/ntHQz PZ/H1FpO08LxZSxotPg0VkxO03NlyL+S41UPELWYDEi3HBHIiYeskYwQggRABIjn/naKtZI2 LHI4h2RjpbZBuWFBDpwAZjs/o2+g0fYVwkPJbPEIk9hiEczXOGBFhDc4xzbJI/iBxJpi+BGj ILp6Ehvl6X2ftbvInb98eZ8684Fxo+Jd8c7xeFtIGgCn7pd9+K4OlsdZO1njr137FRMR/t9P hQLr/Tnx2ZrqODjpr9uf0/VtHH+P3I8DAWjKf1z/55c1dNo04Vf/Hp/zitFgO910d6uaSxen 7OKw/T0+oqrDJDw28kWkPRZgL+YjIBA5GMY1NDrjvvZdfia5cMJx143HrSuAvJ/0RE9LjKbD gpvQ+vwIGFdIG/nonvBpkC0mNcqbnCyL5XTX2/cXudwdf9Wmd3d48fiz/I9nu8D8fZr1uod9 RPpW8YTd2/7qz8nXOz/+Wrq2Glznxe1oPx/99K61/r93/Vm1nJSraaz4Xfy4+Lhr5dtpbd4n 4rdq1RZheQAJIAjAwHnUF76/HnrpvP/flb2FwuOC0tl/3YKZ9kudh2O8CNeSuhJgjB8vQYwM B69zbmIhphegEzGBgNnqcsD9JF6F6vuLxIPyRHH41X3ZqVsXiLmaF19/E7d5k/H8OPmPd9XZ +76mx+rq3i73e37nz9v+EnQ+3ceJysPeKrPcj0LHVZKLx9Bsc3XrffEsrfOr6Xgtdnoslm7n 4+IyRbeNv/9+rPzlGp7recwYcC4ge5bTgCdBGA+mRA0YdjoHw5+RPdyR+hy/AM47vyxazkIm XPof8urWuCJkwvF8GvAJBgoBZVwm3bVF5BArzWxUhfZgcboNr13WqNSvz6mpl65yfTc6nTPN LpeFpYPX4F972VptJhLz/s9pdjQZN1+vsTUpwf++XN+dXaq6ejpejrtbe9Bi/Z9W+8237GUQ sbZZpaW345otUgkf+1gKDsSCv5G/+3hLjznseo+efrZMJ8uiui+ZXZGx9R+mWElf6fjYDDjp Own+idVR/f31Xz1sfXWx/qCA4jrV+SjFIBX5sPZnne9OEXbUfivVQb7sV15gC4o8l5+35sMG Mh4fljeS+lyUGfDQup+ATBB74whTgEDDhiJ1BctiEY59N3hwoIQhgII0hPZf7YTuyCYpgSEZ KKxQIBITrN2foIao/jHvTtUr/UueDjESTsEuhMAvSBjMQQNvxrM9xnWoyn03j8Pn8F7hfH3/ x+K66TE8vC8OyzOZ/lxP87vYK3YrDdLU6TtWX3Zntl2PmvGVtY+9Hdu12IbchmMKcGBmg8NT W1gmcEgBMgCBZ+tkRrB6KgwKoRbQPO4YuOb0mhY+Zx9B4cJpb1M36mrsfo0Gf0fXpvHiOrQ/ J4tL4Pd4esfRrT19Levq3mt789hfMHYoxRaMqIgRGIjADbISAyU8cTwGRMitAC12DcU/gGrg lADZrbqqra6HpRYeX1u5/LmfndeDhLHa+VlEYT+Z31dbK86suHbsPCW/ZNj/fx/x8+Fa3sRq MUhpgEQcgwOYA1ILyQMD30wBwKVQJGAQDhAzWUrYqjK04nQFRTCgpaU9vlpvvXjY2Fyq737P 28fde9tbvy8DL3hp9T7zq/C/v/t9ppuuceHx56v5z4sXrsz2K58Nyq9R4MpJed4MVn7B4sTC YU163pVIEqMYc5GBrARgKajOZBERILcIHeIwNQAnpcBWA5qrLijfOjXq6fZdNxA/EnC6aKXv zzsDp9fp/mB7oIiIERh4xIepqL9CG2ruIw8Ts/fQ9T7vtvf0SM7jm39TuCz/67zsxdDlnew5 v17XVaL1stq997nmaTgWON/3kj4Pj3y/ezn5wfzYBY8K+ODGEQkqsqwI+JBrLZkkGBvKQsgy VZWEBZjEPgFpCi7E9QWYaYXvEb72vOkL7C1Na1Obr9u5/oMMrMaT48v19fcvJ7b8d7xGlRFB poEYjggCOAAU1WHA+sEY1vlxgiBXlARNkAUx7hC3BBrjMgPxxfhddD+NFG+jo/+/bmcy3X/k DuFBEgUWr3guPsVR847n8eczD4fpmGXWZafp1lB70kRyQ8DJJYvq4AZEhJKhSODZhc4HzWCS +TFa78u36fuzreq2H6rOMhO7LRNncTIsn5BIjd//iuNgDZRDoPuST/b1duZ+rhm6+I2Hgn5o GXj8++1pG0ET+Mz1EICIagK9srcgoXhMENq/MtbxqtElbVi1tG22skjAgoHjIALUVF49HoVY EQ+FgN7Uios5IoBUvuDyh/OY5ODR9o+YXYhEVzDM81sFyjcxRQLoABfBFHO/pOM8zVeJeCLn ABySKVAm9QFRkUkUJEFAB4uKoijh1LAph/DOrIQ6P+E+0f2Hzjhnwj9sgFB2R/YtQcipDwLC 1DIi7jQB2P+HwilDsYITY88TgnB5RmhqBKDQiRGgFbFbE0PAKoUgqh9wvwrB7Y/EfGPSGsFk GJHZHRF9HkPkn9h+Se9PgH7Z+gf8nrztT6J/ye1NjpAFOVVCKKIQVCAiHrgARFD1Q+cz1PMF VcxnkyGgEgfS0WFz9TRouIg6mAAuaIhczCx8iAiDkiKNoAGSAqAdlq0oBcR3udSInDg4mK5I iSKvU3UFZhRYhVqLWo8hdIFUdOmEgtlf1SLQliH3T5p9A/0OpMQyInxaxaQ8FpFQLWJAWEYo dEWYfcOMHw1AfjUiK8FmEoLMJYcpxh4ztncOkgkIR7pmZlx+oQuB9Qof98Q8wZHs4KbCrUIT BGMbrLYP3Oldtr8ulo4EwGIlAyYpslWLBeyZMmJgN4OUrlLKi08pOUmwuqlHKcWdg5SHIXJB vGCKgF19EhGHojAlILZClNFKIoIcELIcyKDhkIxMpGRkPkIMhNkZGRkXyEBiLExHMTEdjAwM DAwMDArgLAwHwE2AmwMDAwMDAwHGMDAwLmDDDYGA7YCZzIA6SAaSIOkgKaTSUAaSCoGk0mks GkgP2JqNdGsjrIqus1lCBhBQDWQB+xESxFGopsLFAbCImwg7CCXZTKZTKXjlIgAYIqZSIVkI bd4RxsOAPzSLSHoM8zFkAkKCdQX5vA2sOB4McMGeAPXPT8JqDlMc5uytgWX9St32flMzf4T3 Lg8RA6YizpW4YFaQn2j0teP3JaawwKxvPme34dZkMEJFiApmDZICCG1GTe+MDeGTUGIM9nuJ SEQTNdmLTMiYZF+edRgNijQehouYnIEynZEghQnNIpCJiy+cf6JZfK3WkWSEO6ermMDuINFC XwXwm9e+/NVwa2FSXsLbBkthqM3OdgOc/ILlg9xU+XvwnuQJO/2pDE3T5WfeId96o+FWGz7F g8joIk7cZ5xL3skHoL1vXbxafqBolO8Hrzyw9z3/ECA9tPSgeJJ4Sqno36bpd7cXty3UMqwS uLUGfpXEa2aOWeA0cAXY+xlVM7BMx9+d1iowtlADsMe6jXGV4ahNK2c8ZTao6DMxEajTbAmO b2QLybOEjsa0uTuOlSKNhkS5ph4T0MQd6D4UiHQPjOGDBlHEYW0Yj9AC5hfT74d6Q2+iDRFy umX4xUas4MZ4bg2NEkCSRJYqvHssbfB9ajXdykN+ipDyRpzPRBIXhMhzhPwQasIqtQiCB29F ui5IzZpClHMJsh7lbhPRFQreYLqJIEZlAy9+g5c6ciJgkmA34UoNlNVL2WQkFvjapnwsNo0U YBDbvmEgjhFxNOZYC0VzM2h0RePZNEIhhuBU0HRoOGjbUZciCpgEgEhMAE4mwmqLe5js8h2c 8g2YrWiiuR9pVWEUQZBUMhGiCCEgCsihitqLahhot8pVt8PPPLF7r4oSEr8Pjep2Nk3NXNdu /zeHQYY9wB597sN37nXcd3ke0mA4geNM3tCZ0MR5+h2PZ44cOw8FNiWvGZ3lbdr8OvzvhjRq TVfI6+X3yQnz+6RIKow0+h2pvBy9ko0QcZTpKtPsOThcbtotmNiL2iBaT7RWEtCQ9AmKQXqi BRHqi0bxiBHmyNCksJG9hoCreE9AaY5YmJki3ueYnDS63ph3OMXrZIJqVPoIzpRi9HmKMwOw d2Zy8mY9SRjMUaErYTw5H54KaioTW8Wwbme9iFqxb79yyBi5Gqwl0M2JsbWOBVXSDjm5UezK lWW5X9QrYhjOLM2r1L39zmdHQmYrCbYFsyR9pw1BwuEyE2uwhATjHx448cCgi4JajNiX4Od9 pWoigDGxoTGzlIWmhgbHhHYdMIatxqStQKEwcmiCbXvlRj3Z2oxydmT4FTmwA9uDuow9oDrE NNr1o085UkML4Ws50ZOk7d+2eTekkrB4iD3IDIKT0olSAdgFfY5zvXCnwJzd7wYGCYTpUqlZ AhEJGQixjSbQ0xNjUAhvsXwPf/XfXmSWVy/MHhjfy7fT4sx6kK9ajm2nkNZxW1C8bsqZmddb IvGLq0FQEisjIsQTPOY0eHHV4e0wGEbQeT73XY91wkCSbi0tGdrJUMfJgaDJgIYEEecJUXND Bi/ULzeiej6XpWyTzZmO3K9WY08ebnS+VXrN4ilp2VqPOkrKFUEhnVwJMYmJAusn08KimFHO brY3Wr2rsoyJUhxFDIFNOuApJTy7MJp8dUkGjnxh5whoQU45iXPqwz2YE16m8OwPhbpkj/ST hn7z7HdM5XD9gRhrJOCox4GZngqVcpJ2hN4m1fREkF0UK7jfzFaBVD4pyrEhiUGf6jj3fj/F pKU7SP2qDysNfoSPc3h+iTzT5C1OJLReudM2hAuwjDGj0MT2s/Xh7jUkan0/mREJgESIibHL Joe2z2fifJMO3F9j2SOQkNaXi5SF0EdixJtsUIe0dO4wNG4oPugmyNNht8I0GEGaGA2L6p4u IAdGDeqeN4PAWlyoONg43jcuPPBKghNtD9KqWxSyBIlEhrXrjtQgMIqwIvLEwQCxOZKIMVL4 tgJZYUqxgjcspYLIkQ3jIstEAhNuQTIcwh2G8iLiET3wOa7882mZmO+LqxF4R74ylWefxrBl evPje+uCCbjDyYrOKCnHqALheokdOPrdl43fLZcoirozgQNM2iwrNhTF11ypUfrXPV0mV944 hEG0k5Fnt8GU8+azhV99GuU0Gp74FsRSjc93TLCQRORMnjJwdnQqPCeC1nBg489CnC1aodsC HWOMiy0JNy31LXXc54cO3sIjub0XcCjEw7UaSEhl7NTQNGmOWTfgT7c72a6GuUiTMwL4c6rW hE1hop4DC7pqEDtDWQEiSCzAOS9QJDOIbIpi7UlU/TLCW1DqTXAl0KJrKzto2GJwXdM3Zg2C sYWKRPARSbUAIJGQFJLHCxWbl5kOho9UXIL2jSzUjSOVqKhzvTFtM6gPPEpczc46DRHE6QzC 20+Ydq1z1AQhwoJ5nrNXe+BULHOqGtUMgA+mqGsoALgBEIuJiKhyYKh5jaqGYAfGAGwv56Mj wE+EBLMicCAcBx5+bt0Tm+pX6iCcxzGRlblVD2ywqHMqGJtMdBNms6evd1SipwwhPKD4IBIg ajQaWLvKh4N6/Bu7Je5eqom/O0WzVC15a6odswVDiADg7YtjLJSLwbjwqsQtCy73JXFOTC+N HtRVOdcz41Q6VQ4kE5WUkaI1OZmuAG1mQASypyOWvx6Mw6qYnba9uIqsl9mKh7IgaFpUNqwo QOTm3FVCEJt1YbSHYVDZs6UE4EExPb8W3L6Ok7/hA48aqqkkkJJM8XeC9sfrHwdeh0Wg8Jne kgwgj/TX3Y5nsyz21Q7IW4A9BmYPdcp7l0sy3YHwvU31NSnEcnw+JQrgPTaG1JkudOkhKfrx CSQklskhEQyY++RtiNmIgefhUV7tb8SD378p/EaPdGj1se2Ip9AOS7HQOyvPX4yjShdBdQ34 nCVDdqh6Kob5DQqEQTgQSxxACxZ41Q7O7nKKJOw+Il9SCYoJmh49NMkDf3xPkgcAaefzHdVC 5e4sXrVDSewqG1BLACX6cFQrAlIYuR1DKJ2f/VbB3XdZ+08lcYOfI6ZNA0cSCdZG5SE5Dx1+ jcYPV9F6czNP0/S5NyJc9BvTkQo2jbjlVDUcCd9BPXOgATxYYBpIckEIqx77IQ1jqO4AHSay Kh2gAwWGtBLKhsVClQxW6oYKhr7Nj7CobFQ8SoWEDAu3KMSCYzAqFlQsUqGBFIZFQy2cXYOb hwfDMTe3od1BNVcVVW1UMrqhtVDtNjAxWtJsDE8gx11o0pzMLuTgLhymQkSMImjRosgnhFQi nnMTLPWqHEglZgBrxbb5r9mjt1bjdxjjwnCqEIb0JGRIQSQCQo4WgKhgcEmnWdrioYDBNToc i4QOHpGmbzSb4CbmuaQ265qoZyoZDmzdFlQ2pzRhVCy4qgaOPqAE4bcZoVDWWtvZmMSRkYwd 8U4OiHDYThOfxETo93QZnuV5N2rx1vSb9V3EE5FQrSfOIY9OWFuVdYAaClQ7x2VQwQToIZ8+ Y8LxTm4L7iJB+n7QUJ/IHymfn+nOTE7G7wHbKMzsW6o2xwH7avBTpeqSSSY7Pn+b0W4fRGGW OUZN7u0LGfuAqicOHVg8YLBMpUqyYaoxkBUrhOkWIIKoiRaeN5Q47pNq9MCWJd2a2FaSnBiz 2hBRhqnOrfpeZUpu/MM3rNdZG2dDReTuulz1LC/jRrcrPCtAeZcKj3pXWNYPUADzs23nkco5 NLZZHLa7o5aERKdrFmiUCGoUnusUMLvQaxaJgDEj5Jg/rdVlG6Gz0+fdQFATGM7v3zPPCDh2 A/18yu97EO45rDsy3PLZSmnuPCPlgMMexLstp58K5s4sGkkDpsxJBYXDxzuL21HnUmokIiNp MHzh2XAXbzUrkFgpQGZ0cxIlSdq6IvHEnsjqI7SpUV6FjWfv1Ji3cTvmupuI5wqcaxCB0THd PnK8s9k4RCb15ZeL5JU9fjl5d53iEZ8+iGGNjMwXNhzS1f29PLLqvfNUOXpJGAzZHVMnyYPk aAkkz/qjuYd7PxI4s2hlp3gQcYeQmI3JeFMmnAmrtG92Byblgjn52EV7V0YniNvX7uhj5Fs4 Z4ZdpYy8GfTw3MNvbg0Aguzl3DjRSEGCsYNByhCQqqiDgePoPbyqYCQtThug6O/HS7vyKkmB MaHs9VCW2oYi3ZSgNwnQ5GIQ0ubKdwGNEYy0gy7PpvLIeMVDsIJvvYCdFuXj4qMzZDjrgOmM hxvBawWkkqqlEZBkRhKilE6lQDDXyZ8vBy7OS07Gj2DkZEmAYqp6fTFa0zMLnDgmYZas2BxN ltOPnfCx6wW3TkKPdJjNNWuWPmfThXEbupmyz+ButSoYggtZ8N7JK49D0TzH1J5TQ9Zmx5/Q bv4YE9oHM3Db3LtLnbvnDPiN3SnH3unbw1aaJdO3aiKwQMzBtY6sSjMwewezU3C9QMt2Aa7z BmYPGU9qbtKwbh+m/m9PE7P1vP4zTsh4Ngd35ippSCfqHPcm7jOPk7PHs2y2Xl7FHUqOvLub Te0E9Yy70nqaeTE7WrWXjggl3PTkJxLigj0t3kPTKBhv3uXxwBd8QWMDj7t9Qi0X6OoYCne1 OnkCOoKGIYdlTdKRUY5kDaMCEgBDMJkhJC5/H8LpsGxuPt66Ejn8sAjobOJMNqKNSHlwOO6e Yr8r64Yezz6tz13568zI3ZWtdq8z9RbM1jnchj3DdyW38NWx3nSbd/ToxDJUXDbqTARu3jjx YzT5tH0fnvJVMaYHiYGLw8FaQzMEtFwrhmVzNeEcT5DmpT1yD0fVwJvkcThsOZfD69TjdmDq TUg9gwMui7/Uw7vGqSEkJIp9h1KUQZJJJJJJJJJJI7uzu47uOtmQ/n3yzeQTW+NnSOdZwslw t1su6sJTFbmM9dLtHT1B2Qjo55D5ZaRBmd+ZC25Xt2d/OfWbRgwb2+rLaj2zt8RSk6JuiHZ3 c2NVM3FsMwllKRNclI1eRvds1xRgTWkeHi8zDt8bA0mBn0KHRppT3mDBggpGbaAlCxGEhHto C5hw/M4sE3UrVODt5vqqHvvhWbqz75uOubsZhnPOcMDCSJvwW3wy9GrMwVIUF2Zg8ljn5dsV a+ojI6mkHix7N++dze8+88iPBwAWCpxSpdgEWxdzttvR0gbnMzCiDbQTOyiBwTuylEm9700P Zt6abHl4t+x5c9NdnPvls8xzC5xMeFjEsi5xu4lLlMIRGBP25PL0KQdCdy1n72jWafXOlpzN 96yZ1Tc3QdjMXZjHm5ZbFUjmOB583Z3JlclGeXffUhhch2CASYEx2tXfw3Y4lWaxKx7SebMz BwzanHUOGAbtuw4eA3hecrZ7XYsYTxJ9cjfU2FUcq8tMa5Cyrx47GZg5rBQRtbHUqE8t5wPB jNtljZv5Fh6ZYHnk4/ccmtiY7cZGslRNghJJIqzMF5r01qt250ZP6x8K+hzm40qTRJmYIObj 5Dqjmc0ocMSDfxXGlDTDp46SqNZF0xJuc2cdbpV15q9G/oZmDRgY4ljLym845AamojPQjKmG fFbFJpcJmuz34uRmIHdr8NOezaipPwWICSAl8E91IuIZmCW3zkbktiGc6QNo41hASlvo9+iR rY58PhJxXJcCFEkGdxqqJkeYmVJA9QJTk/McaYFaYX57xzzKCbh78H6JYSadqM0zFO/R1mkb /EiNnrjjXL8x4cpywTNG1o4xi6M+T8jTbgab3oHn721sEHBXWRpfNu8YmQeD4u47Ea4NtZ2Z gxGvtam9mYFwLP1S2m3dlLq25lyTXOeF0mrcz52xQja7rOKsewWkVJDI2YtW3QdDbzLwbqX9 irHkkxjPLeebzPC3Mxvk0Ek00c/JYY0ffju3cSO7vr5KTjMwDktHeK7m4EgsTkiCn6kwMzFm YPEXbr2YeAt7PW1dqETINPHZDFNUPWu/TGWo18SqXRioJMzTMR8L80jq47c76VQyVDLPeOOa ohQ6i0j0nObrl5ETeOcScrLD9VobqFO0bW4bWrB3dsXyfYn6347L5aA3ApylqymNgrPVc1GL R1XnyDeFDjK8VEAw+rWKR8CHpvdtfFcJaNsQWIcYJNr5OfezMG8yO9fo0SMsb2nczM3KCdOW dGAjYC9NgIIxIxiZqzsgpDsvV28udZO7u4ydwNexmYKnjNOfQ6c9vWOOSSSSSSSSQkh72W5I o45IJRxkkjjJJJIyMjjCSKSSSSSSSSSSJSQUkIyMJIRxxxxxkkjjkkkkhJJJJJJJJJJJISSS SSfVFmSW24ZZswxJMzY2EEzuaRnWpN6Lfitebyd/uVOfabOGhicZkpG0iIILQwTq5RGw6KL3 ycGXZgVObh3cTINrukMk6BO6TJI363I2EdJoXmxYTMVNwc6bSR1trc6CYZd6sDvSzV2w/zp1 Xpfa/jkxdTu8+kknfsgu1nmOROAOlqKmQ8XgbbbbbckpQkOMDvwZJFoCYVxnZ59ZmcFzm1F3 XppqJpfaODgO7fnjzOs3sj1SWhiqGCobB9M0dy/Z0CXi0Dbu7UT06Dcbe2fgl3Iw9FgvLJb/ Ur0d7w9+eLAdrLj4dMwN3OUkjRO4NwUsRVsxBhCEUuRTYNwoLF1pCwUlSZpfyYmT1U0qqBdK sJN4FHi3dRUuJvD1LwuaSvyfaRhPs75sjt4P19LiO+e6MA3Zm8TQU7hhwWObNidLnzhNG/Vm sxN2lh180jnL6xq477DoM9A2QLISpfllPczMCK4NI2Ks+amZupSUiesuckSYhmYHZmDOt0J/ CF3yywrK1aVQsqFprO/mqGk2ioYPFp34aK2EUosXxzzSoSnlIwye+WUTSWsV1mrOtMDEzL6E X0yvpjA8s74TismiFJ6lmxiInJ7xe05E5RShW+CuCJ+GnThuXAyH1CA4kHachnaQ1fUY133D ZhZ9m7uOS7cR7v2Xa5MXAcOoTFBctKG9qkjV1I5b5zRV7MoEeN76HVB07p+a+yOapRrmkzJF hzB5iYbxgnA77hRz46MOo0vebldnRbQgZQ0xA4oKRgQkAhO1q5dDpdM+ncsrOehDnLHesK2I cEPm5AQyIR4HYckl5nGB5+74btpT00GtWNoXZkm6Oq/Twuu4cqgMdyhDctejx5mJTuoqLISL NzRtCxbBnfshjRSs54nw343q7FnwdS2kjaaX3WrfdKCiN6IW0KD08Cjcn1rgWB527vbqfBBM Jj6ngPC9cQOsmdYo1p9221CzsSQXXaAiE69EpszBYtVlK8V2ykSwz6O9zPSaaehVwiDce8le v336eveVRGti+g1rlW5VzbajWiifG7EVEYsVitpKMWKjbRtGxktbuvg8rV7o78SiAPWxDzmG OfHo7Gnh7Wfhvoy2awILIgGLUzI2S2UtFvmL6M18yvnb6kJmSFN0hiDQdutk9yprHrTEk83d hWBQPDPH8bh5PS3knG6CgJeTh4sKWd12QvADhhvoTi4fSrEBYwkAatjhZwab0rl+fj6chJSn l2sn5/1/V0n3ycf0tDJZCU5H82SBn9itur17H4aVhh+h+l434Ppj1cqpSqeBXjS64SV4nAp5 Y57nMBvS5vfF/0LAMYGg6jApjRJmMRiIGAZi+04pDSEUdRx5npUfoa3ft+H/7qsp/mda1+r/ 70dnVd/v2/LC3UM5KgFIgpc0mhJA14QYwBgkE0hOBQJidPe7vcpeauKk8mZ+FagA1kvRJkMK jEJnBUIv5BoVMCMcXvN3i7AjkMnezrkyOlmpBPO4FnhXQwYaJOa85JNzEwFTCIjPTQwM2tgQ kYCwgkteoSEF6Py7OCxwnC/x1VyFZYgwmcCSkgMQBAHIiBzG1IxBBV9fEIkzqkFVGaA8ksIA 3y2Llsnjc2/c7Ht/Bz/SwNNDOSjcUWewCDxne7FHG+k21nbYTi3bekp1779hvrXMp905pgOk 5p4M/Jq07miVySI8MSu/7PXdK3nr6vecmcvqZd7h2wd/QnJySJOHEU2YCakz9czLi2dVx6E8 sTPM7YRDxRdcVDuv9M4SwfWiAF4uEtcTl2Zv7H9+9XqvSS7tulMv0va8JCEhvtpTfG6FBBaj TKgyHavsTnzygdhugnkLzys1eshha5N5Cz5DH3nH3TO6r35Dx82Ze16V+n7D2AWoXkjHMRgY GBAhInWzmLUFonGnk/N3k8VZf7ln5IerTqR6kzOPDUeVSktqEQhCQZBhFhFUXYeyvp970/6J b99pvR4Pl4T+12ua3dkr/dgfrxPJav/evYKb2p++YWpreKu/J5ufw9aowDUIljnQcbMy2TLZ ayxWOCSPxfnfmX/C/Zv/c/pzjn4b/nz8ghjfyv0tu/vaU/d9n3/g7Oh9Ai2j+cxUYmNNt3tE 5T31mpmsEazTqGaJT7mbDCLN1G2NC00NmnX/c0XZGRZAOCKZGhyvhoumGlbOkyMDAqcsMM3p 6xikNWLXYRYMa3LdNjV3iONavGzZo3xB8GjJXGyPVEIagxjY28lkE13rbV7PbxbxXeTe3XrU b+BaO3UQMHhAYDHOYrbFWxcEWCroynD1kyAo9Eob0DoFeBCDWBBSppqHNMf+aU7u3whDmQJ2 LiuyAcRQQyTSOs7kzQ0a1wVsK962ZkIbpwYS6x8QMYPnmqfYGB9aIIEI9ejldIxM8oZ8qiUE gdSi20gwdfduHVKaKh9FtcDoNNmDO+wwhMannel4qb7UdVUuOeqM7c1jGPJwZgrrOIYZd7uQ 1DgxLirz2jLOHRj4ijDZIs5ZDCa3jHsY1rMyNeD0Ca5RyqXZOHKPq10w0YocagY9odNjCDa9 L3RtmYtmiBYqizVUUS70x1gfprvwMdh9eFYAYRwHuYYYYdFElbJjBslkXbJqlZEwRBMneAJl aNWwa888E6g1IhrvODOFDhN3b7+MeivSB7ZvY1g4xhJlo0eUyJtM9D35oOjT69Da0TTUmF8X SYhTYrK1Y00veXtvYeYtIbI6NRpLQO4Q+WCN4xMqjKz1GSGrKHgDL3Z9iH0mxmSCkQNzqkTn 4Cs7VaFTfvhangLNel/X+99v+M/fy6lUX4R8n/ZGgzEg7m/05XYYXq+N3Tfa+Th5eXOimY8K IbkUgNnDyebltk8zX6U3HLh7Get5GL7pYF5Phz0z3b9yfNOntgdX9Fz/Pu6ItL8lY6z2aylJ +Kz6XMktHfuZwdvRco5/VWhbEjLkI6tj3nXllbuPrrHLHr7uzh9h9M/PTdo2eg9p9m3oOA7m uyvZikcjHJSOa2meL548ce3GEZ3OnuTxk9V3Sw+na76CljsEkbn3SkOOTodw4f8xPU4Zj2r3 +W7udoYB27THzebPby5nE/Jpd07cgli/OPm71lKN/m0F6i9SHIf5kZQNpCKoh80dL2M9OBo+ Q1VbNjw2J3QgHhFDJwukcYbM3QiFbAAOYxA0q6mH3q2D2t98ne6/bw++mO/o8BmJux5CNJZK i4OA+Xd5YZJcLbMkYynBLHHHzCgug9D0tN5PIP7/gyvI3aByZ3tus8W3XkxOiRT9AjPgkUQw PrD37lplsxl/Yk96pwpialXGItmIRw97muH+AfGlTSYMCODEBej7ZYOuf4sYDBvWsgR/qlFI kED7BeyI4eP8f8Pj67VWNhCQkJCRgE/mgaiHx9/+LZa2s2X8ppENRFQvHVGLACQUZ+0wo5Y2 11TWX711rtlUVXjrNrfqINoElhIjGJEkJHD6vkORT+I9F9H1Xjflf14v/v8fntzw3n8Ul2pe 53X8bAc3g+1abfqJ0CSlDmMD4qdSnp8WZVIVwejs0BMgFCmRbfRN7AQDBIgYaL2RjSE/rWYm H59sLcGO6tTiumMYCeZsGYcSrzivdefz88MhB2NXczXqJsU/nKh3F2H18aP2jvfeWoDwoHCI MgUNVSz4Ojq3SZLMjsUiZrM3l2Tja1ZDCz3VuGgYnODnFbDWA/fySlr3EttZlCEVW5bexard XY8g4lZqT9S0/DjL9eKYZASgpRSDdClHxjmiaF1FkFSLuPiFUNCJsXYZMacV8Z/b01qrwp/Y qDUc4qK0QNTVo4MLFXBNiGrElWNjRIkp7Hl+bg/kNxE6kohNN27Tr7MloCm3x/rqent7FYLL YrKyMHhAVowwLyAgxjpBMjAOgQBz1VRptSInWxdvNtt6bWaxSiEAuKoJA+lZGMZSECrzfrSd G6YagEISEYbCBzEFiAlGNawQAgGN7HkuFna1mA1Xc/T+P4sczz/25vbytJZPa7thuA42KBjd +OUQOO1v+C74n69DvZ9Iw27s+JV4bw83fYN+I8Hp0h14eWW0qIUE4PU2sxwGA6/B1yvLPE3k kklxi6rAGy/FNZphq4UxmH1doWQCbTKyxzf3fhDZ9doreIWmU3jBmZ+G+qHvEX2CJIHNC00q R2Osk/H9o2Qu8b74JCoQd6G8TbUWkt58+GFwMowhA1PBtW5CrR+kFs7fLp0/yvJpNZNUhaIh GY2wfVsnnN6pZhdNElnRyiQ76JD0Ih5SxzyhX0WmD1S0vrP4i3Q6G3Ze9fqx167WVfoVasmq 7q60pK0pw7bBt5enKqIpsoORPBCa6xfhDftFA7o3TodsXoHQMKHXDHBlFgyhg/RyVHL7Sjxz B7GI3tRB5wh7O3kcZJ0Wc54U2eWJeUOpNfn5fDhvObhLvN8zZw4uPQCE7TO/CStdQ98G7omS NZ8eCi8lgBnPGKTr3Z1pd3N2GhD+f1ziEFr508LMW3Ozwa9FmrOdzFFTtqMydZ2DKZEnjHoY lSZo0WeTvrfKRJqO0h6xcMeUpTTPyc54dLLC1yYzpgsgSYa7ngjp5xhErOMO6Ke1IcJrsXYF GUZPteG5e8295tzXMca/24o6c+4BZHSceHivK94VguE3+3DKHjW+J+TUge2h3MDDKP5Qo3ZA 4ED9TtDri1zLF4UfNM8VDkclxmSDHWEVF3FguRV/qxHBsbnZ09+TUYpMVA9b+8w3oe1gXZek WjvDuXGNXlq4cARVBkmzYZMTTi9hH2smxIb5ar+9f05vy9GAs+BMnHszaR8D3j/N4MYcZMG3 zaw2yjoeQJFkSKpaKXwJHbZbkOHe9CxnJoOEltpKkjqU5pHE18F4XoQ6fz+bZLWouvpvhhNd mo4LaNn95IPd9jpJJbA44kiRCMAoP3hd3u++gSIFzCMYvqYEolajPg5zWqJsjdoliIu45BoE oCRc2SQ8YDN0Yt1sY1attIbslg5DC+ckTnSZir9PLBqbXStQ3o8RkW/hdFgwU8HeBhw/JCbw NdgH3FasbEnkFNgRokkQS5qdrHdw2UJBJwdh8792tM2WXliKJiWOtyTAHPIqcElI2IKjFWII CC0RiLFRT+PDcgDM5kmhjKvj2UoaHuhiwGPJ2Li0r2kMb127fBGCGzH+CgPcEtQRnIc4tBA8 EJyO5NNGepCAyNcHmY/XVDHpJoDkdQFMMZ9iHPAjESupZI2ryHM7HYY5sZy6/a25ZbQagGoy bzmzdDm8RSBAmQQO3dul20vDnNDabM3jJC5hDAsXdUcmIjnOCUJAMuEzRJpAbku2ge8bzGhg W0yMCQPklyo1NcTwXOYM62JB5by4w+U2NyO6ZYsw7QJho7PegptLdRnR++ADuEYwDVq6OZ08 JYLmr9FPO/Y+NTDAg5rKyp7tTJrhowDYpBECpwKoSAo874Kj3uVRse5fMLRdDlwO/887rL1e T7exwtb4KH7M/itdnup8v3P7waxmAMYejhRecHtbUqYxakHGTNU2F0ix1+uV6vpUIvQwgOLw vNZTWdgxJkvxPi3y9zblvPimHxw1BzQ2uHx1s+/QPE/1O6ZsW37Oxj+844CV1x22Xsmbbo/J m5M6/xIaBoU9rYCYJZ0YXF0obLBIEHg/oCZFm8Q/Yo3NjM1ncCii2mfcl74O8pMKTnfbkH7h b+c770M3t+DlWZx+lsS+bs7PiE9Pw6RhYMVgXbV1yoY1Q4e28ph2jnMDb6iXCyLCKvAw/Akh CQQ80VJAZMOEor0erQG7nTksHlTFbc66OrcdEvSZGan6vCMOrj4juRZk6xfaoi7+CdIq7H1u y1+TWLej9lfE71/t1KuAmB88HHMXi0qpinfEDngTV69ScG+cWOATz9AlNkyBZ3fArw44VlbT l5a4TX3J2I9Hbto1O9nlGE+Yl4Y31Wy1cBwROKNTc+QtBHqBeWy2ESOvI43SbngnSXTBhE2u 5SbtkPOLmjqfRu+HbLwcFotgs/kzJnIY6ZsXM58uHDc3FEwk4VMNG0FRkgSyyq2Mq2ni6KNM aOXSatIwhJoYRv7tMet+rx2l5Wm7EjWIR0idvQk8j09lZVmZ1kdybmYzbOvxi64Ho8GZZLlu IfaLm8tGujgsHvr2QSLS7lPNDhV9m2TXt9dma9nSZVN571ebp97nzHV7P0536OOcTcBRbS60 IZmnDawDHLKSmLxeetse8Dqsbgbsja1a5CH7sGoa8QtYHGT1Ori9Cd4wSBjqD2b2Uhskc2/x qffuUO8htUDoobHI0YFzQ8ncnFjFWUZktDsnC6a5w56ydMIU6HMNaDW6lniPVH73ft4ynH71 9ePj24NPuLyZnRBXPG0mingHhgnuCccEc++6htjKompuw3u/G9GZ01GW5DA4CycKV0E9Dzex 6U71gvYve+CcQw9ais+Hu1YdckXxbQ1hhmDueONnMEhD+mX7r0pPn9H0um27oUTI17j1TbPX 9rYHATYonXJ27ETRvhiPHJHQ+i8SzRxKu83T7IMzlMxiTY+B5i5bzLz5csAfPfwkr1w8/joP uMFMuhu48rrUvdq4STQoV3IF4Ep+1teZCG0TlGmRVWJYMsVRLenHztDxj1c2HRk1kQZO2qkL 0HYdg2blFlJzQiHEPd8lgyW+8FGTJMWVo6kbxBuPropEnMsvSljNuKIDugqQbUG1taBR/I8y pVzKc2I3Hq+A6Rjzd3G2Cv3OaIDEXRtm0qpnSts7MMMKUuIOm5nEktOTLkcwIt+vTjjxB6W+ yvyMj1245ZWwRnzu0+WDempVcE/nyl3rlPt63KVO1ogygnGhaWyUWfdVm4bKZlNU1DaLTTRe TJ6VjTR7hw91OK6OWt4B33LqmY7aBExN0PvjcTOm5yidvHWOieNNC2ex8LlsnpphBONCHeEh Pv36PTtr6tyU8N8bpk/JLKIrNX4sjLsUm8d2kQ9wNZiW7X4vvH/hOo8+J5d/S090CNMHZkBg cI1dQ47ec3QHvMVyfIgedvp5rcWhZkK96eigEG+Wx/463oylH/vo2fydX/u9bvs/O4+x/tL/ O0UYfKwzt4tXS+y1zJAhJEY5BQPWV3HgV72ZTDhLd935IRWQXlwEiAyQkVCEBSMsMBV+WWC6 0aFKKgqSc21LLWu7qdMpVusmqsm2plFRTaa3dG+08umrT5fu06gGxYJCav05SxpRtJBSKDWo lBMQg4GXIYMaCQAGcDtaGQkjRGOJ+9VrmGahFKUjGyv5OoYY7xUrkGAhtDCDeQySvDGVjGjb MLHJrUMxuZVa3dVzDbp8z4nB9TIDEMGzrwg9gg/EkKBJ96+6wVC9Sn9J/NubaGfq7bALuglk mm0jHdrFL7/ra+e20JCSIaTPIQ1CCDnkkETedPS6KGlxFOkN2E6RwW2hSqa8kkUJGSQ/TiUT aVhLbcheS2oxvpuCbRk0k0rXq+3MYwtEWnAKTFRpuBFs/UcdfVczAXdOZ1XP9/BQv5+W8ZXy vdJ7979Wz/NrdV3s6P5v57p6HmlQQh/tYloHmAgHWGBTZRWf5pO3LH+kMvo/qqfgPxH8eozA C8U9/Or6TSpZBT4QfEvD4wFB/EmlgwqdIQBPOQOZLzSThoe5Mh43N82Czj4pjhL+bqvOx35f K57mMv+P+SJMwOpLY3S6qlEpKiWIwF2Bgg4HnJHQh0lVaEmM/I+8/hmYz2/F9x9D0H9yfsvd PuqU3o/RfsAhxV2T1b6vwXb9qk0/j4Zywn+2pTPOf1OEr0eS7MA/opVP3F9mnkd531ntkS0/ Y/Yfk6GKMEYrzf8PCFgL775hevEq2gzMH4p/QyX828B8fFJJRVSSSSSSdPdIeL6NJbD4CIAZ JAylIVFRgjIDIIn9Lk4953tnD0X1IE0/chnTyCI23eyJEHLrXNE+d6Tj636kkx4de+LwSKyi 5fCOxM7XKUySTpIBCYOZ6qZQ9qPHgGcelYJ7fqvHogysDSNtGFQ1uRxeuIqGH1+33+vfmqbv KbNoVcqF6ocyZ23TJtg4lwDMhFQubVQoAPU3qy09YdXwavVrwYE6gw7wWKqPXWtUP+UqEVDs G/t5FQ/R0Bwz1q4FQ4VQyyVCis5pQTYetpE6LdmjePP088MJmYPxw9TmpQm7gsAVWv9ndgo1 pt+MUlEhH7C0euCHM0d+wKGgRVtGjulDMmfb0N0BY7ilWXeNaYm8sdPG8LYYX74wLslL2MCU k2LZt7FWpTD1YgVZwo2Yq8iMyZ0sObsc67il885nIpQLaZVaTMO/Klm7FS1KpSbhJ+DDDKVp M9Tp3jd+eyGz4DmYzQx6fo8x1adcr/vZbTOu28svdrdqdIhjebqmNfBGsnQ7+Xuoei51kI7j W2K5dNBjnNPwjcLJMzBjjl7XyxjnfiWzEU14obJAA24w2SHLnLnLmE5vnDsU8kOwmTdw5zle dB/nMkedx3eSG9f2oSGuiUFNPy/E+7yHJ5jP52vVGO8bL9pac8Fvxud8F06vrq5ne+/urzLa rR4PgWmJlmJUo9K+g84YAxHFPScARZ1bRe9Dz1Ipz6+GBeVAKRU1oHOqvvqt0wA+4XHwyiNQ D7hMEQA+kCLvQYgVHmGk9MkT55uKok51Lzr7lU9F1LqFdG3KHo+f1/IuDq9AsYUwxE4kc5mt 48ozTGULoxuWTIKizgHVChkzirswsRhqA478xrKe9f3rNHLbn6/exW99Td2k9vlPfqaaY/O6 Tv/vfZ7js/5joa5513kog5pRQW0SFShnLx8pDoSeX+abPN7PslHrHp1R7MML3Ll6lSj2yj8e n3m3te2qGRoATJ53KAlu0FxxRxlXXUXhm3+XwZl2gekzhXSH8Ze9IwMswfYQGAeRyUsp0iEM lJ3eeb8a7pKs1NrlnzM1Y8/T81n2u29YbPRl7eujOww41VdpIPSZ9s9UPNmZbMDu1Q6hUMOW 7IZdCW1u+83ibZNY3e461eg/kmR0FE79Ny2IQqBe9htgK58HRP0IJxiQtsW6ISrXJ1WwJxnB e701Fo68c/biBtyKnnhWtHmqu1AqpJ4u4XbZUGA9XkKDxft7hLAZvT6DzV5EFWwr5pBWb0HB ZIrabFBpQmLOvKwQ0jEY8P0l5S8L4BcrfVfssBa0yyem28rYqqlOtcML0VWpKhSrxatLXwJY Sq9L3vWk5GFrYek1xjQY2RkzHe1au4CQR28NdGl9swOrDyTxWTRnJDNGiaZQ9SkPPOzs1lUN 6zdODsrcXzMcQstWDMTQEQNolkgrIkiQYO/vzXwG7Tfg4Lbc6RUqXteK0ta6pK9Ve8XnUi8N lLBqPNtqZXm831pyPHHerXiezNOL43cYw7ItdtO6PUOGur4KTiMhJ+bFoyUonlYlOVBzgtNA rTgDNaNOGhKBPKQ+UxvhB+PQsqlTXebMqGxE1Sx6uoZwCizy7rDAZVZCKsAUoyGV7/QtKLwz y4z3OjguZ6WxlN8HraKvW8One05zpe1oHIreLUta6hgkJpCNwr7EjAhWB2p3zZWNs1CAF58X 70SVxsGyKhnJnAuO+lTgbBgQNsd9M5Ntn0mYaEYQInCU6kIbAsmkTbwbpJFDGTM1Gkk1MGHk 5MQJfnb5YJP2u8DFAzQKON8hmm2Yx8t56LhDVXVF2sMSYGa4xVsS5ihIO6GzWigTMYMkGIPT whC9+mBqgW4CY8HNpcV5vPnel1ME/8Nl8W+z0Dp4fuQu/WUnp6/d+71P83kjSuslT33T2gCh AoqZ9QUYYlPmrdAnbX7DUDZkYfYDoBgHZNDtDClra0eIqIETpF6UnTkhc85oaBDUC2I5VAmc gZr7p9K5EzeoVCCJUGML5Ul+jhraxv+3GqXgZy+HrRyBkpLicpcBzApWDk1cawcGNxqGcFg2 O60bx3bZgZmY2YGCk3VrT0UNfcCPyl4A87vPlr7cP34VTs7vd7vd5tVGmxsCtxdTEwSgLx7p JMrhTgEHtDZzC8MhAvQwD1pmbu+qqFVJGyQWiBQfvIujRmsqt8ANMO4oLlHzBeCBQD46ckEi WxCPhUG1ZpdLqU3dKQdVUt7ejYucxO1X2SaOI0vIIYdV+f034nlT5pte1PaP+J5P8HyNvueT tbvCV/z5Ox+PU+Lqz9/1+vmeN82T6u8m+B720wNB7e0/t7/Hoe7yKa+ZjQImvLN2oRKMgzy8 eICtry8KYSBJ2V6N344KSQ7oA9t0fRz8eVXqLEObLoYQWJVGRym6wNRBEswFh+mePbK5zFbr Y60Ajbta+5OX8bnqwhRjPwqPDNMscD8I/JKnE/bn9UxYTJpKOeB1gJ1eUsS06jg6qQsb7CX+ ruXROswt5h8uj6aMIjnBkxgZSakP3QgiGi+mrFa+C1KdE+/VIVBKgjIBSEjFAkmSagMM+hlt wAYZt2NTjKic4xWW/9fSAEXDghdt+vgKH+qiF11laIIuv+Nw/rtpk5l/23yf9ilbVmvsfyS8 ZYTgtRYRwWN/KqV5o5f3KrKt7YPqL6vD7n7/y7ufmXDax8n4OD2GwKIawJtzcArQlC0rBsoh 6J5sDp5sVFIDY7/cjZiF4F4GgUQitZWBKbB44gYQSRJGFmSB3Dv03iUEDDl88rLILoy1cSKV YUzimjM5ybJjpWgWxIDNlebX5J201wJdBEiS7Mq6gckpoYw1OnWCI9dkNCBoY2I/NQVDkAA8 7cCFIHTgKPt95tOFyw+w8/S74hCdrBPTc4nKlrtu18EiY+ipTM6mjIT7lFfjkPZEeFE2kPrL sPvgQu2+3rc3CdaDeCJAuRVoETigFII2AbDyXQ+DAoYr1KIQDXUsRImI0UAwSEiCM5XnNvid kCIDD6TEHIExxmOa6ztHR2MjvLL6nx3C6iqWiMpRHamZwqOV4XKJueoQoymZPDhRMkPpQO8m ZDn+ETnpHmNEjzmw2HonxDebipg2aNygAxTXWGJvqMVAuwwdmFjaTqUJxU9NbfkBrQxjqqpJ IBo1abLmgJ7ClBYyrIQGyAUqHfzwEKYmGKaWZsGEypi0Drmj4ZcINGsItMAPSO/yU3/TfIJT 0RBtihWHEg2AujED9gnQHj1HeDgBE9SB6hBHwSHnyG+K52SfWLB8ZgdjQcyIcpv9J2TpVbHa eaHbBEsUWo7tqK7GqxIdMRQtalNoi/QYLzm94cTV2q2lzYI8UsvBp5WoXdhw7UHg5MVgdj6j WAvp6WUeScoPTg8QBxNOY5DfXsz/wjxHTuORxXTvO3qOvqsU5P2TV9HFIpwnEwSIQ4uA4lYI dJDiR6jtceo19BsficDdHC9oqLe5q3PqNqNy01wNTUfBAMkDAkzNNzm0NebJg3iMwnlgVqUE DlzYmMabYDuZcAhoUC+OmHdVcS+rVW6IKEgN4cnXpd4D6oNJ4x1mZkbR8YImIHgAdADzCZmb 5dmk1Ol1A6jtEDPQcceXaI8TDCRBKgh6IepEU1gA2YMCMgQIRYQhMyh40xIP0vhIjAiMBNg0 CQDYihDJBHW6E3JsmnR38UKj3eWwcdcNT8P4nlPFuNpfXrfv//1vz/zsXR5kI9xatMlw7RM9 IdMgpDO4Nv5Q4VZhMzYtCc//Dr6wkAV1G6QoG5Nl49a2mt33DGO4FtpeZsay/SO9EqZyYxmx HIAwBj1FcFfrh6GlPZgZ20UEX24/dsVoFELk8/2Psl9RoTVsBMz540PCYEyhUusVFXtLBYQS olI+dTiNCQ/6j5TekzEfRNncCKwXN310QJmYPoBjdBt3bKHwJj4zHGa/IX4rbbtq+IKNGomF PW9y3xFISRKga5xAMJG9/jPjARP0gPhV2b07GSO21DNCzbFh0Q7iWHtDNctkma17YXkc2g0W VIYMWMQMDDLA8cVYui9MJ2+eoVJXxag5tm5SqYLMCYqTPjtYxLE1QZpUQcGU2o7Sy7CCLOHT kqW3k2ob8L7EZzKFsiIBr6vXHmokvid8GKCHTFSSEYQhIxizGe903IGDJYDyA2QgMHRl4Fhs iLYN3kOCdvA4XlN13m7/f22u+nIIzadW9Gzk3459XrgenD96PqCD003amZHqGh3jM0PAeE6z 0TeUO0okUq9pkpzhnOJn6jLRlN0T+UDMFscXljTAlarUY8hrapS8qVERV/RwLrx3F9M95IYL vpjItnjE/4OLDDNfQvcwLSLkLF2eVcNLRYneD6a+6eBjg5FykWlTFJx9MJs98aYEnsmachMW cvDGlSoI7D1zHxt8Ch8aAliAxWCwYkVbt8HANjGKjvnfb4fsnTfJ7zUDTN5CnOlbyXWoV+Jd oU145BIW2vrVb7P6n2t9p8n1OvR5C8eM0Gnx07uX9zsXDxw3sc31f4CU7QVuTynfKE3CN5vN T0DicTwGNhrYJ6FJIEQ1DYUmKNBv/QvRTSDZPFkLBWJtOAaMHajE0x5VlOvlMzCWFWDbU8s6 UqPSKR178cbFbZLTMlheNAZmew4x9SxPR2vpnaIWGOc8StIu+Ei1BSxdZGNBSzxLFcBTl2Vu vMmZkl6fB25vwOcIJv3Mwm/SOcHlRCxuNAFuPmIGofxGR50KKffBE/AXDBEoopEXpjCUcM4J 2209pxdcIONuLoosEghGJFi/8/83+Z1J09xWq85FHV4JPB2+8cT6r0TT7294y1t+9m8dkKLF bqaZJ7EJReyFTcrgazYWOk3j0TkOA3HhHJsUTRVSkOalDKEHpKucvevIT0IM73NC1I1gYkJs vQJkrZVre15mgzDBrgUmABZMNha70JUVoLwrPYePRyzM8cGZsCHWD54qhXUrRzJ9AzsYPgWm 18Smk8cK2lawkZOTe8N3Ed0R+XAdGR5LHkQAYGaHgEgVT4PqHlJ1nlL0YuQRPLAzblAAb5FD kgvXJIv/NP/KfJ+/2u97bgMdPc5O78+6bz7JeUPtMWb+AjFHeLHeOJoYnkPAaGhsL1V4gsr1 m6pA9ayjYSaQvIqpIzmZxfB5GEIemOBiWSyAihaQDY4XVsLYvclOvjq0WwepQMoJvGKLrAlT Eh/Ce8CTUsMwE88pKoM2RFL452RbKxR1TGKWqmxZ5WJSJFKZ1xwHuWRR6+5G9gIBA2vWWCwQ IQV0u3aTXXhB0iEFigdSDrO6a14fRJmXLqpwa/VOr0Muz9nLp5SRA6N/Qb3iCEu6FT1WXqlx /53u3PXTBJYnadZxNDQ2neN5tKPOtVMZurB2A3iIDweDTTELWMscsaDGVCMttq1GFgWHwtcl euE8a4eCtC5XCHdUeRUIZrQkINdACVnHBlTNeJ13XPD4Js203qBcxAhnbs3Ndul/QGFAvdO2 UMKDpPMd5fbp5Rlz3wdSRgIwtE8MCKVjDw78wAWAtjogMLIKkI57FMDNLgkMQU1vwdZcPSHL X7gHMoA7BECndmciNNiaEyUxMiZEgEhNCePWKnpGDIwOwjLRetAZqE1jJz3E+7kjQCzn++he oodmGMhppo6JdbwRThhwBR5cbIcWbL6qepeFST09crblbcrFMET/I8NLtLS4atDyowESh4tE oCmD1gIkES4bgXnSAidMQH0z0AcEQJycIDMOqVvICVJMS7cHIidn3FPT8vi3ReLGKQShIOAZ OILlrwhL8HXY/u4RkJF8Z4z0TiPTPRPWO6J6Z644eM+CZ/gt7kA0sxR/cq5sqmGtBvMUGqHQ URtwC4gICaMWJMTCTsxgHb3O09gQJhiCQgQhcYMO+aHiDAd8h1nStDe5/SmQHCG+Rgb4005E 5FKM+etKpORT8I3nri8IQG+VgCtmVYaU6OC1ZCQWoQIxhFgJevmkZZYmt9r6ecvL0ppsu2VA Ggg1VVRiKyg5fuzDCw+Of0HsHZ2Pu+fD4lyj0pgDfw0jcQe67hTs4R3LBKZPhX3Hp9vgsiXB SZQSVO+7qP5Ds+OLS/vVnJH7D8RyRTS/xqizOlJShrOeT1sgrBul8CX0GF6r876etvyMtmXN iKjRvdvuLO7u7yEIi+TO7A3494gxjXZh2QbWYEJ9R0l/vPrut1hP1PUtnpnCOXTDMwIv3Bj7 oO+hkIJpjP3R900KVPSuposvmuVb6Hr47XuvjX3l7JZFWDAJXMU53ljccJzoVn3aTGk/gW5m LTCY3r89gmh4rZhqrHt6RKdJz/O919We/79/Zxx+Z0/Mr50zl6YUtKOdTwlNDWqwUQOsU69P 3bG8P/Id7k5yDzyJzafoIl4jz93oC1LJB3ysZJ7K7Qok8DPjLmHSieOFS2dMVWkiKJK7yacp GBaU6pZ3tgnhXwJzelaXzq0JveSVCi7uyruSYazbQxinPTjk305FIuqAR2Vo067lU3HgU0Vk YyUIKTGkV2Jc09u4jbTfPQrgIl7QvWmfcVPnfn/Ps2O5SBrN4HVv7ty2qrcIddSJmBImM6oK WiHrwBA4UAQHPzejz+bOWOm488t4n6O04Jk8fSsgSUATtTq6PJ8+W9KYbz843HLFBInG/qAm 8hN6yt/gtPiQLkG/tJRgEOrBrwSkXq7+HemjGvh2jYgSRM5JxKGb3Lic0n5M/b3Pe7WGhz3q Nmx8ENNeymJEndAm9ffDCQD+jOpJj1qvvd0aRfolMtZ6oy4UmQ843UaqZpze8qM0DqSKgU9G 78FOyECEIaKJN0tuUUuXwvo1vNrvsvi93l5CBwRDPj5bc12jzMdvBSaYmMOVxEN1tsBeJgce CrMq8MCckY2nJG0BAYvJx5+a4hIpeOESOYlopGz9f1R+XO6V3sWk6+qlDGibqurNKuEys90M 5Yl2Sa3sUvlzytspYjm99KN84QNnJzpHDjkSgEcz89PcohsqQ69GkiEIw7mZv40NKs6xzcgh GFXwnNE5BqshFUQJgdMQmDAYuzOPkWImh0YKbprmMe2w1QCZUtyESil2VezK74MNRMCExgmg 1Ee6Wq0m6wxial9HEMdqGxYpX585bXHnIp3ajiEk8m4d6dIkQ3fWh61+HJTgNZG0es88UD6y LrwQgxjwO43EjoOvHpo84aGuDy3JTlpLLMyJv0dM1rPD76IoHp6DiHHfCZbjPa+8jJa0FLmm +axIdVQJ+KIVMRgJhccYeeDtJa9x21njFajOC+K4dSbqtChv0BZXs4plcFTwdO/OTBy+Hz8e Up51QcSfMw0mMzjDHg3NrrWql/Ybh0rQYcY6jUIPnWEMJhN379c3QSiHaOl9ZbMmaSz5jhzV wYmKc+ewcEVR3sSsG5NKTyo7G/zRaJOcH65/ReeTUBZbx7XiCdXILOZTKSnnT4awNibZhLDm k7MXcdSZvDXz9BeelrLjPHjnylc60lyTQuCbAQ2Rsdmdd7wsTw4jtjQ0WMoUR5zjDeguHNUU eW0KWqnCEhCEhJnSTrvY6wkJFH6tJFAi9+Yxc49UG0dPF2qt2a4Di0VXZtdxLSumqxgOcohC 6kWthWJcY8cHlWpx5qYO5yZjU77RbscjdJmS3x+TMyEJBnjcGVOTclG032dNYHrfEnBqHr8r dCLDt2UedyaozyuZ4vnpIPXPBmy3sNDsN9NvJZWdurMkKClqM6kgszzpNpajro7IlS3kl19D 32OY8n15pNDbHZ06yd9l30mxw1tWVZZ2oXR24yIayHTlYdJPccWLzh+PY+36By1RXiGcTrOW Gk5pKeU5TG6M+GBM6cHJ7XwRISQIlxiEHeedYpxOm9GuDXXZle525hbN5H8f3V7h9h9Sggjv 6e2aA3+fVZezBqEim2JxkHvmn/D8nFqrPvqh2O7enrziUYgcg+KzM3UJF2kgYE/5Ao8mUZsH m53Qc/ZafnwLwPrRgeQPKfD8p8AfXgfw569bCLnBbpST8088uvuZ/Eeuvub+qtayskotDEor ZKZqo1bJszUlk1Nm2ZWbQsoZlsn0O964BC5Yh8uOA3RPgIr+wQS8QCS0tVaiNooj9SSA1FS8 kFtFLxC8C8BPz4MgwiDnRF1E46gQ/H9CUCgmuN5OrwtjLrGvNNG9RVDzYKDuTY83qOrtIDtj qKuO1H9swrR4xrTHVT5q/dh5DpJgIxiIWNoAgURsRaDsCG4bJUGEIEbMCzery3tm3a9pdved 5sUk1JKI0UjApdW1NPqPt/y/j1fL+H4P5P6dvgjUZs1CaRXUjrgyDIIbk/BE4xlGgMhTUk3d Fmu2y6KxXWjcRFE0YNAqyMJJ3REQ2JNNYDvd50lva01q5Vpqlu7duVGoaYmzBkYlGgNUzjLo mZCSGMSbWxvZXSMVZMIAa16tLQ0o2DIIgqA5UVBRRi2qtmkBio2ju+W6XnuZnGHRVyNjg89R zlWTUOBo6WnpaWlplWvvwYLwhAM/cTnh7nQmIijyjqB8p5TMwD+ucfv6n0Q+r/ONlein4/wz iw4ZIiQlFHi8WvqEndoO026at9LplNg3W3p3czWasyZqoDGGmen8Y/2uDp6uxBw0m0xgdDCA rXqiV8rXs6iwPsL3cMa7poZVthmoq9Hcg7e5kUZGmwaYw7zzhnbIV3u0TR44hGZwuqHIESjT v2s+K5ezazm+RcoaU2Wo87kgYw20cMBbYr52hVeBowaA2MwYHjfNtJocRjXVNIWmgHpQQl78 YuAoicMSUs6aDya3s5o+ve0zvRUqmYG5WxpvuQd2iO1ZE26Jej7BA0+CN+7LjRQEN16Pijvt Hm9IrgnsiwK8yaB6M5XCcrDU29FQ9tj0nXIO2v4C3ibHf7KbBLH+xsapyG+NU2Tx8Mx//jqB P9za9CJpmlPHrUahsemuTNEw+zMOKEJh8DcK+jRETNq/8oc72gaD7tEmhhyX6BsJ7PsztKt2 /Ef8Khk+rmbo8wai8xH3ZYK7t7I80WLHr5kt44ngk6Ql5B5YgGgdsUcmTnDnzCGO5DJravqH mRxw4PRHgC5/BNPLuyck2/e+OktYsUW6Vr1V9Ygeh4jYIeS3+6PRsbcfDMiWy788lAuHlDAK dOo9QK8iCH4Io9sAGH5WLsueEDNvdbNzgjAsaJlL4ClueOIqFaIO89bub0E7dBWCemd8APnC AUgmQ7kOEvqQsRQuGyhLKhsHCA8wqEIgkLj75UzaChEzRUM1UMZmL3CCcyAvvFzxO6AEyMzc goZmoNCCQQT+JLgAmBYrTrQTHWBdUMgRNA0gm5SwgnAoOFUOfPVKhfbOVC8LBMWhEM6OSIFw 5x6/8U2CAH5kCrUKc3ALQWWfeEs2hVsiyNxUwsvh8WH2biSAZRD8kaiZw5lQ5QQOURxt7hzi yqJSMnTUSGn62jtZwxacbfaOzi4wnEzq9di3euZqoegBDxYqQggSQ9+KhQK8Yh/miHH6pz6i QIAxVPMEfJc+6CAAOR/vkI92/jBox4zGRvSQymQPpVK0EkjDJHUAQNBXixhBYgkap3xAkKo8 8cx+EeAYDt/0qH3k2fFqmJPhVTxKk3YeRPxDdXnnTglGEFSQ6cp66SWjbGdTAh192HAqJ16o HXijvkA9FozrsmVQywlFDWQpSGY5nxmMG8JBKgFe1QpgEEHFcIH54C+wEuFxCdIsRSg/c9P6 noBvNsqHVqhBBKxNKDOpeubs3A90oYFQpMq9XgVFsqBEE98RS6JGFDwR/1X3iNqkqvTFPiN7 7FZB5SB2b0F4wxFt+RW6oZ/6WpvklfB1v2u3rIeU5QEGH+puPjNYqV5D7/g/KYYaYacGyWO3 DxWvIF6AFx+XLzrzrHTOKDtiK9lzBtQx9BlD8UynIoee09GwB6kesCDuSDR4J4ppE4+Si3Im EwFjuCCw1xphAi5QsVGRYkSJ3KjPYGMxByNZ/ZsHRyiS5wh2PuP2VgjXmfeu3PNgBwxUP2VP 3+N5bxFQkJu1Qu7nyhOZVCLzKCZmmoTja4SWn4yAV5f2sDvKhuVDD303riEVSiFBALHzVQuB Rok0krIBA8RwaDBuH5AxCzCbY1IZ0SqUgiURAWrVUAN52FKGsvo0qJBBNiCRqFEEIGFlF2Kh 0nECmY6DH9x9x8AVUIUWADbBkiSSHgKhPIy4xeKHvNSfxK9z1dFecyhje18VLeon7xdwDRlJ ThNoEVTUHj+a/NyO/5X+j737f9b3AyFSRhCetWBn/S7p4gE7wAZ4ZI/UPqIcKdaUmH2KULaH 7dW0KkxxobFxzyMJ8UvB1DV42lQpUIqEQQiIJS1spqpieDVWEYpCGvVHsNRqxIwCQG3PtIpg v53AVswXhYGqke4MJvFsxEQRpDTBsDloLY2zySNPIYDa3ulQ9MbFyNXS0fyDeaTGgYx6VlYx jBhU4NjSIoVNRSDpSlcUj6JZAwMUZTD1/CeB94vtP8R/znxvo/vaOMx4zg1ymvgteqERDZg1 0hDx/KUQPI1UtCFrliiSYDxQdUV03p127aouon359SyDBdf0bY8okA9EBYYa9nUbn7XwByHr U4EYf9nSbA1qtykE2BnnLKh5z4RcmEAOeQJ4vP4w09TAgkIKhMq3KhAAz1iob99haSTXkgbP iQcUE2KhwifkEglnZzii5G/QxYBqPvhD5n2A0QLA/Nw9URbGDp4aQInQAFxi6ncV7SksWsSx nnOmd08/M9WIOL6VnQcv0PZakA32imGJFgAmgicjiyT5mBgBrdCI5qEQzgvzIZ9ZAGNQrDCD hGSemiQZWIaZuA4IrtMWv8cQB1F4/D/BofcDO096QfKiIfg9L2P4ZwgAvch0Y5hmZkQBv4hc JY6YuN5vjseDzE0atb3daN9wiaIuivLIJgVCyoWQTmkEvQTiVCxcBTn+bh0Z7L96YX+4Gmtm 2i58ZZeMD7gZqhwacy2iia7FVx/5BsxxJCRUNXAqG699FOOeQGUQJBePoL7upVfrZeEOz5Sj /CxH9oAirmEUc50M6krUjsbO7LjBgMGpaVrXLq6Ztg04iXm4AtnAa6oXiljKEFQ+C0icuAH3 dBNioUqEVDYAJcVLiXCj9hUNQ8EHC0peEu0kYQGeg9KEbofK2BsMAYwbwIMIxoefGNgbEG9p obyEhGERLIkBiof/BFMnNchvBjYfyZiUhBCyUiUPzh71gd9UOEDycx8bpLpucHsxOC6TntNM eqBEoBE2mVUNyUfCVDilQzHco63QA1hVCwbYIbQuDqfcqhw1Q0GkUpFNFUNuimyCJoKhnoJA 1VQ5pUNYzwzUzjdqhsmihzZom6DcliRgUUWQNh0gdFsPRrS8YqF4DkNE01QpRKV/FGF+oimM rL6corlwucow0JISOM7Bm319SpQ+8JXILAjkfdnI8AGLcUczvaMpDVxXHJjG51KQ6kFwcH6b vB8GjR6QgPQHgKVDiNgOseIicSCZIJzkQP8AOYzW83WZJYskWyobRBIWpUMY5QjCNuSVDFOc U8gAkFDaDgUbQHtAryopdvQGGQwcCT7+JIRzkqnLwEFZD+i7XnFJXLlaSoYkSIpeWcg2QR0Q dQNIm2XYQS8OLPzCB7HtKPSw8QNkyDvyGiOoMHaiZ44gBN6qGASlQviCVnBoB78xbrT568HI cWYHPK7gSEJ5DxPSHT3SHocLJ22im0JRRUnSOJgPo/beX85ZdPshQn8uP7YeHbjukkkoLEnm qSBc7udncs55errwEI933u/GHzavvvp8n+2acVlQYzMMCPSSF6aHF45fO6qR0L/aPzBywOOD iF0CZ8KyKEyaOkqPgZB7op9SSKFTMRichDsMRh0DRo6rOsOnBC0HLO2ihgcFttqbpjjVtO3x jwcX5/xfK/pT9qXcPWnxERD8sv5hJeTzs7Dj4MYF3lgAhYsnT1ZKY/YJoHhuZiOhVCKJjiIr t8Ni85SA5z6Sl4QYlOqddeaMiZyCXqh1CB0kSxxnEoETqJ/p5osEJbm8hRCOImkESgD8hufe sJ+Dyvi9OeWpUqph8wJxwH8FWPM7X3why/MheQlg5QfrFqQ5h4ZbphMoHqo2HyDRaU8UL+h1 mGjWB9H7oPpPyj3EHjbEjGvtnqPqUMyEgWWU3DvEfB/cIZCZZGRMZLjhgZcOO0f7pnBwRJAx pINH0Gl/RGI1ztQTvMBnF5gCOPAWLZnYHhPN+nt3P5PbPIoDMhzSofXLoJ09TMRseobxkAdr tnuM+L1frqhqE0nSJyKh+pDm+au9wmSpuECghOzKdQUsDLMS+uVOLqHS9I7k4lar6TwvJcaf r4jpi0Kqiho3daD70u436vI7IcEvNviM636xkNY/Pejsfp/bPXhi673+fhu0MimdxovUxQuo vK6v6YJ/uwGbj0O3PRnqceOJlOtygJCS6uK5wwSQAQDb2BWjM88RSgTTlBmQ8yzWKPsQxXEy U3gNy8+KtQZHSjczSTDb7/a+jndL+z0phLw4r1ZyP39j+qwSkpBByM5UM3Kmdch561dlg4o+ Wfkl5vnZ/9lbuH/kzVdnRM7RzJV7YnmPwoJpETWqgdeufVpKKgUVR/MjgKYAiHpUE6l6POvK PflnrSqAMIqHaHbB58vL1wNQBAQIqG0WZgFKfihuKu1JEIdcFg68LgpvJCny86MKrdE++Yja WspSoekipeRGER0xd/8Yc+0H0RkFV4WEDNTBqYjmXRDVPpFSzsY1ZIgwjzqo9c/vkOAjUV8J CmBNAVSYag2543z6e1rcHerwm5+ZHx3L50AJrjj8i+X/dVmUjBRuciXEjtNkG6ShwP8R9hbd Do/tcfz4/Qu//uycM9/V+bKeTp3Ci4o3fQSxl9Uo4X6F6c9BLmpHCRmRwcz1J0gOuOoEOEjt QRmTL/r5uO6p9JLRjxYJ/3IzvB+FBuQTvdyqLrFpCrOwLsEd6oHuIXY4Lumw0CQUMEwRgQzx AHKxU/YWCbOvXpaOzv+d067ZAgb1tqDg3UDth5+JwaVn/O1KkAutd0IxiD5FSZ/GJG7eTP90 o6aeB9fAS/Uz/58ZR/e7HVzfzaieGDUlYTAY6JYAhQCUSDOQXLw1vrTzobU12EBnXkPmkkmj OTAXG86O/JC/Wf+3x+pURgfVHsyHdflyMB40Zvo9lmhm2wt2wwIhPMpR/X2zIa1SIYyVUE3V x/ATpgc2N8FT61g+PwZDBuCWMEJo4Ydz7W5ydjwOr/hfjGmOM4rlIlK+sabJ44mF7ogGwu// AYL/NG36FPz+/Hb0ESsJzcogqhqgDuqH9IFihAm+kM0KQVIdPwS53CODEHV7B2gIGL8geRUF jfHDI1y+oInr0NBKpBhmb+F4dDH0zrMLEbmmC8JGIbA4TQOiVtBCR0NhZoQdVMPhr1QgPiAh EJ+3y1qmgPlExZrhsiXyzZ/4ZuZo8x0FOasvjzPdRzVLTpORR6nxgwPdjfFPVO3wdP4YPmB4 +zZbseqdoQMof6ZQN+GFFAoRHEiA9aOE0JBRQeWaAvZH2YmLXcFGM/LvPI+Dgt1kT0UMWPGO CdbCwxdO6kl4SQfFePoXlvoEOT7wFZEjvHfPo/pAvdioIEHOWMZf3PYp7M3tBD2cCXUeYGn+ KfBcO3ImWCTf9pg5/AQYTjlBInM2JoULOMHhvj+k530UU0zVTcb4NyJhM/OrgIzsYB713dP7 2zFBDeuIK27n9/6P6Ohbu5Y/JOcwO8ieXg+YmRhmEvj8WUf8Sj2s93847+xfYrsS1H0l89vI TICfvb/w/QdekqX4Lu3wr+af6jAPpfHBFGk/RmZM+29ml/Qfl02X+Oflm+GfH+LzTnxfiFr+ +GeQZvR/JHNycv5Avuvq8OTtIv3AzsOZtfWcA9YMQpcB/eIBVsF+xqfBpaAtoGjVFLX89Y9z gGxjcHv1TpJ4UwKs7KyZuD7RDfE/jemfHJfxhASJNU/Br7c5tekH+6UVPoLh+r7OyHPH7XEP 1Exvcez/iL9X8hrfZcrmeDYcvuXlifs+yRFvxLkQWEkPEz93L9SuCJ9/8mXwOi5SrpU+of3i kB7v0HYn1vYWmpia2uPL0YrORsRvLVA+djLWXsfD9Z5y4ybn1fe+iLzN3Gn567aXA6WKPdZq RQ7IMy/ZvJIMvXHHE8wHEJuu5WOveNO8ts0LTGnQcQSEY3oQ0A8GeWibvOo76G6jhPO1nmzm W9uVyZZOYjzX2Op4y4CuFv747C8N/vSCagtT84uJCr92AgMK6K7VyyAoQUFi9Wx2BIB2TsUh y4x2LEQpNQJiMsD/xQrVa5yZauIMc1islWgMExYSEJAFiLckoBz8sIIcJrLlUANgveooAQeP FFgksCFXNUmP7llUAXgjV4qrj6NG9aHFhKpIIouUgDMyddQb3HgerglXSYWk6HygRz3yu/Wm lRIM7IHxsuDQBf55og1jB3HEU4o1JwcytbTQEsGkka+ujJUjClaGY+wl2OL/ZpK/qn7/Ybvv kS37RFz1iUiZ9RSVnkVtYkxMnZ4+ieTlHPkTke8KQzZYYo/fLqVPhoM/pDTdSbemj6SNpYM/ oD9ZWtLEGV2+hzdC9+s+jT1Pnpn2Jl9JbxINKdHH4XYYpvGjgVY5cDA+xkPkcSBHmJW4E8lv PhuQvSXMYOzH4Ig2I74mptfDvA9kx9rrbmKH9KmhpvHz+Ju5FRsENETWrl2yUU3A/Q6RkqMj C2GK2pJWZ2ihbqvQWokDxlF1BACgXlP4VdCiwEDGLYhcRMwn1x3DsI8gyobK0VXnjoCgYjQT gfXBQ0Egs6djS5jwNkz69Qgv53Y/Wem5L+y/iXZgZECNv/dx9f2vXrs2qVxc9pgYJFVecd8v MJJgd+3PmiHpORiERm52pG9L8DsAxllEgjoMD0z5K2FD+r3savz/Hp3lyH88BXF8Q+28pAgr EBj8SR+r0QW9YSLFY65BFZzhiiAEBYrAKvNP0jd49WNeMDwLmmqRCsOxQNxrUH5MF2/ua2k4 ycj0B0nLmZzeuYTQlVXGzpIXVVSXf120RcHEM1PECgZGnyF0rHybqtnULTj70BFfI15Hxslw JIzcjYzcTiDkAtazwCI82EMtK9SFc3bMJQFjuzyZEUYqQ5vNkZFBk7+UeAZ86/6pmzB88tYa ySEZ/KhvmL2hJZLcIHTHoo5rh0QCRHPWGPOvICpHejiver0cOnoP4PxKeil9q/YtDs7YOKYi GjBtNxbiTcyZ15UccuPd2t/2hXc7a33zwd8/2vwB8cYQ/U4T/V1pPw3Cc34fWwf5kPS9HO1V Xq9dJjDKAh1x/Shh+HBZNtuqBVNDEU7J3q/lETs4AA9leXjILEWpNqS1jaNrDNZNUpIEJIBE gEdacEsUgI6cfRQNQwUKGc8AogYO1z8d++4Bjwhy9+TpAru2XXvoTu72+/+DGYfi8H2tW1EF Hb5BXgfxdnvbW/jocb55P3nC/WH+38qnwChOLI+4FDR0asUOBAFfCUmAARl7yn+54K4PEArp ARq8MYoFWibMPpMxNI6eRtVEEGcFX9EcDzoPlw/ifhUr0RrORlw+A5rYQwJJbeWObdBJ/G6b 0fwbn9yL+pF0P5KPi+lo5E91ui230ygKh+9/pRigXznAQCgUOY8hZncsewRi2g3s3W8xzwvd 5oNAONOHAeOhj4iRliF1M7OI4INSMB8RG4+JB5EsBn2YQOuQ5RcsfudmtWfukPIkcwC8EHGY OOHt16Bzv3YINzkHheYdtCB2nFOYASMEe0DirXpzNzF5ClGQOa2uQgyxVOYoBR4M143SuQF3 Dm3pNlqNe0TXDKGOSDhUlQQdUfNG1DcyYkhJpn7F3Ln5H8P+N+V1V6TG4fzkc+wdqRX9W4dC B1Gi1y0F/ci4wkDXGelj59mQ9L3XENyz07MDU6Oip9w/eO+deb3yB6c/D/NNz57KLqzMi7Xj CvzpfRW+IL/P5qU5DSU9RR6h3jo8GNGdV0FQoq8KCFghD8XzShh8plzrtwbhD4pwj1275IgT 6ZSfhP3VCBbh8QmoOmzUSPE//e2L7jJL5zJ9Eo5EnDIehL/CQMo/dnhNnPY+gj5AP6pTR/JS NRLxMCqX9eKX6/QoPGQV+kqHAVD4id6qFwH9SoGm1Ffsqh8W4VcSCf4H/rZYQI1eWsqNQS0J VDgQTIAli8xgCfnrD+sxgCXKhZUMwAyiBgPalbr+9EX9QLgAT6o0gmYYwBM3IAJ6+Iqxv+NZ UO5T8MGjdCoRUM8PnkpUNBUNMr5NH9hcZ2y0qEVCiACapUIZ9rFzDUcaoe51dQFcEkX62CQm AD/e9RsYxsYMbH+WsDQ3/3TNDtcLWTZccKkpqpDhcMSEJ9sH/GHN0GYOdiD/whouWuo0BUNQ izs1wIJFQIgn+CoYVQyA/4kGyJjYilwgFlbNnEqG3QTdqhtEE7i9QPbKhEM5UN2iYwUzEQI7 IAmIzA3AAmuqGXbCoSEgAk8Ab4hldcA4ud4R3hTbuGiDAghCEEIIwfzJqI2CiBRCEYUTAhQY HMIciPEinEil0UyuUFKpXQmA6xsHO4iQZCB9qSqGo1GVRySdN1R9A7nkvhl/DydafKzD5aD8 JvYtDfJPmHyz3WMSrWgvQP35BYkTixYt/ZwJljAMDD3O4yHmYErFz3xaCHMyyPjSkJqhQeBI hC2IwYvVwvhkwLxhiDJz9YIdt5jzMfvdCfXwHtJe3SosZ+viINpMYY4Y/A5RkjP0fND9nhB/ lOdr7DF2V1/6FBRdO1J3kuIf8l7RbQP6vwrP9EQP/cfsR/mgvqfb5MuX9DAvobUfHgez3pX/ qN8AhAA4z1nqirt5+Adx3/dcyVRIwJORO7+dj5NxdVm5C3ZX2M0+0X2YTpqpPTHpMJSaFU4V iuXt34bc9IEqLEmP7yZ7U4/r2+L8L7v8UkRX2DJkKnbDe3hJjY1uGCibtmLPNEAp6QS1su1Z Bk2cSFYmY5BcHRyboV4F3ucAXnRUfULH3Td7m/3l5dZCpmxSggbMkJGWot8jk/W6hej1eEqa zXxrD6T69xbwuqlIWIfkwPbEOqPlF/XSXHOx97nX/N5O+04oo1jhNFHBaN2Qdr7D1ljk/VFE YQ9h637HreyeCYPK+7qsYHUWcc7P+4oev2SnD6+7wnoeE7/ui3jz+bVx5xKm0PGzQPsZ9MPE IaoupSaZ/JeWJ+DBIgxRSFJqv2f1fnb1Kr8bfw3u2rCjdd8KFMbn9ACwun9w8ZKzDJyOyXNb YLxyqQFExgQD+dVYJQxIMEL28C8FLKXijAKBwKho2LH6x9Y395k+KaLY40dRKNpTS8eTW8xQ 9SS2rgKgaJ+EIHiqg0HjYb13f+X4/sFrbacAkGyBBP6kcQmVqLD+PEAYooaX97SBlVCRUJ4C ch1qcgaZs+rhUIW5ikvIQS0A+tALMWQJL8YmbYS3rFTdoJxf2EEiob3EAv5yi4wD1v1F9lBN AGSnbgEh9AwdIQMRNRQnSKeyeOaQV+a+2TDiAEyntVQwACciG3IACRdZEEgIEAX62Am71P35 UzNCxFOU1Iez17wEDIt2EoTVDUcwkYQSIRWKqu5HCbpLh0R9cGcs84T1UkjKJKaaqVUjjp6S S9E9cd/ff/DySZ5H/5w2Qv5Fnr/YfD+CftHPpPniUwQZBM5wmEIwrGqUCZf0ShG/pI5CbcJi 06LdVlrH0cOlHbTTOrh8Vi+omDSD22OA8fYF2veEYh7ZK/IZeIyDHDdXoimWNRLHDMdIfQhk O+gffkhJPMnzvV93C6j1OHA9J6jDVu7t8z5BV26yBrpAGKcY4v6X3kFiZ4U68+TRICCD3Rmu DMeemyBZ6/CwI59mG5se28Ylgr1buPziF2k31jCciN92RouBQ2D6oeEex1AUlHDYBoRXY4a5 09eoji3Ie9YmvyYfhfUTxMOPh+97mdeKu/3deb5E/hf2OE8mHJxzWbkgclAcRy/ogsJ877rL 0Tg7HZEqFYBCb7AWSiiMFwIW4NBZqMLTu6Xokkj7nanrhXDXLIkyune6nwixB8kNWqFoR2uO PdrDMVKU/AOH1Pdr/gyQzh/ikX5+fpbZOHB8tYX24cUqB1ffr9ad3po+x8nRPUCqbTzZYOif FkyDpMJukgRW8geZ2boIS8yvHRSGgAhOJXBMzA+xEdpMR74195ILfLc+iF66R+Y+AS/vBB98 /8n7OmwP1iMF88rjcNTaJyDuihI/4ZQBMp6FfXnPsHHP0yF9iVR+sQoj+PcY7Zcmzanzxwg9 SeS7fqgN3lDafQzfYJiJmf91fOpuXimtpVpprCT1WPY+CavAcdf31hU+zwLnXwlddgqxj4rL jxT2m+Pr+6PNZAPWFH/YtX+0etJ7Ii8tDNRouT+ISoF4zubCoraiY9sY8O7BUkCJPJwDA5Ax QwTZBjIETFBq46kt+Bm6cAD1YAmP32PyJVEfPaYd8aWqqHuLsZrKh66EpBMM9RXryLZIYicq 2CrACHqwjLTguWftcSQdA2AI7z+ySBbQIeQQxCkWY/2QpeTJqfjVGemIFVVQ5PxJuKqpNDzi LgHswgn8gWSmIkaImF/gCG4EYWcr/Phln8hAqvH/RMaX4J84eM8Y8j6cyz/mE5kvH4yxdRa0 qhSihJIGWzomGQsAtCo0V0n6uhsQ/8Ge4pryhBdGfPT6No3ShRAz/MfzjI5AyCED3O/RI+aH 6ZHnifRwokdkD5YaYd3nlGhhOqOqgcS9hvPS1IWhWvzhztz6WOGOgYHAYClQgAnPqhEEohC+ rMKosokUq2P/IQLoC2iq0DfAE6IuVRKCYVQkAFlpI2AioXL2SDjVD2KCXIJ8YQh526zEugKZ BEKNIqmgWAkaNsxByGJBOIYwcdjoYAJbNJQYXnwcRUIrEVKFHMIYDYBEo1CCl45WkO8GAjeC JhQSKhgC9UKVCwAlgBYNDOmExquzBhBesitwAJgMCQKhIBh9kBu4quf3AYj+0/ikpOmUtYKO X8Z7PbHZnabs4z4HOfi8YGJQIfyQDHut0trE5QgcJhgIHhP5Os/xD1DAwFQzy/rCduDgB5VQ 5kvDDulQ7M/Jc8AN4QAEN+pjVDtARNZ6L3cA1OsNgBm7UsGM3E1KhSqVxLDA/gtZBu+kjSoQ hRARYAeZUNAmvrmkxBIfbQTYAEQSgASgAgIJQtzaqFyABpiCQ4FQ4jgLIJAUgKbwgnFshqRS na3NZ/jmCCayjSGSCY6RwVCFgyNeAQz2EPtdyT9Mt/KYB5ObrOsNq9fuXMA4s4nGyi59+XfL +JjKLzmsdPN6u5HjOLsWG930D+/5DB8IEYMTKAZyRHr7mdieRMgLEEkJCQkEIxbZ1NmNqBsQ dAqh+0KMNHf1WEzIQTWdKw92IJvpBu4B/HA4fdOBULwOP5FOdXSA5fkKQL+RKG77vY2NM6Wv kmd0Ihfrj608jO1B6Jh8T6EZkcjLQNDXy2kYYegCsYdsor/ehWQdkUvHKMQgOwrTu0E26oZP uT2we0n+7yB2objgN3nQTVgQVDSTVTVVtbCJ1EwdLvClIUJqP8GnuqhuTvo4hEyTLVwJjwrm I7MNpCRC6LIKv5Z71zAh1Eoi0HciB5nmkvookS+8gRHWc8zC30r377mfS+oT9m/AhDDCeXme eIRcvckkly4QpOMRTqU0Af2vvsJHUfnKd85CPjj/M7ssMr/U5dLgnlztzl+Gq/Ro54yno7F5 ipJA2CWPJGsUb1ULyANbc3DtgPuCc6XnSHgFgCrZSMOeJQbXqkLNiGlyJnFjSWOgc1cXobU2 1KrSbakm2pVTZWsekgkAEpa9P3DAwTZ9AyyIMbkKJQfMtiBYkzM2y8oCnPL1RQGIQS45YrGg mM3+RRXo387nlFQsBoVC3aiJY3SkYcqCag5QyDjzkQD1ABDQIFyCfsmKlMIpmIoos2ys1fYz M1day/U7zrxWVgxlE16307i8fMRjSCOsWDSMg3oQsXgQg/2SFjh1kDC4CZiobGCboAOtgQS/ OEEgCbQAIRBMCUIQsJwGq5CbCBRQYgpggnDyqhnkIJgqGCoQoUM0DGhc1Qr9IewPnPkuBbwG HzL+7Q0IrmIdqpxzraC1ipAq6EHRD5PE9K3H0D6vkvepKDrPH9bjVDtEXyLzrlmZqAUOe06E EyAE6EEx0CBZBLkACjnVDMAFuADLSCZlkmDAAm6qxlNm8cBqB9QKDwh4ELiwPCXGwMH67sEU gORBzIWIHZIBSqhHEWK3IDEoqSJJZofA3bu3daoqFKhvUn1DUThMQcnQOQilOlUIWHFS4KO8 2bAd8uRBPCglGFULNFFlQpbUqFlQsqFfzJzIJrW5pOsTPS3COKCZg7ov9oiFEWEUTbndy1vk fp3EsJtNNUN9mCBcqGkgm+4G+3e5TzOSPl97ZNT/mA+b0PLmlrgUKr21dhFRNhUIIMVCBGAT OrRils8ANsqEDCqE9IAUUQNMwBnJ9IuQE2xpbboiIOd1UTlJRiGkQ0go98FMhShGDeUIX3oI zY1aiSSHTGydLC4hUJJJUDTUjG2xsGx+6LDXs59ASc+DW1L4e7gvelefFXxIPm5zDaZu6umy ZhhIsbCNDEGERDmUEuxUIqERMjYhgFgUgqYjQK0iYDBLII+MRfxA0Z2Mh/s2XQFHgLFwsmRk xbDCxgUhcZm4ghGCAUNkTS6QViHkRTADSQG6JQZJkXQpiGQIkAxhBs56qBqEJJIQ0JKKJAMI h9FsiWAhCoh9QCpN+Xbo3pkIkDs8NFw3CZocoqQDnzln4pwXMU9ScwFJ+RDwyHM4OyZgopMm /2eg3vZ7qS+9ULXmEOZg7JiWasmI37nuIwHo6QCyoUAJiIJ0qhFQ1lIJRo2jOczHrjXj7/CM bSG03/BTIM+rImSIZB3bamSSWUl9Jx5rrtFl44kbMGi1kqECWqMJIFQohpZp0Y8t2eRymp4N iy6O0sWQFtpmTjQeAEGApxeDC7JsGKxVpOarFYFaea2sB2AvsKhpQTWJFQzP9QMFboYmGixp VCFRFnmAE4A0B3xAioWPpmAdvC5RRDI0iilBgqkQSIFCoQ0CAYmNYhM1Q3IJABMjcPpKhAiR MSLtVCBjNCCYgiZrqmrXyt8e/cczCYRtEpM2wGMQhAIosIKhveObEEgY8xLKgawENKobBzDj FVPQCyAl1QoXnwRKATCQFXFu/59Ub9DSEKqHUQuCBnFDoDAezHBfkFLhMYAcEcqh0/BkIUO5 Pk3fYpfdU1RG2/Gd13B/eVeq/G+0EIyFkEvjkpoCeciRjCEIRgRxq/Sl2AePCYGwwKxaNw/3 J7B5B1oPmD0fSu+qXohaH+uwLvry5hAAa/7GFe3wFHAdAndtnl4VQOoTlgqGICBo451R7XD2 vUSr8Z+BADQvsTo8KCfUOIUDceEPKLvb8Nk8xK9NUPUNSoWTjQSACcioY+kDxdFfTgBb4ylq VVA2nxwF5FQj8m3g2SYACUolSTBUNYIF1QMTbmwaZ+U8Z5Ptub3X5tQPKePzJl77+4R5vE7T h/xBMbDM4ZwL7HIKPNRIwnpKi85UVCyCeBULB5PbC8oZTUMug7OxP4TvvENQ7SY0JrSecKK4 NbaG1NhTnR9+xRN4/bVLW3+XlFbehknICJBBOdAIqGBFIgnndjb7ZBLguhttu4SLnqlx9H2C vmxqSH0/bOiqryl++eTvd+P3ocT3+wjp/wI6Oa787lT0O7++v8LGeO5O6iG9rULgv3e3VCwp pPLTysG6WD9j/W6cBUmSb4pqSCnUH5l4zAUxR1EOifsWJPN9PifM9sP1YWOQNHZLEnhieb5v zwALOmPGen9HJUPOAGQhR6ioRUORULrcVC2nRaC0nszMCYBh5esaALgvhTAMTnDrDAN55Dhe V1NYNLWwgVvz9IrIR0oa5OlmE5dBOtAIvMO0ORAzAMGzbsPXX/mMcT7Yc1Qkl9KIdFkkbD2T q+gTNOuvvgXH31yj2yegxJYt25gLrJAhj8xz4rV8MGmBIxgz6f88+f/TLhtE7PXNHgMK07jx 2IfSeOfP7ZOrsdn4FjBoEvwGC8uJfcXe3rRPUmpxCNiHs7iilOb9sn10xyabaPPH98sjTN0W /xTagrXmJ7jtBBwD8vKjySpbkKtEtCQQtEVLnteofM5p8Vsv2MwbrQITJAgXvPG4EzH1+o5k clp3ezmn7fhOj1/Xp6TYoEDECNVHnTnMhAMc5AVWCtcwzrSJgxo21zNARr7mdZNMcDqxdEm5 xQdp94+b34fgI3En3PfXuj9WKlBAgEtSUQIQkI6OaR+qKh/Y4Dxnk1yQMTftsDoHeNb2lust ZEbEJSVe9vvvlW19XwQfnbu+vsCcLxQZxGFksf78tViV+X8v5i5kKhhcX0ZxOY977L/FydmZ 0DZfs//gQf6f3cSkPEvmfg2KTpStC1J81ZyKt9RB/pnOHl7qEpwsK18bu78/7T4/zptPD4/n K8DfZn3uICxaSb4KZnEebUtLi+sh3iIiIJc1sIMPvHKCJlnnom+mCl5pAirmZXksClrgylDs 05viaae6H5SwNeq+rnRXW3z6wX2O/gMzMecGZvU514TPW4ww4UiSBGRCPMeh19cm/Pfm1DKk fNTl/PdFYzt3WCjkLJyBOW0kQ1eVmUzbXKn3DltE5b1RkfUYzRltH+72BiPkfIZ7B7ntfDt1 kvq/gMvu7r5w449jWnqjYz24ODgirsBgVaRWpRO1/ZmYCKlcSWOGCU2wfGC2ZNVjJd5O5Duc J99HtXMIXIPSeIO6ZXzPxg+9tvnWbttTW0yzMzMwSmZQgmmYSZTLqta+S37xqvyi0R4LQhR6 cTgNw2DwQubwjYx+lbwsWDmBE5DtRjA7DBDrbIadgfaLIYMQrRRmHOqGAgEVCIpjkTIPZffb L2OUDUdQxw+224qOVMuRLtl8T4CnYoYX0i9McMVTDFCUNJvrCY/Pe5KIlFbJJCSSSSSXFhhm 3DeFvW2ah1HWbTUPGdJkVbjqw1QQcCnShMJIQhCZAk7h23DYuRXNFwHwj0D2xsPEBQ6neHic BPzguDmDhdDDEETeDrA5gro4NLDb+wJhxQsc5ylLuOX7VgeEETujm2UDF2Ow3oMJHqbPQ0/3 Y7XmB5OU10UUQ4h1G7cZGxUzs4DYeUQ+chiHODDSNDpGDgb6FOTE4DkNwbDeIXDlBCftk7vB ITQ7ylKjHeOs1G4HFgFzRSAOiUJizjOC8bWLOTdoZV7KldOG21e+rVvyaIqTVqStERERERER ERERERERERJqJliIiSiqSk1YimWZtW+P9gEwN3iAzYfIE98qF73/e+GjQr2AUaKDjBgXvVfa Ve1W+pV8Va9oIARL3tbwHbF40mgGMidmYF3AA74DedNyklNFkfCBaR83H+E9qscVrpbCKMXz eh8ZNtiPoReV+imwLmRYAv7hgb/Gw5kBRnDKGcM2NmRkOti4rmre5sNY3dAN10nJW6xSapI9 nCjQmtMzQc4CqrowDxB0FTrV00MGI6gqkvUqRZvxIsJqHcbPf+xRp/DrplFFEJCSSSTYPugW sEkvp341V4HEf03Y5mAbgwV4Af4RshHg4JHoENm8daa5cDUaDUCjiziork3gCEEMdwHZVunA GlDSpmGnm1J9ClI/wdy4GAMBOwMm3nI2/mtkNyb5EW+nOjCLUSgM+2JW1G0tOab9VySSSSSD 5RpWtKqvQmEhpkpGImu/T3bqm4wTjBRyzTGuj83QCJ2gROQORA4R34yIEgSIRd8eGI4XlVTV UU7p3E7jsGnSzbBuLYbTstk6EzZoCI2bke0uJxUakN8muAvN3NoGs3Kx6oQHIHeQ4yDE9004 qPgDN5OWSSJNSZ8Yj2l3zM8w9hQNGBtmx/Fr2f46TpKlKUTSAbmkeBurRYaJk2xs2Khw0kSP ThJkVQ0RsgHL7kIa8w5ngA1oqZudmzBjB3AwYiQHkB7ZmQiXxMAMAFOUxBDOdd2DIQL9dMNG FHqOpslQPyzcniIcRoAeJrgMcpEtedJ2dh4K9NEQ3aAxmnECJmHAPdDl7jzHMKbgHI3RkDrq xUCs0wBExBE8IBSa05xyBE8IFYh0KrEigQdkISkEwohJLOpKDWD+vznDxyqpKPYpxjGTXpqQ hK2cRAgQQ1jBufBX+swhIDSyQkmKdM8xMkJxdHS21rZbOC2xzx0y3KY202VoqMYyeekVFXnC BMrGMISyFzuAhItKcRuN7esd4IN6HfoTGwvCTKFIKnKwPS+F9N9NJ4iSjckIi2/JmS2oYQik CLLAYOsrIO+ueEt8W1ozHWxgzxNclMIUNdhKMEmaiBNtmD6ZjAqRp641IouDroRrCDGxcPlc 6esNPRFG0RobRwNIbblggeXMskpOEkpCoWYNw3dAO+2hwW7eE0koAHSEPjBE4MxF5OUIQkkJ 6aocIG3HE1QNJpvtHFV1jp1MJu4oTaPTvajmATaIb29fCeaHXY+BEPIfXH4z6r04x9CqI2Ps QLH9s+X+Iz/tQn99jkd2VPsyvvEhIyUfTIHnhxEPlmf4P31Q97aD/0iD4JNidccYcuWM7OVD YUQ79G7Idj5Is7ntXik1A6g49T6Qn38Dqm9Pv5+KxQ8Gfhq7HxkwfHiCU2aIljY0Rtse7r3f U12te/NbeNbk78ECIEEtyiWMHDoKgp04qDclJwkgMT6JEwAsoieru02B3T9WAA4G5QTbqyPp UBoGirJGqLWTupuJSZHuaSCSDWNvdrX1/ZZqpE0RETSISQQYoMkIQYEWyjtuHnPcAHQhwRD2 5RJEzaGmiIVJA9mKUiQAcaQFXRli7N18YGK1Cbeh541JKek6Cip5imjB6SldJXOQRc6mxqjp goNgICLEB5RA+wVyGAKPn7OIqFhEuRqPCIIXPjyEXEUOU2CCPYsK/GAmpDoyrUZHz4dsumlk mJWQNFoIZVCYGBgjgM0A+kCJB7dkzma5AJ0PxME0iq/dVMdxyFenRQazsH82FkDBCEYMGEgj dw0kOqdRwjsGDBg2IEIUlJw7dGG7hMhAyxVDlBErQxDpIeLBCQTIZIwWCRWEFk1LXyr7+2JN QFi2NoKNEAp40ADxQXM+E1I6otmFcOkNokQsO3vEA1IqhYgKYCG6VDSuF7QmcqiYpRiVYU2C Qh+1duRjsgjDtkJ3AlgLKIdjDDsB8V6T40cE145DqUQsAJwQdIInuGDaMFgATdGzQFhANHdj k0i7KNiEGEREgFywI+XWcZ5YGVBAKGNtGhxed4/GKhi5IMGRtpC4KULTweeIbjYG1NI4kMuU 8tLvAIPoKQF3xJAQNwoJpwBvdPLBqRYwpgGoayiRIilVi2xtrNr3aWszFtEIAMCBFrVaoAco 3yWoxIKidlAICJQQESkVLgjRv3RAh8vg3gOHDm5TnB3BwnFBm9lJrISUkTvbebMpCNUqlQCk 2ap1ociiKcekgMikRkIosQMYlh3IwFyYHdRTKgkQqCqD2mO2VkByIwK2pWkvfADBYk1gNBBC Jd7GJkqGNsKDed6QiLCJhAsgBdioZ4r0SpYog0tKhWNhW0QbipCvJxFhVsSQjpKBoKpAOAzU HrADiQRxTpwOaB9UncuVCVIfyizT6d6mwoKaQNbYHkbiaaGxfUgaKJdtAAxnHdIc34XNp052 fMP9uNB7AV2D0BL/H9TRghCIape8waocURQPDBlwMfOUv7cO3iSYHHQZEYCW/VEFCyPHgc8C ZfBHfXvllQ48HjIkUgxGKpEv6hRzIbiKDQEFGLECfbA/UHxo0EDdL2eyNGZni7IJzcVzGZu9 RI2Qti2+vaoPjFfpHtWCiHolFOXtnas6qpQTS+9PuWNlSQq7RUhrPnO74eAJBDd2FGiKAwU/ Bmfc3/HLOonXsGmAQ4QHAGSorIhKpokALK6UBODl3+KYSYzzHv2NE5hSDZA1Fvi1etWvzJXs kq3tSVutNae3Rr5HFMcNBgbDabBOFUv2SDqpOQ+Z7G8bTJdAQNaHKjYKB5A7cYvvHcyCJ3GU GKRzoq7F4BVDfpzIIegavObX8gbfsVUJAog48KHW9VWOBYSWmYePIBxq6hAOfpszcLIbRDhD VAMoDCKSIBbRvh3x4Q5218PwTVDXKCoB1/3B6oGsQ8mBjCR1kMT1hTdhSCfSGCEGABzBBVh5 skwNJ7iEQogrxClRGgDzWK4gAoe73Y0O9kZYVOHDEPBRjielMomcUzvuOhSdzHEZ2PgAwEr2 ci43jpce/igfEYhaf7uQjpe6isrS9caZFgswraRvZuSBIwhGSEjMsBR6Dc9BDaJvVJGQQIik E6EwlwaolVVHIh5vGb4ewBWBsBHQaSCkrILSSCxIKJIJSIELipKqt40GB5Qdu7eN7KEhaQhQ 2CQzTs7wDD3hNwJooAIDSMcrBY3mbI0NpLdOrL5V5817/g885nntgIS+06F3F04AhpooIRai lQEkIDY1zFiit732vbtERL1y35e0vdu6yByz3qokkQt1g7XgVCAOxdKDi64Uk7xi9Fj2Tsq5 r3iKGtMgCA0EJXS3Xda00sW01UK2LKWWV2V1u1c61V8lPHSTTLm11NSWrqbVrkVNklNU2VvP JfJ1vUnlLdE0uVxMiMhCCyMzhUOAmrN2DIEWKkZlVoSd46gqDPYAicsbHYqpXgp5rUty9VVJ RVVVcgM5YF21SEHR12NJM/+N3bfonMCKHfOwvAYSBpw5TYmyLyRojkQSEiTMRNqiVpkCCTrA /WMVsQJEYPY1IDwsFLjdUWlRMl5OmghhJDCo4h+WmrQa56C8h2MDCTDD2asVOego+Xt46S5m zaZlFkJBZ21y9blGDBssIJWjIIkbhAsRLXf876MxGu5xv+074GhptPmBAOeYca1+M8WGFCgM Uog+Tg8IY/0PitBoPD82Co9brGB+w/Hjf/bPE2fWOJ5gWFqhm/mwzsJMhBsVRGeOUaI4OCCY bINXTmIeDPw/0UDRoWkUzU0PBDmUBrrswoZa1hkLIZzAEJoHEzMAdjENcWVNucPY25ivE1sR 6cQgWikAU4pAWBQeZ0e6yBQ5wmEtb2RMTWoB5AMjrYXHdR/dFhKe/MNnrAFkX1/MAnuutxT2 UkCMiisHAoh6IUmgGQjATIoqKQgQL0SKwAZdgAJgJ6hZBOxxTVjVSMjN/vy4cCfgEMhY640l GxnAiRMhv/KcM7E6QrQ+iOwym48hT4zkep+tghCRTNIR9JtOkwiVG/QIH0J2C5o+kGhXRcip ewgbWgUuBA06rjbBDVMEQ06VLBMKwYbDTQppYFJXLjpXtWJINRsQYXHVUN6iuDyspBROsHg8 e9GtxZqTeQV0iKKIEQzdmP7IZrhmIelOMKUHRpIMEkEaUWmFZSZEkGMeEYCHaUrSjC0TqA6c dCLqksQqqCNpJLjqDG41IWCRuJLrSdWlFMKB2GECNadYiOZrtZhRu2zTtukyNnaJb0Y9IsGh t1Tw7IAjWzRUcN57Aii5Fw0Mz2hkVMEogRxxrFoNgJiYkNpAkGGaapo3uGatMMb2Ol1VzucJ 7u47eH79GI90YILo2XkMOeMBUWwNtBziG0EQhQiUQOgiS42dA3QYawsipgMAuKYgiWuGABYc DRhSEhqE7XVxwZFLFD5vNYPGez8GpG3Eq+EYRJ419xomOV1F7k/nbJRJ5JdKgmCcCPkSGZ6G jAraiSRJBmZKcJEeqP4PTrA6CihwJIjJGPI2T7JavF6Jje3IBg3r1dV9lfmJW+2VullqTJi0 SKS0pSUtmJqX0pL7bWu65UBophVAJUZEpQSFMNmqYQBHhHI6ChD607mZ3MBUO4qfTEwCJEC+ 7ROVR5Z62tuIZxAJIRIQCDxI8PFW0qgamc5wpVAZKdQh5lhgIaUm5ttBMe9oNvBGpGIKxBgo HMt8/FnR+Isd40+IMCiz9oMCjBvMC1iH1g2qq3TSKJzsagiQ1oJAgVw7qTtF0DpT9fxATYUO gNRuoo21VSsCBR6hQAnvQdyoRUIKh3doIp/CNyjfkiPRY8RvFjovUPMzbXQWR8wRQd4JEAhg TkW34TckdsGmFBQIu05Yo81u3RYRNZGW0FhLROHJQE1Hk+rVBuq6hi23NHgdpRlQ1XmQK5QS AcuczdceVAm2hIdCaxTRaX8GoHyqe6bJaI+AvUS+A45NnNnv9o8derMCEbTZHMg9D0afZ+Dl avCae0xzmxHqckrlYIWbGwYTdZsMF6jSGjRVEf5cqJIvY4A2TPE2KBnANIGmKM8O+U7F06J4 UBl7xY8Yk8afVcoMDKg1XwvkgwgVisASII6YABUROcBNiGwVDGLffviEzhhpMRwqdWZrd4nU 57taoIOtFsH1xISLJ0wk55pxiDi1q0AJ1L3wbGicKpJhAUTsoQJCTghDa9KESYNrqyMxwbFY iUDh1VyEtBndwBiWVt08UmxZmCaKwmE02easZ3JSY1pAhyDLe14wZmDKZN8BphpkAMbMcr4p 6wZSiSCdJW+40qkJLtNBE+BJZxbs7M8JViYzGGVVFUzMGOjY5IyE00TngaOYEswmbKszA8tN iZmCpKYY5U75nIWuaDW8xZCMEx2NQlrB0aN8RDeniSBjZmoMfIkJaoGMi2uwx1nTLAnJjPSm zOlnnVmYNYvm7zk+3CG1wioIidL1TLHIVZ0magv1o5JWbFNN0VlPPJhmxFXGaW2WzN0SfLIU G73kygjd8E1NstGuNmTqiANIO7wUx4NuSDG8ydmzXCmeHnCmhsYKjbWSA9WgaWZdRt4aZjYA eCNRlDF4zDCQwyBy0qmLkaCDWHJyxAo0Iak6rhILCCRJWoSlajzZEiV7SyVUcrJ2pzpjjbab TMMm4oM3z0YyTTA45nYcYclya6azox9JGHKKYRMadAscWHTgXZi02qmzGZgrbAJA9QdDSdbi CmQ5bLAs1ibF4wIqOQWibRNE0ZXJUUmoJ8TiKlpihq0cxLXxGoPUvVK7YprTwcHZshi7UQZS XAPtHwmluYnRbOauR6QaTIGMCLYY4oLqQAvBDI+ic4DJRsVzDbNEyL07+QkpbBGckXBJmgyb Rn5lOCU6D4uzqcsZGRtI2b3KbzVyR03ZGmal0k2htpt8vtdT4youIrSnbajC4XJx1J16zmHT dSxZLLy1150TnOC8aXLpR8Qq54Rw0JpibabZIZWrFHhDpQrXvMqJEk9Hdy8oFSmqpO6Muc4V DU4GogQcRxGgswKStzCFwLg8Jga1bDsKDVITEzNpgYopcpcXc7tCZpQ69JMjcaSxkMQuJuLZ AmSAZqhuRKc2NthEBsGOuECSSWVAzNiaAS4G4yDWgITAhGMXSAGAUiywUGvRHUQhISDoyHAM gYhA10CJEyQ1m0DAdLBwBg2A1gFwoB1iKZhgGvOSEgXgQiq30lKliSJudpwpWESRTy2OltJF JEJBUTYOkCBAiBIMJAYoKUII6Fwp3YyyI2R7aK6sAzC5pgjqTZsBR2wah1BsOoF5qGj+KZQu zRihoguYqoUaasECRiSSMZECKQGQ0fBTBtaqA6CCMHIXJgiIX7AothEz3xzOZmPZK1lcWYGI attGJCrVUZUI+0MeeKQaSZy8PDzkMHoIjDqRJBrF6Zm0kE4Hx1IcZpJBz7vTUCFOxLCHLHhl MsbE1BTzkzME5irPIGLtaZNUkQ7T4AwkVTZuRNTvoi+7g/ra6FXYtPKbxENvDADhUXIVnzYi dAIlrgJZUaxDnMtNwOBipgglDQbBNpQKnYzDE0FwBOVFN5BGxrARiBvGrIPoppBe6kJ4wp6A nRQMW24ybYuIHDgN5siaR3ZTFCLBkCKkYhuFTkMw9wDtvbnHzjW0hBJfYUM0XqxYuBglQo5G LoZt23V1HnxlI2X3bnIdkzsFGhtvIQjItBIH0cKRuaEoU4dxJq1Rw9qImHAFwCzjAGjRFesZ IiMglwJvXOW8OkzRUOmIEigB2ROCBG35IkekJmW5is9+SOxAxtjaTfmAa2vaemLPaptt0tFe 21uu1LGhMAgtud2AYGRDgTb2KjRRKp5PCihG5mCu2TdiERzgM5yilCDogxNMiCPyAiTIiyKQ isYECEig7+PBWco5Q7/Bjijg3x9WF0OW90zcAbYSSuTK8EjbYSiyl5cAuIgkbWZglVgYwYxn i1sIe471BRxBYbZzpWzNINFNY5LGxtmE82Q1ugrMet4qzUTR0CUNDdhTGuB41BlZQyK48DHc QgCI6Xqkg2cs7t8Lb50LJtzU0uwTJptvbVW91Y1qKKNNgZo1hp3WjrvRqlANpoR1Etpm1hQO R8QKmGtW6t0ui3QKCGNDYQYMEMTmVO4ng7HKe69gb2Odwl3LjFdCQWCwWKu7aB6UTYgzecF3 3jLqhZUKd/0QTiRIqtw/Ca1EoMDIBQ7a2O0KwN7hgnAI+vEep6jDA1oJ5fcBxGOmUs9qxYYk NoZKsHOB5h+AYx02kTSCdwIJk4OYlUQpTE7eAUJAQgE91Nypm5vT55LAWZIabbbY4SQmO5MY lUI4SZ4vVR6aGjxXzzrtdaacNGnOLfZV3W3Kr7Wk+/ImyCmG9QkESzqAYVsgO9CRCAm0IPTR 8dJLfHlONul7fg66tT1Jxsy2nDjulvDNSrHwR3QQh8Jl55YFSVYTvOsXpJpZTaGjLnMMkEPw kjGmBZBsG5FOZBnLEHQdenCTFoXcJi+mQsID2NJbI2mMGyqpjIJCDSUVIJKTEMkdLAXgPWAN +yC6oZd/gCyawI+OEgEIBAjIqF8CRDMirIyCJApEghvxst7ymtNbNsW0ylLJtNk2ZWmiSyls s2aTbWE0a1drep8VfJrutVdJSsiaDWuiIUrFVhGAEQYEFYaoNoIGFOtVIRQsiNDpQ8pSxEOC BRCCYcs8p0YKKvQ1GjTQEMGYUIUkGJcjyWsFezCF1TAhGEiLFYIGpUIA4GNJdQCDQxIkaVSh yRpirQ0gWFogQAoIAZAUb1Q1DeVLHEQzTQ5JxB6Zzs0HgO1XfQKq53aKnqQ+x4aR+QNgCc8F nClcSO+PHwEEIK4IIhsEkhFkKdwff4QMhtkDMRwja7gFgtr93A9OA5QO2BgHPEAogESKQhvQ oge+DdoGyC3gIaDs9CFe0YBZLUjBqFq35Cpb3W3wrfm2RVzXNS3x0ppYDsBeZoG6c4murAQd pRh6TiH7h/MdG8YdLwOAL/bJgGnYYX+KSEj2hA9KCUBwlvsXAQTne4Gg+c3IYxkKgEKKSRQj RBxxoARZt1MzSDSMQVieQiaSDBiwYy0kokISRLDVjDSfEQcRHXFBuTdrwXQKxVkKwdxjsx2I mUC8KO+GmKHUO4Q9dTn32uzA0p3w2AKbEEa2GSvu7wp4kihFcFGHFyrAjnoJiPBP2/ZUfiw9 wW+efD9PdfMr/OHcHgS0NYzr999vTHXqxztlGZJ+//ebpi7xJOBmyEHrOE2LIkC7dIoawrZL qo2Omrt3uMZxh7nIrSKQY0xjPr0uh0MFwM0221GQqQoj8h+WkLEvUho8ti9yeN98ccZgfPcM Z9u/uGVJtDrgbt0lCXlqPumxXGOod9UduJtgcvo88nfyjeoPaYNDGlACT9KJRiKdhRJsrKH0 vhosjwXtsoo+HhpOE3igk1sMoG4ibpCb2G21fomeG3DQVWrNatVHOqc4A0ArulXz3wqNnzeu lO7PLzkbzWv3CmvpDEBygCfuDN4hDWO6WLbI1fRZCxvnBja5mQSi8wmIQd+O/dMbHbDg8Un1 2h8b2G4Pqm+9xkjpdyxy038n9Q/dbGA5L8b+L6BDeVHmTdEORJ6hIt55I/2Ifd+ked7GSc42 OA/09H0M1/B57X0eFKLT28HbFygBfFRCQFCEVJEknQlCodUEU78S5CSRBhEjAgYmiwjgQG8a IgIdxgAHUQsIGSFgN5Et6bVeolo1VyqvsS7S1srSRCAidSpr/QXIgkgLAgGCqWbmaHXb8ZZE t5bNrW8pinUf6IzNShvIpvAQDeUgkACAIlNkXRVCDbQNAkGw2DJ3kBpEdyIGsG/XD8MSgilH 8tCUejCmcaukdohY1EIrsIkGDIwVkIRQwIoDpAXENohsEQgI2AREEhnMgIACQbmMYgJzJC7k CrRABFQFZQFFIR89nmY2OZFWaM9muBJOnVweXH9LcTtnbOFE1QAImqFAxYAOEgEgyIhGkKAx BCRSQE2M8kjcHI2QbI1CWkK3akgtZGNkblQxCGMJbeTBKFbPyjFfcMgkwpLCZJ7uWuLXN1lj bel5iBq8221yARgSSBjJNAiSCIUY2FvvEBzg427xizaXCKWqgNkgeMGDGuehcg2QpQiO6EpU ogDZAKgIkGBKFplHajRxROR5vQLh3l+YSCgNsYI0WAMwuhARIPRAiQESgRLHlMo5SNbeWb8J U1bxk1WjarG2qiEkJCEQkGEWQzIY4Dy0CY2fUksduwXuoa7/agB8U18b9cUfx7jWImI/pSjw jsf8prqDjDlUTDr1i7EGgXlBPQ13US0GsiLfnt8caoj7LUX0LdWfSlcrWTAGBJSoQVQsFtds WFIXgiXgNckVhAFbeqeATBbqjmTI+G2TqKZ9mNvxwCS524F+2aAU31dbRtCMYKA0JmJkeUBB pFTGwJZBP3PhLTaUPPQUkIKREG7CFkG7FC9oKLysBwPSCjSaApOzG9guoQLQJdGMIOEVkKJZ CHEiHiPAT+xT73tJ5g6ZA/d+rb+D2tNiTvEHI/rFOwZ71sXSiicCcgVzaDC0ukzCvF4qux88 NL5UiGwdAehLZyqvQ41ACKh5sSSMkwu/QeCNxVDBB+OASIBYQ8RgK4MFK9O9MTKhhQSHeQAT Ed2mXjOS4XaFVRsGw6oCaJmCickikisjoQKgPDgtFDhM8VD12ti2o404BOcgsipqDN0AESEI IMQhheXLlEIDqEDIgE+KJ1SDRKG/et+U8HPXLZIWFdba91irK2xj3sVnx+MSowG49MI0gje2 tjGDRGQZcirDISjQtGxtRLHqapEUjpQNwsQ7JMGQuGGFUYMe3qohhFAaqcqyQgWwbUpaDQQ/ KwJAD1WldKCZNtEllTZNQbzRdFAdYPHygBOHBBLw3QA7AmQiGs5E2GaqN0OSIncQIhUZJq32 aphJti0bEa1wA2H4sg5xcYGDIPswE0YPuDFQkGRAYwViCwAiIEHQBEiIWFQlAwXWo/i3jqj6 UP2UnhgmiIDCIjFQiA+JU78JLFU2iodBKIlTgUR1qpzSm3ALyIgPNUB2rAhCBJFkBJJgkkTI lJo9+vqV9GwPIhB67vdIzvfNtcB197kq/qPiesC1pn5gaehNgqcfisEtScpAGgLVOJOVHizk 6Z3DxEFQ29VUCo0yGCUloKUb48/soJcGCKUKhBX9kYnX5KMHRewb5HADTSkmOOmwuujr1Nho OkhzFUUQCrSAaxiSkigoXWAGMKI1AwiC6QRIttMZJJ2tze23qt0ympe6CZVJsRE3dRwzhAd8 B98N89Cxw8/acIWgFNy4FNm22NDbCMJY2rZV9ukOP4/r1ByMiGsInZVA4x3Ihe+0JqXQglSd uI761GB1DgC0sF/SMRcN6wTDBxCxkBhilu+eLlTM9UVyhRFUaiURKiMxpkKsUoVFmQkfrjlg WDiNVXJiRN9JzE09PzL7WvYtaTUyUrLWmWhoMmmSbJtqKNSSZShZlswhikaUykKQ2jRSRWMb ajZkyLWmk2hJASSkk1VNrFJqUxpJKVTa00w0aJVsRVtlKtGtoUtmpiMiwgRCESMRgTI5daPr AYp+9p26hKi6u2WBybUnrKGvWAYicKcqMSAEtNa2+b9t9L1fmqCQyxYJEui83KIX5CDJjVCg oIQgRJABIMCQCEgSJEIQzguQqHUmlTjIr9+60doOfhcVUPUfcamJY7BmKoa7FzYSJUE4x3Tl LBeJpQTBJZ0MFwd9hNEANIMkFDRcoGQSIxCQO0bWARhqlCtjIQQ0AAFBkIGaBtiEZCMjFRbi eNPR4dW/saQib4ixUNQdjs8cKklBQMBGYAUCpvuGhQBH1Fn4gj0FmtHzRyV3ITciqKTuu2k6 xqWqa2jLcQqSuu7ja6WuugKNABEUNt0Z15A/nOg7w50OKesMcdEoO/LBWDAYfZE53lD6XoLm 6DsgPSnZ+PU9CIpq54K0inUI9ID+idb1ChqHP9fsh+hPysd7k55HwAFuCXGpR2kTciDH6GEz jPKHejIhuCNDw5aEIyAWtRl2WhR8sEdOqhS0W0yZIiyIZynQItEAiQBAkJGQhictLQcjQURX EhfBbG7LB1cIQhFS4L5IBt9yHZRwATUAjvBkCoHmMT84qqIt0uOhUCCUQAKigfoJ+Hb2jnA5 xDYdW6k7XUKTdX3o58r7D6vuBRfX+qdS6FIDE4nI9cmc+5KBQhHyjpQ0B6owCsFNJEGA3Iph EiEJJAKSqIqEalJGjlNUOb7VB1cF7kskmYiZam1UyEgqyJIjNbPgHRd/ekdgHtQSRMa3CVux L4rdxMKihDRndjUrG9bxQPi3wUcc7lBPcBtFSRSoRIEBRwfMi5GesQwZSbFggqFoWxRIGoIp oFQzwQ98F+yQh1CoWx3B1gFxQOYe3DwZIJjYjtFA1tDvEaBIMEOA+0sWDPQ49CCk0ihSSSsy FRtEKlEmglGxhBpUoUpcagVA0mDQ0svAlO4hGOBEwVDCIpQUABFlIeDcCGxuBoQRxMTYad6t KbtFT4/Yr19pRv0HAyNaGfHTWPWuEBaA2IbQbIiJjn5Unh6WhYjCMCRIAwhIC48J1KJnDpRW II4DYRcwx9IURTyMA8hmUbQE2ARMl+4HYIkIwEdJuzHYgcDqSz65DA7hRacskP35aXMga7x2 EQLJvHw4CoauNN9VMYqAUdiUqSRhE3FFQj4aKuqEq86lGllWo2tvF9GrbL8AiiwH1tsVe48w Ozi8FVVVCdNb/7ae34C4aEB1AcNXjjSQIRACJfLe/mvkwCFLCVKwQo7pyhg/EaTaUpdJO8VY AypuwAyYmZx1e8yHocGqWbrNFU4E/D0TYLjbGxCgPEHgKggeXywHgY0CyJEQgLAgUeKrfQCt pE3pjVDMQMwBaBEz901lQndFOQiCnUCQbDNwIoUqFrm4EblGk08+3tEhLW+Wr+MwbQMf0H+E 2/qekfe+cHH6xA6HpA2/N1UqHDXg610Y3NFSWpos3lFNEKiXlMolX80D09L/7o/MDSe7br5j sWPfyKqHnmB4jHvLAfmA9Rr5Na2QxrESRJmkkFMuA6CdncK+0O241QTZE8JkqjLAcnIwwQUo xclzF8cDMotFcSyvxXOMcDA75QuKy/KPKUdo4vio15EdoO/5CHQgxgkPsSQ9uer90r4r+0Qh EjOYQ2HrmPFwHKe4XvE2hoCJ7VqU96BCKGoHoPuG0x0wxAYqHYy0u0NRgmsV2HGay7OjLCZb hMBwYsApCBGSMGDKpXveVeZjNbynYGiErKlLNmZZrXjcMJFDKDa2ZpGT2bU2htyFUHDnQ5xd m9tucr4rT3ylKsyo2oT65vxkgkRgRkCcBSFIJBdwKLZQR4qxgmC8GHaLtkUYKdpTL3tEUfNM 4KFmtIc95dBYuWCZRBzth8chiwJMJg2JtDYbWxpMRRclijAjG4idLKA2WVDY8eR9wFl4SQkh CqTXY6MkcfqD3jJXo3twmitjMCxP4ik5wllSBBKHUoxUJ903+LvYm9z9NMlmWK3x7jwl9TQl gntm1EefXRU56KPxdVcRF34PHAyjaDlYitY93F0S65dqyfakvXgtXDd19JYLD3kUPmCB1CNQ U0wkCOxBhagagAVACg3YkGkYMUUEYgRVEI8qRAIK33MDM2xKo7hUOQE24AhkP8JsQ0cu3sgJ oKQ0lhSKmA5rOTT5gf2DgA7OzS8vLJqurLHK/96KjwjlyBD2MTjZM6ESjdTkF+E7BznO4xTM KQlK1EYn7spGA4QkoKVGCoQUqJFi0FYOoxkGJttJsQerTTG0tEc02gRA0TBCIZYMjtjAeQkR BDYIINt4wb+q0ZhkhxQM22a1vRrBjgxga0MgN1jaGLLlVx7aSCpYzBKkIRIQ2AKluRJZWzAa WMYfJCL1kfLJNgiIY9tOOIg19O1WMKEHCgWU0LjIqtoslpYjaKDZUIpZGIM+47lvLzq8FGqr 53zu19ktYA+WiMQuWjuMgujKhmf3pAA2KmwY2VHRQUKmRFiRaICSCVGngcFsAl7Hcw0et3Wr 4SAYgAnHmAZkRA1UrcEUIh6LEKDWgmBZcGNKJUaGQbQkRjYPTCAJGmsaQikmsA9WsSNGKQet XqWatd2LzKIYxZExgoEBVFQMqGQWQ+Yki4pgYALysCQGwkYqmAqsVHEq7W3LW3LbytZZkpNs rZfeEF9NRN4KIKRMZrX28jIhgAzB3CESR9yCe1YEX1YgWC6OgOGeUiE8FeC1EjtMiIBpIpjh FQCiCCsIyCiLpKomcWsMiI0oDph1ktnakJCUSqIaE+EaSwEaGFKRAHLAENttjbgAZiQsCH9/ KRAmouIUdUVSk0kUTBtHPRlIDBMYCawELDgiGKDyBcHamAjln2WwInwG6iy6TE1nrfNMdNV9 WYbhziUArnC0LQBIwRKmu92yS+2wPegKGBMSK0RePEQaUS4JviNWwWtq6rWq5bbq3WyUDClp YpqbUbWTTKIpJ69LvKmKvVJeq8rzRskapiraZeAVCRUKGA0BBBusFGxEshZQIjqSJSBRcXJe RbJpGdlPa1QEGBTQu1AIpgFBEFBj+ugQYNpsNJxjjHI7RDxUzQbjlOU5Q7NnTmo6u0O72vbm /Ds5Jwj1s3rSH0JAz9n84zKhgdkwPEOb64aN1mpMINejFyNW5ltwtcSI5HBCHHBx2gZckjbG 6ZkhJceem9a4GtN7eioVZSYra4aTEPHCmcVfhs3tQh7rDt8evnl+pT6Bx2iRIpilthdbiQNg nDw2A/BlguZAJdQ1q7QRMaAC1hD5x+KCuZOL34UIpYB4ioZjZAwDBQJIiRhIECBEKGAUREMk Ty65Op40EwTWcwA/wgiQdaB9A84yKhPtEaJCoPIiURUMwAg3L0NIJSJjiqGVyMVgJFCEFX6y hpH0uEFEgAm4fR46Lh6M4zmE0NoQYjIEYhBYLAc4QuNoqdOip+F84dsv0QpFDkikIsFhA5DA ETsAJKXADu6m87A2NBGvSIaxkAXpMSYxJAJMy8EDXgqmjAVhADAQSoRihIhqEkdaIGCAI6UR v2WhFE5cSxL1I222UfysScedLHyat6gI8wB9UhAXmcgPPadKs0RoVB0AMZAjnsaRbwq6i4AE ViIRhN5d8TaIYJcAwGBCK2NKgjsIDIChmgBcUUC9SDpjFpIgEYkSRwbQzNSLuBFPWq98PPlC Xx33t9P8h937fJn248XsB9hznOHOHA282c8AD51eb5+vmXXv5nPNUZfM0HTtgfmdog2216bn 2yGOQb+VBjS58LEDoIJJmVa2EnUxzfO6JsMIgkAE51RDhEgY+A7hJJWswHjLHRCBJkWKFIzg 4MIQqobZtDNDEJYoI0IbbiCjEgYmMtGOBpjtGMiKtgMS1IKn1FiZcJBp8pIHaYHLwoQYRe+J UkFWaEBGxwYAQZGEGIkjGnKyYXbR0621Xv/BMvPPVuqgwWNYoOAKDDBp0tAoBkUE0iAgRRSO uMF20DKoPQyQGEWEZFkQIRkA0Jmg0E6SaqaqVT2SZY/aVDNC4qZCG9USEYgnWZaZsGazhGP3 EdbV2SkWSpV71vXxTMm1FqWZbJIjaSwyprzb45fohAOoV0AJcpILnCEGSQdM0h5oOkQNB3Q0 pxBw2ISUG+eY8JQB68HfSfPAkNuYCWkFE+ginCieR0HEW7vXn4pO+b1mo4Sto9HIc1NCdnyK 3S2UQ6RVapUL3EeQLtyEfaosI60xAEs7lUOOJvoFU6thUaAIWgB64gCtxqUt6caRQu57nB5g G8xoMWcTvKA5WbGA8hgwSSQhJ/EXF8s9/39A/5R7oh35Houu6HOQLZhY+IUGeGocUbLiOREf N+m/l6wx60D9BRjl5M9xpULvXXhmZmYHii138CtE4MBPphVIdMhCRdMMWYtLHKqIm3KQbW1m 2NpsfZAcg2x9bSs2+3MdgMY2JbAgwGfeYECnP5jawEd73rA9uediNqa6ZiZAlgCg1hYSwPOr 9eGRirRdaUcldzkFjlJJCCG5GDiXi0IzgiZWQEqiMoBKSCMINRXACKWMr49aRMxDBNu4QIe/ AhDg4PTeG/vcNJ/UhCFpip3tjyfP4vVMLj5BQUO7IZ+aVT3RSOgnZhGrgHxIix7LyF9JuKcE YFpH1L8bdwIDmhtODJYiL7vyDftMJQ0/OYliuLx7lMrrS9srrS88lzwXpdC7N6G/ZWlpXFYO TppoRlGp5fGWth2wgNmhHbURXTC53Sv7I87UKYfrxhTXECr9Mb975U8ozWDqNBVmQ4IVcril bakC7VPpyomol4Z++2LUKmUKIdfUPukqyJ1qEoB6BS4XX9gnaMyhFiKcQCQzOEAwPoD9uAUe SPmDx9nQ93hryqG0M0NqcB6AhicK4CJlHUMFOJYQjAIkCHEbSgMg0gibcC4RACmrGJqxbQJj u9TIxxD11p3rHZOpkfYtaCdVAqHP6CyEgQjIoMDRiZChGGsbRsOKcYYcIepccU4h4wtDXzFg k35UqchEDcjyCKVZ1DsOo7lZ6MzBgURN+LC8Z3S2AIpuw96FACZlnTyGnolIQ3u9dBxwkmOF LFZBHiPGYVt1wYMBpoIiqkHXHBkQnlrToUNMVHvwjIRViEdh0gEFNo4i+HSgaoZZPCoWIvco mWgB7RkkGGtEFfTO225m6PMHLw7Htui7WxR23n5TdD1WJxj7HXy+vebwFobeVE8oQ2DZLzkZ 0ByF8kwXYNoNH5uMpBHPICJ0riHoeXy90KGNsqDJQH0PJkOZdO4ekdw2HOpv+q9nvKiG4bHW ohq9PAN8hJxXKbZGehK58BcSEkuB4jfLprCGGkKCjWQ9UsphBUvdpKEWkYoRJKIQQtSQEiVI bIRRoCHYsN7wglNLUqKTIWSthmUreS0y17za5PW6tIKJCjAiGgIhgfCIEBhhzQrSoxKFTJCb qUGgoREkSbhQiSC1RAiPiqIloJFNNRlUUZBkjIKEjBjJJIoyJwQwENdkXNRT9LPfGIiDxImw RQ4IqhBQIJ0pUQowo2mrAGFF0OtMGARYyuNyA+0LgnYIawONz6JoNtoTaaZtiw5wqBiO0gQk SiGxvQ4niQYJlYh0F2uIWFWWmvPXW1teFMfhbtdrerkoFhGaKEqOZSr6wMAMIGgQEhW17UpY s1J266sVaS9zeOa5piKFYTgGDIIWimxgDKJg0U3RFpoSAyIj1b9CKZFwAgQxSFpUA0BgtmK6 TAELyWGALBvEWgHMMeIeusHYgEvMBcrYx4gOuOyYrCKN1FyVMRyghcSnQcKKJiaWIGJNGaLp zrcghAUaOJE08AMINuDRjoDMVXAybIjuii2gEGCBC4VaxaYLCDY2oq4IR6BhECGxrDBlF7pl wwkhVx4V7QxjQ/aqb9l+DSwK1NEcJGhRLHDQEc5Ea1zSEcOUua2+95azV3K2Telm90p9xBCu rrKvcz1dV48a+v3aNZRNQDVhUBKyoNCm2NKBt1DSVDVFdGUA+eHk4PaDaDgmRwiKQATjNHbI RC4F0+dWKiR5ZzEVBsqQiGimkcYwhBZJBIm1QLPEcRL0WRTtLpbqhYDI0FNjVAAuyAnUILkl tjHleY2r6Rh5wcYKoEdiGJGm9fXeTCtLGCcy8hVrQliRgjGk8FoNJLWjTtQlCEQmhsYiILRI ohsPQYiA20ZFEiERXFEdI6A1hbb5bPG870kECxCqVClQIqGaXFUKYc4YWsB50Fe2BpixAC/s gRMQwdnCAABYR2wBeqHYPtGAhASIQgSIXANgt0kjHWAjmiGSLc7ZgfxgQFDFAYqGIlBFFAgi hMAj2gAihwFxzneNIDdAgKbQkWcC9TAhBpNCAKZgJSJ3oqhAGJvqGtNQIRE2OwCWAH5YlrUm jSVpT6F522No0WiJEZBkVC+BBv2CgEWypaApB3x61e3rR666Uu63FLa7VNuhik3duZYmsWyr rSpum2GhDFyCKo6yiqhASFx894AAYJp76gAqBgQEkKCioohxpAoBsI4Z67lvX/P3Byxzs7c0 uVg8oPckZICBwoG0kl6F890fCfzi27zo9w/bLFZKhRyRX75N8syP/yPAeAwMTyC7TSewYK6z 9Y3XuRck4/C7+GeWTPLmnSwLmLJNS31OTWHF2dhIHDJmtFutEtKuvIC+qxJBBYwS2e1hs4yH FOLF5INnPtndrBJBuEZMmZgS++EXyuNMwMM4rOsZEVcLSQ488YF0OTWHgnbcARTSlQ1Z1fJB 9CUSQNP53WFGnyg4rx9ekaSBjOkNSXgf6irSNdsVH/1mZgwgmAxN1dZSwgLQ93s+OTpVng9W Q8lEiVDSVJGl8rEm3VB1kPiGYSvLAsSvLJxpwCO4YEvSLjOsxeZ4h1pfPGTM0wg1CU5hJFC5 KtsuA5OZgOxgWK2NBDJ6NlZBiUJ5OjWEvL67Fo0R6e8t0FN7nFhNTTCaO1TJ0tIoCSChYuol ChZyvEqiqVttZmDpxBpVJ19gvhRLF5jqEEhUUEsdODUGoyYK2hDwkk6ogYm2EGBCRtvstbL0 M0UbDT0DZiU5OzrtrkgctjIMGiON7HK8pGOwuVUY8eVwK02m6DfGXowbQwQCGxSAyO+MAVmZ 44sbENoDZV0zsiL1k81mQ8Gs5G47mHXXf0OvUy6S4Yai6XV69OC5+XoKDVjDYA0wbD73RZFX C8hJIQoFdhwWHSbS5Cjx0GQxEwEXfRhuUsIb51AAUFTweA5WSKtRhQ7O9CuSDBh0e1EBgCDA UUkhAFYMWi15beq9zd7r1rfDX4evQFAWdwrpBuBQWQigc0aRgohBvoM1BaYikgKhdueyDxCo UAc5peEkaUS0EdEGGilWRoIq3FLAItjEhSa7KrpCWtY204BE00NtVSKlii5YySXjgxwIQBoi KUBQRQCEUsMXEM8raAQ1xkIxkhBlKvV70BIUiB5Z9j9c68kxmIN+JBeXu90nhr7F8NfCeTbv az5putq1FfZJ/XUQ/dUQvac9BUfF7ae9LEqAj3spQ4pJH+SNuoH+07P+vcb1ensOX0/N+3iX Ty3/8urcs2+/QvxK6fLZfN9mjyUOurvJCP/ZbkvWkI29ycb7SVMMXz9pJ7+wWDecddW8JqK8 DYsVD4wBrCQAIjCYI1XTW1RUnLa88u19v+OzNlFrHAhHzdjbG19mQUfzD4EJPi0/VWCwySRE JI55/QOv2cjqkGRChBgMYMjjGCOi3yHSQki/SQh+eGRDYzGMZh+rhmN44dzvM1r3Vd18xdR8 u868miE/OdiCuHOnN3diDHTu1yJUSYsZiLjpYXmOI/gkXC3UGTEWsdxJ1dE0hlU/cCzM/NEl Pti2lO0QvIBpQ0NHnHnFOwaiZVvtgdME1DrtjWCqahVVBJUkYyQGTpjrrudW2lwOrWm1pq23 2sj6qdLR8llDS2CRiwM1MgRKXUM4BLIqUZKBGK1nE85DOEbjOA24UJuAVrJWmlWzK1pNKJa0 Ihxa8cBAxMdZrNDETYcJawqaSlcBjAIHFmoUZ8CWtCkQt3P6HhWD3HjQoHBDFBOGPFAQ/dkj hBhBH7o8i7W95VQDPrMTEmc0DUANqbsA6e/UZ14np+NI+85Q1Q8zTtPZCUQ233ksJIEkbwQy yH3VtCSEkJISQkhCZX+xYHsHaHseqGI+lIpBgRoa8ynXfCg7xHqBTToRUKCMSRiAhCWVIPf+ hD8Ip8A8XcGiFX4IaIXQaIVfDDMPEjegEaeywbQPCUoBbjvcG2+Jg74iwhGIQVBYLFdvSvjW RQ3JCEQBc7tg6lwsD0mRyWz5FsGd7CXIKGuzCg2wSU2nAY1pGhhDWxJBcaMJpJBpgIqSQiYR IBNWCAIYto3VHCNj+8XBs20JLQVPVFk8iBC4KhoV2IDEHFI3ePZstgBtNaiG4cTe9OixDkkK EDyWBkgJh5vAS+YZxsIs/rNHun2h7Gtggtw+BXBvhqA0VGWVRkCQkQ0o0SETbI5I0xjrczoH CYxsXrPmx9CULD2eepwJ8UvfwYLcsZ11dciIKMkZCqqwXskj1PCIsOAZGN+Ui46QGEiif7It Co8i7CYxNgEYGDKzV+EVt6p5JJsZaJtdXa3SdXW5pl0t6m17WlmslsTruuRDRB4VTIRhRgxo MBqsklCRLgNprOzFUzDzeUDYSbpIsCIsCUhItC4zujhMZDZDdyS0hRUY4yw3BYxLnBMk8UhF KEtRkKoay53EUL0ElA3Skttm+NxaTfLy6+OJub27fJ0QkSI00PPGKHlhghm/fenDdmQaG20V FPb7LWLhRqIjTbZpIKhUYEaGyRP3guXgEuU76m1Q/uhNwm+5bpqewVUkoYJknBJFUoP4AgQZ v65QZlipA3JqAKDJXl0iks+vHRrgndjO8bjBlWtbIT0EMglzpoZ28749H+V3ZaECDzHKGeaY kM878TQNhJ0uaj+UsV+I7+2vzmGIbL+3+0/k5efX2PT+4djapJePfvfoZjAIbGV9Fn3ARm93 DkJpgqpBx2gTpH3LjLv/GtC9LlJjgKn6SrCx/FK4G5MziZ1/CPJmycgcWhtt+u1igSNmKO5s BuMJbufSrKDkZMW9m8mcFdSCZ2Ee7hizPJuD6WFDeNTBPAUfwIeM3MY0w/B5S7alM6kHQIOD gPO0AKIClY0LL1wXU4EYzuxRgUynBiecj9nwUzAY2TdBJVFIeEJrlrhQPWwbCGsgomcK5Z0n lauBjhRoyJixMyCo4imEgdMwKrazIfSQ6KXdVDBZBgJq0OTljNOf8kdOANMR0ssZpYNEKLwH yyDWhznuAEUQc93OxJB2cjLc3HSA+jru2bMxPjdFe2azNszwFSQcYuDi5ycKJINXc33GsRqb qTIJDkkXWoBBC4K9HdgYykztMqhTHqIZBTXO5JsZlatIAGUmUtaOYOGVlqrnA1fHLFVym21L uev9eq25b5UuY+OPy3/QiX4YAfhxQacPcoVtJsQPxXZTbf1flzwutOXarnXdzI6KWiQojfZN /AsXVI0kUqaDw8zNNkNucnmeI8wjxDXqePPIEjkabg0HqWga1IDkT0qOwKMq9neGbISB6hQg ejIGc8qVBEsAGGymHbgp6oGN8knpxcbmsa8ZxrubahCCz0pFZzlNTLKo49oTGF0qVcuw3qIB v6Bxb6iPBAAfq7NzlstribLqVYrba8MCzMcY8aSltnZjafonmkMYBQfC+ESiRzZTYpXIe3vm Bm5voexw222IKzkbaG+YTXkS2UQcUBqtgP3YXOO4WFk9f5EEaGCgQipCKwxlEQCCne5iCNK3 BOSL1Re58C4FHA2HGWAELHrGQr7RV0uRAYY4FkSAsBEsEAgxHzcGEcIWJEDBUNZe6xAXQm/R PTzRocYl6LBBHrIXkiPJyRGNsiTemRkaArdLbj9N6E+NLedNagHkV09mi72I83G+EgDZ2Npi 0GCxNNdp+gUGwhiOTAkhFrEYQh3YB5Tg8AYgZBAhmNbUEYOhzVT4IoeX+bMwyMgYkHNQm3Tj UEg+/M6XYEwwduZdmIYDfNvqt3BixyGOwQDZSPvalTiLDkIGBQUDRRSUi0DzJDQCJihdIECw FJQxoDB2Xp8iGA84KJAT98SlADYh0BeA9CKnfzcB1Ao6WKh3yg9dK6QQxgib4ejatfMla14R NpNlLVqUjBYEJEkDeYKBkinIZ2cGCSLIpCI4zOEvACAJ2PWdT13NnDDGAinFwDNTWadtmwRV NcBLwAAlikCoFjJZIxX82EIzTwdGLQ05EUlcCiIxsGLMNZjQD1CMNMI6JoY0MXkYuAXuKGWF jecTBzxOCTA41Qw7JYagujtbfr/og2ISCkl80xUDd2tOvbYiPwmwsZXphIkIrIJJCNEEGLZU N4O2FqHOjFsQpcHGE4GjEOAAX0xMXbGMQk5CuGFEGj8xTymH9iHh3Jr2vmKqlJJVLIwj8xy8 3K7LWubbS7xtumZrTU103EnNu0hmTi2+77qr4fVPCWGCAfvpA2EJFe4REVLvKImlQLIeKIVB EIpFWIHt0s6tqNuMQ24qEEKJIMRsqOvcCnewjuAGAiwCjBNT4ADs8RmAATQKGMUbpZxUGE2o Zde4wDFhAmbpbv9SgNsh4VSY9HTujvemDzd5q/bb6r395pBT3aGXt19OtSQj3OHlLFTZpKOz 3moNjRiRwNGMhCSISXkO5WBY3iqa6OxWwze2RDeivmTcb1wgI00oQFQikYxNlY6T1EkFo3YQ JBphI3DLGqo2nCFqKNpQBkUUSZHUkEKVSswBGKgA0hoWRtIQKEIkBLjASClC0WFQsxULqhUV DIq8VC5Ad7tLkBhEE0EY/U1hwbncboPYDzXEDpESJCOuJIgVAKYJgCJHmecKG5giRUvY3jcZ gluwAY6ZsMQTeBHMoIRVIQgQGBGglEpWcA00IQFIDtAsFsrlIa+n86+Vvo2+7rX0dKkg2gtL S0tZSslZaUpqazappqtQRisQPMIak+IJADWmwUlASJZQCMF6VPVBR7wjqcHGHolg7EM6EWnp pXG4o0qEFAYBBRNTEUYmlpWWJM2aZYNkNEbGtqDVUyms2hkyabUjQgRcVDwJzzxkGeVJAzMh SHkHmpSqMCkC05JZuDBF9Iwuo+nzYKhIGIrvcuvgNiojq7hxh2GRkDQIqGpQdxT4aza+bL4t 7yIQqVKtfH8ye9+U3OwiHQQzicRgQOQeEOsEJMC6ew2bD6P/MfPVNwFDisCKe53M3yOqgLAB YGvPuGrqZh1ji4PLVa1wVBzuQw64dB/6NXw+Kf658KZgQ/1dX5dOeUunTpnIct+FRh2RLN2Y +9vOwahn+F23Deb7zb6BjAn7FPkmGOaREzWAGj2NcuiAgkhmuKqhHRv+mIo3Afu9btJnf36P 8QmjvbOyReemWLRvmyr0GkDVnwARMpypaAnOBBhCHQeMQsQgj0Kh9CvOgvviOY/ACKHz/WM6 q0MpgwYUnNRCkYBIQjUIMKRqVFwVBbXfOs281355DNtcrVuqVe19tfL1X15/KqKNRo2Ker4f sVBj5bCXSASZ7wJgSTaE9sgAWIkiSue8MUbZWwSeJDkCAYYyJmCIhIkvTDfMLsEepDXCkDXB V55189EiQETYXHjB56CWirtyBl589/aISCGNXOEPnoJdsgbBJqToOr+qb0IxJCEGQkGMDRAN Q4qOjz41sGiCF8mveq9I4DoWEJPXNtwukU76UJkh2q5oG1IaR6hIV8BNlWbQ1TVAwqbCCEFV 3om8lomxhOrx54JbiQC7PvwqghskUPghuAzYrZzTAUsiCZFb/U0qEI6o9warYVUwrMDvQAhA WIO0AoNYH4O5JIQVCH7yiGtogJiiAeOEIklQojUFkFxeDQQMDC4Au9+WMiocWF4qGCCoXmNv QPMdwaiXcKdxynVhzAnU7rBByBXhBwaxDT1H2AgL/XRhKlLst68oMjfgqqA2wcpWxUPRhzCG qYrNulqpNf1P9R0p+B2Ydozqty+YBPyGCMiyIAG0Nr22iSAEhAXxQJflgYlvY3967L6XKGki PtXmoJHQel8Cfyhxj3gDCP7RsAMtTQcFQ857PyIcARhJtT1Zrk3vw2Cy/B8ELDYJEnv9f6IB 2QIByxj5wBIUgbGAJl8Z9CU1SrKKJBzuYnuOeV6hVMWNUIrqLQJIEIqmVoMx5+IGq26CXqoY az/vzQxTOa1Uy376jf8RmPUQGgsCDTUIeaQqHyekPEqBuYIIP5/4x4/XPQ9DzH1hduI39glI XdPRwOVZkyRTtLmEUjw1UMAwfQLeFwClA/yfxH5LpdVTr+4H2TvjoB7X628CGZcihospxkQP XYoeBDABXQmkALiCH97mGKYuGEQ0IUHcme5+fHL+cRO0lXukRn7L2SHqs/C3DzvzBxLxzCNO evWJse8zvC4zwzlj0RxYabnpxRtQIXepuhmWtN76/5hllxh5w/TAtlGgMTyiWLTElE6Q5TaA lx/TIPkP7275mCXikTOiiBFPkOcGq4aZB3YLQpUA4keOjfDAI+OUSSMYRwjC66h6KwJM4DuM UpAdYCRFOwrMe7SIZw5lIc/FAvGIsCKAxDADz4CUcUkWE0H+Fu4fUjosQhJJJJO77B1J1Yna D4D+8vdx3lJz3P7qsRoAol47o2+EhgIutETFOBotaxA3WbZoeIBD4Ox83b1xxgNFOOKbN3QS +FvxY0ard45l1OBI5EEwaCmnSFBqCQinRyjY1CQv3Zs0U8wy3jZjINUsBcKuDaOw6RZIxrB7 pCxsibbGzAQO0ETUBYFpIuGluYhksAowUKAT2ZnHA0XBjQDDwhTLe4xPgRCECMGQYwYQB3kF bmAnCH7MGZ7jvcnkIej2nz/4nQPlgbLv4FgjYx0cZD9eG0LVxSZ2ATpRFR8Q1AxZTST+c+ka T0UMGaaEOBKmhwgvAVdMgRCLDY0sVra/HtbFrGttFVvbemKS8Bwg/r3wJbyAj09FIGBxmBp5 Ob0d4P++a3ekDVKKy/o1b7w5WNOpPvFzuSCSXxSAJm6qfqUx+McBFZEX8SpFBgmf8ejS/HHo K1Ne1F1cddcZtRtc2zuund9x+ivhAP4Z8MqNAO4ISE19d1ByGnq7T93Qz9DaqmbmZor1YAWU uQA6xIoB20BJACREwIDigGKofIofkHAc0RcUCQQqkTcZYuDynZXVQ+5/5BmM2YCmKMkITlxJ ohq2KC2DgGB4fYP0Tn3mwbd7JDNAkiQJg+CB7R09PnRtLgheLpc626RB9QQHgZB+r+1bYU+3 J8dpUkdRRZ8kZ5JJTVY6rf6bmaW5oPLGB5IsbJccKFnpYcscnm65YMyXN6g8MKe2J29hxgjm ROMgvIbzee7Mz4uYzi/wbmCa9hA6byo4/Je2uCiNnpjS5rFzu0uBHAUjISfjP0si0AyFYQX4 B+N2L7hefI8v9wqYLuGEryaBB9iqEvHOWsPqkvthbM9lh/humYz/WkngY+0HOzfv/6C+Wjrc QJRuccZJf8T71A49IhG1N/WK9jbX4mwQhRAbL4szBeXLE+AwekaH431DPvIPAh0OKBmH37Xa 1xkyk+69C+udZ/L8RuOMVh0WY+6tMkgo7q39NNyEUTOoTQa4tvE4t9qTliVuRUHnYBr4Ku9j UIwhYbd5d6C/YggfpPd2pLW9rLaSq0ZKWWNhYRB2A3jtmPHKeevXGQ7qAgZ2NpRITTWptrRZ YptiRWY/BtdI7xuMw4IOMHq97ewxEyiZ1xdxHz8NCgoJ1Y3aeaQa2NRttpt/B+CveD5cQ44E gOkFaSNFbb+S/RT/1dW47Qi5DkYL0AbYHWKhD0fzLAa0P2pXkGsMN9Dzc2IQ5DLrW5jPcB2Z gN2eKBgN+aUfQnparWYwY9FVsbDyxRNodeBIk0HlrBaCptSC3Eoj/Lof1cfxgG7Dn87u7tQ1 tbyVqn8Gvq1PN60RfT/Q3aSxpJ8byvn++IPuSe+mEfyT3PlGx4+jsG0h8304x834vE/c5ybC Zwse9GOEyDsFytPssckYY2y47IGSNpSxWpB7wblA578EDNmFz9Tqgo5kYdZHj4fl0el1rgQe RgiDOTOWH614NMVrrAAD7FnHEPYXKgYHomXZkHprD1AxM0jiz3/4TIwCJcwYlKWIBZsj52WA DqzR6Jy9HHpDWxXXF6keyRsrbrkS774ALBUh7KqkbGhhL+h/r8IP4W35aP0cA2f6w+TLQV9C ZBNDZBpn52nkDbB9/tH2phwaYxm7xnDRh+diIWmA3ARox8lH0l9aStBU7v8j80+AXOFkWLF5 vNIJRVoNvwU1Fh9+gPm3oTpwK+WJ9hgBoYoHBAwkwcAodeVGM5afU+IIvIhvfqSYrWIYao2x jSJcVPfj0PXGqlnswZdU4KrsyNbIozkb9rrn7XN0A5a/WIAGkFYLGggyR1QK+BUMbKWiF4An 8EA9aKZQXEm6km3zluverqSb7lXw6+GUmpKTJpkZTRpZVvg3nttTXr6Xz71r0Ki1vfTKSXv9 vL73tbm28UghzPtbv2/+VmsZ6vj2B57WD1RkpphAY2mweQxoVAY65R/tGAitvbB4QRRG5L5K voTvoO3km5XPLq5MkKPOdnXV5dvneczNT6vbep5NCXXYq6V67kbnnldi1yvKd4lw6SVEiyC2 CKmB1GAcISGhtVqNT6DcyXji/sl2wLYvO7Bokbb0tdMy2Xt25q0WWlrQbwGkkoITEdzD8uJQ 1k3INSE7bmI8vXcrG/Efcd7S9Tb20VjncMXZujoIsqt+aLUyqXA3ssLkiA+hEKIoii17m3Se uvd26t42mzYvTovLq3TaTSmKsVM1Rtkq0kkyXvtvv15F7O2KkkJXXunXPKJKSwjCaMyQmjUE V7N0C0gYSYUys0iYJkTQpRGhIkxaTBjM0zGmSVlm2SqZnjvSu52eNdOXYzr3LW8ojKDebSVd cuZWo1ptGir06NLavNpRCbIsiEuUQhYkLNFMKo7tUKbsg4PJ5bDwmk37wm5ibiFNTRAhvwzx HjTerP1zZUt5dKbmbc54DhPlw434TH9i4p0mzZZpjqcae5ED3UCz0Qp1T8X3MbJR9sueprcs ZV9epupLFtRtc0boZLUX5iW6lE4aMhImdoUiWIpCCPRRG61ASBIkIKSA8GIfyQSxOc+P9Ez/ T3jUDOAwGj+X+ZR548bFbXga8KiEkztvYG0O9li1JeQePuzdbDbewiSRrywiaDHh4E1cfxYJ LRWG4dq9tPUU2Wm8VXVm3g3yo9uedZZV6lNurdPK63mWrxcq/NNXrK9MJ9NrV02zEJJlosvc Hji7AQJdKVa9hQFoHwjwvxsXxpXsfM28W6ny/lt97LeedUVGFO01RwjA6LEe0mZuWKIDa40a xB/XxInSOAJBqYgQMIpAlQghUVZuLWAiDhxcr8a1skbT3Ev3WsYbus5nskJJHFrTSTkUxhua /Rgn53YLi+8DnYH7I350yMgMCxgOZ9AfnMD0uiUiHnPgLn6nnDBfbmSDhFJIEKIDNFZFWmOS ICKIGKKMgwYw7vqIJR6i16lFq/oJvby63F3V9/+tHeeujUgqMAg6Y0kTKN57sH8kXAAv82Nl JGylU80MFsQM86t+qc4ZDD0/V9jdfGCSDDWQPJdEidwgHZYMF1drCiJ3Q+aI+agFj6JnvrNL 5s8Ce6Nu40hgEamr7mGOZ+ZgkwAm5CIqOxkVZ/yyXMj3j3rYwKIlQFALy8UVE7WqotqpbyN8 FdqktjajV+B86Skpm0lZ86/K7zfVm3upIHTbXtLaSRNFm2lWSpKk2pNjaZTzpbeTabbLksaE 2jFFk0k82vF14k8rXY3lKteK2jayY1KUFKDbGMYwjCkgJtq21qpUYohKvNeXdCHl3KvGvG28 UVGiYecRDjpAON+mczcloHA468e65MTDHEd25mAGeSkecFXhXUh1PuTUqOVhz62ARoSjaHdx 7vPPA+DupjUSa3wbff9bX8zJAsEHuILRAahl0MuY5kqZmRfEA7CxeUh00H7dGlw94Ig/WHM+ u9sLTs/U0XDJ13ttr+A8CyRg3GEbiTaS8TjRqQzogPpGs1MVEOCWWXMbIH5JaHyxqDaOPE7t i7xy/1K/VuHCYy1k6h/0ZyDhN5yHZIbex/qRn11JsmViDt1kTHTiEh3LopbxLMZu5ibTGwz+ iONgsU7ItlFjQWNJX9Q4tbjiOp10u29enRTslURC3pKemQdZ7Q2ZBUMMtx/hLlZZ14jjmicS abjiFvHb/IjnGdLw4RThAmeH9mXOzzxDI5kkgfdJoZ9yZvEkN9P9W5Nm/umwuvR2h/lKGtB9 65iI6pHxiY6m4ncdM6GwYigzamOJ9aeGxvWf3mR95KgqIxykolJdyKzKol4cJagqYMhudrtC 5smZkU75A0xcz620e7TJ9Tnqw4tXH1Qk6xTWG8F/U3MEzVbBXPzXaIMIJkIfuScPVDcTHbPc DNpFxjcLZd+N0er18fte169B6Xy0G+t9z9ZciAI3c6SBQ2h4v/W3IPgkwyOkuQZ/bD6I1FrQ iGiB2EaDCEh/yzUgKdmjH7WDpRbF2cdD45waogarR8xAySYNgFzVoGlNs7f9eN9BgybHKGxQ yIJ+wQOZyRi9wSQQYf2O/dt2/E/C9CwJEYjOIHPSNk1MnN59ZEOgSDxtjkyPek/PjU8yffmS O9Mgm+RFNB7xOSmmk7qHDHKWJt2+8q18axgJxJBi7kImpr+7g2AsNJl2cd0yuhOGAjiaNtKM 8F8Nh+ef7h+fw9HDQRzcMhat47xJVDaFHj8rIRsWRTmgLvb9wt4DjtowebJISbA4EMkW+VPb tHfVyiu+Lwkliv3gMjcLcm+eLT06Ddsqa9TjZXZ2Mnca/T+583mw24lsExgQ8RtFl7ft0alv 5Y8KkoG19qRt3G8od6+w2Sdx0WlDEERrJISEhZDtziDNuw67kaaZ4H9I/c97o+r+Jp0d08fo ePXaNHn6iXiDvULmip2FpEoOS1Hq7R9OVCoUu5y/jxdsCcTk0hSTOiE2fi5xCTJC9R2dMkjx mgx0H8LShfPPGDqFRUFNM4mEsRf1DGZNn4GW4mfX2w5nM85U3/uolRkpKgv6jMGZi3ervahd EIt53lkPydnGjJ0hKBY/k+A62xOBUQjgbNkGxZngzLjQSIQtVEesQPPZAEU2ZZMv8RuOqlsB NF3ME2QhsfxF80zPyjbS4unPLKc2TZfvxyKL9iIOc0mjMymGyNpDUMk+aOnQlZSKJ1moFCJo xaP958csFMMcvpyAulg75IggfJJ3QiS77Md4R4UzJMAkMOIAgy02iNW6NhQMMXycd84iExYM pGXYSxwrsxsb1IyJGTkpEndkhIZxriYGRdtZF2/SHrgODqyHQ31qISRIdi+1PQEGGH9Usdq+ YNn7he4nP5HD7j3n5/w7bFY3lTmJhEiSIVoWu1iFF2nRq6rY2tQNp1267k6WjwTtL/0og/Ag HGx5LeQkhQh/TC+CH3oH0z93zp/KdJf9r/ah2pA/2lqJGvOGgIG2BWAkwXLIy5H5XRLuf2c3 m305jUAerIXAqgdb2sAl/RGGNrjlHiXCoMYUDEq6JOgoKJ0FQKCiTQuKthYTyahUgUCFjpOM ic1eSh6jmeZMJ++Zpw+HcGhp5lt1x4W+l78vk/Mi2/Vv4fhzPvfwNOaTUq/N2PHzvTxJNuOE qSMEzphJhISSP6QnsOwrB0Ef9mWmeVz6Lvdd299qd5qlJy2yZQOJQKYmRfD8KWwxbYro+V9U 3dDa2N6oTeL0IgNidbHSSdbEx6De2gQhDhqYmRlLprhhISDBdSnJyZDs44OxiBtyE7SdBQHM dxm2d6JM2L/yvifZTnNxJL8j637R/rPr3PvChVLlc4u8fjt/167JWP9ClVcXIQzceB7Re/rQ eIsSRbs2RLs/y/wJEhISmm+b/JdjyJu8d1G/8o8B6v3Z+eT5c6Oex9VlwOPcD6/Cc5N0MR9Q SspfZvzmVnLY4c9WjMqqQ7GU2KiP+SDAPjA+C0n9vLHNM7VUqJJIMhpZK/gBsfNUj8wGCH4h DaGfV7IVig+cioMgASAMgCofHtfNegwJP5C6j4s8AkCMD4AnT7gOn5B1AztzTqm3zUJvdzYU FWCNFUNYZmTtP3R5Q8Rhy9sbI/4Y/RWCAQ5I5Sjr/0N/4w8XyeZ/eki/x/DQf8nX7GdFweiM Iw6FTzcDzB+H4i3KQuRhRCjshwgfLgF/RBlc56LQDeyCA8gxyAUExDDA2IbNl/3uL7e//MOW 93Y/Gw73UTaNia3AuROUNAwNNmXd+S4boT22WL+LV/uvQ+njM9jbZ6gS9NYdLEWZ1UTc5aUh yUMn7Z5O/9rp9WwaF2VLyQkNgsogpJp+cHommvRakoZ6k0ZMToevrMGZUiaoxkwwudH9E9k8 7veAxOAhy5CaTFfa9vdUwoqcKnumajWjbbjmiR1fyMWGBuqOS/O2gQToIIW0e5E4jx5dWYdf lt9LyF8Iye946PBgI0f6Ynh23ALc0MbI0QP3oIHWYp/hwA8gRCpdTQ5hY5oopSGn0J0x8P5G rnhqZYQNM8sa117L1JR88m9LXr2Dklex+113/U7nltoMze5OXnYCdkilTnIyIBVNtR/pmDjB uMCpg5wV0fRmfdIkJ1xgckhyXT2XWSfwZTpTljt7eaX9U7ajpohqarBBz4YdbPrcsfilhApE D281hgaiA8wdFm9H/QOYCbl0R9GJguPEBRNX308v6T7Vi0wkZ/8jVqT04GMy7c2t92yBDUKG t0QpJHAL4+iRaE0YI9HDzp3h0m58gXhsGCw/xyjRB2odUE+3seqwJSwfaMU1H5L+6+H1RASU RFr9D73wK81Zb19bswMeIEBqXPF3znb6Qj1djSqLxHyoKx2UHB5f8pVxHJj60nKSKD95Bqf6 CNISPTZGFsD+WVDkfUe48J9MYc5+QjTjt9bmkoOI7m1CRxd1A5Clztthvshvwze3NCAoMYEu zhsNVMMD7KTwZPmoUlCjVx0zb2bFjToPEe9Ja0F0LU5LmxP3iDUXJG5fbikm2pISZv9Js1/g C/sD7NDS1xLLoiCNIgScQ+Yk45EnY3ZG8fAMJhgiuGAPHMQ6Q0tEOkNrk2KRvybK+JZnZ9CP QCQ7fUp1JTQ1MMGEhsSvNlkH3xc2HVn1b54CZkkyNyHQ83GM9BAcPristmhr/K9//M/0WPuj m0PEpGxvfgzQHNB8xur7mVpkUHY5lhSSZBgvEeQGsZjDGeM+HpMOvsPQf3h7bTcIps/qz7Xw YYGmHh9NDp6Z4a+x9K3Hz8nY/ZKqHt9nol7knEEuR5DQcwnhOs4R/ZZgiUjA5WLHvmYmI4o4 jn/n9i526jnp9fCJfyX81ddyYAhNpHP5jWIjrlqcAom6PR2Di9TUfd+33WaaCOgMkPuSEzu6 R+uJuatt9xuKx1kw2an/F+jU9b/v0/t9szs8ekcDV9ybecHgagG4+yHY8Y0G8P8nf+Z6gl8w 8fyv/V93/v/W/bba+u+9MLgeg8C5kO6B/BBCk7HDmEHrcmnCYjGYLn+3+ftcBx328puszPwb kq9tGouTKwfkyoGcm7Q+/+R+fy8/1PoZG3ocd1sgEQnNiDpPtiRt2MfzkG0SfUUDXjiuMXzS ZVGKON5op1YyiodOUL0mRKJE6QFSRwfNkELohzWr/EIxXJ0HQIb4yH6fK7zk2vu+sPkeDm8e bP07Df1z5yCTkCaHZs22N8Qfc3ynKAtne2/xPwvoOm2ZrqfpEOLkglD0EkISQZ7fn3OzbvgP 3Avibm/tdPZl6y/wGzbHYiG3P0dg7Db24CNiUhqngJGwPxZ/l7KtBpeDnWTJJGnOwdMEC18C t/xlrMzd3Yx9if4MDn2Prsw4kglsdh00kZNseZ+cIdCJMwbKHzxvGDa3ZuM7nSI1PSF/IWNG quDCwWJuJQRrcpV7vNJvcyEEPERRw4OMxYIcG2bV/Ttz4Np3Hzjfq0pLngeHPBueD8wtymKV wNYzjL0nCHzj++ONMsyhEXiTjlzZv+xD2X6hm8kd24UjYoOcl1ueltYg3y52pRT1HN52I2hg P1cSj+HMl0Jrgg455dEMdA7vgJicLBHi5+cXPzl1+Y31Zmc7azxdVJEG1b9zO00Bs3CJi2bA /ZSCgjJnMWCMTJxZTyEEJjHJhDJUWm9sw9FfzfwflzEoxAftGQjpA6COQJ4UYyDjrA5w2alM E4yTAKrjQ8ZW/dyEVVtFs6j05Sju8YaWj0ULihyaeIIa272y7HJuvcvz/m2/F9UxPZNnUbdh Mf84R0gG+U0DbxpMNqf7nq/yrnIs5zM460fyqOhNmiEsnISZBmMwd1E5n3BAelsOlP8j9h04 Y5eke/jpX4Bst0SCKko3TfcRgQJNURj6R5n05t7NnGOmUKIRODbGcocD5XlR/ZovN5R/aODq sB980xpBP9mpPKCqofpvdR+urW3waE6TwrlSyGomoPc5ZEQJXpdB9qidshz5pDikhNlqfWfK oZYonH2cfiRENyR/AR/DR+xQZo+NZ1zjjrY7uOfMN+2Dw9FAs0um6XQSrl0V/41zfz/xChhy eiOhkDdijXYJ7NiOQqKc2ckCAoJ/+6cSDn+b9dvn++7LcJXuz4o31GnUX1y/qUKvN3h0S/y3 NrE7cX70+mcjkUcqcenKSOT9OEiQv9jXrVESlp0bOwtj0vaWC5s1hffzI7kyp0ktnDqWcdNc AmhoXaOt5i1KoPek7CKoOp0FbQ1DbW3F2TRhRDJHPCNtEHSh6IX7JxxFh3K7qJEt7ZBhRTZJ q0SbxC64ZnJbMbzKQzx3KSC5LBu5NzYWSOi1NO7a0VbpMBI25SEVwjYVqFDmFUt2lsZwWuQI lOo+yfvXR98INdeCT/KmHZF2Ohu0ravg4tGeAzSeZk7ShYqzRRrwWkFWQ+mE5lk2kKuEFZlj DH/glf9zS38ac+jNvU5n/lqYr1a7ryzrRvL5Obmw441AGG+LX/7y4sceI97bIFFGOsc/i5Ex Gx2fbOsx3nYo6JKUK7nftKKTE2XrjzRMTjhaB/WLYTnR6dKkdTRB5DUld8hcsw57daDO6Q22 4Qa6dM40Q6w51i9vZBaDluEUb9vjLjI/c82taHqIrL4oRjabMkCROMbahllYztap5cdmQhjg SON9Rq3sdK132OkCM9ZhX3wkIIEyZ/SFltOONQIZCD3oo9H6Aftd6TE57/9GzfO/dfJ+6+ty n2YWJMeP6Jop56SnOBFF/0/edxdj+xKtDNmWDuxOdXnyrlTPiPYQAE9QP0WJSy2RGxUL2GRv wCwYAZlcy0xEQJf1dgZSeBfb0YfntPE4S3jU0hbiNrDGCG4fiATRuLaa9AhlO9lPgwuMrgb7 0jztBVJEgVyYvYIhOA7oQEXUw8+ntdibiAyWWIVLEHBEKtM7QpTf4ikrph/ZntJH5k5KG+cH fG345KkY5yxtRoX9ZXxDaPA9Ny1Z+bz548NMfYWthXaWG8TeNL6XZNtDzZd5FbEvFmaxHjyE cwcCbiLcdr30yvPaqPXfc/J1/NFJ/N9xe3Ptr/Iz4gfTGtfWeWI5926zffr6+VZ19l+EyUCF jGHinzo17r2QLc+9UTP6hPDL4sW8f+ZWVhwm7ezsMgG0i5TbC7HtztQwiFwgVE+tu12TOYBP hjREJHKP8uBaB4JeEauxWRL6HDTWlbqymvP1PMTbEUE3eFkIWLySh2wUJm502YmmmkmHZBtV FX5FZ62piritFThaqgocLG0UNqjss8gTFkaSdKQku+Phi/K1PmoOWdHtg+0ly0O1nkZrd77g V1mKn3l4zm9U+6btaV/6acO1Dpkmurqv6l33GDRHxLXy/jlGlwcsKP09YNGSQktFfpc2++oE a71Hy7uQJC5Koo/b/D4OktlR6ooxTzJ6FiY5ilyPM3JVkkDUxSpjSIIAAjOAIv8+/lVQzIM4 2ERMJR0V4IG7HFDY2c2CsefpEtdVqQDAm9GzMhiYAAFOsq3jA5TnHnSad7DofFQa56dq9CBU Q55RCRWHl/bUohgx+I1FW/kebnH+A/9GzZGI+mahu0xn0uZpwEYVP95hiQ/4pwwwEQkM+uEW SVTIBLqcEtw5tyv6f5r9LFRmYq3cqEJzk6gloJFgQNcOQmiXh82O4inYvlZCQOO1OyPnvReO WikIqflfnSK/QPRUTErfFxH+D5MvFcNEGr2JdeF/g6FD2TsePnbj0/4WAxBCu1rfgwCBQz/s oArpJHLHIoAAZRIUyR6VFGDYi52MDO4I1FYdwApJF7zEhl0Htli4FwNYRmzRsbT7Xud5/n8R z/WENX5e00G6vYTULJpAQvFUie3NXVlQ7IrmCYeyAim5g04Ltoes9V+f5r52iZ+Z08ZwBVYV gVYp2sDptKxhuQ3yIHB00jmxD5Zu/9/v/b/k9OTaObvNI2sfLW/fJiooQcE908o2kDbtsF5p l6caUopdBVCBF3vcfX/v6bxflfc0Azxy7Q89+OL+vGtfm/GXn41jzWqzFXxntD1dehsX2LEk gsSJqmGwgUeF4fpPb3PGmXfsCx4Rts+53v3PA2P0N95DUMfBzqXEEkkeL0icUETEbWPGJiOl yp6+3ejxsqgwgbDs5/gev27A3bmXhocHJwIIQgzogciJd5PXCDYX0Xyr0wnyXZ9SHmBhIP/D s7Xb1y2DvOu7PqDO2u7NTbXBU+ibNSztrbchJ8QHFzEIXgGF3HMBBcLqHABxBMzY8T6GmKzd iUcEsYNRJgGUPEOmKNUgm8d5I5xr/5/9vGh/3m7r2BV2PfvLfLs8eO/+Va/Q+K33HgyO1ei9 iIcwuRA4kROHCM3EOgiBqZyAYS4IwOCeG02j7j6/Od18rJy5DNm/mkTQ0BKDPE6uIYbmjc6G h4facX7b9vz1Lnd+Z7PDbAOBkjnNFdvADoAQT8hFgGOcgMIFgcBGmD97gYMgyG0JRISVgxAd RBKPqHyjicbiDGSCGv29qj9frrwM44OOQuJHHwyITRE1DH7Q0b70r4j+DC1i6lD8g6KwqFg0 tL3sMJ1mU7uB53bGlLG3tZseyPCK1dXnjLLqLEo7KijZupuloHMlVNSQ855zmDPfdMTX9ri/ xMYFX67mvJ7Xp+1+p8i3sD0OsqI62xrGdDcyxoSsGyVbDCjipin5xKianOk6o64tajIazq8L 4vVddvfe4jNdJesQG7+uc0RTmGizucyaSkhJuZMe44M2W8a/yfvt3WiJ23F45ts/yPneRqvJ TcR4+BjGNSwCIIhQ0k0kEY5Jnp0HBG4YKEYxjZgguEqDmEM0qWj3W66knZWGRvuy+DI8f3+x Uf72tD+v+ay18llJf1ib7UXah/P+D5zjlRHf8dyByE7Vb+qL6jCcZgoOlP0T6sH35yQGIG9M AbsxwgxCiF+nwN90+Lt3kfDUVu4nayHmebvZvrUFHO7/p4G1p7b5Tjc7WGM5H4b/s2GPvLI7 /T5Nbh9d8FTyHnyS39UfvVl85nhksjjR6bfT/iw4F2hGMAYxhI5PmHzT7H7P9Yt6Gyz6Fj/X ElLDN6oH2ICPiAICwiw1g/8Po9WXsFixGe3+JH4R7yoeyVq9ViXcyuO2PSsnjuX6eOvUxbPq cr9b70uvnrHaf9d2a2j/10slTcE8TwrB3YrPQ2Rh18R7v/ua8vYvk7hdBge6tfvK7lk/rC/9 rvXxyNXtbPe/ls6kqoaven6GvsOh9a+4ngD2MVobCG/73P2YWYsn72DF38XSdH6D3/nF4Tyd z5o43zPmSiMrKVasFsBVPdF9QtR5k7qJ4J6IpSwITOqUecLkbT37j1+XrNHAtp9T9ZflwPe+ fRMLmp3+mllIldqrjveTB1umyddudr6eX4E/rqfw6UeIRtLhVMWNrd5D96eaobPH5NBaNNkP r9fqVtxsPOj2zCcDc67YbfDezuaKDiLQOh/Xvuc5839SPDpR/wz1frFczbJbu+pdhienyv5k s/RNtHS57HZq3xNJYPd4WJ81Xp09ZIix8Xu/T6PQ3dzte5r1v5eBHWr3j88RguHIB9iuqRJ/ 96PYkuTgwui2zW5fFZ4aTPb/37jl+r5dvT42/j05p/ldp5M59jYdMS1qaxuN/tX8Hz7HL6Ha +fALr+uODM9ebw93sdm5uxyNy2X4fxT5yHDt3sjU3bmh5vK1nw0/YZ4ezaazzv66HmWuDzWu rqb17V6vfGGmHf+T9Nh8X3B9n/7/yd8e6H3PthKPeQRAN3BuiIAo+BCQmrdb7+j3tjoYfkf7 lIHnkAIBIqJ+Wb07a1f6dFl+mWv06raq/JttXqYyJ1JAQzVHPz5RZcgoB7UoAiSaQfxxsigd V/39pmn1CN8Rk/yLUWkUB5uB3ZE/whOU8wWENKCMiIjIJmmdGTazNlS4j4Nik8KXEDBCRCiI QgXUiAiPNQckcck+G5LCOEkKFdGhp0bFZBnbCYyT3rNfFtbdNaascdcGz9tDTHYYyulbVvGF bmWSLGMHmYYqGRDjI2VkSGmSEmkh0DJRJO8D7Pb/APL/E9PtMv59wMc8sc/zP3Wd8E/rY51c o1bZnVOEsJaBqafc4smPDttqXLmlqJnjV/wPVzhwwvVlzWqbQ2btewYrw6aIeLmsb34DRkJB ayxjhD5fD2N/fMVXT+RzL9R0q7YxpRtAVWgyULDwMdWlJV3GiyMV6S5gTlpCi+e6bSsNJ0m3 VJZwhCUObmZuShrJFMv1lJabEVnKYxkxwve1BwZKQ7JKpq6aweQhB5ZY2VlMiVqmFJRsrUTQ 02wgREGxxkGxwbZjZa1ks1i0tRtpnDudri9enr17UntzZEXjiIhYaLpbqGkZpnnr16eu9vbu auPT0mXT+NvHrdptyvdkzWGGNyJjg+3j08aTMaF3cp5evPB110l47KaZlx2lZ4BkbyYxuW1y V0hXHq2WKo44yru1DhwhHCEjOJE21XtqZvRhkGNRNptQcGNyEBg1BmUmoGDLmoVNtkyaY2Zl qjW8Iajijxl4yt1kg5jttipCOeC2jOYSj066wuoohYg02IfC5+kE+PERHEQQ48iByQIGMAYA wNlwgKhYOxNjhjXSgq+yU26NYdKMzzO/per0/cru/PYUsj850Z7UMEG9TJoxvO7HRSvnH7nZ 0FuQrdeZCvg0XSGk8ezF8GmHfmdd+TCT5W88Yf5TZD6hx5gdnQz/q/52OqfYI1uyzOVrP3yk rChf7n6QP+N9MX9h4ddnJ6X/YNK8HPhEwPGj5PmsW6s3kXfLNiDcme7jQR87w9VDs2eqa5Qv 9J5efJV36gza+kNKU1aq87bk6hWzwWyU9mKGQfD9wQRaw7UG7ugXDtNDy12d2ZeF+ztFubsv 4/6837g2h5dR+t2qrhthVNrbMO19lrzSv260DKD74XZ/TIjEB/785LI7ckwpOvUUmCwA/PCT UrWPofmN/vFNaUsTXbVaKGILcs3w9Kn7n9Cdkxba40tNwrlDdo8hIzyGDSA2uFeuMwOTjLu8 rdLcb2J5RdapRw2uSirhi9rNkbbjPD1BnRphth4hlRqB9gQPsMYBcGHZQb3/xGX98Jd3khUG MADeMdgbEfMNSPAPjDcakewNLxjGXn95sHsj3BnkExwkD2MkVIRogyUtzVjVzlXDbXNtyjaL W5fPlXNJt4yctdKOnd2uUaNcubFad3TRjctc5qTbJtRpMYRoCoP/f3Sga1BETbCMAiKNYK3I xqNyiq7u2uWtzGsbWd1FrlubUc1uZNo07t0rGsiXOW3NGp3W3NaMbmxY0bdli0VxKuVjWrmo 5VcqijVG5uVy4YuluRcrcq6WotqLlZ3W0mq6bmqLXK3Ko2umLlyosatdKNjRa6Vrmo2jW5WL QVd3a5Vc1zYrmjHLcS3Nd3QWuVcrm1yJNq5VGqNsbcuaJ3V0rudbmo0aua5qNuarhWOa1FrG qLu7Jc3LVy2NsUWo0bmoitbm6Vc2uctzXMba6RRtG1cotGq6a5bFGo23d2orm0RXNy5tEXDV GosWTdK5ucxtc3Lmt0rprlU660ai5oNuandY3MV0sSWKotjbRzGrlrlXuuu2LaixrxG25ctR VzUYrcLla5oquWiiwaNuW47qrklFysbUbbc5RyrmrcrlXK4WLkWK0GuaiiiyWNVc26bRzbc5 XLc2I101zbpqjY1oqLVzhq5u7tUXNh3av/8XckU4UJC4OQ+L --------------050803030603010300020209-- From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 18:43:21 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CAC8106566B; Thu, 10 Feb 2011 18:43:21 +0000 (UTC) (envelope-from yongari@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id F01EE8FC1D; Thu, 10 Feb 2011 18:43:20 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1AIhKsA063557; Thu, 10 Feb 2011 18:43:20 GMT (envelope-from yongari@freefall.freebsd.org) Received: (from yongari@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1AIhKMQ063553; Thu, 10 Feb 2011 18:43:20 GMT (envelope-from yongari) Date: Thu, 10 Feb 2011 18:43:20 GMT Message-Id: <201102101843.p1AIhKMQ063553@freefall.freebsd.org> To: rmh@debian.org, yongari@FreeBSD.org, freebsd-net@FreeBSD.org, yongari@FreeBSD.org From: yongari@FreeBSD.org Cc: Subject: Re: kern/154591: [msk] [panic] if_msk driver causes kernel panic (fatal trap while in kernel mode) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 18:43:21 -0000 Synopsis: [msk] [panic] if_msk driver causes kernel panic (fatal trap while in kernel mode) State-Changed-From-To: open->feedback State-Changed-By: yongari State-Changed-When: Thu Feb 10 18:41:33 UTC 2011 State-Changed-Why: It seems there are two issues here. o Station address read by driver was ff:ff:ff:ff:ff:ff o PHY attach failure triggered panic I have no idea for the first issue. It seems you have 88E8056 Yukon Ultra controller which is known to have several stability issues with msk(4). Due to lack of hardware I was not able to track down the root cause of the issues. However I can't reproduce the panic on my box when I patched the driver to fail attaching a PHY. By chance are you using VIMAGE enabled kernel? If yes, how about removing VIMAGE in your configuration and test again? Responsible-Changed-From-To: freebsd-net->yongari Responsible-Changed-By: yongari Responsible-Changed-When: Thu Feb 10 18:41:33 UTC 2011 Responsible-Changed-Why: Grab. http://www.freebsd.org/cgi/query-pr.cgi?pr=154591 From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 19:16:33 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09229106566C for ; Thu, 10 Feb 2011 19:16:33 +0000 (UTC) (envelope-from jayb@zoe.braeburn.org) Received: from mail167.messagelabs.com (mail167.messagelabs.com [216.82.253.179]) by mx1.freebsd.org (Postfix) with ESMTP id B69AC8FC13 for ; Thu, 10 Feb 2011 19:16:32 +0000 (UTC) X-VirusChecked: Checked X-Env-Sender: jayb@zoe.braeburn.org X-Msg-Ref: server-10.tower-167.messagelabs.com!1297363791!27187982!1 X-StarScan-Version: 6.2.9; banners=-,-,- X-Originating-IP: [144.160.20.145] Received: (qmail 28410 invoked from network); 10 Feb 2011 18:49:51 -0000 Received: from sbcsmtp6.sbc.com (HELO mlpd192.enaf.sfdc.sbc.com) (144.160.20.145) by server-10.tower-167.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 10 Feb 2011 18:49:51 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id p1AIoEXc025723 for ; Thu, 10 Feb 2011 13:50:14 -0500 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id p1AIoBoW025682 for ; Thu, 10 Feb 2011 13:50:11 -0500 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id p1AInmSa029797 for ; Thu, 10 Feb 2011 13:49:48 -0500 Received: from oz.mt.att.com (oz.mt.att.com [135.16.165.23]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id p1AIngPL029583 for ; Thu, 10 Feb 2011 13:49:43 -0500 Received: by oz.mt.att.com (Postfix, from userid 500) id 853662BF2C; Thu, 10 Feb 2011 13:49:42 -0500 (EST) X-Mailer: emacs 21.2.1 (via feedmail 8 I); VM 7.18 under Emacs 21.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <19796.13124.805432.342309@oz.mt.att.com> Date: Thu, 10 Feb 2011 13:49:40 -0500 From: Jay Borkenhagen To: freebsd-net@freebsd.org X-GPG-Fingerprint: DDDB 542E D988 94D0 82D3 D198 7DED 6648 2308 D3C0 Sender: jayb@zoe.braeburn.org Subject: cannot set mtu > 1500 on em0 optical Gig-E X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Jay Borkenhagen List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 19:16:33 -0000 Hi, For quite a while now it appears that mtu on my gigabit ethernet fiber em0 cards cannot be configured higher than the default 1500 bytes. Here's what I get when I attempt to set it lower and higher on a brand new 8.2-RC3 box: === sandbox# uname -v FreeBSD 8.2-RC3 #0: Sun Jan 30 06:52:51 UTC 2011 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC sandbox# sandbox# ifconfig em0 em0: flags=8843 metric 0 mtu 1500 options=18 ether 00:d0:b7:7c:f8:01 inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255 media: Ethernet autoselect status: no carrier sandbox# ifconfig em0 mtu 1499 sandbox# ifconfig em0 | grep mtu em0: flags=8843 metric 0 mtu 1499 sandbox# ifconfig em0 mtu 1501 ifconfig: ioctl (set mtu): Invalid argument sandbox# ifconfig em0 | grep mtu em0: flags=8843 metric 0 mtu 1499 sandbox# ifconfig em0 mtu 1500 sandbox# ifconfig em0 | grep mtu em0: flags=8843 metric 0 mtu 1500 sandbox# sandbox# pciconf -lv | sed -n '/em0@/,/em1/p' em0@pci0:0:2:0: class=0x020000 card=0x10008086 chip=0x10008086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82542 Gigabit Ethernet Controller' class = network subclass = ethernet em1@pci0:0:4:0: class=0x020000 card=0x10008086 chip=0x10008086 rev=0x03 hdr=0x00 sandbox# === However, back in the 6.2 days it allowed it: === [jayb@cactus ~]$ uname -v FreeBSD 6.2-PRERELEASE #0: Tue Dec 26 16:53:18 EST 2006 root@cactus.iplabs.att.com:/usr/obj/usr/src/sys/CACTUS [jayb@cactus ~]$ [jayb@cactus ~]$ ifconfig em0 em0: flags=8843 mtu 1500 options=8 inet 10.0.1.152 netmask 0xfffffffc broadcast 10.0.1.155 ether 00:90:27:e3:0a:67 media: Ethernet autoselect status: no carrier [jayb@cactus ~]$ [jayb@cactus ~]$ sudo ifconfig em0 mtu 9000 [jayb@cactus ~]$ ifconfig em0 | grep mtu em0: flags=8843 mtu 9000 [jayb@cactus ~]$ [jayb@cactus ~]$ pciconf -lv | sed -n '/em0@/,/none0@/p' em0@pci0:2:0: class=0x020000 card=0x10008086 chip=0x10008086 rev=0x02 hdr=0x00 vendor = 'Intel Corporation' device = '82542 Gigabit Ethernet Controller' class = network subclass = ethernet none0@pci0:4:0: class=0x088000 card=0x0767101e chip=0x9063101e rev=0x04 hdr=0x00 [jayb@cactus ~]$ === Can this be made to work again? Thanks. Jay B. From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 19:33:32 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2417106566B for ; Thu, 10 Feb 2011 19:33:32 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta08.westchester.pa.mail.comcast.net (qmta08.westchester.pa.mail.comcast.net [76.96.62.80]) by mx1.freebsd.org (Postfix) with ESMTP id 624E28FC0A for ; Thu, 10 Feb 2011 19:33:32 +0000 (UTC) Received: from omta20.westchester.pa.mail.comcast.net ([76.96.62.71]) by qmta08.westchester.pa.mail.comcast.net with comcast id 6KJe1g0081YDfWL58KLJfX; Thu, 10 Feb 2011 19:20:18 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta20.westchester.pa.mail.comcast.net with comcast id 6KLG1g00f2tehsa3gKLHVe; Thu, 10 Feb 2011 19:20:18 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5DA089B425; Thu, 10 Feb 2011 11:20:15 -0800 (PST) Date: Thu, 10 Feb 2011 11:20:15 -0800 From: Jeremy Chadwick To: Boris Kochergin Message-ID: <20110210192015.GA64265@icarus.home.lan> References: <20110210155622.GA60117@icarus.home.lan> <4D541B0C.6000909@acm.poly.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D541B0C.6000909@acm.poly.edu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org Subject: Re: Reliable PCI wifi cards, and layer 7 filtering X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 19:33:32 -0000 On Thu, Feb 10, 2011 at 12:06:20PM -0500, Boris Kochergin wrote: > On 02/10/11 10:56, Jeremy Chadwick wrote: > >(I was considering cross-posting this to freebsd-pf but decided against > >it, instead starting here first. Please keep me CC'd as I'm not > >subscribed to freebsd-net) > > > >I'm looking into the possibility of using my home FreeBSD box as my home > >firewall/NAT box, to replace my Linksys E2000 router (which runs Linux, > >specifically the TomatoUSB firmware). > > > >I plan on using pf for the NAT and firewall layer. ipfw will not be > >used (I have long since moved away from it). I've got solutions for > >everything except two items: > > > >1) Wireless hardware support > > - What consumer PCI cards are known to be reliable and have good > > support on FreeBSD? It looks like anything that relies on ath(4) > > might be a good choice, but I'm not sure what specific chipset is > > considered decent/worthwhile, or if there's a specific model of > > card from Vendor X(tm) which works great. > > I have a ton of Atheros 5212s deployed as access points. They are > solid. Some field-tested implementations: > > - Netgear WPN311NAR > - D-Link WDA-2320 Thank you very much -- exactly the kind of answer I was hoping for. Sadly, neither of these products are manufactured and have been EOL'd per the vendors. This doesn't sound like a wise purchase (for me anyway). See below however. > > - The card and driver need to support both 802.11b and 802.11g > > simultaneously. 802.11n (for the future) would also be good. > > Simultaneous 802.11b and 802.11g works, but the 5212 chipset does > not support 802.11n. This helps a lot, thank you! I dug around and it looks like the D-Link DWA-547 uses the AR5416 chipset, which is supported per ath(4)'s man page. This card also offers 802.11n, but I'm not too focused on that at this point. There's also the D-Link DWA-552 which uses the AR5416, but I see on some Mac forums people are getting very frustrated with it (who knows what the driver on OS X is like compared to FreeBSDs though; I imagine ours is much more reliable). Anyway, getting 802.11b and 802.11g working would be a fantastic start. I imagine if I have issues with the 802.11n piece (when I get there) I can talk to Adrian about it. > > - Driver or OS needs 128-bit WEP -- this is not a joke, I really do > > have devices which do not do WPA or WPA2. > > This works. > > > - MAC address filtering is needed too, but it looks like that's > > already available (looking at ifconfig(8) man page). > > Indeed. Awesome. Perfect. > >2) Layer 7 filtering > {snipping your response -- but thank you VERY MUCH for it} As it turns out, I realised I can solve this in an indirect way: by simply using a different piece of software that doesn't pull in the ads. The software I'm referring to is Yahoo Messenger and Windows Live, by the way. By switching to Miranda IM, I can avoid all of this. Pshew! Time for me to make some hardware purchases and give things a shot. Today so far has been super horrible for me (fighting with my landlord and neighbours over stupid stuff, resulting in my loss of sleep, and some family matters), so the above information is surprisingly a very positive moment for me. I'll probably blog about my experience migrating from the E2000 to FreeBSD. It'll be a learning experience for sure, but a fun one. First time I've ever delved into the wireless AP world too... -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 20:28:15 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 475081065670 for ; Thu, 10 Feb 2011 20:28:15 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id ED8948FC0C for ; Thu, 10 Feb 2011 20:28:14 +0000 (UTC) Received: by qwj9 with SMTP id 9so1024144qwj.13 for ; Thu, 10 Feb 2011 12:28:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=KjVpCLAqdMu/mR5uX6R/oBamsLwACYj4rZID9jCiV4w=; b=GumnATnVnvEYJm64+P+RFcuiq2Um6TTZxoqu8rymtnpZjH1b65V4E+Cs3MEl3Q4mFK QtplXN7ZBOrJEcMP70zMNA6Nh4s60VyP8Isai/1D5iaIz09cMd0UpWC7X4kslwWQqM00 MCj40JZk/evX7xZHE3fhtIIJAIV8H9YEabwqU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=n/OOUOvP+EIWqCzC/dkWT7rzX5I45E4SdvzVRyAspYzL6hey8kFLiQGBVhQZ74tzDo ybsny1Dsp08XT38etjKqjR88BngUhO/lsYwYyPVxJzNRll1+6CSVT4DbaU++GX8lbs22 SU33rA3ZdTIl65rxjSFmkWGXtWC83dFeQ//L0= MIME-Version: 1.0 Received: by 10.229.231.21 with SMTP id jo21mr16648831qcb.119.1297369694032; Thu, 10 Feb 2011 12:28:14 -0800 (PST) Received: by 10.229.102.87 with HTTP; Thu, 10 Feb 2011 12:28:13 -0800 (PST) In-Reply-To: <19796.13124.805432.342309@oz.mt.att.com> References: <19796.13124.805432.342309@oz.mt.att.com> Date: Thu, 10 Feb 2011 23:28:13 +0300 Message-ID: From: Sergey Kandaurov To: Jay Borkenhagen Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: cannot set mtu > 1500 on em0 optical Gig-E X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 20:28:15 -0000 On 10 February 2011 21:49, Jay Borkenhagen wrote: > Hi, > > For quite a while now it appears that mtu on my gigabit ethernet fiber > em0 cards cannot be configured higher than the default 1500 bytes. > > Here's what I get when I attempt to set it lower and higher on a brand > new 8.2-RC3 box: > > =3D=3D=3D > sandbox# uname -v > FreeBSD 8.2-RC3 #0: Sun Jan 30 06:52:51 UTC 2011 =A0 =A0 root@almeida.cse= .buffalo.edu:/usr/obj/usr/src/sys/GENERIC > sandbox# > > sandbox# ifconfig em0 > em0: flags=3D8843 metric 0 mtu 15= 00 > =A0 =A0 options=3D18 > =A0 =A0 ether 00:d0:b7:7c:f8:01 > =A0 =A0 inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255 > =A0 =A0 media: Ethernet autoselect > =A0 =A0 status: no carrier > sandbox# ifconfig em0 mtu 1499 > sandbox# ifconfig em0 | grep mtu > em0: flags=3D8843 metric 0 mtu 14= 99 > sandbox# ifconfig em0 mtu 1501 > ifconfig: ioctl (set mtu): Invalid argument > sandbox# ifconfig em0 | grep mtu > em0: flags=3D8843 metric 0 mtu 14= 99 > sandbox# ifconfig em0 mtu 1500 > sandbox# ifconfig em0 | grep mtu > em0: flags=3D8843 metric 0 mtu 15= 00 > sandbox# > > sandbox# pciconf -lv | sed -n '/em0@/,/em1/p' > em0@pci0:0:2:0: =A0class=3D0x020000 card=3D0x10008086 chip=3D0x10008086 r= ev=3D0x03 hdr=3D0x00 > =A0 =A0vendor =A0 =A0 =3D 'Intel Corporation' > =A0 =A0device =A0 =A0 =3D '82542 Gigabit Ethernet Controller' > =A0 =A0class =A0 =A0 =A0=3D network > =A0 =A0subclass =A0 =3D ethernet > em1@pci0:0:4:0: =A0class=3D0x020000 card=3D0x10008086 chip=3D0x10008086 r= ev=3D0x03 hdr=3D0x00 > sandbox# > > =3D=3D=3D > > However, back in the 6.2 days it allowed it: > > =3D=3D=3D > > [jayb@cactus ~]$ uname -v > FreeBSD 6.2-PRERELEASE #0: Tue Dec 26 16:53:18 EST 2006 =A0 =A0 root@cact= us.iplabs.att.com:/usr/obj/usr/src/sys/CACTUS > [jayb@cactus ~]$ > > [jayb@cactus ~]$ ifconfig em0 > em0: flags=3D8843 mtu 1500 > =A0 =A0 options=3D8 > =A0 =A0 inet 10.0.1.152 netmask 0xfffffffc broadcast 10.0.1.155 > =A0 =A0 ether 00:90:27:e3:0a:67 > =A0 =A0 media: Ethernet autoselect > =A0 =A0 status: no carrier > [jayb@cactus ~]$ > > [jayb@cactus ~]$ sudo ifconfig em0 mtu 9000 > [jayb@cactus ~]$ ifconfig em0 | grep mtu > em0: flags=3D8843 mtu 9000 > [jayb@cactus ~]$ > > [jayb@cactus ~]$ pciconf -lv | sed -n '/em0@/,/none0@/p' > em0@pci0:2:0: =A0 =A0class=3D0x020000 card=3D0x10008086 chip=3D0x10008086= rev=3D0x02 hdr=3D0x00 > =A0 =A0vendor =A0 =3D 'Intel Corporation' > =A0 =A0device =A0 =3D '82542 Gigabit Ethernet Controller' > =A0 =A0class =A0 =A0=3D network > =A0 =A0subclass =3D ethernet > none0@pci0:4:0: class=3D0x088000 card=3D0x0767101e chip=3D0x9063101e rev= =3D0x04 hdr=3D0x00 > [jayb@cactus ~]$ > Hmm.. Did jumbo frames really work on these cards for you? Intel claims: "Adapters based on the Intel=AE 82542 and 82573V/E controller do not support Jumbo Frames. These correspond to the following product names: * Intel=AE PRO/1000 Gigabit Server Adapter * Intel=AE PRO/1000 PM Network Connection " As seen from code, the behavior was changed to match the description ("keep old 82542 from using jumbo frames") and merged to 6-STABLE before 6.3. --=20 wbr, pluknet From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 20:40:07 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 972391065679 for ; Thu, 10 Feb 2011 20:40:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 712FD8FC13 for ; Thu, 10 Feb 2011 20:40:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1AKe6wY082074 for ; Thu, 10 Feb 2011 20:40:06 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1AKe6C3082073; Thu, 10 Feb 2011 20:40:06 GMT (envelope-from gnats) Date: Thu, 10 Feb 2011 20:40:06 GMT Message-Id: <201102102040.p1AKe6C3082073@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Jacek Zapala Cc: Subject: Re: kern/127050: [carp] ipv6 does not work on carp interfaces [regression] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Jacek Zapala List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 20:40:08 -0000 The following reply was made to PR kern/127050; it has been noted by GNATS. From: Jacek Zapala To: bug-followup@FreeBSD.org, maddog2k@maddog2k.net Cc: Subject: Re: kern/127050: [carp] ipv6 does not work on carp interfaces [regression] Date: Mon, 07 Feb 2011 14:30:21 +0100 Bug is still present in 8.0. I have found that system joins multicast group only for the first address. For example: for address xxxx:xxxx:xxxx:x::110:166 it joins ff02:1::1:ff10:166 as shown by netstat -ia (where it listens for example for neighbour solicitation messages), but for the second address assigned to the same carp4 interface it does not join another group. It works if I chose the second address with the same last 3 bytes as the first address - because it falls to the same multicast group. Regards, Jacek From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 21:33:23 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22C63106566B; Thu, 10 Feb 2011 21:33:23 +0000 (UTC) (envelope-from rmh.aybabtu@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id B89D88FC0C; Thu, 10 Feb 2011 21:33:22 +0000 (UTC) Received: by iwn39 with SMTP id 39so1750989iwn.13 for ; Thu, 10 Feb 2011 13:33:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=eNMAzoxv1idat2G+hVrFWajP8hsWjnigHNmMV0m4JlI=; b=lkQqoMHyN8kfXGwNarV/0L9EjjV5I3TNQAuN5ATMGVleV1go3K7P0FD22e8I/5VqlA N/2P6v0lXcIsVcNrZWqnn6vHjEtaiPX853LFrOgvjTTePCQnfKamY/004dQiPX6UhxaS FppQQZIqTRkqG4eJs6F3bWRZwBu6UQzLfs2AQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=GeD5fntaw1J/IgN9AAf9zeuEy7bQyOuv4YHZjy3ii9SsZvOvCvseR/lpTkYA1/saw9 1KbkDpEUa2B+L2sasheQ7WmskuPjtnb1+7KBlmjWIaMXyK1TZt2z0j83+hViiMjhki6z I6e6aKoldb755XfYxbr+vE9IFrTObPx3yIZgQ= MIME-Version: 1.0 Received: by 10.231.33.195 with SMTP id i3mr23067765ibd.161.1297372061710; Thu, 10 Feb 2011 13:07:41 -0800 (PST) Sender: rmh.aybabtu@gmail.com Received: by 10.231.166.193 with HTTP; Thu, 10 Feb 2011 13:07:41 -0800 (PST) In-Reply-To: <201102101843.p1AIhKMQ063553@freefall.freebsd.org> References: <201102101843.p1AIhKMQ063553@freefall.freebsd.org> Date: Thu, 10 Feb 2011 22:07:41 +0100 X-Google-Sender-Auth: o6s0IXZ57tHeAPoM8vCxbfIjQl4 Message-ID: From: Robert Millan To: yongari@freebsd.org Content-Type: text/plain; charset=UTF-8 Cc: freebsd-net@freebsd.org, bug-followup@FreeBSD.org Subject: Re: kern/154591: [msk] [panic] if_msk driver causes kernel panic (fatal trap while in kernel mode) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 21:33:23 -0000 2011/2/10 : > By chance are you using VIMAGE > enabled kernel? No, I used the default amd64/conf/GENERIC from current trunk. -- Robert Millan From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 21:37:20 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44112106564A for ; Thu, 10 Feb 2011 21:37:20 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id EF69E8FC15 for ; Thu, 10 Feb 2011 21:37:19 +0000 (UTC) Received: by gyc15 with SMTP id 15so812258gyc.13 for ; Thu, 10 Feb 2011 13:37:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ZtBa2b8i3qscFlemFi1Nkem5tFcrcllNt/DoyOmUMmc=; b=DiU+lo3n4CSBTSJlwdA55CoQpyjivWAKcA2Yr/O5inFaoRX2NrNWi+qvf3zvyOmyEc PFKcZ8xq3cvrj7S8cAql7LQdMyDXCszsi5g2PusQKoZP/PvooZ69srg6LGuJnoDQtHhv zHnYeNA/6Rstgngvs5nu72r51DvfK2IuaRPr0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=OTyj843jqhORsZUxvr5meChVNwtA7hNplUY5UsJ7ytETc6TpIW3c6dhL2ZZTp0UY8y d7aOqtxRIMVkcG69ITCSqQ0lnmDWv68bM+W8t2NEOznbj1wpAn/8eqib16/p2L4myGrp KcTVGAYuFTlDYvstOVhdJ1vZhthTyKjM4npPc= MIME-Version: 1.0 Received: by 10.236.110.14 with SMTP id t14mr2773266yhg.57.1297373838456; Thu, 10 Feb 2011 13:37:18 -0800 (PST) Received: by 10.147.167.5 with HTTP; Thu, 10 Feb 2011 13:37:18 -0800 (PST) In-Reply-To: References: <19796.13124.805432.342309@oz.mt.att.com> Date: Thu, 10 Feb 2011 13:37:18 -0800 Message-ID: From: Jack Vogel To: Sergey Kandaurov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org, Jay Borkenhagen Subject: Re: cannot set mtu > 1500 on em0 optical Gig-E X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 21:37:20 -0000 Thanks Sergey :) Jack On Thu, Feb 10, 2011 at 12:28 PM, Sergey Kandaurov wrote= : > On 10 February 2011 21:49, Jay Borkenhagen wrote: > > Hi, > > > > For quite a while now it appears that mtu on my gigabit ethernet fiber > > em0 cards cannot be configured higher than the default 1500 bytes. > > > > Here's what I get when I attempt to set it lower and higher on a brand > > new 8.2-RC3 box: > > > > =3D=3D=3D > > sandbox# uname -v > > FreeBSD 8.2-RC3 #0: Sun Jan 30 06:52:51 UTC 2011 > root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC > > sandbox# > > > > sandbox# ifconfig em0 > > em0: flags=3D8843 metric 0 mtu = 1500 > > options=3D18 > > ether 00:d0:b7:7c:f8:01 > > inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255 > > media: Ethernet autoselect > > status: no carrier > > sandbox# ifconfig em0 mtu 1499 > > sandbox# ifconfig em0 | grep mtu > > em0: flags=3D8843 metric 0 mtu = 1499 > > sandbox# ifconfig em0 mtu 1501 > > ifconfig: ioctl (set mtu): Invalid argument > > sandbox# ifconfig em0 | grep mtu > > em0: flags=3D8843 metric 0 mtu = 1499 > > sandbox# ifconfig em0 mtu 1500 > > sandbox# ifconfig em0 | grep mtu > > em0: flags=3D8843 metric 0 mtu = 1500 > > sandbox# > > > > sandbox# pciconf -lv | sed -n '/em0@/,/em1/p' > > em0@pci0:0:2:0: class=3D0x020000 card=3D0x10008086 chip=3D0x10008086 r= ev=3D0x03 > hdr=3D0x00 > > vendor =3D 'Intel Corporation' > > device =3D '82542 Gigabit Ethernet Controller' > > class =3D network > > subclass =3D ethernet > > em1@pci0:0:4:0: class=3D0x020000 card=3D0x10008086 chip=3D0x10008086 r= ev=3D0x03 > hdr=3D0x00 > > sandbox# > > > > =3D=3D=3D > > > > However, back in the 6.2 days it allowed it: > > > > =3D=3D=3D > > > > [jayb@cactus ~]$ uname -v > > FreeBSD 6.2-PRERELEASE #0: Tue Dec 26 16:53:18 EST 2006 > root@cactus.iplabs.att.com:/usr/obj/usr/src/sys/CACTUS > > [jayb@cactus ~]$ > > > > [jayb@cactus ~]$ ifconfig em0 > > em0: flags=3D8843 mtu 1500 > > options=3D8 > > inet 10.0.1.152 netmask 0xfffffffc broadcast 10.0.1.155 > > ether 00:90:27:e3:0a:67 > > media: Ethernet autoselect > > status: no carrier > > [jayb@cactus ~]$ > > > > [jayb@cactus ~]$ sudo ifconfig em0 mtu 9000 > > [jayb@cactus ~]$ ifconfig em0 | grep mtu > > em0: flags=3D8843 mtu 9000 > > [jayb@cactus ~]$ > > > > [jayb@cactus ~]$ pciconf -lv | sed -n '/em0@/,/none0@/p' > > em0@pci0:2:0: class=3D0x020000 card=3D0x10008086 chip=3D0x10008086 r= ev=3D0x02 > hdr=3D0x00 > > vendor =3D 'Intel Corporation' > > device =3D '82542 Gigabit Ethernet Controller' > > class =3D network > > subclass =3D ethernet > > none0@pci0:4:0: class=3D0x088000 card=3D0x0767101e chip=3D0x9063101e re= v=3D0x04 > hdr=3D0x00 > > [jayb@cactus ~]$ > > > > Hmm.. Did jumbo frames really work on these cards for you? Intel claims: > "Adapters based on the Intel=AE 82542 and 82573V/E controller do not > support Jumbo Frames. These correspond to the following product names: > > * Intel=AE PRO/1000 Gigabit Server Adapter > * Intel=AE PRO/1000 PM Network Connection " > > As seen from code, the behavior was changed to match the description ("ke= ep > old 82542 from using jumbo frames") and merged to 6-STABLE before 6.3. > > -- > wbr, > pluknet > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 22:05:23 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C9B4C1065670 for ; Thu, 10 Feb 2011 22:05:23 +0000 (UTC) (envelope-from jayb@zoe.braeburn.org) Received: from mail119.messagelabs.com (mail119.messagelabs.com [216.82.241.195]) by mx1.freebsd.org (Postfix) with ESMTP id 670C68FC0A for ; Thu, 10 Feb 2011 22:05:23 +0000 (UTC) X-VirusChecked: Checked X-Env-Sender: jayb@zoe.braeburn.org X-Msg-Ref: server-10.tower-119.messagelabs.com!1297375522!4828014!1 X-StarScan-Version: 6.2.9; banners=-,-,- X-Originating-IP: [144.160.20.145] Received: (qmail 22064 invoked from network); 10 Feb 2011 22:05:22 -0000 Received: from sbcsmtp6.sbc.com (HELO mlpd192.enaf.sfdc.sbc.com) (144.160.20.145) by server-10.tower-119.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 10 Feb 2011 22:05:22 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id p1AM5jO4025834 for ; Thu, 10 Feb 2011 17:05:45 -0500 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id p1AM5dBo025748 for ; Thu, 10 Feb 2011 17:05:40 -0500 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id p1AM5Gmq021960 for ; Thu, 10 Feb 2011 17:05:16 -0500 Received: from oz.mt.att.com (oz.mt.att.com [135.16.165.23]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id p1AM5DDa021849 for ; Thu, 10 Feb 2011 17:05:13 -0500 Received: by oz.mt.att.com (Postfix, from userid 500) id B051F2BF2C; Thu, 10 Feb 2011 17:05:00 -0500 (EST) X-Mailer: emacs 21.2.1 (via feedmail 8 I); VM 7.18 under Emacs 21.2.1 Message-ID: <19796.24842.872566.716691@oz.mt.att.com> Date: Thu, 10 Feb 2011 17:04:58 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable From: Jay Borkenhagen To: Jack Vogel In-Reply-To: References: <19796.13124.805432.342309@oz.mt.att.com> X-GPG-Fingerprint: DDDB 542E D988 94D0 82D3 D198 7DED 6648 2308 D3C0 Sender: jayb@zoe.braeburn.org Cc: freebsd-net@freebsd.org, Sergey Kandaurov Subject: Re: cannot set mtu > 1500 on em0 optical Gig-E X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Jay Borkenhagen List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 22:05:23 -0000 Jack Vogel writes: > Thanks Sergey :) >=20 [...] > > > > Hmm.. Did jumbo frames really work on these cards for you=3F Intel= > > claims: "Adapters based on the Intel=AE 82542 and 82573V/E > > controller do not support Jumbo Frames. These correspond to the > > following product names: > > > > * Intel=AE PRO/1000 Gigabit Server Adapter > > * Intel=AE PRO/1000 PM Network Connection " > > > > As seen from code, the behavior was changed to match the > > description ("keep old 82542 from using jumbo frames") and merged > > to 6-STABLE before 6.3. > > Well, I've got tcpdumps showing DF-bit-set pings succeeding to and from such a NIC with MTU >> 1500, on a box running 6.2-RELEASE. (pciconf data below.) In this particular case I have MTU=3D4470, which= is really good enough for what I need, and DF pings succeed up to 'ping -s 4442' Maybe Intel thought they couldn't handle MTUs approaching 9k, or maybe they had problems approaching linerate -- neither of which are concerns in my application... em0@pci0:2:0: class=3D0x020000 card=3D0x10008086 chip=3D0x10008086 re= v=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82542 Gigabit Ethernet Controller' class =3D network subclass =3D ethernet From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 22:16:34 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C9E7106566C for ; Thu, 10 Feb 2011 22:16:34 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5309A8FC15 for ; Thu, 10 Feb 2011 22:16:34 +0000 (UTC) Received: by ywl2 with SMTP id 2so834241ywl.13 for ; Thu, 10 Feb 2011 14:16:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Nb8P3tHaEcvoz4bhd+X6m2rI11tAqdfd634GofBPxyk=; b=qPjgX2K7XD89RxVvH7z0cBS28kmzWbnVzErB8X9MoltHk/HLgUhXz2Uwt3mvN/qzpI NotC8k8OhQ5Fo5Ei5KPBHNLciHidfnIt8ZToppEG37UPyba0xj69r+Yp2w6U16kCayPR xQLeLWSWy7X3nn2JNuVhFZqkJMpeEVbcvKX7A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Xy2kcHCkJcj7Qh2cXXoWqzAB8szGIpfF+QiOCZRucvtw2D32kNyuwE57X89eJIY4tN nwXpkLorLRVe0d7us7VEXGhuaN8tzmVnoz7w5GuvIoyHMYsSXMuUUj6vGKOzW0r2+2WS dvqiqknWvC2oaEJTkaIbzMXmetp16Du1FS1aM= MIME-Version: 1.0 Received: by 10.150.148.18 with SMTP id v18mr4097422ybd.54.1297376193596; Thu, 10 Feb 2011 14:16:33 -0800 (PST) Received: by 10.147.167.5 with HTTP; Thu, 10 Feb 2011 14:16:33 -0800 (PST) In-Reply-To: <19796.24842.872566.716691@oz.mt.att.com> References: <19796.13124.805432.342309@oz.mt.att.com> <19796.24842.872566.716691@oz.mt.att.com> Date: Thu, 10 Feb 2011 14:16:33 -0800 Message-ID: From: Jack Vogel To: Jay Borkenhagen Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org, Sergey Kandaurov Subject: Re: cannot set mtu > 1500 on em0 optical Gig-E X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 22:16:34 -0000 Ahem, its our OLDEST 1G part, it was ancient when I took this job 5 years ago :) And the code is working according to spec, jumbos are not supported on the device, that's not the same as saying its not possible. We only have one or two of these devices laying around, and I get a chuckle any time I've asked to use one to repro something, adapters aren't that expensive, time to think about an upgrade :) Jack On Thu, Feb 10, 2011 at 2:04 PM, Jay Borkenhagen wrote: > Jack Vogel writes: > > Thanks Sergey :) > > > [...] > > > > > > Hmm.. Did jumbo frames really work on these cards for you? Intel > > > claims: "Adapters based on the Intel=AE 82542 and 82573V/E > > > controller do not support Jumbo Frames. These correspond to the > > > following product names: > > > > > > * Intel=AE PRO/1000 Gigabit Server Adapter > > > * Intel=AE PRO/1000 PM Network Connection " > > > > > > As seen from code, the behavior was changed to match the > > > description ("keep old 82542 from using jumbo frames") and merged > > > to 6-STABLE before 6.3. > > > > > > Well, I've got tcpdumps showing DF-bit-set pings succeeding to and > from such a NIC with MTU >> 1500, on a box running 6.2-RELEASE. > (pciconf data below.) In this particular case I have MTU=3D4470, which > is really good enough for what I need, and DF pings succeed up to > 'ping -s 4442' > > Maybe Intel thought they couldn't handle MTUs approaching 9k, or maybe > they had problems approaching linerate -- neither of which are > concerns in my application... > > > em0@pci0:2:0: class=3D0x020000 card=3D0x10008086 chip=3D0x10008086 rev= =3D0x03 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D '82542 Gigabit Ethernet Controller' > class =3D network > subclass =3D ethernet > From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 22:23:41 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D420E1065673 for ; Thu, 10 Feb 2011 22:23:41 +0000 (UTC) (envelope-from lists@rewt.org.uk) Received: from frankie.stf.rewt.org.uk (frankie.stf.rewt.org.uk [91.208.177.187]) by mx1.freebsd.org (Postfix) with ESMTP id 88A9A8FC19 for ; Thu, 10 Feb 2011 22:23:41 +0000 (UTC) Received: from [172.16.11.86] (jwh-laptop.barbary [172.16.11.86]) (Authenticated sender: jwh.lists) by frankie.stf.rewt.org.uk (Postfix) with ESMTPA id 47A8950843; Thu, 10 Feb 2011 22:23:06 +0000 (UTC) Message-ID: <4D54656A.8080507@rewt.org.uk> Date: Thu, 10 Feb 2011 22:23:38 +0000 From: Joe Holden User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <20110210155622.GA60117@icarus.home.lan> In-Reply-To: <20110210155622.GA60117@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd@jdc.parodius.com Subject: Re: Reliable PCI wifi cards, and layer 7 filtering X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 22:23:41 -0000 On 10/02/2011 15:56, Jeremy Chadwick wrote: > (I was considering cross-posting this to freebsd-pf but decided against > it, instead starting here first. Please keep me CC'd as I'm not > subscribed to freebsd-net) > > I'm looking into the possibility of using my home FreeBSD box as my home > firewall/NAT box, to replace my Linksys E2000 router (which runs Linux, > specifically the TomatoUSB firmware). > > I plan on using pf for the NAT and firewall layer. ipfw will not be > used (I have long since moved away from it). I've got solutions for > everything except two items: > > 1) Wireless hardware support > - What consumer PCI cards are known to be reliable and have good > support on FreeBSD? It looks like anything that relies on ath(4) > might be a good choice, but I'm not sure what specific chipset is > considered decent/worthwhile, or if there's a specific model of > card from Vendor X(tm) which works great. > - The card and driver need to support both 802.11b and 802.11g > simultaneously. 802.11n (for the future) would also be good. > - Driver or OS needs 128-bit WEP -- this is not a joke, I really do > have devices which do not do WPA or WPA2. > - MAC address filtering is needed too, but it looks like that's > already available (looking at ifconfig(8) man page). > > 2) Layer 7 filtering > - Specifically, the ability to block outbound packets in real-time > which contain certain data in the TCP data portion of the packet. > - More details: there are some HTTP-based requests which some > software I use on XP submits to a server pool to return some ads. > Filtering by IP address isn't possible since the A records of > the FQDN often change. The software in question does not honour > system proxy settings, so use of a proxy (Apache, squid, etc.) > as a solution will not work. > - I filter based on GET parameters or the HTTP: Host header. Thus, > the matching mechanism doesn't need regex; simple substring matches > (e.g. strcasestr()) would work fine. > - Linux has kernel modules called ipt_web and xt_web which can do > exactly this. They return TCP RST to the client which submit the > packet, and never forwarding the original packet out the WAN. > There is 'ipfw-classifyd' which has been somewhat improved by the pfsense team in order to support pf - I don't have the exact url to hand, but IIRC it is hosted on googlecode somewhere. It does what you describe, uses regex to match payload. HTH > Item #2 above seems to be the kicker. Is there anything in the works > regarding such a capability? I'd be more than happy to test out code or > whatever. > Thanks, J From owner-freebsd-net@FreeBSD.ORG Thu Feb 10 23:08:06 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11689106564A for ; Thu, 10 Feb 2011 23:08:06 +0000 (UTC) (envelope-from sullrich@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9A0A88FC14 for ; Thu, 10 Feb 2011 23:08:05 +0000 (UTC) Received: by eyf6 with SMTP id 6so1150697eyf.13 for ; Thu, 10 Feb 2011 15:08:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=PgvcUuuxYxnxFbQ+23dXPKxfFMfrNHmTfI3BgJswdpY=; b=BqK6YU7mCCH8wXIRyLIVKwkEvqI4nnYZC/W+SkFE/lpXoUN1t9NN+5H38Y2HUXC9TJ YFdFRr2FANi7wVnofTg5feb/fiF+DyRlaZEVCuwGzp4QuKwtDHlWnHTIWav4likx6jM9 Y14Yu2Cz0v6CX9vLJUYoQYtwJwlUlwNAY9P68= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=KtLWhG3FXghXtcGu7/p/rB3RUfEUbyXoyf9dVcCIGeWFcQN0OYAlrRqHa5RdCQ4uDC rO8rTf2CrjcQVDQ4RSsxlNngVkG440k4Nx5okyXpnQ8XAFJBPCQs38vE0+7e9JquHxta RQlbQK9rqHJ+jTJTnSePY9HfhgVa96/eDRluM= Received: by 10.213.104.147 with SMTP id p19mr4457426ebo.77.1297377645233; Thu, 10 Feb 2011 14:40:45 -0800 (PST) MIME-Version: 1.0 Received: by 10.213.101.7 with HTTP; Thu, 10 Feb 2011 14:40:25 -0800 (PST) In-Reply-To: <4D54656A.8080507@rewt.org.uk> References: <20110210155622.GA60117@icarus.home.lan> <4D54656A.8080507@rewt.org.uk> From: Scott Ullrich Date: Thu, 10 Feb 2011 17:40:25 -0500 Message-ID: To: Joe Holden Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, freebsd@jdc.parodius.com Subject: Re: Reliable PCI wifi cards, and layer 7 filtering X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 23:08:06 -0000 On Thu, Feb 10, 2011 at 5:23 PM, Joe Holden wrote: > On 10/02/2011 15:56, Jeremy Chadwick wrote: > There is 'ipfw-classifyd' which has been somewhat improved by the pfsense > team in order to support pf - I don't have the exact url to hand, but IIRC > it is hosted on googlecode somewhere. > > It does what you describe, uses regex to match payload. Hi, Our custom code is held here: https://rcs.pfsense.org/projects/pfsense-tools Simply check out the tools repo and inside pfPorts/ you will find all our goodies including ipfw-classifyd. Hope this helps! Scott From owner-freebsd-net@FreeBSD.ORG Fri Feb 11 03:15:37 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B7F91065672; Fri, 11 Feb 2011 03:15:37 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 71C068FC12; Fri, 11 Feb 2011 03:15:37 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1B3FbTd006398; Fri, 11 Feb 2011 03:15:37 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1B3Fb2S006394; Fri, 11 Feb 2011 03:15:37 GMT (envelope-from linimon) Date: Fri, 11 Feb 2011 03:15:37 GMT Message-Id: <201102110315.p1B3Fb2S006394@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/154667: [fxp] link stage change on multicast join/leave X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 03:15:37 -0000 Old Synopsis: link stage change on multicast join/leave (fxp) New Synopsis: [fxp] link stage change on multicast join/leave Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Fri Feb 11 03:15:17 UTC 2011 Responsible-Changed-Why: reclassify and assign. http://www.freebsd.org/cgi/query-pr.cgi?pr=154667 From owner-freebsd-net@FreeBSD.ORG Fri Feb 11 07:57:39 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C550E106564A for ; Fri, 11 Feb 2011 07:57:39 +0000 (UTC) (envelope-from cmb@pfsense.org) Received: from mail.pfsense.org (mail.pfsense.org [69.64.6.29]) by mx1.freebsd.org (Postfix) with ESMTP id 8BB2B8FC14 for ; Fri, 11 Feb 2011 07:57:39 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.pfsense.org (Postfix) with ESMTP id 8F81322E35 for ; Fri, 11 Feb 2011 02:40:54 -0500 (EST) X-Virus-Scanned: amavisd-new at mail.pfsense.org Received: from mail.pfsense.org ([127.0.0.1]) by localhost (localhost [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ehS+EJA6vnlp for ; Fri, 11 Feb 2011 02:40:32 -0500 (EST) Received: from [10.0.64.48] (99-89-64-69.uvs.lsvlky.sbcglobal.net [99.89.64.69]) by mail.pfsense.org (Postfix) with ESMTPSA id DEC9D24F8F for ; Fri, 11 Feb 2011 02:38:08 -0500 (EST) Message-ID: <4D54E75E.1020202@pfsense.org> Date: Fri, 11 Feb 2011 02:38:06 -0500 From: Chris Buechler User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <20110210155622.GA60117@icarus.home.lan> <4D54656A.8080507@rewt.org.uk> In-Reply-To: <4D54656A.8080507@rewt.org.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Reliable PCI wifi cards, and layer 7 filtering X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 07:57:39 -0000 On 2/10/2011 5:23 PM, Joe Holden wrote: > On 10/02/2011 15:56, Jeremy Chadwick wrote: >> (I was considering cross-posting this to freebsd-pf but decided against >> it, instead starting here first. Please keep me CC'd as I'm not >> subscribed to freebsd-net) >> >> I'm looking into the possibility of using my home FreeBSD box as my home >> firewall/NAT box, to replace my Linksys E2000 router (which runs Linux, >> specifically the TomatoUSB firmware). >> >> I plan on using pf for the NAT and firewall layer. ipfw will not be >> used (I have long since moved away from it). I've got solutions for >> everything except two items: >> >> 1) Wireless hardware support >> - What consumer PCI cards are known to be reliable and have good >> support on FreeBSD? It looks like anything that relies on ath(4) >> might be a good choice, but I'm not sure what specific chipset is >> considered decent/worthwhile, or if there's a specific model of >> card from Vendor X(tm) which works great. >> - The card and driver need to support both 802.11b and 802.11g >> simultaneously. 802.11n (for the future) would also be good. >> - Driver or OS needs 128-bit WEP -- this is not a joke, I really do >> have devices which do not do WPA or WPA2. >> - MAC address filtering is needed too, but it looks like that's >> already available (looking at ifconfig(8) man page). >> >> 2) Layer 7 filtering >> - Specifically, the ability to block outbound packets in real-time >> which contain certain data in the TCP data portion of the packet. >> - More details: there are some HTTP-based requests which some >> software I use on XP submits to a server pool to return some ads. >> Filtering by IP address isn't possible since the A records of >> the FQDN often change. The software in question does not honour >> system proxy settings, so use of a proxy (Apache, squid, etc.) >> as a solution will not work. >> - I filter based on GET parameters or the HTTP: Host header. Thus, >> the matching mechanism doesn't need regex; simple substring >> matches >> (e.g. strcasestr()) would work fine. >> - Linux has kernel modules called ipt_web and xt_web which can do >> exactly this. They return TCP RST to the client which submit the >> packet, and never forwarding the original packet out the WAN. >> > There is 'ipfw-classifyd' which has been somewhat improved by the > pfsense team in order to support pf - I don't have the exact url to > hand, but IIRC it is hosted on googlecode somewhere. It's in git at rcs.pfsense.org in the tools repo. Note divert + PF in FreeBSD is also specific to patches we use that aren't in stock FreeBSD yet, you can easily apply those to RELENG_8_1 though. Kernel patches are also in the tools repo. All of it's BSD licensed, you're welcome to grab whatever you want to use. From owner-freebsd-net@FreeBSD.ORG Fri Feb 11 08:08:44 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AEBF106566B; Fri, 11 Feb 2011 08:08:44 +0000 (UTC) (envelope-from monthadar@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id CB8F88FC0C; Fri, 11 Feb 2011 08:08:43 +0000 (UTC) Received: by wyf19 with SMTP id 19so2216676wyf.13 for ; Fri, 11 Feb 2011 00:08:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=KAtirXgTFn1tZySJhzYBzNaivMuYx5BO0m01CAAUcAg=; b=C30YAB2kSwE2j35J80qg4ceeyaG4PjNYYF/Vm/mp0SHxRPAGM11ZU8yOZITa+1tDNB B2Ghcs3yFbKnWyPSzVVzhlaLXeFIXToPVVOWeBAxvGLJY7Mi4pklcXEEQksMr0OySg32 siHanLu7d4nBIcbcrfIqr+sNFQ9Ewnxb0J+DU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=nDiWdyESquTreZyOURpzI7NemkgNyqXHjYDd0RQDjU2rc5BEPNZGdd+ccTq23rnoWr d4Cmn08L2P/KOLun723lQsisrGuE23iR5lOXFZH725GFYi1C9l1wqCxMKESuiF6Ql6/A 3H+VCYGx7Y3hPDAZmY5ICyvr2htv+itD8YxMw= MIME-Version: 1.0 Received: by 10.227.157.9 with SMTP id z9mr167228wbw.79.1297411722748; Fri, 11 Feb 2011 00:08:42 -0800 (PST) Received: by 10.227.134.137 with HTTP; Fri, 11 Feb 2011 00:08:42 -0800 (PST) In-Reply-To: References: <201102101119.07350.bschmidt@freebsd.org> <201102101209.19983.bschmidt@freebsd.org> Date: Fri, 11 Feb 2011 09:08:42 +0100 Message-ID: From: Monthadar Al Jaberi To: bschmidt@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org Subject: Re: 80211s HWMP problem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 08:08:44 -0000 On Thu, Feb 10, 2011 at 12:17 PM, Monthadar Al Jaberi wrote: > On Thu, Feb 10, 2011 at 12:09 PM, Bernhard Schmidt wrote: >> On Thursday, February 10, 2011 11:42:05 Monthadar Al Jaberi wrote: >>> Okej, I run all four scenarios: >>> >>> 0->2 OK, but the first "ping -c 1" got lost >>> 1->3 OK, >>> 2->0 OK, >>> 3->1 FAIL, works after "ping -c 1" 1->2 >>> >>> looks like path is built correct in one way only? :S >> >> I have no clue about the mesh code.. but, that smells like there are a >> few frames discard which are supposed to fill the ARP table. Can I talk >> you into dumping frames on the interfaces to figure out where those >> discards happen? > > please :) Sorry, but what was it you want me to dump? > >> >>> On Thu, Feb 10, 2011 at 11:19 AM, Bernhard Schmidt >> wrote: >>> > On Thursday, February 10, 2011 11:06:41 Monthadar Al Jaberi wrote: >>> >> Hej, wanna check if anyone encountered this problem: >>> >> I setup 4 mesh nodes in a link topology ( 0 <-> 1 <-> 2 <-> 3) >>> >> >>> >> But I cant ping from 3 to 0, or 0 to 3, without first ping between >>> >> the nodes to fill the hwmp route tables. >>> > >>> > Pinging 0 -> 2, 1 -> 3, 2 -> 0 and 3 -> 1 on freshly started nodes >>> > works though? >>> > >>> > -- >>> > Bernhard >> >> -- >> Bernhard >> > > > > -- > //Monthadar Al Jaberi > -- //Monthadar Al Jaberi From owner-freebsd-net@FreeBSD.ORG Fri Feb 11 16:06:48 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B696D106566C; Fri, 11 Feb 2011 16:06:48 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8BBC38FC14; Fri, 11 Feb 2011 16:06:48 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1BG6mYY037767; Fri, 11 Feb 2011 16:06:48 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1BG6m9G037763; Fri, 11 Feb 2011 16:06:48 GMT (envelope-from linimon) Date: Fri, 11 Feb 2011 16:06:48 GMT Message-Id: <201102111606.p1BG6m9G037763@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/154679: [em] Fatal trap 12: "em1 taskq" only at startup (8.1-RELEASE) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 16:06:48 -0000 Old Synopsis: [Fatal trap 12] "em1 taskq" only at startup (8.1-RELEASE) New Synopsis: [em] Fatal trap 12: "em1 taskq" only at startup (8.1-RELEASE) Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Fri Feb 11 16:06:24 UTC 2011 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=154679 From owner-freebsd-net@FreeBSD.ORG Fri Feb 11 18:40:07 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C1D8106566C for ; Fri, 11 Feb 2011 18:40:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6AFF58FC19 for ; Fri, 11 Feb 2011 18:40:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1BIe7Bd097732 for ; Fri, 11 Feb 2011 18:40:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1BIe7m5097731; Fri, 11 Feb 2011 18:40:07 GMT (envelope-from gnats) Date: Fri, 11 Feb 2011 18:40:07 GMT Message-Id: <201102111840.p1BIe7m5097731@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: "Vladislav V. Prodan" Cc: Subject: Re: kern/154557: [tcp] Freeze tcp-session of the clients, if in the gateway is used quagga X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "Vladislav V. Prodan" List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 18:40:07 -0000 The following reply was made to PR kern/154557; it has been noted by GNATS. From: "Vladislav V. Prodan" To: bug-followup@FreeBSD.org, universite@ukr.net Cc: Subject: Re: kern/154557: [tcp] Freeze tcp-session of the clients, if in the gateway is used quagga Date: Fri, 11 Feb 2011 20:31:03 +0200 Put in mpd.conf option: set iface enable tcpmssfix and it worked. -- Vladislav V. Prodan VVP24-UANIC +38[067]4584408 +38[099]4060508 vlad11@jabber.ru From owner-freebsd-net@FreeBSD.ORG Fri Feb 11 19:53:57 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF6731065679 for ; Fri, 11 Feb 2011 19:53:57 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id A92908FC0A for ; Fri, 11 Feb 2011 19:53:57 +0000 (UTC) Received: by pwi10 with SMTP id 10so649695pwi.13 for ; Fri, 11 Feb 2011 11:53:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=uIPVg2CGnO2Pptk63Irf1RsVDrKlOEmZ6UuZFdJ0qe8=; b=auQcy2zFzW1rIt3a9hahIrkoWPDKdf5GBGwsVlutaQqKa4DPs3mpY5FplYPRvzl6rP Z1sD24e48O1ehQ8knLd+Kr1vbSbJ+aK1LwAGeJqWeB6Ger+IqIxxIaCwcISlHNokhD24 7uQh7cblgowEspNiQeP6JWfNFMyhi886l4PXw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=HvqsBdXai5MHIrJWoikEKrVIDGCRjGRnguj67QWT8ETRVVfPkgxQgIqAxcPiXY8VCz LOtFdXi1lS3yqh+80Y5Xw178k9YdWv8racbyOVkvtbp6JcjrzJo9Z4nug28LK1s1expq hSQLRZtuRk/oWGNCR1ilaInfe/H/spBgZZRPo= MIME-Version: 1.0 Received: by 10.142.142.5 with SMTP id p5mr666222wfd.257.1297454037111; Fri, 11 Feb 2011 11:53:57 -0800 (PST) Received: by 10.142.246.3 with HTTP; Fri, 11 Feb 2011 11:53:57 -0800 (PST) In-Reply-To: <12838373-FE96-443E-8979-AF5408705BF0@freebsd.org> References: <12838373-FE96-443E-8979-AF5408705BF0@freebsd.org> Date: Fri, 11 Feb 2011 14:53:57 -0500 Message-ID: From: Karim Fodil-Lemelin To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 19:53:58 -0000 Hi, I see a commit was made in current (r218530 | jfv | 2011-02-10 20:00:26 -0500 (Thu, 10 Feb 2011)). Is that commit done to address this issue? And if so Is there any MFC planned for 7.4 for this? Thanks, Karim. 2011/2/9 Michael Tuexen > On Feb 9, 2011, at 6:35 PM, Jack Vogel wrote: > > > OK, but the question is why does the ring get totally consumed this way, > the > > ring has 1024 descriptors, it seems unintuitive that that whole quantity > can be > > used without some being recharged. Do you see the system mbuf pool being > > depleted at the same time? > That was the test case I created: I set up a server accepting connections > but not reading anything. So the driver passes the mbufs to the transport > stack and they are not consumed. Then the problem occurs. Then I kill the > server. Now there are mbufs available again, but the driver doesn't know. > > I had the impression that these were the circumstances in which the problem > showed up (mbuf allocations failing). > > > > Since you can reproduce it, do me a favor, in rxeof, change the > processed > > value from 8 to 4 and then 1, effectively call refresh every descriptor, > see if > > that eliminates the issue. > I will do. Need to see if I can do it remotely, since I'm not in my lab > right now. Can do it tomorrow for sure. > > But I do not think that this solves the problem, since I did the things > very slowly and you call it at least when you are leaving rxeof. > > Best regards > Michael > > > > Thanks for your help, > > > > Jack > > > > > > On Wed, Feb 9, 2011 at 2:36 AM, Michael Tuexen > wrote: > > Hi Jack, > > > > I could recreate the problem. When the problem occurs, we see > > > > rx_nxt_check = n > > rx_nxt_refresh = n + 1 > > > > (This was also reported in a mail from Karim) > > > > This means that the *whole* receive ring has no buffers anymore. This can > > occur if, for some amount of time, no clusters are available. > > > > Now outside of the driver, at some point of time, clusters are freed. > > I don't think that igb_refresh_mbufs() gets called, since it only gets > > called from igb_rxeof(), which gets called when a packet has been > received, > > which can not happen since the receive ring is empty. So how can the > driver > > know? I have no idea. Maybe we can periodically check for such an event > > and call igb_refresh_mbufs(). > > > > Does this make sense to you? > > > > Best regards > > Michael > > > > > > On Feb 9, 2011, at 8:32 AM, Jack Vogel wrote: > > > > > Hmmm, well so much for that theory :) > > > > > > Jack > > > > > > > > > On Tue, Feb 8, 2011 at 4:06 PM, Karim Fodil-Lemelin < > fodillemlinkarim@gmail.com> wrote: > > > > > > > > > 2011/2/8 Jack Vogel > > > > > > > > > I have been following this, and thinking about it. I still am working > from a theoretical > > > standpoint, but based on a patch I got quite a long time back and never > quite groked, > > > I believe now that I might have a solution. > > > > > > The original PR and patch was kern/150516 from Beezar Liu, I was never > quite comfortable > > > with the code changes, nor convinced that it was a real issue and not a > misunderstanding. > > > However I think now that this very report might be behind what we are > seeing today. I have > > > a slightly different approach to solving it, of course it remains to be > seen if it handles it > > > properly. > > > > > > Please try the patch I've attached, I'm open to further correction or > polishing of the > > > changes. And thanks to Beezar for his original report and changes, this > is not for em, > > > but if this eliminates the problem its clearly needed in all drivers. > > > > > > Jack > > > > > > > > > Hi Jack, > > > > > > Thanks for your help. I tried your patch and it didn't work so I added > a couple of printf to see if the added code was getting hit: > > > > > > --- a/freebsd/sys/dev/e1000/if_igb.c > > > --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c > > > @@ -612,7 +612,7 @@ igb_attach(device_t dev) > > > device_get_nameunit(dev)); > > > > > > INIT_DEBUGOUT("igb_attach: end"); > > > - > > > + printf("this driver has a patch from Jack Vogel\n"); > > > return (0); > > > > > > err_late: > > > @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, int > *done) > > > struct mbuf *sendmp, *mh, *mp; > > > struct igb_rx_buf *rxbuf; > > > u16 hlen, plen, hdr, vtag; > > > + int commit; > > > bool eop = FALSE; > > > > > > cur = &rxr->rx_base[i]; > > > @@ -4255,10 +4256,23 @@ next_desc: > > > bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, > > > BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); > > > > > > + commit = i; /* capture the old index */ > > > + > > > /* Advance our pointers to the next descriptor. */ > > > if (++i == adapter->num_rx_desc) > > > i = 0; > > > /* > > > + ** Sanity test for ring full, if this > > > + ** happens we need to refresh immediately > > > + ** or refresh may deadlock. > > > + */ > > > + if (i == rxr->next_to_refresh) { > > > + igb_refresh_mbufs(rxr, commit); > > > + printf("igb_refresh_mbufs called with commit > %d\n", commit); > > > + processed = 0; > > > + } > > > + > > > + /* > > > ** Send to the stack or LRO > > > */ > > > if (sendmp != NULL) { > > > > > > Here is the results: > > > > > > # dmesg | grep Vogel > > > this driver has a patch from Jack Vogel > > > this driver has a patch from Jack Vogel > > > > > > # netstat -m > > > 60453/52707/113160 mbufs in use (current/cache/total) > > > 48416/51584/100000/100000 mbuf clusters in use > (current/cache/total/max) > > > 2894/690 mbuf+clusters out of packet secondary zone in use > (current/cache) > > > 11946/854/12800/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > > > 164834K/119760K/284595K bytes allocated to network > (current/cache/total) > > > 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > > > 0/4/6656 sfbufs in use (current/peak/max) > > > 0 requests for sfbufs denied > > > 0 requests for sfbufs delayed > > > 0 requests for I/O initiated by sendfile > > > 0 calls to protocol drain routines > > > # dmesg | grep commit > > > > > > At this point RX has hung. > > > > > > Somehow the check (i == rxr->next_to_refresh) is never true in this > case. Also, I did read kern/150516 and couldn't wrap my head around the > patch for the em driver that Beezar Liu suggested. > > > > > > Regards, > > > > > > Karim. > > > > > > > > > > > > From owner-freebsd-net@FreeBSD.ORG Fri Feb 11 19:57:09 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A78E2106564A for ; Fri, 11 Feb 2011 19:57:09 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5522C8FC16 for ; Fri, 11 Feb 2011 19:57:09 +0000 (UTC) Received: by gwj21 with SMTP id 21so1260417gwj.13 for ; Fri, 11 Feb 2011 11:57:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=u9+wCxdOFfvSdx/FU3OJGWFrewqFworiuvpI+IlfJ1o=; b=Rbnv96CAuEl7lras+f0ndLvR9nGCMnt2NWaMpgkX8Ab6L3n6rBJ2QAJuzVazXJcPGD fGkPmZbr8DdZiR0fL2+rG+cA4woDuCDg09osLpK+Li62U85NIQUoJZngjSBuPcTIFvUG PPq8gOPKPsk2uqYIzHcwrxN3UimgTGN4MyckY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=A1wadKZpyLutUDU5qYEfdUyvcR0XLTyxiTMqtHIpwg2flETuF9lnav0i+voy/Nejt2 tHOaOaAGpcNODN+8Wjti8gQ1lrohIWjaDSaLMf2jU1BPidKzsSItlOFU8mokJn1F9Hll dw/lrE54JTOhdZQgozZbiVBBov9NaD6H5PnBY= MIME-Version: 1.0 Received: by 10.147.34.2 with SMTP id m2mr1130219yaj.2.1297454228180; Fri, 11 Feb 2011 11:57:08 -0800 (PST) Received: by 10.147.167.5 with HTTP; Fri, 11 Feb 2011 11:57:08 -0800 (PST) In-Reply-To: References: <12838373-FE96-443E-8979-AF5408705BF0@freebsd.org> Date: Fri, 11 Feb 2011 11:57:08 -0800 Message-ID: From: Jack Vogel To: Karim Fodil-Lemelin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 19:57:09 -0000 Yes, it was the fix for this, Mike has tested and confirmed it eliminates the problem, and yes I will MFC the relevant bits back to 8 and 7 for you asap. Jack On Fri, Feb 11, 2011 at 11:53 AM, Karim Fodil-Lemelin < fodillemlinkarim@gmail.com> wrote: > Hi, > > I see a commit was made in current (r218530 | jfv | 2011-02-10 20:00:26 > -0500 (Thu, 10 Feb 2011)). Is that commit done to address this issue? > > And if so Is there any MFC planned for 7.4 for this? > > Thanks, > > Karim. > > 2011/2/9 Michael Tuexen > > On Feb 9, 2011, at 6:35 PM, Jack Vogel wrote: >> >> > OK, but the question is why does the ring get totally consumed this way, >> the >> > ring has 1024 descriptors, it seems unintuitive that that whole quantity >> can be >> > used without some being recharged. Do you see the system mbuf pool being >> > depleted at the same time? >> That was the test case I created: I set up a server accepting connections >> but not reading anything. So the driver passes the mbufs to the transport >> stack and they are not consumed. Then the problem occurs. Then I kill the >> server. Now there are mbufs available again, but the driver doesn't know. >> >> I had the impression that these were the circumstances in which the >> problem >> showed up (mbuf allocations failing). >> > >> > Since you can reproduce it, do me a favor, in rxeof, change the >> processed >> > value from 8 to 4 and then 1, effectively call refresh every descriptor, >> see if >> > that eliminates the issue. >> I will do. Need to see if I can do it remotely, since I'm not in my lab >> right now. Can do it tomorrow for sure. >> >> But I do not think that this solves the problem, since I did the things >> very slowly and you call it at least when you are leaving rxeof. >> >> Best regards >> Michael >> > >> > Thanks for your help, >> > >> > Jack >> > >> > >> > On Wed, Feb 9, 2011 at 2:36 AM, Michael Tuexen >> wrote: >> > Hi Jack, >> > >> > I could recreate the problem. When the problem occurs, we see >> > >> > rx_nxt_check = n >> > rx_nxt_refresh = n + 1 >> > >> > (This was also reported in a mail from Karim) >> > >> > This means that the *whole* receive ring has no buffers anymore. This >> can >> > occur if, for some amount of time, no clusters are available. >> > >> > Now outside of the driver, at some point of time, clusters are freed. >> > I don't think that igb_refresh_mbufs() gets called, since it only gets >> > called from igb_rxeof(), which gets called when a packet has been >> received, >> > which can not happen since the receive ring is empty. So how can the >> driver >> > know? I have no idea. Maybe we can periodically check for such an event >> > and call igb_refresh_mbufs(). >> > >> > Does this make sense to you? >> > >> > Best regards >> > Michael >> > >> > >> > On Feb 9, 2011, at 8:32 AM, Jack Vogel wrote: >> > >> > > Hmmm, well so much for that theory :) >> > > >> > > Jack >> > > >> > > >> > > On Tue, Feb 8, 2011 at 4:06 PM, Karim Fodil-Lemelin < >> fodillemlinkarim@gmail.com> wrote: >> > > >> > > >> > > 2011/2/8 Jack Vogel >> > > >> > > >> > > I have been following this, and thinking about it. I still am working >> from a theoretical >> > > standpoint, but based on a patch I got quite a long time back and >> never quite groked, >> > > I believe now that I might have a solution. >> > > >> > > The original PR and patch was kern/150516 from Beezar Liu, I was >> never quite comfortable >> > > with the code changes, nor convinced that it was a real issue and not >> a misunderstanding. >> > > However I think now that this very report might be behind what we are >> seeing today. I have >> > > a slightly different approach to solving it, of course it remains to >> be seen if it handles it >> > > properly. >> > > >> > > Please try the patch I've attached, I'm open to further correction or >> polishing of the >> > > changes. And thanks to Beezar for his original report and changes, >> this is not for em, >> > > but if this eliminates the problem its clearly needed in all drivers. >> > > >> > > Jack >> > > >> > > >> > > Hi Jack, >> > > >> > > Thanks for your help. I tried your patch and it didn't work so I added >> a couple of printf to see if the added code was getting hit: >> > > >> > > --- a/freebsd/sys/dev/e1000/if_igb.c >> > > --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c >> > > @@ -612,7 +612,7 @@ igb_attach(device_t dev) >> > > device_get_nameunit(dev)); >> > > >> > > INIT_DEBUGOUT("igb_attach: end"); >> > > - >> > > + printf("this driver has a patch from Jack Vogel\n"); >> > > return (0); >> > > >> > > err_late: >> > > @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, int >> *done) >> > > struct mbuf *sendmp, *mh, *mp; >> > > struct igb_rx_buf *rxbuf; >> > > u16 hlen, plen, hdr, vtag; >> > > + int commit; >> > > bool eop = FALSE; >> > > >> > > cur = &rxr->rx_base[i]; >> > > @@ -4255,10 +4256,23 @@ next_desc: >> > > bus_dmamap_sync(rxr->rxdma.dma_tag, >> rxr->rxdma.dma_map, >> > > BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); >> > > >> > > + commit = i; /* capture the old index */ >> > > + >> > > /* Advance our pointers to the next descriptor. */ >> > > if (++i == adapter->num_rx_desc) >> > > i = 0; >> > > /* >> > > + ** Sanity test for ring full, if this >> > > + ** happens we need to refresh immediately >> > > + ** or refresh may deadlock. >> > > + */ >> > > + if (i == rxr->next_to_refresh) { >> > > + igb_refresh_mbufs(rxr, commit); >> > > + printf("igb_refresh_mbufs called with commit >> %d\n", commit); >> > > + processed = 0; >> > > + } >> > > + >> > > + /* >> > > ** Send to the stack or LRO >> > > */ >> > > if (sendmp != NULL) { >> > > >> > > Here is the results: >> > > >> > > # dmesg | grep Vogel >> > > this driver has a patch from Jack Vogel >> > > this driver has a patch from Jack Vogel >> > > >> > > # netstat -m >> > > 60453/52707/113160 mbufs in use (current/cache/total) >> > > 48416/51584/100000/100000 mbuf clusters in use >> (current/cache/total/max) >> > > 2894/690 mbuf+clusters out of packet secondary zone in use >> (current/cache) >> > > 11946/854/12800/12800 4k (page size) jumbo clusters in use >> (current/cache/total/max) >> > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) >> > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) >> > > 164834K/119760K/284595K bytes allocated to network >> (current/cache/total) >> > > 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >> > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >> > > 0/4/6656 sfbufs in use (current/peak/max) >> > > 0 requests for sfbufs denied >> > > 0 requests for sfbufs delayed >> > > 0 requests for I/O initiated by sendfile >> > > 0 calls to protocol drain routines >> > > # dmesg | grep commit >> > > >> > > At this point RX has hung. >> > > >> > > Somehow the check (i == rxr->next_to_refresh) is never true in this >> case. Also, I did read kern/150516 and couldn't wrap my head around the >> patch for the em driver that Beezar Liu suggested. >> > > >> > > Regards, >> > > >> > > Karim. >> > > >> > > >> > >> > >> >> > From owner-freebsd-net@FreeBSD.ORG Fri Feb 11 22:24:34 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 066A51065670 for ; Fri, 11 Feb 2011 22:24:34 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id C24408FC1D for ; Fri, 11 Feb 2011 22:24:33 +0000 (UTC) Received: by pzk32 with SMTP id 32so543593pzk.13 for ; Fri, 11 Feb 2011 14:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=bL099MCNGOS2aU/b2tvspGYtFjTTkKqUg9uFBc26QBc=; b=wz+BdiSFLCMYCpSdgqQ3JhqHJDM7l14jhNWfdF8lvcWJ83+1GV2n8sCgUtV2zwv2ea XnV/U58iIgPWP96im8u4DrInIf40xc4c7kNpo4CbKLV+uJt5GkHitD7wCnXHgKwbmN1i +RLwqnu+zdcTQ06EvD3NaJbM7r6vJIwzoqLRI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=JEZjMbFJZo0JeANyrzQAnGbYb7Bnbb/G4Ndfn+LMqd1tyTMk8FPPe55+vWGlvzr4bC W1Z2s+6v80ZN3zPFXX7rCfrtl+ltDN3G21x/2pByn2U3utLN3uTbRukIO7rF0R7enzM7 kJ9sZX59IcznxO2dhaRU59iLAbbLhEoDrq6cc= MIME-Version: 1.0 Received: by 10.142.131.17 with SMTP id e17mr755747wfd.371.1297463073215; Fri, 11 Feb 2011 14:24:33 -0800 (PST) Received: by 10.142.246.3 with HTTP; Fri, 11 Feb 2011 14:24:33 -0800 (PST) In-Reply-To: References: <12838373-FE96-443E-8979-AF5408705BF0@freebsd.org> Date: Fri, 11 Feb 2011 17:24:33 -0500 Message-ID: From: Karim Fodil-Lemelin To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 22:24:34 -0000 This is great news! Thanks! Btw we've confirmed that your patch (from current) is indeed working for us as well. 2011/2/11 Jack Vogel > Yes, it was the fix for this, Mike has tested and confirmed it eliminates > the problem, and yes > I will MFC the relevant bits back to 8 and 7 for you asap. > > Jack > > > > On Fri, Feb 11, 2011 at 11:53 AM, Karim Fodil-Lemelin < > fodillemlinkarim@gmail.com> wrote: > >> Hi, >> >> I see a commit was made in current (r218530 | jfv | 2011-02-10 20:00:26 >> -0500 (Thu, 10 Feb 2011)). Is that commit done to address this issue? >> >> And if so Is there any MFC planned for 7.4 for this? >> >> Thanks, >> >> Karim. >> >> 2011/2/9 Michael Tuexen >> >> On Feb 9, 2011, at 6:35 PM, Jack Vogel wrote: >>> >>> > OK, but the question is why does the ring get totally consumed this >>> way, the >>> > ring has 1024 descriptors, it seems unintuitive that that whole >>> quantity can be >>> > used without some being recharged. Do you see the system mbuf pool >>> being >>> > depleted at the same time? >>> That was the test case I created: I set up a server accepting connections >>> but not reading anything. So the driver passes the mbufs to the transport >>> stack and they are not consumed. Then the problem occurs. Then I kill the >>> server. Now there are mbufs available again, but the driver doesn't know. >>> >>> I had the impression that these were the circumstances in which the >>> problem >>> showed up (mbuf allocations failing). >>> > >>> > Since you can reproduce it, do me a favor, in rxeof, change the >>> processed >>> > value from 8 to 4 and then 1, effectively call refresh every >>> descriptor, see if >>> > that eliminates the issue. >>> I will do. Need to see if I can do it remotely, since I'm not in my lab >>> right now. Can do it tomorrow for sure. >>> >>> But I do not think that this solves the problem, since I did the things >>> very slowly and you call it at least when you are leaving rxeof. >>> >>> Best regards >>> Michael >>> > >>> > Thanks for your help, >>> > >>> > Jack >>> > >>> > >>> > On Wed, Feb 9, 2011 at 2:36 AM, Michael Tuexen >>> wrote: >>> > Hi Jack, >>> > >>> > I could recreate the problem. When the problem occurs, we see >>> > >>> > rx_nxt_check = n >>> > rx_nxt_refresh = n + 1 >>> > >>> > (This was also reported in a mail from Karim) >>> > >>> > This means that the *whole* receive ring has no buffers anymore. This >>> can >>> > occur if, for some amount of time, no clusters are available. >>> > >>> > Now outside of the driver, at some point of time, clusters are freed. >>> > I don't think that igb_refresh_mbufs() gets called, since it only gets >>> > called from igb_rxeof(), which gets called when a packet has been >>> received, >>> > which can not happen since the receive ring is empty. So how can the >>> driver >>> > know? I have no idea. Maybe we can periodically check for such an event >>> > and call igb_refresh_mbufs(). >>> > >>> > Does this make sense to you? >>> > >>> > Best regards >>> > Michael >>> > >>> > >>> > On Feb 9, 2011, at 8:32 AM, Jack Vogel wrote: >>> > >>> > > Hmmm, well so much for that theory :) >>> > > >>> > > Jack >>> > > >>> > > >>> > > On Tue, Feb 8, 2011 at 4:06 PM, Karim Fodil-Lemelin < >>> fodillemlinkarim@gmail.com> wrote: >>> > > >>> > > >>> > > 2011/2/8 Jack Vogel >>> > > >>> > > >>> > > I have been following this, and thinking about it. I still am working >>> from a theoretical >>> > > standpoint, but based on a patch I got quite a long time back and >>> never quite groked, >>> > > I believe now that I might have a solution. >>> > > >>> > > The original PR and patch was kern/150516 from Beezar Liu, I was >>> never quite comfortable >>> > > with the code changes, nor convinced that it was a real issue and not >>> a misunderstanding. >>> > > However I think now that this very report might be behind what we are >>> seeing today. I have >>> > > a slightly different approach to solving it, of course it remains to >>> be seen if it handles it >>> > > properly. >>> > > >>> > > Please try the patch I've attached, I'm open to further correction or >>> polishing of the >>> > > changes. And thanks to Beezar for his original report and changes, >>> this is not for em, >>> > > but if this eliminates the problem its clearly needed in all drivers. >>> > > >>> > > Jack >>> > > >>> > > >>> > > Hi Jack, >>> > > >>> > > Thanks for your help. I tried your patch and it didn't work so I >>> added a couple of printf to see if the added code was getting hit: >>> > > >>> > > --- a/freebsd/sys/dev/e1000/if_igb.c >>> > > --More--(byte 1253)+++ b/freebsd/sys/dev/e1000/if_igb.c >>> > > @@ -612,7 +612,7 @@ igb_attach(device_t dev) >>> > > device_get_nameunit(dev)); >>> > > >>> > > INIT_DEBUGOUT("igb_attach: end"); >>> > > - >>> > > + printf("this driver has a patch from Jack Vogel\n"); >>> > > return (0); >>> > > >>> > > err_late: >>> > > @@ -4131,6 +4131,7 @@ igb_rxeof(struct igb_queue *que, int count, int >>> *done) >>> > > struct mbuf *sendmp, *mh, *mp; >>> > > struct igb_rx_buf *rxbuf; >>> > > u16 hlen, plen, hdr, vtag; >>> > > + int commit; >>> > > bool eop = FALSE; >>> > > >>> > > cur = &rxr->rx_base[i]; >>> > > @@ -4255,10 +4256,23 @@ next_desc: >>> > > bus_dmamap_sync(rxr->rxdma.dma_tag, >>> rxr->rxdma.dma_map, >>> > > BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); >>> > > >>> > > + commit = i; /* capture the old index */ >>> > > + >>> > > /* Advance our pointers to the next descriptor. */ >>> > > if (++i == adapter->num_rx_desc) >>> > > i = 0; >>> > > /* >>> > > + ** Sanity test for ring full, if this >>> > > + ** happens we need to refresh immediately >>> > > + ** or refresh may deadlock. >>> > > + */ >>> > > + if (i == rxr->next_to_refresh) { >>> > > + igb_refresh_mbufs(rxr, commit); >>> > > + printf("igb_refresh_mbufs called with commit >>> %d\n", commit); >>> > > + processed = 0; >>> > > + } >>> > > + >>> > > + /* >>> > > ** Send to the stack or LRO >>> > > */ >>> > > if (sendmp != NULL) { >>> > > >>> > > Here is the results: >>> > > >>> > > # dmesg | grep Vogel >>> > > this driver has a patch from Jack Vogel >>> > > this driver has a patch from Jack Vogel >>> > > >>> > > # netstat -m >>> > > 60453/52707/113160 mbufs in use (current/cache/total) >>> > > 48416/51584/100000/100000 mbuf clusters in use >>> (current/cache/total/max) >>> > > 2894/690 mbuf+clusters out of packet secondary zone in use >>> (current/cache) >>> > > 11946/854/12800/12800 4k (page size) jumbo clusters in use >>> (current/cache/total/max) >>> > > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) >>> > > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) >>> > > 164834K/119760K/284595K bytes allocated to network >>> (current/cache/total) >>> > > 0/339/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) >>> > > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >>> > > 0/4/6656 sfbufs in use (current/peak/max) >>> > > 0 requests for sfbufs denied >>> > > 0 requests for sfbufs delayed >>> > > 0 requests for I/O initiated by sendfile >>> > > 0 calls to protocol drain routines >>> > > # dmesg | grep commit >>> > > >>> > > At this point RX has hung. >>> > > >>> > > Somehow the check (i == rxr->next_to_refresh) is never true in this >>> case. Also, I did read kern/150516 and couldn't wrap my head around the >>> patch for the em driver that Beezar Liu suggested. >>> > > >>> > > Regards, >>> > > >>> > > Karim. >>> > > >>> > > >>> > >>> > >>> >>> >> > From owner-freebsd-net@FreeBSD.ORG Sat Feb 12 02:20:19 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9DAF106564A for ; Sat, 12 Feb 2011 02:20:19 +0000 (UTC) (envelope-from beezarliu@yahoo.com.cn) Received: from nm5.bullet.mail.bf1.yahoo.com (nm5.bullet.mail.bf1.yahoo.com [98.139.212.164]) by mx1.freebsd.org (Postfix) with SMTP id 8C7B48FC0C for ; Sat, 12 Feb 2011 02:20:19 +0000 (UTC) Received: from [98.139.212.151] by nm5.bullet.mail.bf1.yahoo.com with NNFMP; 12 Feb 2011 02:07:54 -0000 Received: from [98.139.213.4] by tm8.bullet.mail.bf1.yahoo.com with NNFMP; 12 Feb 2011 02:07:54 -0000 Received: from [127.0.0.1] by smtp104.mail.bf1.yahoo.com with NNFMP; 12 Feb 2011 02:07:54 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1297476474; bh=goi1OCs7Hf1e9TjwNzKN5AogPn1UwXdDYLMEFs1o+I4=; h=X-Yahoo-Newman-Id:Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:Date:From:To:Cc:References:Subject:Message-ID:X-mailer:Mime-Version:Content-Type:Content-Transfer-Encoding; b=s8HJ+h/gL1Rl4HOKEz4DmqVyB7Z7Sj7KBtf3xOxUjE5T9sO+iGZ0pXkC1nHUfGKJmQbmPFCCdDisXZ2i75rqBlVnRXH+ldwAR4VartZwZJaqG2zrSIL4C0cAxvumu4pVMwr1pgXeVnisICYgPBKxtuhC0vh8Y4NFNvRSvVLtVxc= X-Yahoo-Newman-Id: 320931.59842.bm@smtp104.mail.bf1.yahoo.com Received: from china (beezarliu@221.137.121.170 with login) by smtp104.mail.bf1.yahoo.com with SMTP; 11 Feb 2011 18:07:53 -0800 PST X-Yahoo-SMTP: YP5UPy2swBBHZGZlvbmOrntlD3fotw-- X-YMail-OSG: TuqwBXQVM1n4xhSzyktNugYKizmEmHXd6n9iOPpz.l2_B_5 FMVXJ2xVrCw6LI7K3Q7r87vAppP0eh3qkLkqUnIlQP0zLA8y_7j89wc6Jkuu 73mhgvzRWqSm3LKF4VqzV4U5UDhG16wuvLsSwEWF23oCR.Cx0aXtDz1KfOSY w_R46HA0Yxz_Mb20onCm7ahfL4BBZA73GeQyjKNLthjnVWSU1GjFNgpeSsXh 3T9dtvoaCdXMgkLgTi6QnAXsP64BsMtQUGPFfSEoxPhj4aWwbBpHcfdhYIcP lEtg9NTJiUTTnZ6zhrg-- X-Yahoo-Newman-Property: ymail-3 Date: Sat, 12 Feb 2011 10:07:52 +0800 From: "beezarliu" To: "Jack Vogel" , "Karim Fodil-Lemelin" References: , , , , , , , , , <12838373-FE96-443E-8979-AF5408705BF0@freebsd.org>, , Message-ID: <201102121007466138878@yahoo.com.cn> X-mailer: Foxmail 6, 10, 201, 20 [cn] Mime-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 12 Feb 2011 02:55:01 +0000 Cc: "freebsd-net@freebsd.org" Subject: Re: Re: igb driver RX (was TX) hangs when out of mbuf clusters X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 02:20:20 -0000 On 2011-02-12 03:57:42, Jack Vogel wrote: >Yes, it was the fix for this, Mike has tested and confirmed it eliminates >the problem, and yes >I will MFC the relevant bits back to 8 and 7 for you asap. > >Jack > > >On Fri, Feb 11, 2011 at 11:53 AM, Karim Fodil-Lemelin < >fodillemlinkarim@gmail.com> wrote: > >> Hi, >> >> I see a commit was made in current (r218530 | jfv | 2011-02-10 20:00:26 >> -0500 (Thu, 10 Feb 2011)). Is that commit done to address this issue? >> >> And if so Is there any MFC planned for 7.4 for this? >> >> Thanks, >> >> Karim. >> I think it's r218583 that is trying to fix the issue. :) Jack, did you try the fix in PR kern/150516, I think it's better to handle some corner cases, such as mbuf shortage, NIC re-initialization... Thanks! Beezar __________________________________________________ ¸Ï¿ì×¢²áÑÅ»¢³¬´óÈÝÁ¿Ãâ·ÑÓÊÏä? http://cn.mail.yahoo.com From owner-freebsd-net@FreeBSD.ORG Sat Feb 12 08:04:58 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0B0F106564A; Sat, 12 Feb 2011 08:04:58 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) by mx1.freebsd.org (Postfix) with ESMTP id A7C298FC0C; Sat, 12 Feb 2011 08:04:57 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id p1C84sRT014374; Sat, 12 Feb 2011 14:04:54 +0600 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4D563F21.2070802@rdtc.ru> Date: Sat, 12 Feb 2011 14:04:49 +0600 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: Gleb Smirnoff References: <4D3011DB.9050900@frasunek.com> <4D30458D.30007@sentex.net> <4D309983.70709@rdtc.ru> <201101141437.55421.jhb@freebsd.org> <4D46575A.802@rdtc.ru> <4D4670C2.4050500@freebsd.org> <4D48513C.40503@rdtc.ru> <20110201185026.GB62007@glebius.int.ru> In-Reply-To: <20110201185026.GB62007@glebius.int.ru> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, John Baldwin Subject: Re: panic: bufwrite: buffer is not busy??? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 08:04:58 -0000 On 02.02.2011 00:50, Gleb Smirnoff wrote: > > This looks like ng_pppoe_disconnect() was called with NULL argument. > > Can you add KDB_TRACE option to kernel? Your boxes for some reason can't > dump core, but with this option we will have at least trace. > Same box, new panic today with new trace. Still no crashdump because of second panic while dumping core. It seems the source of the problem is still somewhere in NETGRAPH. This time it looks as garbage and not NULL... Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0x34646dc7 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff803d3e20 stack pointer = 0x28:0xffffff80000fc8d0 frame pointer = 0x28:0xffffff80000fc910 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (irq256: em0:rx 0) trap number = 12 panic: page fault cpuid = 2 KDB: stack backtrace: X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37 panic() at 0xffffffff8030b567 = panic+0x187 dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330 dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f trap() at 0xffffffff804c155f = trap+0x3df calltrap() at 0xffffffff804a8de4 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff803d3e20, rsp = 0xffffff80000fc8d0, rbp = 0xffffff80000fc910 --- ng_address_hook() at 0xffffffff803d3e20 = ng_address_hook+0x190 ng_rmnode_self() at 0xffffffff803daf65 = ng_rmnode_self+0x1c95 ng_rmnode_self() at 0xffffffff803db43d = ng_rmnode_self+0x216d ip_fastforward() at 0xffffffff80406356 = ip_fastforward+0x7e6 ether_demux() at 0xffffffff803c1841 = ether_demux+0x131 ether_vlanencap() at 0xffffffff803c1c2d = ether_vlanencap+0x27d ata_sii_chipinit() at 0xffffffff801e725a = ata_sii_chipinit+0xe50a ata_sii_chipinit() at 0xffffffff801e75d4 = ata_sii_chipinit+0xe884 intr_event_execute_handlers() at 0xffffffff802e4c44 = intr_event_execute_handlers+0x104 swi_add() at 0xffffffff802e62f5 = swi_add+0x285 fork_exit() at 0xffffffff802e26d8 = fork_exit+0x118 fork_trampoline() at 0xffffffff804a92ae = fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000fccf0, rbp = 0 --- Uptime: 4d1h33m30s Dumping 4087 MB (3 chunks) chunk 0: 1MB (150 pages) ... ok chunk 1: 3575MB (915088 pages)Sleeping thread (tid 100034, pid 12) owns a non-sleepable lock sched_switch() at 0xffffffff80330819 = sched_switch+0xf9 mi_switch() at 0xffffffff80313ac6 = mi_switch+0x176 sleepq_timedwait() at 0xffffffff803478a2 = sleepq_timedwait+0x42 _sleep() at 0xffffffff80314041 = _sleep+0x301 ucom_attach() at 0xffffffff8029756b = ucom_attach+0x129b ucom_attach() at 0xffffffff802976a4 = ucom_attach+0x13d4 ucom_attach() at 0xffffffff8029772b = ucom_attach+0x145b iicbus_release_bus() at 0xffffffff80230b88 = iicbus_release_bus+0x16f8 sc_puts() at 0xffffffff80263bec = sc_puts+0x24c sc_attach_unit() at 0xffffffff8026698c = sc_attach_unit+0x56c cncheckc() at 0xffffffff802cbd55 = cncheckc+0x65 dumpsys() at 0xffffffff804f2865 = dumpsys+0x4d5 isa_nmi() at 0xffffffff804f228c = isa_nmi+0x76c dumpsys() at 0xffffffff804f2623 = dumpsys+0x293 kproc_shutdown() at 0xffffffff8030ac39 = kproc_shutdown+0x1d9 kproc_shutdown() at 0xffffffff8030b120 = kproc_shutdown+0x6c0 panic() at 0xffffffff8030b551 = panic+0x171 dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330 dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f trap() at 0xffffffff804c155f = trap+0x3df calltrap() at 0xffffffff804a8de4 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff803d3e20, rsp = 0xffffff80000fc8d0, rbp = 0xffffff80000fc910 --- ng_address_hook() at 0xffffffff803d3e20 = ng_address_hook+0x190 ng_rmnode_self() at 0xffffffff803daf65 = ng_rmnode_self+0x1c95 ng_rmnode_self() at 0xffffffff803db43d = ng_rmnode_self+0x216d ip_fastforward() at 0xffffffff80406356 = ip_fastforward+0x7e6 ether_demux() at 0xffffffff803c1841 = ether_demux+0x131 ether_vlanencap() at 0xffffffff803c1c2d = ether_vlanencap+0x27d ata_sii_chipinit() at 0xffffffff801e725a = ata_sii_chipinit+0xe50a ata_sii_chipinit() at 0xffffffff801e75d4 = ata_sii_chipinit+0xe884 intr_event_execute_handlers() at 0xffffffff802e4c44 = intr_event_execute_handlers+0x104 swi_add() at 0xffffffff802e62f5 = swi_add+0x285 fork_exit() at 0xffffffff802e26d8 = fork_exit+0x118 fork_trampoline() at 0xffffffff804a92ae = fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000fccf0, rbp = 0 --- panic: sleeping thread cpuid = 2 (gdb) l *0xffffffff803d3e20 0xffffffff803d3e20 is in ng_address_hook (netgraph.h:191). 186 int line); 187 188 static __inline void 189 _chkhook(hook_p hook, char *file, int line) 190 { 191 if (hook->hk_magic != HK_MAGIC) { 192 printf("Accessing freed hook "); 193 dumphook(hook, file, line); 194 } 195 hook->lastline = line; (gdb) x/i 0xffffffff803d3e20 0xffffffff803d3e20 : cmpl $0x78573011,0x64(%r12) From owner-freebsd-net@FreeBSD.ORG Sat Feb 12 09:36:55 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D394610656A3 for ; Sat, 12 Feb 2011 09:36:55 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 652748FC0A for ; Sat, 12 Feb 2011 09:36:55 +0000 (UTC) Received: by ewy24 with SMTP id 24so1728029ewy.13 for ; Sat, 12 Feb 2011 01:36:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:content-type:content-transfer-encoding :date:subject:to:message-id:mime-version:x-mailer; bh=LFqJcMKw+G3K3LbfP3b4P/O+0T0qywXszb/KqnSBvg0=; b=Qm0Z4TAvESMy9ovKDakfro7M311zIPyD9vDNU7ZfVQaVHC5XetS81VescLfQjnAnHd KJtC0fa3Uyq6oUqSpo6PXN07qVLL2JYHcM/WaNey/91U4zsUV+IQ4Vv8qeuTmw4IKihM nNzWSXAoJH8OfYeXVPRqXfQnafT0WHJB1Gm0Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:date:subject:to :message-id:mime-version:x-mailer; b=eCL6a/DFyb/1r+xxAXcNFVYPyCPxDZ8uXFDxNuGSbqM4dFXMgGiJXDjNSCY83YTBAL yLbOBGq0HawSXC0406y825PwMONqWoq1iqxVEMdauRTERIF1VENOYJ3fqPvW2TrWI23q o/ZNULvAs5IybLmfEzMwRXA5r+BkQYtteyLlE= Received: by 10.213.35.209 with SMTP id q17mr1488979ebd.53.1297503414190; Sat, 12 Feb 2011 01:36:54 -0800 (PST) Received: from mytaht-2.totalterror.net (93-152-151-19.ddns.onlinedirect.bg [93.152.151.19]) by mx.google.com with ESMTPS id u1sm253010eeh.4.2011.02.12.01.36.52 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 12 Feb 2011 01:36:53 -0800 (PST) From: Nikolay Denev Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Sat, 12 Feb 2011 11:36:50 +0200 To: freebsd-net@freebsd.org Message-Id: <7C507431-3902-4B7A-B33F-51ECCFFF8306@gmail.com> Mime-Version: 1.0 (Apple Message framework v1082) X-Mailer: Apple Mail (2.1082) Subject: option RADIX_MPATH, RT_LINK_IS_UP() and interface routes. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 09:36:55 -0000 Hello, A quick glance through sys/netinet/ip_output.c shows that interface = routes are short-circuited and not checked for RT_LINK_IS_UP as gateway routes are. Consider the following scenario : A pair of redundant routers : RTR1 and RTR2. Each having dedicated uplink to some ISP and both run BGP, and they also = have a dedicated cross-connection. On the LAN side, they share a IP using CARP. Uplink1 Uplink2 | | | | +------+ +------+ | RTR1 |---| RTR2 | +------+ +------+ | | | | +---+----------+---+ | LAN | +------------------+ Now, if the cable on RTR1 connecting it to the LAN is disconnected, RTR2 will become carp master and will start receiving packets from = clients on LAN and they will be routed ok. But form the ISP point of view the best path to the network is via RTR1, = so the incoming traffic will still be routed thru RTR1 because it's Uplink1 interface is UP and = the BGP session established. This will cause the packets destined to the LAN to be effectively = blackholed, because of the interface route on RTR1. When using kernel with RADIX_MPATH and ospf on both routers RTR1 will = have two routes to the LAN, one interface and one via the crossconnect to RTR2 but still, the interface route will be consen, = regardless of link state up or down. I'm thinking about checking for RT_LINK_IS_UP on interface routes, or = clear the RTF_UP flag on those routes when interface goes link down. Any = other solutions/ideas? Cheers, Nikolay From owner-freebsd-net@FreeBSD.ORG Sat Feb 12 12:50:13 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 607B9106564A for ; Sat, 12 Feb 2011 12:50:13 +0000 (UTC) (envelope-from nvass@gmx.com) Received: from mailout-eu.gmx.com (mailout-eu.gmx.com [213.165.64.42]) by mx1.freebsd.org (Postfix) with SMTP id C18EC8FC1D for ; Sat, 12 Feb 2011 12:50:12 +0000 (UTC) Received: (qmail invoked by alias); 12 Feb 2011 12:50:10 -0000 Received: from adsl-15.109.242.166.tellas.gr (EHLO [192.168.73.192]) [109.242.166.15] by mail.gmx.com (mp-eu004) with SMTP; 12 Feb 2011 13:50:10 +0100 X-Authenticated: #46156728 X-Provags-ID: V01U2FsdGVkX1+hQ+8PSYuhV3XY7hRmerr3jtNnEz+A8kTq6tW/z+ YgfYNfCKrF7mJ1 Message-ID: <4D5681DE.5010908@gmx.com> Date: Sat, 12 Feb 2011 14:49:34 +0200 From: Nikos Vassiliadis User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Nikolay Denev References: <7C507431-3902-4B7A-B33F-51ECCFFF8306@gmail.com> In-Reply-To: <7C507431-3902-4B7A-B33F-51ECCFFF8306@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 Cc: freebsd-net@freebsd.org Subject: Re: option RADIX_MPATH, RT_LINK_IS_UP() and interface routes. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 12:50:13 -0000 On 2/12/2011 11:36 AM, Nikolay Denev wrote: > Hello, > > A quick glance through sys/netinet/ip_output.c shows that interface routes are short-circuited and > not checked for RT_LINK_IS_UP as gateway routes are. > > Consider the following scenario : > > A pair of redundant routers : RTR1 and RTR2. > Each having dedicated uplink to some ISP and both run BGP, and they also have a dedicated cross-connection. > On the LAN side, they share a IP using CARP. > > Uplink1 Uplink2 > | | > | | > +------+ +------+ > | RTR1 |---| RTR2 | > +------+ +------+ > | | > | | > +---+----------+---+ > | LAN | > +------------------+ > > Now, if the cable on RTR1 connecting it to the LAN is disconnected, > RTR2 will become carp master and will start receiving packets from clients on LAN and they will be routed ok. > But form the ISP point of view the best path to the network is via RTR1, so the incoming traffic > will still be routed thru RTR1 because it's Uplink1 interface is UP and the BGP session established. > This will cause the packets destined to the LAN to be effectively blackholed, because of the interface route on RTR1. > When using kernel with RADIX_MPATH and ospf on both routers RTR1 will have two routes to the LAN, one interface and one via > the crossconnect to RTR2 but still, the interface route will be consen, regardless of link state up or down. > > I'm thinking about checking for RT_LINK_IS_UP on interface routes, or clear the RTF_UP flag on those routes when interface goes link down. Any other solutions/ideas? You could try sysutils/heartbeat which is similar in concept with CARP, but runs in userspace and gives you the ability to run scripts in case of a change. It can also ping any IP address to monitor a link's health, that is, "if I cannot get replies from hosta in my LAN, something must be wrong with me, I'll notify the backup host and change my status to backup". Heartbeat wikipedia article: http://en.wikipedia.org/wiki/Linux-HA HTH, Nikos From owner-freebsd-net@FreeBSD.ORG Sat Feb 12 23:10:56 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48AA51065674; Sat, 12 Feb 2011 23:10:56 +0000 (UTC) (envelope-from yongari@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1A1BE8FC0C; Sat, 12 Feb 2011 23:10:56 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1CNAtdX074945; Sat, 12 Feb 2011 23:10:55 GMT (envelope-from yongari@freefall.freebsd.org) Received: (from yongari@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1CNAtna074857; Sat, 12 Feb 2011 23:10:55 GMT (envelope-from yongari) Date: Sat, 12 Feb 2011 23:10:55 GMT Message-Id: <201102122310.p1CNAtna074857@freefall.freebsd.org> To: michiel@boland.org, yongari@FreeBSD.org, freebsd-net@FreeBSD.org, yongari@FreeBSD.org From: yongari@FreeBSD.org Cc: Subject: Re: kern/154667: [fxp] link stage change on multicast join/leave X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 23:10:56 -0000 Synopsis: [fxp] link stage change on multicast join/leave State-Changed-From-To: open->feedback State-Changed-By: yongari State-Changed-When: Sat Feb 12 23:10:20 UTC 2011 State-Changed-Why: Would you try patch at the following URL? http://people.freebsd.org/~yongari/fxp/fxp.mcast.diff Responsible-Changed-From-To: freebsd-net->yongari Responsible-Changed-By: yongari Responsible-Changed-When: Sat Feb 12 23:10:20 UTC 2011 Responsible-Changed-Why: Grab. http://www.freebsd.org/cgi/query-pr.cgi?pr=154667