From owner-freebsd-net@FreeBSD.ORG Tue Aug 10 09:11:51 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D94F106567C for ; Tue, 10 Aug 2010 09:11:51 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 95EC18FC12 for ; Tue, 10 Aug 2010 09:11:50 +0000 (UTC) Received: (qmail 69686 invoked from network); 10 Aug 2010 07:41:53 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 10 Aug 2010 07:41:53 -0000 Message-ID: <4C6117D5.2070207@freebsd.org> Date: Tue, 10 Aug 2010 11:11:49 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.11) Gecko/20100711 Thunderbird/3.0.6 MIME-Version: 1.0 To: Seth Jeacopello References: <4C5DE0D5.7090802@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Server sporadically sending unexpected RST to Client X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Aug 2010 09:11:51 -0000 On 09.08.2010 15:03, Seth Jeacopello wrote: > Thanks for the quick reply Andre; we have some new information. > > First I took some time to review some of the tcpdumps per your > recommendation and have not found /any/ reuse (with most dumps spanning > approx. a one hour time frame and the problem occurring toward the end of > the time frame). OK. I thought this to be the most likely source of the problem. > The client system is another FreeBSD system (we are unsure of the version at > this time). If there is no port reuse then the client OS shouldn't matter. > You may be correct about the syncache simply showing the symptoms; as we dug > deeper we began looking at changes in netisr, in particular the direct > dispatch policy modifications. We've run some tests over the weekend and > found something that seems to work for us. > > We've found that moving from 'Always Direct' to 'Hybrid' mode seems to > resolve the issue for us without any noticed consequences (setting > net.isr.direct_force=0). Can anyone comment on this setting and let us know > of any downsides or problems that may occur running in this mode? I haven't worked on the netisr code but a quick glance suggest that running in hybrid mode should be fine and not cause any further problems. > We believe that this problem is also only isolated to one of our Server > platforms (testing on our other platform is still on-going, though initial > results look good). OK. > Both platforms are Intel based (current generation vs. last generation) with > various differences, though the one that may be most relative is the change > of the on-board NIC from being 'em' based to 'igb' based (that is the > systems with the issue all have 'em' based NICs vs. 'igb' of the newer > systems). This could be red-herring as well, though I feel it's probably a > good idea to include as much information as possible when troubleshooting. It is unlikely that the network card or the driver has anything to do with it. > Thank you for all of your help and I look forward to hearing any further > thoughts on this issue. Please try the attached patch so I get better information from syncache_socket on the particular error that comes up. Socket creation and PCB setup are very complicated areas. -- Andre Index: tcp_syncache.c =================================================================== --- tcp_syncache.c (revision 211131) +++ tcp_syncache.c (working copy) @@ -627,6 +627,7 @@ struct inpcb *inp = NULL; struct socket *so; struct tcpcb *tp; + int error = 0; char *s; INP_INFO_WLOCK_ASSERT(&V_tcbinfo); @@ -675,7 +676,7 @@ } #endif inp->inp_lport = sc->sc_inc.inc_lport; - if (in_pcbinshash(inp) != 0) { + if ((error = in_pcbinshash(inp)) != 0) { /* * Undo the assignments above if we failed to * put the PCB on the hash lists. @@ -687,6 +688,12 @@ #endif inp->inp_laddr.s_addr = INADDR_ANY; inp->inp_lport = 0; + if ((s = tcp_log_addrs(&sc->sc_inc, NULL, NULL, NULL))) { + log(LOG_DEBUG, "%s; %s: in_pcbinshash failed " + "with error %i\n", + s, __func__, error); + free(s, M_TCPLOG); + } goto abort; } #ifdef IPSEC @@ -721,9 +728,15 @@ laddr6 = inp->in6p_laddr; if (IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_laddr)) inp->in6p_laddr = sc->sc_inc.inc6_laddr; - if (in6_pcbconnect(inp, (struct sockaddr *)&sin6, - thread0.td_ucred)) { + if ((error = in6_pcbconnect(inp, (struct sockaddr *)&sin6, + thread0.td_ucred)) != 0) { inp->in6p_laddr = laddr6; + if ((s = tcp_log_addrs(&sc->sc_inc, NULL, NULL, NULL))) { + log(LOG_DEBUG, "%s; %s: in6_pcbconnect failed " + "with error %i\n", + s, __func__, error); + free(s, M_TCPLOG); + } goto abort; } /* Override flowlabel from in6_pcbconnect. */ @@ -750,9 +763,15 @@ laddr = inp->inp_laddr; if (inp->inp_laddr.s_addr == INADDR_ANY) inp->inp_laddr = sc->sc_inc.inc_laddr; - if (in_pcbconnect(inp, (struct sockaddr *)&sin, - thread0.td_ucred)) { + if ((error = in_pcbconnect(inp, (struct sockaddr *)&sin, + thread0.td_ucred)) != 0) { inp->inp_laddr = laddr; + if ((s = tcp_log_addrs(&sc->sc_inc, NULL, NULL, NULL))) { + log(LOG_DEBUG, "%s; %s: in_pcbconnect failed " + "with error %i\n", + s, __func__, error); + free(s, M_TCPLOG); + } goto abort; } }