From owner-freebsd-hackers@freebsd.org Fri Dec 11 21:30:49 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61B67A04F32 for ; Fri, 11 Dec 2015 21:30:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D512114F0 for ; Fri, 11 Dec 2015 21:30:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBBLUhQd093156 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 11 Dec 2015 23:30:43 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBBLUhQd093156 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBBLUgMW093143; Fri, 11 Dec 2015 23:30:42 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 11 Dec 2015 23:30:42 +0200 From: Konstantin Belousov To: Steve Kargl , freebsd-hackers@freebsd.org Subject: Re: sonewconn issue? Message-ID: <20151211213042.GK82577@kib.kiev.ua> References: <20151211160354.GA1005@troutmask.apl.washington.edu> <566B06B0.5060809@FreeBSD.org> <20151211193034.GA3100@troutmask.apl.washington.edu> <20151211210024.GA2613@nparhar-pc> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151211210024.GA2613@nparhar-pc> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Dec 2015 21:30:49 -0000 On Fri, Dec 11, 2015 at 01:00:24PM -0800, Navdeep Parhar wrote: > On Fri, Dec 11, 2015 at 11:30:34AM -0800, Steve Kargl wrote: > > On Fri, Dec 11, 2015 at 09:24:00AM -0800, Navdeep Parhar wrote: > > > On 12/11/2015 08:03, Steve Kargl wrote: > > > > Last night my system became wedged. Inspection of/var/log/messages > > > > revealed 290 messages of the form > > > > > > > > Dec 10 19:27:49 troutmask kernel: sonewconn: pcb 0xfffff8000a76c4b0: > > > > Listen queue overflow: 16 already in queue awaiting > > > > acceptance (1 occurrences) > > > > > > > > During this time, I could connect to the webserver on the system. > > > > ssh into the box would connect, but I never got an actaul session. > > > > This morning I found the console unresponse. > > > > > > > > Any suggestions for identifying the rogue process? > > > > > > > > > > Try "netstat -aLnp tcp" and look at the "Listen" column to see which > > > sockets are backlogged. Then "sockstat -4l" to identify the process > > > that owns the socket. > > > > > > > Thanks for the suggestions. I'll keep this in mind if the > > problem re-appears. Unfortunately, I had no access to the > > box. ssh didn't work. console was unresponse. No serial > > console. > > If you can get into ddb after this happens then you can look up the 4 > tuple from the inpcb (its address is in the message displayed by > sonewconn). The local port is probably the interesting part if it's > a server. > > db> show inpcb > db> show tcpcb > > Regards, > Navdeep > > > > > I forgot to mention that this is FreeBSD-current circa > > Nov 9th, 2015. The box gets updated about once a month > > or so. I believe that the network messages are the consequences of some deadlock which stopped the usermode. If you do get at the ddb prompt, start with the ps command.