From owner-freebsd-net@FreeBSD.ORG Fri Oct 3 17:32:01 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F0F7CB3E; Fri, 3 Oct 2014 17:32:00 +0000 (UTC) Received: from mail-qc0-x231.google.com (mail-qc0-x231.google.com [IPv6:2607:f8b0:400d:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9EC63C79; Fri, 3 Oct 2014 17:32:00 +0000 (UTC) Received: by mail-qc0-f177.google.com with SMTP id c9so1334468qcz.8 for ; Fri, 03 Oct 2014 10:31:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=xOtc314qXnPlVEXUrVn8rMskN9JXDdJSyCjopq+/EaU=; b=VfoPe/mqljkRkMhvrrrJxNhTtA+YTiuDiDxxuCj+Q6KJD3XeIJN9edGyLb3EfSPsZi 60Yhq5nFv4poYl0GfZgNsV+Z7xXCRcJZL1SuLHLaK9GXKxczT4oG96wJqYFtW0Pt0DXU FbaZlf2h9rSzd1Otfal56nnW/LQ5N8VtZcMLpQ3LafzIIuWAmIzKst3MmbzFgjt8Hy+A 15tbrxPJgccEGi4amc2GphTyc/ZIlW5VOx+O49DC3zFcCJ+PB2JUETpiEwv3I9gYyMW5 UasKWftfX607/rjSOgKkqGQcCO9dTWaavLvpOmX0flJvdZW/VFpwNHjn3DsqjRZChNUm ahrA== MIME-Version: 1.0 X-Received: by 10.224.60.129 with SMTP id p1mr5572068qah.99.1412357516721; Fri, 03 Oct 2014 10:31:56 -0700 (PDT) Sender: hiren.panchasara@gmail.com Received: by 10.96.168.194 with HTTP; Fri, 3 Oct 2014 10:31:56 -0700 (PDT) In-Reply-To: <20141003080150.GL29361@mordor.lan> References: <20141002164202.GI29361@mordor.lan> <20141002181958.GJ29361@mordor.lan> <20141003080150.GL29361@mordor.lan> Date: Fri, 3 Oct 2014 10:31:56 -0700 X-Google-Sender-Auth: Dfij3evTil2z_1b8Sb5CUNJfED0 Message-ID: Subject: Re: [jcigar@ulb.ac.be: Listen queue overflow: 8 already in queue awaiting acceptance] From: hiren panchasara To: Julien Cigar Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" , FreeBSD Stable Mailing List X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 17:32:01 -0000 On Fri, Oct 3, 2014 at 1:01 AM, Julien Cigar wrote: > On Thu, Oct 02, 2014 at 04:36:49PM -0700, hiren panchasara wrote: >> On Thu, Oct 2, 2014 at 11:19 AM, Julien Cigar wrote: >> > On Thu, Oct 02, 2014 at 10:24:13AM -0700, hiren panchasara wrote: >> >> On Thu, Oct 2, 2014 at 9:42 AM, Julien Cigar wrote: >> >> > sorry for cross-posting, I'm forwarding this as it seems that part of >> >> > the problem is also related to: >> >> > https://lists.freebsd.org/pipermail/freebsd-net/2014-September/039664.html >> >> >> >> Umm, this looks like a different problem than the subject of this email. >> > >> > yes and no, seems the same hardware (HP and igb) and I have also some >> > "requests for mbufs denied" (https://dpaste.de/t8kJ/raw) without any >> > reasons. I should add that the box hanged a week ago and I had to do a >> > hard reboot, I have the feeling that it's somewhat related to this >> > problem .. >> > >> I suggest you try to debug these 2 problems separately. Did you get a >> chance to look at kgdb to find the culprit process as I suggested >> below? > > I tried what you suggested, but I get a "No struct type named inpcb" > Any idea ? :) Is your kernel not build with debug symbols? Can you share your kernconf? I have following in my kernconf: makeoptions DEBUG=-g options KDB options KDB_TRACE options DDB cheers, Hiren >> >> cheers, >> Hiren >> >> > >> >> > I also wonder if something has been fixed in -STABLE in this area .. >> >> > >> >> > (please keep me in CC as I'm not subscribed on freebsd-net@ an >> >> > freebsd-stable@) >> >> > >> >> > -- >> >> > Julien Cigar >> >> > Belgian Biodiversity Platform (http://www.biodiversity.be) >> >> > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 >> >> > No trees were killed in the creation of this message. >> >> > However, many electrons were terribly inconvenienced. >> >> > >> >> > >> >> > ---------- Forwarded message ---------- >> >> > From: Julien Cigar >> >> > To: freebsd-questions@freebsd.org >> >> > Cc: >> >> > Date: Thu, 2 Oct 2014 11:52:06 +0200 >> >> > Subject: Listen queue overflow: 8 already in queue awaiting acceptance >> >> > Hello, >> >> > >> >> > I'm running 10-RELEASE on a HP Proliant DL160 Gen8 and I'm seeing the >> >> > following in my kernel logs: >> >> > sonewconn: pcb 0xfffff8010e561310: Listen queue overflow: 8 already in >> >> > queue awaiting acceptance >> >> >> >> This usually means the application is not keeping up with the incoming >> >> connections. >> >> > >> >> > I already raised kern.ipc.soacceptqueue to 1024 and netstat -naA | grep >> >> > "fffff8010e561310" returns nothing >> >> >> >> This is the usual way of finding the culprit process. If this doesn't >> >> return anything, it probably means that it is a short-lived process. >> >> >> >> Here is an example of what you could do: >> >> >> >> sonewconn: pcb 0xfffff8008f40cb10: Listen queue overflow: 1 already in queue >> >> awaiting acceptance >> >> >> >> From kgdb, >> >> (kgdb) p ((struct inpcb *)0xfffff8008f40cb10)->inp_inc >> >> $3 = {inc_flags = 0 '\0', inc_len = 0 '\0', inc_fibnum = 0, inc_ie = {ie_fport >> >> = 0, ie_lport = 10295, ie_dependfaddr = { >> >> ie46_foreign = {ia46_pad32 = {0, 0, 0}, ia46_addr4 = {s_addr = 0}}, >> >> ie6_foreign = {__u6_addr = { >> >> __u6_addr8 = '\0' , __u6_addr16 = {0, 0, 0, 0, 0, >> >> 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}}, >> >> ie_dependladdr = {ie46_local = {ia46_pad32 = {0, 0, 0}, ia46_addr4 = >> >> {s_addr = 0}}, ie6_local = {__u6_addr = { >> >> __u6_addr8 = '\0' , __u6_addr16 = {0, 0, 0, 0, 0, >> >> 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}}}} >> >> >> >> Here, ie_lport = 10295 which is in n/w byte order and converting it to host >> >> byte order, 10295 -> 0x2837 and swapping them gives us 0x3728 which is 14120. >> >> >> >> Now, use sockstat to find out what process is on that port: >> >> >> >> $ sockstat -l | grep 14120 >> >> >> >> cheers, >> >> Hiren >> > >> > -- >> > Julien Cigar >> > Belgian Biodiversity Platform (http://www.biodiversity.be) >> > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 >> > No trees were killed in the creation of this message. >> > However, many electrons were terribly inconvenienced. > > -- > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced.