From owner-freebsd-stable@FreeBSD.ORG Fri Oct 3 08:01:56 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 94CA969A; Fri, 3 Oct 2014 08:01:56 +0000 (UTC) Received: from relaygateway01.edpnet.net (relaygateway01.edpnet.net [212.71.1.210]) by mx1.freebsd.org (Postfix) with ESMTP id BD8FF6E8; Fri, 3 Oct 2014 08:01:54 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqIGAB5XLlRNbXB9/2dsb2JhbABfgw5TWIMCx2wMh0sCgQYXAXuEAwEBAQMBAQIgBC8jEAsRAwECAQkTDgICDwUUERYOE4g2DAmoR48zhjoBF49EEQE/EQcGgnGBVAWGK5AFgkGBSYMCAYEsg0GRDoIggUU7LwGBDoE7AQEB X-IPAS-Result: AqIGAB5XLlRNbXB9/2dsb2JhbABfgw5TWIMCx2wMh0sCgQYXAXuEAwEBAQMBAQIgBC8jEAsRAwECAQkTDgICDwUUERYOE4g2DAmoR48zhjoBF49EEQE/EQcGgnGBVAWGK5AFgkGBSYMCAYEsg0GRDoIggUU7LwGBDoE7AQEB X-IronPort-AV: E=Sophos;i="5.04,645,1406584800"; d="scan'208";a="277900906" Received: from 77.109.112.125.adsl.dyn.edpnet.net (HELO mordor.lan) ([77.109.112.125]) by relaygateway01.edpnet.net with ESMTP/TLS/DHE-RSA-AES256-SHA; 03 Oct 2014 09:29:17 +0200 Date: Fri, 3 Oct 2014 10:01:50 +0200 From: Julien Cigar To: hiren panchasara Subject: Re: [jcigar@ulb.ac.be: Listen queue overflow: 8 already in queue awaiting acceptance] Message-ID: <20141003080150.GL29361@mordor.lan> References: <20141002164202.GI29361@mordor.lan> <20141002181958.GJ29361@mordor.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="rBvYE4HrSVE32kP+" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "freebsd-net@freebsd.org" , FreeBSD Stable Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 08:01:56 -0000 --rBvYE4HrSVE32kP+ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Oct 02, 2014 at 04:36:49PM -0700, hiren panchasara wrote: > On Thu, Oct 2, 2014 at 11:19 AM, Julien Cigar wrote: > > On Thu, Oct 02, 2014 at 10:24:13AM -0700, hiren panchasara wrote: > >> On Thu, Oct 2, 2014 at 9:42 AM, Julien Cigar wrote: > >> > sorry for cross-posting, I'm forwarding this as it seems that part of > >> > the problem is also related to: > >> > https://lists.freebsd.org/pipermail/freebsd-net/2014-September/03966= 4.html > >> > >> Umm, this looks like a different problem than the subject of this emai= l. > > > > yes and no, seems the same hardware (HP and igb) and I have also some > > "requests for mbufs denied" (https://dpaste.de/t8kJ/raw) without any > > reasons. I should add that the box hanged a week ago and I had to do a > > hard reboot, I have the feeling that it's somewhat related to this > > problem .. > > > I suggest you try to debug these 2 problems separately. Did you get a > chance to look at kgdb to find the culprit process as I suggested > below? I tried what you suggested, but I get a "No struct type named inpcb" Any idea ? :) >=20 > cheers, > Hiren > >> > > >> > I also wonder if something has been fixed in -STABLE in this area .. > >> > > >> > (please keep me in CC as I'm not subscribed on freebsd-net@ an > >> > freebsd-stable@) > >> > > >> > -- > >> > Julien Cigar > >> > Belgian Biodiversity Platform (http://www.biodiversity.be) > >> > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > >> > No trees were killed in the creation of this message. > >> > However, many electrons were terribly inconvenienced. > >> > > >> > > >> > ---------- Forwarded message ---------- > >> > From: Julien Cigar > >> > To: freebsd-questions@freebsd.org > >> > Cc: > >> > Date: Thu, 2 Oct 2014 11:52:06 +0200 > >> > Subject: Listen queue overflow: 8 already in queue awaiting acceptan= ce > >> > Hello, > >> > > >> > I'm running 10-RELEASE on a HP Proliant DL160 Gen8 and I'm seeing the > >> > following in my kernel logs: > >> > sonewconn: pcb 0xfffff8010e561310: Listen queue overflow: 8 already = in > >> > queue awaiting acceptance > >> > >> This usually means the application is not keeping up with the incoming > >> connections. > >> > > >> > I already raised kern.ipc.soacceptqueue to 1024 and netstat -naA | = grep > >> > "fffff8010e561310" returns nothing > >> > >> This is the usual way of finding the culprit process. If this doesn't > >> return anything, it probably means that it is a short-lived process. > >> > >> Here is an example of what you could do: > >> > >> sonewconn: pcb 0xfffff8008f40cb10: Listen queue overflow: 1 already in= queue > >> awaiting acceptance > >> > >> From kgdb, > >> (kgdb) p ((struct inpcb *)0xfffff8008f40cb10)->inp_inc > >> $3 =3D {inc_flags =3D 0 '\0', inc_len =3D 0 '\0', inc_fibnum =3D 0, in= c_ie =3D {ie_fport > >> =3D 0, ie_lport =3D 10295, ie_dependfaddr =3D { > >> ie46_foreign =3D {ia46_pad32 =3D {0, 0, 0}, ia46_addr4 =3D {s_ad= dr =3D 0}}, > >> ie6_foreign =3D {__u6_addr =3D { > >> __u6_addr8 =3D '\0' , __u6_addr16 =3D {0, = 0, 0, 0, 0, > >> 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}}, > >> ie_dependladdr =3D {ie46_local =3D {ia46_pad32 =3D {0, 0, 0}, ia46= _addr4 =3D > >> {s_addr =3D 0}}, ie6_local =3D {__u6_addr =3D { > >> __u6_addr8 =3D '\0' , __u6_addr16 =3D {0, = 0, 0, 0, 0, > >> 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}}}} > >> > >> Here, ie_lport =3D 10295 which is in n/w byte order and converting it = to host > >> byte order, 10295 -> 0x2837 and swapping them gives us 0x3728 which is= 14120. > >> > >> Now, use sockstat to find out what process is on that port: > >> > >> $ sockstat -l | grep 14120 > >> > >> cheers, > >> Hiren > > > > -- > > Julien Cigar > > Belgian Biodiversity Platform (http://www.biodiversity.be) > > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > > No trees were killed in the creation of this message. > > However, many electrons were terribly inconvenienced. --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --rBvYE4HrSVE32kP+ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJULlfuAAoJEAi2KiTKQR5pFawP/3iWom25X33Tl/4aELt/ouph TkfTbXa50/tZmD3MwAHFLBsOHzp2eAy8r16gHDo/enduBEPsmi8FpANJL7uStH1g +/PEWrYz9/PTD/6en92JETGcYiYkMwZog1PljBoUscZcK/aGvOrNmPSFqniexUBP /1qRvDW5uCnIGHIOTZ2NZoBxFXeYBTiGc5ovE+n5K5b8GLalbqr6h+q2slbmC2Bt fwrLhSBAlscz/fqrLD/i8tZZHaQ5ZAaJUG6dFDiUk8eTqNwPQ7+SZJX6erN2ceAH zrUJhIEr3B6cr3UNlCCa/qCpbb6LqrMfRX6Hcx1IOU48P1qHVjJ9QW6xSYCAd3ky L8CNlAx95Qqbde8cUAEUa2AhtrJs7Aa0C7AItsxpfgBAPLNCnbRCeUw3JECFzIec jzcvkl8TolE0ut7SRqG7yE35D+37AzBsAZRT2kFuv7XODzd8+NsxcNNmd2waUM2D SM9fa1MFqffJ2arioXe6AJ4kASCwK7fB57d+sQI2/AJkfakSvwfh285+2uR+LNes ADLmHkl1uR47L6sTC906gwxnE6wyS10TjLGO6TwFCs15BiFtqS3VyRy9WGVzUmpo y874H893zIOuWTm1qtyPw+ylHDpV0U8htkO1DcvqXpeH8RePYDZHU+VgrWUo5JtJ I6G6D715ydd+cwwx3TBJ =CFZe -----END PGP SIGNATURE----- --rBvYE4HrSVE32kP+--