From owner-freebsd-arch@FreeBSD.ORG Sat Jun 7 01:16:20 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 75EE5624; Sat, 7 Jun 2014 01:16:20 +0000 (UTC) Received: from mail-qg0-x234.google.com (mail-qg0-x234.google.com [IPv6:2607:f8b0:400d:c04::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 15E47251F; Sat, 7 Jun 2014 01:16:20 +0000 (UTC) Received: by mail-qg0-f52.google.com with SMTP id a108so6105032qge.11 for ; Fri, 06 Jun 2014 18:16:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=QP345DZG4URYtqtQyrmPwJB9IgIGioA+EI0e1no4sPw=; b=AtL/AiB3WHZUMhPha+ACcrdZFpm5FKwdB0TXcVqOfj8ONaybyd5lGm+c0TmBPHHWwg EQsiG0k7KaXKYY0+rS2g/AGBn0KzFvptQZh9ZBCNulDVaf/Wn2P6XJcowckANtloOvAK X5t5hDkBMCFnuUqbJ7lkHYRNpzVYR+wDXw02NdTzPbuSdHLvuRU7wFrOMqAaPQmchKCe KkHebkSg6EPrZF7UkrR7ZjLT3WXCzRu3zzA8EGZBA2CKjATT8MDTVNeaVRTzl+K0c5Zq NfdqKQq5W3olYYmR3wjahMkI4cVk7gkEUEGTfxZikFgneunNPLpBEBdiZULMcY/BIMvj m5xw== MIME-Version: 1.0 X-Received: by 10.229.57.129 with SMTP id c1mr14201796qch.7.1402103779207; Fri, 06 Jun 2014 18:16:19 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.43.134 with HTTP; Fri, 6 Jun 2014 18:16:19 -0700 (PDT) In-Reply-To: References: Date: Fri, 6 Jun 2014 18:16:19 -0700 X-Google-Sender-Auth: TQNFhzv309iVcoy_ZokfndYNyyk Message-ID: Subject: Re: Seeing ENOTCAPABLE from write FDs in kqueue? From: Adrian Chadd To: Robert Watson , Pawel Jakub Dawidek , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Jun 2014 01:16:20 -0000 ping? -a On 2 June 2014 10:58, Adrian Chadd wrote: > Hi guys/all, > > I've been tinkering with my RSS stuff where I have multiple listen > sockets, one per worker thread, all terminating high throughput TCP > transactions. It's all HTTP; I'm using libevent, evhttp and a very > small amount of glue code to achieve this. > > libevent craps out from time to time because occasionally one of the > events in kqueue returns ENOTCAPABLE. > > ENOTCAPABLE: ident=3D1686, filter=3D-2, flags=3D0x00004000, fflags=3D0x00= 000000, data=3D93 > > ENOTCAPABLE: ident=3D1324, filter=3D-2, flags=3D0x00004000, fflags=3D0x00= 000000, data=3D93 > > ENOTCAPABLE: ident=3D1740, filter=3D-2, flags=3D0x00004000, fflags=3D0x00= 000000, data=3D93 > > ENOTCAPABLE: ident=3D1628, filter=3D-2, flags=3D0x00004000, fflags=3D0x00= 000000, data=3D93 > > ENOTCAPABLE: ident=3D1199, filter=3D-2, flags=3D0x00004000, fflags=3D0x00= 000000, data=3D93 > > ENOTCAPABLE: ident=3D818, filter=3D-2, flags=3D0x00004000, fflags=3D0x000= 00000, data=3D93 > > ENOTCAPABLE: ident=3D389, filter=3D-2, flags=3D0x00004000, fflags=3D0x000= 00000, data=3D93 > > It's happening on the various data FDs; not on the listen socket. > > What I'm seeing from ktrace: > > 27770 rss-http CAP operation requires , descriptor holds > > > 27770 rss-http CAP operation requires , descriptor holds <> > > 27770 rss-http CAP operation requires , descriptor holds <> > > 27770 rss-http CAP operation requires , descriptor holds <> > > 27770 rss-http CAP operation requires , descriptor holds <> > > 27770 rss-http CAP operation requires , descriptor holds <> > > .. so, why exactly would I be seeing this? Is this some capability > race where we have an FD setup but no capability yet assigned when > it's added into the kqueue set? > > It's happening under high throughput (> 30,000 TCP sessions a second.) > > Where would I continue debugging this? > > Thanks! > > > -a