From owner-freebsd-stable@FreeBSD.ORG Wed Nov 14 01:12:45 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AAB45455 for ; Wed, 14 Nov 2012 01:12:45 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7708D8FC08 for ; Wed, 14 Nov 2012 01:12:45 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id wz12so797886pbc.13 for ; Tue, 13 Nov 2012 17:12:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=mEIQrGlSu/cqPlVWZGIXkdfXp58h3Al2JonbvilAmU0=; b=q6bIC1Uhy67Lt/zfv6OBW55+Pc9runc316oCQ5u6OLAZ4pyxcHF2jWm1bnpA4VT/3w UzycKVoxwargAG/pkn0+slkPOm8Jx9gV/Z0fEqPpt+mlr++neZuXAvFeX2aYWJbIIsEi A+/TeXBxkSU6b5SiC+kOChADxIKXiDTOtpprQ4/QLLxhtxZhjVfpbI+9hIBRB+K7RPZg F8rMTMKS4ZSolNXiw6cuIMKpeEEMp31psuKb6fbeU/XLcq+htpNs7BVs+8gyMOpgiJF5 Vnr4PpqjuxiV7c8QKybA/OZCpVKykkz6sntI3x6rrnFVy3pTvbMUw8dabDvP+upU0P4P W6Vg== MIME-Version: 1.0 Received: by 10.68.209.166 with SMTP id mn6mr45085829pbc.95.1352855564876; Tue, 13 Nov 2012 17:12:44 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.68.124.130 with HTTP; Tue, 13 Nov 2012 17:12:44 -0800 (PST) In-Reply-To: <007F7A73-75F6-48A6-9C01-E7C179CDA48A@hostpoint.ch> References: <6908B498-6978-4995-B081-8D504ECB5C0A@hostpoint.ch> <007F7A73-75F6-48A6-9C01-E7C179CDA48A@hostpoint.ch> Date: Tue, 13 Nov 2012 17:12:44 -0800 X-Google-Sender-Auth: ErqwiNyiZ8lGq8lr74WU3HKc2V4 Message-ID: Subject: Re: thread taskq / unp_gc() using 100% cpu and stalling unix socket IPC From: Adrian Chadd To: Markus Gebert Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 01:12:45 -0000 Oh lordie, just hack the kernel to make IP_BINDANY usable by any uid, not just root. I was hoping that capabilitiies would actually be useful these days, but apparently not. :( Then you can stop this FD exchange nonsense and this problem should go away= . :) Adrian On 13 November 2012 16:41, Markus Gebert wrote= : > > On 13.11.2012, at 19:30, Markus Gebert wrote= : > >> To me it looks like the unix socket GC is triggered way too often and/or= running too long, which uses cpu and worse, causes a lot of contention aro= und the unp_list_lock which in turn causes delays for all processes relayin= g on unix sockets for IPC. >> >> I don't know why the unp_gc() is called so often and what's triggering t= his. > > I have a guess now. Dovecot and relayd both use unix sockets heavily. Acc= ording to dtrace uipc_detach() gets called quite often by dovecot closing u= nix sockets. Each time uipc_detach() is called unp_gc_task is taskqueue_enq= ueue()d if fds are inflight. > > in uipc_detach(): > 682 if (local_unp_rights) > 683 taskqueue_enqueue(taskqueue_thread, &unp_gc_task)= ; > > We use relayd in a way that keeps the source address of the client when c= onnecting to the backend server (transparent load balancing). This requires= IP_BINDANY on the socket which cannot be set by unprivileged processes, so= relayd sends the socket fd to the parent process just to set the socket op= tion and send it back. This means an fd gets transferred twice for every ne= w backend connection. > > So we have dovecot calling uipc_detach() often and relayd making it likel= y that fds are inflight (unp_rights > 0). With a certain amount of load thi= s could cause unp_gc_task to be added to the thread taskq too often, slowin= g everything unix socket related down by holding global locks in unp_gc(). > > I don't know if the slowdown can even cause a negative feedback loop at s= ome point by inreasing the chance of fds being inflight. This would explain= why sometimes the condition goes away by itself and sometimes requires int= ervention (taking load away for a moment). > > I'll look into a way to (dis)prove all this tomorrow. Ideas still welcome= :-). > > > Markus > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"