From owner-freebsd-arch@freebsd.org Tue Jan 5 15:20:06 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8282DA63A73 for ; Tue, 5 Jan 2016 15:20:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 64A4318B8 for ; Tue, 5 Jan 2016 15:20:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id D341BB946 for ; Tue, 5 Jan 2016 10:20:02 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Expected taskqueue behavior Date: Tue, 05 Jan 2016 07:19:52 -0800 Message-ID: <9174153.lS3QljxB7A@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 05 Jan 2016 10:20:02 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jan 2016 15:20:06 -0000 Given this bit of code: void cpu_hog(void *context, int pending) { for (;;) ; } static struct taskqueue *tq; static struct task t; void mod_init(void) { int i; TASK_INIT(&t, 0, cpu_hog, NULL); tq = taskqueue_create("test", M_WAITOK, taskqueue_thread_enqueue, &tq); taskqueue_start_threads(&tq, mp_ncpu, PRI_MAX_IDLE, "test taskq"); for (i = 0; i < mp_ncpu; i++) { taskqueue_enqueue(tq, &t); pause("foo", 1); } } How many threads would you expect to be busy? The taskqueue(9) manpage does not describe what happens when you queue a task that is already executing. Currently our implementation will allow the task to execute on a second thread (or a third, fourth, fifth, etc.). I think this behavior is probably "surprising" as I expect most folks assume that a task won't be executed concurrently and rely on this assumption to avoid the need for certain types of serialization in task handlers. Linux workqueues explicitly document work-items as being non-reentrant. >From https://www.kernel.org/doc/Documentation/workqueue.txt Note that the flag WQ_NON_REENTRANT no longer exists as all workqueues are now non-reentrant - any work item is guaranteed to be executed by at most one worker system-wide at any given time. Windows work queues also forbid this though indirectly becuase they forbid extra queueing (which bumps 'pending' on FreeBSD) outright. >From https://msdn.microsoft.com/en-us/library/windows/hardware/ff564587%28v=vs.85%29.aspx Do not call IoQueueWorkItem or IoQueueWorkItemEx to queue a work item that is already in the queue. In checked builds, this error causes a bug check. In retail builds, the error is not detected but can cause corruption of system data structures. I think our current behavior is "surprising" and is a bug from when taskqueue thread pools were added (the case simply wasn't handled). At a minimum we should document the expected behavior (and fix it if its wrong). If we don't change our current behavior then the Linux KPI workqueue implementation will need additional changes as it can no longer simply call taskqueue_enqueue(). -- John Baldwin From owner-freebsd-arch@freebsd.org Wed Jan 6 17:08:43 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41C8BA65A5F for ; Wed, 6 Jan 2016 17:08:43 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [IPv6:2001:470:1f05:b76::196]) by mx1.freebsd.org (Postfix) with ESMTP id 36C15186F; Wed, 6 Jan 2016 17:08:43 +0000 (UTC) (envelope-from bright@mu.org) Received: from Alfreds-MacBook-Pro-2.local (unknown [IPv6:2601:645:8001:cee1:d5d7:5a09:788d:3b60]) by elvis.mu.org (Postfix) with ESMTPSA id 95765345A921; Wed, 6 Jan 2016 09:08:42 -0800 (PST) Subject: Re: Expected taskqueue behavior To: John Baldwin , freebsd-arch@freebsd.org References: <9174153.lS3QljxB7A@ralph.baldwin.cx> From: Alfred Perlstein Message-ID: <568D4A1A.5080204@mu.org> Date: Wed, 6 Jan 2016 09:08:42 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <9174153.lS3QljxB7A@ralph.baldwin.cx> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jan 2016 17:08:43 -0000 On 1/5/16 7:19 AM, John Baldwin wrote: > Given this bit of code: > > void > cpu_hog(void *context, int pending) > { > > for (;;) > ; > } > > static struct taskqueue *tq; > static struct task t; > > void > mod_init(void) > { > int i; > > TASK_INIT(&t, 0, cpu_hog, NULL); > tq = taskqueue_create("test", M_WAITOK, taskqueue_thread_enqueue, > &tq); > taskqueue_start_threads(&tq, mp_ncpu, PRI_MAX_IDLE, "test taskq"); > > for (i = 0; i < mp_ncpu; i++) { > taskqueue_enqueue(tq, &t); > pause("foo", 1); > } > } > > How many threads would you expect to be busy? The taskqueue(9) manpage > does not describe what happens when you queue a task that is already > executing. Currently our implementation will allow the task to execute > on a second thread (or a third, fourth, fifth, etc.). I think this > behavior is probably "surprising" as I expect most folks assume that > a task won't be executed concurrently and rely on this assumption to > avoid the need for certain types of serialization in task handlers. > > Linux workqueues explicitly document work-items as being non-reentrant. > > From https://www.kernel.org/doc/Documentation/workqueue.txt > > > Note that the flag WQ_NON_REENTRANT no longer exists as all workqueues > are now non-reentrant - any work item is guaranteed to be executed by > at most one worker system-wide at any given time. > > > Windows work queues also forbid this though indirectly becuase they > forbid extra queueing (which bumps 'pending' on FreeBSD) outright. > > From https://msdn.microsoft.com/en-us/library/windows/hardware/ff564587%28v=vs.85%29.aspx > > > Do not call IoQueueWorkItem or IoQueueWorkItemEx to queue a work item that > is already in the queue. In checked builds, this error causes a bug check. In > retail builds, the error is not detected but can cause corruption of system > data structures. > > > I think our current behavior is "surprising" and is a bug from when > taskqueue thread pools were added (the case simply wasn't handled). > > At a minimum we should document the expected behavior (and fix it if > its wrong). If we don't change our current behavior then the Linux > KPI workqueue implementation will need additional changes as it can > no longer simply call taskqueue_enqueue(). > Agreed. I had to debug numerous problems due to this in a Javascript application where the behavior was similar. -Alfred