From owner-freebsd-arch@FreeBSD.ORG Tue Feb 11 18:56:47 2014 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0DADED42; Tue, 11 Feb 2014 18:56:47 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C46881499; Tue, 11 Feb 2014 18:56:46 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s1BIueiC054785 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 11 Feb 2014 10:56:40 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s1BIueUM054784; Tue, 11 Feb 2014 10:56:40 -0800 (PST) (envelope-from jmg) Date: Tue, 11 Feb 2014 10:56:39 -0800 From: John-Mark Gurney To: security@FreeBSD.org, arch@FreeBSD.org Subject: CFR: unifing sha256 userland/kernel implementation... Message-ID: <20140211185639.GK34851@funkthat.com> Mail-Followup-To: security@FreeBSD.org, arch@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 11 Feb 2014 10:56:40 -0800 (PST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Feb 2014 18:56:47 -0000 I did some performance testing on sha256, and found that the libmd version is significantly faster, ~20%, than the kernel version. Even if you enable SHA2_UNROLL_TRANSFORM (which isn't the default), the version in libmd is still faster. So, this patch moves libmd's sha256c.c and sha256.h into the kernel, and adapts the userland to pull the version from the kernel. This change removes sha256 from the existing sha2.c file, and does some minor cleanup of types in sha2. I have tested this w/ ZFS using sha256 checksums, and a ZFS made pre-patch is read fine by a kernel post patch. I have also run the tests in lib/libmd and they all pass fine. Passes buildworld/buildkernel/installkernel/reboot/installworld/reboot/test. Patch: https://www.funkthat.com/~jmg/sha256.kern.patch Following stats are in seconds to digest 100000 10000-byte blocks, calculated using sha256 -t: $ ministat soft.times kernsoft.times x soft.times + kernsoft.times +------------------------------------------------------------------------------+ |x xx xx +++ + +| | |___________AM_________| |_______M_____A______________| | +------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 5 6.775387 8.279581 7.848128 7.792094 0.60912664 + 5 8.997429 10.768921 9.090787 9.4359144 0.75040822 Difference at 95.0% confidence 1.64382 +/- 0.99674 21.096% +/- 12.7917% (Student's t, pooled s = 0.683428) This is in preperation of bringing in an SSE4 accelerated version of sha256 (for both userland and kernel) that sees a 2x performance increase. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Tue Feb 11 19:49:44 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CE962F9; Tue, 11 Feb 2014 19:49:44 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A83A519A0; Tue, 11 Feb 2014 19:49:44 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 9BF7BB981; Tue, 11 Feb 2014 14:49:43 -0500 (EST) From: John Baldwin To: freebsd-hackers@freebsd.org Subject: Re: core dump vs kern.ipc.shm_use_phys Date: Tue, 11 Feb 2014 13:21:02 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <52DFAC31.6030905@FreeBSD.org> In-Reply-To: <52DFAC31.6030905@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201402111321.02294.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 11 Feb 2014 14:49:43 -0500 (EST) Cc: Andriy Gapon , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Feb 2014 19:49:44 -0000 On Wednesday, January 22, 2014 6:32:01 am Andriy Gapon wrote: > > I seems that if kern.ipc.shm_use_phys is enabled then shared memory regions are > not included into a coredump. Seems that each_writable_segment() in > sys/kern/imgact_elf.c skips OBJT_PHYS objects. Hmm, that may be a feature. I often map large shared memory segments with MAP_NOCORE on purpose. Is it to tell if a given OBJT_PHYS object is a SYSV SHM object? (I assume it isn't.) -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Wed Feb 12 00:39:09 2014 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 65BC2BBE; Wed, 12 Feb 2014 00:39:09 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2414815BA; Wed, 12 Feb 2014 00:39:08 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s1C0d7Cr059263 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 11 Feb 2014 16:39:08 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s1C0d7Qc059262; Tue, 11 Feb 2014 16:39:07 -0800 (PST) (envelope-from jmg) Date: Tue, 11 Feb 2014 16:39:07 -0800 From: John-Mark Gurney To: freebsd-security@FreeBSD.org, arch@FreeBSD.org Subject: Re: CFR: unifing sha256 userland/kernel implementation... Message-ID: <20140212003907.GM34851@funkthat.com> Mail-Followup-To: freebsd-security@FreeBSD.org, arch@FreeBSD.org References: <20140211185639.GK34851@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140211185639.GK34851@funkthat.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 11 Feb 2014 16:39:08 -0800 (PST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Feb 2014 00:39:09 -0000 John-Mark Gurney wrote this message on Tue, Feb 11, 2014 at 10:56 -0800: > I did some performance testing on sha256, and found that the libmd > version is significantly faster, ~20%, than the kernel version. Even > if you enable SHA2_UNROLL_TRANSFORM (which isn't the default), the > version in libmd is still faster. > > So, this patch moves libmd's sha256c.c and sha256.h into the kernel, > and adapts the userland to pull the version from the kernel. This > change removes sha256 from the existing sha2.c file, and does some > minor cleanup of types in sha2. > > I have tested this w/ ZFS using sha256 checksums, and a ZFS made > pre-patch is read fine by a kernel post patch. I have also run > the tests in lib/libmd and they all pass fine. Passes > buildworld/buildkernel/installkernel/reboot/installworld/reboot/test. > > Patch: > https://www.funkthat.com/~jmg/sha256.kern.patch > > Following stats are in seconds to digest 100000 10000-byte blocks, > calculated using sha256 -t: > $ ministat soft.times kernsoft.times > x soft.times > + kernsoft.times > +------------------------------------------------------------------------------+ > |x xx xx +++ + +| > | |___________AM_________| |_______M_____A______________| | > +------------------------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 5 6.775387 8.279581 7.848128 7.792094 0.60912664 > + 5 8.997429 10.768921 9.090787 9.4359144 0.75040822 > Difference at 95.0% confidence > 1.64382 +/- 0.99674 > 21.096% +/- 12.7917% > (Student's t, pooled s = 0.683428) > > This is in preperation of bringing in an SSE4 accelerated version of > sha256 (for both userland and kernel) that sees a 2x performance > increase. Sorry, security@ != freebsd-security@... This is now going to the correct email.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Wed Feb 12 02:47:16 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 365E3A0B; Wed, 12 Feb 2014 02:47:16 +0000 (UTC) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D86AB1666; Wed, 12 Feb 2014 02:47:15 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id i13so13094160qae.13 for ; Tue, 11 Feb 2014 18:47:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=4niL9FfnftPB1j7gfKOyMfw6OrBlF2+ci16hFCF2hJA=; b=pQsS3qLDL/90RrgEN3fBHTLoKxtiTRAqwXPcSRRKEzXVaXvcnk/GR+zQ0Imj2W18Y+ eSPKxjMBagXkihg2B80zJxSM3zmPrncdBwO6EM12yyjML8ZABepFhSey6voYqrC5SXI2 jR2v8PFJ7o+gzs3cE6x0/LyYE6CUCQlDm0ktUz2MVzUsV8PSwHEOWFsRKgDC92DNisEI xWD2HvMMHbhXjxbLmmcgjggfE5AeIjtoNhkLEn9bxRj54gtEOmejJe6BAzW5XVzkkLFR iJiPkPhXJzCUTvoT4I/KqpkStc8hOvi2+SSWYfdDrGzXzHllCNCeEXqb5vvv1CPq7XiN Jdtw== MIME-Version: 1.0 X-Received: by 10.224.11.136 with SMTP id t8mr62871475qat.26.1392173234336; Tue, 11 Feb 2014 18:47:14 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.52.8 with HTTP; Tue, 11 Feb 2014 18:47:14 -0800 (PST) Date: Tue, 11 Feb 2014 18:47:14 -0800 X-Google-Sender-Auth: e3EkuRPK5KGFmk_WZRtv80vt8o8 Message-ID: Subject: [flowtable] don't insert a flowtable entry if the lle isn't yet valid From: Adrian Chadd To: FreeBSD Net , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Feb 2014 02:47:16 -0000 Hi, Some of the collisions stem from the flowtable entry being created and inserted before the lle has completed - subsequent lookups will fail the LLE_VALID check, but inserting them will immediately fail due to a collision. This patch: * doesn't insert a flowtable entry until the lle is valid; * adds a counter to netstat to log when this happens. http://people.freebsd.org/~adrian/netflix/20140211-flowtable-no-insert-on-arp-not-done.diff I'd like to commit this to -HEAD and backport it to -10. Thanks! -a From owner-freebsd-arch@FreeBSD.ORG Thu Feb 13 08:34:40 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 91CC339B for ; Thu, 13 Feb 2014 08:34:40 +0000 (UTC) Received: from mail-pd0-f175.google.com (mail-pd0-f175.google.com [209.85.192.175]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5A40B18A6 for ; Thu, 13 Feb 2014 08:34:40 +0000 (UTC) Received: by mail-pd0-f175.google.com with SMTP id w10so10197094pde.34 for ; Thu, 13 Feb 2014 00:34:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=Z4qH/RRzvgN1CO556ltRSLY2irfWUIV/vxW0A+0TaB8=; b=X3YuLVizQzTGUISiRm50TCyeUUYMUykbByD5/yuChjfPSOwgDUIveoYSUCqnBZJwap kNIZM9y/NFdXMWUwKtTMyxzqqBIVnxgxkKeCyJSFkTR3VdiLBcd1XVxJqK/tFSeytiWd PSTEHFQtWbE3bSIRVJ0cHCHNV4ok0wRCwtt0JC3fO6k8XtYGJ+aXIM/zQPSxwBqdpEN5 vxuWVDKUideF3GdGxyEJKAfDP2YBvZjamZTV48p/qKm3ttcp6NOiDXZXY4xAZ5/zMtNW C1chRUdtzXiy1qb54dMpjvae1Vt5JTHrwNerp/IP0Abc66xSc0AHk8X4TRLnHnciD0gR ay0Q== X-Gm-Message-State: ALoCoQnIxbb5PJ/MsPuR5hloZJFCXoRnhGs7IaED9qVGNL3iOJy1UjkQkX5nlNp6PLN5MdvvH2kS X-Received: by 10.68.35.129 with SMTP id h1mr156589pbj.163.1392280474271; Thu, 13 Feb 2014 00:34:34 -0800 (PST) Received: from Michaels-MacBook-Pro.local (c-98-246-202-204.hsd1.or.comcast.net. [98.246.202.204]) by mx.google.com with ESMTPSA id vg1sm3930187pbc.44.2014.02.13.00.34.32 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 13 Feb 2014 00:34:33 -0800 (PST) Message-ID: <52FC8397.1040604@callfortesting.org> Date: Thu, 13 Feb 2014 00:34:31 -0800 From: Michael Dexter User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Nathan Whitehorn , Jilles Tjoelker Subject: Re: Automatic enabling of serial console References: <5275C597.6070702@freebsd.org> <97944047-D575-4E2E-B687-9871DFE058E3@fisglobal.com> <52769CFE.5080707@freebsd.org> <5281340E.8080009@callfortesting.org> <52813E53.20403@freebsd.org> <5281441E.7060806@freebsd.org> <529A6862.7060308@freebsd.org> <20131201123442.GA6818@stack.nl> <52C9C8C3.7050108@freebsd.org> <52CB3E01.8040605@callfortesting.org> <52CB6809.7090808@freebsd.org> In-Reply-To: <52CB6809.7090808@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-arch@freebsd.org" , Devin Teske , "Teske, Devin" , Peter Grehan , Current Current X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Feb 2014 08:34:40 -0000 Hello all, I am curious if any progress has been made with regard to the automatic enabling of serial consoles? All the best, Michael From owner-freebsd-arch@FreeBSD.ORG Thu Feb 13 19:11:28 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4BC53C14; Thu, 13 Feb 2014 19:11:28 +0000 (UTC) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id F1A601748; Thu, 13 Feb 2014 19:11:27 +0000 (UTC) Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57]) by alto.onthenet.com.au (Postfix) with ESMTPS id DB8641247E; Fri, 14 Feb 2014 05:11:25 +1000 (EST) Received: from Peter-Grehans-MacBook-Pro-2.local (c-50-156-65-192.hsd1.ca.comcast.net [50.156.65.192]) by dommail.onthenet.com.au (MOS 4.2.4-GA) with ESMTP id BRU64225 (AUTH peterg@ptree32.com.au); Fri, 14 Feb 2014 05:11:20 +1000 Message-ID: <52FD18D2.1050007@freebsd.org> Date: Thu, 13 Feb 2014 11:11:14 -0800 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Michael Dexter Subject: Re: Automatic enabling of serial console References: <5275C597.6070702@freebsd.org> <97944047-D575-4E2E-B687-9871DFE058E3@fisglobal.com> <52769CFE.5080707@freebsd.org> <5281340E.8080009@callfortesting.org> <52813E53.20403@freebsd.org> <5281441E.7060806@freebsd.org> <529A6862.7060308@freebsd.org> <20131201123442.GA6818@stack.nl> <52C9C8C3.7050108@freebsd.org> <52CB3E01.8040605@callfortesting.org> <52CB6809.7090808@freebsd.org> <52FC8397.1040604@callfortesting.org> In-Reply-To: <52FC8397.1040604@callfortesting.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Jilles Tjoelker , "Teske, Devin" , Current Current , Nathan Whitehorn , "freebsd-arch@freebsd.org" , Devin Teske X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Feb 2014 19:11:28 -0000 Hi Michael, > I am curious if any progress has been made with regard to the automatic > enabling of serial consoles? Nathan W checked in the base code for this in r260913. I'll make the change to the amd64/i386 ttys file after some more testing. later, Peter. From owner-freebsd-arch@FreeBSD.ORG Thu Feb 13 23:57:34 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 64141376; Thu, 13 Feb 2014 23:57:34 +0000 (UTC) Received: from mail-qc0-x22c.google.com (mail-qc0-x22c.google.com [IPv6:2607:f8b0:400d:c01::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0FCC31359; Thu, 13 Feb 2014 23:57:33 +0000 (UTC) Received: by mail-qc0-f172.google.com with SMTP id c9so19191327qcz.3 for ; Thu, 13 Feb 2014 15:57:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=cz3U0btyXW8MwXJuj5oXkC8m9w5u4r8gLJ6IfDdUCeU=; b=d1OZ6lB8+z9iX8cldj+Pfm4lXHdqQKo+SNqdR4Z+9lqcsq6r7V8kDYMep0maQ6YpTE oTXQtBXusFnoVHpqMEWP/vvkQvlJOBmcJlyoGL2oB4SVkN3hsPqO3Izf59okQuRHlHL2 NgktTfJI9gobd+He/1EpzbLtp9KqPofCA72MrqD4tfAXjjXX8/vnk+KdIJkqxsUXb7V7 BYsdK0VdHHQ63R1Bm/wSnwC48wS/A202Tic9ngqDKAFPY9uovDYt8gvVtTWTJFV6xidQ imZv+Wl1NtvMChdbtfhr0Y9WfXpSl/sUU+tk0ZwJbSxz1r7Vf7jl2By5xiEENzaZDcPx 2OSA== MIME-Version: 1.0 X-Received: by 10.140.50.131 with SMTP id s3mr7347333qga.12.1392335853133; Thu, 13 Feb 2014 15:57:33 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.16.10 with HTTP; Thu, 13 Feb 2014 15:57:33 -0800 (PST) Date: Thu, 13 Feb 2014 15:57:33 -0800 X-Google-Sender-Auth: rVGTpt9nbFWANg8oBRjJkIXiVQk Message-ID: Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? From: Adrian Chadd To: "freebsd-arch@freebsd.org" , "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Feb 2014 23:57:34 -0000 Hi, Whilst digging into collisions in the flowtable code I discovered that a bunch of them are due to some of the flowtable_lookup() code running on a different CPU - the contention on the l2/l3 lookup lock(s) was enough to block things so they'd get an obvious chance to be migrated. So this led me to wonder whether in a fully preemptive kernel, a running kernel thread would stay on the current CPU until it hit a very specific subset of things (exited to userland, hit a lock, etc.) Apparently (according to kan and rwatson) this is how we define fully preemptive - it's not just interruptable at almost any point, but the running task may be interrupted and rescheduled on a different CPU outside of specific critical sections. This means that for the flowtable case, the current setup (without atomics to maintain the lists) can only work if all threads doing work with the flowtable structures (ie, lookup, insert, purge) have to be CPU pinned. Otherwise we may have the situation where: sequentually: * lookup occurs on CPU A; * lookup succeeds on CPU A for some almost-expired entry; * preemption occurs, and it gets scheduled to CPU B; then simultaneously: * CPU A's flowtable purge code runs, and decides to purge entries including the current one; * the code now running on CPU B has an item from the CPU A flowtable, and dereferences it as it's being freed, leading to potential badness. Now, it's a ridiculously small window of opportunity, but I'd rather the code be written to be correct and mostly-fast versus very fast and potentially exploding. I'm sure those in operations would agree. :-) So, my questions: * is this actually how fully pre-emptive kernels _may_ behave? * I believe there's a difference between what 4BSD and ULE will do here - is this the case? What are the scheduler behaviours? * are there any other areas in the kernel that rely on pcpu uma zones / curcpu indexes for things outside of sched_pin() ? Thanks, -a From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 02:37:05 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CA09D9E1; Fri, 14 Feb 2014 02:37:05 +0000 (UTC) Received: from mail-ob0-x22b.google.com (mail-ob0-x22b.google.com [IPv6:2607:f8b0:4003:c01::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 740EC1FF2; Fri, 14 Feb 2014 02:37:05 +0000 (UTC) Received: by mail-ob0-f171.google.com with SMTP id wp4so13291458obc.16 for ; Thu, 13 Feb 2014 18:37:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=MERphlNKfDE1BAVIvYiPemycFmItkkla/PRxwB3dpxM=; b=KiEsJ8C7bnC51BcfqhM8FStxMpvZIZjUYzzLbNXrcoGoQx1L2tuxg63IBCfl9fkbH2 +G6wVB/38tnwx7NKlaFi0q2OMamqPvKLms2SkIg2vtxtGjNZqC0GnLsqniznupmRMI/l AQuQx667IRktJxrSBn+/HWfGTZs5L0HoLPPc/yy1GRAFHV2tYh4gWZMO8rftLZY7MylQ iFTKLcDIf4CwyaJ5fhEq/b5KS6kWZykuudpK1ckJt9tapXiymvxxBbiSB8ItVbqE2X/l m9BY6nGlkOO29PoG7GSgqy+cDk9vA5Nak4Z0jRpnVRJEavjVHj0Fd0x03CMRTotXi13P zc7Q== MIME-Version: 1.0 X-Received: by 10.182.113.195 with SMTP id ja3mr4262074obb.46.1392345424631; Thu, 13 Feb 2014 18:37:04 -0800 (PST) Received: by 10.76.130.196 with HTTP; Thu, 13 Feb 2014 18:37:04 -0800 (PST) In-Reply-To: References: Date: Thu, 13 Feb 2014 21:37:04 -0500 Message-ID: Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? From: Ryan Stone To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 02:37:05 -0000 On Thu, Feb 13, 2014 at 6:57 PM, Adrian Chadd wrote: > sequentually: > > * lookup occurs on CPU A; > * lookup succeeds on CPU A for some almost-expired entry; > * preemption occurs, and it gets scheduled to CPU B; > > then simultaneously: > > * CPU A's flowtable purge code runs, and decides to purge entries > including the current one; > * the code now running on CPU B has an item from the CPU A flowtable, > and dereferences it as it's being freed, leading to potential badness. This kind of scenario is definitely possible. All of the FreeBSD kernel code that deals with lockless per-cpu data structures that I have seen (e.g. uma) has used critical_enter()/critical_exit() to prevent preemption, and have been careful to invalidate their references to the per-cpu data if they have to drop the critical section. I don't believe that sched_pin() is good enough because I don't believe that it handles the scenario when thread A gets a reference and then is preempted, thread B frees the entry, and then A is scheduled and uses the now-freed entry. However I'm really not familiar at all with flowtable so maybe there's something preventing that. From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 03:06:09 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BED17444; Fri, 14 Feb 2014 03:06:09 +0000 (UTC) Received: from mail-qc0-x231.google.com (mail-qc0-x231.google.com [IPv6:2607:f8b0:400d:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6AD0B15A7; Fri, 14 Feb 2014 03:06:09 +0000 (UTC) Received: by mail-qc0-f177.google.com with SMTP id i8so19073631qcq.36 for ; Thu, 13 Feb 2014 19:06:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=+jioV4xyVCq1T00uG72ww/G+bXecll9niyHinUnbWKk=; b=Vj8r3VYM1y7MtT/BiWRwTs6KQJ1zda90xdUJb0isC8K/6gKnYXyY2XWekbn74LIXOv huLHFOt4ntEpHHwFqL6UE4YaEaji8wPlWnaj6bR0sb0obnHmWQEBMKYAUxHZmgiUpzH8 3cr/7oZGkIcEkI4aBdYIKR+YiuBh4Y841GKfslmjBNIcZONFnv9mmfxGsMLh8ry1Ho97 X7OGZg4F3uRIzmgv783PtCGn0QrVmHxA2iQt7ECG4vcXJ2PkcmWJ+iF8RqqcOM9O3Re+ VGqypWDzsDnGcB9p99v5+7gJ82qgnJhbTnQS7JTvd/kPlRSBBB92DtafDn2PS+CMJJ84 37Fw== MIME-Version: 1.0 X-Received: by 10.224.11.136 with SMTP id t8mr8929293qat.26.1392347167800; Thu, 13 Feb 2014 19:06:07 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.16.10 with HTTP; Thu, 13 Feb 2014 19:06:07 -0800 (PST) In-Reply-To: References: Date: Thu, 13 Feb 2014 19:06:07 -0800 X-Google-Sender-Auth: Mev_E25aX-jq1TG6UrCYnr52XZM Message-ID: Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? From: Adrian Chadd To: Ryan Stone Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 03:06:09 -0000 On 13 February 2014 18:37, Ryan Stone wrote: > On Thu, Feb 13, 2014 at 6:57 PM, Adrian Chadd wrote: >> sequentually: >> >> * lookup occurs on CPU A; >> * lookup succeeds on CPU A for some almost-expired entry; >> * preemption occurs, and it gets scheduled to CPU B; >> >> then simultaneously: >> >> * CPU A's flowtable purge code runs, and decides to purge entries >> including the current one; >> * the code now running on CPU B has an item from the CPU A flowtable, >> and dereferences it as it's being freed, leading to potential badness. > > This kind of scenario is definitely possible. All of the FreeBSD > kernel code that deals with lockless per-cpu data structures that I > have seen (e.g. uma) has used critical_enter()/critical_exit() to > prevent preemption, and have been careful to invalidate their > references to the per-cpu data if they have to drop the critical > section. > > I don't believe that sched_pin() is good enough because I don't > believe that it handles the scenario when thread A gets a reference > and then is preempted, thread B frees the entry, and then A is > scheduled and uses the now-freed entry. However I'm really not > familiar at all with flowtable so maybe there's something preventing > that. Oh man, I hate you for bringing that up. So, both flowtable_lookup_common() and flowtable_insert() does do a critical_enter() / critical_exit() during the list operations, so that's ok. It won't be pre-empted. If I wrap flowtable_lookup() with sched_pin() so it covers both the lookup path and the insert path, then it won't get scheduled on another CPU. Then the critical sections around the list operations guarantee it won't end up with a freed entry. The fl entry uptime is updated in the critical section so it shouldn't end up overlapping with the flow purger in a way that has it removed. So, I think that's okay. I think once i add sched_pin() in the right spots, this crap won't use a stale entry. Well, as long as the entry get used before the flow cleaner picks it up. Phew! -adrian From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 09:22:35 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B7D30568; Fri, 14 Feb 2014 09:22:35 +0000 (UTC) Received: from mail-vc0-x230.google.com (mail-vc0-x230.google.com [IPv6:2607:f8b0:400c:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5BE6E1222; Fri, 14 Feb 2014 09:22:35 +0000 (UTC) Received: by mail-vc0-f176.google.com with SMTP id la4so9138450vcb.35 for ; Fri, 14 Feb 2014 01:22:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ivC0a4n2HHy8ApjJC3TkB6Ds8nhM2LrQ8JT0ClCKB40=; b=JzoyMgcYDC5ghKjOx3kfR+8LvG9d8d3fYW+gknqJVQZVCJSgNVqIo38fRgL/t5Dlj+ OZT2ZkQzDeLUEyTsCKB6s/JEM2C+5h1R5Xnatcljkp85Eb4BOcJESE8UPTLUdNi0q9Ds eOQ7y34SPUEnm0h879JmRPrIaKHfjlBLtCjLelph6G8WafHKLoJz0lRbjpabQhzd6Kq/ VdNwiPJVX/jhI8KpUSNDL+bDkXwxI4xoVdcgEFFlOw8V+cIX7gVNNB31iUPuvRBZ7Yhm +QgPnClFP954GalGd3PTBAHHAnEGneq4Kfa+8LcoOI2ivWdK59Z2ONi8N9axSO2o+lyo 0xXw== MIME-Version: 1.0 X-Received: by 10.52.247.231 with SMTP id yh7mr325212vdc.34.1392369754283; Fri, 14 Feb 2014 01:22:34 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.220.113.199 with HTTP; Fri, 14 Feb 2014 01:22:34 -0800 (PST) In-Reply-To: References: Date: Fri, 14 Feb 2014 01:22:34 -0800 X-Google-Sender-Auth: R5K41CJg3nunNfEWN3a5V0YcGOk Message-ID: Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? From: Adrian Chadd To: Ryan Stone Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 09:22:35 -0000 Ok, so now I remember the other odd thing. I was seeing the sending context(s) jumping from one CPU to another during flowtable_insert_common(), around the locking bits. But I thread pinned all the sender user threads! So, why would the senders still be scheduled on other CPUs if I've pinned the userland threads? (and yes, I verified that the userland threads weren't moving around.) -a From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 14:25:25 2014 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3F1C0774; Fri, 14 Feb 2014 14:25:25 +0000 (UTC) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id 00E871C42; Fri, 14 Feb 2014 14:25:23 +0000 (UTC) Received: from localhost (58.wheelsystems.com [83.12.187.58]) by mail.dawidek.net (Postfix) with ESMTPSA id E4DF1C61; Fri, 14 Feb 2014 15:25:16 +0100 (CET) Date: Fri, 14 Feb 2014 15:26:52 +0100 From: Pawel Jakub Dawidek To: freebsd-security@FreeBSD.org, arch@FreeBSD.org Subject: Re: CFR: unifing sha256 userland/kernel implementation... Message-ID: <20140214142652.GA1661@garage.freebsd.pl> References: <20140211185639.GK34851@funkthat.com> <20140212003907.GM34851@funkthat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="huq684BweRXVnRxX" Content-Disposition: inline In-Reply-To: <20140212003907.GM34851@funkthat.com> X-OS: FreeBSD 11.0-CURRENT amd64 User-Agent: Mutt/1.5.22 (2013-10-16) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 14:25:25 -0000 --huq684BweRXVnRxX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Feb 11, 2014 at 04:39:07PM -0800, John-Mark Gurney wrote: > John-Mark Gurney wrote this message on Tue, Feb 11, 2014 at 10:56 -0800: > > I did some performance testing on sha256, and found that the libmd > > version is significantly faster, ~20%, than the kernel version. Even > > if you enable SHA2_UNROLL_TRANSFORM (which isn't the default), the > > version in libmd is still faster. > >=20 > > So, this patch moves libmd's sha256c.c and sha256.h into the kernel, > > and adapts the userland to pull the version from the kernel. This > > change removes sha256 from the existing sha2.c file, and does some > > minor cleanup of types in sha2. > >=20 > > I have tested this w/ ZFS using sha256 checksums, and a ZFS made > > pre-patch is read fine by a kernel post patch. I have also run > > the tests in lib/libmd and they all pass fine. Passes > > buildworld/buildkernel/installkernel/reboot/installworld/reboot/test. > >=20 > > Patch: > > https://www.funkthat.com/~jmg/sha256.kern.patch > >=20 > > Following stats are in seconds to digest 100000 10000-byte blocks, > > calculated using sha256 -t: > > $ ministat soft.times kernsoft.times=20 > > x soft.times > > + kernsoft.times > > +----------------------------------------------------------------------= --------+ > > |x xx xx +++ + = +| > > | |___________AM_________| |_______M_____A______________| = | > > +----------------------------------------------------------------------= --------+ > > N Min Max Median Avg St= ddev > > x 5 6.775387 8.279581 7.848128 7.792094 0.6091= 2664 > > + 5 8.997429 10.768921 9.090787 9.4359144 0.7504= 0822 > > Difference at 95.0% confidence > > 1.64382 +/- 0.99674 > > 21.096% +/- 12.7917% > > (Student's t, pooled s =3D 0.683428) > >=20 > > This is in preperation of bringing in an SSE4 accelerated version of > > sha256 (for both userland and kernel) that sees a 2x performance > > increase. I can't wait:) --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://mobter.com --huq684BweRXVnRxX Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlL+J6wACgkQForvXbEpPzQndgCglshTIuytaOOPgOPHPoGBE9D5 kHcAoNRC15/8Gk2aUD+6AtD7akEr/8ng =IhpC -----END PGP SIGNATURE----- --huq684BweRXVnRxX-- From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 17:51:34 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8D04A11B; Fri, 14 Feb 2014 17:51:34 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5D8021453; Fri, 14 Feb 2014 17:51:34 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C96A7B917; Fri, 14 Feb 2014 12:51:30 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? Date: Fri, 14 Feb 2014 11:39:48 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201402141139.49158.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 14 Feb 2014 12:51:30 -0500 (EST) Cc: "freebsd-hackers@freebsd.org" , Adrian Chadd , Ryan Stone X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 17:51:34 -0000 On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: > Ok, so now I remember the other odd thing. > > I was seeing the sending context(s) jumping from one CPU to another > during flowtable_insert_common(), around the locking bits. > > But I thread pinned all the sender user threads! > > So, why would the senders still be scheduled on other CPUs if I've > pinned the userland threads? > > (and yes, I verified that the userland threads weren't moving around.) Can you clarify a bit? It's not clear how sender thraeds differ from userland threads differ from sender user threads. (I.e. one reading is that these are all the same thing and should thus all be pinned (I assume you mean using cpuset to bind them to specific cores rather than sched_pin)) -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 17:55:05 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3B8135BC; Fri, 14 Feb 2014 17:55:05 +0000 (UTC) Received: from mail-qc0-x229.google.com (mail-qc0-x229.google.com [IPv6:2607:f8b0:400d:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C63C714B0; Fri, 14 Feb 2014 17:55:04 +0000 (UTC) Received: by mail-qc0-f169.google.com with SMTP id w7so20684898qcr.28 for ; Fri, 14 Feb 2014 09:55:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=bjmYs8IWxXTAY/Adfo+BHnmFW552Egd/N3gmaf2V0MI=; b=NykmRvtWx1snhjJvcyRxii/54BtRZKYfkDCyrvo9i6vqO/alk1+yy3NutonXZWiRnl GgCFuBJhRFMmQnoC0D3rcgH/Xp7Nt3/G7CSNxyxcmf1zyjw1ftQBlnRtgNmg2fb3mRHB /WA/4j+vmT3A49WSrV3UeLzD+WY71cQKZxgUnqRrEoWRzq7aUGGomuNvdA2EWlSTS3c1 9V9+GgpuYdQNQq9EuY0b00PY/S3A7PrWHonmrzQD2dk78S++9Sq6dFkhpJI5jP5AKyhc HrVe66T3Ezj4iC9nUK4zDIBQaIb1GFsxwBtmuZpiolMpPX5F0mlGeWMsLE6VJyokWrO+ 3qFg== MIME-Version: 1.0 X-Received: by 10.224.11.136 with SMTP id t8mr15755403qat.26.1392400503873; Fri, 14 Feb 2014 09:55:03 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.16.10 with HTTP; Fri, 14 Feb 2014 09:55:03 -0800 (PST) In-Reply-To: <201402141139.49158.jhb@freebsd.org> References: <201402141139.49158.jhb@freebsd.org> Date: Fri, 14 Feb 2014 09:55:03 -0800 X-Google-Sender-Auth: CZZ-aGEmfEN_lLThkm6dptv_s7U Message-ID: Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 17:55:05 -0000 On 14 February 2014 08:39, John Baldwin wrote: > On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: >> Ok, so now I remember the other odd thing. >> >> I was seeing the sending context(s) jumping from one CPU to another >> during flowtable_insert_common(), around the locking bits. >> >> But I thread pinned all the sender user threads! >> >> So, why would the senders still be scheduled on other CPUs if I've >> pinned the userland threads? >> >> (and yes, I verified that the userland threads weren't moving around.) > > Can you clarify a bit? It's not clear how sender thraeds differ from > userland threads differ from sender user threads. (I.e. one reading > is that these are all the same thing and should thus all be pinned > (I assume you mean using cpuset to bind them to specific cores rather > than sched_pin)) Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: * the userland threads are using the cpuset call to map a thread into a cpuset, yes * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as the userland threads -a From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 18:18:54 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C9364E11; Fri, 14 Feb 2014 18:18:54 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 96C4416EE; Fri, 14 Feb 2014 18:18:54 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 92A71B9DF; Fri, 14 Feb 2014 13:18:53 -0500 (EST) From: John Baldwin To: Adrian Chadd Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? Date: Fri, 14 Feb 2014 13:18:44 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <201402141139.49158.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201402141318.44743.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 14 Feb 2014 13:18:53 -0500 (EST) Cc: "freebsd-hackers@freebsd.org" , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 18:18:54 -0000 On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote: > On 14 February 2014 08:39, John Baldwin wrote: > > On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: > >> Ok, so now I remember the other odd thing. > >> > >> I was seeing the sending context(s) jumping from one CPU to another > >> during flowtable_insert_common(), around the locking bits. > >> > >> But I thread pinned all the sender user threads! > >> > >> So, why would the senders still be scheduled on other CPUs if I've > >> pinned the userland threads? > >> > >> (and yes, I verified that the userland threads weren't moving around.) > > > > Can you clarify a bit? It's not clear how sender thraeds differ from > > userland threads differ from sender user threads. (I.e. one reading > > is that these are all the same thing and should thus all be pinned > > (I assume you mean using cpuset to bind them to specific cores rather > > than sched_pin)) > > Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: > > * the userland threads are using the cpuset call to map a thread into > a cpuset, yes > * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as > the userland threads If they are all cpuset to a single CPU, they should not migrate, though I think sched_bind() can override that. However, that requires code to explicitly call sched_bind() which should be rare. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 18:33:38 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2A87D5FB for ; Fri, 14 Feb 2014 18:33:38 +0000 (UTC) Received: from mail-la0-f46.google.com (mail-la0-f46.google.com [209.85.215.46]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 98C6F187F for ; Fri, 14 Feb 2014 18:33:37 +0000 (UTC) Received: by mail-la0-f46.google.com with SMTP id b8so9504066lan.19 for ; Fri, 14 Feb 2014 10:33:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=lxweHO6XgFZ880npI09YRyplbMA5yctEUxQ5ZIF0V+M=; b=iBvuTeLK80askuGh1QQfon903Uge3nn+5LzAtn60cRqdp56TQI6cUz6l/dwBBEjai8 B7vtec9UJi6m/B+I2YK6okCZ9er5UlAkJY6kb3d2mkSn5WANPPGN/lubTsDloPCC7kBq ML4v9cRdKoq80MlBrtwe5kOdBCH0o6aFBw2eP0prbfr58XZLgvCbDKaj+1Fm0EiJI8vi ivJsCSKv8QZNfmQzk3PoOZNoU0qkxxra4zJciMyUyDa0beXAiBM5G2Jn/NCViNoesjg0 HMS//ryQhN7rk92kJobVU3vX2h3xVVeqMMYAGrU/uxiKakupchCl1zGYmGbXua8GI1PE 2/8A== X-Gm-Message-State: ALoCoQnijjNWMIH6b2GQ5ZxSSwUaBWlUM3zj05+TWr8Yp/b+mVOesvcRxpeTS52Onv9Vi322rrMc X-Received: by 10.112.160.161 with SMTP id xl1mr10184lbb.71.1392402368412; Fri, 14 Feb 2014 10:26:08 -0800 (PST) Received: from [192.168.1.2] ([89.169.173.68]) by mx.google.com with ESMTPSA id cu8sm6711323lbb.12.2014.02.14.10.26.07 for (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 14 Feb 2014 10:26:07 -0800 (PST) Message-ID: <52FE5FBF.3090104@freebsd.org> Date: Fri, 14 Feb 2014 22:26:07 +0400 From: Andrey Chernov User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: John Baldwin , Adrian Chadd Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? References: <201402141139.49158.jhb@freebsd.org> <201402141318.44743.jhb@freebsd.org> In-Reply-To: <201402141318.44743.jhb@freebsd.org> X-Enigmail-Version: 1.7a1pre Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 18:33:38 -0000 On 14.02.2014 22:18, John Baldwin wrote: > On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote: >> On 14 February 2014 08:39, John Baldwin wrote: >>> On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: >>>> Ok, so now I remember the other odd thing. >>>> >>>> I was seeing the sending context(s) jumping from one CPU to another >>>> during flowtable_insert_common(), around the locking bits. >>>> >>>> But I thread pinned all the sender user threads! >>>> >>>> So, why would the senders still be scheduled on other CPUs if I've >>>> pinned the userland threads? >>>> >>>> (and yes, I verified that the userland threads weren't moving around.) >>> >>> Can you clarify a bit? It's not clear how sender thraeds differ from >>> userland threads differ from sender user threads. (I.e. one reading >>> is that these are all the same thing and should thus all be pinned >>> (I assume you mean using cpuset to bind them to specific cores rather >>> than sched_pin)) >> >> Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: >> >> * the userland threads are using the cpuset call to map a thread into >> a cpuset, yes >> * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as >> the userland threads > > If they are all cpuset to a single CPU, they should not migrate, though > I think sched_bind() can override that. However, that requires code to > explicitly call sched_bind() which should be rare. > Due to this bug, not fixed yet, the real picture is more complex: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 -- http://ache.vniz.net/ From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 19:10:44 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 700EE2D6; Fri, 14 Feb 2014 19:10:44 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 434551B9F; Fri, 14 Feb 2014 19:10:44 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 990C7B9A0; Fri, 14 Feb 2014 14:10:40 -0500 (EST) From: John Baldwin To: Andrey Chernov Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? Date: Fri, 14 Feb 2014 14:10:29 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> In-Reply-To: <52FE5FBF.3090104@freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="koi8-r" Content-Transfer-Encoding: 7bit Message-Id: <201402141410.29325.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 14 Feb 2014 14:10:40 -0500 (EST) Cc: "freebsd-hackers@freebsd.org" , Adrian Chadd , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 19:10:44 -0000 On Friday, February 14, 2014 1:26:07 pm Andrey Chernov wrote: > On 14.02.2014 22:18, John Baldwin wrote: > > On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote: > >> On 14 February 2014 08:39, John Baldwin wrote: > >>> On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: > >>>> Ok, so now I remember the other odd thing. > >>>> > >>>> I was seeing the sending context(s) jumping from one CPU to another > >>>> during flowtable_insert_common(), around the locking bits. > >>>> > >>>> But I thread pinned all the sender user threads! > >>>> > >>>> So, why would the senders still be scheduled on other CPUs if I've > >>>> pinned the userland threads? > >>>> > >>>> (and yes, I verified that the userland threads weren't moving around.) > >>> > >>> Can you clarify a bit? It's not clear how sender thraeds differ from > >>> userland threads differ from sender user threads. (I.e. one reading > >>> is that these are all the same thing and should thus all be pinned > >>> (I assume you mean using cpuset to bind them to specific cores rather > >>> than sched_pin)) > >> > >> Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: > >> > >> * the userland threads are using the cpuset call to map a thread into > >> a cpuset, yes > >> * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as > >> the userland threads > > > > If they are all cpuset to a single CPU, they should not migrate, though > > I think sched_bind() can override that. However, that requires code to > > explicitly call sched_bind() which should be rare. > > > > Due to this bug, not fixed yet, the real picture is more complex: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 Eh, that bug report has no useful details, as in, it doesn't list the actual commands run. If you do 'cpuset -l 6 -s 1' to force all processes to only use CPU6, then yes, of course the other CPUs are idle because that's what you _asked_ for. AFAICT, that is all the original reporter did. At work we regularly add and remove CPUs from the default set (set 1) on hundreds of machines every day with ULE without any issues. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 19:18:31 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5809B5DE; Fri, 14 Feb 2014 19:18:31 +0000 (UTC) Received: from mail-qc0-x235.google.com (mail-qc0-x235.google.com [IPv6:2607:f8b0:400d:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E3E271BF4; Fri, 14 Feb 2014 19:18:30 +0000 (UTC) Received: by mail-qc0-f181.google.com with SMTP id e9so20437027qcy.26 for ; Fri, 14 Feb 2014 11:18:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=jsS5cQgxKWF0vUwePrX33qb3N38GlytJ045P/Pc/MyM=; b=ZrVoWZAu+Zzsd450YJPyL8vJShIYYS3yNeIHfPHDBcFk+GNQ89Es9lG5k2t1ek0+I3 un/88l3ydkG2Z+Z6gMq4mvzed9ZFgCyRGD/OoWg0/9Rwtz4JlWv6/3DH7id9CwEqtqwp M/wGlljgaBCERAfGWLGIKcHqBx4i/4rYBDpwrd9xigN6gHJDX5Fp5GHJGGd1xKz3opdP iCLuTwXFlQ/DBDrVswPcUWoc2qZmkmBHYazuRydLkgM94V43d0b5wYyeki/y8qTiKhfW sD3XefnghiAgVa+kvAAu4OcH34m0k12NIyKqhdQ627x/1AxWJIX77S1iySretoCiYX2c vWhw== MIME-Version: 1.0 X-Received: by 10.229.179.5 with SMTP id bo5mr16190937qcb.21.1392405510104; Fri, 14 Feb 2014 11:18:30 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.16.10 with HTTP; Fri, 14 Feb 2014 11:18:30 -0800 (PST) In-Reply-To: <201402141318.44743.jhb@freebsd.org> References: <201402141139.49158.jhb@freebsd.org> <201402141318.44743.jhb@freebsd.org> Date: Fri, 14 Feb 2014 11:18:30 -0800 X-Google-Sender-Auth: QEybOPM7Tbhp6qOcVV4ns4tncyg Message-ID: Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 19:18:31 -0000 On 14 February 2014 10:18, John Baldwin wrote: > If they are all cpuset to a single CPU, they should not migrate, though > I think sched_bind() can override that. However, that requires code to > explicitly call sched_bind() which should be rare. Yup. That's why I'm confused. I'm rebuilding -HEAD now with the latest flowtable changes. I'll add in my debugging afterward and trigger the particular scenario where it's behaving badly and do some more diagnostics. Thanks, -a From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 22:08:47 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 65C42DFC for ; Fri, 14 Feb 2014 22:08:47 +0000 (UTC) Received: from mail-ee0-f47.google.com (mail-ee0-f47.google.com [74.125.83.47]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E81331BB7 for ; Fri, 14 Feb 2014 22:08:46 +0000 (UTC) Received: by mail-ee0-f47.google.com with SMTP id d49so5900750eek.20 for ; Fri, 14 Feb 2014 14:08:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=D6/BAheko3zijfo+zJbs/3UvzngaykW0h1HWOqOH22g=; b=MiWULXupVGt0owzmVT9EGjmBjF5DPcCXs4G22zNmNa2QOV/6g3VttCjrE+2pJgXgWv 61TtZm8gA6R0d3+Hnmb0lByNCddHWuNCBg2x1UxCw7XkcfN7ajfMXCXrqowtSInKDFoZ y1dR6Rp1O6CqRSE62rXN6Iq2o6FnlGzCHHtdQvZSariZMGKOTz+TqK9eo1QP6NRaZL5q B5+Na7G+lXuOwSIPkGnYrFw9Ra+xeRRBLi4DO6I3j8HRYeYFwJJ9aRE4V9ic1UGrCx/z QkZv/8+9yRlLCZ4FuGtwtk9UXoefgzdto8fBlwRvMx0e5babNdW3NcyNxsmyD79EIdbA UMQw== X-Gm-Message-State: ALoCoQmpgcY9qB/SLc4HKUYnvNa40cyw6yF9ONXq39Co6C/EsuZ6aQF/8K6KcO4ehjLN6TH53H6N X-Received: by 10.14.203.197 with SMTP id f45mr2188243eeo.90.1392415719153; Fri, 14 Feb 2014 14:08:39 -0800 (PST) Received: from [192.168.1.2] ([89.169.173.68]) by mx.google.com with ESMTPSA id y47sm24635115eel.14.2014.02.14.14.08.38 for (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 14 Feb 2014 14:08:38 -0800 (PST) Message-ID: <52FE93E6.6030705@freebsd.org> Date: Sat, 15 Feb 2014 02:08:38 +0400 From: Andrey Chernov User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: John Baldwin Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> In-Reply-To: <201402141410.29325.jhb@freebsd.org> X-Enigmail-Version: 1.7a1pre Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Adrian Chadd , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 22:08:47 -0000 On 14.02.2014 23:10, John Baldwin wrote: >> Due to this bug, not fixed yet, the real picture is more complex: >> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 > > Eh, that bug report has no useful details, as in, it doesn't list the > actual commands run. If you do 'cpuset -l 6 -s 1' to force all > processes to only use CPU6, then yes, of course the other CPUs are idle > because that's what you _asked_ for. AFAICT, that is all the original > reporter did. At work we regularly add and remove CPUs from the > default set (set 1) on hundreds of machines every day with ULE without > any issues. Probably original report lack certain commands, but I provide the link to the port which reproduces this bug too. All threads there are assigned to the _different_ CPUs and appears as result on single one with SCHED_ULE (not with SCHED_4BSD). And it is what original reporter mean too. It surely happens, maybe not the first time, but on 2nd-3rd. It means that cpuset_setaffinity() is completely broken form SCHED_ULE at least for 3 years. -- http://ache.vniz.net/ From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 22:57:21 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 472FFBC9; Fri, 14 Feb 2014 22:57:21 +0000 (UTC) Received: from mail-qc0-x229.google.com (mail-qc0-x229.google.com [IPv6:2607:f8b0:400d:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CF900105A; Fri, 14 Feb 2014 22:57:20 +0000 (UTC) Received: by mail-qc0-f169.google.com with SMTP id w7so21352854qcr.0 for ; Fri, 14 Feb 2014 14:57:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=dyaQZ1O5yEnbLCb25tmt99o5o9IQoLAU+LgPmywsIvQ=; b=BUmkYv3u0Q9Xt+0CwwVZ1zjlj7/Z5XHysaSZabgZ4ieowgLRp2MlZWasAQ+0t8qGmJ +gKztJK/q3xEUaZieEQubZ43El//eQCcOgIR+ho+24CMWQgaxSpIgnN8v4fGBjWexrKR mZDsutgmTcp3vpyAJ9+/oSben9byJgSVJr7v/VLZG+xkVdunxtfSqWfoQ77mUV8IzNfz 67g4stqyIrqyv6PXnl6qFcfcr5VVVecby9FfbAXUEsL0I2wEd8GJiEUMQAwCc1FZhRk9 ZFjobHyNK7jE2o6z79juyAGQSIt2NqHJ+V/Ki0XIJUdD+2BKva2Gmz3Qp4HzOUGUv3/U 3IoQ== MIME-Version: 1.0 X-Received: by 10.224.61.2 with SMTP id r2mr18094159qah.49.1392418640030; Fri, 14 Feb 2014 14:57:20 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.16.10 with HTTP; Fri, 14 Feb 2014 14:57:19 -0800 (PST) In-Reply-To: References: <201402141139.49158.jhb@freebsd.org> <201402141318.44743.jhb@freebsd.org> Date: Fri, 14 Feb 2014 14:57:19 -0800 X-Google-Sender-Auth: vlg3IRlPpAsLpOvGMj4V9O8uvdc Message-ID: Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 22:57:21 -0000 [snip] So it turns out that the threads somehow migrating between CPUs during flowtable_lookup_common() is the clock swi(s), which I'm guessing are driving the per-CPU TCP callwheel timeouts. It turns out the per-CPU clock swis aren't CPU-pinned, so they can be preempted and migrated. I'm not sure if this is correct behaviour. I'll experiment with pinning these to their base CPU and see if this causes issues. Thanks, -a From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 23:02:58 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04E6EDA9 for ; Fri, 14 Feb 2014 23:02:58 +0000 (UTC) Received: from mail-ea0-f180.google.com (mail-ea0-f180.google.com [209.85.215.180]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 83BA710FD for ; Fri, 14 Feb 2014 23:02:57 +0000 (UTC) Received: by mail-ea0-f180.google.com with SMTP id o10so6065782eaj.11 for ; Fri, 14 Feb 2014 15:02:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=x8xxOQBDIs7K3LPirTh/iZf1+TrjzpM+a/QZiY5ZdHk=; b=BOK55qZodWTkRw8l/s/LOZjplpTszAwxnJCLH4J7ikwwR0+15jtAh28lC/SFP2S7P3 P3LsyJYApS9XH4hMILHNy7NJ7yM1suLEfUuf6dnFMGsaWKxJQiuCtKXtmkbAgAmQSOQR BgGj+rSxiypkQ/emRIFfCataz9K8O1HkSugFQAbsb9shK50MWCQXrUk9U20WbmLv/X8f 1g8lP+CR7lZLtjBiK1PHRGh/qNkdLthADJRQRqqlXtXUG86NreAMuMaDWOvzwn/kLwiB EEihvBctBHAXPb4W8iQSEMGFeQgGqOcJ1Oel7H9uwtCtSRV/q1mJHzEWwtTaQPSJgCk6 PAUg== X-Gm-Message-State: ALoCoQlB04hEovpnv78ebIRwHrRMRKyojoo+jA8p4hrvpGROXlQL0Fn6yAHcVGPv2Zu9xuq518Kx X-Received: by 10.14.87.129 with SMTP id y1mr12051788eee.38.1392417375195; Fri, 14 Feb 2014 14:36:15 -0800 (PST) Received: from [192.168.1.2] ([89.169.173.68]) by mx.google.com with ESMTPSA id 46sm24812320ees.4.2014.02.14.14.36.14 for (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 14 Feb 2014 14:36:14 -0800 (PST) Message-ID: <52FE9A5E.5050300@freebsd.org> Date: Sat, 15 Feb 2014 02:36:14 +0400 From: Andrey Chernov User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: John Baldwin Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> In-Reply-To: <52FE93E6.6030705@freebsd.org> X-Enigmail-Version: 1.7a1pre Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Adrian Chadd , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 23:02:58 -0000 On 15.02.2014 2:08, Andrey Chernov wrote: > On 14.02.2014 23:10, John Baldwin wrote: > >>> Due to this bug, not fixed yet, the real picture is more complex: >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 >> >> Eh, that bug report has no useful details, as in, it doesn't list the >> actual commands run. If you do 'cpuset -l 6 -s 1' to force all >> processes to only use CPU6, then yes, of course the other CPUs are idle >> because that's what you _asked_ for. AFAICT, that is all the original >> reporter did. At work we regularly add and remove CPUs from the >> default set (set 1) on hundreds of machines every day with ULE without >> any issues. > > Probably original report lack certain commands, but I provide the link > to the port which reproduces this bug too. All threads there are > assigned to the _different_ CPUs and appears as result on single one > with SCHED_ULE (not with SCHED_4BSD). And it is what original reporter > mean too. It surely happens, maybe not the first time, but on 2nd-3rd. > It means that cpuset_setaffinity() is completely broken form SCHED_ULE > at least for 3 years. > This is code example from cpuminer port, in case you are interested, it is very simple: static inline void affine_to_cpu(int id, int cpu) { cpuset_t set; CPU_ZERO(&set); CPU_SET(cpu, &set); cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); } ... num_processors = sysconf(_SC_NPROCESSORS_CONF); if (!opt_n_threads) opt_n_threads = num_processors; ... In the thread itself: struct thr_info *mythr = userdata; int thr_id = mythr->id; if (num_processors > 1 && opt_n_threads % num_processors == 0) { if (!opt_quiet) applog(LOG_INFO, "Binding thread %d to cpu %d", thr_id, thr_id % num_processors); affine_to_cpu(thr_id, thr_id % num_processors); } -- http://ache.vniz.net/ From owner-freebsd-arch@FreeBSD.ORG Sat Feb 15 00:00:06 2014 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EC25564A; Sat, 15 Feb 2014 00:00:06 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 83E51149D; Sat, 15 Feb 2014 00:00:04 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id CAA20878; Sat, 15 Feb 2014 02:00:02 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1WESfq-000OIM-70; Sat, 15 Feb 2014 02:00:02 +0200 Message-ID: <52FEADC9.2040608@FreeBSD.org> Date: Sat, 15 Feb 2014 01:59:05 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Andrey Chernov , John Baldwin Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> <52FE9A5E.5050300@freebsd.org> In-Reply-To: <52FE9A5E.5050300@freebsd.org> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Feb 2014 00:00:07 -0000 on 15/02/2014 00:36 Andrey Chernov said the following: > On 15.02.2014 2:08, Andrey Chernov wrote: >> On 14.02.2014 23:10, John Baldwin wrote: >> >>>> Due to this bug, not fixed yet, the real picture is more complex: >>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 >>> >>> Eh, that bug report has no useful details, as in, it doesn't list the >>> actual commands run. If you do 'cpuset -l 6 -s 1' to force all >>> processes to only use CPU6, then yes, of course the other CPUs are idle >>> because that's what you _asked_ for. AFAICT, that is all the original >>> reporter did. At work we regularly add and remove CPUs from the >>> default set (set 1) on hundreds of machines every day with ULE without >>> any issues. >> >> Probably original report lack certain commands, but I provide the link >> to the port which reproduces this bug too. All threads there are >> assigned to the _different_ CPUs and appears as result on single one >> with SCHED_ULE (not with SCHED_4BSD). And it is what original reporter >> mean too. It surely happens, maybe not the first time, but on 2nd-3rd. >> It means that cpuset_setaffinity() is completely broken form SCHED_ULE >> at least for 3 years. >> > > This is code example from cpuminer port, in case you are interested, it is very simple: > > static inline void affine_to_cpu(int id, int cpu) > { > cpuset_t set; > CPU_ZERO(&set); > CPU_SET(cpu, &set); > cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); I think that CPU_WHICH_TID should have been used here. > } > ... > num_processors = sysconf(_SC_NPROCESSORS_CONF); > if (!opt_n_threads) > opt_n_threads = num_processors; > ... > In the thread itself: > struct thr_info *mythr = userdata; > int thr_id = mythr->id; > > if (num_processors > 1 && opt_n_threads % num_processors == 0) { > if (!opt_quiet) > applog(LOG_INFO, "Binding thread %d to cpu %d", > thr_id, thr_id % num_processors); > affine_to_cpu(thr_id, thr_id % num_processors); > } > -- Andriy Gapon From owner-freebsd-arch@FreeBSD.ORG Sat Feb 15 00:11:02 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E1488884; Sat, 15 Feb 2014 00:11:01 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B33B91566; Sat, 15 Feb 2014 00:11:01 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s1F0B0hl017264 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 14 Feb 2014 16:11:00 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s1F0B0ZW017263; Fri, 14 Feb 2014 16:11:00 -0800 (PST) (envelope-from jmg) Date: Fri, 14 Feb 2014 16:11:00 -0800 From: John-Mark Gurney To: Andriy Gapon Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? Message-ID: <20140215001100.GS34851@funkthat.com> Mail-Followup-To: Andriy Gapon , Andrey Chernov , John Baldwin , "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> <52FE9A5E.5050300@freebsd.org> <52FEADC9.2040608@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <52FEADC9.2040608@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Fri, 14 Feb 2014 16:11:00 -0800 (PST) Cc: "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Feb 2014 00:11:02 -0000 Andriy Gapon wrote this message on Sat, Feb 15, 2014 at 01:59 +0200: > on 15/02/2014 00:36 Andrey Chernov said the following: > > On 15.02.2014 2:08, Andrey Chernov wrote: > >> On 14.02.2014 23:10, John Baldwin wrote: > >> > >>>> Due to this bug, not fixed yet, the real picture is more complex: > >>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 > >>> > >>> Eh, that bug report has no useful details, as in, it doesn't list the > >>> actual commands run. If you do 'cpuset -l 6 -s 1' to force all > >>> processes to only use CPU6, then yes, of course the other CPUs are idle > >>> because that's what you _asked_ for. AFAICT, that is all the original > >>> reporter did. At work we regularly add and remove CPUs from the > >>> default set (set 1) on hundreds of machines every day with ULE without > >>> any issues. > >> > >> Probably original report lack certain commands, but I provide the link > >> to the port which reproduces this bug too. All threads there are > >> assigned to the _different_ CPUs and appears as result on single one > >> with SCHED_ULE (not with SCHED_4BSD). And it is what original reporter > >> mean too. It surely happens, maybe not the first time, but on 2nd-3rd. > >> It means that cpuset_setaffinity() is completely broken form SCHED_ULE > >> at least for 3 years. > >> > > > > This is code example from cpuminer port, in case you are interested, it is very simple: > > > > static inline void affine_to_cpu(int id, int cpu) > > { > > cpuset_t set; > > CPU_ZERO(&set); > > CPU_SET(cpu, &set); > > cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); > > I think that CPU_WHICH_TID should have been used here. I agree... cpuset(2): The which argument determines how the value of id is interpreted and is of type cpuwhich_t. The which argument may have the following values: CPU_WHICH_TID id is lwpid_t (thread id) CPU_WHICH_PID id is pid_t (process id) CPU_WHICH_CPUSET id is a cpusetid_t (cpuset id) CPU_WHICH_IRQ id is an irq number An id of '-1' may be used with a which of CPU_WHICH_TID, CPU_WHICH_PID, or CPU_WHICH_CPUSET to mean the current thread, process, or current thread's cpuset. All cpuset syscalls allow this usage. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Sat Feb 15 01:59:28 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6FD493B3 for ; Sat, 15 Feb 2014 01:59:28 +0000 (UTC) Received: from mail-la0-f46.google.com (mail-la0-f46.google.com [209.85.215.46]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E18521B9C for ; Sat, 15 Feb 2014 01:59:27 +0000 (UTC) Received: by mail-la0-f46.google.com with SMTP id b8so9732664lan.5 for ; Fri, 14 Feb 2014 17:59:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=crcNec/eWEFO9B2ZEEMjiB9e7w9iwi4nq9pGBSlNO8A=; b=LeW4yudpXeUHH15cVeIhYLcY6m6g6mEKWtZkszfOJdQMTHA+V9Q0DB8T0Nn8+nN8nn QywwyDarENCrwhXTOKfbAWz7C06AfLHj5cp+q5Xii/iLtzaZnukVpA9GRNqI07cbx2i7 aMWObQQIQ94nDgvogouaMk9TrzMzViIwnxrF1q77n1khhF93qRNDYfm4UcJ0UE1laMps gGqvady029YEWjV673XWMOT1RBwUUy6BNeyn0PlbxGF028hILMEWe5+CPrqsmQb9S44C I5LvKkH1FQaFF/TfkVXU77uNegCeP1GBeRvHRAeOGNplXaUpF+/0HjSWuyXjopxBuUjt Cj9Q== X-Gm-Message-State: ALoCoQmM/o0YcZom6wMmYzrGs23si4UcKqC/hrDxmNxm7G6ZvelF0pODf8N3S+9pWixnSBqa4FPn X-Received: by 10.112.236.3 with SMTP id uq3mr7325815lbc.14.1392429565787; Fri, 14 Feb 2014 17:59:25 -0800 (PST) Received: from [192.168.1.2] ([89.169.173.68]) by mx.google.com with ESMTPSA id gi5sm7913379lbc.4.2014.02.14.17.59.24 for (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 14 Feb 2014 17:59:25 -0800 (PST) Message-ID: <52FEC9FC.20707@freebsd.org> Date: Sat, 15 Feb 2014 05:59:24 +0400 From: Andrey Chernov User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Andriy Gapon , John Baldwin Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> <52FE9A5E.5050300@freebsd.org> <52FEADC9.2040608@FreeBSD.org> In-Reply-To: <52FEADC9.2040608@FreeBSD.org> X-Enigmail-Version: 1.7a1pre Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Feb 2014 01:59:28 -0000 On 15.02.2014 3:59, Andriy Gapon wrote: >> This is code example from cpuminer port, in case you are interested, it is very simple: >> >> static inline void affine_to_cpu(int id, int cpu) >> { >> cpuset_t set; >> CPU_ZERO(&set); >> CPU_SET(cpu, &set); >> cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); > > I think that CPU_WHICH_TID should have been used here. > You are right, thanx! -- http://ache.vniz.net/ From owner-freebsd-arch@FreeBSD.ORG Sat Feb 15 02:30:42 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DDCEE824 for ; Sat, 15 Feb 2014 02:30:41 +0000 (UTC) Received: from mail-lb0-f180.google.com (mail-lb0-f180.google.com [209.85.217.180]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5AE961DAB for ; Sat, 15 Feb 2014 02:30:40 +0000 (UTC) Received: by mail-lb0-f180.google.com with SMTP id n15so9995226lbi.11 for ; Fri, 14 Feb 2014 18:30:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=mxuMC739luJjPFi9wwM5jy8RiUDPXezylsHk+FHZ3rI=; b=OPNrK/qVONOyTMEaE7lu6Nq0bNSiS7BLLdtwtMJHzyr+eaN5yvHm4ewVLUaW5ABLbe xHqqegkeH6Ol3bsGIJniNbNNMF0/gMPcZjrxGttkM+ML3lF2kQcWLDXe/x8gHbeVddc8 k/LaIs+/UEW7PRALHpeN5pFlPBk+SyGSIB7fN3GCfqp/gPkIlW6OdYHsJgVGEQqKJRlf Ofcmacktl7eS7ZNaLQeZz+OyOd65b4tI+foieNv1Zgn2ILv/nL1nnOXe1tms3a8zAPNg xXSeXYmNMUiPP1hFPlmPSFbOOgomOjIpYeOekVl9dn4d76v7tTURk+LiidSYzZYpPDpq FDFQ== X-Gm-Message-State: ALoCoQnhwXltC2ipzWZfagF0hbcuS+yKeMu1QnPZ2QlHSpJnGb3rCsJi3ai/nYQh8gR1JEiumuUK X-Received: by 10.152.201.197 with SMTP id kc5mr14074lac.77.1392431439118; Fri, 14 Feb 2014 18:30:39 -0800 (PST) Received: from [192.168.1.2] ([89.169.173.68]) by mx.google.com with ESMTPSA id w2sm11141662lad.4.2014.02.14.18.30.38 for (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 14 Feb 2014 18:30:38 -0800 (PST) Message-ID: <52FED14E.50304@freebsd.org> Date: Sat, 15 Feb 2014 06:30:38 +0400 From: Andrey Chernov User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Andriy Gapon , John Baldwin , "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> <52FE9A5E.5050300@freebsd.org> <52FEADC9.2040608@FreeBSD.org> <20140215001100.GS34851@funkthat.com> In-Reply-To: <20140215001100.GS34851@funkthat.com> X-Enigmail-Version: 1.7a1pre Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Feb 2014 02:30:42 -0000 On 15.02.2014 4:11, John-Mark Gurney wrote: >>> This is code example from cpuminer port, in case you are interested, it is very simple: >>> >>> static inline void affine_to_cpu(int id, int cpu) >>> { >>> cpuset_t set; >>> CPU_ZERO(&set); >>> CPU_SET(cpu, &set); >>> cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); >> >> I think that CPU_WHICH_TID should have been used here. > > I agree... cpuset(2): > The which argument determines how the value of id is interpreted and is > of type cpuwhich_t. The which argument may have the following values: > > CPU_WHICH_TID id is lwpid_t (thread id) > CPU_WHICH_PID id is pid_t (process id) > CPU_WHICH_CPUSET id is a cpusetid_t (cpuset id) > CPU_WHICH_IRQ id is an irq number > > An id of '-1' may be used with a which of CPU_WHICH_TID, CPU_WHICH_PID, > or CPU_WHICH_CPUSET to mean the current thread, process, or current > thread's cpuset. All cpuset syscalls allow this usage. The question still remains: why SCHED_ULE and SCHED_4BSD do different things here on CPU_WHICH_CPUSET == -1 (current thread's cpuset)? It looks like SCHED_ULE changes per/process mask while SCHED_4BSD change per/thread mask in that case. -- http://ache.vniz.net/