From owner-freebsd-current@FreeBSD.ORG Tue Feb 24 23:14:56 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C974F106564A for ; Tue, 24 Feb 2009 23:14:56 +0000 (UTC) (envelope-from justin.teller@gmail.com) Received: from mail-gx0-f176.google.com (mail-gx0-f176.google.com [209.85.217.176]) by mx1.freebsd.org (Postfix) with ESMTP id 6F3FA8FC08 for ; Tue, 24 Feb 2009 23:14:56 +0000 (UTC) (envelope-from justin.teller@gmail.com) Received: by gxk24 with SMTP id 24so9729453gxk.19 for ; Tue, 24 Feb 2009 15:14:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:cc:content-type; bh=80Qo/ZhM8+Fabr1+ZUplUxCG+MN+86qeqf0jF20wCdo=; b=jNZb+uCu0lJ6cpQEmqpq8g0+3gZZhJyJp2YjbVrSWLDMFgUCgfJYBm4Bw46Af3PQCP Oisigksl2eKUrDo9LYFez7zWXogpvyC13+7IVM8br46140A/i+INWSSqVc2qCpct4D6Y 8NYTf6HrEIVkNJSjsjtK3/prAskWbLCvAHSCM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; b=s5J6J7+o6X98GNdyGfGcmQ4tssWQpJm2lNp8wTZhSc8dBFIqTqJvg/q5XXA2TGefQ0 BEs69OqEm4nRppUhGf8Jznu6zkF7nE/ZlzFGaIUAHtgWH/EOfVQiiZcJbXvDkaw6loSa 8whHvoKYp3r+/D2Qw3SxsOVAwvBrLR6llPTnA= MIME-Version: 1.0 Received: by 10.150.136.12 with SMTP id j12mr229532ybd.18.1235515571108; Tue, 24 Feb 2009 14:46:11 -0800 (PST) Date: Tue, 24 Feb 2009 14:46:11 -0800 Message-ID: From: Justin Teller To: freebsd-current@freebsd.org Content-Type: multipart/mixed; boundary=000e0cd5d000bafbb20463b1e2f8 Cc: Subject: Possible case of sched_ule never honoring cpu_set affinity? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Feb 2009 23:14:57 -0000 --000e0cd5d000bafbb20463b1e2f8 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit I believe I've found a case where sched_ule can never honor cpu_set affinity, and I've attached a suggestion to fix it -- if I'm way off on how this should be implemented, please let me know! Imagine the case where sched_affinity is called and the thread is on a RUNQ on the wrong CPU -- sched_affinity simply exits, relying on something else in the scheduler to migrate the thread if need be. The next time the thread is chosen to run, it runs on the wrong CPU. Worse, if the thread never goes to sleep or is chosen to be moved by the load balancer, the thread will continue running on the wrong CPU indefinitely. Attached is a suggestion of how to change the scheduler to honor affinity in this case. With the attached diff, sched_ule will allow a thread to run on the wrong CPU for one slice. Then when the thread moves through sched_switch, if it was running on the wrong CPU, it will be migrated to the right CPU. I've written a test where one thread will bind another thread to a particular CPU (using cpusets) then waits until it's running on the right CPU before binding it to a different CPU (and continuing ad nauseam). Without the change, the test will sometimes hang waiting for the second thread to get on the correct CPU -- with the change, it works every time. Thanks! -Justin --000e0cd5d000bafbb20463b1e2f8 Content-Type: application/octet-stream; name="sched_ule.affinity_change.diff" Content-Disposition: attachment; filename="sched_ule.affinity_change.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_frl5gcn80 LS0tIHNjaGVkX3VsZS5jCTIwMDktMDItMDMgMTU6NDc6MTguMDAwMDAwMDAwIC0wODAwCisrKyBz Y2hlZF91bGVfdXBkYXRlZC5jCTIwMDktMDItMjQgMTQ6MjI6NTMuMDAwMDAwMDAwIC0wODAwCkBA IC0xODI2LDYgKzE4MjYsOCBAQAogCQlzcnFmbGFnID0gKGZsYWdzICYgU1dfUFJFRU1QVCkgPwog CQkgICAgU1JRX09VUlNFTEZ8U1JRX1lJRUxESU5HfFNSUV9QUkVFTVBURUQgOgogCQkgICAgU1JR X09VUlNFTEZ8U1JRX1lJRUxESU5HOworCQlpZiAoVEhSRUFEX0NBTl9TQ0hFRCh0ZCwgdHMtPnRz X2NwdSkpCisJCQl0cy0+dHNfY3B1ID0gc2NoZWRfcGlja2NwdSh0ZCwwKTsKIAkJaWYgKHRzLT50 c19jcHUgPT0gY3B1aWQpCiAJCQl0ZHFfcnVucV9hZGQodGRxLCB0ZCwgc3JxZmxhZyk7CiAJCWVs c2UK --000e0cd5d000bafbb20463b1e2f8--