From owner-freebsd-current@FreeBSD.ORG Thu Aug 26 20:03:40 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 080071065695 for ; Thu, 26 Aug 2010 20:03:40 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8F60C8FC12 for ; Thu, 26 Aug 2010 20:03:39 +0000 (UTC) Received: by eyx24 with SMTP id 24so1857390eyx.13 for ; Thu, 26 Aug 2010 13:03:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=/2q83W3kbgFioJeitCefUvryK8AeBuKrnLDN7dndL+E=; b=lkgDCupk8P249aYxRkYRtBf/gPo/SgSjWvKeyXLR/7r+WPpSLYAaWQOECFic/uNOwi NF4vdCU/CpntjBq2iYJs+41gSztpn1x2eA5TPoURGAlW+jMhUfZoFOsjHGrZUH8FqqGg lLOSfsztEhdsb272J9RgWubbpWNLjy6YvKlgg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; b=e8X/lgb3wbeppzhEPvCSdIEvB6rw7KhzVQjaPZ+b4+08jxO0ZHjYERwJ93poWOluEU swo07fXfseCpLxrLP2SmL/qBCKy8hxKJdXa6ZTCmZ+hN9Z0heKwxGCefANF4qwqcWRZK MTwptdiVVO6q8Oe2xy9oWVB66t6JMCD0bdYDM= MIME-Version: 1.0 Received: by 10.213.114.83 with SMTP id d19mr213361ebq.7.1282853018285; Thu, 26 Aug 2010 13:03:38 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.213.20.144 with HTTP; Thu, 26 Aug 2010 13:03:38 -0700 (PDT) Date: Thu, 26 Aug 2010 13:03:38 -0700 X-Google-Sender-Auth: Ywk1QEnRRcKMSnafRa55xCYdCvs Message-ID: From: mdf@FreeBSD.org To: freebsd-current@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Jeff Roberson Subject: sched_pin() bug in SCHED_ULE X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Aug 2010 20:03:40 -0000 Back at the beginning of August I posted about issues with sched_pin() and witness_warn(): http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032553.html After a lot of debugging I think I've basically found the issue. I found this bug on stable/7, but looking at the commit log for CURRENT I don't see any reason the bug doesn't exist there. This analysis is specific to amd64/i386 but the problem is likely to exist on most arches. The basic problem is that in both tdq_move() and sched_setcpu() can manipulate the ts->ts_cpu variable of another process, which may be running. This means that a process running on CPU N may be set to run on CPU M the next context switch. Then, that process may call sched_pin(), then grab a PCPU variable. An IPI_PREEMPT can come in, causing the thread to call mi_switch() in ipi_bitmap_handler(). sched_switch() will then notice that ts->ts_cpu is not PCPU_GET(cpuid), and call sched_switch_migrate(), migrating the thread to the intended CPU. Thus after sched_pin() and potentially before using any PCPU variable the process can get migrated to another CPU, and now it is using a PCPU variable for the wrong processor. Given that ts_cpu can be set by other threads, it doesn't seem worth checking at sched_pin time whether ts_cpu is not the same as td_oncpu, because to make the check would require taking the thread_lock. Instead, I propose adding a check for !THREAD_CAN_MIGRATE() before calling sched_switch_migrate(). However, sched_pin() is also used by sched_bind(9) to force the thread to stay on the intended cpu. I would handle this by adding a TSF_BINDING state that is different from TSF_BOUND. The thing that has me wondering whether this is all correct is that I don't see any checks in tdq_move() or sched_setcpu() for either TSF_BOUND or THREAD_CAN_MIGRATE(). Perhaps that logic is managed in the various calling functions; in that case I would propose adding asserts that the thread is THREAD_CAN_MIGRATE() unless it's marked TSF_BINDING. Does this analysis seem correct? Thanks, matthew