From owner-freebsd-hackers@FreeBSD.ORG Thu Jul 29 23:39:03 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C90D106566B for ; Thu, 29 Jul 2010 23:39:03 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 167308FC17 for ; Thu, 29 Jul 2010 23:39:02 +0000 (UTC) Received: by iwn35 with SMTP id 35so939082iwn.13 for ; Thu, 29 Jul 2010 16:39:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=srbXnpSkwnZyWReN2hEzf3ACyGvYQNjoyKDQ2bOvOmM=; b=Lw8PqhJT7jzmhrrw0mb4DelDo6R7ni5ImEh+khweawC/vLa0g+0k5HjLBnDalhAaYe w0FC3vfL+HVMtlcuNOLWLRHWk10WZQj/OQGm4zVNOAGaLhCRulckdh28TKgrz0mm3Q+1 i8o92pn23drjfNBcUWXpGNQcf82H+tO9O2aUY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; b=GuBzI/LH20/o4wBarwm2qn/9hv8ddPKnIgsvQc8mTr8oDYqJIStJxBA7p183x82J0q vcE1Euz0vS5vJ8Hv4yQz5G9bbumHQfktJUvASXGLGD47sxFDeDESJ7A8qELfvfFgmWdo ZbGwwFum7dRihK/0FN58fddWLCLiy6p+9pvXk= MIME-Version: 1.0 Received: by 10.42.9.69 with SMTP id l5mr186837icl.80.1280446742146; Thu, 29 Jul 2010 16:39:02 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.42.6.85 with HTTP; Thu, 29 Jul 2010 16:39:02 -0700 (PDT) Date: Thu, 29 Jul 2010 16:39:02 -0700 X-Google-Sender-Auth: 4ouVY9hWjuzZ2dhMKYwYcWwNxKs Message-ID: From: mdf@FreeBSD.org To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: sched_pin() versus PCPU_GET X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2010 23:39:03 -0000 We've seen a few instances at work where witness_warn() in ast() indicates the sched lock is still held, but the place it claims it was held by is in fact sometimes not possible to keep the lock, like: thread_lock(td); td->td_flags &= ~TDF_SELECT; thread_unlock(td); What I was wondering is, even though the assembly I see in objdump -S for witness_warn has the increment of td_pinned before the PCPU_GET: ffffffff802db210: 65 48 8b 1c 25 00 00 mov %gs:0x0,%rbx ffffffff802db217: 00 00 ffffffff802db219: ff 83 04 01 00 00 incl 0x104(%rbx) * Pin the thread in order to avoid problems with thread migration. * Once that all verifies are passed about spinlocks ownership, * the thread is in a safe path and it can be unpinned. */ sched_pin(); lock_list = PCPU_GET(spinlocks); ffffffff802db21f: 65 48 8b 04 25 48 00 mov %gs:0x48,%rax ffffffff802db226: 00 00 if (lock_list != NULL && lock_list->ll_count != 0) { ffffffff802db228: 48 85 c0 test %rax,%rax * Pin the thread in order to avoid problems with thread migration. * Once that all verifies are passed about spinlocks ownership, * the thread is in a safe path and it can be unpinned. */ sched_pin(); lock_list = PCPU_GET(spinlocks); ffffffff802db22b: 48 89 85 f0 fe ff ff mov %rax,-0x110(%rbp) ffffffff802db232: 48 89 85 f8 fe ff ff mov %rax,-0x108(%rbp) if (lock_list != NULL && lock_list->ll_count != 0) { ffffffff802db239: 0f 84 ff 00 00 00 je ffffffff802db33e ffffffff802db23f: 44 8b 60 50 mov 0x50(%rax),%r12d is it possible for the hardware to do any re-ordering here? The reason I'm suspicious is not just that the code doesn't have a lock leak at the indicated point, but in one instance I can see in the dump that the lock_list local from witness_warn is from the pcpu structure for CPU 0 (and I was warned about sched lock 0), but the thread id in panic_cpu is 2. So clearly the thread was being migrated right around panic time. This is the amd64 kernel on stable/7. I'm not sure exactly what kind of hardware; it's a 4-way Intel chip from about 3 or 4 years ago IIRC. So... do we need some kind of barrier in the code for sched_pin() for it to really do what it claims? Could the hardware have re-ordered the "mov %gs:0x48,%rax" PCPU_GET to before the sched_pin() increment? Thanks, matthew