From owner-freebsd-hackers@freebsd.org Tue Jul 7 10:10:21 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DCB098F778 for ; Tue, 7 Jul 2015 10:10:21 +0000 (UTC) (envelope-from pjalaber@gmail.com) Received: from mail-qg0-x232.google.com (mail-qg0-x232.google.com [IPv6:2607:f8b0:400d:c04::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2EC671069 for ; Tue, 7 Jul 2015 10:10:21 +0000 (UTC) (envelope-from pjalaber@gmail.com) Received: by qgii30 with SMTP id i30so81698251qgi.1 for ; Tue, 07 Jul 2015 03:10:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=FPP0BIWpmg5w5szR9FmQSQPOIAmGRH9Cn9hDiLGQzxc=; b=vTJU6zL4AeHnnTZHfK7sDibg4b71H4qCzus4raZbDf3zvvrmKn81XGEmRt0mZPT5AC TuINzrk6QkXH77+17usfjtgNfsuisLxYtN5ygkx6i+lfwPasglM+QMQ0wmfyGcXXEehH VAQyrMVk3sO6iK5QYnCqALILuwR7p7Mrw+IV0rqDLwm4GeH4jtN208RSjmugnowhyTBu XKppcnuNI/qeGDVoxnJ3paCCArcEHNUuOgzFeQiztl0Jvrufe7dH+nvO0BOJUfCsLEfz cABp/1f1B2OTsfE075Bww9U2DouuWW2tt956GR+SDmkWb0Rg1A7/UflwtsOavVKhoHtE +iRg== MIME-Version: 1.0 X-Received: by 10.55.33.213 with SMTP id f82mr5280510qki.107.1436263819965; Tue, 07 Jul 2015 03:10:19 -0700 (PDT) Received: by 10.140.36.233 with HTTP; Tue, 7 Jul 2015 03:10:19 -0700 (PDT) Date: Tue, 7 Jul 2015 12:10:19 +0200 Message-ID: Subject: adaptive rwlock deadlock From: Philippe Jalaber To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2015 10:10:21 -0000 Hi, I am facing a strange problem using the network stack and adaptive rwlocks running Freebsd 9.3. Basically I can reproduce the problem with 3 threads: 1) thread 1 has taken the rwlock of structure inpcb in exclusive mode in tcp_input.c. This thread also runs my own code and repeatedly takes a rwlock (called g_rwlock) in shared mode and releases it, until a shared object is marked not "busy" any more: rwlock(inp_lock); .... do { // thread is active waiting in the loop rlock(g_rwlock); o = find(); if ( o == NULL ) break; busy = o.busy; if (o != NULL && busy) runlock(g_rwlock); } while ( busy ); if ( o != NULL ) { // do something with o .... } runlock(g_rwlock); .... 2) thread 2 wants to set the shared object as "ready". So it tries to take g_rwlock in exclusive mode and is blocked in _rw_wlock_hard@kern_rwlock.c:815 "turnstile_wait(ts, rw_owner(rw), TS_EXCLUSIVE_QUEUE)" because thread 1 has already taken it in shared mode: wlock(g_rwlock); o = find(); if ( o != NULL ) o.busy = 1; wunlock(g_rwlock); // o is busy so work on it without any lock .... wlock(g_rwlock); // thread is blocked here o.busy = 0; maybe_delete(o); wunlock(g_rwlock); 3) thread 3 spins on the same inpcb rwlock than thread 1 in _rw_wlock_hard@kern_rwlock.c:721 "while ((struct thread*)RW_OWNER(rw->rw_lock) == owner && TD_IS_RUNNING(owner)) " My target machine has two cpus. Thread 1 is pinned to cpu 0. Thread 2 and Thread 3 are pinned to cpu 1. Thread 1 and Thread 2 have a priority of 28. Thread 3 has a priority of 127 Now what seems to happen is that when thread 1 calls runlock(g_rwlock), it calls turnstile_broadcast@kern_rwlock.c:650, but thread 2 never regains control because thread 3 is spinning on the inpcb rwlock. Also the condition TD_IS_RUNNING(owner) is always true because thread 1 is active waiting in a loop. So the 3 threads deadlock. Note that if I compile the kernel without adaptive rwlocks it works without any problem. A workaround is to add a call to "sched_relinquish(curthread)" in thread 1 in the loop just after the call to runlock. I am also wondering about the code in _rw_runlock after "turnstile_broadcast(ts, queue)". Isn't the flag RW_LOCK_WRITE_WAITERS definitely lost if the other thread which is blocked in turnstile_wait never regains control ? Thank you for your time, Regards, Philippe