From owner-freebsd-arm@freebsd.org Wed Jan 27 17:18:37 2016 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35672A6F379 for ; Wed, 27 Jan 2016 17:18:37 +0000 (UTC) (envelope-from wma@semihalf.com) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B9231A9B for ; Wed, 27 Jan 2016 17:18:36 +0000 (UTC) (envelope-from wma@semihalf.com) Received: by mail-io0-x234.google.com with SMTP id f81so26266841iof.0 for ; Wed, 27 Jan 2016 09:18:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to:cc:content-type; bh=rBuuie9fECRnmujtpSSXZhNaC7HT9u+/cmqcJOdCLyU=; b=NeVYTaQjpQ+BZYuJW8pMT0fd06zKtiK3ZwQDaC7WJRMaYPagWd+n3i6m8D6fyjfjes Uw1u4Dibt82zd5plEtc21B5dlOCFqoSUaWd+D3I5dTnoQZ9FmJCzBUzUikcX2pPyL0ne ntEu7YV1tBl6o1UApyaax23rZzeR/yoncTNR6qUo5WZ9zaQ9TZoQFWq8IfGNncHg/J8q CJJt6tUka50geJxisYPCw+AdnMNjNwaXdc+OGDtSnf+zfH62Kx+fTLjLAEVmtJ1JqJy2 DsZwyF+7YJet2uPQcrNSl9hNqn/3Ar+RqJb0L4HAvZSZt/MD8xGB3y84NJ9xZKeFCAwB 9MuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc :content-type; bh=rBuuie9fECRnmujtpSSXZhNaC7HT9u+/cmqcJOdCLyU=; b=GvmJ7kblsuJdAxOQ4PmBicIwe6uw+nGxOIBiY3/wkPk65w/D5jJQw4AddRh5Ijri6u +s97IzNrzYCCbxdTfdbUU4uFEKqTeMMdpt8/vW788Ie4OFdO+VNXg5GC4RZAddTFVxZv 8sG5kyWa6t7yGJxlGqS7HoO+53/xY8bXY53O67LwbWxPmid3GMjWPJaTug9xnJ9DILo6 DUcMWf716d2NGYGGMAjWUXyZJg/RIcd1FB4lrhZ8pG9v0ZquC9Tln4xHJmQj4jskUIWj PVD4avw7q5eEYI8yH0/WjPzGxeMF5sGuae9M3irMyNjWVLFPYNp30oTs2pRcYBiRJEpm 3fQA== X-Gm-Message-State: AG10YOQkJvcG/kMkIPHaBTT7eEVAMbWyD3O32UhEgAJlmlHgkuimhYze+u3u/jzkF4NgacnBlKItapcIyHjLeA== X-Received: by 10.107.3.37 with SMTP id 37mr33571660iod.182.1453915116314; Wed, 27 Jan 2016 09:18:36 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.154.19 with HTTP; Wed, 27 Jan 2016 09:18:16 -0800 (PST) From: Wojciech Macek Date: Wed, 27 Jan 2016 18:18:16 +0100 Message-ID: Subject: SCHED_ULE race condition, fix proposal To: developers@freebsd.org, freebsd-arm@freebsd.org Cc: Olivier Houchard , arm64-dev Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 17:18:37 -0000 Hello, I've encountered a very nasty race condition during debugging armv8 HWPMC. It seems that ULE scheduler can execute the same thread on two different CPUs at the same time... Here is the scenario. The PMC driver must execute some of the code on the CPU0. To ensure that, a process migration is triggered as following: thread_lock(curthread); sched_bind(curthread, cpu); thread_unlock(curthread); KASSERT(curthread->td_oncpu == cpu, ("[pmc,%d] CPU not bound [cpu=%d, curr=%d]", __LINE__, cpu, curthread->td_oncpu)); That causes the context switch and (finally) execution of sched_switch() function. The code correctly detects migration and calls sched_switch_migrate. That function is supposed to add current thread to the runqueue of another CPU ("tdn" variable). So it does: tdq_lock_pair(tdn, tdq); tdq_add(tdn, td, flags); tdq_notify(tdn, td); TDQ_UNLOCK(tdn); spinlock_exit(); But that sometimes is causing a crash, because the other CPU is staring to process mi_switch as soon as the IPI arrives (via tdq_notify) and the runqueue lock is released. The problem is, that the thread does not contain valid register set, because its context was not yet stored - that happens later in machine dependent cpu_switch function. In another words, the sched_switch run on the CPU we want the thread to migrate onto restores thread context before it was actually stored on another core - that causes setting regs/pc/lt to some junk data and crash. I'd like to discuss a possible solution for this. I think it would be reasonable to extend cpu_switch to be capable of releasing a lock as the last thing it does after storing everything into the PCB. We could then remove the "TDQ_UNLOCK(tdn);" from the sched_switch_migrate and be sure that in the situation of migration nobody is allowed to touch the target runqueue until the migrating process finishes storing its context. But first I'd like to discuss some possible alternatives and maybe find another solution, because any change in this area will impact all supported architectures. Regards, Wojtek wma@freebsd.org