From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 20 00:10:14 2015 Return-Path: Delivered-To: freebsd-hackers@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 69B32E1C for ; Sat, 20 Jun 2015 00:10:14 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: from mail-ig0-x229.google.com (mail-ig0-x229.google.com [IPv6:2607:f8b0:4001:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3834ACCD for ; Sat, 20 Jun 2015 00:10:14 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: by igblz2 with SMTP id lz2so23330085igb.1 for ; Fri, 19 Jun 2015 17:10:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=tYd80MZg2nQiRBS7Rboxu5PafQ9jJpdPtxnjqdDXLnk=; b=qborA2zjW5MK67tQqIUz8IwjmSkOL22vKEjyvpwFwsV0s0vcflbtI/ntkOtM+xy5Mj z62yJ+4cXdafIXxCVJB5K80PXU12K4WFc3cX8kle4uoJhZcYPG6rL37kqTklIwJLdGGW cxvYlTJfIvzE3SAmz2Abl3UKS/b4LfkPVCsCB96tvTiZLgF2gsQuj8T1SVHB8xNJeYuR ZFskP7uwCTJvWk8NIZOowfE6vMXseNmnn6HJajq3Xe4egfFoEI+Dn1ciHiVaLCjA/MbP 6W2uIoUmFMQyERV4l+gkdhqQvP8yi3wBS9MLH1L1CTyXgKEW5Vkhgu+6ZUVBYSnH3Cvs fIyg== MIME-Version: 1.0 X-Received: by 10.42.113.133 with SMTP id c5mr14777369icq.67.1434759011928; Fri, 19 Jun 2015 17:10:11 -0700 (PDT) Received: by 10.64.2.132 with HTTP; Fri, 19 Jun 2015 17:10:11 -0700 (PDT) Date: Fri, 19 Jun 2015 17:10:11 -0700 Message-ID: Subject: Re: Realtime process CPU starvation From: Dieter BSD To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Jun 2015 00:10:14 -0000 Chris typed: > I have a process running at realtime priority 0 > under FreeBSD 10.0. The main thread needs to run every 10 ms, #include FreeBSD is not a true real time OS. Please submit your patch to make it one. > How can a thread with the highest realtime priority not run for such a > lengthy period? Some device drivers do things like for(ever) { DELAY(MAXINT); } which does wonders for true-real-time jobs. :-( (and sucks even for non-real-time jobs) I have also found that kernel printf(9) can spoil true-real-time jobs. Hint: log(9) is your friend. I run a lot of true-real-time jobs (needs something like < 4ms) on a uniprocessor, and most of the time they run fine. Even running several of these jobs at once. Carefully crafted userland program running at rtprio 5. It took a *lot* of work finding the problems and a lot of kernel hacking (mostly device drivers) to get this far. This is with 8.2. Tried 10.1 but it is severly broken (in non-real-time-ways) and is unusable. :-( But even now I still get a true-real-time failure occasionally (and data lost forever :-( ) and no clue why. :-( > ULE scheduler Has problems. I frequently run jobs that should be either cpu-bound or disk-bound, but observe a significant percentage of cpu idle time *and* the disk i/o rate is far lower than it should be. No clue what is going on there. Fortunately these jobs are not true-real-time. I guess I'll have to call these jobs "scheduler-bound". > I was suspicious of the ZFS filesystem How much grief would it be to try a different filesystem (e.g. FFS)? (I run FFS with softdep.) > During this 500 ms, two of the four CPUs were idle. Hmmm, could the scheduler be trying to always run your true-real-time job on the same cpu and it is waiting for that cpu to become free? Is there a way to ask the scheduler to run your rt job on cpu0 and zfs on cpu1? That might help rule zfs in or out as a problem. (Clearly I am not a scheduler wizard.) Maybe see if there is a way to instrument the scheduler without having the instrumentation creating its own rt or scheduling problem?