From owner-freebsd-virtualization@FreeBSD.ORG Fri Jun 5 23:03:32 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F0B84A6E for ; Fri, 5 Jun 2015 23:03:31 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-oi0-x22b.google.com (unknown [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B0C211C98 for ; Fri, 5 Jun 2015 23:03:31 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by oiww2 with SMTP id w2so63457457oiw.0 for ; Fri, 05 Jun 2015 16:03:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=c6TP8KRYcbNW++cdwIFS3SyKFUVIRgCFhnRzuYncA/Q=; b=wxowSLealvCyAIJ9gK6f7Jg8B+tvle2OwihxQlRw3ZcPRY/5hcs2WVM7v5TGcvh/km Zz4t19d2vUZ7vQP8zhiZkVxviB64oCjokw5euDJrNWyuHB+RFNYnDs6JcF9UkShcfTZ8 nuk+fRPLhVCkGGbkwPA3qvwdFBlfAyPZdNMtU8zu2oV1+5C1u8uBihA5Wja0MzZ11Uix NYbgECRojODgL2FucxAZzEEL9aSvoxxdTMtVCU2GV5Dw1XG9n2E0BwmR3hle4mS7uiCv y5lAsJWSkJ5CQDwcr1eF1MQlnkwafZxWnx4CpeTq5PcIlnTxD2Kixs5aE1vC/x4xPcSh DTgQ== MIME-Version: 1.0 X-Received: by 10.60.155.97 with SMTP id vv1mr5028114oeb.15.1433545410656; Fri, 05 Jun 2015 16:03:30 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Fri, 5 Jun 2015 16:03:30 -0700 (PDT) Date: Sat, 6 Jun 2015 02:03:30 +0300 Message-ID: Subject: Implementation of gang-scheduling for bhyve VCPUs From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jun 2015 23:03:32 -0000 Hello, I have broken the implementation of gang-scheduling in three steps: 1. Marking threads that should be gang scheduled - add a custom field "int gang" to the FreeBSD thread structure, defined in proc.h - create a custom ioctl request code VM_SET_GANG, that will set the 'gang' field of the VCPU thread, to the gang it is part of - in bhyverun.c, in the fbsdrun_start_thread() function that is invoked when a new pthread is created (a new VCPU is added), call the ioctl function with VM_SET_GANG request code 2.When a thread that must be gang scheduled is selected from a CPU's thread queue, inform other CPUs so they can schedule a thread belonging to that gang - add two new fields to struct tdq: bool gang_leader /* set if CPU is gang leader */ int scheduled_gang /* set if current CPU needs to search it's queue for a thread belonging to scheduled_gang */ - in sched_ule.c, in the sched_choose() function, after a thread is selected, check if it is part of a gang (has the gang field set) - if so, mark the CPU it is on as gang-leader (set gang_leader in tdq), and use function smp_rendezvous() to send IPI to all other CPUs to run schedule_gang() /* custom function defined in sched_ule.c*/ - in function schedule_gang(), set field scheduled_gang, and inform the processor it needs to reschedule (not yet sure how) 3. Scheduling rest of threads - in function tdq_choose(), check first if the tdq's scheduled_gang field is set. If so, reset scheduled_gang field to 0 and return the first thread in the tdq belonging to the gang. If sched_gang==0 or no thread belonging to the gang can be found continue with normal thread selection. This is the implementation of gang-scheduling on FreeBSD the way I have sketched it. If anyone has any thoughts/suggestions, what will work, what will not work, what could have been done better/simpler/easier, anything at all, I would really appreciate and welcome it. Thank you, Stefan