From owner-freebsd-virtualization@freebsd.org Mon Dec 12 12:27:46 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BCF8C736F6 for ; Mon, 12 Dec 2016 12:27:46 +0000 (UTC) (envelope-from ian.campbell@docker.com) Received: from mail-wj0-x22f.google.com (mail-wj0-x22f.google.com [IPv6:2a00:1450:400c:c01::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3C0A4EB7 for ; Mon, 12 Dec 2016 12:27:46 +0000 (UTC) (envelope-from ian.campbell@docker.com) Received: by mail-wj0-x22f.google.com with SMTP id tk12so70349623wjb.3 for ; Mon, 12 Dec 2016 04:27:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=docker.com; s=google; h=mime-version:from:date:message-id:subject:to:cc; bh=grAdJj/7HVPQ7EBpC0VnIxhqJnL7hL6fveShgVVnYrY=; b=D7ZXdgOxgz2CEEqw2jkwYhfS8F9Wu15wBYN+tavaXeCqrlgsWRj74yJla0CWinx64Z r2ZZJ7+oQCGx2J9pt4/98ni2W8XOfZg4MyvTO+vtUGPYZ1s2wfUsBIvWVqUbwstFxQ0g lKmL9eP6KC4g4sYX9N8Ugt/7GiBYC0erhIS0M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=grAdJj/7HVPQ7EBpC0VnIxhqJnL7hL6fveShgVVnYrY=; b=ZpG5obizoG1e3G4pIdpRGuj/6+QEKy7EFM62h5ZaQLAKa/VN4z4NlPwfAJMFItFtzn ohv9wsV3ZXQ238VyPVhPSHtU36nzfm4Xo6EMX6bS9VDUOv4rf0nNhUSUa+Nx6tsLgksM pF58cA9tiXom25aLQvcR6lHtqM2fZnMvv0QcxNAPQwEPr0M3GAUV+/siyEz8tdi7uUMw 6mssUEt5F3P53K4LwZCetFndYBf/pmXW7+V0unAquFQySTFf5uicrcrIttsFelQJTe9G 1/BQ1UMw+P/c0vwcltAc3P0LPbtW6+gZMIsm1HZPcbBxM7ZczETUEbEcKL7WFQBznAG3 Snzw== X-Gm-Message-State: AKaTC034zFOp51ZMYtQQSj8d2KllnkCWjOfPzvBDc2sQq1NYBO6UbuTE/LlCtnsI92DIyvl0X8osQv+nuDR5iwhw X-Received: by 10.194.105.228 with SMTP id gp4mr59936237wjb.208.1481545664247; Mon, 12 Dec 2016 04:27:44 -0800 (PST) MIME-Version: 1.0 Received: by 10.194.107.231 with HTTP; Mon, 12 Dec 2016 04:27:43 -0800 (PST) From: Ian Campbell Date: Mon, 12 Dec 2016 12:27:43 +0000 Message-ID: Subject: Query about bhyve's blockif_cancel and the signalling mechanisms To: freebsd-virtualization@freebsd.org Cc: anil@recoil.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2016 12:27:46 -0000 Hello, Recently I've been investigating a blk io hang/deadlock[0] in Hyperkit (a bhyve derived hypervisor for Mac's using the OSX Hypervisor framework). While I think I understand that issue (TL;DR: OSX kevent EVFILT_SIGNAL does not receive signals directed to specific threads rather than entire processes, which is different from FreeBSD where it does) there are some aspects of the code which I'm unsure about the mechanisms by which they work on FreeBSD. To recap my understanding of the mechanisms at work (glossing over the queue handling and condvars involved etc), the bhyve block_if infrastructure registers a callback for SIGCONT with the mevent subsystem, which is a kevent/kqueue thing which delivers events to the main thread (mevent_dispatch is the last thing in main()) it also sets SIGCONT to SIG_IGN. When a disk controller device model wants to cancel a block request (e.g. in ahci_port_stop) it calls blockif_cancel which sends a SIGCONT to the blkio thread which has claimed the request, notionally to kick it out of whatever blocking system call it is in and cause it to return an error to the device model. The signal handler callback (in the mevent thread) then kicks the thread which sent the signal to indicate completion of the cancellation. I go in to some more detail of how I think things work in [0] but that's the gist of it. The main thing I do not follow is whether or not the blkio thread is actually interrupted at all when the signal has been configured to be delivered via the kevent/kqueue mechanisms to a 3rd unrelated thread. I've dug around in the FreeBSD kevent and signal man pages but I cannot find any part which describes anything of the semantics which bhyve seems to be relying on (which seems to be that the system call in the target thread will return EINTR at some point before the thread which is "handling" the signal via kevent/kqueue sees that event). Have I missed something here or is bhyve relying on some subtle underlying semantics? I have a secondary concern which is what happens if the IO thread is on its way to making a blocking system call in blockif_proc but has not actually done so when the signal is delivered. It seems like it would simply carry on and make the blocking call with perhaps unexpected consequences (i/o getting wedged, perhaps only until a second reset attempt). I've not actually seen this happening though and there's a chance I'm simply over thinking things after staring at them for so long! I should say that I've mostly been looking at the hyperkit code here but AFAICT it has not diverged in any relevant ways from the bhyve code (although it might now have to in the future given the differing semantics of EVFILT_SIGNAL on OSX). Thanks, Ian. [0] https://github.com/docker/hyperkit/issues/94