From owner-freebsd-virtualization@freebsd.org Wed Dec 14 12:11:49 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E743C779B8 for ; Wed, 14 Dec 2016 12:11:49 +0000 (UTC) (envelope-from ian.campbell@docker.com) Received: from mail-wj0-x22f.google.com (mail-wj0-x22f.google.com [IPv6:2a00:1450:400c:c01::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A1EA1834 for ; Wed, 14 Dec 2016 12:11:49 +0000 (UTC) (envelope-from ian.campbell@docker.com) Received: by mail-wj0-x22f.google.com with SMTP id v7so30607202wjy.2 for ; Wed, 14 Dec 2016 04:11:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=docker.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=fAcxnqxYSQ2Zj0U39FjBnB1caJ7EHUhBluTak8gyeEY=; b=EA6KL29T84VgmzsGsh6nEy2AZcHLQnexxNILsyYUhu+WrQifHtkF8IwYnGFZJopBgf Zt+NS/UMoP40HhKmZqkOq6bDZdffBDY2a4xAW66HkJOwsRdiumMGGbjUi/5XtjtPtNBB /+1dVeDDQwB/bftp2NRzl9jpZCvcouj0ihrbs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=fAcxnqxYSQ2Zj0U39FjBnB1caJ7EHUhBluTak8gyeEY=; b=VrHV8eJl1NG5e/8lVhKZrjbEkTaTIRo6b8K2UuSBrEykeNKyUn8t16QHhSm/TssuWq QxMWnRTu7RtkGWW2ne7MYXcxsaAA7kQYpa34aXkaOmK23ujp2qVYv750/ZrwdKWxs1qk AvPtoKbDiEAzsIuCXpXQpn3pqlwQfs+zWREReQB2FCt3ahByXLLJnvaNCy/j5R/vDxAD Ti9K4pPl//l4WsUS+cZwyCZny+3c8jFSrhUeQvH6Y6Ad91Z0vE8/OorJEBzlYk0jomYC h+BTBlXESXhlwMeEvKN9ZXylk81VRrgoAcpzXuQ2Jx/2PKo1XufXDBVfL0DqzxZRKlG0 PNkQ== X-Gm-Message-State: AKaTC01DQ2UorCYr24HryJIFXaNTsmRFIh1jHFKzqTIB1i+QWS2RJLv0lj7CaHATccJEdWvVxjXEAO5RqpqWm+Ox X-Received: by 10.28.113.76 with SMTP id m73mr6630209wmc.114.1481717507201; Wed, 14 Dec 2016 04:11:47 -0800 (PST) MIME-Version: 1.0 Received: by 10.194.107.231 with HTTP; Wed, 14 Dec 2016 04:11:46 -0800 (PST) In-Reply-To: <631f775d-8d61-55ba-1e7b-8ce4fcadcbf3@freebsd.org> References: <631f775d-8d61-55ba-1e7b-8ce4fcadcbf3@freebsd.org> From: Ian Campbell Date: Wed, 14 Dec 2016 12:11:46 +0000 Message-ID: Subject: Re: Query about bhyve's blockif_cancel and the signalling mechanisms To: Peter Grehan Cc: Tycho Nightingale , freebsd-virtualization@freebsd.org, Anil Madhavapeddy Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2016 12:11:49 -0000 On 13 December 2016 at 06:32, Peter Grehan wrote: >> I've dug around in the FreeBSD kevent and signal man pages but I >> cannot find any part which describes anything of the semantics which >> bhyve seems to be relying on (which seems to be that the system call >> in the target thread will return EINTR at some point before the thread >> which is "handling" the signal via kevent/kqueue sees that event). >> >> Have I missed something here or is bhyve relying on some subtle >> underlying semantics? > > > I didn't think it too FreeBSD-specific - if a thread is blocked in a system > call, sending a signal should force it to exit on most Unices. Sure, the thing I was unsure about was whether that was the case when the application has asked for signal delivery via some other mechanism than the traditional asynchronous signal handlers such as kevent. Especially when the signal itself if SIG_IGN'd (as SIGCONT is in bhyve in blockif_init). The attached test tries to mimic what bhyve is doing, it has a blocked io thread and a thread handling kevents, with the signal set to SIG_IGN. A signal is then sent to the io thread. Running it on FreeBSD 11[0] it outputs: main: starting kevent thread main: starting io thread kevent thread: opening kqueue kevent thread: setting up SIGCONT event kevent thread: waiting for SIGCONT event io thread: blocking in read main: sending SIGCONT io thread main: waiting for io thread to complete kevent thread: received SIGCONT event and then blocks indefinitely. The kevent thread has seen the SIGCONT but the read in the io thread has not been interrupted. If I #define USE_HANDLER at the same time as USE_KEVENT then instead I see: main: starting kevent thread kevent thread: opening kqueue main: starting io thread kevent thread: setting up SIGCONT event io thread: blocking in read kevent thread: waiting for SIGCONT event main: sending SIGCONT io thread sigcont: received SIGCONT main: waiting for io thread to complete io thread: Read from pipe returned errno=4: Interrupted system call io thread: exiting main: waiting for kevent thread to complete kevent thread: received SIGCONT event main: all done i.e. the signal is seen by both the kevent thread and the async handler in the io thread, the latter of which causes EINTR from the read. bhyve sets SIGCONT to SIG_IGN so falls into the first case I think. If there is really a bhyve bug here I think it should be fixable with a dummy SIGCONT handler alongside the kevent thread. I think it wouldn't be reliant on a specific ordering of kevent vs async handler invocation due to the while loop on status == BUSY in blockif_cancel which Tycho opened my eyes to earlier. Ian. [0] ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/11.0-RELEASE/amd64/Latest/FreeBSD-11.0-RELEASE-amd64.qcow2.xz fed to virt-manager.