From owner-freebsd-xen@freebsd.org Sun Jul 29 13:43:37 2018 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1222110578E4 for ; Sun, 29 Jul 2018 13:43:37 +0000 (UTC) (envelope-from pratiy0100@gmail.com) Received: from mail-it0-f53.google.com (mail-it0-f53.google.com [209.85.214.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 826A181A98; Sun, 29 Jul 2018 13:43:36 +0000 (UTC) (envelope-from pratiy0100@gmail.com) Received: by mail-it0-f53.google.com with SMTP id s7-v6so13762957itb.4; Sun, 29 Jul 2018 06:43:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=E35pgb/USmEMXS3r/HI0zKRgyYN0PpBTWM/lDvSXfLc=; b=OxxnoNrH2dgW9W75Y1s36LQSl5+LDJFTT7dOFsCjSGXaEDvqSakSz03X+Zm7GLn8nH kQvO0YNNiQsqBIOLAn5sqdxHvSo9puGAEFcrzOGSn/e21aFH9Vjlhw1cQyu25FB5OGHf Zs01qG2dHYwfWGsVuZuwzz13Md/4ZFTxWOU27bUYlRMvsm/SYOQbJRXOrB3g2mihK9cf 09f3bOAkNceqH4OOQ+4Jw3Cq2rv6EOZOjHC7VdRLDzqvAp3zxqq/cak0TsmKi9uKO8ie a7rr7vOihvx1DX9C+4u40fTI9iHCRGICaLj8iFQZQqSd49iC3KToyAufCoNRThtNapCE Hu7A== X-Gm-Message-State: AOUpUlFpqjssMdE0mdw9rUCf97iHfrg+MBTRljJhpDHY0o0eyrY2nLUk 987WRfh5kTleuctJzZU6DWyNLN5ZLQpaog== X-Google-Smtp-Source: AAOMgpfWfJSXWrsm8XBr0r7c5yI9gOmpnVpb3Jfs04OdR7meq7mypdZLNBtey0wZwXQH/uifD2R5sA== X-Received: by 2002:a24:2206:: with SMTP id o6-v6mr11354644ito.112.1532867972973; Sun, 29 Jul 2018 05:39:32 -0700 (PDT) Received: from mail-io0-f181.google.com (mail-io0-f181.google.com. [209.85.223.181]) by smtp.gmail.com with ESMTPSA id t4-v6sm2983034iof.55.2018.07.29.05.39.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 29 Jul 2018 05:39:32 -0700 (PDT) Received: by mail-io0-f181.google.com with SMTP id l7-v6so7707446ioj.1; Sun, 29 Jul 2018 05:39:32 -0700 (PDT) X-Received: by 2002:a6b:c844:: with SMTP id y65-v6mr10703672iof.187.1532867972491; Sun, 29 Jul 2018 05:39:32 -0700 (PDT) MIME-Version: 1.0 From: Pratyush Yadav Date: Sun, 29 Jul 2018 18:08:56 +0530 X-Gmail-Original-Message-ID: Message-ID: Subject: Recursion in non-recursive mutex when using the grant table free callbacks To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: Akshay Jaggi , Edward Napierala , FreeBSD-Xen Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jul 2018 13:43:37 -0000 Hi, Currently, the grant table free callbacks can not work. This is because of a recursion on a non-recursive mutex that causes a kernel panic. The cause of the recursion is: check_free_callbacks() is always called with the lock gnttab_list_lock held. So, the callback function is called with the lock held. So, when the client uses any of the grant reference allocation methods get_free_entries() is called, which tries to acquire gnttab_list_lock(grant_table.c:77 [0]), causing a recursion on the lock. I'm not sure what the correct fix would be though. One way I can think of is that check_free_callback() should be called without the lock held. But with this fix, it is possible for the callback to be called even though the grant references it needs are not available. This would happen when another thread takes those references while the current thread has completed the check if(gnttab_free_count >= callback->count) but has not yet called the callback (grant_table,c:105 [1]). I think a better way to fix this would be to have a check in get_free_entries() whether the current thread holds the lock, so it does not try to acquire the lock if the current thread already holds it. If you agree, I will send a patch. [0] https://github.com/freebsd/freebsd/blob/master/sys/dev/xen/grant_table/grant_table.c#L77 [1] https://github.com/freebsd/freebsd/blob/master/sys/dev/xen/grant_table/grant_table.c#L105 -- Regards, Pratyush Yadav From owner-freebsd-xen@freebsd.org Sun Jul 29 22:30:32 2018 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F751106588C for ; Sun, 29 Jul 2018 22:30:32 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 848367CE80; Sun, 29 Jul 2018 22:30:31 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: by mail-lj1-f181.google.com with SMTP id j19-v6so8756993ljc.7; Sun, 29 Jul 2018 15:30:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vS9H3r7Y/87a7vyrILh9R9mS3kAk1s3iimbsMfOp3RU=; b=Y9uOsZQKl3phRo8wYnfH3HwkY+YrTOoBsyhOEy+rjT+vxTgBnonFqtz1hEJPpmCg2N X/x3TXA0N93JQbtut4+xYN93L1FfAcEaH1D5UUwE7AbjQTxESEPmZ+yBSIODQGMxuBHR gmmDwVgKTGl8MKuRJEf9xmIWN4SNwCrJSGy3p9Tne9R2yqNnCIH/Pn05Ndd6G3FeO74c YHfgG1WVp//JpPsbzfNM7S9dbLlrW3k/CN/U/AJ2BmSt/QTA/+ACtjsePDBIAbHrBDGA 6ROm7hoMPAl0QRabXsEpMYACGT/p/0qwJRAj0HlQ4kcYpK/2RDOs7DeZe88W73CrQKe3 wWUQ== X-Gm-Message-State: AOUpUlHiBSKgiiLg2Eh2lrVw7HGE7kdWTktemTYOGAs8OCNH4D+6IzLE 2k9XSUAfyBo9EM4ywtPoT/opbfk+gVY= X-Google-Smtp-Source: AAOMgpeho0sPaomiOXdMpXzte2n9GI7ZkaGOSh+uZaYik8bktcdF3PDzYuVAIw2UReVSmXwpNI6bmw== X-Received: by 2002:a2e:8950:: with SMTP id b16-v6mr5802384ljk.111.1532896919843; Sun, 29 Jul 2018 13:41:59 -0700 (PDT) Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com. [209.85.167.47]) by smtp.gmail.com with ESMTPSA id m9-v6sm1317879lfg.0.2018.07.29.13.41.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 29 Jul 2018 13:41:59 -0700 (PDT) Received: by mail-lf1-f47.google.com with SMTP id u14-v6so6735791lfu.0; Sun, 29 Jul 2018 13:41:59 -0700 (PDT) X-Received: by 2002:a19:c403:: with SMTP id u3-v6mr8380254lff.87.1532896919118; Sun, 29 Jul 2018 13:41:59 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Akshay Jaggi Date: Sun, 29 Jul 2018 21:41:47 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Recursion in non-recursive mutex when using the grant table free callbacks To: Pratyush Yadav Cc: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Edward Napierala , FreeBSD-Xen Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jul 2018 22:30:32 -0000 Roger would be a better person to answer this and I would wait for him to respond, but here are my 2 cents. First let's finalize and record the semantics of the callbacks. What guarantee do we want to provide callbacks about locks held (no locks held OR x,y,z locks held)? This is mostly a product decision. Usually callbacks are called with no locks held. Post this problem is simple. If the list_lock would be held during callbacks, we need to provide a lock-free function (which assumes lock would be held before function is called and use that) and if the lock would not be held we should fix the locking to ensure the case you mentioned should not happen. WDYT? Regards, Akshay On Sun, 29 Jul 2018 at 14:43, Pratyush Yadav wrote: > Hi, > > Currently, the grant table free callbacks can not work. This is > because of a recursion on a non-recursive mutex that causes a kernel > panic. The cause of the recursion is: check_free_callbacks() is always > called with the lock gnttab_list_lock held. So, the callback function > is called with the lock held. So, when the client uses any of the > grant reference allocation methods get_free_entries() is called, which > tries to acquire gnttab_list_lock(grant_table.c:77 [0]), causing a > recursion on the lock. > > I'm not sure what the correct fix would be though. One way I can think > of is that check_free_callback() should be called without the lock > held. But with this fix, it is possible for the callback to be called > even though the grant references it needs are not available. This > would happen when another thread takes those references while the > current thread has completed the check if(gnttab_free_count >= > callback->count) but has not yet called the callback > (grant_table,c:105 [1]). > > I think a better way to fix this would be to have a check in > get_free_entries() whether the current thread holds the lock, so it > does not try to acquire the lock if the current thread already holds > it. > > If you agree, I will send a patch. > > > > [0] > https://github.com/freebsd/freebsd/blob/master/sys/dev/xen/grant_table/grant_table.c#L77 > [1] > https://github.com/freebsd/freebsd/blob/master/sys/dev/xen/grant_table/grant_table.c#L105 > > -- > Regards, > Pratyush Yadav > From owner-freebsd-xen@freebsd.org Mon Jul 30 09:32:05 2018 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 947B110507D8 for ; Mon, 30 Jul 2018 09:32:05 +0000 (UTC) (envelope-from royger@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BE079045B; Mon, 30 Jul 2018 09:32:05 +0000 (UTC) (envelope-from royger@freebsd.org) Received: from localhost (95.red-88-3-119.dynamicip.rima-tde.net [88.3.119.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: royger) by smtp.freebsd.org (Postfix) with ESMTPSA id AA9269292; Mon, 30 Jul 2018 09:32:04 +0000 (UTC) (envelope-from royger@freebsd.org) Date: Mon, 30 Jul 2018 11:31:55 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Pratyush Yadav Cc: FreeBSD-Xen , Akshay Jaggi , Edward Napierala Subject: Re: Recursion in non-recursive mutex when using the grant table free callbacks Message-ID: <20180730093155.c6ihgafia5ducw65@mac.bytemobile.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 09:32:05 -0000 On Sun, Jul 29, 2018 at 06:08:56PM +0530, Pratyush Yadav wrote: > Hi, > > Currently, the grant table free callbacks can not work. This is > because of a recursion on a non-recursive mutex that causes a kernel > panic. The cause of the recursion is: check_free_callbacks() is always > called with the lock gnttab_list_lock held. So, the callback function > is called with the lock held. So, when the client uses any of the > grant reference allocation methods get_free_entries() is called, which > tries to acquire gnttab_list_lock(grant_table.c:77 [0]), causing a > recursion on the lock. > > I'm not sure what the correct fix would be though. One way I can think > of is that check_free_callback() should be called without the lock > held. But with this fix, it is possible for the callback to be called > even though the grant references it needs are not available. This > would happen when another thread takes those references while the > current thread has completed the check if(gnttab_free_count >= > callback->count) but has not yet called the callback > (grant_table,c:105 [1]). > > I think a better way to fix this would be to have a check in > get_free_entries() whether the current thread holds the lock, so it > does not try to acquire the lock if the current thread already holds > it. I agree in the analysis, however I think the proper solution is to use a recursive lock. Roger. From owner-freebsd-xen@freebsd.org Mon Jul 30 10:19:34 2018 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 340F010518CB for ; Mon, 30 Jul 2018 10:19:34 +0000 (UTC) (envelope-from royger@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C62EE91C51; Mon, 30 Jul 2018 10:19:33 +0000 (UTC) (envelope-from royger@FreeBSD.org) Received: from localhost (95.red-88-3-119.dynamicip.rima-tde.net [88.3.119.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: royger) by smtp.freebsd.org (Postfix) with ESMTPSA id 23830969A; Mon, 30 Jul 2018 10:19:32 +0000 (UTC) (envelope-from royger@FreeBSD.org) Date: Mon, 30 Jul 2018 12:19:24 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Roger Pau =?utf-8?B?TW9ubsOp?= Cc: freebsd-xen@freebsd.org Subject: Re: HEADS UP: merged PVHv2 support and future plans Message-ID: <20180730101924.aatt5ug33kg6dfil@mac.bytemobile.com> References: <20180719090444.jojyziar2c6wy735@mac> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20180719090444.jojyziar2c6wy735@mac> User-Agent: NeoMutt/20180716 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 10:19:34 -0000 On Thu, Jul 19, 2018 at 11:04:44AM +0200, Roger Pau Monné wrote: [...] > After introducing PVHv2 PVHv1 was deprecated and PVHv1 has been > removed from the hypervisor in recent versions, that's why the Xen > ports package is still stuck with Xen 4.7, because later versions > removed PVHv1 support. With the addition of PVHv2 to FreeBSD the port > can be updated to newer Xen versions and we can move forward. Packages for Xen 4.11 are now available in pkg. Remember that in order to use those you need a very recent FreeBSD kernel (r336475 or newer) and the Xen command line has slightly changed, so dom0=pvh must be used instead of dom0pvh in order to boot. I will see about changing this in the handbook. Roger. From owner-freebsd-xen@freebsd.org Mon Jul 30 10:36:49 2018 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE58F1051D5F for ; Mon, 30 Jul 2018 10:36:48 +0000 (UTC) (envelope-from pratiy0100@gmail.com) Received: from mail-it0-f54.google.com (mail-it0-f54.google.com [209.85.214.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78501927F2; Mon, 30 Jul 2018 10:36:48 +0000 (UTC) (envelope-from pratiy0100@gmail.com) Received: by mail-it0-f54.google.com with SMTP id g191-v6so5481299ita.0; Mon, 30 Jul 2018 03:36:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=XvPtrARKHwUeZA6eZ9WBWv9uVfhEt64+mgVfA525ySQ=; b=c/MuVH+O+R3InNKdG0m6cdxFhBDPbwEUf9V+EfD00jR3T8ZqwfIYuE6mcajQctMjxK QhiLZDs6p8h2noivSzaoxy4hDaomVnW+Z/G3gjf9p0TpScRtcMS4dQUdMLgjqjgCJ5p9 jcpgSO44eOgXmVS9LDCTGmSEtlXfdacrPJ7Q/8Q/fyxddd78Yk1iJ59jxPKBb6tSJstn dwFy0ctGwriPK/IEFWzbN6U0lvnuRGt+mS+voTbQZ2M9wcz30y/gCZ1AKo/a2hoieJAV QGbYM/daF23lHeU+tt89CE8IZM+lyRNw5MZuePhRt/heSl7qxAvWM7QpujuxJSI4R4tY 4LyQ== X-Gm-Message-State: AOUpUlFe+2u709lRa4cT9soctQJqw9/AKJ2H0LXJ0CUJLBBdGUX6gO82 jrl4rId1We23YQXheeJ+Ozix6QH438GfeQ== X-Google-Smtp-Source: AAOMgpegQ3zvg/MgrtSFGKZk4/kQz4Bub1j7nvJpJkJECQOQhZemswXgQqWtMqJ1w+0ai7fvj+L6oA== X-Received: by 2002:a02:380c:: with SMTP id b12-v6mr15986033jaa.125.1532946596399; Mon, 30 Jul 2018 03:29:56 -0700 (PDT) Received: from mail-it0-f41.google.com (mail-it0-f41.google.com. [209.85.214.41]) by smtp.gmail.com with ESMTPSA id j11-v6sm6448849itj.6.2018.07.30.03.29.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 30 Jul 2018 03:29:56 -0700 (PDT) Received: by mail-it0-f41.google.com with SMTP id d16-v6so8923782itj.0; Mon, 30 Jul 2018 03:29:56 -0700 (PDT) X-Received: by 2002:a24:5f92:: with SMTP id r140-v6mr14227746itb.95.1532946595871; Mon, 30 Jul 2018 03:29:55 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Pratyush Yadav Date: Mon, 30 Jul 2018 15:59:19 +0530 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Recursion in non-recursive mutex when using the grant table free callbacks To: Akshay Jaggi Cc: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Edward Napierala , FreeBSD-Xen Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 10:36:49 -0000 On Mon, Jul 30, 2018 at 4:00 AM Akshay Jaggi wrote: > > Roger would be a better person to answer this and I would wait for him to= respond, but here are my 2 cents. > > First let's finalize and record the semantics of the callbacks. What guar= antee do we want to provide callbacks about locks held (no locks held OR x,= y,z locks held)? This is mostly a product decision. Usually callbacks are c= alled with no locks held. > Post this problem is simple. If the list_lock would be held during callba= cks, we need to provide a lock-free function (which assumes lock would be h= eld before function is called and use that) and if the lock would not be he= ld we should fix the locking to ensure the case you mentioned should not ha= ppen. > > WDYT? The guarantee we want to provide the callback is that the allocation of grant references can not fail inside the callback (unless they try to allocate more refs than initially specified). This means that we can't call the callback without the lock held otherwise during the execution of the callback it is possible for some other thread to get the grant references, and the grant refs the callbacks needs would not be available. So a change in the locking has to be done. On Mon, Jul 30, 2018 at 3:02 PM Roger Pau Monn=C3=A9 w= rote: > I agree in the analysis, however I think the proper solution is to use > a recursive lock. All right. I'll send in the patch. -- Regards, Pratyush Yadav