From owner-freebsd-arch@freebsd.org Mon Mar 12 13:17:52 2018 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C4796F45305; Mon, 12 Mar 2018 13:17:52 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f49.google.com (mail-lf0-f49.google.com [209.85.215.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 512517AA0C; Mon, 12 Mar 2018 13:17:52 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f49.google.com with SMTP id q69-v6so23124566lfi.10; Mon, 12 Mar 2018 06:17:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=5jBSZcD7KR50Fj82AkHGF6h336mA1tGGLVXNTXAcH5c=; b=i53biMxorStuutX2Bb08VH0JVq2ugjn4MmeNQzbtMrvT86JPrr0BsIzAzNMg9p07UQ dXZzpkrN4GoZOxJuoDDDEjeMEq92+BnQ2/4SW9pNt3+pr1h86jNmP5OKU2yCsFrnAQES knA/BuzdmthjvCgY7lOq0uTrhnAMFjPjqW/xKwwRPPE8UrgUwznZ5/eKABUEI9aE/FIm lfuYcwgKZIYehYC5+cbrnRkev+ViJmZmmpfJgluMO4TWhuwgqrRoOoqbQgwGT9WhwQLi p8rDDMuFOlT1teuWv57Wv2Otkr4yl84i1mr2P3Mt766OiDD5Y4admoBL0GfbuwVsXidF C6CQ== X-Gm-Message-State: AElRT7EGPL9QsYqZLcL26Qs8GyKw4LsufhuYc0GtRS+J+RskZirMPol+ DlMv9tdhHNzdM+30it5/WOPhKPBf X-Google-Smtp-Source: AG47ELseJmK9YZf2riyV4lybw+SdZX9VsFxeudKefuuUYpATjSKV5ixwNxlIF6Hm+RnyR3+7kKU5bA== X-Received: by 10.46.14.10 with SMTP id 10mr5377960ljo.64.1520860670389; Mon, 12 Mar 2018 06:17:50 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id v3sm1759720ljd.59.2018.03.12.06.17.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Mar 2018 06:17:49 -0700 (PDT) To: freebsd-geom@FreeBSD.org, freebsd-arch@FreeBSD.ORG From: Andriy Gapon Subject: geom->access problem and workaround Message-ID: <809d9254-ee56-59d8-69a4-08838e985cea@FreeBSD.org> Date: Mon, 12 Mar 2018 15:17:48 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Mar 2018 13:17:53 -0000 According to Poul-Henning (phk@), the principal author of GEOM, a GEOM class's access method was intended to be a light-weight operation involving mostly access counts. That is, it should be (have been) close in spirit to what g_access() function does. The method is only called from g_access and it is always done under the GEOM topology lock (like with most GEOM "control plane" methods). The lock ensures that the method and the function operate on a consistent state of the topology and all geoms in it. In reality, many classes have their access method do a lot more than just checking and modifying access bits. And often, what the method does is incompatible with the topology lock. Some examples. g_mirror_access() has to drop and reacquire the topology lock to avoid a LOR (deadlock) because the method needs to use the class's internal sc_lock. zvol_geom_access() also has to drop and reacquire the topology lock when it interacts with ZFS internals involving many locks. The main issue here is that ZFS is both above the GEOM when ZFS uses GEOM for the storage access and it is "below" the GEOM when ZFS is accessed through the ZVOL provider. g_disk_access() -> daopen(). In this case the topology lock is never dropped, but the operation issues multiple SCSI commands and waits for their completion. So, if something goes wrong and takes a long time to complete then the whole topology will be frozen for all that time. [Perhaps doing the lock dance would be a better alternative] But, of course, dropping the lock does not come free. It opens races where two (at least) sets of incompatible access counts may get granted. Or a special action, that should be done only on a first access to a geom, could be executed more than once. Bringing everything to conformance with the original design would be an ideal solution, but it will take a lot of work both in the individual nonconforming classes and in at least some of their consumers. It seems to require moving all the complex operations from access methods to the GEOM "data plane". E.g, doing those things upon the first I/O operation. Or having a new special BIO_GETATTR (kind of) operation that could be executed after g_access() but before the actual I/O is allowed. I am proposing an interim solution, so really a workaround, for the problem of dropping the topology lock: https://reviews.freebsd.org/D14533 That workaround cannot guarantee, of course, the complete stability of the topology, but it prevents concurrent calls to access methods. The idea is very simple. Before calling a geom's access method the geom is marked with a special flag unless the flag is already set in which case the code waits until the flag is cleared. The flag is cleared after the call, of course. The topology lock is released while waiting for the flag. I think that having this new flag may help to get more visibility into the problem. P.S. The workaround does not help daopen() at all. -- Andriy Gapon From owner-freebsd-arch@freebsd.org Mon Mar 12 17:11:42 2018 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A61BF2C980 for ; Mon, 12 Mar 2018 17:11:42 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22b.google.com (mail-it0-x22b.google.com [IPv6:2607:f8b0:4001:c0b::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1882D86B1A for ; Mon, 12 Mar 2018 17:11:42 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22b.google.com with SMTP id u5-v6so12450465itc.1 for ; Mon, 12 Mar 2018 10:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=NC9/9+VsdTpO39szZttPOl2mb16vBPjCeI1+teosK9g=; b=x9AH66wYYEGuk4uF9zD9nWrIo7Zx4HidSqcIlmDq33UFZRDnWtvr8rHBbcdKpYsiGV U3vBhNgvWHhp1otRpgenmhq2W23sz785a4cJKoY5OfEOeQO+NMbdfv3ZG04q4J8g0GSD nFufGqLXQEdnT7mifrvUkedFUxVxExkw9qlwYdkq6BBb5Gx9qbu6/N9j6B9K+/EOogIi GkJPUNrykKrKzi7hg7RhPgPN2ERm/T1tdBNerv49m2H3FPitknZQ6xngXMwFZ0YBTFXD OgRpzS1M5bOcyp07H2vdfjmnszpZI/ONHBw0RhEjaEiAc2C3nzx50mB5ihLtVM+seJpq binw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=NC9/9+VsdTpO39szZttPOl2mb16vBPjCeI1+teosK9g=; b=UMiyQhOLw+zzWGd7TJLtG11eQytU6qFtj5YE8gs7Kr6i1nE/ncWg2l7cCT4ZbG7wTP rXNAdGABCe4kg3bTysD0i7XWgypTowM5zfIuYWhdvoqCREnu7aXundh1aq4PiHqwC51O kDVM0rOXnxI0RhZOhIZV5VaypJuK+TJm6tHzVvaZz4FU8cXkXx9JFr0ANf6NLP8ar2Zn ZPMAgS9gSwcJnpZGagPXfii8+RfeHO/unukThiecEjla4qVjl57V3gCHgAfFobpEYNDi s4qAgs9ZvLRFhxlvaIT2I91g86JSVFivy7svwRVVY7Ir8eHVrGvp36wst6hmSrJAtwuN oP5A== X-Gm-Message-State: AElRT7Ez6TDR9b7E3NgwcMOce7COT8bgyQUAJGlYvhxbUmUj8jK1L3Zh Nc07DFQOt4MDwkyKUkSK/hH7YrAbKTRAPmwZIF7SZQ== X-Google-Smtp-Source: AG47ELvbknfpMTWBnOUrSYKNunuJ3wNZNf2om76XMuedjgAsGyCh/ndyOBD7U0ZrnE/Y1XpfsmA6XF+G793AvmIVamo= X-Received: by 10.36.16.147 with SMTP id 141mr9583195ity.73.1520874701268; Mon, 12 Mar 2018 10:11:41 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.203.196 with HTTP; Mon, 12 Mar 2018 10:11:40 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <809d9254-ee56-59d8-69a4-08838e985cea@FreeBSD.org> References: <809d9254-ee56-59d8-69a4-08838e985cea@FreeBSD.org> From: Warner Losh Date: Mon, 12 Mar 2018 11:11:40 -0600 X-Google-Sender-Auth: 8wO9USNQdf8tekqXZOBFR5dBO20 Message-ID: Subject: Re: geom->access problem and workaround To: Andriy Gapon Cc: freebsd-geom@freebsd.org, "freebsd-arch@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Mar 2018 17:11:42 -0000 On Mon, Mar 12, 2018 at 7:17 AM, Andriy Gapon wrote: > > According to Poul-Henning (phk@), the principal author of GEOM, a GEOM > class's > access method was intended to be a light-weight operation involving mostly > access counts. That is, it should be (have been) close in spirit to what > g_access() function does. The method is only called from g_access and it > is > always done under the GEOM topology lock (like with most GEOM "control > plane" > methods). The lock ensures that the method and the function operate on a > consistent state of the topology and all geoms in it. > > In reality, many classes have their access method do a lot more than just > checking and modifying access bits. And often, what the method does is > incompatible with the topology lock. > > Some examples. > g_mirror_access() has to drop and reacquire the topology lock to avoid a > LOR > (deadlock) because the method needs to use the class's internal sc_lock. > > zvol_geom_access() also has to drop and reacquire the topology lock when it > interacts with ZFS internals involving many locks. The main issue here is > that > ZFS is both above the GEOM when ZFS uses GEOM for the storage access and > it is > "below" the GEOM when ZFS is accessed through the ZVOL provider. > > g_disk_access() -> daopen(). In this case the topology lock is never > dropped, > but the operation issues multiple SCSI commands and waits for their > completion. > So, if something goes wrong and takes a long time to complete then the > whole > topology will be frozen for all that time. > [Perhaps doing the lock dance would be a better alternative] > > But, of course, dropping the lock does not come free. > It opens races where two (at least) sets of incompatible access counts may > get > granted. Or a special action, that should be done only on a first access > to a > geom, could be executed more than once. > > Bringing everything to conformance with the original design would be an > ideal > solution, but it will take a lot of work both in the individual > nonconforming > classes and in at least some of their consumers. It seems to require > moving all > the complex operations from access methods to the GEOM "data plane". E.g, > doing > those things upon the first I/O operation. Or having a new special > BIO_GETATTR > (kind of) operation that could be executed after g_access() but before the > actual I/O is allowed. > > I am proposing an interim solution, so really a workaround, for the > problem of > dropping the topology lock: > > https://reviews.freebsd.org/D14533 > > That workaround cannot guarantee, of course, the complete stability of the > topology, but it prevents concurrent calls to access methods. > The idea is very simple. Before calling a geom's access method the geom is > marked with a special flag unless the flag is already set in which case > the code > waits until the flag is cleared. The flag is cleared after the call, of > course. > The topology lock is released while waiting for the flag. > > I think that having this new flag may help to get more visibility into the > problem. > > P.S. > The workaround does not help daopen() at all. The storage layer generally doesn't expect higher-level locks around calls to it, and feels that it's free to sleep in the open routine for resources to become available. This is true across most 'open' routines (eg, tty will wait for the right signals, etc). In a world of removable media, I'm not sure that one can avoid this. But I'm not sure that calling open on the underlying device is at all compatible with the design goal of access being cheap. I think you can't have both: either you open the device, and cope with the fact that open may sleep, or it looks like you'll have broken code. Once we've updated the access counts, we can drop the topology lock to call open. If it succeeds, all is good. If it fails, then we have to reacquire it to "unaccess" the device after the failure... However, that doesn't help with the concurrent attempts to do first open for the device. g_disk_access will still have issues of sleeping indefinitely, which of course can lead to deadlock in complicated geometry situations (I say of course, but I'm not 100% sure). The whole reason that daopen may (but not always) sleep is that it may need to do I/O to the device to get it's media-status / size / SN if it' s a removable device... Just like with the RO flag, we'd want the open routine to fail if it can't reasonably access the device. Warner From owner-freebsd-arch@freebsd.org Mon Mar 12 18:07:29 2018 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61FE2F31FB7; Mon, 12 Mar 2018 18:07:29 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id E2A0A69817; Mon, 12 Mar 2018 18:07:28 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) by phk.freebsd.dk (Postfix) with ESMTP id A915327374; Mon, 12 Mar 2018 18:07:19 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.15.2/8.15.2) with ESMTPS id w2CI73BO056621 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 12 Mar 2018 18:07:03 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.15.2/8.15.2/Submit) id w2CI72Hc056620; Mon, 12 Mar 2018 18:07:02 GMT (envelope-from phk) To: Warner Losh cc: Andriy Gapon , "freebsd-arch@freebsd.org" , freebsd-geom@freebsd.org Subject: Re: geom->access problem and workaround In-reply-to: From: "Poul-Henning Kamp" References: <809d9254-ee56-59d8-69a4-08838e985cea@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <56618.1520878022.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Mon, 12 Mar 2018 18:07:02 +0000 Message-ID: <56619.1520878022@critter.freebsd.dk> X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Mar 2018 18:07:29 -0000 -------- In message , Warner Losh writ es: >The storage layer generally doesn't expect higher-level locks around call= s >to it, and feels that it's free to sleep in the open routine for resource= s >to become available. This is true across most 'open' routines (eg, tty wi= ll >wait for the right signals, etc). In a world of removable media, I'm not >sure that one can avoid this. The original intent was that we would. Things would probably have been clearer if I had called it g_reserve() instead of g_access(). Removable media state was supposed to be a job for the driver(s background polling), and the geom event queue was supposed to do what needed to be done as a result of the g_access() calls. The primary reason is that messing around with the geom topology is a global operation, in order to keep things simple[1], and we really don't want global locks held for any amount of time and certainly not for mechanical-movement / failure-retry kinds of time. The secondary reason was to be able to present a consistent and precise view of the system *without* opening devices, so that disk maintenance tools would not spin up all disks, rattle all drawers and bang all doors before telling you what you wanted to know. >But I'm not sure that calling open on the underlying device is at all >compatible with the design goal of access being cheap. I think you can't >have both: either you open the device, and cope with the fact that open m= ay >sleep, or it looks like you'll have broken code. Once we've updated the >access counts, we can drop the topology lock to call open. So this is where it gets slightly tricky: When you open /dev/foobar, do you open the media or only a drivemechanism that *may* hold a media ? For any normal "hard-disk", there is no difference. But for floppies, CDROMs, ZIP drives, WORM drives, Robots-with-ATA-disks and other interesting hardware, which were relevant when GEOM was designed, and to some extent still are, you only open the drive, and will have to find out next if it has a media in it or not. In particular CDROMs forced this design decision, because the ioctls to open & close the tray on CDROM drives operated on the media access device node, and too many ports knew about that. Compare that with a tape-changer, which has one device node for the robotic parts and another for (each of) the tape drive(s). If we want to have an architectural sound way to do slow operations before any "user-I/O" is initiated, the right way to do so is to define new BIO_OPEN and BIO_CLOSE operation, and insist via asserts than all BIO_{READ|WRITE|DELETE} are wrapped in these. BIO_GETATTR should probably not require a BIO_OPEN/BIO_CLOSE. Poul-Henning [1] The alternative would be to have different sub-trees, each of which can be locked individually, but that requires a LOT of housekeeping and class-complexity in order to find out what those sub-trees actually are. -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe = Never attribute to malice what can adequately be explained by incompetence= . From owner-freebsd-arch@freebsd.org Fri Mar 16 16:44:50 2018 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E7D0F5CD4A; Fri, 16 Mar 2018 16:44:50 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED7E268A27; Fri, 16 Mar 2018 16:44:49 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 857B12603AE; Fri, 16 Mar 2018 17:44:47 +0100 (CET) Subject: Re: [HEADS UP] - OFED/RDMA stack update To: Konstantin Belousov , Navdeep Parhar Cc: "'freebsd-infiniband@freebsd.org'" , freebsd-drivers , Meny Yossefi , "'FreeBSD-stable@FreeBSD.org'" , freebsd-arch References: <1519683699.47932.5.camel@FreeBSD.org> <20180226224311.GT94212@kib.kiev.ua> From: Hans Petter Selasky Message-ID: <3027f48e-0ba8-555d-df23-d638303cb125@selasky.org> Date: Fri, 16 Mar 2018 17:44:41 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <20180226224311.GT94212@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Mar 2018 16:44:50 -0000 Hi, The bsd_rdma_4_9_stable_11 projects branch is close to being merged into FreeBSD 11-stable. Mellanox plans to merge no later than 12:00 CEST TUE 20th of March 2018, unless objections are received. A compatibility header file has been created, ib_verbs_compat.h, which offers full source compatibility to existing OFED kernel applications, as a response to the raised conserns. User-space compatibility is maintained through library symbol versioning. https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/ofed/include/rdma/ib_verbs_compat.h An example client for this header file can be found here: https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/contrib/rdma/krping_compat/ Currently the cxgb and cxgbe i-Warp drivers are not building, and will be stubbed from the kernel build before the branch is merged, unless Chelsio can add patches for these. Here is a quick and dirty patch to make the bsd_rdma_4_9_stable_11 branch build: > diff --git a/sys/modules/Makefile b/sys/modules/Makefile > index 6b005c854d7..b918a208f21 100644 > --- a/sys/modules/Makefile > +++ b/sys/modules/Makefile > @@ -530,7 +530,7 @@ _txp= txp > .if ${MK_SOURCELESS_UCODE} != "no" && ${MACHINE_CPUARCH} != "arm" && \ > ${MACHINE_ARCH:C/mips(el)?/mips/} != "mips" && \ > ${MACHINE_ARCH} != "powerpc" && ${MACHINE_CPUARCH} != "riscv" > -_cxgbe= cxgbe > +#_cxgbe= cxgbe > .endif > > .if ${MK_TESTS} != "no" || defined(ALL_MODULES) > @@ -554,7 +554,7 @@ _vpo= vpo > _sym= sym > # intr_disable() is a macro, causes problems > .if ${MK_SOURCELESS_UCODE} != "no" > -_cxgb= cxgb > +#_cxgb= cxgb > .endif > .endif --HPS From owner-freebsd-arch@freebsd.org Sat Mar 17 19:52:36 2018 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF4D8F64340; Sat, 17 Mar 2018 19:52:35 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pl0-x243.google.com (mail-pl0-x243.google.com [IPv6:2607:f8b0:400e:c01::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5CA6A6BD12; Sat, 17 Mar 2018 19:52:35 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pl0-x243.google.com with SMTP id m22-v6so7845160pls.5; Sat, 17 Mar 2018 12:52:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-disposition:in-reply-to:user-agent; bh=WOzsTN7kJIZXhrUPU7HNhxhkX4eeYLTa2xaccMXJujQ=; b=BVZJ9zowhwLecc6zkDlWSd+Tq4NTIyuPncmeT3XTOwl+cVEdodCZcG8DywxQ2ir/qz XO0Bq6VGM5jpl9PcvzY832ZYq7rPgg+DqNbjvDPZn7Q61EJaueD8YaRDZJQXV1DZRC7g c8aOtZcSRWCNBeuNoH4q7P2BdMgfkQNrxLalyFfeJgeansfXshGJgO9ahP5BObes3pbF 5v3idzqy/sku37j1p5CvFK2WSMVtMCobMljtPmf41tgEO0j2PzCkXg0S9ylVUN2VG2w0 WzJWfbjAQR06GggY+4MQdkBjgb3fZWl7a9ETAB8TEdNqzGO8qkUDCovuXVQScvmypQ6V gCng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to:user-agent; bh=WOzsTN7kJIZXhrUPU7HNhxhkX4eeYLTa2xaccMXJujQ=; b=cFPJAmt4cLcMDW1ZOhV/vtt2goyFYfiz7klzQ2GZo0l8VGi5xYEhbrh/bfuJ62xLJx JQkP6lumPIZ3o8Nmce4qP4oQb6BVXyCF+eNdiUcNLp6ZdRFRDBn0Ka5cwPWS0RNvx2dk MjnNXYXfEaDQamfzOwnbV4AUySBkH4fHBldBSSFbrhDitBfwruD1PDvFHO56jyYXcmYq 0lMzD/Bw+I5oDmHNeL1rYH6IagktqcfIi+HsAnBdbsBTFTkpezWPeYJbrMqoMi0aaIps pdHCK92QAfRyAOVsxvamY5Q79GuP97CRZWa6E+Uyg7LGkJJM87oNzD+e+d9v/ckyYlt2 mNfg== X-Gm-Message-State: AElRT7EWWf/w6LkpXZ+Hn4smSPrS6+4XaSs/tgQ9+Kdf2kqlL/ikLGaY EYD1kceNcfqef1/AjwR9908bfvNT X-Google-Smtp-Source: AG47ELs97s0plY6rCVbH/YAcdfo6b7Ua0WSTdAV0Skj5tzbDeuc1GwaXEoE4CENw1CdI/yo/MILccQ== X-Received: by 2002:a17:902:108a:: with SMTP id c10-v6mr1466917pla.22.1521316354379; Sat, 17 Mar 2018 12:52:34 -0700 (PDT) Received: from ox ([2601:641:c000:b800:c55e:d964:7a77:a05]) by smtp.gmail.com with ESMTPSA id v6sm20058516pfm.2.2018.03.17.12.52.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 17 Mar 2018 12:52:33 -0700 (PDT) Sender: Navdeep Parhar Date: Sat, 17 Mar 2018 12:52:16 -0700 From: Navdeep Parhar To: Hans Petter Selasky Cc: Konstantin Belousov , "'freebsd-infiniband@freebsd.org'" , freebsd-drivers , Meny Yossefi , "'FreeBSD-stable@FreeBSD.org'" , freebsd-arch Subject: Re: [HEADS UP] - OFED/RDMA stack update Message-ID: <20180317195200.GA5223@ox> Mail-Followup-To: Hans Petter Selasky , Konstantin Belousov , "'freebsd-infiniband@freebsd.org'" , freebsd-drivers , Meny Yossefi , "'FreeBSD-stable@FreeBSD.org'" , freebsd-arch References: <1519683699.47932.5.camel@FreeBSD.org> <20180226224311.GT94212@kib.kiev.ua> <3027f48e-0ba8-555d-df23-d638303cb125@selasky.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3027f48e-0ba8-555d-df23-d638303cb125@selasky.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Mar 2018 19:52:36 -0000 Hold your horses. Do you have confirmation from the affected party that the shims are adequate for them? I have been waiting for that before looking at this branch. Is the iw_cxgbe breakage a simple merge conflict as previously discussed or do the shims require driver changes? If they don't then it should be possible to drop the iw_cxgbe from head into this branch and it should just work, is that correct? Regards, Navdeep On Fri, Mar 16, 2018 at 05:44:41PM +0100, Hans Petter Selasky wrote: > Hi, > > The bsd_rdma_4_9_stable_11 projects branch is close to being merged into > FreeBSD 11-stable. Mellanox plans to merge no later than 12:00 CEST TUE 20th > of March 2018, unless objections are received. > > A compatibility header file has been created, ib_verbs_compat.h, which > offers full source compatibility to existing OFED kernel applications, as a > response to the raised conserns. User-space compatibility is maintained > through library symbol versioning. > > https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/ofed/include/rdma/ib_verbs_compat.h > > An example client for this header file can be found here: > > https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/contrib/rdma/krping_compat/ > > Currently the cxgb and cxgbe i-Warp drivers are not building, and will be > stubbed from the kernel build before the branch is merged, unless Chelsio > can add patches for these. > > Here is a quick and dirty patch to make the bsd_rdma_4_9_stable_11 branch > build: > > >diff --git a/sys/modules/Makefile b/sys/modules/Makefile > >index 6b005c854d7..b918a208f21 100644 > >--- a/sys/modules/Makefile > >+++ b/sys/modules/Makefile > >@@ -530,7 +530,7 @@ _txp= txp > > .if ${MK_SOURCELESS_UCODE} != "no" && ${MACHINE_CPUARCH} != "arm" && \ > > ${MACHINE_ARCH:C/mips(el)?/mips/} != "mips" && \ > > ${MACHINE_ARCH} != "powerpc" && ${MACHINE_CPUARCH} != "riscv" > >-_cxgbe= cxgbe > >+#_cxgbe= cxgbe > > .endif > > .if ${MK_TESTS} != "no" || defined(ALL_MODULES) > >@@ -554,7 +554,7 @@ _vpo= vpo > > _sym= sym > > # intr_disable() is a macro, causes problems > > .if ${MK_SOURCELESS_UCODE} != "no" > >-_cxgb= cxgb > >+#_cxgb= cxgb > > .endif > > .endif > > --HPS From owner-freebsd-arch@freebsd.org Sat Mar 17 20:03:48 2018 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 036A2F64F74; Sat, 17 Mar 2018 20:03:48 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8E9376C3DB; Sat, 17 Mar 2018 20:03:47 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 98064260370; Sat, 17 Mar 2018 21:03:45 +0100 (CET) Subject: Re: [HEADS UP] - OFED/RDMA stack update To: Konstantin Belousov , "'freebsd-infiniband@freebsd.org'" , freebsd-drivers , Meny Yossefi , "'FreeBSD-stable@FreeBSD.org'" , freebsd-arch References: <1519683699.47932.5.camel@FreeBSD.org> <20180226224311.GT94212@kib.kiev.ua> <3027f48e-0ba8-555d-df23-d638303cb125@selasky.org> <20180317195200.GA5223@ox> From: Hans Petter Selasky Message-ID: <6d451a3b-a635-08b1-f8d4-52fdc48083d6@selasky.org> Date: Sat, 17 Mar 2018 21:03:40 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <20180317195200.GA5223@ox> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Mar 2018 20:03:48 -0000 On 03/17/18 20:52, Navdeep Parhar wrote: > Hold your horses. Do you have confirmation from the affected party that > the shims are adequate for them? I have been waiting for that before > looking at this branch. Hi Navdeep, Mellanox has received an API list from at least one party, and has taken the action to support all the required APIs. > Is the iw_cxgbe breakage a simple merge conflict as previously discussed > or do the shims require driver changes? It is a merge conflict. The code already compiles in 12-current. > If they don't then it should be > possible to drop the iw_cxgbe from head into this branch and it should > just work, is that correct? Yes, it shouldn't be too hard to do and I would appreciate if Chelsio could throw some resources at this ASAP. I believe the new code will benefit everyone using Infiniband/RoCE and iWarp with FreeBSD. --HPS