From owner-freebsd-arch@FreeBSD.ORG Thu Mar 19 10:12:00 2009 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65EFD1065672; Thu, 19 Mar 2009 10:12:00 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206045082.chello.pl [87.206.45.82]) by mx1.freebsd.org (Postfix) with ESMTP id C56DB8FC24; Thu, 19 Mar 2009 10:11:59 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id BC16D45C98; Thu, 19 Mar 2009 10:44:30 +0100 (CET) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 4A34D45B26; Thu, 19 Mar 2009 10:44:25 +0100 (CET) Date: Thu, 19 Mar 2009 10:45:05 +0100 From: Pawel Jakub Dawidek To: Luigi Rizzo Message-ID: <20090319094505.GA1539@garage.freebsd.pl> References: <20090319081936.GA32750@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Dxnq1zWXvFF0Q93v" Content-Disposition: inline In-Reply-To: <20090319081936.GA32750@onelab2.iet.unipi.it> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 8.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: fabio@gandalf.sssup.it, geom@freebsd.org, arch@freebsd.org, ivan@freebsd.org Subject: Re: RFC: adding 'proxy' nodes to provider ports (with patch) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Mar 2009 10:12:00 -0000 --Dxnq1zWXvFF0Q93v Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 19, 2009 at 09:19:36AM +0100, Luigi Rizzo wrote: > Hi, >=20 > Fabio Checconi and I have been thinking on how to implement "proxy" > geom nodes, i.e. nodes that have exactly 1 provider and 1 consumer > port, do not do any data transformation, and can be transparently > inserted or removed on top of a provider port while the tree is > actively moving data. >=20 > Our immediate need was to add/remove a scheduler while a disk is > mounted, but there are possibly other uses e.g. if one wants to > "sniff" the traffic through a disk, or do other ops that are > transparent for the data stream. >=20 > We would like opinion on the following solution, which seems > extremely simple in terms of implementation. >=20 > The idea is to intercept requests coming on a provider port, pp, and > redirect them to a geom node acting as a proxy if the port > is configured in this way: >=20 > +=3D=3D=3D=3D=3D...=3D=3D=3D...=3D=3D=3D=3D=3D=3D+ > H H > H H > H H > +=3D=3D=3D=3D=3D...=3D=3D=3D=3D=3D=3D cp =3D=3D+ > | +---------------+ > V | V > +=3D=3D=3D=3D=3D.....=3D=3D=3D=3D pp =3D=3D+ | +=3D=3D=3D=3D= =3D=3D=3D proxy_pp =3D=3D+ > H 'ad0s1' H | H H > H ------->--+ H H > H gp -------<--+ H proxy_node H > H H | H H > +=3D=3D=3D=3D=3D=3D=3D....=3D=3D=3D...=3D=3D=3D+ | +=3D=3D=3D= =3D=3D=3D=3D proxy_cp =3D=3D+ > | V > +---------------+ >=20 > Normally the proxy does not exist, and the geom tree works as it does now. >=20 > When we create a 'proxy' node, with something like >=20 > geom my_proxy_class proxy ad0s1 >=20 > we do something very similar to a 'create', but: >=20 > - the proxy node is marked specially in gp->flags, so the core will > not generate a g_new_provider_event when the provider port is created > (this means there is no taste() call and nobody should be able > to attach to the port). >=20 > - the provider port we attach to is linked, with two pointers, > to the provider and consumer ports of the proxy_node. >=20 > In this situation, g_io_request() finds that port pp has a proxy attached > to it, and immediately redirects the requests to the proxy, which > does everything a geom node does (cloning requests, etc). > When the proxy wants to pass the request down, it sends it again to pp, > but now there is no redirection because the source can be identified > as the proxy. The pointers in the bio insure a correct flow of the > requests on the reverse path. The one advantage I see for this over using regular GEOM rules is that new consumers go through proxy automatically. When I was working on similar functionality I more wanted to do something like this: consumer1 consumer2 \ / \ / provider Insert the proxy in the middle of any provider-consumer pair: consumer1 consumer2 | | proxy_provider | | / proxy_consumer / \ / provider This can be done (almost I think) atomically: /* First attach to the destination provider. */ g_attach(proxy_consumer, provider); /* Then switch original consumer to use proxy_provider (should be almost at= omic). */ consumer1->provider =3D proxy_provider; /* handle access counts */ In-flight I/O requests know how to go back, because they have source and destination stored in bio_from and bio_to fields, so no races here. > Disconnecting a proxy is almost trivial: apart from handling possible > races on the data path, we just need to clear pp->proxy_pp and pp->proxy_= cp. > After that, we can just send the regular destroy events to the proxy > node, who will have to take care of flushing any pending bio's (e.g. > see our geom_sched node that already does this). >=20 > Overall the change is very small (see attached patch): > a couple of lines in g_io_request, two extra fields in the g_provider, > and the addition of a flag to gp->flags to control the generation > of g_new_provider_event. > There is basically no overhead on regular operation, and only > a couple of extra pointers in struct g_provider (we use a spare > bit in gp->flags to mark G_GEOM_PROXY nodes). >=20 > The only things missing in the patch should be: >=20 > - a check to avoid races on creation&destruction of a proxy. > I am not so sure on how to achieve this, but creation and destruction > are rare and can normally wait, so we could just piggyback the > small critical section (manipulating pp->proxy_cp and pp->proxy_cp) > into some other piece of code that is guaranteed to be race-free. >=20 > - a check to prevent attaching to a provider port of a proxy > (not a problem, i believe); >=20 > - a check to prevent attaching a proxy to a provider port that already > has one. Of course you can attach a proxy to another proxy, and > if you want to change the order it is as simple as removing the > existing proxy and reattaching it after the new one. Could you provide link for the patch, as it was removed from your e-mail? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Dxnq1zWXvFF0Q93v Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFJwhQhForvXbEpPzQRAqtkAJ9zME7wMe4RZMsBYdAURm/9voIr/QCgwAim NK2iPFV6J/jPGqpLNhH3Cl8= =Fzxm -----END PGP SIGNATURE----- --Dxnq1zWXvFF0Q93v--