Date: Tue, 3 Jan 2017 19:01:36 +0100 From: Domagoj Stolfa <domagoj.stolfa@gmail.com> To: Mark Johnston <markj@FreeBSD.org> Cc: freebsd-dtrace@freebsd.org Subject: Re: RFC: Changes in DTrace to allow for distributed operation Message-ID: <20170103180136.GA5635@freebsd> In-Reply-To: <20170102220029.GC46812@wkstn-mjohnston.west.isilon.com> References: <20161230173747.GB46006@freebsd-laptop> <20170102220029.GC46812@wkstn-mjohnston.west.isilon.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--nFreZHaLTZJo0R7j Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, > I think some concrete examples of what you'd like to be able to > accomplish with this would help a lot. Is the goal here to be able to > trace multiple independent systems at once and somehow aggregate their > trace records? That's what "distributed tracing" suggests to me, but > from reading your email it seems as though you're primarily interested > in some form of remote tracing whereby one could execute something like >=20 > # dtrace -n 'fbt$guest::kern_ioctl:entry' The ultimate goal is indeed to have multiple DTrace instances cooperate and trace multiple independent systems at once. With the proposed implementatio= n, remote tracing would implicitly be implemented, as one could easily attach = to a remote machine that one has access to and trace it. The reasoning behind doing this in DTrace itself, as opposed to a daemon, is that global state could easily be kept in the scripts then. We could have a= ll types of variables that we do now and use them. In addition to that, instance-local variables could be added and allow for a concise way to repr= esent instance-wise aggregation. One could create scripts that allow for certain probes to be fired only if a certain condition is satisfied that happened on another instance. This coul= d aid in debugging interaction between certain software, such as a web server and= a database on two different machines among other things, while giving one full disclosure of the events taking place in both instances. The scripts would be written in the following way: # dtrace -n 'instance:provider:module:function:name' for a number of reasons: (1) It allows for tracing of multiple providers in an arbitrary number of different OS instances (2) With these semantics, even userland probes can be traced, for example, = once could write: # dtrace -n 'vm-[a-zA-Z0-9]:pid$target:libc:memcpy:entry' resulting in each of the userland probes in each of the instances(not necessarily virtual machines) that match glob being installed and reporting= back to the host. > on a hypervisor and get records (in real time?) from a guest. Assuming > I'm not completely off-base, this is a cool idea, but I think your > objective needs to be more clearly defined before it's possible to > evaluate the merits of different designs, especially when you're > proposing adding new concepts to the core DTrace code. In the case of a hypervisor, this approach allows for real time reporting of events using the hypercall ABI that I've implemented for that purpose[1]. T= his would also allow for destructive actions to be implemented in the case of virtual machines. This would prove to be more problematic for remote machin= es, as we don't want to wait for host's response(which could take a very long t= ime over the network) in order to do something. In such a case, the host could during initialization ask the guest DTrace to perform a certain action once a probe fires. This of course requires one to trust that DTrace instance to do the right thing. Some of these changes already exist and are being implemented in my GitHub repository[2]. I'm currently working on a PoC that allows for some of the simpler things to operate, which would allow for benchmarks and stress test= ing of the core concept. I hope that's cleared up some of the vagueness regarding the problem I am t= rying to solve. [1] https://reviews.freebsd.org/D8100 [2] https://github.com/dstolfa/freebsd/tree/dtrace-vm --=20 Best regards, Domagoj Stolfa --nFreZHaLTZJo0R7j Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEHQB+y96lmmv+IXofwxT+ikb0YU4FAlhr5v4ACgkQwxT+ikb0 YU4Wkgf/ZNu04+8CMebwWuL9vsOuFGMmGeT05YEUc5q3xAVc3SCpQO3UWlgGTILo djtjHttnRw5JMtmQyFlFT6we862iIWmZ/Ej+l1pmhbqgK2sHHdaD25d0PK4JlrH+ Ceb1IxQh0NXDgm40BdNq+wVF2YwIIa4WFnQ3ABNJjMmmGMrpcJVNJJyT9MYJPQ1X YGnm6ZcUzAHi3VNiexM6X1puG7lHYRVr0KHmjAFoaxUxuuqIYVxZWd4BMNht6h1u s9WXuXGmUjfYtMd07Mbs3GTFQ4RFNOT8Lz1DRcwQogkeqRUtQT3DEd8jE/qFdQQ2 MuUzv3+WuGxREMc2+47hGeNa7jVqiA== =M0re -----END PGP SIGNATURE----- --nFreZHaLTZJo0R7j--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170103180136.GA5635>