From owner-freebsd-arch@FreeBSD.ORG Sun Jul 3 19:54:08 2005 Return-Path: X-Original-To: arch@freebsd.org Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9E65016A41C for ; Sun, 3 Jul 2005 19:54:08 +0000 (GMT) (envelope-from peadar.edwards@gmail.com) Received: from zproxy.gmail.com (zproxy.gmail.com [64.233.162.192]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3F83A43D48 for ; Sun, 3 Jul 2005 19:54:08 +0000 (GMT) (envelope-from peadar.edwards@gmail.com) Received: by zproxy.gmail.com with SMTP id 8so310781nzo for ; Sun, 03 Jul 2005 12:54:07 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=EWgJa4LXLXxWQfrIBalvZvRQ8Bc3S6yVmSZI4vL7we7zyI5ewPsQcbxjiwvuMMumW4Zds6CvcIQTfGQsJjuZ1ZEgy6q5KitjcmHNCjf/YBncWfge0LVLzfsamJAI76+fcfj3LBxAZSk7Y0/rZZJPxi2tdi4OQVopI76tjO3crZk= Received: by 10.36.65.7 with SMTP id n7mr469268nza; Sun, 03 Jul 2005 11:13:32 -0700 (PDT) Received: by 10.36.68.15 with HTTP; Sun, 3 Jul 2005 11:13:32 -0700 (PDT) Message-ID: <34cb7c8405070311131cd1ca8a@mail.gmail.com> Date: Sun, 3 Jul 2005 19:13:32 +0100 From: Peter Edwards To: Robert Watson In-Reply-To: <20050701155757.A36905@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <20050701132104.GA95135@freefall.freebsd.org> <20050701155757.A36905@fledge.watson.org> X-Mailman-Approved-At: Mon, 04 Jul 2005 12:50:00 +0000 Cc: Peter Edwards , arch@freebsd.org Subject: Re: ktrace and KTR_DROP X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Peter Edwards List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Jul 2005 19:54:08 -0000 On 7/1/05, Robert Watson wrote: >=20 > There are two benefits to the current ktrace dispatch model: >=20 > (1) Avoiding untimely sleeping in the execution paths of threads that are > being traced. >=20 > (2) Allowing the traced thread to run ahead asynchronously, hopefully > impacting performance less. >=20 Agreed, as long as the rate it needs to trace at doesn't exceed the rate th= e trace can be recorded at: otherwise, it needs to be flow controlled, either= by calling vn_write, or by waiting for another thread to do so. Just to re-ite= rate, I understand that not slowing down the target process is probably beneficia= l in some circumstances, but you can only do that by loosing trace information, = and in the vast majority of cases that's exactly what you don't want. > One of the things I've been thinking for a few years is that I think I > actually preferred the old model better, there processes (now threads) > would hang a "current record" off of their process (now thread) structure= , > and fill it in as they went along. The upsides of this have to do with > the downsides of the current model: that you don't allow fully > asynchronous execition of the threads with respect to queueing the record= s > to disk, so you don't run into "drop" scenarios, instead slowing down the > process. Likewise, the downsides. >=20 > In the audit code, we pull from a common record queue, but we allocate th= e > record when the system call starts for each process -- if there aren't > records available (or various other reliability-related conditions fail, > such as adequate disk space), we stall the thread entering the kernel > until we can satisfy its record allocation requirements. I don't follow: looking at the code from RELENG_4, it looks pretty similar = to what's there already: create a requst, call ktrwrite, free request. By add= ing the flow control with the patch, you just end up waiting for the free queue= to be non-empty, rather than waiting to aquire the vnode lock and do the write= . Am I missing something? Do I need to go back further in time? >=20 > There are two cases where I really run into problems with the current > model: >=20 > (1) When I'm interacting with a slow file system, such as NFS over > 100mbps, I will always lose records, because it doesn't take long fo= r > the process to get quite ahead of the write-behind. It doesn't even need NFS: syscall throughput is much better than I/O throughput :-) >=20 > (2) When I trace more than one process at a time, the volume of records > overwhelms the write-behind. As long as you need to do physical writes for the syscalls, I don't know how you'l ever avoid that... >=20 > Write coalescing/etc is already provided "for free" by pushing the writes > down into the file system, so other than slowing down the traced process = a > little, I think we don't lose much by moving back to this model. And if > we pre-commit the record storage on system call entry (with the exception > of paths, which generally require potential sleeps anyway), we probably > won't hurt performance all that much, and avoid sleeping in bad places. I'm not sure I follow: are you saying you'd rather move the ktrace resource aquisition out of where the sleep will be more critical? I don't see how you can avoid having the ktrace'd thread wait for as long as it takes for the writer thread to catch up with the system to some point