From owner-freebsd-arch@FreeBSD.ORG Mon Sep 1 03:10:53 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF5BEA2B; Mon, 1 Sep 2014 03:10:53 +0000 (UTC) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C04A61E5D; Mon, 1 Sep 2014 03:10:52 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.9/8.14.9) with ESMTP id s813AnQ9089758 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 1 Sep 2014 07:10:49 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.9/8.14.9/Submit) id s813AnHP089757; Mon, 1 Sep 2014 07:10:49 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Mon, 1 Sep 2014 07:10:49 +0400 From: Gleb Smirnoff To: Alfred Perlstein Subject: Re: [CFT/review] new sendfile(2) Message-ID: <20140901031049.GF7693@glebius.int.ru> References: <20140529102054.GX50679@FreeBSD.org> <20140729232404.GF43962@funkthat.com> <20140831165022.GE7693@FreeBSD.org> <540382E2.3040004@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <540382E2.3040004@freebsd.org> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Sep 2014 03:10:53 -0000 Alfred, On Sun, Aug 31, 2014 at 01:17:38PM -0700, Alfred Perlstein wrote: A> > J> > 1) Split of socket buffer sb_cc field into sb_acc and sb_ccc. Where A> > J> > sb_acc stands for "available character count" and sb_ccc is "claimed A> > J> > character count". This allows us to write a data to a socket, that is A> > J> > not ready yet. The data sits in the socket, consumes its space, and A> > J> > keeps itself in the right order with earlier or later writes to socket. A> > J> > But it can be send only after it is marked as ready. This change is A> > J> > split across many files. A> > J> A> > J> This change really should be split out and possibly committed seperately A> > J> after a review by the proper people... A> > A> > Of course. It actually makes 80% of the volume of the patch. A> A> This change has high value, although it has a lot of changes for what A> appears to be an interesting edge case. A> A> As I read this it really confused me, can't this be accomplished by A> utilizing the socket's callback and pointer parameter instead? A> A> Basically you would put all that accounting inside a struct hung off of A> so->sb_snd.sb_upcallarg and set a callback to do your queuing. A> A> That is how you can async drive thread to queue more data, in fact by A> using aio to read/write to the socket from a stream. A> A> It should be relatively simple, the only tricky part being that you'll A> need to watch your locks and sleeps inside the so->sb_snd.sb_upcall A> function. A> A> Basically move the sb_acc and all of that into a special struct hung off A> of so->sb_snd.sb_upcallarg and leverage so->sb_snd.sb_upcall to queue A> more data as space becomes available. A> A> At least that's how I would have tried to accomplish this... but maybe A> you went down this path and hit a non-starter? AFAIU your proposal, you suggest to go even harder and make sendfile(2) not blocking on socket buffer being full. So that we can do sendfile(fd, s, 0, 1 Gb, ...) and return immediately. And kernel would bounce between disk and socket buffer until the end. This is another step towards Poul-Henning idea, that is in near email. No, I am not there yet :) My sendfile(2) simply doesn't block on the disk I/O. If the socket buffer is full it will either block or return EAGAIN, if it is SS_NBIO. The change to socket buffers in the patch is quite primitive. It touches so many files just because it substitutes bare access to sockbuf field with an inline. -- Totus tuus, Glebius.