From owner-svn-src-head@FreeBSD.ORG Thu Sep 3 19:45:29 2009 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B7A7106566C; Thu, 3 Sep 2009 19:45:29 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-ew0-f208.google.com (mail-ew0-f208.google.com [209.85.219.208]) by mx1.freebsd.org (Postfix) with ESMTP id 3A0398FC14; Thu, 3 Sep 2009 19:45:27 +0000 (UTC) Received: by ewy4 with SMTP id 4so262839ewy.36 for ; Thu, 03 Sep 2009 12:45:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type:content-transfer-encoding; bh=igxB8U5ZDSnbf4nvW1GwYh39JxjG/iUZTdY5CNQg38w=; b=DG48w1l3CB/xu34BKgxNSlrl2OKrp4vS1U33H/ge1rRJbp4abx6D0WUL1HXuPJXBZO LFrgJx+9ZOgteFZSArIKV8FNw08yXyrmgWGreA7z30A1epyMwsizRcEu8Fie2NZNZKuH g0EWRN8cgNvAOOdcZGv6Ri5En8jpKsHVHEEjU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=RC6uixJRa+RvL09/C+8Pm0SjAm6kOIr//5GkmCXVtkaMVxWTTuohboI3x39rXTaoUC 9H1eL3ZMs6qh2VJzRK54aq+11r3nckF1/G7K/MBJOtuoRhnIV6yo4fyoOpEJbVrm0xgt Tg0rzldSzRLLZvqGoIgdFOP3SyTYb2tV0QAPQ= MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.216.7.67 with SMTP id 45mr592161weo.95.1252007127089; Thu, 03 Sep 2009 12:45:27 -0700 (PDT) In-Reply-To: <20090903114121.C20031@pooker.samsco.org> References: <200909031237.n83CbIgk032551@svn.freebsd.org> <1872D962-9297-4C45-9F73-4BB823C49D74@samsco.org> <4A9FD8B4.2080605@FreeBSD.org> <20090903095224.N20031@pooker.samsco.org> <9bbcef730909031037y4aecd692t4812718b1fd7e78e@mail.gmail.com> <20090903114121.C20031@pooker.samsco.org> From: Ivan Voras Date: Thu, 3 Sep 2009 21:45:07 +0200 X-Google-Sender-Auth: 185079f3d5a3088a Message-ID: <9bbcef730909031245o7c380925sd29b2cc976c4d7dd@mail.gmail.com> To: Scott Long Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: svn-src-head@freebsd.org, Alexander Motin , src-committers@freebsd.org, svn-src-all@freebsd.org Subject: Re: svn commit: r196777 - head/sys/dev/ahci X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Sep 2009 19:45:29 -0000 2009/9/3 Scott Long : > The problem is lack of kernel address space, not lack of RAM, but that's Yes. > just semantics in this discussion. =C2=A0I've tested with increasing MAXP= HYS in > increments to 1M. =C2=A0Performance increases logrithmically, and effecti= vely > hits a max at 512K for the variety of controllers that I tested. =C2=A0Th= e gain > from 64K to 128K is huge, the gain from 128K to 256K is ok, the gain from > 256k to 512k is measurable but less significant, and the gain from 512k t= o > 1m is almost not measurable. >From what I've seen with iostat, all machines I can currently test (SATA, various SCSI-like RAID, QLogic FC HBA), 64 k is the maximum transfer size, which I assume is DFLTPHYS. From your tests it's apparent it is suboptimal... > I have simple patches to increase MAXPHYS. =C2=A0The introduction of the = the > maxio paramter in the CAM SIM interface is there in preparation for this. > However, a _LOT_LOT_LOT_ of drivers in the tree falsely assume that MAXPH= YS > and DEFLTPHYS are 128k and 64k respectively, and size their data structur= es > accordingly. =C2=A0Changing these values will cause the drivers to fail i= n bad > ways. =C2=A0So an audit needs to be done. =C2=A0Also, MAXPHYS is abused b= y the swapper > in the struct-buf, so that needs to be reviewed as well. Since GEOM is ok with it, couldn't a migration path be to introduce a new tunable, something like LARGE_MAXPHYS (or to rename it completely to MAX_IO) or whatever and use it (after reviewing cases of abuses) in new and better maintained drivers? If the new size is larger than the old one, maybe the maximum damage would be a little bit of wasted space now and then? I can test mostly ciss and mfi. I see mfi uses it in one place that looks kind of logical: 334 /* 335 * Get information needed for sizing the contiguous memory for= the 336 * frame pool. Size down the sgl parameter since we know that 337 * we will never need more than what's required for MAXPHYS. 338 * It would be nice if these constants were available at runti= me 339 * instead of compile time. 340 */ 341 status =3D sc->mfi_read_fw_status(sc); 342 sc->mfi_max_fw_cmds =3D status & MFI_FWSTATE_MAXCMD_MASK; 343 max_fw_sge =3D (status & MFI_FWSTATE_MAXSGL_MASK) >> 16; 344 sc->mfi_max_sge =3D min(max_fw_sge, ((MAXPHYS / PAGE_SIZE) + 1= )); But ciss doesn't reference it at all so either it deviously assumes it or is independent of it. > Even though kernel address space is less restricted on 64bit platforms, i= t's > still not free and limitless. =C2=A0Large I/O's requires more work in the= VM to > assign address space, and in turn causes more lock contention. =C2=A0I ha= ven't > done any practical measurements of this on common workloads, but I can > anecdotally say that I see increased lock contention from it in locking > profiles. =C2=A0If FreeBSD wants to seriously increase MAXPHYS, this need= s to be > looked at and either proven to not be important, or fixed. I think the major benefits would be for the server-side crowd using RAID modes with some kind of striping. --=20 f+rEnSIBITAhITAhLR1nM9F4cIs5KJrhbcsVtUIt7K1MhWJy1A=3D=3D