From owner-freebsd-hackers@FreeBSD.ORG Wed Sep 29 16:03:03 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B936106564A for ; Wed, 29 Sep 2010 16:03:03 +0000 (UTC) (envelope-from crossd@cs.rpi.edu) Received: from cliffclavin.cs.rpi.edu (cliffclavin.cs.rpi.edu [128.113.126.25]) by mx1.freebsd.org (Postfix) with ESMTP id 38AF68FC17 for ; Wed, 29 Sep 2010 16:03:02 +0000 (UTC) X-Hash: SCtCte|9a8bd3d2a61412997e2c6491aa4e35c5011afd78|726e88d60504330b352482fcd659de87 X-Countries: United States X-SMTP-From: accepted static-98-113-167-42.nycmny.fios.verizon.net [98.113.167.42] (dcross2.office.okcupid.com) {United States} DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=cs.rpi.edu; h= message-id:date:from:mime-version:to:subject:content-type :content-transfer-encoding; s=default; i=crossd@cs.rpi.edu; t= 1285774885; x=1286379685; l=1587; bh=GiUWohVUMt5GXUUIAqcOMOIkHyU =; b=lTZRt453joYSVhpA2njzkzbHelgBiBFHcSG7+ldWe6ORNwNFTHhJe7SMqUR 4HMR3jyic31mIu3DWPHXzUdqtyw031+gOWkHGpCvhmouahNVejm2l/5bzx9dODCj k6sf609EB1Aztw48Ow1YZ8Uk3wOweSPbyWCz2NHrFoQt3ZP8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cs.rpi.edu; h=message-id :date:from:mime-version:to:subject:content-type :content-transfer-encoding; q=dns; s=default; b=kyZXH/vH+zbLKYi0 xlMoWbLRqwU4fil0laa0Ltbbb02ilkJ6pZx+j3gYiqdETZBWpPJ/f54t9zAD4iFK vcnRThzdKHDMVZbzZlRqfOk5KSMttHqf4CfQ/S+Suundl4T/8qk/Umzszc9oK8jY JBDWylV0d9uT1GpQMRPlHO+qcgs= X-Spam-Score: -2.6 () ALL_TRUSTED,AWL,BAYES_00 X-Spam-Report: Spam Report from cliffclavin.cs.rpi.edu (SA:3.2.5): -1.8 ALL_TRUSTED Passed through trusted hosts only via SMTP -0.9 BAYES_00 BODY: Bayesian spam probability is 0 to 1% [score: 0.0000] 0.1 AWL AWL: From: address is in the auto white-list X-Spam-Info: -2.6; ALL_TRUSTED,AWL,BAYES_00 X-Spam-Scanned-By: cliffclavin.cs.rpi.edu using SpamAssassin 3.2.5 (hard limit 15) Authentication-Results: cliffclavin.cs.rpi.edu; DKIM=neutral (none) header.from=crossd@cs.rpi.edu; SPF=neutral (mfrom; Mechanism '?all' matched) smtp.mail=crossd@cs.rpi.edu X-Auth-Passed: cliffclavin.cs.rpi.edu:o8TFfLWH034003 Auth:crossd X-Virus-Scanned-By: cliffclavin.cs.rpi.edu Received: from dcross2.office.okcupid.com (static-98-113-167-42.nycmny.fios.verizon.net [98.113.167.42]) (authenticated bits=0) by cliffclavin.cs.rpi.edu (8.14.3/8.14.3) with ESMTP id o8TFfLWH034003 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 29 Sep 2010 11:41:22 -0400 (EDT) (envelope-from crossd@cs.rpi.edu) Message-ID: <4CA35DD5.7070900@cs.rpi.edu> Date: Wed, 29 Sep 2010 11:40:05 -0400 From: David Cross User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.8) Gecko/20100329 Thunderbird/3.0.3 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on 128.113.126.25 Subject: Unnecessary reads on write load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Sep 2010 16:03:03 -0000 Tracking down a performance issue where the system apparently needlessly reads on a 100% write load... consider the following C test case: (more after the code) #include #include #include #include int main(int argc, char **argv) { unsigned char dir1[4], dir2[4], filename[15], pathname[1024], buffer[130944]; unsigned int filenum, count, filesize; int fd; arc4random_buf(buffer, 131072); count=atoi(argv[1]); for(;count>0;count--) { filenum=arc4random(); filesize=arc4random_uniform(130944) + 128; sprintf(filename, "%08x", filenum); strncpy(dir1,filename,3); strncpy(dir2,filename+3, 3); dir1[3]=dir2[3]=0; sprintf(pathname, "%s/%s/%s", dir1, dir2, filename); fd=open(pathname, O_CREAT | O_WRONLY, 0644); if (fd < 0) { sprintf(pathname, "%s/%s", dir1, dir2); if (mkdir(pathname, 0755) < 0) { mkdir(dir1, 0755); mkdir(pathname, 0755); }; sprintf(pathname, "%s/%s/%s", dir1, dir2, filename); fd=open(pathname, O_CREAT | O_WRONLY, 0644); } if (fd <0) continue; write(fd,buffer, filesize); close(fd); } return 0; } In running that in an empty directory (it takes one argument, the number of files to create), I see that it spends most of its time in BIORD?!. If I have a debugging kernel I can see that its all in NAMI cache misses, and doing block fetches... but "why?" the only directories that exist are ones that it just created, and should therefore be in the cache, right? Any ideas? Give it a try yourself and tell me what you get. -- David E. Cross