From owner-freebsd-hackers@freebsd.org Thu Jan 23 13:40:23 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F1CA81F26FD for ; Thu, 23 Jan 2020 13:40:23 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 483Ngq1r16z4f09 for ; Thu, 23 Jan 2020 13:40:22 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id 00NDeHZ9083528 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 23 Jan 2020 14:40:18 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1579786818; bh=FsFeT3rC7NufXegyyDQ3B9zSdYMGULMDBKECo9V6qxU=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=qcnUnbqgseqViE2p1nzggKsMMo0KdqagHsd6HycC42cGrro3ZIo4qGuKAv/gqum9H ZPimJVQKLN20bWARx/IQA5C2YJExJJGdL1Fdu5zPCxGY28Vz7v/mbt5RWckWHBOx0U hyYhHUb967Z2h/j06g/wfniIgtnxgn3LTIjqF9fE= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id 00NDeHl3083525; Thu, 23 Jan 2020 14:40:17 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Thu, 23 Jan 2020 14:40:17 +0100 (CET) From: Wojciech Puchar To: Miroslav Lachman <000.fbsd@quip.cz> cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: slow directory operation on huge dirs In-Reply-To: <9b426949-be87-106a-46c3-f1b6a2e5bb83@quip.cz> Message-ID: References: <9b426949-be87-106a-46c3-f1b6a2e5bb83@quip.cz> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 483Ngq1r16z4f09 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=qcnUnbqg; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-3.93 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-2.63)[ip: (-6.95), ipnet: 194.1.144.0/24(-3.47), asn: 43476(-2.78), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jan 2020 13:40:24 -0000 >> this part works fine. The problem begins where it's already cached. >> >> To get the problem you need BOTH of: >> >> - large maxvnodes like half million >> - large directory (in order of 100000 files) > > I remember very similar problem with slow access of directories with 80 000+ > files. It was always slow like frozen hell. There is some recommendation to first - we have DIRHASH option and i have it enabled. second - it is not just a problem of huge directory but both huge directory and large maxvnodes settings. Large directories works fast when kern.maxvnodes is small and there is enough dirhash_maxmem. it's clearly an error somewhere.