From owner-freebsd-questions@FreeBSD.ORG Thu Mar 26 18:17:24 2015 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 00072575 for ; Thu, 26 Mar 2015 18:17:23 +0000 (UTC) Received: from fallback1.mail.ru (fallback1.mail.ru [94.100.181.184]) by mx1.freebsd.org (Postfix) with ESMTP id 358E6262 for ; Thu, 26 Mar 2015 18:17:22 +0000 (UTC) Received: from smtp26.mail.ru (smtp26.mail.ru [94.100.181.181]) by fallback1.mail.ru (mPOP.Fallback_MX) with ESMTP id 7B8ED542311E for ; Thu, 26 Mar 2015 20:46:26 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:Subject:To:MIME-Version:From:Date:Message-ID; bh=nrdOcr+Vd28G7WT5K/IlMf0deQKBj6L5TtEHv/bb22E=; b=kIn7YsIfi35rvLjPjdI4yF9dzoQPjN2tVFWLLzGEFc44+uzru8wYtOFD5zs6QNOyYYPlrMDWU8H79+f4BwOhIrguidJdgptqn1d3j5x2rJsGO9ZwYNRClW3lahCh5WWIT6rU1nCuO2sO1mQFPCf2T8GwB/JIn6Ug0pKOz/VW3Vg=; Received: from [109.188.127.13] (port=52239 helo=[192.168.0.12]) by smtp26.mail.ru with esmtpa (envelope-from ) id 1YbBrF-00048r-Im for freebsd-questions@freebsd.org; Thu, 26 Mar 2015 20:46:17 +0300 Message-ID: <55144606.7000003@artem.ru> Date: Thu, 26 Mar 2015 20:46:46 +0300 From: Artem Kuchin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-questions@freebsd.org Subject: rm -rf stalls server Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mras: Ok X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2015 18:17:24 -0000 Hello! I have a server with 100+ sites running nginx, apache,mysql The disk system is 2 TOSHIBA 3TB disk in geom mirror FS is UFS SU+J tunefs -p / tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) enabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L) root Ram is 32GB load averages: 1.83, 1.77, 1.83 usually about 500 processes are started Everything run nice and smooth. Periodically i need to run fs intensive tasks, like tar for backup. That kills server after 5 minutes of running. I worked it around using pv limiting bandwidth to 10M. But, sometimes i need to delete huge amount of small files in different dirs. Yes, i just need to do it. So, i do rm -rf tree_root After i do it in 5 minutes process count goes over 1000, mysql is flooded with request it cannot complete. I can barely type anything. Top shown tons of processes in 'ufs' status. So, I ctrl-C rm command and kill -9 apache, otherwise it will take like 20 minutes to resume normal operations. Then i do /usr/bin/nice -n 20 rm -rf tree_root And it seems like rm -rf now is playing nice :) But, if load for web sites rises even for about 30% during this command server goes to stall again, 1000 process, hundereds of stuck mysql requests, tons of processes in ufs state. As i understand, freebsd does not have ionice So, any idea how to make this rm -rf work reaaaaaaly slow, not creating any load on hdd or metadata locks (i think it is actually fs meta data locks fault). Artem