From owner-freebsd-hackers@freebsd.org Thu Jul 2 07:40:56 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2C92634C1B3 for ; Thu, 2 Jul 2020 07:40:56 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49y94l05ksz4Rjc; Thu, 2 Jul 2020 07:40:54 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f169.google.com with SMTP id h22so23387879lji.9; Thu, 02 Jul 2020 00:40:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=6aSiaPdRXICB2YqnHiQGRrsesVRRv23djAu4zL/CCqA=; b=U5gKQRGvANN9lnBq4kCcDcCffRCiuVtosx1tOHXK+LppXwg0y40pY4yUEXfMm/OCLh c9kTdOdQEP81ZZ4Pa30jXOjYykXylHQ+rD2JgUj3i2CdH+E7mWgETg/qrRyq/mgikfW6 ecLFT3zCawpdSy+XHVv8GjrzqRxvSEpdTKvVMom06RXbuPnpRcZ1RmVjggRVV4jwdjCL pAzqCVSU+YmwH+jFbIPQy4kygYPXFv3GNc4ylc4VtEvLxU6kcQ/XHpSvF8psDO792v1a s74AuNQefiT+COUP/QQyye2jpa00nvnJIwnbFilocskDGV17PV/vjg6NoXIMVjx0SVIf Dkxw== X-Gm-Message-State: AOAM531LiyEeJA43n399H2TTJ6tO3m7PUHW1yqK1Cu1ZlL2BUdnOW7aQ VtKvVGAjhGMX/WnexrbdIEpY2687swM= X-Google-Smtp-Source: ABdhPJxuQ5MzUclE0+CMSb4FMuTp0kUgAlCQf8f3/A6ATWmwHJFWdYKAsLjLq8JmkQvWPIODGIVUYA== X-Received: by 2002:a2e:730c:: with SMTP id o12mr9744697ljc.165.1593675652818; Thu, 02 Jul 2020 00:40:52 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id a8sm2669528ljk.138.2020.07.02.00.40.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 02 Jul 2020 00:40:52 -0700 (PDT) Subject: Re: Constant load of 1 on a recent 12-STABLE To: Gordon Bergling , Allan Jude Cc: freebsd-hackers@freebsd.org References: <20200603101607.GA80381@lion.0xfce3.net> <20200603202929.GA65032@lion.0xfce3.net> <8b1498ea-e343-506e-79c7-c25b594808f0@freebsd.org> <20200604123720.GA63595@lion.0xfce3.net> <20200623072332.GA54653@lion.0xfce3.net> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <0a10a722-3eec-456b-d771-97a49af89ad1@FreeBSD.org> Date: Thu, 2 Jul 2020 10:40:50 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <20200623072332.GA54653@lion.0xfce3.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49y94l05ksz4Rjc X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-1.83 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[FreeBSD.org]; MID_RHS_MATCH_FROM(0.00)[]; NEURAL_HAM_LONG(-0.91)[-0.906]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-0.63)[-0.630]; NEURAL_HAM_SHORT(-0.30)[-0.295]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.169:from]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.169:from]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[93.72.151.96:received] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jul 2020 07:40:56 -0000 On 23/06/2020 10:23, Gordon Bergling wrote: > What is maybe also related to the problem is the following top output, > collected by 'top -HS'. The three zfskern threads generate little load, but that > constant. The annoying part of the problem is, that the load of 1 is reported and > that the hostsystem schedules the thread over all 4 physical cores and keeps the > clockrate at the highest frequency. > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 155 ki31 0B 64K CPU2 2 17:28 97.39% idle{idle: cpu2} > 11 root 155 ki31 0B 64K CPU3 3 17:29 96.78% idle{idle: cpu3} > 11 root 155 ki31 0B 64K CPU1 1 17:29 96.40% idle{idle: cpu1} > 11 root 155 ki31 0B 64K RUN 0 17:25 96.13% idle{idle: cpu0} > 8 root -8 - 0B 1040K mmp->m 2 0:44 4.32% zfskern{mmp_thread_enter} > 8 root -8 - 0B 1040K mmp->m 1 0:44 4.28% zfskern{mmp_thread_enter} > 8 root -8 - 0B 1040K mmp->m 3 0:44 4.25% zfskern{mmp_thread_enter} > > Does anyone has some insight where zfskern is spawned? I maybe can bisect then the resulting change > which leads to this behaviour. That's strange. Those mmp threads shouldn't really be consuming much CPU. Normally they should just sleep all the time. Do you use the new ZFS multi-modifier protection feature? I guess not. Do you have 'multihost' property set on any of your pools? The change you were looking for is r361383 MFC r354804,r354806,r354807,r354885: 10499 Multi-modifier protection (MMP) -- Andriy Gapon