From owner-freebsd-hackers@freebsd.org Mon Aug 20 19:25:42 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51AC9107A18B for ; Mon, 20 Aug 2018 19:25:42 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it0-f48.google.com (mail-it0-f48.google.com [209.85.214.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1E837D792 for ; Mon, 20 Aug 2018 19:25:41 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it0-f48.google.com with SMTP id p81-v6so953643itp.1 for ; Mon, 20 Aug 2018 12:25:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=JcmqHHUvjOnEaywohfzsNz2hytgnwR3n3FqGMSitU2E=; b=QM3JQYFuppBHVmfba3mRjWBxfmmVgr73MxdUnyTUJaG/6RjuXL6neBiOzOlpnpMwke x9G34k287YPVOdaMpA3V8tFow3jtR3CoOpoW3G5C2TQ6oHglOOc04fvJ07RVcgDgNXCt imY4D5B98qWvxlpIYjudZawOjOS59FMSYU8MlY09VKYboF8C5YTNMXqwuNkExUsQqpvj 7HCRf0scg088sR4Dg545d79DAff/b53MQ/BPINCt5UmbOOIXICjfeqQ7TnxTfwzuSn0l DPnPY4ieAsfAypfkNoHX9Lrz6AyeBTYG+hPKNCKj48Y+lZ4S9owSfEitOQIGXSk2/80e csHA== X-Gm-Message-State: AOUpUlEqfJT1IYnl6w2JqMam7AAY0cOi6N1SMz4MuPnfpMKIv0Ib+tcQ qcTGxKa+9DCdSWOaqnXf396o5Y/S X-Google-Smtp-Source: AA+uWPxboQYpD7mLsSsiMM5CJUNxXAnTpFwdRkmS5QJW1FB7u0PGL7kqHpx7HyqgdvohD3tXcJ0GLA== X-Received: by 2002:a24:e408:: with SMTP id o8-v6mr14843962ith.77.1534791449972; Mon, 20 Aug 2018 11:57:29 -0700 (PDT) Received: from mail-it0-f49.google.com (mail-it0-f49.google.com. [209.85.214.49]) by smtp.gmail.com with ESMTPSA id p13-v6sm286721itp.3.2018.08.20.11.57.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Aug 2018 11:57:29 -0700 (PDT) Received: by mail-it0-f49.google.com with SMTP id 139-v6so839940itf.0 for ; Mon, 20 Aug 2018 11:57:29 -0700 (PDT) X-Received: by 2002:a24:144:: with SMTP id 65-v6mr13220367itk.62.1534791449552; Mon, 20 Aug 2018 11:57:29 -0700 (PDT) MIME-Version: 1.0 Reply-To: cem@freebsd.org Received: by 2002:a02:b472:0:0:0:0:0 with HTTP; Mon, 20 Aug 2018 11:57:29 -0700 (PDT) In-Reply-To: References: From: Conrad Meyer Date: Mon, 20 Aug 2018 11:57:29 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: rand_harvestq high cpu usage when /dev/urandom is used To: Ali Abdallah Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 19:25:42 -0000 Hi Ali, On Mon, Aug 20, 2018 at 4:27 AM, Ali Abdallah wrote: > Hello, > > I was just sorting randomly some jpg image files using: > > ls *.jpg | sort -R --random-source=/dev/urandom > > The above command never exited. Later I noticed that > one of my CPU is always running 100%. top -S reveals that it is > rand_harvestq kernel service. > > Is this is a bug? This occurs on 12-ALPHA1 and 11.2 There is probably at least one bug in sort(1). sort has special behavior if the --random-source matches its default (/dev/random), but otherwise doesn't understand device files or pipes very well. Since urandom isn't exactly the same path as /dev/random, sort fails pretty hard. sort attempts to seed its internal RNG by MD5ing the provided random source path. For its default path, /dev/random, it grabs at most MAX_DEFAULT_RANDOM_SEED_DATA_SIZE (or 1024) bytes. This is hugely excessive and MD5ing it is totally unnecessary, but still mostly harmless. For non-default files, it just passes the path to MD5File, which will read() until EOF. Since /dev/urandom will never return EOF, sort --random-source=/dev/urandom will get stuck in MD5File forever. This is totally stupid. I'm not sure why rand_harvestq would take 100% of a CPU core even as a result of excessive consumption of /dev/urandom, but it is certainly possible that sort(1) is consuming 100% of a CPU core reading from urandom and MD5ing the result. All the best, Conrad