From owner-freebsd-arm@freebsd.org Sun Apr 19 16:56:52 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A67FA27AE2B for ; Sun, 19 Apr 2020 16:56:52 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2k.ore.mailhop.org (outbound2k.ore.mailhop.org [54.148.219.64]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 494wwN1dXkz4ZxC for ; Sun, 19 Apr 2020 16:56:51 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1587315410; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=RRiWNqxz8Lj4JMj/Svmz9VV5lxEoh05WU9/JNDgjKG/QWi4QkV/MTpTUf8ylpc2s7UEPTgbH31bIw jzwhaCxH4DnRLEqL394VWBisjR+ZCJe3JVINzv3mZl3fKZuGsWHUbaS/QbLE+bFLrUeEsHD7dbbxel NAKSlwmeiP4dCntW3HfT6Utvq/dt39memyIqIgBuDXJo8MWLD1608056mQ6yG+RbYdhpnIckCW5JWM d+yYp8pMBOig/F6O4IrhX46Ropo5XdMuQ3fxCCZEPJWWDkYNI11aQaFel0MJmyOz1TRzVuOwjbQ4Ht LY6swZjomDrT9zN1u934/QnF9dohKEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:dkim-signature:from; bh=aYZjg8fR66Ebsvs+mwq6nelt1fIPSD0t8i5y5RZjJsk=; b=ltrVd4mdMZd5JkJ6ZPmJJn3yS/MJGVH/ex59s1sUR5JkzuKpJ+LBRTkmP3OEHBk4SVvpbMgZCr6xB QTJ/APo6BXrCuF8r9Qw7mppMPDb60ZTDydruaqGOnI4ilOWHtzWvkuXrF0I1pbVBp7oHI7Gi7T6f+K cyOv0SFU+5UjLwEemUll+ZDVjRH20HsFR5JdNadwpLlaNhmlRIApWRpLOVZeEfpB+98mh6CvAORMmY HzWryqzimTOrG0cErHKSiYyMaIdLyRpAVN4jSW3WaIyaHTKhlzTeN+6bfLmNGtR1vqHQpQ97wOAgNM mgu7K0vFtsoMCcsfF0Y4pex2t/lDWxw== ARC-Authentication-Results: i=1; outbound4.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:from; bh=aYZjg8fR66Ebsvs+mwq6nelt1fIPSD0t8i5y5RZjJsk=; b=q5aaZaMvaY8PyGki0OU9+SwND1V2F4s7kheLaZyN++MxZSZolmEF0uHatwPXmJkQo+kCaIs9zGPhL 3Y7pMd5pu53OU98tq/qz0fiv82LtwODYUI1EKqq8eUczeBEuOX+hRvLGrW6MvCxxjKLZSr19yjTQBs 6gLbL62q4QptNA6I1rZTGY0I5aU+18oR7HFNtMXPAZLhZ5ETTVQNbeT35Rw7FwwDHcgk2bh4CGOi+l T4lLL5RC8vwvP1pvTx0JSAwP9A7oOOLzznH84cDyAvSfblMkD0O13f7GWPA10Q2NpI+6zFnhuf2W3z dXiHdT3yuR0iCy8aUmcdcMsLYmVICFw== X-MHO-RoutePath: aGlwcGll X-MHO-User: c1dd6683-825e-11ea-a065-6d02e42e573a X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (c-67-177-211-60.hsd1.co.comcast.net [67.177.211.60]) by outbound4.ore.mailhop.org (Halon) with ESMTPSA id c1dd6683-825e-11ea-a065-6d02e42e573a; Sun, 19 Apr 2020 16:56:49 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id 03JGulPr087230; Sun, 19 Apr 2020 10:56:47 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: Subject: Re: Multithreaded latency too random. From: Ian Lepore To: Andrey Smagin , freebsd-arm@freebsd.org Date: Sun, 19 Apr 2020 10:56:47 -0600 In-Reply-To: <1587258233.899122295@f226.i.mail.ru> References: <1587258233.899122295@f226.i.mail.ru> Content-Type: text/plain; charset="windows-1251" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 494wwN1dXkz4ZxC X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.94 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.97)[-0.969,0]; NEURAL_HAM_LONG(-0.97)[-0.968,0]; ASN(0.00)[asn:16509, ipnet:54.148.0.0/15, country:US] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Apr 2020 16:56:52 -0000 On Sun, 2020-04-19 at 04:03 +0300, Andrey Smagin via freebsd-arm wrote: > I wrote simple multithreaded example : > > main.cpp: > #include > #include > #include > int dTimeMax = 0; > int dTimeMin = 0x7FFFFFFF; > int Nanosec; > bool ThreadExit = false; > void Thread() > { > timespec T1,T2; > for( ; !ThreadExit ; ) > { > clock_gettime( CLOCK_REALTIME_PRECISE, &T1 ); > clock_gettime( CLOCK_REALTIME_PRECISE, &T2 ); > Nanosec = ( T2.tv_sec - T1.tv_sec )*1000000000 + T2.tv_nsec - > T1.tv_nsec ; > if ( Nanosec > dTimeMax ) dTimeMax = Nanosec; > if ( Nanosec < dTimeMin ) dTimeMin = Nanosec; > } > } > int main( int argc, char *argv[] ) > { > std::thread Thr( Thread ); > for( int N=0; N<100000; ++N ) > printf("%d \t%d \t%d\n", dTimeMax, dTimeMin, Nanosec ); > ThreadExit=true; > Thr.join(); > } > > On my PC > dTimeMax — dTimeMin < 8000 in console over ssh. > dTimeMax — dTimeMin < 5000 in tmpfs file output. > > On my Orange-Pi Zero board it produce very random values: > dTimeMax — dTimeMin > 2 000 000 in console over ssh . > dTimeMax — dTimeMin > 15 000 in tmpfs file output. > > What can help disable interrupt of thread ? > > Set kern.hz=250000 not change result. > > FreeBSD orangepi-zero 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r359929: > Thu Apr 16 06:10:42 MSK 2020 /usr/src/arm.armv7/sys/GENERIC-NODEBUG > arm > > to compile: c++ main.cpp -lthr > > > The only thing I find surprising about that is the 2ms of variability in the arm console over ssh case. The 15us in the tmpfs case is actually better than I would have guessed. I wonder what it is about ssh over console? Is it the network activity? It might be interesting to run the tmpfs test while having iperf or something similiar running in another window to load up the network. I wonder if dtrace might be able to answer any questions here? I'm not very good with dtrace, but I suspect we have some people reading here who might suggest some of the canned dtrace scripts that could help figure out what's causing the high variability in timing in the ssh case. kern.hz doesn't really control anything anymore like it did in the old days. It has some indirect effects on performance, but nothing like it did in the days before eventtimers and the tickless kernel changes. -- Ian