From owner-freebsd-virtualization@FreeBSD.ORG Fri May 30 16:44:41 2014 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E6DD7B8B for ; Fri, 30 May 2014 16:44:41 +0000 (UTC) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 90101290B for ; Fri, 30 May 2014 16:44:40 +0000 (UTC) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 76289153A9E for ; Fri, 30 May 2014 18:44:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HoOIckkV_mq2; Fri, 30 May 2014 18:44:30 +0200 (CEST) Received: from [IPv6:2001:4cb8:3:1:5cbb:e5e6:6f86:6c35] (unknown [IPv6:2001:4cb8:3:1:5cbb:e5e6:6f86:6c35]) by smtp.digiware.nl (Postfix) with ESMTP id 8168F153AA1 for ; Fri, 30 May 2014 18:44:30 +0200 (CEST) Message-ID: <5388B56D.1000501@digiware.nl> Date: Fri, 30 May 2014 18:44:29 +0200 From: Willem Jan Withagen Organization: Digiware Management b.v. User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: virtualization@freebsd.org Subject: Bheve: Slow linux syscalls on AMD Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 May 2014 16:44:42 -0000 Hi, In my quest to find why Linux is so slow on AMD I executed strace -T -c ls -aslR / > /dev/null On both a Linux running on a real CPU (just a mere Celeron): % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 25.26 0.235827 1 180706 lstat64 23.06 0.215235 1 255949 255942 getxattr 17.65 0.164726 6 29720 getdents64 13.41 0.125168 1 180698 stat64 12.75 0.119004 1 130339 130339 lgetxattr 4.28 0.039929 2 22656 86 readlink 2.08 0.019399 1 14859 openat 0.70 0.006569 0 14910 close 0.54 0.005085 0 14909 fstat64 0.15 0.001417 0 3135 write 0.10 0.000896 0 2831 clock_gettime 0.03 0.000263 3 83 35 open 0.00 0.000000 0 18 read 0.00 0.000000 0 1 execve 0.00 0.000000 0 12 12 access 0.00 0.000000 0 8 brk 0.00 0.000000 0 3 3 ioctl 0.00 0.000000 0 35 munmap 0.00 0.000000 0 12 mprotect 0.00 0.000000 0 57 _llseek 0.00 0.000000 0 67 mmap2 0.00 0.000000 0 1 set_thread_area 0.00 0.000000 0 2 2 statfs64 0.00 0.000000 0 4 socket 0.00 0.000000 0 4 4 connect ------ ----------- ----------- --------- --------- ---------------- 100.00 0.933518 851019 386423 total And on a virtualized Linux: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 55.83 30.198355 666 45331 getdents 17.99 9.733976 429 22665 5 openat 16.62 8.988507 396 22674 close 9.10 4.920301 217 22673 fstat 0.37 0.201331 223 901 write 0.03 0.015656 447 35 23 open 0.02 0.008853 328 27 mmap 0.02 0.008239 358 23 brk 0.01 0.007008 501 14 mprotect 0.01 0.003540 354 10 read 0.01 0.003146 393 8 8 access 0.00 0.001808 452 4 munmap 0.00 0.000660 660 1 mremap 0.00 0.000615 308 2 2 statfs 0.00 0.000582 194 3 3 ioctl 0.00 0.000254 254 1 execve 0.00 0.000236 236 1 stat 0.00 0.000193 193 1 arch_prctl ------ ----------- ----------- --------- --------- ---------------- 100.00 54.093260 114374 41 total One cannot really compare the factual results, but what is significat is the orders of magnitude difference in syscall time. So I ripped this from the net: Which turns out to be a BAD test, since linux caches the getpid value. ==== #include #include #include #include int foo(){ return(10); } long nanosec(struct timeval t){ /* Calculate nanoseconds in a timeval structure */ return((t.tv_sec*1000000+t.tv_usec)*1000); } main(){ int i,j,res; long N_iterations=1000000; /* A million iterations */ float avgTimeSysCall, avgTimeFuncCall; struct timeval t1, t2; /* Find average time for System call */ res=gettimeofday(&t1,NULL); assert(res==0); for (i=0;i