Date: Fri, 16 Feb 2007 11:48:07 -0700 From: John Giacomoni <John.Giacomoni@colorado.edu> To: Daniel Eischen <deischen@freebsd.org>, Julian Elischer <jelischer@ironport.com> Cc: freebsd-hackers@freebsd.org Subject: Re: pin/bind a pthread to a processor? (take 2) Message-ID: <1959ECB1-61D4-4034-BB10-83D68DA01B13@colorado.edu> In-Reply-To: <Pine.GSO.4.64.0702092348170.12150@sea.ntplx.net> References: <368841.53689.qm@web32903.mail.mud.yahoo.com> <45CD31D3.8040106@ironport.com> <Pine.GSO.4.64.0702092348170.12150@sea.ntplx.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail-24-70959062 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Dan Eischen and Julian Elischer, I'd really like the ability to "pin" to a processor. However, I think I need something stronger than immediately rescheduling the current process as seems to be done in the 4bsd scheduler. I'd like to "own" the cpu as much as possible until the "pinned" thread yields it or terminates. I've run some user-space and kernel-space tests in FreeBSD 6.2 with preemption turned off that look as follows: I created a pthread with both libpthread and libthr that had PTHREAD_SCOPE_SYSTEM set with pthread_attr_setscope. The thread then calls a system call module I created that calls sched_bind and sched_pin on the thread. The same is true for the kernel, only I create a thread in the kernel with kthread_create and then bind and pin it. the bound thread in both cases iterates 10,000,000 times, and for each iteration 1) takes a timestamp with rdtsc() (x86) 2) spins until the delta from that timestamp is >= 100 3) classifies the delta as short or long, short is a delta <= 400 4) repeat Short trials are on average 102 cycles (good) Long trials are on average 950-1100 cycles (bad) There are 950-1100 long trials for every 10,000,000 trial run. This means I'm getting an abnormal trial 1300-1550 times a second While this is not the end of the world, it is causing me difficulties with my experiments as I'm trying to process 1,488,095 Ethernet frames per second, meaning a new 64B frame is ready every 672ns. These abnormalities are causing me to require queuing and reduce the maximum processing time to account for it, which is less than ideal :) Right now I have a setup that lets me stream 100% of frames from an input handler thread in the kernel to a user-space thread and back to an output handler kernel thread for all frames >=96B, and 95% of the frames for 64B (I can't seem to generate more than that on my hardware). Therefore it looks like I need to find a way to eliminate all non- application requested tasks including the periodic scheduler and any callout code. Do either of you have any thoughts or ideas? I'm attaching the test code I used to generate these numbers. Thanks in advance for any help! John Giacomoni --Apple-Mail-24-70959062 Content-Transfer-Encoding: base64 Content-Type: application/octet-stream; x-unix-mode=0644; name=pintest.tgz Content-Disposition: attachment; filename=pintest.tgz H4sIAFX71UUAA+0ca3PaSHK/ol8xm2wSsLERTyd2vFUYZFt1GDgEybrqqlhZGmzFQqIk4cde+X77 dc+MhAT4kU3sVGqny4WkmZ7unu559LRanjleRMPol+cEVVV36nUC1/JOvZy+JkDKar1RrzbU6o5K 1HKtXG78QurPKpWAeRiZAYjy5dwxLX/q34f3WL3oR3L9SWDG7V+Cq2W67rPweLr9AU0tg/3r9VpF 2v8lIGX/8DZ8Hh5fY//aTh3tX6vvSPu/BMT2v3zGAfB19q+i/RtVVdr/JSBt/+faAL7K/o0a2n+n Upf2fwlYtn/pxLykE8el35EH6KNRq91v/3KjnPh/dWb/nVoV7K9+RxnuhX+4/V+T2OBk4gckdKYz uMWRQAPiTwjsCfigKP1B74iQfValdHvjk2YXnjx/anqK0jrsNI8MsrlPtsLI3rc+fFCUbRhQ7tym 5ONZaG/PAv98e3r5u/KjOywhAyvzH3+2re/K45H5X61Xaov5X8fz3w5c5fx/CShtbClkg7T82W3g nF9EJG8VSPnDhw+kGYZmQD7D4KCBO/dsRGu6LmFoIQloSIMram9DOVYNqO2EUeCczSPH94jp2WQe UuJ4JPTngUVZyZnjmcEtrjTTsEiuneiCwKKDV38eIZWpbzsTxzKRRpGYASUzGkydKKI2gSXkyrHh JrowI/jBFct1/WvHOyeW79kONgqRCrab0mgX78vbS6KFbFnjMlk+rE9TGALQncgEWZGqeeZfYZXQ CBIB8PzIsWgRMJyQuEAPySzYsu5lZQKmlms6UxqgjkhlVRBgmNJILAj0056DcM8jC+G9FJRs35pP qReZsdFKYA8f6gMyNcHwjumGC8UzgyHhdDfiATA81g1i9A6Hn5sDjcA97Bif9LbWJgenUKmR5mh4 3BuQZrdNWr3ucKAfjIa9gUH+/LNpAP67d1jFRln3lGh/9AeaYRBooJ/0OzqQAbqDZneoa0aR6N1W Z9TWu0dFAlRItzckHf1EHwLasFdEdkhotSXpHZITbdA6hsfmgd7Rh6dMoEN92EV2hygg6TcHQ701 6jQHpD8a9HsGo4bdautGq9PUT7T2NgEhgDHRPmndITGOm51Oupvwl+nlgQYSNg86jBRjA71s6wOt NcTuLO5aoDMQrlMkRl9r6Xij/aFBT5qD06Iga2j/HgESVCK1dvOkeQR9yz+iFTBIazTQTlBe0IMx OjCG+nA01MhRr9c2kBSQN7TBJ72lGXuk0zOYwkaGVgQmwyZjD1RAW1AN9wcjQ2d607tDbTAY9Yd6 r1tAQse9z6AYELYJrdtMx70u6zPoqDc4RbqoD2aCIvl8rEH5AFXKtNZEXRigvdYQqaUwgSvoc5jq LOlqRx39SOu2NKztIaHPuqEVwGK6gQg65/y5ecr6OGLdR1uBbPw2NXSLzKJEPyTN9icdhRfIMA4M XYyZ3iFSMkatY6H9eBb8dhhQemC0d0kYWKXwApaiEr0x0asKS5euXRIOVYntunyzLV7BKlVlq25J fV+qvCequlv+sAvb4ozCJCTazYz8BtRLivI68arA0XL87Yvf00W3YUx/tSK6ndFwtRhW3LlLl8oj 23XOsmVzD+a8nS2bmtaF49GSNZtP5p6FlQr4E8oUltI83MBCfG4ViQVaIBsb8HBVUP6r5Pgz9exZ FOwpOUQUUo+9+RRKYHGZWxHhoo1hgwYE+NlTlFypRCYOLG8iXiFwsIkZbV/RIMR1bB/82L+oP8lj aQEIAhbe5uGKrfOvRPNXhSJ5GyOlZEAKSNA2IxP82OjKdJG5wMinMAtYDpvHPPBIXoWnO+WncnGX 4j/J8PmeHuDD/l9ZbezsxOf/qlrG+M8OuITS/3sJkP6f9P+k/yf9P+n/vbD/xx2XxX7LncDt99tl UlHVWkn9gI5g+f1uZWcXbmZf7HvcwPtdu5kZmNM1xYFvPdEPhHLXty7XYM8jerPW+YTFZLX8kgYe XeOTXkYXATXttYSiNaKH1gVl2Ar4gVsIypXvwuLlUuawORYsb1Eusn2Pgg+n7qUQlQTBBZ+VzAGx URuDZwmeT37xBN5AVFzUug7sPuDWrW+sPKUxuLxEQFIjwKZuZO4l1bZPFrgZHOhMYEehlS+QLc5m 0eqOXF+gAvIc82Mic4zA3VNeXdgjBH3UFbVc+Y5NLqgL+22e3aO/jt5tql5ZqX+gbxETk1vhARx/ 9jDKPVSqlQWK5c89RCmLgOL9xMyr84e5AcLYfQK3sbv/ULce5RMbdQml3FigMCvyXoEVZgFUT/Kv uAVIMPc82Nf/471CE7FjEXgSNr7GV3IxLjwskHLT6GaMk3nMxuxbNpVYATv7sCckkbfmAZ+Vxcqi BpvERObefWRwVIFrYJ7BqQ1ECFgTJZeMBDGEF30Gtyef6NcBlRLnI9Mv3GxuFtbMhntJLWuWiScm 5PKEiMGZxJPmd5JnSJtVVS2s45sdIJv7y1N3GcQoIZubqzh3hLrgGj/I5AEWdyslkWhw/xLBjMCm WqI4sBX1MqZajBzr1oL9CrDflNUvc6AnCnDomWceOKum60S3MQKMsKKS4yy2IqH0fPqxUBIKQTax VDE3m4ZOAKMIerFL3jQYQ9f3zjMFcNYeY+EussSCyI9MV6BwEVBvpTxjtBWzK3KLxeyLJLlBpfHi gggszD0YM6W5h9PgSfOFNcinn7Mz6PEZk2ickfKoLWZ0soWV8UFsk2N640RxkGHNvhYpl9NbIEPy IoDCm5GNyC6SzJqd3iEx/pI788+A0TyWhtkMzw18uRErSCyGBb8RzfO6IumOOp34V2V/YpV6Y4NN UNyE7rXpMLL4qlGsZJGPAR0nvIh7LjazX5mIOBVxYLmUzvJvQUpO/+bmBkiXUag7brtz54qfmPjh RfTcxNiTh2cpn6DylHtJqRk5hWxcsIVRluI8JX7uAaZ/csfnHesYSuHR6+TVKXPYhJGEYTg64eYa i6d97KtazOVypQ3AGMPB8Bzb5jhaUo5EsfxuLy2CP5mENGLnbU7uGvTAVSIEIXBYhIsPp2t+bC+l x05MYB+ODWPj1GjBeSbDACN87JTIXg6DbsG/8U27hIPbXKWnYGkyErl3STb4tcgYWtOVYYnFNAhA idx5y4Vw5LQuSB5w2VCwTFg4T3rtcafXbJNdGBvcZATGrOgl8uXivbHZqsA7hmMldwaj4nIvRWbU vZ8Q7xiQmgT+9AFiNp2YczdiRGLhtV4fTl/GqN9PY94lQ4jhsUEkVD0GaUYdLS/icEXylnOCm8wo KbL+8ekGUnxz/Gc5/vcM6R+P5n9U1QbG/9RKvVavVFj+R7XSkPG/l4Cl/I+zuePauEKzdYMdW5Pl Q4TaX3/1GTfmwI64FTzelvF4W4bjbWVXre7WKuDPitOt8i+YCPtxpF0xBi0jedq2lpJKLoGBTCr5 Jsjm/z5L+P+x+V+v1+L8j3qj2thh879RlfP/JUDG/2X8X8b/Zfxfxv9l/P/l4/+Ph/lXj5/oprIs ZVy3EldVqDFzCk1eDbAMExphyDZ9Qn1qsATDhJFN9vfJIrLD4hKLeIk5pTGBLeJfioBJEuhLocL+ NoGDuRcz/BUhxlc4s1/xiMmaZeK6CyLZyO7Doap0bDcSQd2lqG6OH/zLMa0H4lWpLq1EzJ4kxyJm lg2aRXZKEvUJkiy0wWNnC33cLUI16vcJ1DwtTiPDNP+YMM1qlOY5gzTPCEvnv+cI//yd+E+5Ls9/ LwIy/vPPhqXv/39I/LdSVZP4T73C4j8VGf95GXi27/867UWZO+O+rvI6UwqF8jPBHwzL8/8ZPv97 PP+b7/+Y/11u1DD/u7Gjyvc/LwIy/ivjvzL+K+O/Mv4rv//7Ht//LSpn63K7ReHYm2WzuJM0bpaN Fon0Vh5eylal8oGzFTLn+7Gc74y6HXtjbbo3q1mb6P0N323mYiXkJin7ZUrjPPBF4VrMJURMHM79 3PnfuTjxO5fN+Mb4ekQx7sIdE1DmNWWujZgdSnqelEULzLAUY4O1sy6odRliprNojK9c6JSE4FPQ VLIlz9JNf2GLUXmkjc6EiK/HjdPhoK/98vZvfXiLqvDPz/mrHJ+cgQnsRDBwTIA6vf/zXGwNPlY6 +fQV0/UrkdfMUISKrGgObTm2QCObpKqqMTJgvWM+GRycbmCAX5uzGbU5FxbLD8GZdUF/lPxFAz+D mDbEcieUxdyQefIMfkie/GQ5T/6bsuKJkuMUtyZxWnz68adIixczD05oeMz4G3Nv7SKG2/VSmvss neYu3p1kMt2f8L8FYhoR3O0tHs0oCrDMXFfmiST4HFsUhdQ8yZ2Y7A0horI+46u39ApKYM3DtPIM TccD+d/OTP5hAGLtk/7weKA122Oj1etr+OJuqJ0sywLEGTq2LXL6hXXSZAQQkbWEkkjOfzvDl1JI J52mX0jvEvwVKHtnDWsqG4+pDUFsLcnnRQwPufMm78J4C1nbda+w3LlzGoHXlxdieWnd/E9dg5xo wsOdQugimYpxU/G6L6MrF2yU+QpAdC1xHcvLWoBL8hkCTiHRLFnH2ZhnTUIfLW6c9Eutk35iCEwj iHcXOJEDIbr8HUN6fH/xYQijIpIx/qOjQBIkSJAgQYIECRIkSJAgQYIECRIkSJAgQYIECRIkSJAg 4WeH/wMHGDv9AHgAAA== --Apple-Mail-24-70959062 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed On Feb 9, 2007, at 9:53 PM, Daniel Eischen wrote: > On Fri, 9 Feb 2007, Julian Elischer wrote: > >> Peter Holmes wrote: >>> This is something I am interested in doing as well. I had >>> corresponded with Julian Eischen & Daniel Elischer about this. >> >> Dan Eischen and Julian Elischer :-) > > Yes, it needed byte reversal or something. That would indeed be > a strange union! > > I think it would be nice to have something like Solaris pbind(1) > and processor_bind(2). > > -- > DE > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers- > unsubscribe@freebsd.org" -- John.Giacomoni@colorado.edu University of Colorado at Boulder Department of Computer Science Engineering Center, ECCR 1B50 430 UCB Boulder, CO 80303-0430 USA --Apple-Mail-24-70959062 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed --Apple-Mail-24-70959062--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1959ECB1-61D4-4034-BB10-83D68DA01B13>