From owner-freebsd-hackers@FreeBSD.ORG Sun Jun 23 00:03:18 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 40B4E616 for ; Sun, 23 Jun 2013 00:03:18 +0000 (UTC) (envelope-from doconnor@gsoft.com.au) Received: from cain.gsoft.com.au (cain.gsoft.com.au [203.31.81.10]) by mx1.freebsd.org (Postfix) with ESMTP id B67E315F9 for ; Sun, 23 Jun 2013 00:03:17 +0000 (UTC) Received: from ur.dons.net.au (ppp118-210-177-45.lns20.adl6.internode.on.net [118.210.177.45]) (authenticated bits=0) by cain.gsoft.com.au (8.14.4/8.14.3) with ESMTP id r5N02vYN012641 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sun, 23 Jun 2013 09:33:06 +0930 (CST) (envelope-from doconnor@gsoft.com.au) Subject: Re: KDE installation Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Content-Type: text/plain; charset=us-ascii From: "Daniel O'Connor" In-Reply-To: Date: Sun, 23 Jun 2013 09:32:57 +0930 Content-Transfer-Encoding: quoted-printable Message-Id: <87A9FE3C-26B4-4510-A8A3-3BB1BBD81EDE@gsoft.com.au> References: To: Ruben Dario Alvarado Tejada X-Mailer: Apple Mail (2.1508) X-Spam-Score: 0.163 () BAYES_00,RDNS_DYNAMIC X-Scanned-By: MIMEDefang 2.67 on 203.31.81.10 Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jun 2013 00:03:18 -0000 On 23/06/2013, at 9:22, Ruben Dario Alvarado Tejada = wrote: > I wonder if there is a functional procedure or guide on how to install = KDE? > I have followed several online documentation in relation to install = KDE and > nothing has worked for me, I guess I homitido a previous step before > installing KDE. > I made the installation with pkg_add-r kde | kde4, make install clean, > etc and I can not activate the interface. After install log out & in and replace any .xinitrc with.. /usr/local/kde4/bin/startkde Then run startx. Alternatively you can put kdm4_enable=3D"YES" and reboot or run (as root) /usr/local/kde4/etc/rc.d/kdm4 start Then select KDE as your session type and login. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-hackers@FreeBSD.ORG Mon Jun 24 14:27:09 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5A07DEE9 for ; Mon, 24 Jun 2013 14:27:09 +0000 (UTC) (envelope-from dcherednik@roshianokatachi.com) Received: from smtp.nanocore.sportcomitet.org (unknown [IPv6:2a01:4f8:d13:2941::1:3]) by mx1.freebsd.org (Postfix) with ESMTP id 92BD81AAF for ; Mon, 24 Jun 2013 14:27:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.nanocore.sportcomitet.org (Postfix) with SMTP id 4DA5FC0259 for ; Mon, 24 Jun 2013 18:27:07 +0400 (MSK) Received: from webmail.roshianokatachi.com (unknown [192.168.113.121]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: dcherednik@roshianokatachi.com) by smtp.nanocore.sportcomitet.org (Postfix) with ESMTPSA id C36E3C0188 for ; Mon, 24 Jun 2013 18:27:05 +0400 (MSK) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_6d72d94acfd6e7bc8332c22a3d0ea929" Date: Mon, 24 Jun 2013 18:27:06 +0400 From: Daniil Cherednik To: Subject: panic in =?UTF-8?Q?g=5Fio=5Fschedule=5Fdown?= Message-ID: X-Sender: dcherednik@roshianokatachi.com User-Agent: Roundcube Webmail/0.7.1 X-DSPAM-Result: Innocent X-DSPAM-Processed: Mon Jun 24 18:27:06 2013 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 24,51c8573a13362741013760 X-DSPAM-Factors: 27, ffffffff803d8910+#+#+#+34, 0.40000, 0x10+#+#+data16, 0.40000, attached+#+#+are, 0.40000, data16+ffffffff803d890e, 0.40000, look+#+#+have, 0.40000, Date*06+0400, 0.40000, ffffffff803d88be+4c, 0.40000, 48+#+#+mov, 0.40000, 48+#+#+mov, 0.40000, Do+#+#+#+do, 0.40000, c1+#+#+sar, 0.40000, c1+#+#+sar, 0.40000, EPERM+#+#+return, 0.40000, schedule+#+#+ffffffff803d88f8, 0.40000, test+#+#+ffffffff803d88ce, 0.40000, Date*Jun+#+#+27, 0.40000, rbx+#+#+e8, 0.40000, 90+00, 0.40000, break+#+#+WRITE, 0.40000, nop+#+#+89, 0.40000, was+#+and, 0.40000, Received*Postfix+with, 0.40000, 00+#+0x5, 0.40000, 66+#+#+#+data16, 0.40000, 66+#+#+#+data16, 0.40000, was+inlined, 0.40000, esi+#+4c, 0.40000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2013 14:27:09 -0000 --=_6d72d94acfd6e7bc8332c22a3d0ea929 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=UTF-8; format=flowed Hello. I have got panic, see attached file We are using FreeBSD 8.3-amd64 but this part of code wasn`t modified in HEAD part of disassembled code: ffffffff803d88b8: 4c 8b 43 18 mov 0x18(%rbx),%r8 ffffffff803d88bc: 89 c6 mov %eax,%esi ffffffff803d88be: 4c 89 c2 mov %r8,%rdx ffffffff803d88c1: 4c 89 c0 mov %r8,%rax ffffffff803d88c4: 48 c1 fa 3f sar $0x3f,%rdx ffffffff803d88c8: 48 f7 fe idiv %rsi ffffffff803d88cb: 48 85 d2 test %rdx,%rdx ffffffff803d88ce: 0f 85 29 01 00 00 jne ffffffff803d89fd ffffffff803d88d4: 48 8b 93 90 00 00 00 mov 0x90(%rbx),%rdx ffffffff803d88db: 48 89 d0 mov %rdx,%rax ffffffff803d88de: 48 c1 fa 3f sar $0x3f,%rdx ffffffff803d88e2: 48 f7 fe idiv %rsi ffffffff803d88e5: 48 85 d2 test %rdx,%rdx ffffffff803d88e8: 0f 85 0f 01 00 00 jne ffffffff803d89fd ffffffff803d88ee: 4d 85 c0 test %r8,%r8 ffffffff803d88f1: 78 05 js ffffffff803d88f8 ffffffff803d88f3: 4d 39 c1 cmp %r8,%r9 ffffffff803d88f6: 7d 60 jge ffffffff803d8958 ffffffff803d88f8: be 05 00 00 00 mov $0x5,%esi ffffffff803d88fd: 66 data16 ffffffff803d88fe: 66 data16 ffffffff803d88ff: 90 nop ffffffff803d8900: 48 89 df mov %rbx,%rdi ffffffff803d8903: e8 68 f7 ff ff callq ffffffff803d8070 ffffffff803d8908: e9 e3 fe ff ff jmpq ffffffff803d87f0 ffffffff803d890d: 66 data16 ffffffff803d890e: 66 data16 ffffffff803d890f: 90 nop ffffffff803d8910: 44 8b 5a 34 mov 0x34(%rdx),%r11d ffffffff803d8914: 45 85 db test %r11d,%r11d g_io_check function was inlined and quite good optimized. But it look like we have possibility to get access to wrong address in g_io_check(struct bio *bp) function here: if (cp->acr == 0) return (EPERM); break; case BIO_WRITE: case BIO_DELETE: case BIO_FLUSH: if (cp->acw == 0) return (EPERM); break; default: return (EPERM); for example in g_io_deliver we have additional checking: cp = bp->bio_from; if (cp == NULL) { bp->bio_error = error; bp->bio_done(bp); return; } also in this function we have KASSERTed bp and pp. So the question is in which case bio_from can be equal to null? Do we have to do some additional checks in g_io_check? --=_6d72d94acfd6e7bc8332c22a3d0ea929 Content-Transfer-Encoding: base64 Content-Type: application/octet-stream; name=panic-g_io_schedule_down.gif Content-Disposition: attachment; filename=panic-g_io_schedule_down.gif R0lGODlhgAJNAecAAAAAAAEBAQICAgMDAwQEBAUFBQYGBgcHBwgICAkJCQoKCgsLCwwMDA0NDQ4O Dg8PDxAQEBERERISEhMTExQUFBUVFRYWFhcXFxgYGBkZGRoaGhsbGxwcHB0dHR4eHh8fHyAgICEh ISIiIiMjIyQkJCUlJSYmJicnJygoKCkpKSoqKisrKywsLC0tLS4uLi8vLzAwMDExMTIyMjMzMzQ0 NDU1NTY2Njc3Nzg4ODk5OTo6Ojs7Ozw8PD09PT4+Pj8/P0BAQEFBQUJCQkNDQ0REREVFRUZGRkdH R0hISElJSUpKSktLS0xMTE1NTU5OTk9PT1BQUFFRUVJSUlNTU1RUVFVVVVZWVldXV1hYWFlZWVpa WltbW1xcXF1dXV5eXl9fX2BgYGFhYWJiYmNjY2RkZGVlZWZmZmdnZ2hoaGlpaWpqamtra2xsbG1t bW5ubm9vb3BwcHFxcXJycnNzc3R0dHV1dXZ2dnd3d3h4eHl5eXp6ent7e3x8fH19fX5+fn9/f4CA gIGBgYKCgoODg4SEhIWFhYaGhoeHh4iIiImJiYqKiouLi4yMjI2NjY6Ojo+Pj5CQkJGRkZKSkpOT k5SUlJWVlZaWlpeXl5iYmJmZmZqampubm5ycnJ2dnZ6enp+fn6CgoKGhoaKioqOjo6SkpKWlpaam pqenp6ioqKmpqaqqqqurq6ysrK2tra6urq+vr7CwsLGxsbKysrOzs7S0tLW1tba2tre3t7i4uLm5 ubq6uru7u7y8vL29vb6+vr+/v8DAwMHBwcLCwsPDw8TExMXFxcbGxsfHx8jIyMnJycrKysvLy8zM zM3Nzc7Ozs/Pz9DQ0NHR0dLS0tPT09TU1NXV1dbW1tfX19jY2NnZ2dra2tvb29zc3N3d3d7e3t/f 3+Dg4OHh4eLi4uPj4+Tk5OXl5ebm5ufn5+jo6Onp6erq6uvr6+zs7O3t7e7u7u/v7/Dw8PHx8fLy 8vPz8/T09PX19fb29vf39/j4+Pn5+fr6+vv7+/z8/P39/f7+/v///yH+EUNyZWF0ZWQgd2l0aCBH SU1QACwAAAAAgAJNAQAI/gARCBxIsKDBgwgTKlzIsKFDhwZQoVLlypCDhxgzatyIwIBAjxxDihxJ sqTJhhElwrJz8aTLlwxBwpxJs6bNmwtl4tzJs2dNnTuB+hxKtKjRo0iT+hSqtKnTp1CjSp1KNSTT k1elZk26tarXl12/ioUZdizRsmbT5kx4FSRal2+tqp1Lt65dp3Hv6t1LsyXfv4AN5h0bVubgwBsP Iw6smGBWj40XO5ZMubLlo5GhevR7ObHJzJ1Dix5NuqkBAwA6FkQLAHRphwBSP5S9MDZtgbFh3x64 u2Zug70P/m5ou2Dxm8MZHleY3DhH28GJ916OO/pL6NN3L8cO3Ppz7yeh/hMUP344d96xu6ZEZRUV LM5swbuWa7M5QgD48MFPiN9DbXj41OIXfvoplx98+O1EIIIFMgfgfsIBWMt4EoZkH3APEgcgKf81 aBw+/qGnHID5hYgRABnyho89HqLo4UYXjqcOifiYgEBLLpLAG4DGXIQiiyVSCCJEBr6YUYxC4mMK jiSWIqSACKAYT35DRkmiJxA6JhFnp7XmWACojXeaY13qZAAo7Cm3nWxhVpccecxRB2eUcq5Jp5BZ lnfbeePho6ObtLlogJ/o7TkndVbio9OcwgVaqKHaAQgeAOq8+FugAA4aoqBV0mleam+myJ+od0a5 o6KdwulilnuSClyl/hftgA8ER7q66qk+Agppq0YaJ8+s+NmIG4ksDNvigZqe2umJde4K6aIAJsDc rw70l2i1Vd6KG7XWLhjAsgcFgOZ4aMJiBm9oSmTBRxKh0tKZpqDiChkEBeDeiFT6tyoAWuiHbX5Q JpofBw5OiQ/BUVLZCpP5Ghugj/m5aOKCAzFFYIMokhiwp9lq7ICPmfZJLW7q/IqPLzhSOWB+v9pY rcej6geAELP+i08xKt5zrDy/mhihorLt2zE+AQgJbX4LJ4yPi+vidmCHbQq9acjKvsJwm31O/HTM L9d8Yqa8DgggAjRDYAA8JgsIcn69fOx0r/cNPaiNlAI9ndzgHgSU/otBKo0xjUPjU8rKcHdXo9Ko HS6offj9WR2hCp2ZZkfjRrSu5A5YPpC4kwOAZuaoND1bfqiB5CJ+TMzq5tAIcKH6fWNv4S/H+j7Y Ldgd/yqG11/jI22UsFor8JAuJqwvPiZz5uLvUuMqJPOVJu6zcZXiNwXGsaUaLZ7/kahv8ITiR2LK yRub+J8/4rMG77DBQ+K6LkLQLT46/y1z4PiI7vT0+PINQZ4Rep/zjHW7+9XOgM+BR9sSlTj4oQ1p xlHg2hp4JNKFCh8qqtY9XDFB4dXqYQ47BciAtAvlwWNjLlpg5EQxOcxpLiKuiFLoNve5j1RuhhUD j99MQTLAfYxv/kPjF/siuDTZ/Q+IxwMa8dzXsEQhIHX/q46GMGgsbB3QilYC0tAQlKLmGet3xtoR jfgnxvrVDIkDDGOHEECFWZ0Ngdfi3rBMxrrrRbFUDsKgEd+ILfSJqm6Zy9av2Mg+D+IRdgazI450 WDw7VnFHU+pFFQuIRRHlRFsXo5LZHpSsR/pNk4sskuDEZqmxkYyOPkMSroJmuyW6UXtUnKRy0OQW U8QQADhcjyswgK4Wfk5zAuEcAD01v3vQ0WwWxB8Zd2SyJErvbUEiEB4FRcRhMnCSxEumjgiEGkwp ioi8Ad/EqPY2MTjTS/zBRzzMiEwljjOWb+PCMgnYr3Y+85qn/pIJN81DTjXmUYj2FB4133bP0ymS dqeqjTxYBNC1DfN09Vyb0fJ3rfm5M41cAw5F9wnL/eBHdMWrjW1gqU9JyRNxFwTg4oY0qIty9J1X M6RB7CUvQrxLIq4oxLu6hApedkQlFkGAvd5jAFiILiKwUCiVipEyVAkrk08lFXZyoy1oZmukm1oq xFTWp8KpKD9MVRq2nkqlbFGpbTY7Vn4qsdVKXmxwfepEU/HBCZWSzqxNvNiS+mQJu+a1rEncWiYJ hw9WbHVrb3soYK0aWLXiQ66M3aoHKcacxXoynX/ND2SVhoA5vFI/nbxY0hJbJAbl54hUwtJXTeu1 qgqHjgG7/tgCM4lXfKhQrJW17MU2ezG2rnZblh2JLlniFcpGiY5UskBLuwIA2FoTUb9hbnB0aJIL UVeVxmlOenCCXT2piT/f6e59pgMguErxOYiNE+N2pSf2lgcm4hVOekXa3vp69zVICYt4jVubvBER hfOVr5HwY02S8Ddh+Cqwi2J4KvNqpLuujdnGfgYhQ0KYRvprn6Xw8UdXHYmRMDvV1JDGsMKW2MER HsmBmaVhP1IJYXwbbaJuqxZVsSlQ232UdN+UY0DZ904pVZOzNDqx7WQqVTxuFhHZm5XoovNQ2u3n z7KLm3zircifOmSKiQjGUnmzk54qaeHCxpxKmW0Mr5uN/q06XKCgQRmdl30tHAV2RUra2awsKnCj ktxe98YZdtHMkOsy18oR61kqaNwXG9pMpbo6jKKjMljTeus2NApsthH7FcJwW9kpMY9Gtx1pVNl2 tT7leVgk4i3yHBs4SuRpX3I4Y2rJdyzv5TFqhTaf0ZIEVwIx7be3ZiUcB+owJ5nvblrzakUBdEcN Yy2OYazn2Uw2Wr6hOMCNujJKs+nSZMIURmOEZon7xrdQY3ui27Qds1/KQDirxYLCxg+aK529ESNg fc0G56LpfecDFpREaM73FBnm1nCv9HjGLGUsvQjtLx7bvwRUH/Yma0qMBjDQwMpm+RJmj5G1O3f4 UEPN/g6NIpM5EIH4STjBBQoPIGVYpv0jkcD/I+k0ns6VWGR47/a60kmTSIWZzGoTGz46oB0tZdWK R9IO/kEKnk7mafXAy0y22RpTCUrpAyUQddQ1ABz0ZwBYwhkN7u3hsW53d9xvF3O9qmSVPK9rxhis wOzakAIv3LAD3BHxTnS7xwwBIid0xg0tRyvhvZFpvjAGryd4+UGuqoBkOQbrCUYLT6rlv/q64hFg RIdCEh9LZ7s439uhaln1tHwE8/4+9M4uHzJu+UGxVCv+9r6dlz/IdSM87sGizqvoiqo3C3Sy2XFN LpflBpun4aNZ9k8mcZpS9nvMTDd6jm4T3vkkouly/q3rr47hnFsRX1kDStKurkH5p9tjfozX/fbv 88vV7BAA0H58uX3130Vsbb0t/qFfAcB1USR9YBdwEmVlJ4dM59RRwQYuH7VtRQYPMhYbELhI/SSA 3bF/7pcpOkA36xdkxLFR6qQoshKAKPIMW5U4wvJuZ9VUoVVWLqY8U9V3uoVV0BRWg9VVikVqYiVQ ZUVWYHVYjtUjxzVsWtUnYeU3NKZRqII+LghNx6A8IIRZ+eGDU/hJc+VYmMZVwCaFtcVtiHUxsaVb EZMq5/ZVtrdlXUVuVEILTcV5n+VWCkM4fjV0foNaPyhuZogjyJWCaAhcOug3QkhbQheFnNZfp5VW /lRIWBPGFxD2hRy2IsnVUg81IwCzKEN2eziWZR0BKRyhHjASX6byEbdHEoeGX7DXDAiCXspmikVX imDBiuFRhnmUH/qDXW03bxPkNAYXMmLnAAmAH1NCDLkodgFIIpv2ej0BjIVzEaYDT4U4FPMhGfjh erAIjTwRjdX4ZtmVZMKWOYzneRkVUeVWaWNjR2cjKb6HSS23NI5UPOnoYRfoZ5fyY30nJxXIjX42 Xs2SiaBIevIYNPTFjT82j49Cj8RWjQiZkB1iWykzJaREbjSSbjLDbKTFH8FjR16UfrpHIv1Cjn8D cLrXKytGRAGiLKGnRD4HJJsGeSRSbSGIZ39j/jJAF5Gj8nMlZoNMxJC6kkcF1zA69yFLAyA+BzDj dkCON2I6OTxMaDIuQxYK+ZR6wX2JwjzYh1LxBnVqtjQSJ3j/lnwMBErQximolyH9CDwox3YkUj97 h0Fk0Fprd1Fxpo73I5GVJBzgQzdoOZefdDj3UX0WtH1zFiFTIjtcWUAP0G9OZFFRcnRQ+YqNeRdk WXgXw4bQ1Aqolh+aNymUaHwPBJGocnL/Uoy+g02flTi8Y4uReS1ut45aJylD4Jb2QyWOI31ymXPN R2GI+ZMJxoM6aH15hABSsJGyuW3bVH2sWVsoVHGPuZxBURKRwRS36XfDtyk0KH7/B5sqtY4t/rVJ wzY2bQkBAcBJhOlPgGQA3/kvQuSRfvVM0lRAg4J2VtJMCUVQb5IfJ0Wec9eF8xR53MaeUgkbzTed jHVr9wdnQcdZFlU6j0KG+PB9CukWa/ETnZhflyFac7V0dESZmQR0wTWSv4VvkmhRGBRwkthZspYf qkUxixaA49dWhpiUsrVVtKJIdbeC0DRh+PGE0MQLbRVaI+VTAnaU0NQIbQVzohQ+ZYUygAJxVmiG VsN6A7pDQKhaKlJ1zHmlWNoT2JgRrugZhtilImE6dGSlY9FcVFINYFpZKZiln2EaKHEXb7GlbEoX WyGnc3qn1+iYWWqneCoYfdoZfIoZXkoS/oEqGYWKEYf6p5oxE4chp4mqqJDKpkLxqE9BqcIVqZia EZZqqTTBqXoDGPNBqZ6aqaRaqliBqKaaqqQxqqraqq76qkrBqngBq6Ehq7V6E4phq7R6GRD6mLrq nMtpp78qFsO6qoSKEMW6q8rap2Vhq8m6rIr6rNA6rRRKrdZ6rdg6qJYhrdnard76rRMqip8KruQa E+WKGNw6rel6ruzaptC6ru16q5i6pdEIr9jKqjGoHNJBXVUGrHGyjfyaFIgyG9ZlIdOlHWVJZd/B EwNbXRIar5OBkCtyTHZFjQKRMvC4N6O5GtfBZjfTZQmLHBLSpfhxan1isQkGlCvjcQ8m/osxg7Iu oYxpWlmO86YQyxVRwRRCsS/qQIIXZIk91ofZJ0UCWZARm4eJRXGrSEwGuY8GYUJLaLReVnj4xB+O 0m73dSog27Db2GelhzU2Nk1vxiswe17PQmSFoquFeqiDobP+qq3I6qeFwahZCQC0QJVE6TTF5x82 I0n89yFA4h8t9Wt+Y15SQotrlF0M+lDwUHNS+mhI6WA/syhU0mtSCXkTy5e42UY/tHsDg4c2RyK3 YFcmmzH5YQvZuXhjlx88FzEAUjtAMpP5wUF/m08NKDAohoHWNrMP0aiu2haqEaFwIa7HOhlvt1er 822FooAdUnmN26D6J3kIEFEZdSpU/to7CLA7/IZz7ilT+iQPKqcnxDN6cQmPp4IfneeOs7M8csR0 PIlN6CcprtSV+GBE4XmWcEl0GKJ3bySkw+NvRtqpNnupb8sWS+GukTOuWhG8QoIAvvW/8wm5Efyb CYVvXeOZYZdmFLZKCvtPUHTB/dktm6m5G6ws4XO5s+eM+2sPatk1biiaUCtGdPgznxaYrzIljkYp hzc2gxCSOXecrecg/JuXP4xKuEoVcdGrYcqxvhoeAFI09MmgfMBL7NZ+mKUHPkVsVWxp40U17qu/ GAIk1weXB7pPzCUpBRrCsASUU7Cfq9aaGBScHnk0zWg3+BJONqx9IGh0Q2uWtgk0/lBsxRiCKvNW fwkIfkUhrHNqrzVBR4S1LPgRhpk1SgaCdWX1yC/4ovkxxsNZkdUbpQLqN2Q6Ho5shvoJTZKLH6iY W4PChBhUNn7jDHO1yn5Dy7CHD321g0zKWYJwoviQopectLWlancIexdBjH6jo5/kyvgghMzKxAjJ yCURIwErZObhsp+owM+IiWVFpOGxJlz7u256p0osvLxaFLyrivKlfLMxwk2JjCqGzcaMttOMYfE5 w1UhzdZ4wEtMt5ShyAScyNY8kABZKOusXqq0OIwZfk5r0PR4tA0bwDuZUYT5Rt0ElXnRrJp6r6Uq rTHGJC5XUTBJK8u7n8hl0lSC/rzbrIS/zEyI6zcPKWMCZikYKGdUAgIWOZGz8i1UQrs4ARr6XI1B fbPm7BDkq76CV5csic/xzM76yDqvCcPwW3om7NRT5D8r4zNDTatbPRDpPBddXRSpmSgA4FnciS1A WpsSTUyTAiqQDGKdfDpRDY63Atd4XJd7ltCYhw/t6NZWna1BHdZEXdQNEaCdecgRXMU2bXtKeE9g 3FX/Jp/keZuAlsYozdS6uDQAaMjVKtDfWs4cnbPa+lYEl4a1RaUHOs02+mcsvUVpiLrQlIRAScnO V11imB+O5hWCncCD7aeLShozux3yHNEi0YjfBG7HXdzCgRiMktfdEc3duqkF/kzYvF0xcYtoI9uy 6/chZZtbCra08mWy6DU9jTMVJGvPMQN6UPvXvT2rT7nbD3ZkWIZj/Vq+yT1e9e0p7/WzlpSP7CtS jMOAV6VlDe1j1dXceGIAfc0buvxOuzzd7W2K8E3d1x3fS5OGs4XCO8PY1EOEpeu6kIYf8pmRcVi7 ZshDJZ3ivva6DlNtO4zZFRRcr6LT9oQAxLhIeFOzEb6tO34Q9MOyCNV2HUVKqavUHTYl1KvWkINQ rH2B+ONiEMki38l0ZkyfQG3dMXeIvsagE/JxGdbj77rRp1qmKYxxfYRRFgjZ5X1ZiCeady15Z7g9 s+hKQpqRY0OiOVlbhlu5/l890XmXebAZW369l0ME5oZuGVpcdopNm/lxfpX1KxGonAepNI7Obouu nALGnopu2Mld5UtKnTVd3GXFpMoIAMG5loNCMIrtNMd46IBqFxMOzVguEMEMyrf9WFA4yiprhpB1 g30iyVVY20g4h6dc274uykBICkAouQzbhAtiAOlbhGY4c65e7YERzgC+3DNRzcGqFyGrpc4BG+I+ GtyOFOU+6/w8Fy4i0wBqKyqs2uztz92uzaK+tMUaLB+o4/JF7uCtFAT26uYu31JLkP4kyM4ttb+H 8H3KJ1arif4dtKBC36HoJgBL0PC83w/vbqtnIPO9awpPsEVr4B6B4O0F/rRHw0h2EvJMO/CHgsb0 /rSlkdF97De3MNVVRGtL/jNRyDcopLsak+7eftJzbpQsZ3LFJllCKdJZJd6Wxjee8DaEK2CEyzfE TNsco+/byJcx+jYtU4Pn7bgXg9ovTlev5rm/XGKqJtmIxHxBGTiyW4UXzFQgHIYGg/X57JQLMbMc BuTVqeFQq1LG2Z0+WSnst6vvZ/Pqy52RDb1nTXdyfvhKPywsYsF5F3L6cb+Fz+RAVCzpjQ+czzF8 KFXZs6YaBnjru39CDsCkTsRKy7h6pDqmu0e4g8gfV2fMm5B8Wne7+ErKW/Ab3I25mefx/pQMP9Mn 2ZMAYES1xzrql7+Y/pu3dj6iIzdlsj/24zt4XAPna1zVpIfuBASAIIzYWbf6+Gvk775kgefCZn02 BlNbRPoyZkT02y/vekqsVeubvun7BLIEbqxEY3w+AIEAAD58BgZ6EEjQoAEEDR06ZPhQ4kSKFS1e xJhR48aHAwkSRGgRwMiDCQuWFAgvHkiTCx0CgFewI76YpRy8jNmRJD6EAOSd5NkQpkyMQyO2HClU YUmPBoNWbMp06VOTe3pODZkS382LMH8GjfrUI8uoRykCULdVKtCsHtVkPQvPHkutR5safLnz6kkA QmM6QGlyDFyOhSEa1mgWcUbFixF83fryIz4SSj9WNonPlwOuCXtx/hb58bNlfL9AjxX7sVZnx60l snYdW/ZEnx9NcUUdcijrsa1uouZNsPPY1ZnxFc9cHHhXyMM/8sI9Wbf0rtSNwx2IPDfOyF3/zvyo Wy9p5GeFoxw7WrJv790FNpdMcDTqYqdF/z5P1TPsi40Zz27Iv40ElI1AiZJyrS+L/DNQKNoUTAxA nTiCUEKKGrSQpArjKigihjTMCMQQEcxLsZECOBAAE1VMccUVKxRRKJJ0AgBF2miE0KgVO4oJr9A+ Ys/C1jziT0gLMTSyIiSTXCwiIidacqAiE8TnQOEkXNKyKa3ckkkvtSLIgvbwscmvj8TMi8czi4qp keFiKkRNQ97E/ocSNf1Qs5AEuJuTu0a4w4eRPcHEB82hAoVtIENjwqdPMCuhk84wv6QUsSwrxfSh Sy010qwZEfgwRoE+NTPHvkhEEMaRPO2RxhRRFRUqUkclkVYcYYxJVVhHzfTVWmn7boutAFMQLyKZ Cra73cy8iY1hUTXTALF6nLagaqXVjdpsrd0W22jB+gtZfCBAYI5xpR13IAys7Okvb3dDD5+fcEvr pjXG7bJXffe9MDYP+aXURHmVhYdRT3BjNLWv9hpKTHirrVOylVKbS1JjfoPnq126hInRbY+z70qB 1PlKvYE8NhOfTvDDp+F9UdPvwHBjgiAAhRCgAl8ZwVWLZpYR/u75XIUMGANfo9RklahokWaau6Tt wkorwGgedCSqxvMoob205o67qYcFOGwANx1QbNcwJFtNoAEDaylc3YNpJWchMIBmcR3wllk1X6pX XLpjlrlKYfEO9+rvpA5s2brQyxgfYSHIN+yhDoXco8RjYm9yRiE/Ky3K2W657XvmVYpR3OSZS1l8 Rlfdnnnx+yo605VyHW5GM891c5a1S/jjVuKjS2qhPNfdbOOPx8gstJNM27HGFEdrq7x7o3NvNc4t vN1n91ab2bsHwgyqXInAXi3D8ak6+6796t1MBMTQuVcNN5Q5nrnkAHuolb1nnyD8QeMV++6BD3Od Jia30ZvX/roXIMUl0H0LdGD02Nau2hWtcqfiGQS+F7PLEYxRRQMg8oTUPBGW0HhHQ5xYxmOZtyzN Mk7ZCwzjs4a2tAop+DBBCsMXF5SFJXwolOBAcmiZHeQwKjmy4csmAzjgAYALRqOWw3oklVw9DmOt a9kTK6ctF3axLk3zIgrDmMSwGJEmLRuC0Nhyw8qUMS/wqI+MclW6lq0gfiY8DKjwSEI89tFKBFHO vN41mfIkJHOTcU54SBMkpWjnI82IDkE28x7z7RBY7slMHDP5m3mBbyaTzMx8EOlH8JBFOAaYghov Q5qricw4bbxSKv/2kUiKrDa2HKVxTkdLVtYSbh/RJGow/jOWYE5miMSspS8pw0rCkNKZI3zm2ZhE PwpRUza/6kqIXBnNE85KVrWK1Vm8+c0WvcqcOnERjtSZF3B6E5ukssuEUvQgbHLTnhG6J5REKKXF NBBAT9Jm5PJ5vEMVakxBOtTv5Jkyg7anE2p6KHdKoaaJcscTg9LK/sBUJkItak0dXR93EDiTbfHm cAxFU5MG6ryVYqmEopocjTy1KgVak5260tWt1ITTNNWzQNGcX3vwlsYLvsQkE0xhDQkWGaISi4Hf 6ta1pBpVqiZVagZ4IuGkt4R00cwA8OPMQAYYKfOUFILL0gLYWmo2PmKqrS9tX0IWRhqNHmqkQ0FA WjkX/peveBQfCMVHxfzyFcDOFUyHbFlMUrrWfi6xmV4bSs1uxtW9Wi2GibNl3+w2NCuK8YtOA+Nn uxgW7X2NbjMrn2mvSK+TzJRxJ3GagoLoM8bWtqVvdcxS/jUUjNIKLH2rVmA4hABZ1s18Sl1bB4Hm lGE2zoq29ZLmCFI5gnQQobdrz+fQJZXURad20YkdHVUXO/xMTFKsuCLljKve3E13tYMlCEcL+rGG wiR1UkzsdHsLXf5+CbdKlE/1AMWz3+BkJYW8JACUkFrleu98fIPMthBAhjveFjFBFUnB/Kezodw1 XA7GDhJTBz/O6W9tEBSeAlUcQfV5ZYCPC0DjHIe9/gHiQ697gh5wtafBauEMioKj8UqI2l8iF1lC GmolPshw2batsS4NFd/EmKxUyCZ5DpdFotL0dZT/GuaWwaMNPlYCAL1+TVocWFyTZWimqgW2ZbI0 c32TKFo6e9bOZIyaT+biRKENEACPQ1fLLEgsG5LWJFzTCgAGPTlAnxEAlBWokSU96YdAhqNKAawi M4NeyyAAf3u9ZG/HIl/V/Ey4A8k0yHRJ6QxR50kWDDRLQKeZRCbSlJGBdS6Xs2pe73rXmSG1prdz HeOkBpC+nI6wrTPssHWZ1c/2sk8pBdARUQjaAAsnPXWFoWzT85zs/LaMNjQrmJLb3L96Zz1VJZJ1 /l57X852t0aGUtFnUrs68ZZfwhZ7yb8CjUy8eduk2sPIoRAcc8P5SXkKriaFgslk+mYomkH6QO6o RzJmDTN9H9ur/R4Pt/DGt/i4JaMAittUDrrpiNq5chwljackl3Y+yYZhkeNtcAlAkLE6SeAGC4/C RQWVbLn45KoSHapHf1hpESCFoN0kZ6eNTCrxI1gHkrTqFxceVyMNzTyG3KV6pLQYLadY0sxninIO sHf6mjJH2lLD/6bdTz7mCpa5zOsw2zjivDo0LeJmLWs+qvf2LhM431nLhn/aTuMjQ3h5FbU1e/wV q5YWl7DzWrriOW29rqTN2/OMtpsLiX9jWa2k/q58gB9uVo2Lt/Mlt/XRYlxM5rb1PoKcO5+z3DLN 9PBjZzi/4zIJd3HZXaWENyGzS8h3abcSSVVUuuVj7+9LfDivEORgV6SLfQU+30RvruOdB789/YnX ziasjW+fbt0mxsTbM12Dm0UuiEPaMWHnxP205y/NOyePAT7dTLWArOE5roVCi9SBM624Lkwav5NK saproKRLiRoDNBmjrLr5ijJLMQiEQK3Iq64aQLSosaYKP36xPdtqEMRbPHGRllxrLTaRshtSKidR H8uQAyxrCDMjQYJyrNDYs8KLojSbMhUbiD1bNGqRuJgCrSQsldASu6mApT1rNCgUmlz5OUIj/gpD q401mxwtQhfGW503q7ARdLcSVCmN0LVcIrYnMRfqiq8QuqTgIIhUC6QBlIy2wwdR4qUxHBJXOyUS i7VlmjWLozZhOioD0KtY2yVcysMv86VGBB47NLbs48NlKzVmskRiM44hEsNesb2P85Is+S+burBt mjZWgw2Qi7kMGSdt+w9vmyd5oyZRTDmVC7dReZFa9DJM6bZXTJNNDBgTEq6BGghWI8OrS5KC2rfA EbjCGAs4jBx7S4gQgbj2AEDuMDh8GCkEpBONGsVKQUY2sR1MgrZiXIytExCScK3nuRqaKopzc62T E4h207+YijRyLJtj3MWrK7eVE8VlASGn/go64Ukr/NMKUWsnlBO3wDEQCTIA9wsgHzGKqfqW5ho5 o7swMpKtdqOVfvwOWXpI2crDBnQmewSYtjqUumIUSLG8I2QURuCYuDoURxgw7XEKvyqT6uu3S5y0 S8G79vCRzEjA7LudLWm8dBmaNVy9yMK/BxyY5VIU2vml1Mm7opwlhvDBw5szFFRCLxMzsphB7cMH luy9mtuNLjSAe5m+cKw8MSRJC9GxFFoXQqHJYASPysOrDdKeeKgxyFujDRRJX6SQt0sprNEt2GO/ 7nOv3NMeyrlI9OGelECdL3QAHMsJFKMNd5A+0OEu0qEkXEI+n2Ad2SHFwHTMq8Ih9yEf/qhzCk2k DeJJzOp6Pb+4Lz8iySUxED5Km++QrbfUwIQ5TGDDGNMLQvr7iNOCDCPaLE3EK8A0uQp5Hvv7MYI4 pvtICdOzpI5IB3gYoPvTCgQCyzX0O3ljQOjxTQ4bz8oUCXVgFAyctxNbMZFcQHEsOaF6odcCAB7A HutMz9p5rqEQxJPCK49kK+bkiDwz0L4szK4gzBYMOlejlqfrwlSZQcHIOxEiyS87Td8DAB0wSqAY Jqz5CYPoR0drqoh0mC/8M/OUz0tSyG+5FwOqEvBoIahwtBeNM49KvK3UyvekUXxYgmSTToYyoyqJ uSFsGfzcooocCTLKFRHkiLYsSfED/pJagkQgZSTagAw7PAW/I70nKdGPmKRj0Y+B0CR8eqYLnQwW YKUhJQhqqKVr4BhjCrzCm4wfYkDzSLtQCqsrWTQwDY4rDTNFepLH8TX46DVdM1RjNA9v4FIhcsCP 6IY4/Qg1fRIkhZnp/AjWjFICra0vAyVONdMjixGay8dZnA1oPFVZVFBV/Sf6QZJjgUWQZBRjSB/Y asdU3KcNQRIo1cOu4yZePR6CbKyhTD5BdbMVVRMw8yN/ohA3E1YvQ0/HoMvACUreUzXfU9Y+4i1Q rRSQIxtgFUP94xBp8agOTNJ8KcvBuTBSSQp+1EjzuFPSNEhYOTdZPBlk3YjEYVcu/hLTvUCqmiMu 4Iu2XVk5uyjVgeBMblXYhS0KyPjNAP0xMvsxaQwaUPPV+pmUsUO7ay2o6wycZ31ARrlJGZOvuPI9 kCU9MOEdLduZEDtMqqQ97tNYQ/mIg0kJRuE0+ry25VEp5TkShg27nJC6qTGsZbGp57NYikUAyoJM Hy2fhBjStTxZaaW+FrNaOpzaxiSI80NQ5LMMWlCTMm3NzCTNf60+//Ot0sqbrA1VoCVGtxWJg82J p/sak1ILcUja4WmcoSU5cASAqwzY01q/YapEne1FcHS90vq7aX3MwyVLcIHJ8cSgkuLIiXmu+cTT 1XSfHYhOfHABreAxTX1Dmduy/raF286DGY+9vQw1PAAYMvPYMyFQUYoVSqU5opt6WMN7xkLLs6+k jilZUqyM0R/R3KxBFTwDCiOiPE+i0QEygBlQUXSt09EiCKgFXeaVlTlbWHA93Un7svkwVAwNQ+DR vb8cXaFIVGQyDovD0ulV1FLKWeopRBVK2bNw2PirjlXKDEjqzF8iiIspNtEtpQyVGo7ZCUpN3zQ9 qoFQ0x7N1O7t3nO7R6+bKWtrLHyNjyJh1dhg1lVkN5GADPbN19GUH3ei1+NlWdpYv2pAVw/uL+41 IRg23Q+JCUeiOk4EOxlmRhIOs2c91eGNrvF01gUBO+zVJ16Ft3u9NBM7vho7/lHi+yeTZZJ/GUln GtVTeblQcVdT3Uh5xGKw+6Ys8xbKBOIu5uJSJSVxFSecepuX4xWChUc3prMwvik37mDJfafflBVb PGN2zci+lWNmjFYTdZ8yY9Y1LlKaQYDXheCLPUaVuLWhEMu/ApReqOSZFNKUiSONjUuR41jtZUpA wmS5nA0oxVDVhUqEIVa52tqUQSyX0VjFnZa1w0QNZhTvfLu7ApLYOitGEUSrQeUeHYqPAaWOvV6e QbRDQbB8rdqoGxecy5WrRFeHFWDBG9/XMB4dNhuI3aK5OMSrRSqjWFyQ5Av5A5MFgxz+EOfTvFcF xONlCqImw9ygVZa3pLjo/tHCxjHkxoG1r8yJvqtbn7ybtV2huCEIicvAs6PIbJXGlehnBK0SiLRa RvkJ1WRcxBXAmyjCFxSeDS4K17xml8I/bW6N24TWlrm/3UDnpCSweLxZ/U0PKv0NzDRbob1m4qyW RAHQ9jHeSLw0fylhF+aOyauk+ZMx3eNmDnS/1ZueKppd7+Bpd4GYHhu0hI5q6RlTm5JBIDutGvPp 1fKkApyMrtbfvsXolkAAzj1XtQEAdICAjtPiRE6d2TMMAdnVXr0mSGZd5GUuNkIKUwFmk2PnqTgm 7W1fEb29FoXBS0SyCl3W+wXHrf5Xk6Chv9ZRvn6t4PGIKZhKt3EwWCLS/uBiAhV1Es9e7PdN7PoE 7MXlGWkZHHSZz+WM7Ph4LZcbSA3eQdweM0hrZH0JZVcai2IGplra5BBmGYIoJknipJB04DAda+Bx Bl86pp1oI8iI7roWRurs15kwmY8AYEbsJfLFuJlY5lLCQ7OlRJj2U9JIbs2O1wG2NeceblYC7Zv4 UoLgX9QmDQC2DDwUjuMuTh8m32D+2Sr+uh/GYE2ZJqCCDP72uMLALVw1a2ktCh7mxXqz8AKX1gzv bU4laYHxcCuOCcdOCRHeYRLXp0Al6Q5ncVZDRz1KlZ46uYKt49CiY5iT433McaFeKTXmt2p+TzRW irbwY6PaSBi5GQJd/vFSbvFfjGKCIAWwbjJYri6yAxObBUtS4+gLwUmGsfKE8rdq9bepRnFuwtAK jUg0P8+y/iaIoTvEyRnCqbEr9YnybnIWdzaGoGIDRwxwjj188Mjq80OFtio2c2QOVNfqozCtCmep nks1YxFKkto9AuMjO7tk7GWs5RCv/MBcwQe0ZGnsUJnvu/NS715F3k06xKt9rpJ+9mpZpjK/FT28 uh6oa/TI2M0II41ggzvkMcl8FN+8faEUfFk/B5faSb8JTYlsNPXOW/LOC1HW7hYwut1Cp1C1q12o gWgvLGfBFo/6NR4BP3N5g7Ie1XaGrk9IbxuHtXWCfm97evZmH9CW/iIOZOtfRyQPUxvTO3xJuEng PA3uWgpTyICO9RXw/sBu3w7w6jCZgp4fDFI2gohfhWgy9E5wecd4rlPwVtwnM3/3jFdQX2EO2wDw MxHweAf5Sl8pLmOeeBtGoPr4IuvEjT9pSbHzlMd502XOBukysfviPk5FUv3jAMqyRClyCaf0oA76 M4oktiHw2kL5nC9dCAfMIZS74IuJI5zvlNhyYGGUTS6cNlpetPtJMMd6PQ5XyG6PB0F7qXd7t4rg nV4J9kxZvAIrDaZppuBNufDqLVq7dc4Whw7p3o7Pm3/7cTz8Q2fyJ9+bq6xdABhtYdfAenmXo55Y ELLA+mv1wd9E/v3zD7xKrmeL+k/cxNGf4oTfZoKobNEytKYt9g/sXWspwpbxwwSdY19X+UxRR+vI 3is8uzI/8MXPfYmwzXcDP9N/Uo5XeZa/toGwheUG7mWLeUqyeLIgHbodtrFwBAA/eAId955MfIxH /uG3sH1Kcsbi8biNRdwvw7cNv/Gf+vA3/4uf5xOKCU4gSIQVR34qIfiX//8HCAQCBxIsaPAgwoQK FzJs6PBhQQASAQg0gGAiRYEYLQ6cGFEiR43w8DXESNBkR48pJS4EMDIAPg8rQ7K8eHKkg5v4MkLs 6fMn0KAKQwotavQo0qRKlzJt6vQpVIguR+KzoBEf1lY5Xd7D/uop50Ws+MBepFqKLACsZBFOxWo1 LFZTYF2KlVk27lqDdImmxXvXQVq7d0/igycvZtTEihczRkq0MeTIkidTjkoXwRR8EDRKREwXMGKR YzWqGxvY5+XMELZ2lvn59N7TbAt3XTvRM07Zf3Xmrez7N/Dfj4MTL268OF0ASzQDporbND4SHXGK 3vqRYfItzOlidY3VAGx48eoq/Kyz+18DoXdPJ3nc6PDg8d/Tr2//fv35JeFR9Y4P/HOgCXbZVf/p Zl55hqHXF4B29bXgdwDoJxpNEXpm4YAj8YXVGoLh9yGIIconInwkFjUhatR1JNaF0DkoVi9zifUL Wmq1pGKB/uhddNuLWfXW0WE25hjag0SKdRZhvvxoIpP4ofjTk01KOeV7fS3pUE16/WTlSSgVWcuV FVE5JpmLRYnAmWWquaaJXEaEDwZHtaWjU2nWySaeeeq5p0922jcRXyrtqFGXPBHmoV4osaWhhAT5 WdCjfPoWqaSVQkWppVJyZ8lcClb1l2xpjWdbawdZVCRiPwIQJD6khBlUmphmytSrs+Ypq625HlSa gAg0R5IemhmQ23oI3kWnUBUiCquuzZpYq7PRfvgjrgVRx1NyXAhLbIajXVQaBLodhJGh41IkrrQG VZsuu3e22+S6R2EoUpD+NdjeigaupxeL+76p77LvCjww/qQEG/ynWGCK9q+ODyoMF7JCPYjkwbnG q+fFFWt8VHwegUchTYomVa5UJLNl8kNZRlRZxky1vDGUJr7saLOUzpwQPl0hsMdY6gX5VVjjueUr Zw0tmVYCW+KYkLFi7uctYdAmOyjMJ1Zd9c0DZf3nSAhou9penw4mx1hbjZR0S4LeFajKin72MdNL 79g2SO1ZR2FKVHNGN9+GNn21pVJPtjWJhBP3cnJEdv31XQBIwRxcr3JnJHkQU8ydVgurdjSj+eKT +bFyLfw1aOT1RddbE6ON+epU1YJ25IDLfmtPhktmO2Nc4QP0YGNsB489Q0sVIV9nFxpgqCOhG5Gn b82N/rziOxWLo0uHLecAa2lJBwCvaZmwtvcV4z47y8K5u9D4lIWdetdq/A4s5HozLVYlOp0XoE0i jVdJmJfNUfbkXHOt6ewPLV3Dl2qOhY/vmScj3HOOdPJGvglmClfpK1FkoHWmy5ANbMqLHgDoADm6 wO5kpSoQH+LEINjMa21iW9RORAgB9QAMLvdyofP6ogcHKQ9INVxhi8DjN7nVjk0X7BMFEXJEWsWM SRkTCwI6SEOsIAlVAxISW1hFsatcjkU9+pyMSCKG+NGsSF+kHBgLhAD3reYqmStSjcQSwQc1I4z4 KEYcBeckoJxpiUn8o9My5TeUsYVjllEZZ+zkR6YN/pJmqJFKlyCJH7Ulck0iMyEhs6ZIRwrsZZss SfSklhY9SiYtwWPe7hJzM5eAjkESg1pLQokl9yBsVDrhX5Pa5pyAtQcfMQKk7JIHy7jtJG99K9Qg EWmuY9qtPerhZdHQ5MBzJbOayLxm0XQJy74kJGTaJOSbeDhMGIZMfiI7JyWXac1mUshf15HmqaiJ zb2tk57GHKK3zEMDzejtXIcBzffsCcyB8YhC4AyLPQ7jH9TlqJVU6USnvNK/XfqKO7isjjNvZBrf lY6hEPtlWeolkuBFz5dbwYpH/3YReTxsNliB6FUS6s6PCBNu85PpQkeSuoT10hS9rIVGG0YV0LFn /m0aBcwYHUDDlIrFoVRhX5DY55fT6ZQzGiKNaUYyw+8gYAjx68wcxWIMgi1yTAEUyVM1Crvq4WMN ZXueAF0EvpkWSIhYvSF70kKViUqPo80JXgezFz0EaOF9XttWDcsyHrd6cJyrKgV2TlhUGAo1qked zmGoMMJSveY5eOWXgTgSm+nlM2f/TJD0GPvXtr61oGN7nwPcp1RGKW4kjJ0iVljQuKd2dIHtqQZa OhNQ45ByoEfpXnQ0R0a9HHBhy2nsggb4LQiWB0akYVW3DDhO5gGvK5pJwGWeq8CAJke85dUMW7FC 3q49DmxcZd4x+EonAhGzV4NJIDHtt7mzNlA0/pVzKT6qCLz/Gkul3GVRYxEwRuiCcL/KK+wMPTVY 37URQdwj6Q74eReQvqaXvr2UcT/UwuRA+Ep9mYI4K/TDea3woBhpkTmvKhoX44NV71PxDV0iNB56 1zWHAQ9NZJwvVPavhXBB8fwSm5zNJXkMKc7XvYC4vQi5uKAt1kmQ/xNLeHj3xj4E0HpJOkdPTdlA 2BJygWSkvOf+Sj1WGa29TlPcSpVVXZP04pDISy5s4cMWJxVLHLESVqz4OUcP27NEWFVHQ9sRjysF 9EpMwqDkwtFzsmRnQ7fCqrykRUlpxg6LhhsWXhRZjkOa455X5OlKD8kuU/T0R+eCaBZxGEZ2/nwj i4JLLhu2CItFWu82l/agX7I6yYKOnRKE1S/y4nnOjiGOs+HFFGV2RCPykKlYLHC0gxYskpbOdrS1 xjAWaZvas9RyvjhNyEaWhNvlQZmLbeyWK20wnYTBtluEaG7U9GveDJmQmz4dYpcxps6WIWJYxFkr Az+tN6NUjEsO/V6gpAU7xuOXwi1OyxQ5drvTxqfwKNsLtP7XfIMbOMrvkrRyCXPhckNnlvo743vO c271nKwr06ZNFMl8RRm/bM3dhhObJofmkQ662uhrnsJiryYfW5VcrbQ82qX8YIjrmmYrfMLYaJTo CYsoPkR33+3UGEI7qSrEgOpftR/LoY5l/mmp8SFxqkQQhoTdbE1d3LkcuQLsfV9Y1kHTY7gwFSt/ PxasMSdrGb9GqzD5jwFIxxl/Tb3qlhfkB8VZ2ZCPiz8htzK3ZkxaA44nqYKNa6/gjNGI2IOoJhy9 RoW5+Rd23rJ7g544D0Q9z1P49KDyLAvxnByquFd6V0zjdEB6+cMt/+CHURhdwjVY/HaeJIGf3Pak G1K5a9d+7f3VsWXOPezWVS/yUH7159vcuI1n5L8XJwKYsFz98nNyDNQ+d5RP38Gw2f67caA6UNfq Tc7/QMDKjcTlnAvsNR8DWsr62JAwAQCTVR8ALBgNRdmIacgL9cUWPFldBR8GhhZhoNvq/rkUXvWF k6GWm0Fgg81fe1Sgsq2YkmlgDmFFB+LNBwYRCDZKul1FV8BgY60gEGVXAxYhn3ATG5Bd5ZgRPXkE E0IMGsEUXKzRV6VRsT3hg/hUphGGqwhcdZVcWrgeaCEAGShhw+CZzTkhGmLhkYQRGVYhrkFaqzFa jogdgLWIUg2CGUJUGIUbwRlh1fmhU0Bdv2jbu3mbxPSEs+1bZO1bOc3TrkEhuAHicRgcJW5MwCVI 4k3clnicUmQiI91PnKCWdnHf9h3bJTqEJaai5cEctsAcTeGTbXAiJukEt7lipCVTLPIGRsVA/WHL VdjS7zHQP4UPKx4jMjKLxHjKFy3a/pxsoq9RhTFElMPJg6MZ2zWaknolSH/kiKONH1ZMow9CDXdk Y9F9S1bFoEVkWIWpCBCJWljAo3GsYjI2BT024GV41WY8T/ZBR91ZGHKN2dtdIyPpnicujD4K1v1B x3BZGLGUmVgsJHcUnzEWFXfAmrURZMzcY8HV454skR8RzpK9T3Sh1wL2XM7dREulH2lpnOOQJO6x E/eMBj5Nh0wtmGgc3mBUh0EK4nuEpEcGpXHomNkl3PvhSw4eH5F94QlixRxAU1l04xAe5cIkZQ+G xWEAQIk94I4wnpExXLcJpZkoUWRwpFhiibAJn1qEilhI4YNI4VVwihca276kBVxW/t8sms5axsSf 4QMn2BHv3KF/5ATPbNUa0k8eoYmamOVZagxjchL6gBbu2Nu4tJvJ8KDKFYW7EQrEMeJJWKZl3mLd rEQA1FxjnqYgbdxT0EUjgJLk4EPwcJo8NEIYyUrFuWZisObC4UMJjYscSpB/0R5zYUUhgB0lgJ1w oqZykslj7IXRCRR0klOxjObMHV17lFDTcFMjFg8JEtMrmpNElKZ1EtMzjadRmWd2NtfbJGHTZcTT 7SXqhYew/A/2LKd9Uklz4sOOdRTaTUyn2B45jWCAyYhIQQxcFhgRrcpfKqZvUpGM7Fh12UNX5JTY PAjvzInzVN+GvNSK4FSOXJRF/jonqAzddxiAak1enCGPVq3FY94nIKGITzYcZuED2dSna13GifKL TJFKTx4PEZZgR8RDKfRmovSoWtkka2mdML1hffJLPNSGj1ZEdkqWhfVLXE2kUqEUaXWR6SjPXsWo i4ZpZewfB8Fk7gUWDGHnLkXU/fyoDZmfHbJktzyN/ZQhg+VeErYRXrYpUganm2ZmSHVF792FW+YG xpXFPYxHAYKpmDYqZPRFCpkNCQLRKKYXVIooCx7fHqjQV67fFPJikjUlPgzCKCbZplYnC1Yql0UM KuGYbAAIPqRQpvapwInKTizqA45WqSaaXGojAAiCCzKns7Soi6ZFFeWapf1c/hPexm8+IRfZERq5 XlpcKBbRpYdM66sAiFyUjq+tobDNWrOi4Ul1IRTKBLeK4bgNJgLclpXmSKDR1R86qryODMV1a78Z oglRHLVdJrslIpaAmreAh7xVxT6mDGb2hL252yUtE9uw3GbOK8RaxkFCBV3c5SwJ44r05qqIXUp2 IrQ8XCkhYHHZKqepZoxlG2oNqJ/SaogQa8RKyjkOCjNFZy2KBkwMiKDUhCzaIqiWn87h0y32UDbd 3MIyKHSGDc7GnAP17HkO4NDtFzBKnWfwCgJcD8Qw6ssuZ7XYKqWh1HjsFFbEiKSy6k0IDQfgYI7Q gqTaF6AGjVC9HUY2KPKl/sU9HIY8juB4lBldfBGHeV5ylu3bbqjcYVWWNuT66RUqSgTjdY9WuRKb Td5C/ko78mbWVm7u7N54fB9cLYzmVl9XxWC1Whly/SmCvh36LdMCkiKSMtmNdg3T5Zf8bUbZbRNY USXOuWO/9COWng5pEZuXYsU+giVZuajLGhLH2Q/pnFXjSB4Fakdj2QZFdRbpHi4tlsUWySnL2p3d 9J8AJgc7cs5OJOT+KVB8pu5OymQAYsWCSarbAcZIHYZ22CjC3Ufx/pEF2YdFcASm+AkHipN4zuop Xqp4BC6UBZ9RVmW+pKAaMa1kHgiHQCVcKDCmTmUAcyMB5yCnJtb5Fogi/uyqkCBuCN3Yp6heWmRl /2Gq5SJRWCrFEdVvY6TFLWhauJoazqVagSHrHJZrjsRtWqxkWhRa7MQSGN7Rbv7dFXor7e5agW1a suawfxLG9YKW2eVEnuIWGiGGFd/tRzYRF3fxv6WwMjJNvZKj0JDbsEwsdqRMZ3qmzVVmoVzlRZRx ttEQ1tqvqYDxHcertNzmJ8IDND6sb36syRqFNnHecO5Oye7qSNTPFuPxlMxHlLgwmUywPKFn0qVl d9Ys2pLnc7YxerojdczAdkQtfIZTO0GwI/OJJC+nKdkt4Y3EF2UjMwKpSBSDEJvUVVzbh81GovoW VaGROFZwV6LbTEqu/itNIJVuHZql8rus8gRdHTl6XrIpaXINRuxOlkicbpGaLwwZUPDo4+3B33NZ se4SH3hp6T+GLXJO2Y6mckgAJTNj3salhpnOmPOWznCsCrryKQKjlv2sr/ImxzSvrUjULT6s710c mgK6xujSskHU8Vg6s8F8Ujy/sA16IAATZTW/qd2Y2I85MD4ocJLdYNNSsKVeJQhz73/oYNfqC7BJ NGQ+BUwHUuBIG1m2S/r0cF+SI4sIpMOZYhB/IbKkhUbOBvThMBQCG5FKMR4iwD0/SDaGGhRCdExX NHDM9MtecjFpyVBsSb8uhImNYLRRUsJ6cSOKJhv7BlVb9SMHU9nV/khsvsm0wUNUZzLCovEGvfWK XNyKYG0h/20vWaw9sjVhP2rSaTX0BtukVnJ0nhMmP4ljb/W7zewG3+5igydlO2TPTDPVPF1D4y8r 5m9hn09Vo5IveY7+CRv1BonewnKO9C1JKSuQ+MItR7XQaLF/uV+sYZpFzi4DoVR/WtfKDrNoMa46 GgCFXWDlnSVWX+K6+FHohTNvp2c0j8TmAIYyk8RAq9SqrCTq/uP88uR0GimQImjwcPbmjp309fRu AS/hlRRtjnbKNbciMgb+UVR5q/Y8L049N85+VW/EyVeqVPZw3g35TjdfbzA936mkAgARjJA7qoPQ 3HNCb8VnIyN9/su3YrBYp3YnCqMgRpv0ABcLVixlqII0kpngDpIWd24oGYR4BhZoEINw7+WqkRUR /eK4huv45Q0btDIbVtQ1UF+FOB7xEibo3FarYBJGnEoxxx4mVrDCrc0i0Bj52+6mXo7GU38dfWR4 1Xn5MQKyWIem+og5ErmNks8NTa35G7f5SrD5m1uy07UxpyEdIt05ZU5Snrv53sB5n/O5mss50ZI1 yZQ1cB46zWqmmetrWhO6wx6so9OU4SDNwZVc7qT5UkxOhlYfUdHFFnm6TmhhTmoXZLVHqYvG5VDF hTqHVOGD2ppy24FdS5UuVpwtiWi6Rs36SHT6SHiCTvB6gP36/k95WKuXFD7YumWfrvbkt2KT7U13 c27iNzHpOi6Lhi23R2pXe0JH+2Vn9p4v3S9WW6K3G2bv4mQzYjEbQPxit5Sesrs3TvSUpwv5b7zX e/miHgKwIx2ji3ymXnQ/CDUaQNZ19XV4uzLxKLoPHdOxuzS1k0DKO9K2Ewi8e9j0o7+PhmqccU6I coSNxgqQXVw7dDwyu3blhApomMH208vteRqq7sdvhtM5kAbqlgvRfNjYvAa+ND7gPD7AhM27DdCz fCv7MnCjEdvNCTQ1njoiwINDlzYL8d5WVLP6LbKbHxriZfRdIHJ7WUZ1Pao2Dpbx7AsWsIqifDKv eIr+qNLr/ikonV2F4nd6VX1E2FjS54bjlehTo9mH6z3fe7iQQV2U2b1mBIDgh0vhEzQ62tVKfBjX GRThRt/w7MRhSBXRDdixMw3d11f0uZIOcP0mx2zThj4K29hvq6VUq65/Pe5tCOTnsr0tEl9v3XuK jMeglm6+B6tU/K4Shg/ilpaCEOgHz66aBf9exdRphUXx+5cuNDjswwXHwr1ZQKva677rWxz8miRJ YOZnJKRmUn9vTVmQAL/vB3/4G39plb9/rRr1n/FEsn/zkxDJAXVAjziP1Xp74/6iWM/vFOhlsCMh lwZAAICHDx8EBwAIIiSBAKE9efgcIGCIT148iBIRPrw4/lGevY0IB37EFzKiRJMGTKZUuZJlSoH4 XA5EMKTgwZD4TGAcCICmQZYA1IVcUpPhwFoRXyJVCQBAy5gwoxBNavIlgKE+fzJl6rSoUKkDe5Wc SnVgSZ0bz5otii/BU7Vjz7bFqA4iQg9F71nUUpBhU7trD+K7m/SvS8Fk2c6F51EMX4xNnaJ8iWBM TYQEj54FMIVoVq1cBTrEt+UrvlZiB8pN65Y14rdlXf8VeI/gXggB4GnsaQAebXw7SqOGaAAnYOI5 AV9dO7OgQI3AsW7lupyLZZmkIawFUB3rSq3SW4Z+KKR0M+Gv0QJuvdr19JPuQROcclc7VYIGFOq0 iG9h/nh8Hq2yDh/iLJhowMLUA20glJ6KKST6ftKIIAi9GwkfALCzaUB8ClyLwbM+9PCpAJ4K8SUT FyzRvgMPa+hC7IhjcbD7ZKPxsMfgOQ3HATGycLPS3DvRJa3400kjCpeSsMjw5NlJuRNvFBIxFHlE DDLNVOxRxokAhJGgAYMoKMYBezpIysvwKxLNwqrizKAToSOONgPKTNApKTW7yqb9+vvpywnv9JG8 NxeMMsUpswQxURHhazS+sAwkyKzLALUz0C9nvIgKy77MjD0mv5z0y9OQgs3RIP/MNCI3x4QIqT/F ghUjCWMNVUtJZ7U1UrQoLaXWSint5NcovxT21mGJ/sW0vTtpvRUfWs47FVVlL4vIyy+zY6jZXYft 1llRCTJ2V/ooTRbQcs01ZVhyU3WWXWU/jVDXceOV1k9qcVVDTFm5/dZfbXUt963PqCL4MfD6Qrgv 76QzcVnvhlS40WrtjS+9ijHOWOONOe54Ylw9NlCtkEku2eSTN6Y4ZJVRbrnjl/DxFGYOzwL2pg4t fcm8twy9uMf+ZsbZ5BBdLtroo5FOWumlmW7a6aehTomuwAYry4CreBNzUw0jYoJTkC21j1xTl2rR 6spGjlrttdlu2+234Yaa6KPnjhsBG6uuayAIstaQ77Je8mmsExEmcuwDIzZb784YNvi7Kx+3W/LJ /imv3PLLMUe67pSoxIeYUi8M6badRMctdIIEN1WgkCBdiyDkQrOZWtJRV82wdltt8U/+0s7cd7qX 3vx3koUfXtrijcd4c+Gnzm91gna7Kc7FCFIuQcLUBLxnBJQoyHQHiKt6v8a6ey95lJE/X+Pi01ff /ffhbxlvLi/EmqIBpdh3zgyZOnRN3bekoQDxbX4guZC+yve29lFugS7rXfwgqLYGRpCCK8JHMdbl LNj9KScAUNK78NG6cgHtTySEV7mQVEEVrpCFbptgC2EoLYmdbIagCVINT3WlitWtcDrEmA9fBkQa ChEBmyNixFrSwCOuzGCg8eF3ItaUF55qivE7/kAMzRc3lrmninBBVQoN45EHfgyMLGmfQMyjpSra p2RbpCHZvmgxn+XwRh4r3M2mAxeYyQyPCqSi3daIRY5BMY8KI6QTewgewkWxQuSCjA5fYjutLBBB h2zYIw+Zs/vc8I5VymLBcAjKpTQxPE205IIgScqJOBIj60kYieBTScdBBj8H62S01uCqS/ZIdZIK n3oqk8CNBfKTmSPm+Y4pw5sgxz9lEU4Ix+i6w2jogqgJCQdqVolJMUVx4KujHkPCiWgmjGau8wRS mhc9j8QsWkuBBwblKML/vE5BX+qTn8SIGGjWk56u4+ODaqajg3WzkmSDma8KyaaQ6IgwdBBT/m7C JZYzAc5q9zEAAkE3Mm7mjWvj9JhHIZhMQdpLe2VcFgDcUZc6VuhADMLe4RAAI1T27FwVhdDgZmpS aXJUNjvBh21eWq+JtIKT34Sjd0qaR9Ul1T8HeiRTJyNTmCh0diq9qUxAhKCf3GQhQa0KPvgwnIpi YFeKg9mbEmLU9FymEZoRHUjhtkCRmpErcI2MMZU2v3rKJaiXitlqICcTNzmnqhBASIH6er2brHQp zSMoPsgKFHvcoyMPtaoakeqKaM6sT5OxWJp0ej3H6pRSDJ0aQxCDAPIJZD+P5alaUSMhk3o1qc7x jWWpdlJ4UE8OpTHEpAzqF7OJRhCMG+lx/iGIQqXCRI1abVyNtrSca+HHpQXU63KaW7z/ZQq0HFkn bitJEHfOcUXd1RIbSNuuelZ3qowd5UYNZN7JOJSAU2Vvd7eb2ux+Frr4vR+2+hZekTwEAHMQEIF2 dN8aETiXwkSu8h6stu9E6XHCxcyvdCHRCv9JZp2qFWW+xk53jThSCI0UtGz5GeVGqsMbUpVWESJP 8t6uUj1qXSnhmzAoUqrFItbxdyTU4xYjwE2BIYhAV4zCVxHExJRSV4q3kuQvnVONAPRmevF1kWDm zgNLNk0GIxxm9wXZy3+ywKTkaGa7tiyUzvrSmUGDEpcucWIoe6SEIIUugsA5lrvb85pL/ilDexnw vxbI1nsTOVcx/07Ri36Ze9GXMaJ5MY+QThKSN5kxhNDQ0nFz6VEj49ysPGXPd9rtCUMiLkevrdGr Zlom48MCUEKOlphU5XMLtrBZM7JesF7lY1r5sPhyEpKHQiQQeVgcW+Y62JFLKK11vexd91qVbHpk a6YyFKRAjiF0wc9TL+tqcY87h8tMqLJhBk/CRE8jaXxg7PCBzcu8pEMHfWYpphTYIyfutYdLm0Dg 6deAI8Q390Tql5iJ4yWlG5110UFzTr1PU68zUy85XDVrNnBU6tNT5Hwt7M7UvL3FCCVx+pO6r9u2 VnOR3JIWM11+beqq5varg2VqeAQr/iB8oK1vv+wrYWmWlF/udC2G7dm/fXynHIdtvLlVyYdmxlHi dNWnmzLszbcKE23bZCDkC7BrdRISxEL1Jv0ua97OmoB5M5aqX15fyyu3cq7I3W4pb3pPLxsa2tAm Ozat5wA1xD3wFmcq+kbAVbDXQe11ULz5GbY7BYpzeGH3Uub9yWgB47zcfPdvKjX4eAFABAHFdPAc Re1ZEJC/q+d9b/11POWd8xA3teUlIhSu2eGee93Tq6n+3ZKLAIx1pB5JUc1Vk16rspvGEx75JSyv K1maps7KB8sE+Xx5Xa+7xXbe6KSlXs+gbt3oVmW62QfhN0HmIgwdOOjit/zuWb37/uUlbcKOjJyU CfLk+FrZ2vc3KK1+pcdkDFg8jMQm7J4QosU8aoQMo+M8o/92TL0oZeDuozhiJDBAzv94ZVsiZQAp 8IS+RP+SzHDSTFVADK2Y7FeMAdDoLtIkQngAjXPGrQWLyW3IrKz+LNAqRs84hOR+YtSYTWMgB88G hs6U7oeIEAchywGUgtgsRkkswAff69X+JPLg7wpZIgaRSWlewgEnBtRSBmx6BKlkjJgQYgEHwgq/ cMZ+SAwngmFIrZxkriQkA45eQg0bpYFoEAv5sA9B5JtmKdqETofmxpJqTZQWxYsuI5aYgkroDEpu KhAVyWoesRGDkNaeyNgUS8su/mKXIkVUHgK4XMzlpuNDkmkP/XDRWg0V20gTYUacDITeUI++3i2c lswADMDAUJBXKMIj/M075AEPaYzKJmI/7qIWPcl1SGHJZHE5rA4Z8eHJ0ITICkLt5unQCO4hHMl/ KGpfBsQAMiRnDEoT/+jp+pAVUzEdLeQjyG4gGkyAOo1wkqWgEgMxVMOzkuQWxokEhSqMQjEtFMoj bAMeZyuntIQex8uCaup0aiJG6BE9YkU0tDAdKTJk0PG4wIktqIr0CMUj9G+ryg4x8E3Y8PHxYiLg JK+fKG8OqcKxYEcmnpGwvDAmZAsxiEq/cJI9ngcfeit1olEUIVLYKnJ4LnIo/osmmfDkf4DGFQsF m3pvEjcEZ/DEQObj+YSybKLLQLbA+zwpvxjlD4UmSZxqSuRwKi9jDW4q/eYJAIJJQxCsL3aiYOKy +IxSguqyIjlwAssMXKjRIL4tAgnCF5ZsVUJsBQ3jFHgG09iQxHpEGP8lUgSzX7TEAfSkwiwM4yhG 9HbxxiYCJe/F+kQGAWyDy8qqq/jF1YryaVLzLiXn1hjRJVrrzcTiaarNCAtpSEoRVHaHzzzjh5Dt TmhTYmYokHyNNe/KOI/ncrowBiktUJwmI9bKDcutxcxwMdfCM88IDM1x01pGeBDCInjGOo9Qnzrt LuluNZFzvQARAKDOL/St/gllDtoSjdlSKZGKba3ssz7vMzwZsZMesVA6SxLf8/Rkrj2jTcfok9ek LV5MKZMcdKkuQtt0rZYWND05ZhXlhq5ekIXMMiTOyciacTLe0T+Ib94Got6+BKHk0SmB7uKKYT1c RCzUQSP4iCA0zmcScLNGJVaMcb1IjaES4kRRr078hPgcciBAKM9mqt6+7+JuTI/mSUbXSVxghhj7 p0oIY+Qsqi3tLXjM0W5ACj2vUExXYh1lNNyeh3wIEgHGCJwGgju2radoYz8gQO1k4uFWDwPtEdsW r9vGD9Q8iDM9A7biA6LOw/FiZ+u+KrTAySKsjvbgi7Vo4yEIBSZY5eYU/hFCac4g0eUusuasQFRx 3i/CyFTlVqIFS3VpMtLpokrnasF2sq5BBpKzuk1JKhUARtPvGFRTe6omTZIq5GEZ+Umtkuhhfk4m etImHuIm66lBsIPoZuNPbjVXw00c2THv1OG2mAmnKAsfNBN0lFRCrm8L/ahpUhU+0PMik9JGljIZ ocQpL2UrzwKW4gvviKMyv4/xWGRbmRJLXTK+KBSzrvJ2LO8y6ICsChT7wK9EaCZt2Ksqv1IpPQQf yoS1QHMR6ZFKXIouuq9eIUktCQxfoxJgDrRjztVC7/I0KSXPVFa8+KfC7CMzBEYhTRA6NMlGf+UD 8SGNqgVRv8QzESKN/kKz0iZvqNq0ZY9sL1eEI39MkdxuZkeMYvjHDgswUnSWZdH0SyLTr6ysL70J Z1F2Q8PWY5DnZE9VCXmTZBZJbW1zkFzz1ZQkBx+wP7FvCpHobhGxQh20QQ0JiG5NiPRTQYMIaYoT C82WgrjzQmWuxgZJOmtwB0OSbAm3j2TuSY1CFCOv9kQxLITjRiUu6mqGpggVMbZWbMrm4kQXXlNG O9WWcuHncNMxKd1TcAsXI1IqIoh0YmY3byv0AZdoKkgD0KIscKftS3+sniJC9JjQEyHxD9t16v7w cI4jejOvmw4P4vhiUL6HGh3AGsGmOWPOdKdiL/wGAbZGhnY3bGo3/sWQN/XEU3HP1mRdZm4uUnhM kQsNcZbetkf2o4MIDElvReMAKjxEzjE+5nTYJZ/0g3HF0p7mMHDsykTLiVaJTjdV0jsKuL5K7sAS sV81NlHyK0vBa+S60SeFg7r4DZRaJIMDbG8iOEgneOPCrtSaqmjvbm9gddYwUZHW11FgF3MmUlp2 R18RrqwSLuuAyyPqhEjaFQEGRfJAtdw8osjsZDKsRwjpIoodxWK9rvEMT02T0yVSClTXzjRJAikq SyRIgpfUco054ntZB3TO6lNFh45RJ6PmQiOaLHJBtyrqmHaMK0gg6h2BjkVzjglXlyEdzIOcr1OL aHdCa2zdpyQn/qPIZuYlLyR3HeQhyKAz2mw52rKKdwKMfVdB4oE2VC8IZc5reUw4AED13m19gcIr fFKEUsOVMnWtsMpY/3WWoeehfGNQXoI2ekIna+uYWa+yqJUgSPmTJ4NLZWJQ3NJmcSyUgKJbNbki f3h4LoMKpFdhx4b4/ANAqBh8F2UlLTYeP4hZyDk4lstE+DGdZ2ude8eAAO9d6fIr9blD3dXYQnie 6GRfCIxV7ueez8RG9DVNEhoh/BdABoU4FoQHHMOcW7KfL8RrsqNFmUW9/MQifsTBJHn3EAJI5yXJ WJdmB1YhwcXHKNqHFZJf2ZCR009CngGRW3rQ2oVitoxfoFZg/tA4YLYFaiPFMHdlg0CwiNHFqDFu V0yIAIu4qFMaI75rpRfupHHaho9yaCTnmLYZ2Ag0j+BwZdxwb522LkYpilIJKwvRmfkWchxX095a cyzUjcLwfUO6WI1mrubKscbVMLAyiJ15oZ7JCx9SMVDNVXD5cvmpRpPurh1b/t6HFYkGU03JFXso 2PT2pKvF5yTFudgkVJa3s6sacXrkHxn02kRGNjCvdx/bfVCxq1sbfiumT7uCAPu3RzwiW+TRRcEF pLgp4bgpQF+vJO3jJXtUJ71NOuRRaCIV+SJ3cmDb0aLbS2O7ZWh7MvLHpu1wqTxCIzpvVNewdJcz dK/qYgTC/pYtFbEjSSfWiaF6Q1pD2+3oZU2m+22CuLrxO4LI7kLAmLjxcSdRhzd81auR9z3BTU/j AkZl4ls30egUz1bNJHx7BIRULb+HyXA35r4tXGMMsraXxECWoLwrmjisLkYOtKV320PML8EUHCaC 10z8h10n9lrClynkQbNsqTwtknh2SNzuV6+TBh3rG4vA5zjx+sgRYJ3XIodj7FewljAzWmt7GzSU hIhBE6qTekV623446FYE08ig3MjwgRd+RSwk5Ms3XLZlkIXYZ83TfMeNxtmG1jfbqK1pd39/SAg/ uWQTRnD7/Kyd6AVtzRJvs4gIXNPCeu520AVdwmTozDtv/i1g//zN42cdBVUI43rQQoKxGTuG80/D y2Yia0+QdXmpYcZy8YGoz6Igv7HBiENCmDsTmasftzg67brWI9nI40is23AgPFMiqHnUKR144rxQ 6g21dY14DyaK4NmaQ3F6hY7wYANPE+/ZEhRRrJ16A8W0q1i0f5HUCpJ2KtV9TZjj2ql4h/cp6hOt /TxQQA6zlbs99zzmlJvZ37ndDw7ppFcOq3nOBDF9ERTY5n3YY0kmtszvlnuGJzhgIm6PsZphAEih M09L2+uIx2ufgKI5Fg4f+ISfyDyfVxn12HOUpH0stX2r4uEhKFWAyjn9Vntt/YkZAZh+0O1+rIkg 0PuC/gdV6rLnT31DdXfeT1k1juOjW510H285JpT0ImgCkbNOxCS44oTUdUhFgsOyXLGwVTtvU2lj nep0Mp5xEzd1OHR8OUEH6gG5Uhv77sCO1s+iYz3IN6KpDtnRN570wrotJHYBdNIE6GPiu6xDnLNO vr5P3nQ1dsp5u6917J0uJXue5oLiHkSjUh0AOqDx8RlaxZWtmAbHsWStfU/qtKjZO1JK4ovCImRA 5xAoazq2KDwCTgk+jFMLa455+wQIlkVrX7hL1NzikXr5+1Sv2gnY84yDWAt8tH0bQo0OaG6v6Jwn bhe5UGa1LPSEwH1Zdq43T4duMhjcv3k14nkfIEt+/lNtvypmwIAbi/WS+eEbZC6IH/S7nUcwup7f +8qtePQOHk1lAvb5PPZJfcUBAgA8fAbwbcEHwYFAfAw9IHi40ADEgQrxOUQAgGFDBA4eepxIMKPD jCEtYsRnDx8AKggL4jt58aNHkgZEniwZk+SUmDJv1sRnAqTEnjIBRITI8GdQmhmXJm1aFJ8FohDV 4XNgQMtVhQMBTEHYcWFHjO6uFry40GvLpyYxDhw6k+GWnG9BVkSrsS1Rpib5jhyIAJ8QhG4HAqXq EypfpyEBII6LbwdjpY/dErT78afKw1SZOhb68SjSkiRAh6ZYObXq1axbu35NFS7s1vI0NhtLkmFp /pIdmYAVG1qjsbAMsbY1ahRf6c61NQatyLAX7rxBb+IrNrazRuyjGVZHyh1xbunQiz/OvdF6+NzE pkfPbj3hXvNr1ubNaf7m3e7HNZYirpED8GV0W3fS6VfQcsjZJN59/DHW0RC/oVYZetVZ2F2BFeLj i3v4aKgdQwVmBB9k1ZQYFUPcoefhVR65hE8tAOY3Goqz3Yhjjjo+JtuOr312XlKwoXegjwumZpSP NyaJI5BIvtajkhAhhySVOy4koo1K8iblbFHG1mWYCHwpZplmnomYljiSOd9qJKJJFJlY6rUXhXDO 5yKPDaoZHJ93nqbRVOIZRuefhrJpKGuIIloZ/qOJPuroo5JWyaRpUwIgW1ozfebkked5ihGomWEm IQShzoTZbKKuauWlqq3aVWitWgYXk53OuheoCzrZk1gGsBTWZzVZ1hERW02KbLLKnilRpMs+Cy2g 6WXUHF4MVdLiX1cxOCdPKWo0Ul6FAsqVfRJ9lVBEvLpJKFqGrUfohQFWGC9SKXGW23r4NLcURdy2 26CD6Ck36LYDQeCSAVm1dPBCECQQbcStOSuxTEM5SzGzFYuZsWtPdWpnqAxaNpO/fX18ZlqGpcuQ TVzFwxAHrnlml00kYZRpyL3WVhPIeV6qF3AYWVXRbk91PHNXK0OX0W5KMyRfskgjDSfVG1+N/nXW bmpUJMklO/f1TEQ33V14nekqaq/qGKYEWJYVuRACS7h96obvEYsRZPi0kp3QAePDSt+Afa1b2DUT bF3Xuba663nw3FPb3PIttCI8hmkVtdapWa15555//uiCQTO0U2KjGxRuSRciN+5MzRU+sLfBdYV5 uWeBBIDkYencK+tG43TpyHFXyDp++CyR+k/Gz2U6WsX/Dbu4sseVUu4TEiTVSdWX6qfnnIMOfvji 6/i6jMHxzR9PGZlv3cn+dd8kdfohUGr73/m91+sa5mY+evreHSKGdCg4A2yf+vABN4HlxWxiQk/q OiKF38gPUuOr2PcqiEEoSeyCOuIga7iU/sEQbsxGHgxTCUWIwhSqcIUsPFQLXwhDFZ5QhsuaYQxh Y8MY5vCGPOyhD3/Ywav16EvN6kkRHRWlGe4QiGW6GBPX9MQoShFaJ1ziFLEGPw0KEVlWvKIXKxip jMGvi14kowjN+MU0qnGNKByilDD2LA+iUVlwceND5qgnI1osa2Fkox/beCM04hExNhzkH5PVuClB ZFN35FTaenekoVQqeJua5K4ciZxIVeqSwQNSIitZyUkuzpLrOqQpGwVFPRIyjxEzJA9txchhaQce tfHE7rbVl7ocxDjyuJeg8PS6ydUFKTBbUS/zkxFCreUe7/GTQEbUy2AW5HWp04jMQIK+/oxAjiG/ VBuhFCSkKo3Jk46R5Sm1eM50MpEk1mOZStClpYhkTyw2M4zDyoUAzBVMIegwFZbYh7dU1Uwd6XoL Pq65z68phKD4hGfcaidPQcUNXRDr1VswFZzLBOmdEuRo5lboSnWaUKRYMwotD+oYk8KMA6UEidzM YrKRqANmGvFnV7jXma4cCABjwwcIFArUrwEAboDZAd0s+jPLAKAUQj1IQVUyBoa5KKoCSosYjhqa 1/0nNPdoBfFO6oGUCgRmFmgpSVX5w5CeNYibm01tjlkccXHTRnGLKsJi6pZ7QY0r95peyWC204si LqB44xQ8UgKvl8xNQDlN6lj3hpvD/k4LYAZATRaul54NHRB9zNlXXhwgV6lkca011CEXy9glLNEl s3V6CQ2kWpGlWKUgiy1X9+KWTXjQNGh2omc/SVYKw76kdiHyGQJqIUmDStSgMqPJS22KPb8qcmRu oYUzDfNLLHWTtGAColpZ+F1EMgQBM+go/QjTWMUItl8u0sH1HCUaYt3uloACwC5tBwBxVLUrrcto rS6qX5I5FDBfqWrLjgMYlnwUMsoLapxu8lzoIMCpo+WuhbUWXvC6JlO7HU77CMYloyLss+d5HW5q s5/EheV1UINRTV2sOO2sSK4jHq9dYYSA+kyOIf3TyISxmtEYuUdNXcwwOuFU4TNe6fhzSAykwgwQ AIWZdclU3lghq+xHI3OsJ6A4BSq+DGZTdVdJWj6kbMo8KiyrGbVp1hgoUGEKMH85yRNbM5rqGMg7 1tnOY/ojmpf8ZjmjIhViVuOf+VznQysZ0T8ExZvj/GVXFDqtZBafohk9Qky38kyBFjSdNc1kVmL6 0qBmIyggLedJj7nUrF5lq0f66mgFGtWSjvWLmmjrXB9Z123OkUTAnApUuEIVsFB1lnmN7GQre9k+ otifSf3EJJ4S2tBmNhhDLeo9WvuFTQbkFDNc7W0vusrhBl+5M+jBT4ubrVV7Y6XZvW4criYgADs= --=_6d72d94acfd6e7bc8332c22a3d0ea929-- From owner-freebsd-hackers@FreeBSD.ORG Mon Jun 24 20:25:40 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8E9E4248; Mon, 24 Jun 2013 20:25:40 +0000 (UTC) (envelope-from pali.gabor@gmail.com) Received: from mail-oa0-x22b.google.com (mail-oa0-x22b.google.com [IPv6:2607:f8b0:4003:c02::22b]) by mx1.freebsd.org (Postfix) with ESMTP id 463D01E0C; Mon, 24 Jun 2013 20:25:40 +0000 (UTC) Received: by mail-oa0-f43.google.com with SMTP id i7so12486332oag.2 for ; Mon, 24 Jun 2013 13:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=9SsyWRw7xiE+S09ASoaBTe7veElVaj/zuZ965YRSXdQ=; b=lkiCp748CCsBaRk+WO4eQXBHdxIRZILTpN+O6Vk9fJxLMm9+hKJwTMT/d5mWUPBw+H cyaX2pNLFFHYaH1LHDohxgJTmWMgRu/w4HrWvMxM90lXNdS298az637+mxpRAmbANT81 lykr5gI6bkUkMwQlTfuiGCbjINNyby1Ut+7+WdEe83hMsl9rU038NOFa3+lwAZClYCSO DAquROYrDFODiF01N960TTmUzDTD4PkHRx4gDHEkwv4mmkRLMJ83OEBn9vz1DqQivLGx Xf8pPRQU44t8CW/0TYYfBmlK7gklo+ZLigZRwNbTZEpu9DnN6AOo2qVpXzK6MRsaQjrv SFag== MIME-Version: 1.0 X-Received: by 10.60.36.226 with SMTP id t2mr10968115oej.6.1372105539887; Mon, 24 Jun 2013 13:25:39 -0700 (PDT) Sender: pali.gabor@gmail.com Received: by 10.182.27.101 with HTTP; Mon, 24 Jun 2013 13:25:39 -0700 (PDT) In-Reply-To: References: Date: Mon, 24 Jun 2013 22:25:39 +0200 X-Google-Sender-Auth: d-H1FpeH3sxgVubwCRJV1umnHSo Message-ID: Subject: Re: FreeBSD 2013-Q2 status reports! From: Gabor Pali To: developers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2013 20:25:40 -0000 On Mon, Jun 17, 2013 at 7:14 PM, Isabell Long wrote: > It's that time again! On behalf of monthly@, I would like to inform > you that the next submission date for the April to June quarterly > status reports is July 7th, 2013 - less than a month away. Note that you have a little less than 2 weeks left to prepare and send us your reports. > They don't have to be very long - anything that lets people know what > is going on inside FreeBSD is useful. Note that submission of reports > is not restricted to committers - anyone who is doing anything > interesting and FreeBSD-related can write one! > > The preferred and easiest submission method is to use the XML > generator linked to from > http://www.freebsd.org/news/status/status.html, with the result > emailed as an attachment to monthly@FreeBSD.org. On that page, there > is also a link to an XML template which can be filled out manually and > attached if preferred. > > To enable compilation and publication of the Q2 report as soon as > possible for the July 7th deadline, please be prompt with any report > submissions you may have. > > I look forward to compiling the report for 2013 Q2. Many thanks, > > Isabell. > (Hat: monthly@) From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 25 19:44:33 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A4B11BC8; Tue, 25 Jun 2013 19:44:33 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6725B10E5; Tue, 25 Jun 2013 19:44:33 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r5PJiVhm020070; Tue, 25 Jun 2013 15:44:31 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r5PJiV97020067; Tue, 25 Jun 2013 15:44:31 -0400 (EDT) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20937.62239.626943.350086@hergotha.csail.mit.edu> Date: Tue, 25 Jun 2013 15:44:31 -0400 From: Garrett Wollman To: hackers@freebsd.org, ports@freebsd.org Subject: rc.d scripts to control multiple instances of the same daemon? X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Tue, 25 Jun 2013 15:44:32 -0400 (EDT) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 19:44:33 -0000 I'm in the process of (re)writing an rc.d script for kadmind (security/krb5). Unlike the main Kerberos daemon, kadmind needs to have a separate instance for each realm on the server -- it can't support multiple realms in a single process. What I need to be able to do: 1) Have different flags and pidfiles for each instance. 2) Be able to start, stop, restart, and status each individual instance by giving its name on the command line. 3) Have all instances start/stop automatically when a specific instance isn't specified. I've looked around for examples of good practice to emulate, and haven't found much. The closest to what I want looks to be vboxheadless, but I'm uncomfortable with the amount of mechanism from rc.subr that it needs to reimplement. Are there any better examples? -GAWollman From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 25 20:12:18 2013 Return-Path: Delivered-To: hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A8498A4E; Tue, 25 Jun 2013 20:12:18 +0000 (UTC) (envelope-from ian@FreeBSD.org) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) by mx1.freebsd.org (Postfix) with ESMTP id 804EE12BC; Tue, 25 Jun 2013 20:12:18 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1UrZb2-0005bD-66; Tue, 25 Jun 2013 20:12:12 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id r5PKC9Pl001014; Tue, 25 Jun 2013 14:12:09 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX19JCODgVcEHrM5GTHs+uewb Subject: Re: rc.d scripts to control multiple instances of the same daemon? From: Ian Lepore To: Garrett Wollman In-Reply-To: <20937.62239.626943.350086@hergotha.csail.mit.edu> References: <20937.62239.626943.350086@hergotha.csail.mit.edu> Content-Type: text/plain; charset="us-ascii" Date: Tue, 25 Jun 2013 14:12:09 -0600 Message-ID: <1372191129.1109.90.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: ports@FreeBSD.org, hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 20:12:18 -0000 On Tue, 2013-06-25 at 15:44 -0400, Garrett Wollman wrote: > I'm in the process of (re)writing an rc.d script for kadmind > (security/krb5). Unlike the main Kerberos daemon, kadmind needs to > have a separate instance for each realm on the server -- it can't > support multiple realms in a single process. What I need to be able > to do: > > 1) Have different flags and pidfiles for each instance. > 2) Be able to start, stop, restart, and status each individual > instance by giving its name on the command line. > 3) Have all instances start/stop automatically when a specific > instance isn't specified. > > I've looked around for examples of good practice to emulate, and > haven't found much. The closest to what I want looks to be > vboxheadless, but I'm uncomfortable with the amount of mechanism from > rc.subr that it needs to reimplement. Are there any better examples? The one like that I use the most is "service netif restart fpx0" but I'm not sure the complex network stuff will be the cleanest example of anything except how to do complex network stuff. :) -- Ian From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 25 19:56:31 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1854FF78 for ; Tue, 25 Jun 2013 19:56:31 +0000 (UTC) (envelope-from ohauer@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) by mx1.freebsd.org (Postfix) with ESMTP id AB8141189 for ; Tue, 25 Jun 2013 19:56:30 +0000 (UTC) Received: from mailout-de.gmx.net ([10.1.76.19]) by mrigmx.server.lan (mrigmx001) with ESMTP (Nemesis) id 0MEHTC-1V2WdU1lMm-00FWZp for ; Tue, 25 Jun 2013 21:56:24 +0200 Received: (qmail invoked by alias); 25 Jun 2013 19:56:24 -0000 Received: from p578be941.dip0.t-ipconnect.de (EHLO [192.168.0.100]) [87.139.233.65] by mail.gmx.net (mp019) with SMTP; 25 Jun 2013 21:56:24 +0200 X-Authenticated: #1956535 X-Provags-ID: V01U2FsdGVkX1+tvalV52St3Ulr54PBWEK6g9XB9KhxPWF9FMi7Br VHzVXUnwYne1sg Message-ID: <51C9F5E7.1040607@gmx.de> Date: Tue, 25 Jun 2013 21:56:23 +0200 From: olli hauer User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Garrett Wollman Subject: Re: rc.d scripts to control multiple instances of the same daemon? References: <20937.62239.626943.350086@hergotha.csail.mit.edu> In-Reply-To: <20937.62239.626943.350086@hergotha.csail.mit.edu> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 X-Mailman-Approved-At: Tue, 25 Jun 2013 20:36:05 +0000 Cc: ports@freebsd.org, hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 19:56:31 -0000 On 2013-06-25 21:44, Garrett Wollman wrote: > I'm in the process of (re)writing an rc.d script for kadmind > (security/krb5). Unlike the main Kerberos daemon, kadmind needs to > have a separate instance for each realm on the server -- it can't > support multiple realms in a single process. What I need to be able > to do: > > 1) Have different flags and pidfiles for each instance. > 2) Be able to start, stop, restart, and status each individual > instance by giving its name on the command line. > 3) Have all instances start/stop automatically when a specific > instance isn't specified. > > I've looked around for examples of good practice to emulate, and > haven't found much. The closest to what I want looks to be > vboxheadless, but I'm uncomfortable with the amount of mechanism from > rc.subr that it needs to reimplement. Are there any better examples? > > -GAWollman > Take a look into the openvpn rc script (works with softlinks). A more complex script that supports multiple instances can be found in the www/apache2x ports. -- olli From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 25 20:37:48 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3DDA25B3; Tue, 25 Jun 2013 20:37:48 +0000 (UTC) (envelope-from gahr@FreeBSD.org) Received: from cpanel09.rubas.ch (cpanel09.rubas.ch [195.182.222.79]) by mx1.freebsd.org (Postfix) with ESMTP id F02F7156D; Tue, 25 Jun 2013 20:37:47 +0000 (UTC) Received: from 98-41.199-178.cust.bluewin.ch ([178.199.41.98]:60667 helo=gahrfit.gahr.ch) by cpanel09.rubas.ch with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80) (envelope-from ) id 1UrZzf-000xjQ-M8; Tue, 25 Jun 2013 22:37:40 +0200 Date: Tue, 25 Jun 2013 22:37:36 +0200 From: Pietro Cerutti To: Garrett Wollman Subject: Re: rc.d scripts to control multiple instances of the same daemon? Message-ID: <20130625203734.GD32654@gahrfit.gahr.ch> References: <20937.62239.626943.350086@hergotha.csail.mit.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="d9ADC0YsG2v16Js0" Content-Disposition: inline In-Reply-To: <20937.62239.626943.350086@hergotha.csail.mit.edu> X-PGP-Key: 0x9571F78E X-PGP-Fingerprint: 1203 92B5 3919 AF84 9B97 28D6 C0C2 6A98 9571 F78E User-Agent: Mutt/1.5.21 (2010-09-15) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel09.rubas.ch X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - FreeBSD.org X-Get-Message-Sender-Via: cpanel09.rubas.ch: authenticated_id: gahr@gahr.ch Cc: ports@freebsd.org, hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: gahr@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 20:37:48 -0000 --d9ADC0YsG2v16Js0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2013-Jun-25, 15:44, Garrett Wollman wrote: > I'm in the process of (re)writing an rc.d script for kadmind > (security/krb5). Unlike the main Kerberos daemon, kadmind needs to > have a separate instance for each realm on the server -- it can't > support multiple realms in a single process. What I need to be able > to do: >=20 > 1) Have different flags and pidfiles for each instance. > 2) Be able to start, stop, restart, and status each individual > instance by giving its name on the command line. > 3) Have all instances start/stop automatically when a specific > instance isn't specified. >=20 > I've looked around for examples of good practice to emulate, and > haven't found much. The closest to what I want looks to be > vboxheadless, but I'm uncomfortable with the amount of mechanism from > rc.subr that it needs to reimplement. Are there any better examples? Wouldn't something like this be enough? =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #!/bin/sh =2E /etc/rc.subr name=3D"kadmind" rcvar=3D${name}_enable start_cmd=3D"kadmind_start $2" load_rc_config $name kadmind_start() { if [ -z $1 ]; then echo "starting all instances" else echo "starting instance $1" fi } run_rc_command "$1" =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # /usr/local/etc/rc.d/kadmind start starting all instances # /usr/local/etc/rc.d/kadmind start myinst starting instance myinst --=20 Pietro Cerutti The FreeBSD Project gahr@FreeBSD.org PGP Public Key: http://gahr.ch/pgp --d9ADC0YsG2v16Js0 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iEYEARECAAYFAlHJ/44ACgkQwMJqmJVx944dUQCfXpZpkBM0hu+ycjlJJAy0tJLQ /VsAoNPUIY1yZl9dLBc0WOmPx3uyJZUW =djit -----END PGP SIGNATURE----- --d9ADC0YsG2v16Js0-- From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 25 20:41:04 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 30F197EC; Tue, 25 Jun 2013 20:41:04 +0000 (UTC) (envelope-from baptiste.daroussin@gmail.com) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) by mx1.freebsd.org (Postfix) with ESMTP id 6E1A41603; Tue, 25 Jun 2013 20:41:03 +0000 (UTC) Received: by mail-wg0-f54.google.com with SMTP id n11so9983799wgh.9 for ; Tue, 25 Jun 2013 13:41:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=MgiLg/jIt0eeYj8QZPCQ7KYHXEK3dCaKYpF+4b3wBlc=; b=J9OcvungpsMB82UjIH778Nrgo518bHkXuk3PMP+kUay4YdCaEIPFf2rSMo9+fK5ZXU 8hybwDBOFZWyEhPscpHKZ92jqfUI4kR/2DrjTbwshJ1bDh0n9dC661YUnV6LeJDFiQDX WoSkylVXvh/++Yqc4EtxgZ7N1XsKFotXott2OlmyOolFb3nrBlMIJJ6om48tdlb/B0jh s2gZT3CpQ40puIRAcmXQbpPkcBouDdGCYs1beudoEooY58fPAQjSGGGVie0+NJp+V4AM K55OxHpqfQy3GvKIdfjlVUPeTJv65D4Sb2zN0V8xDlp8XAyNH0Z0WvYvdXnAmgAYDINN WP7g== X-Received: by 10.180.87.162 with SMTP id az2mr10408922wib.10.1372192862675; Tue, 25 Jun 2013 13:41:02 -0700 (PDT) Received: from ithaqua.etoilebsd.net (ithaqua.etoilebsd.net. [37.59.37.188]) by mx.google.com with ESMTPSA id s19sm6538245wik.11.2013.06.25.13.41.01 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 25 Jun 2013 13:41:01 -0700 (PDT) Sender: Baptiste Daroussin Date: Tue, 25 Jun 2013 22:40:59 +0200 From: Baptiste Daroussin To: Garrett Wollman Subject: Re: rc.d scripts to control multiple instances of the same daemon? Message-ID: <20130625204059.GE93612@ithaqua.etoilebsd.net> References: <20937.62239.626943.350086@hergotha.csail.mit.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="mR8QP4gmHujQHb1c" Content-Disposition: inline In-Reply-To: <20937.62239.626943.350086@hergotha.csail.mit.edu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: ports@freebsd.org, hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2013 20:41:04 -0000 --mR8QP4gmHujQHb1c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 25, 2013 at 03:44:31PM -0400, Garrett Wollman wrote: > I'm in the process of (re)writing an rc.d script for kadmind > (security/krb5). Unlike the main Kerberos daemon, kadmind needs to > have a separate instance for each realm on the server -- it can't > support multiple realms in a single process. What I need to be able > to do: >=20 > 1) Have different flags and pidfiles for each instance. > 2) Be able to start, stop, restart, and status each individual > instance by giving its name on the command line. > 3) Have all instances start/stop automatically when a specific > instance isn't specified. >=20 > I've looked around for examples of good practice to emulate, and > haven't found much. The closest to what I want looks to be > vboxheadless, but I'm uncomfortable with the amount of mechanism from > rc.subr that it needs to reimplement. Are there any better examples? >=20 > -GAWollman >=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" This is a simple multi instance rc.d script: http://svnweb.freebsd.org/ports/head/www/fcgiwrap/files/fcgiwrap.in?revisio= n=3D307542&view=3Dmarkup regards, Bapt --mR8QP4gmHujQHb1c Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlHKAFsACgkQ8kTtMUmk6ExvigCgoz3XDihcKC/fS3u1vpTm0ZLZ SO4AoKLFTTIINM+PNjz48h57lyimXcBM =EHLw -----END PGP SIGNATURE----- --mR8QP4gmHujQHb1c-- From owner-freebsd-hackers@FreeBSD.ORG Wed Jun 26 07:55:09 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4A5176B8 for ; Wed, 26 Jun 2013 07:55:09 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: from relay.andxor.it (relay.andxor.it [195.223.2.3]) by mx1.freebsd.org (Postfix) with ESMTP id 85A831493 for ; Wed, 26 Jun 2013 07:55:08 +0000 (UTC) Received: (qmail 36930 invoked from network); 26 Jun 2013 07:55:07 -0000 Received: from alex.andxor.it (a.premoli@andxor.it@192.168.2.30) by relay.andxor.it with ESMTPSA; 26 Jun 2013 07:55:07 -0000 Message-ID: <51CA9E5B.4010909@FreeBSD.org> Date: Wed, 26 Jun 2013 09:55:07 +0200 From: Alex Dupre User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:20.0) Gecko/20100101 Firefox/20.0 SeaMonkey/2.17.1 MIME-Version: 1.0 To: Garrett Wollman , hackers@freebsd.org, ports@freebsd.org Subject: Re: rc.d scripts to control multiple instances of the same daemon? References: <20937.62239.626943.350086@hergotha.csail.mit.edu> In-Reply-To: <20937.62239.626943.350086@hergotha.csail.mit.edu> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 07:55:09 -0000 Garrett Wollman ha scritto: > I've looked around for examples of good practice to emulate, and > haven't found much. The closest to what I want looks to be > vboxheadless, but I'm uncomfortable with the amount of mechanism from > rc.subr that it needs to reimplement. Are there any better examples? Basically there are two types of implementation with different pro/cons: profiles and symlinks. vboxheadless is in the first category, tomcat7 in the second one: http://svnweb.freebsd.org/ports/head/www/tomcat7/files/tomcat7.in?revision=307489&view=markup -- Alex Dupre From owner-freebsd-hackers@FreeBSD.ORG Wed Jun 26 16:11:51 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 81F4B63F for ; Wed, 26 Jun 2013 16:11:51 +0000 (UTC) (envelope-from torek@torek.net) Received: from elf.torek.net (50-73-42-1-utah.hfc.comcastbusiness.net [50.73.42.1]) by mx1.freebsd.org (Postfix) with ESMTP id 51EE91E0B for ; Wed, 26 Jun 2013 16:11:50 +0000 (UTC) Received: from elf.torek.net (localhost [127.0.0.1]) by elf.torek.net (8.14.5/8.14.5) with ESMTP id r5QGBie4025519 for ; Wed, 26 Jun 2013 10:11:44 -0600 (MDT) (envelope-from torek@torek.net) Message-Id: <201306261611.r5QGBie4025519@elf.torek.net> From: Chris Torek To: freebsd-hackers@freebsd.org Subject: expanding amd64 past the 1TB limit Date: Wed, 26 Jun 2013 10:11:44 -0600 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (elf.torek.net [127.0.0.1]); Wed, 26 Jun 2013 10:11:44 -0600 (MDT) X-Mailman-Approved-At: Wed, 26 Jun 2013 16:22:19 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 16:11:51 -0000 (Note: Last week I asked about this on the freebsd-current list. It turned out slightly harder than I thought, as the 512GB kernel virtual area is based on what fits into a single L4 page table entry.) I was asked to expand the kernel limits for amd64 systems. While I do not have a system with enough RAM to test this for real, the changes below seem to boot and run OK. I went just a little bit wild in create_pagetables(). :-) The lines with the casts got long (and hard to read) so I shortened them (but I still needed the map I drew of the page tables...). If using ptoa() like this is OK, probably there should be a few more of those, e.g., in the changes to pmap_pinit(). Anyway, I wonder if some form of this patch (perhaps even without the #ifdefs) might be accepted back. I'm not sure about the KPML4BASE name, but it clearly needs to be different from KPML4I. (At first I was considering moving KERNBASE too but the branch offsets seem to be the real limiting factor here.) Possibly dumb question: around the comment "this replaces some of the KPTphys entries above", would it be possible to reclaim a few pages by calculating in advance where the 2MB page mappings obviate the need for the underlying KPTphys pages, and just offset things? Another note: one could get rid of the "power of 2" requirement for NDMPML4E. It arises from the translation between direct mapped virtual and physical addresses (being |= and &=~), but the same result can be achieved by adding and subtracting an offset, which would allow the base and limit to be arbitrary, rather than a power of two. (Still, it did not seem worth doing here.) Chris diff --git a/amd64/amd64/pmap.c b/amd64/amd64/pmap.c index 272158d..acf5af2 100644 --- a/amd64/amd64/pmap.c +++ b/amd64/amd64/pmap.c @@ -534,6 +534,10 @@ static void create_pagetables(vm_paddr_t *firstaddr) { int i, j, ndm1g, nkpdpe; + pt_entry_t *pt_p; + pd_entry_t *pd_p; + pdp_entry_t *pdp_p; + pml4_entry_t *p4_p; /* Allocate page table pages for the direct map */ ndmpdp = (ptoa(Maxmem) + NBPDP - 1) >> PDPSHIFT; @@ -556,6 +560,10 @@ create_pagetables(vm_paddr_t *firstaddr) * bootstrap. We defer this until after all memory-size dependent * allocations are done (e.g. direct map), so that we don't have to * build in too much slop in our estimate. + * + * Note that when NKPML4E > 1, we have an empty page underneath + * all but the KPML4I'th one, so we need NKPML4E-1 extra (zeroed) + * pages. (pmap_enter requires a PD page to exist for each KPML4E.) */ nkpt_init(*firstaddr); nkpdpe = NKPDPE(nkpt); @@ -564,32 +572,26 @@ create_pagetables(vm_paddr_t *firstaddr) KPDphys = allocpages(firstaddr, nkpdpe); /* Fill in the underlying page table pages */ - /* Read-only from zero to physfree */ + /* Nominally read-only (but really R/W) from zero to physfree */ /* XXX not fully used, underneath 2M pages */ - for (i = 0; (i << PAGE_SHIFT) < *firstaddr; i++) { - ((pt_entry_t *)KPTphys)[i] = i << PAGE_SHIFT; - ((pt_entry_t *)KPTphys)[i] |= PG_RW | PG_V | PG_G; - } + pt_p = (pt_entry_t *)KPTphys; + for (i = 0; ptoa(i) < *firstaddr; i++) + pt_p[i] = ptoa(i) | PG_RW | PG_V | PG_G; /* Now map the page tables at their location within PTmap */ - for (i = 0; i < nkpt; i++) { - ((pd_entry_t *)KPDphys)[i] = KPTphys + (i << PAGE_SHIFT); - ((pd_entry_t *)KPDphys)[i] |= PG_RW | PG_V; - } + pd_p = (pd_entry_t *)KPDphys; + for (i = 0; i < nkpt; i++) + pd_p[i] = (KPTphys + ptoa(i)) | PG_RW | PG_V; /* Map from zero to end of allocations under 2M pages */ /* This replaces some of the KPTphys entries above */ - for (i = 0; (i << PDRSHIFT) < *firstaddr; i++) { - ((pd_entry_t *)KPDphys)[i] = i << PDRSHIFT; - ((pd_entry_t *)KPDphys)[i] |= PG_RW | PG_V | PG_PS | PG_G; - } + for (i = 0; (i << PDRSHIFT) < *firstaddr; i++) + pd_p[i] = (i << PDRSHIFT) | PG_RW | PG_V | PG_PS | PG_G; - /* And connect up the PD to the PDP */ - for (i = 0; i < nkpdpe; i++) { - ((pdp_entry_t *)KPDPphys)[i + KPDPI] = KPDphys + - (i << PAGE_SHIFT); - ((pdp_entry_t *)KPDPphys)[i + KPDPI] |= PG_RW | PG_V | PG_U; - } + /* And connect up the PD to the PDP (leaving room for L4 pages) */ + pdp_p = (pdp_entry_t *)(KPDPphys + ptoa(KPML4I - KPML4BASE)); + for (i = 0; i < nkpdpe; i++) + pdp_p[i + KPDPI] = (KPDphys + ptoa(i)) | PG_RW | PG_V | PG_U; /* * Now, set up the direct map region using 2MB and/or 1GB pages. If @@ -599,37 +601,41 @@ create_pagetables(vm_paddr_t *firstaddr) * memory, pmap_change_attr() will demote any 2MB or 1GB page mappings * that are partially used. */ + pd_p = (pd_entry_t *)DMPDphys; for (i = NPDEPG * ndm1g, j = 0; i < NPDEPG * ndmpdp; i++, j++) { - ((pd_entry_t *)DMPDphys)[j] = (vm_paddr_t)i << PDRSHIFT; + pd_p[j] = (vm_paddr_t)i << PDRSHIFT; /* Preset PG_M and PG_A because demotion expects it. */ - ((pd_entry_t *)DMPDphys)[j] |= PG_RW | PG_V | PG_PS | PG_G | + pd_p[j] |= PG_RW | PG_V | PG_PS | PG_G | PG_M | PG_A; } + pdp_p = (pdp_entry_t *)DMPDPphys; for (i = 0; i < ndm1g; i++) { - ((pdp_entry_t *)DMPDPphys)[i] = (vm_paddr_t)i << PDPSHIFT; + pdp_p[i] = (vm_paddr_t)i << PDPSHIFT; /* Preset PG_M and PG_A because demotion expects it. */ - ((pdp_entry_t *)DMPDPphys)[i] |= PG_RW | PG_V | PG_PS | PG_G | + pdp_p[i] |= PG_RW | PG_V | PG_PS | PG_G | PG_M | PG_A; } for (j = 0; i < ndmpdp; i++, j++) { - ((pdp_entry_t *)DMPDPphys)[i] = DMPDphys + (j << PAGE_SHIFT); - ((pdp_entry_t *)DMPDPphys)[i] |= PG_RW | PG_V | PG_U; + pdp_p[i] = DMPDphys + ptoa(j); + pdp_p[i] |= PG_RW | PG_V | PG_U; } /* And recursively map PML4 to itself in order to get PTmap */ - ((pdp_entry_t *)KPML4phys)[PML4PML4I] = KPML4phys; - ((pdp_entry_t *)KPML4phys)[PML4PML4I] |= PG_RW | PG_V | PG_U; + p4_p = (pml4_entry_t *)KPML4phys; + p4_p[PML4PML4I] = KPML4phys; + p4_p[PML4PML4I] |= PG_RW | PG_V | PG_U; /* Connect the Direct Map slot(s) up to the PML4. */ for (i = 0; i < NDMPML4E; i++) { - ((pdp_entry_t *)KPML4phys)[DMPML4I + i] = DMPDPphys + - (i << PAGE_SHIFT); - ((pdp_entry_t *)KPML4phys)[DMPML4I + i] |= PG_RW | PG_V | PG_U; + p4_p[DMPML4I + i] = DMPDPphys + ptoa(i); + p4_p[DMPML4I + i] |= PG_RW | PG_V | PG_U; } - /* Connect the KVA slot up to the PML4 */ - ((pdp_entry_t *)KPML4phys)[KPML4I] = KPDPphys; - ((pdp_entry_t *)KPML4phys)[KPML4I] |= PG_RW | PG_V | PG_U; + /* Connect the KVA slots up to the PML4 */ + for (i = 0; i < NKPML4E; i++) { + p4_p[KPML4BASE + i] = KPDPphys + ptoa(i); + p4_p[KPML4BASE + i] |= PG_RW | PG_V | PG_U; + } } /* @@ -1688,7 +1694,10 @@ pmap_pinit(pmap_t pmap) pagezero(pmap->pm_pml4); /* Wire in kernel global address entries. */ - pmap->pm_pml4[KPML4I] = KPDPphys | PG_RW | PG_V | PG_U; + for (i = 0; i < NKPML4E; i++) { + pmap->pm_pml4[KPML4BASE + i] = (KPDPphys + (i << PAGE_SHIFT)) | + PG_RW | PG_V | PG_U; + } for (i = 0; i < NDMPML4E; i++) { pmap->pm_pml4[DMPML4I + i] = (DMPDPphys + (i << PAGE_SHIFT)) | PG_RW | PG_V | PG_U; @@ -1944,7 +1953,8 @@ pmap_release(pmap_t pmap) m = PHYS_TO_VM_PAGE(pmap->pm_pml4[PML4PML4I] & PG_FRAME); - pmap->pm_pml4[KPML4I] = 0; /* KVA */ + for (i = 0; i < NKPML4E; i++) /* KVA */ + pmap->pm_pml4[KPML4BASE + i] = 0; for (i = 0; i < NDMPML4E; i++) /* Direct Map */ pmap->pm_pml4[DMPML4I + i] = 0; pmap->pm_pml4[PML4PML4I] = 0; /* Recursive Mapping */ diff --git a/amd64/include/pmap.h b/amd64/include/pmap.h index 6d76ec3..58d1c9d 100644 --- a/amd64/include/pmap.h +++ b/amd64/include/pmap.h @@ -113,7 +113,17 @@ ((unsigned long)(l2) << PDRSHIFT) | \ ((unsigned long)(l1) << PAGE_SHIFT)) -#define NKPML4E 1 /* number of kernel PML4 slots */ +/* + * Number of kernel PML4 slots. Can be anywhere from 1 to 64 or so, + * but setting it larger than NDMPML4E makes no sense. + * + * Each slot provides .5 TB of kernel virtual space. + */ +#ifdef AMD64_HUGE +#define NKPML4E 16 +#else +#define NKPML4E 1 +#endif #define NUPML4E (NPML4EPG/2) /* number of userland PML4 pages */ #define NUPDPE (NUPML4E*NPDPEPG)/* number of userland PDP pages */ @@ -121,20 +131,39 @@ /* * NDMPML4E is the number of PML4 entries that are used to implement the - * direct map. It must be a power of two. + * direct map. It must be a power of two, and should generally exceed + * NKPML4E. The maximum possible value is 64; using 128 will make the + * direct map intrude into the recursive page table map. */ +#ifdef AMD64_HUGE +#define NDMPML4E 32 +#else #define NDMPML4E 2 +#endif /* - * The *PDI values control the layout of virtual memory. The starting address + * These values control the layout of virtual memory. The starting address * of the direct map, which is controlled by DMPML4I, must be a multiple of * its size. (See the PHYS_TO_DMAP() and DMAP_TO_PHYS() macros.) + * + * Note: KPML4I is the index of the (single) level 4 page that maps + * the KVA that holds KERNBASE, while KPML4BASE is the index of the + * first level 4 page that maps VM_MIN_KERNEL_ADDRESS. If NKPML4E + * is 1, these are the same, otherwise KPML4BASE < KPML4I and extra + * level 4 PDEs are needed to map from VM_MIN_KERNEL_ADDRESS up to + * KERNBASE. Similarly, if KMPL4I < NKPML4E, extra level 4 PDEs are + * needed to map from somewhere-above-KERNBASE to VM_MAX_KERNEL_ADDRESS. + * + * (KPML4I combines with KPDPI to choose where KERNBASE starts. + * Or, in other words, KPML4I provides bits 39..46 of KERNBASE, + * and KPDPI provides bits 30..38.) */ #define PML4PML4I (NPML4EPG/2) /* Index of recursive pml4 mapping */ -#define KPML4I (NPML4EPG-1) /* Top 512GB for KVM */ -#define DMPML4I rounddown(KPML4I - NDMPML4E, NDMPML4E) /* Below KVM */ +#define KPML4BASE (NPML4EPG-NKPML4E) /* KVM at highest addresses */ +#define DMPML4I rounddown(KPML4BASE-NDMPML4E, NDMPML4E) /* Below KVM */ +#define KPML4I (NPML4EPG-1) #define KPDPI (NPDPEPG-2) /* kernbase at -2GB */ /* diff --git a/amd64/include/vmparam.h b/amd64/include/vmparam.h index 33f62bd..47a8ef8 100644 --- a/amd64/include/vmparam.h +++ b/amd64/include/vmparam.h @@ -145,18 +145,26 @@ * 0x0000000000000000 - 0x00007fffffffffff user map * 0x0000800000000000 - 0xffff7fffffffffff does not exist (hole) * 0xffff800000000000 - 0xffff804020100fff recursive page table (512GB slot) +#ifdef AMD64_HUGE + * 0xffff804020101000 - 0xffffdfffffffffff unused + * 0xffffe00000000000 - 0xffffefffffffffff 16TB direct map + * 0xfffff00000000000 - 0xfffff7ffffffffff unused + * 0xfffff80000000000 - 0xffffffffffffffff 8TB kernel map +#else * 0xffff804020101000 - 0xfffffdffffffffff unused * 0xfffffe0000000000 - 0xfffffeffffffffff 1TB direct map * 0xffffff0000000000 - 0xffffff7fffffffff unused * 0xffffff8000000000 - 0xffffffffffffffff 512GB kernel map +#endif * * Within the kernel map: * * 0xffffffff80000000 KERNBASE */ -#define VM_MAX_KERNEL_ADDRESS KVADDR(KPML4I, NPDPEPG-1, NPDEPG-1, NPTEPG-1) -#define VM_MIN_KERNEL_ADDRESS KVADDR(KPML4I, NPDPEPG-512, 0, 0) +#define VM_MIN_KERNEL_ADDRESS KVADDR(KPML4BASE, 0, 0, 0) +#define VM_MAX_KERNEL_ADDRESS KVADDR(KPML4BASE + NKPML4E - 1, \ + NPDPEPG-1, NPDEPG-1, NPTEPG-1) #define DMAP_MIN_ADDRESS KVADDR(DMPML4I, 0, 0, 0) #define DMAP_MAX_ADDRESS KVADDR(DMPML4I + NDMPML4E, 0, 0, 0) diff --git a/conf/options.amd64 b/conf/options.amd64 index 90348b7..f3ce505 100644 --- a/conf/options.amd64 +++ b/conf/options.amd64 @@ -1,6 +1,7 @@ # $FreeBSD$ # Options specific to AMD64 platform kernels +AMD64_HUGE opt_global.h AUTO_EOI_1 opt_auto_eoi.h AUTO_EOI_2 opt_auto_eoi.h COUNT_XINVLTLB_HITS opt_smp.h From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 00:02:07 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0DE0F6C5 for ; Thu, 27 Jun 2013 00:02:07 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-yh0-x235.google.com (mail-yh0-x235.google.com [IPv6:2607:f8b0:4002:c01::235]) by mx1.freebsd.org (Postfix) with ESMTP id C41AA17C3 for ; Thu, 27 Jun 2013 00:02:06 +0000 (UTC) Received: by mail-yh0-f53.google.com with SMTP id a41so79499yho.40 for ; Wed, 26 Jun 2013 17:02:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=from:content-type:subject:message-id:date:to:mime-version:x-mailer; bh=9lRvwp1m8oIs/YsYfW+HkTQB7ayvQtM2w21998gVkVQ=; b=IeT2P706+uRD9xB6hLFHDO0zp6UG/+r7yju4fMX0hI7i/L56ZeBTExdqINSJnWDSbD e1XAVhdXxa/pqunwfAbSmwBfaGwAL1qj8s1MmBDwds6l10g2Agr0ZbLnB+TTB+JrFbNj /eWuo9lEtSgNMcwgLMLdTPhRYtw8Xf2bAetG8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:content-type:subject:message-id:date:to:mime-version:x-mailer :x-gm-message-state; bh=9lRvwp1m8oIs/YsYfW+HkTQB7ayvQtM2w21998gVkVQ=; b=aR8rplCV6XW8Hz84MCy+l0uI/h+Vxs5gzdHoxbFDhcoO9E4G7+wEmKDy671HPbDMGL 4f8N9INrTrfkb1Z6SEx/yfsod2cKV7TEdKvMAuluuryVwg8uP4TofETYFXsm7K1xLoCa yMALEiT7swassYzq7alnReUsBYvBeKsUVkv666+4J285WrTlzLfO4q9oslexaC7bwc3P xy8xubeFbFdnvsFk2Wcs4BJy3O2IOZDS6APzHatB4dB4IqqJoolbJe3iG/dlQC2PWX8A w7gcHZOo6cBzQGeRXROyx2b6PFk8/8M1QIMMk8WjBiItVP6n4UMdJNpfUQ7p//4fE+uy 1eYQ== X-Received: by 10.236.25.36 with SMTP id y24mr3285043yhy.129.1372291325758; Wed, 26 Jun 2013 17:02:05 -0700 (PDT) Received: from unassigned.v6.your.org ([2001:4978:1:45:3cc2:51b:e158:a115]) by mx.google.com with ESMTPSA id j5sm438369yhk.20.2013.06.26.17.02.04 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 26 Jun 2013 17:02:05 -0700 (PDT) From: Kevin Day Content-Type: multipart/signed; boundary="Apple-Mail=_17EED475-0486-4923-862D-97DDBBA66FA1"; protocol="application/pkcs7-signature"; micalg=sha1 Subject: Can't use gcc in a clang built world Message-Id: Date: Wed, 26 Jun 2013 19:02:02 -0500 To: "freebsd-hackers@freebsd.org Hackers" Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) X-Mailer: Apple Mail (2.1508) X-Gm-Message-State: ALoCoQnR0L5JH6jHf9bygHlrQ7gzXyVKPSL9yDe1h1Mn+9M0++1Km9ZETbPRaAPMe/nc0eX2SRCb X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 00:02:07 -0000 --Apple-Mail=_17EED475-0486-4923-862D-97DDBBA66FA1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Are you supposed to be able to use gcc to build userland binaries if you = built world with clang? I'm on -CURRENT as of a few days ago (using armv6 but i'm not sure if = that matters). If I buildworld with clang, then attempt to compile some = userland binaries with gcc, I'll get missing symbols like: CCLD pcretest ./.libs/libpcre.so: undefined reference to `__clear_cache' *** [pcretest] Error code 1 If I look at /lib/libgcc that symbol isn't there: # readelf -a /lib/libgcc_s.so.1 | grep clear # If I rebuild /usr/src/gnu/libgcc with gcc, then try again it works. The = symbol is now there: # readelf -a /lib/libgcc_s.so.1 | grep clear 94: 00003b94 48 FUNC GLOBAL DEFAULT 12 = __clear_cache@@GCC_3.0 # I can build pcre correctly. I thought one of the goals was to be able to = use both interchangeably on the same system. Is this broken, or one of = the casualties of making clang default now? Do we need two different = versions of some libraries depending on which compiler is being used? -- Kevin --Apple-Mail=_17EED475-0486-4923-862D-97DDBBA66FA1 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPLzCCBN0w ggPFoAMCAQICEHGS++YZX6xNEoV0cTSiGKcwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0Ix GzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwR Q29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0w NDAxMDEwMDAwMDBaFw0yODEyMzEyMzU5NTlaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQx FzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsx ITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJz dC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAsjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIx B8dOtINknS4p1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8 om+rWV6lL8/K2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHG TPNpsaguG7bUMSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7Nl yP0e03RiqhjKaJMeoYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4IBJzCCASMwHwYDVR0j BBgwFoAUoBEKIz6W8Qfs4q8p74Klf9AwpLQwHQYDVR0OBBYEFImCZ33EnSZwAEu0UEh83j2uBG59 MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr BgEFBQcDBDARBgNVHSAECjAIMAYGBFUdIAAwewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5j b21vZG9jYS5jb20vQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwu Y29tb2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDARBglghkgBhvhCAQEEBAMCAQYw DQYJKoZIhvcNAQEFBQADggEBAJ2Vyzy4fqUJxB6/C8LHdo45PJTGEKpPDMngq4RdiVTgZTvzbRx8 NywlVF+WIfw3hJGdFdwUT4HPVB1rbEVgxy35l1FM+WbKPKCCjKbI8OLp1Er57D9Wyd12jMOCAU9s APMeGmF0BEcDqcZAV5G8ZSLFJ2dPV9tkWtmNH7qGL/QGrpxp7en0zykX2OBKnxogL5dMUbtGB8SK N04g4wkxaMeexIud6H4RvDJoEJYRmETYKlFgTYjrdDrfQwYyyDlWjDoRUtNBpEMD9O3vMyfbOeAU TibJ2PU54om4k123KSZB6rObroP8d3XK6Mq1/uJlSmM+RMTQw16Hc6mYHK9/FX8wggUaMIIEAqAD AgECAhBtGeqnGU9qMyLmIjJ6qnHeMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYDVQQGEwJVUzELMAkG A1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNU IE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVRO LVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMB4XDTExMDQyODAwMDAw MFoXDTIwMDUzMDEwNDgzOFowgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNo ZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTkwNwYD VQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCShIRbS1eY1F4vi6ThQMijU1hfZmXxMk73nzJ9 VdB4TFW3QpTg+SdxB8XGaaS5MsTxQBqQzCdWYn8XtXFpruUgG+TLY15gyqJB9mrho/+43x9IbWVD jCouK2M4d9+xF6zC2oIC1tQyatRnbyATj1w1+uVUgK/YcQodNwoCUFNslR2pEBS0mZVZEjH/CaLS TNxS297iQAFbSGjdxUq04O0kHzqvcV8H46y/FDuwJXFoPfQP1hdYRhWBPGiLi4MPbXohV+Y0sNsy fuNK4aVScmQmkU6lkg//4LFg/RpvaFGZY40ai6XMQpubfSJj06mg/M6ekN9EGfRcWzW6FvOnm//B AgMBAAGjggFLMIIBRzAfBgNVHSMEGDAWgBSJgmd9xJ0mcABLtFBIfN49rgRufTAdBgNVHQ4EFgQU ehNOAHRbxnhjZCfBL+KgW7x5xXswDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAw EQYDVR0gBAowCDAGBgRVHSAAMFgGA1UdHwRRME8wTaBLoEmGR2h0dHA6Ly9jcmwudXNlcnRydXN0 LmNvbS9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kRW1haWwuY3JsMHQGCCsG AQUFBwEBBGgwZjA9BggrBgEFBQcwAoYxaHR0cDovL2NydC51c2VydHJ1c3QuY29tL1VUTkFkZFRy dXN0Q2xpZW50X0NBLmNydDAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTAN BgkqhkiG9w0BAQUFAAOCAQEAhda+eFdVbTN/RFL+QtUGqAEDgIr7DbL9Sr/2r0FJ9RtaxdKtG3Nu PukmfOZMmMEwKN/L+0I8oSU+CnXW0D05hmbRoZu1TZtvryhsHa/l6nRaqNqxwPF1ei+eupN5yv7i kR5WdLL4jdPgQ3Ib7Y/9YDkgR/uLrzplSDyYPaUlv73vYOBJ5RbI6z9Dg/Dg7g3B080zX5vQvWBq szv++tTJOjwf7Zv/m0kzvkIpOYPuM2kugp1FTahp2oAbHj3SGl18R5mlmwhtEpmG1l1XBxunML5L SUS4kH7K0Xk467Qz+qA6XSZYnmFVGLQh1ZnV4ENAQjC+6qXnlNKw/vN1+X9u5zCCBSwwggQUoAMC AQICEQDbETdDYf7wYKjx8ymk38yAMA0GCSqGSIb3DQEBBQUAMIGTMQswCQYDVQQGEwJHQjEbMBkG A1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01P RE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQg U2VjdXJlIEVtYWlsIENBMB4XDTEzMDYxNjAwMDAwMFoXDTE0MDYxNjIzNTk1OVowJjEkMCIGCSqG SIb3DQEJARYVdG9hc3R5QGRyYWdvbmRhdGEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEAvoIO+cLWLe7YYAGV/WdoWC85K8uIgstlYMg/bC8eGbC7AY/nuQXpRV5+xlTXgN7qry/m 6XErlaw1U3rmwlNyjMhJdYaPZclywBKKpYnc3sp0q2A6naeVmOF/t4QDImtfc3sV7SaEkIr7zssK MFTnkOX57g1r3MuiYoHBx1cMaWXYCJ5LDzsynwHGAExYuziRzXcu4sRZ1HBJlQ8hM3yhTTGGOQv1 H1ky13a1RxXC+uoTtYFyrxdBgPUd4eGF1tILHtK9NXnU6lhey90wDa2jmQOJQErgYuYPZriSuBXz QobK7tGcjMBgBQ1U+gxaTyThbXgxfb1MTjDx46hSl8Z35wIDAQABo4IB5TCCAeEwHwYDVR0jBBgw FoAUehNOAHRbxnhjZCfBL+KgW7x5xXswHQYDVR0OBBYEFO9wHp89I1B980w64KR38bmtuHFYMA4G A1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIx AQMFAjARBglghkgBhvhCAQEEBAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggr BgEFBQcCARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwVwYDVR0fBFAwTjBMoEqgSIZG aHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09NT0RPQ2xpZW50QXV0aGVudGljYXRpb25hbmRTZWN1 cmVFbWFpbENBLmNybDCBiAYIKwYBBQUHAQEEfDB6MFIGCCsGAQUFBzAChkZodHRwOi8vY3J0LmNv bW9kb2NhLmNvbS9DT01PRE9DbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3J0 MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21vZG9jYS5jb20wIAYDVR0RBBkwF4EVdG9hc3R5 QGRyYWdvbmRhdGEuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQCBaQ8dcaprzzREiMtsc2UtOPSHFiCy dcd5OjE6BN+pkcQozhx3nol9dFKJ+YfGvIxIjHmDGFTOgJgJvjRZ0D1Hw2WJCEtyD+U6yi/cnDFu Ksl039qafzbah6ft2r+GM0QufuFmrBi/bTWU3lGuhL8TKOvsWeLFkyGqtv9AJz2vg7j7dpxutLQY NWnrt7nS2x6p4f1LXu3iwczefyNNFUYwE9zXAT0Uwn48g2iijuf9vekfpqtHBmfSu0tSfd3FS3JC hmFp1fMxnWOnuZ529HFtGeYzr1K8Tp+JEVPjzPCxymVFsZ945Vzj0kc0DT3f9N5Gdw6uybrUwupM NHJJCB9VMYIDrjCCA6oCAQEwgakwgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1h bmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTkw NwYDVQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EC EQDbETdDYf7wYKjx8ymk38yAMAkGBSsOAwIaBQCgggHZMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0B BwEwHAYJKoZIhvcNAQkFMQ8XDTEzMDYyNzAwMDIwM1owIwYJKoZIhvcNAQkEMRYEFHPM6rDs5DE4 Bx7sVEE7Oy4qrMcTMIG6BgkrBgEEAYI3EAQxgawwgakwgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQI ExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBD QSBMaW1pdGVkMTkwNwYDVQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1 cmUgRW1haWwgQ0ECEQDbETdDYf7wYKjx8ymk38yAMIG8BgsqhkiG9w0BCRACCzGBrKCBqTCBkzEL MAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9y ZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0 aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIRANsRN0Nh/vBgqPHzKaTfzIAwDQYJKoZI hvcNAQEBBQAEggEAPbsKuS9MC8pDQEcOpNaZ+Yessn48WNHyyoSWBePfN7i4h8llfSa5BDBuou1/ 2HFJ999vqd12K4JKzatPW/QNkNmOaOq1aj9mJ4LRfDbC4XdZXc8IXbygg3RiwKzh1CHyucYAieDi XCOiocMW1Nbd0DRWcTU8ATHrCyOMo1D9DimXtP7s7+Ns5JocM0X6QND/JKP1EAcKuavaCT0ZUV4i BkNu7NV6YPp1AsoJi4Tpku22VZXlsbmWwhmXsIdw1moVoRsQvgsbqLYln77MtHKwNh1KJoPEmQmL D1qsNrh3z6wSYZLXG+XaR/CBJv1JFhueFMt1MfrhOeSM6UEkZOpiAQAAAAAAAA== --Apple-Mail=_17EED475-0486-4923-862D-97DDBBA66FA1-- From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 07:43:45 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EA997F51 for ; Thu, 27 Jun 2013 07:43:45 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id A9E9F1FED for ; Thu, 27 Jun 2013 07:43:44 +0000 (UTC) Received: from jre-mbp.elischer.org (ppp121-45-226-51.lns20.per1.internode.on.net [121.45.226.51]) (authenticated bits=0) by vps1.elischer.org (8.14.5/8.14.5) with ESMTP id r5R7hXgo056923 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 27 Jun 2013 00:43:37 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <51CBED22.6090702@freebsd.org> Date: Thu, 27 Jun 2013 15:43:30 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Chris Torek Subject: Re: expanding amd64 past the 1TB limit References: <201306261611.r5QGBie4025519@elf.torek.net> In-Reply-To: <201306261611.r5QGBie4025519@elf.torek.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 07:43:46 -0000 On 6/27/13 12:11 AM, Chris Torek wrote: [...] > them (but I still needed the map I drew of the page tables...). care to share? :-) From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 09:00:15 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4199BACC for ; Thu, 27 Jun 2013 09:00:15 +0000 (UTC) (envelope-from briansan24@gmail.com) Received: from mail-la0-x232.google.com (mail-la0-x232.google.com [IPv6:2a00:1450:4010:c03::232]) by mx1.freebsd.org (Postfix) with ESMTP id C51971372 for ; Thu, 27 Jun 2013 09:00:14 +0000 (UTC) Received: by mail-la0-f50.google.com with SMTP id dy20so505463lab.37 for ; Thu, 27 Jun 2013 02:00:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=OUY0DcyY/XeIyNeypPHDKA+8GZD6HMxe4YqOhSN/eVI=; b=o2G6N/0XYzVOGdI+EYz/9NSmdNvEirEFKyKtLJIDePu0B5feA8FWNeudJAVTW7wlez TBqWU8dxVkRUicm8J8rhVwtQzYZGZqlB0f3HCBb5yxCABwWLTxz4XFOG3uE/kK9/maW7 rx+xC/IDmPb4j05/9Kaj09uvzI1eu/Db6qH8llqVpIUOSoQKyeeGDexA5Shx/n9BiSQK Sxd0fQ8KejExnIa+xq8T7Ib3uVmWwQ0y8fyYgCjjCmnbhWF3Ad+UBT+m/qAUt48c1e97 6JBJ0GQP5r6cUtnJ4ZV5AnBFdcIYoMK02qI90KWq82pzM/0TIxPIY5ZoreESQbs+aron ZP3Q== MIME-Version: 1.0 X-Received: by 10.112.200.9 with SMTP id jo9mr2773761lbc.54.1372323613698; Thu, 27 Jun 2013 02:00:13 -0700 (PDT) Received: by 10.114.9.227 with HTTP; Thu, 27 Jun 2013 02:00:13 -0700 (PDT) Date: Thu, 27 Jun 2013 05:00:13 -0400 Message-ID: Subject: bsd boot sequence From: Brian Kim To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 09:00:15 -0000 howdy all, As a junior computer engineering major who has dreams of developing an operating system more ubiquitous than ms windows, I have come to appreciate the complexity and elegance of both the freebsd os and community. While achieving proficiency in C programming yet lacking in both UNIX utility know-how and shell scripting, I would rate my hacker savvy to be on an intermediate level. By the time I graduate from college, I hope to have a thorough understanding of the bsd kernel on the lowest level. Don't get me wrong though: while my end-game may be to develop my own operating system, I will forever be a contributor to freebsd. As my understanding of computers develops, you will undoubtedly see my name appear more frequently on the freebsd-hacker emails. Unfortunately, I am just not at the level of understanding just yet. To start, I was wondering if someone could briefly explain the operations and function calls that occur at boot time. I wish to thoroughly examine the freebsd source code but I'm afraid the sheer volume of code that exists leaves me with no real starting point for my research. Thank you all for your time. -- Best Wishes, Brian Kim From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 11:06:11 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BC617358 for ; Thu, 27 Jun 2013 11:06:11 +0000 (UTC) (envelope-from dimitry@andric.com) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) by mx1.freebsd.org (Postfix) with ESMTP id 815841B6D for ; Thu, 27 Jun 2013 11:06:11 +0000 (UTC) Received: from spaceball.andric.com (spaceball.andric.com [IPv6:2001:7b8:3a7:0:204:4bff:fe01:de8a]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 70E775C43; Thu, 27 Jun 2013 13:06:08 +0200 (CEST) Message-ID: <51CC1C9F.7080403@andric.com> Date: Thu, 27 Jun 2013 13:06:07 +0200 From: Dimitry Andric User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Thunderbird/22.0 MIME-Version: 1.0 To: Kevin Day , "freebsd-hackers@freebsd.org Hackers" Subject: Re: Can't use gcc in a clang built world References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Andrew Turner X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 11:06:11 -0000 On 2013-06-27 02:02, Kevin Day wrote: > Are you supposed to be able to use gcc to build userland binaries if you built world with clang? > > I'm on -CURRENT as of a few days ago (using armv6 but i'm not sure if that matters). Yes, the arch matters a lot. For arm, adding __clear_cache() to libgcc was explicitly disabled by Andrew here: http://svnweb.freebsd.org/base?view=revision&revision=244382 "Don't provide clear_cache or the __sync_* functions on ARM with clang as they are provided by clang as builtin functions." Maybe those functions should be in libgcc after all, if other programs depend on this. -Dimitry From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 11:59:05 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DCC1E452 for ; Thu, 27 Jun 2013 11:59:05 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) by mx1.freebsd.org (Postfix) with ESMTP id A13B31E66 for ; Thu, 27 Jun 2013 11:59:05 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.7/8.14.7) with ESMTP id r5RBx5EV000181; Thu, 27 Jun 2013 05:59:05 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.7/8.14.7/Submit) with ESMTP id r5RBx4pZ000178; Thu, 27 Jun 2013 05:59:05 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Thu, 27 Jun 2013 05:59:04 -0600 (MDT) From: Warren Block To: Brian Kim Subject: Re: bsd boot sequence In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Thu, 27 Jun 2013 05:59:05 -0600 (MDT) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 11:59:05 -0000 On Thu, 27 Jun 2013, Brian Kim wrote: > To start, I was wondering if someone could briefly explain the operations > and function calls that occur at boot time. I wish to thoroughly examine > the freebsd source code but I'm afraid the sheer volume of code that exists > leaves me with no real starting point for my research. These should be a start. man 8 boot http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot.html From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 18:22:30 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4A8DE541; Thu, 27 Jun 2013 18:22:30 +0000 (UTC) (envelope-from torek@elf.torek.net) Received: from elf.torek.net (50-73-42-1-utah.hfc.comcastbusiness.net [50.73.42.1]) by mx1.freebsd.org (Postfix) with ESMTP id 0E5BB181E; Thu, 27 Jun 2013 18:22:29 +0000 (UTC) Received: from elf.torek.net (localhost [127.0.0.1]) by elf.torek.net (8.14.5/8.14.5) with ESMTP id r5RIMMNN051953; Thu, 27 Jun 2013 12:22:22 -0600 (MDT) (envelope-from torek@elf.torek.net) Received: (from torek@localhost) by elf.torek.net (8.14.5/8.14.5/Submit) id r5RIMMCU051952; Thu, 27 Jun 2013 12:22:22 -0600 (MDT) (envelope-from torek) Date: Thu, 27 Jun 2013 12:22:22 -0600 (MDT) From: Chris Torek Message-Id: <201306271822.r5RIMMCU051952@elf.torek.net> To: julian@freebsd.org Subject: Re: expanding amd64 past the 1TB limit In-Reply-To: <51CBED22.6090702@freebsd.org> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (elf.torek.net [127.0.0.1]); Thu, 27 Jun 2013 12:22:22 -0600 (MDT) X-Mailman-Approved-At: Thu, 27 Jun 2013 18:44:32 +0000 Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 18:22:30 -0000 >> .. (but I still needed the map I drew of the page tables...). >care to share? :-) It's on paper (I need to get a whiteboard...). If I can ASCIIfy it ... Chris From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 18:54:22 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E5BDFFF9 for ; Thu, 27 Jun 2013 18:54:22 +0000 (UTC) (envelope-from andrew@fubar.geek.nz) Received: from nibbler.fubar.geek.nz (nibbler.fubar.geek.nz [199.48.134.198]) by mx1.freebsd.org (Postfix) with ESMTP id CC9E81943 for ; Thu, 27 Jun 2013 18:54:22 +0000 (UTC) Received: from bender.Home (97e76fc9.skybroadband.com [151.231.111.201]) by nibbler.fubar.geek.nz (Postfix) with ESMTPSA id 817EE5E1D5; Thu, 27 Jun 2013 18:48:42 +0000 (UTC) Date: Thu, 27 Jun 2013 19:48:35 +0100 From: Andrew Turner To: Dimitry Andric Subject: Re: Can't use gcc in a clang built world Message-ID: <20130627194835.4b1a7408@bender.Home> In-Reply-To: <51CC1C9F.7080403@andric.com> References: <51CC1C9F.7080403@andric.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org Hackers" , Kevin Day X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 18:54:23 -0000 On Thu, 27 Jun 2013 13:06:07 +0200 Dimitry Andric wrote: > On 2013-06-27 02:02, Kevin Day wrote: > > Are you supposed to be able to use gcc to build userland binaries > > if you built world with clang? > > > > I'm on -CURRENT as of a few days ago (using armv6 but i'm not sure > > if that matters). > > Yes, the arch matters a lot. For arm, adding __clear_cache() to > libgcc was explicitly disabled by Andrew here: > > http://svnweb.freebsd.org/base?view=revision&revision=244382 > > "Don't provide clear_cache or the __sync_* functions on ARM with clang > as they are provided by clang as builtin functions." > > Maybe those functions should be in libgcc after all, if other programs > depend on this. The reason to disable __clear_cache is incorrect in r244382 as it is a builtin in clang, but calls into an external copy of __clear_cache. The reason __clear_cache was disabled was because of a bug in clang where it is unable to compile a builtin function, however I only found this out recently. The issue with clang has been fixed, and, as of r251791 __clear_cache is enabled in compiler-rt. Andrew > > -Dimitry > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 20:33:47 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 48A01BA for ; Thu, 27 Jun 2013 20:33:47 +0000 (UTC) (envelope-from dimitry@andric.com) Received: from tensor.andric.com (tensor.andric.com [IPv6:2001:7b8:3a7:1:2d0:b7ff:fea0:8c26]) by mx1.freebsd.org (Postfix) with ESMTP id 0FF5C1DAD for ; Thu, 27 Jun 2013 20:33:47 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7::4de1:c0cf:265b:ec55] (unknown [IPv6:2001:7b8:3a7:0:4de1:c0cf:265b:ec55]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id E52005C43; Thu, 27 Jun 2013 22:33:43 +0200 (CEST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: Can't use gcc in a clang built world From: Dimitry Andric In-Reply-To: <20130627194835.4b1a7408@bender.Home> Date: Thu, 27 Jun 2013 22:33:35 +0200 Content-Transfer-Encoding: 7bit Message-Id: References: <51CC1C9F.7080403@andric.com> <20130627194835.4b1a7408@bender.Home> To: Andrew Turner X-Mailer: Apple Mail (2.1508) Cc: "freebsd-hackers@freebsd.org Hackers" , Kevin Day X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 20:33:47 -0000 On Jun 27, 2013, at 20:48, Andrew Turner wrote: > On Thu, 27 Jun 2013 13:06:07 +0200 > Dimitry Andric wrote: > >> On 2013-06-27 02:02, Kevin Day wrote: >>> Are you supposed to be able to use gcc to build userland binaries >>> if you built world with clang? >>> >>> I'm on -CURRENT as of a few days ago (using armv6 but i'm not sure >>> if that matters). >> >> Yes, the arch matters a lot. For arm, adding __clear_cache() to >> libgcc was explicitly disabled by Andrew here: >> >> http://svnweb.freebsd.org/base?view=revision&revision=244382 >> >> "Don't provide clear_cache or the __sync_* functions on ARM with clang >> as they are provided by clang as builtin functions." >> >> Maybe those functions should be in libgcc after all, if other programs >> depend on this. > > The reason to disable __clear_cache is incorrect in r244382 as it is a > builtin in clang, but calls into an external copy of __clear_cache. The > reason __clear_cache was disabled was because of a bug in clang where > it is unable to compile a builtin function, however I only found this > out recently. > > The issue with clang has been fixed, and, as of r251791 __clear_cache > is enabled in compiler-rt. Okay, but apparently compilers such as gcc do not use compiler-rt at all, but expect the function to be available in libgcc instead. So since clang can compile the definition on arm now, shall we re-enable it for libgcc too? -Dimitry From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 21:20:20 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7AE58B0E for ; Thu, 27 Jun 2013 21:20:20 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 534011F4A for ; Thu, 27 Jun 2013 21:20:20 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A7E9BB941; Thu, 27 Jun 2013 17:20:19 -0400 (EDT) From: John Baldwin To: freebsd-hackers@freebsd.org Subject: Re: Exposing driver's GPIOs through gpiobus Date: Thu, 27 Jun 2013 13:49:32 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201306271349.32203.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 27 Jun 2013 17:20:19 -0400 (EDT) Cc: Ryan Stone X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 21:20:20 -0000 On Wednesday, June 12, 2013 11:36:04 pm Ryan Stone wrote: > At $WORK we have some custom boards with multi-port uarts managed by puc. > The uart devices happen to provide some GPIOs, and our hardware designers > have appropriated those GPIOs for various purposes entirely unrelated to > the uart. > > I'm looking for a clean way to provide access to the GPIOs. It occurred to > me that this was a problem that should be solved through newbus, and lo and > behold I have discovered that FreeBSD provides a gpiobus driver that seems > suitable. I've been playing around with this for a couple of days and I > have a solutions that is working, but there are aspects that I am unhappy > with. I also quite unfamiliar with newbus, so there easily could be better > ways to approach the problem that I haven't thought of. > > What I ended up doing was making gpiobus and gpioc children of the puc > bus. In puc_bfe_attach() I create two new child devices of the puc device > with device_add_child(), one with the gpioc devclass and one with the > gpiobus devclass. I then attach both children with > device_probe_and_attach(). I make the puc_pci driver itself provide > implementations of the various gpio methods (like gpio_pin_get) so they can > be inherited by the child devices. > > Things start to get somewhat messy in the gpio client code. I have the > same image running on many different hardware types, so I can't use device > hints to create a child device of the gpiobus. Instead my kernel module > tracks down the device_t for the puc, finds the gpiobus child, and uses > BUS_ADD_CHILD to create a child of the gpiobus. I had to add a new gpiobus > method to allocate GPIO pins to my driver instance. Once that's done, I > can toggle GPIOs and whatnot using methods on my driver instance. > > The things that I'm most unhappy with (newbus-wise, anyway) are: > > 1) By default the gpioc and gpiobus drivers were claiming the uart children > of the puc. I had to decrease their priority in bus_probe to > BUS_PROBE_LOW_PRIORITY to work around the problem. I really don't think > that was the right solution. I guess I could introduce a new device that > is a child of the puc, make sure that it will not claim the uarts, and then > make the gpioc and gpiobus children of this device. > > 2) I'm not sure how to clean up my child device when my module is > unloaded. Currently I am checking if it already exists on load and reusing > it if so. I may be missing something obvious here. Just leave the device around and reuse it. In your identify routine do something like this: static void agp_i810_identify(driver_t *driver, device_t parent) { if (device_find_child(parent, "agp", -1) == NULL && agp_i810_match(parent)) device_add_child(parent, "agp", -1); } > > 3) I really don't like the way that I am adding my child to gpiobus. Upon > writing this it occurs to me that device_identify would be the canonical > way to do this. Previously it wasn't clear to me how to fit > device_identify into the current architecture of the gpio client but I see > how it can be done now. Yes, device_identify is what you want. I think it will also solve problem 1 for you as well. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 21:20:22 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B1DB4B0F for ; Thu, 27 Jun 2013 21:20:22 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 908211F4B for ; Thu, 27 Jun 2013 21:20:22 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 01B8AB94B; Thu, 27 Jun 2013 17:20:22 -0400 (EDT) From: John Baldwin To: freebsd-hackers@freebsd.org Subject: Re: BUS_PROBE_NOWILDCARD behaviour doesn't seem to match DEVICE_PROBE(9) Date: Thu, 27 Jun 2013 13:51:54 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201306271351.54680.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 27 Jun 2013 17:20:22 -0400 (EDT) Cc: Ryan Stone X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 21:20:22 -0000 On Thursday, June 20, 2013 10:54:39 am Ryan Stone wrote: > http://www.freebsd.org/cgi/man.cgi?query=DEVICE_PROBE&apropos=0&sektion=0&manpath=FreeBSD%208.2- RELEASE&format=html > > DEVICE_PROBE(9) has this to say about BUS_PROBE_NOWILDCARD: > > The driver expects its parent to tell it which children to manage and no > probing is really done. The device only matches if its parent bus > specifically said to use this driver. Perhaps run this by Warner to make sure? (There is also a new-bus@ list FYI). -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 21:26:50 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 177BCFDA for ; Thu, 27 Jun 2013 21:26:50 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-x22e.google.com (mail-ee0-x22e.google.com [IPv6:2a00:1450:4013:c00::22e]) by mx1.freebsd.org (Postfix) with ESMTP id A5BFD1F9D for ; Thu, 27 Jun 2013 21:26:49 +0000 (UTC) Received: by mail-ee0-f46.google.com with SMTP id d41so655910eek.33 for ; Thu, 27 Jun 2013 14:26:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=UAiQgGwAe+Yj/ChRDXoxuuxVoZuwV+CGeSbOcCqL+IE=; b=zBBs6g620UKEI5RQ5/14MPNRmvogHNO45MLlUDKoHwuG9VvY8AFmCjr1v+oUvRIVWx kKiQHtmHk6ueDbNe7FmqdmbwaNkteDe/tIWqMM5pJLwUs2n57UrLn1vSPX6FyFVvbBHl +JBY90sa35yQOAnG+tW+pzwiVmD1UEwEpUIf0zOTJGIRgY/Dfz9IZprFjyScjMLnCGPr jwazps8Sgt4kVprniurLp5DeXnGC+ZqplzkaQ3jWDtpxmcKRTzzm0kjVYomKduqcMyqm G7mqBbWesandMxP9x7WTm1ZZTf2DHimJDXB38qj5i6swANwpDFnxSb45K3jIUie8EOg8 3uTg== X-Received: by 10.15.34.129 with SMTP id e1mr10845767eev.80.1372368408613; Thu, 27 Jun 2013 14:26:48 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPSA id y1sm6335306eew.3.2013.06.27.14.26.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 27 Jun 2013 14:26:47 -0700 (PDT) Sender: Alexander Motin Message-ID: <51CCAE14.6040504@FreeBSD.org> Date: Fri, 28 Jun 2013 00:26:44 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: hackers@freebsd.org Subject: b_freelist TAILQ/SLIST Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 21:26:50 -0000 Hi. While doing some profiles of GEOM/CAM IOPS scalability, on some test patterns I've noticed serious congestion with spinning on global pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is already very simple, I've tried to optimize probably the only thing possible there: switch bswlist from TAILQ to SLIST. As I can see, b_freelist field of struct buf is really used as TAILQ in some other places, so I've just added another SLIST_ENTRY field. And result appeared to be surprising -- I can no longer reproduce the issue at all. May be it was just unlucky synchronization of specific test, but I've seen in on two different systems and rechecked results with/without patch three times. The present patch is here: http://people.freebsd.org/~mav/buf_slist.patch The question is how to do it better? What is the KPI/KBI policy for struct buf? I could replace b_freelist by a union and keep KBI, but partially break KPI. Or I could add another field, probably breaking KBI, but keeping KPI. Or I could do something handmade with no breakage. Or this change is just a bad idea? -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 21:23:37 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 906CDEB5; Thu, 27 Jun 2013 21:23:37 +0000 (UTC) (envelope-from torek@elf.torek.net) Received: from elf.torek.net (50-73-42-1-utah.hfc.comcastbusiness.net [50.73.42.1]) by mx1.freebsd.org (Postfix) with ESMTP id 4D9271F7A; Thu, 27 Jun 2013 21:23:37 +0000 (UTC) Received: from elf.torek.net (localhost [127.0.0.1]) by elf.torek.net (8.14.5/8.14.5) with ESMTP id r5RLNadb064879; Thu, 27 Jun 2013 15:23:36 -0600 (MDT) (envelope-from torek@elf.torek.net) Received: (from torek@localhost) by elf.torek.net (8.14.5/8.14.5/Submit) id r5RLNapo064878; Thu, 27 Jun 2013 15:23:36 -0600 (MDT) (envelope-from torek) Date: Thu, 27 Jun 2013 15:23:36 -0600 (MDT) From: Chris Torek Message-Id: <201306272123.r5RLNapo064878@elf.torek.net> To: julian@freebsd.org Subject: Re: expanding amd64 past the 1TB limit In-Reply-To: <51CBED22.6090702@freebsd.org> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (elf.torek.net [127.0.0.1]); Thu, 27 Jun 2013 15:23:36 -0600 (MDT) X-Mailman-Approved-At: Thu, 27 Jun 2013 21:42:50 +0000 Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 21:23:37 -0000 OK, I wasted :-) way too much time, but here's a text file that can be comment-ified or stored somewhere alongside the code or whatever... (While drawing this I realized that there's at least one "wasted" page if the machine has .5 TB or less: we can just leave zero slots in the corresponding L4 direct-map entries. But that would require switching to the bcopy() method also mentioned below. Or indexing into vmspace0.vm_pmap.pm_pml4, which is basically the same thing.) Chris ----- There are six -- or sometimes five -- sets of pages allocated here at boot time to map physical memory in two ways. Note that each page, regardless of level, stores 512 PTEs (or PDEs or PDPs, but let's just use PTE here and prefix it with "level" as needed: 4, 3, 2, or 1.) There is one page for the top level, L4, page table entries. Each L4 PTE maps 512 GB of space. Unless it's marked "invalid", no L4 PTE can be marked "stop here": it either is marked as "this address is invalid", or it points to one physically-adressed page full of L3 PTEs. Eventually, those L3 PTEs will map-or-reject half a terabyte. 512 entries, each mapping .5 TB, allow us to map 256 TB, which is as much as the hardware supports (there are, in effect, only 48 virtual address bits: the top 16 bits must match the 47th bit). The L4 entry halfway down, at PML4PML4I, is set to point back to this page itself; that's the "recursive page table" for user space, which we do nothing else with at boot time. We need (up to) NDMPML4E pages, each holding 512 L3 PTEs, for the direct map space. If the processor supports 1 GB pages, an L3 PTE can be marked with "stop here" and these L3 PTEs each grant (or forbid) access to 1 GB of physical space at a time. A system with, say, 3 GB of RAM starting at 0 can map it all with three L3 PTEs: "address 0 is valid for 1GB", "address 1GB is valid for 1GB", "address 2GB is valid for 1GB". The remaining L3 PTEs are zero, making the remaining address space invalid. If the processor does not support 1 GB pages, or if there is less than 1 GB of RAM "at the end" (e.g., if the system has 4.5 GB), the L3 PTEs may need to point to more pages holding L2 PTEs. These L2 PTEs always support 2 MB pages. Each page of L2 PTEs maps 1 GB. So a machine with 4.5 GB and 1 GB mappings needs one L3 page with four valid 1 GB L3 PTEs and then one L3 PTE pointing to one page of L2 PTEs. That one page of L2 PTEs is half-filled, containing 256 2MB-sized PTEs, mapping the 512 MB. The remaining half of that page is zero, making the remaining addresses invalid. Pictorially, and adding the names of the physical page(s), thus far we have this. (Note, the L4 PTE page is drawn more than twice as tall as the L3 and L2 pages, just to get space for arrows.) LEVEL 4: LEVEL 3: LEVEL 2: _._ KPML4phys v \ +---------+ | | 0: | | |---------| | | 1: | | DMPDPphys DMPDphys ( ... ) | .-> +---------+ +----------------+ | 127: | | / | 0: 0GB | .-> | 0: 4GB | |---------| | | | 1: 1GB | / | 1: 4GB+2MB | PML4PML4I: | 128: *--|--/ | | 2: 2GB | / | 2: 4GB+4MB | |---------| | | 3: 3GB | / ( ... ) | 129: | | | 4: *--|-/ | 255: 4.5GB-2MB | | ... | | | 5: | | 256: | ________ |---------| | ( ... ) | 257: | / DMPML4I: | *--|-----/ | 511: | ( ... ) NDMPML4E |---------| +---------+ +----------------+ \________ | *--|---------> | 0: | |---------| | 1: | | | | 2: | (These are used only |---------| | 3: | if the system has more | ... | ( ... ) than 512 GB) ( |---------| ) | 509: | ( | 510: see below ) | 510: | ( |---------| ) | 511: | ( | 511: see below ) +---------+ +---------+ If the hardware supports 1GB pages, "ndm1g" is the number of gigabyte entries (4 in the example above). Otherwise it's just zero. Meanwhile "ndmpdp" is the number of gigabytes of RAM that need to be mapped, in this case 5. Thus, if ndmpdp > ndm1g, we need ndmpdp-ndm1g pages to hold some L2 PTEs. Now we get to the weirder case of the kernel itself (both its non-direct-mapped dynamically allocated virtual memory, and its text/data/bss). The branch offset limitations encourage the placement of the kernel's text, etc., in the last 2 GB of virtual space, i.e., starting at 0xffff.ffff.f800.0000. But, we want a reasonable amount of room for dynamic VM. So we give the kernel at least 512 GB of VM -- that's one L4 PTE -- while making sure that the text snuggles up close to the end of the space, in that last 2 GB of the at-least-512-GB area. Meanwhile, the boot loader has loaded the kernel into relatively low physical memory addresses. If KPML4I is 511 (and it actually is), this uses the final L4 slot to map the kernel. If we want to allow kernel VM to have more than 512 GB available, though, we need extra space below KPML4I, i.e., starting at KPMLBASE. So we allocate NKPML4E pages that we set up as L3 PTEs, and point the last NKPML4E slots in the L4 page table here. If NKPML4E is 4, for instance, we will have this: last part of KPML4phys: ( ... ) .----> [page #0 of all-zero L3 PTEs] | DMPML4I | / ( ... ) | .--> [page #1 of all-zero L3 PTEs] | 507: | | / | 508: *--|--/ | .-> [page #2 of all-zero L3 PTEs] | 509: *--|----/ | | 510: *--|------/ | 511: *--|---------> [page #3 of L3 PTEs, see below] +---------+ The reason for having those "empty" (all-zero) PTE pages is that whenever new processes are created, in pmap_pinit(), they have their (new) L4 PTE page set up to point to the *same* physical pages that the kernel is using. Thus, if the kernel creates or destroys any level-3-or-below mapping by writing into any of the above four pages, that mapping is also created/destroyed in all processes. Similarly, the NDMPML4 pages starting at DMPDPphys are mapped identically in all processes. The kernel can therefore "borrow" a user pmap at any time, i.e., there's no need to adjust the CPU's CR4 on entry to the kernel. (If we used bcopy() to copy the kernel pmap's NKPML4E and NDMPML4E entries into the new pmap, the L3 pages would not have to be physically contiguous, but the KVA ones would still all have to exist. It's free to allocate physically contiguous pages here anyway though.) So, the last NKPML4E slots in KPML4phys point to the following page tables, which use all of L3, L2, and L1 style PTEs. (Note that we did not need any L1 PTEs for the direct map, which always uses 2MB or 1GB super-pages.) LEVEL 3: LEVEL 2: LEVEL 1: (assuming NKPML4=4) (nkpt pages) KPDPphys KPTphys +---------+ +---------------+ page 0 | 0: | .-> | 0: 0 KB | | 1: | / | 1: 4 KB | | 2: | / | 2: 8 KB | | 3: | / | 3: 12 KB | ( ... ) | ( ... ) | 509: | | | 509: 2MB-12KB | | 510: | | | 510: 2MB-8KB | | 511: | | | 511: 2MB-4KB | +---------+ | +---------------+ page 1 | 0: | | .-> | 0: 2 MB | | 1: | | / | 1: 2MB+4KB | | 2: | | | ( ... ) | 3: | | | ( ... ) ( ... ) | | +---------------+ | 509: | | | .-> ( ... ) | 510: | | | | ( ... ) | 511: | KPDphys | | | +---------------+ +---------+ +---------+ | | | ..( ... ... ... ) page 2 | 0: | .---> | 0: *--|--/ | | . [etc] | 1: | / | 1: *--|---/ | . | 2: | | | 2: *--|-----/ . | 3: | | | 3: *--|---.... ( ... ) | ( ... ) | 509: | | | 509: ...| | 510: | | | 510: ...| | 511: | | | 511: ...| +---------+ | +---------+ page 3 | 0: | | .-> | 0: ...| | 1: | | / ( ... ) | 2: | | | ( ... ) | 3: | | | ( ... ) ( ... ) | | ( ... ) | 509: | | | ( ... ) | 510: *--|--/ | ( ... ) | 511: *--|----/ | 511: | +---------+ +---------+ There are nkpdpe pages at KPDphys, where nkpdpe is either 1 or 2. One page maps 1 GB, and the other page maps the remaining 1 GB. Remember that kernel text+data+bss lives in the final 2 GB of the virtual address space, so there cannot be more than 2 GB. These one or two pages map nkpt pages at KPTphys. From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 27 22:32:03 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 98B69810; Thu, 27 Jun 2013 22:32:03 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-ob0-x232.google.com (mail-ob0-x232.google.com [IPv6:2607:f8b0:4003:c01::232]) by mx1.freebsd.org (Postfix) with ESMTP id 4187812F8; Thu, 27 Jun 2013 22:32:03 +0000 (UTC) Received: by mail-ob0-f178.google.com with SMTP id fb19so1297059obc.37 for ; Thu, 27 Jun 2013 15:32:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=xO9MKfzYYNaypX6l/3YkRbPJMRIzjJNjqzNn6+7mdsM=; b=EwMRL9NkWckttByA4JVq6W53dBnnsrq5qw9PHqHbNrMh0sGWv8Cja8Rdmo7O7cRsoS J05iI70SvlsukoqP1qHBdHR0EmrrQ8HodYOGOMCglNBaL5+x+Zm9vpY1VSD1takHjcNC Ses78+tmfpx0HDNrPWMCQeBHs8z24BaA4dEcNuOBeZ8oXARdK0tsBfV+cTlbLg5f38c+ 3s+gyhTtMyVL2WGGC3p1LmnIF/L86kw5rZ8Mnym6dQ6kJd1bdcCFsj/kwxGrKap/K2id oZPx93AaAdoHWBxDC6KHypFr7RcR6vxB+5jm5kOZrg0Vxb3iDMqvwSU6K5Y6Dqpq6cG1 qvmQ== MIME-Version: 1.0 X-Received: by 10.60.83.116 with SMTP id p20mr3897168oey.83.1372372322743; Thu, 27 Jun 2013 15:32:02 -0700 (PDT) Received: by 10.182.227.231 with HTTP; Thu, 27 Jun 2013 15:32:02 -0700 (PDT) In-Reply-To: <201306272123.r5RLNapo064878@elf.torek.net> References: <51CBED22.6090702@freebsd.org> <201306272123.r5RLNapo064878@elf.torek.net> Date: Fri, 28 Jun 2013 00:32:02 +0200 Message-ID: Subject: Re: expanding amd64 past the 1TB limit From: Oliver Pinter To: Chris Torek Content-Type: text/plain; charset=ISO-8859-1 Cc: alc@freebsd.org, freebsd-hackers@freebsd.org, kib@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 22:32:03 -0000 On 6/27/13, Chris Torek wrote: > OK, I wasted :-) way too much time, but here's a text file that > can be comment-ified or stored somewhere alongside the code or > whatever... > > (While drawing this I realized that there's at least one "wasted" > page if the machine has .5 TB or less: we can just leave zero > slots in the corresponding L4 direct-map entries. But that would > require switching to the bcopy() method also mentioned below. Or > indexing into vmspace0.vm_pmap.pm_pml4, which is basically the > same thing.) > > Chris > > ----- > > There are six -- or sometimes five -- sets of pages allocated here > at boot time to map physical memory in two ways. Note that each > page, regardless of level, stores 512 PTEs (or PDEs or PDPs, but > let's just use PTE here and prefix it with "level" as needed: 4, > 3, 2, or 1.) > > There is one page for the top level, L4, page table entries. Each > L4 PTE maps 512 GB of space. Unless it's marked "invalid", no L4 > PTE can be marked "stop here": it either is marked as "this > address is invalid", or it points to one physically-adressed page > full of L3 PTEs. Eventually, those L3 PTEs will map-or-reject > half a terabyte. 512 entries, each mapping .5 TB, allow us to map > 256 TB, which is as much as the hardware supports (there are, in > effect, only 48 virtual address bits: the top 16 bits must match > the 47th bit). > > The L4 entry halfway down, at PML4PML4I, is set to point back to > this page itself; that's the "recursive page table" for user > space, which we do nothing else with at boot time. > > We need (up to) NDMPML4E pages, each holding 512 L3 PTEs, for the > direct map space. If the processor supports 1 GB pages, an L3 PTE > can be marked with "stop here" and these L3 PTEs each grant (or > forbid) access to 1 GB of physical space at a time. A system > with, say, 3 GB of RAM starting at 0 can map it all with three L3 > PTEs: "address 0 is valid for 1GB", "address 1GB is valid for > 1GB", "address 2GB is valid for 1GB". The remaining L3 PTEs are > zero, making the remaining address space invalid. > > If the processor does not support 1 GB pages, or if there is less > than 1 GB of RAM "at the end" (e.g., if the system has 4.5 GB), > the L3 PTEs may need to point to more pages holding L2 PTEs. > These L2 PTEs always support 2 MB pages. Each page of L2 PTEs > maps 1 GB. So a machine with 4.5 GB and 1 GB mappings needs one L3 > page with four valid 1 GB L3 PTEs and then one L3 PTE pointing to > one page of L2 PTEs. That one page of L2 PTEs is half-filled, > containing 256 2MB-sized PTEs, mapping the 512 MB. The remaining > half of that page is zero, making the remaining addresses invalid. > > Pictorially, and adding the names of the physical page(s), thus > far we have this. (Note, the L4 PTE page is drawn more than twice > as tall as the L3 and L2 pages, just to get space for arrows.) > > LEVEL 4: LEVEL 3: LEVEL 2: > _._ > KPML4phys v \ > +---------+ | > | 0: | | > |---------| | > | 1: | | DMPDPphys DMPDphys > ( ... ) | .-> +---------+ +----------------+ > | 127: | | / | 0: 0GB | .-> | 0: 4GB | > |---------| | | | 1: 1GB | / | 1: 4GB+2MB | > PML4PML4I: | 128: *--|--/ | | 2: 2GB | / | 2: 4GB+4MB | > |---------| | | 3: 3GB | / ( ... ) > | 129: | | | 4: *--|-/ | 255: 4.5GB-2MB | > | ... | | | 5: | | 256: | > ________ |---------| | ( ... ) | 257: | > / DMPML4I: | *--|-----/ | 511: | ( ... ) > NDMPML4E |---------| +---------+ +----------------+ > \________ | *--|---------> | 0: | > |---------| | 1: | > | | | 2: | (These are used only > |---------| | 3: | if the system has more > | ... | ( ... ) than 512 GB) > ( |---------| ) | 509: | > ( | 510: see below ) | 510: | > ( |---------| ) | 511: | > ( | 511: see below ) +---------+ > +---------+ > > If the hardware supports 1GB pages, "ndm1g" is the number of > gigabyte entries (4 in the example above). Otherwise it's just > zero. Meanwhile "ndmpdp" is the number of gigabytes of RAM that > need to be mapped, in this case 5. Thus, if ndmpdp > ndm1g, we > need ndmpdp-ndm1g pages to hold some L2 PTEs. > > Now we get to the weirder case of the kernel itself (both its > non-direct-mapped dynamically allocated virtual memory, and its > text/data/bss). The branch offset limitations encourage the > placement of the kernel's text, etc., in the last 2 GB of virtual > space, i.e., starting at 0xffff.ffff.f800.0000. But, we want > a reasonable amount of room for dynamic VM. So we give the kernel > at least 512 GB of VM -- that's one L4 PTE -- while making sure that > the text snuggles up close to the end of the space, in that last 2 GB > of the at-least-512-GB area. > > Meanwhile, the boot loader has loaded the kernel into relatively > low physical memory addresses. > > If KPML4I is 511 (and it actually is), this uses the final L4 slot > to map the kernel. If we want to allow kernel VM to have more > than 512 GB available, though, we need extra space below KPML4I, > i.e., starting at KPMLBASE. So we allocate NKPML4E pages that > we set up as L3 PTEs, and point the last NKPML4E slots in the L4 > page table here. If NKPML4E is 4, for instance, we will have > this: > > last part of KPML4phys: > ( ... ) .----> [page #0 of all-zero L3 PTEs] > | DMPML4I | / > ( ... ) | .--> [page #1 of all-zero L3 PTEs] > | 507: | | / > | 508: *--|--/ | .-> [page #2 of all-zero L3 PTEs] > | 509: *--|----/ | > | 510: *--|------/ > | 511: *--|---------> [page #3 of L3 PTEs, see below] > +---------+ > > The reason for having those "empty" (all-zero) PTE pages is that > whenever new processes are created, in pmap_pinit(), they have > their (new) L4 PTE page set up to point to the *same* physical > pages that the kernel is using. Thus, if the kernel creates or > destroys any level-3-or-below mapping by writing into any of the > above four pages, that mapping is also created/destroyed in all > processes. Similarly, the NDMPML4 pages starting at DMPDPphys are > mapped identically in all processes. The kernel can therefore > "borrow" a user pmap at any time, i.e., there's no need to adjust > the CPU's CR4 on entry to the kernel. > > (If we used bcopy() to copy the kernel pmap's NKPML4E and NDMPML4E > entries into the new pmap, the L3 pages would not have to be > physically contiguous, but the KVA ones would still all have to > exist. It's free to allocate physically contiguous pages here > anyway though.) > > So, the last NKPML4E slots in KPML4phys point to the following > page tables, which use all of L3, L2, and L1 style PTEs. (Note > that we did not need any L1 PTEs for the direct map, which always > uses 2MB or 1GB super-pages.) > > LEVEL 3: LEVEL 2: LEVEL 1: > > (assuming NKPML4=4) (nkpt pages) > KPDPphys KPTphys > +---------+ +---------------+ > page 0 | 0: | .-> | 0: 0 KB | > | 1: | / | 1: 4 KB | > | 2: | / | 2: 8 KB | > | 3: | / | 3: 12 KB | > ( ... ) | ( ... ) > | 509: | | | 509: 2MB-12KB | > | 510: | | | 510: 2MB-8KB | > | 511: | | | 511: 2MB-4KB | > +---------+ | +---------------+ > page 1 | 0: | | .-> | 0: 2 MB | > | 1: | | / | 1: 2MB+4KB | > | 2: | | | ( ... ) > | 3: | | | ( ... ) > ( ... ) | | +---------------+ > | 509: | | | .-> ( ... ) > | 510: | | | | ( ... ) > | 511: | KPDphys | | | +---------------+ > +---------+ +---------+ | | | ..( ... ... ... ) > page 2 | 0: | .---> | 0: *--|--/ | | . [etc] > | 1: | / | 1: *--|---/ | . > | 2: | | | 2: *--|-----/ . > | 3: | | | 3: *--|---.... > ( ... ) | ( ... ) > | 509: | | | 509: ...| > | 510: | | | 510: ...| > | 511: | | | 511: ...| > +---------+ | +---------+ > page 3 | 0: | | .-> | 0: ...| > | 1: | | / ( ... ) > | 2: | | | ( ... ) > | 3: | | | ( ... ) > ( ... ) | | ( ... ) > | 509: | | | ( ... ) > | 510: *--|--/ | ( ... ) > | 511: *--|----/ | 511: | > +---------+ +---------+ > > There are nkpdpe pages at KPDphys, where nkpdpe is either 1 or 2. > One page maps 1 GB, and the other page maps the remaining 1 GB. > Remember that kernel text+data+bss lives in the final 2 GB of the > virtual address space, so there cannot be more than 2 GB. These > one or two pages map nkpt pages at KPTphys. added two VM guru, to CC > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 04:15:25 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4D86A703 for ; Fri, 28 Jun 2013 04:15:25 +0000 (UTC) (envelope-from shonnur@chelsio.com) Received: from stargate.chelsio.com (stargate.chelsio.com [67.207.112.58]) by mx1.freebsd.org (Postfix) with ESMTP id 1F96910D5 for ; Fri, 28 Jun 2013 04:15:24 +0000 (UTC) Received: from maui.asicdesigners.com (maui.asicdesigners.com [10.192.180.15]) by stargate.chelsio.com (8.13.1/8.13.1) with SMTP id r5S4FOo3017378 for ; Thu, 27 Jun 2013 21:15:24 -0700 Received: from corona.asicdesigners.com ([10.192.160.6]) by maui.asicdesigners.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Jun 2013 21:15:24 -0700 Received: from NICE.asicdesigners.com (10.192.160.7) by corona.asicdesigners.com (10.192.160.6) with Microsoft SMTP Server (TLS) id 8.3.83.0; Thu, 27 Jun 2013 21:15:24 -0700 Received: from NICE.asicdesigners.com ([fe80::51b2:ba95:9d72:babc]) by nice.asicdesigners.com ([fe80::51b2:ba95:9d72:babc%15]) with mapi id 14.02.0247.003; Thu, 27 Jun 2013 21:15:24 -0700 From: Sreenivasa Honnur To: "freebsd-hackers@freebsd.org" Subject: IPv6 - sobind fails with 49 Thread-Topic: IPv6 - sobind fails with 49 Thread-Index: Ac47MtKWUXzqp/7TS3iQhjr60aw7/w4gdA6QAABSCsA= Date: Fri, 28 Jun 2013 04:15:23 +0000 Message-ID: References: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-puzzleid: {E9A6B737-D8C0-41F2-AE88-2C65ECA71EC2} x-cr-hashedpuzzle: AVTP Bgqu BsOR CmeC D+wT Em2F FQMd FeMs Fm88 Fz4W Gfsd IA6w IWFF IyyE JJN+ JQiI; 1; ZgByAGUAZQBiAHMAZAAtAGgAYQBjAGsAZQByAHMAQABmAHIAZQBlAGIAcwBkAC4AbwByAGcA; Sosha1_v1; 7; {E9A6B737-D8C0-41F2-AE88-2C65ECA71EC2}; cwBoAG8AbgBuAHUAcgBAAGMAaABlAGwAcwBpAG8ALgBjAG8AbQA=; Fri, 28 Jun 2013 04:15:20 GMT;SQBQAHYANgAgAC0AIABzAG8AYgBpAG4AZAAgAGYAYQBpAGwAcwAgAHcAaQB0AGgAIAA0ADkA x-originating-ip: [10.193.190.128] MIME-Version: 1.0 X-OriginalArrivalTime: 28 Jun 2013 04:15:24.0641 (UTC) FILETIME=[1CE62510:01CE73B6] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 04:15:25 -0000 I am writing a kernel socket program which binds to a IPv6 address, so bind= always fails with 49. Below is the code snippet I am using, is something w= rong here? roundhay# uname -a FreeBSD roundhay 9.1-RELEASE FreeBSD 9.1-RELEASE #2: Mon Apr 8 16:15:06 IS= T 2013 root@roundhay:/usr/obj/home/freebsd.org/sys/TOED amd64 Ifconfig/ping6 output: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D cxgbe1: flags=3D8843 metric 0 mtu 1= 500 options=3D6c07bb ether 00:07:43:11:89:88 inet6 2010::102 prefixlen 64 inet6 fe80::207:43ff:fe11:8988%cxgbe1 prefixlen 64 scopeid 0xd inet6 fe80::1%cxgbe1 prefixlen 64 scopeid 0xd nd6 options=3D21 media: Ethernet 10Gbase-SR status: active roundhay# ping6 2010::101 PING6(56=3D40+8+8 bytes) 2010::102 --> 2010::101 16 bytes from 2010::101, icmp_seq=3D0 hlim=3D64 time=3D0.915 ms 16 bytes from 2010::101, icmp_seq=3D1 hlim=3D64 time=3D0.168 ms ^C --- 2010::101 ping6 statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/std-dev =3D 0.168/0.541/0.915/0.374 ms Code: struct sockaddr_in6 saddr6; rv =3D socreate(AF_INET6, &sock, SOCK_STREAM, IPPROTO_TCP, td->td_ucred, td); if (rv !=3D 0) { printf("sock create ipv6 %s failed %d.\n", tbuf, rv); return NULL; } saddr6.sin6_family =3D AF_INET6; rv =3D inet_pton(AF_INET6, "2010::102", &saddr6.sin6_addr);= =3D=3D> returns 1, which indicates it's a valid IPV6 address printf("inet_pton retunred:%d\n", rv); saddr6.sin6_port =3D htons(3260); saddr6.sin6_len =3D sizeof(saddr6); rv =3D sobind(sock, (struct sockaddr *)&saddr6, td); =3D=3D= > fails with return value of 49 From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 06:57:37 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8A0FC5B8; Fri, 28 Jun 2013 06:57:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 2C91B188C; Fri, 28 Jun 2013 06:57:36 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r5S6vWZl027592; Fri, 28 Jun 2013 09:57:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r5S6vWZl027592 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r5S6vWrE027591; Fri, 28 Jun 2013 09:57:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 28 Jun 2013 09:57:32 +0300 From: Konstantin Belousov To: Alexander Motin Subject: Re: b_freelist TAILQ/SLIST Message-ID: <20130628065732.GL91021@kib.kiev.ua> References: <51CCAE14.6040504@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/UjIn1MmbqnRGnkw" Content-Disposition: inline In-Reply-To: <51CCAE14.6040504@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 06:57:37 -0000 --/UjIn1MmbqnRGnkw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: > Hi. >=20 > While doing some profiles of GEOM/CAM IOPS scalability, on some test=20 > patterns I've noticed serious congestion with spinning on global=20 > pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is=20 > already very simple, I've tried to optimize probably the only thing=20 > possible there: switch bswlist from TAILQ to SLIST. As I can see,=20 > b_freelist field of struct buf is really used as TAILQ in some other=20 > places, so I've just added another SLIST_ENTRY field. And result=20 > appeared to be surprising -- I can no longer reproduce the issue at all.= =20 > May be it was just unlucky synchronization of specific test, but I've=20 > seen in on two different systems and rechecked results with/without=20 > patch three times. This is too unbelievable. Could it be, e.g. some cache line conflicts which cause the trashing, in fact ? Does it help if you add void *b_pad before b_freelist instead of adding b_freeslist ? >=20 > The present patch is here: > http://people.freebsd.org/~mav/buf_slist.patch >=20 > The question is how to do it better? What is the KPI/KBI policy for=20 > struct buf? I could replace b_freelist by a union and keep KBI, but=20 > partially break KPI. Or I could add another field, probably breaking=20 > KBI, but keeping KPI. Or I could do something handmade with no breakage.= =20 > Or this change is just a bad idea? The same question about using union for b_freelist/b_freeslist, does the effect of magically fixing the contention still there if b_freeslist is on the same offset as the b_freelist ? There are no K{B,P}I policy for struct buf in HEAD, just change it as it fits. --/UjIn1MmbqnRGnkw Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzTPcAAoJEJDCuSvBvK1BG4cP/3xuV0DRg/9kEyhaSXC1LNbf iSXvD/sWYTA/ttKSht5iBQaKDUR0zOFY/2Ik3NKdFMhFTGh/Wsp7+pODB9ZNd3Oe 9SRjAkzImOkwD+vJQBGsQ+Oll4Nt607gnFbQptkGZdEmArl3uqqpAglbEF8fgSws ew2yt6xLfjPv3OpPjI4YuWH40+VxJH1rEH+h2ne0Fm63EJgV+IOiNSB8JM1LWgCJ jT5TM72n5iAZPM3FlVr/oSnKjJVa4UpIfQtU9AuqDhsnGZl1ZXdgOw+eU+Er3cTq GdpzVIY/dg6qMIr20Y+BnLUKt136YNEZHXXbj1Qt32ndikNAo5VGXuCR2WeIS/Qt PdL7YfQcp3zCX/wxtkM1BFHTU2zZwAwMgL9OZkLsWLtV4H6H0/lpiwMihO3g5KNG gqxBpvuGdx1hkz3jqvHpQRMisv/HhdxokOqiZUOLoyNyOm8KMCMXIGyzbtHVIk2E P/4Ls1DhP95DQuG6Ejbj/K39owqqNt7OTcpkiwBwqVbIV0yfgVb/tQRnoO1meLJr +WU5TXRjA8WZsaarhVVPRk0oB5vBITTQott7+s6amC/JLJ6Yk6ec0kdP5iMuoSXM s6/kSrPkUvriNJ8obZxtgR+wx30oy8fWXuigvbGSknG+PICV4oreTNMa5Om2666T 8Y3Q93ljI6fDWsJqoW/1 =B8dd -----END PGP SIGNATURE----- --/UjIn1MmbqnRGnkw-- From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 08:41:37 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8729E8DE; Fri, 28 Jun 2013 08:41:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id D4FBE1BF1; Fri, 28 Jun 2013 08:41:36 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r5S8fWnr062050; Fri, 28 Jun 2013 11:41:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r5S8fWnr062050 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r5S8fW6g062049; Fri, 28 Jun 2013 11:41:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 28 Jun 2013 11:41:32 +0300 From: Konstantin Belousov To: Chris Torek Subject: Re: expanding amd64 past the 1TB limit Message-ID: <20130628084132.GM91021@kib.kiev.ua> References: <51CBED22.6090702@freebsd.org> <201306272123.r5RLNapo064878@elf.torek.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2gpokFnphsYkgNXa" Content-Disposition: inline In-Reply-To: <201306272123.r5RLNapo064878@elf.torek.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 08:41:37 -0000 --2gpokFnphsYkgNXa Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jun 27, 2013 at 03:23:36PM -0600, Chris Torek wrote: > OK, I wasted :-) way too much time, but here's a text file that > can be comment-ified or stored somewhere alongside the code or > whatever... I think it would be a nice addition to the VM article in the doc collection. The content is correct and useful, IMO. See some notes below. > The reason for having those "empty" (all-zero) PTE pages is that > whenever new processes are created, in pmap_pinit(), they have > their (new) L4 PTE page set up to point to the *same* physical > pages that the kernel is using. Thus, if the kernel creates or > destroys any level-3-or-below mapping by writing into any of the > above four pages, that mapping is also created/destroyed in all > processes. Similarly, the NDMPML4 pages starting at DMPDPphys are > mapped identically in all processes. The kernel can therefore > "borrow" a user pmap at any time, i.e., there's no need to adjust > the CPU's CR4 on entry to the kernel. It is %cr3. >=20 > (If we used bcopy() to copy the kernel pmap's NKPML4E and NDMPML4E > entries into the new pmap, the L3 pages would not have to be > physically contiguous, but the KVA ones would still all have to > exist. It's free to allocate physically contiguous pages here > anyway though.) I do not see how the physical continuity of the allocated page table pages is relevant there. Copying the L4 or L3 PTEs would cause serious complications. In fact, this is how i386 operates, since due to limited VA, pmap does not have a luxurly of allocating entire L(top-1) page table pages to the kernel. As result, pmap_kenter_pde() have to iterate over the whole pmap list, which is maintained just to be able to update KVA mappings. >=20 > So, the last NKPML4E slots in KPML4phys point to the following > page tables, which use all of L3, L2, and L1 style PTEs. (Note > that we did not need any L1 PTEs for the direct map, which always > uses 2MB or 1GB super-pages.) This is not quite true. In the initial state, indeed all PTEs for direct map are superpages, either 1G or 2M. But Intel states that a situation when the physical page has mappings with different caching modes causes undefined behaviour. As result, if a page is remapped with non-write back caching attributes, the direct map has to demote the superpage and adjust the mapping attribute of the page frame for the page. As result, it is possible to have small mappings in the direct map. See pmap_page_set_memattr(), which delegates the work to pmap_change_attr(_locked). --2gpokFnphsYkgNXa Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzUw7AAoJEJDCuSvBvK1B/dgP/iEyZew3Kh9QRcyNjHQe/XE+ u9dHpKoCLNsAFXJJqTLXZcaNiSQCsABjC8fgyaA4ruUZw/3xUyffAWML0MMAfJe5 GKsQpV6cxjO9RHh4xrXi62lf87/G7If7ltcKnmwEfLzqBmPt77Wu2qiV2RIF32mK 8QyggtfEX9RIU1hIfsOmvuRiwyp9Z8GOvSrXEPI6ngzTWbSfxulHNzkzi+9Hd1qj hmG5emune641WQ/zcwka0gNWlivyOskioeujRdVszPM9EawZEVcwOTcrNiV7SbuF B6+q+G4LNgbh4kmAb2R2N3Z4diS9K/V1r13M+QCK0POgZ5/wu7vv+uPjVYKT3G3T 2PFnq8d8WfYSEfxcVHDdyGdPxoYXlBdRqO5vAtDhjYlfteC6LqOeIZIrwSksH5cg 9LFC/qmvwRUygkLk2J3qe+f7ZgqnN1956dj604HlLl97EjA62e5kd251wv+MP8vf Bpy90XGHleimkkI3wn+ikZAzZMxQMEgxKxUVBNcjYwXdf2pGYYQ6F8DNYxACqAlq HvYLDucrY0NJOq3olc22Y23SNlOu3I/Ly0YFrs/BEzhMOzjUxLHSk4u4tYsdrgRj CBDH5GUEVVvLOoNoT1w+q8xeEqBjt7RJoo1QwVQt0hJTG61xq4eOMTKsmJ7KPh5E Rwoe17b0pKrPW6tbtmsW =xV7w -----END PGP SIGNATURE----- --2gpokFnphsYkgNXa-- From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 08:57:19 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0386BADB for ; Fri, 28 Jun 2013 08:57:19 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-x22b.google.com (mail-ee0-x22b.google.com [IPv6:2a00:1450:4013:c00::22b]) by mx1.freebsd.org (Postfix) with ESMTP id 906F11C6B for ; Fri, 28 Jun 2013 08:57:18 +0000 (UTC) Received: by mail-ee0-f43.google.com with SMTP id l10so883056eei.16 for ; Fri, 28 Jun 2013 01:57:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=NOlaBcYu9p0Qfa5ad1C3VqsT7kQZ3ybtiMGjYYyBsG8=; b=ucvA3OEZxBpGlUh3rYA1+Msa3RtdGtTo6N2JjB+1pK2aRBynNl5XvFH0ZXQnxSvHL+ h1fvjyHlahwpLCVfc7fa0N4en774nU8FBqfYF0CAFsGa30iBM15xXsuyrVbef5H4sHgC 1BVLSiRcMoVd4daIjbYrvqc/PMiga+91pSZx5rEuuzDV7m+gwEwLEUALnjOsM2A6Ttce QpDsm/jSkQc3S4m0GdA+2ObhIPafJokrCBWa9il1RuECg2dQSNcxNUV/H9CjLO8K/s1s 18XVlUh3R8bpcMX7/p9O1CEA0InhXj3l7c+Cd50LHPI0e/N+RtWGlrqCsPzqDf1JzITA diMw== X-Received: by 10.14.209.197 with SMTP id s45mr12844172eeo.108.1372409837688; Fri, 28 Jun 2013 01:57:17 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPSA id l42sm9114329eeo.14.2013.06.28.01.57.15 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 28 Jun 2013 01:57:16 -0700 (PDT) Sender: Alexander Motin Message-ID: <51CD4FEA.7030605@FreeBSD.org> Date: Fri, 28 Jun 2013 11:57:14 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: b_freelist TAILQ/SLIST References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> In-Reply-To: <20130628065732.GL91021@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 08:57:19 -0000 On 28.06.2013 09:57, Konstantin Belousov wrote: > On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: >> While doing some profiles of GEOM/CAM IOPS scalability, on some test >> patterns I've noticed serious congestion with spinning on global >> pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is >> already very simple, I've tried to optimize probably the only thing >> possible there: switch bswlist from TAILQ to SLIST. As I can see, >> b_freelist field of struct buf is really used as TAILQ in some other >> places, so I've just added another SLIST_ENTRY field. And result >> appeared to be surprising -- I can no longer reproduce the issue at all. >> May be it was just unlucky synchronization of specific test, but I've >> seen in on two different systems and rechecked results with/without >> patch three times. > This is too unbelievable. I understand that it looks like a magic. I was very surprised to see contention there at all, but `pmcstat -n 10000000 -TS unhalted-cycles` shows it too often and repeatable: PMC: [CPU_CLK_UNHALTED_CORE] Samples: 28052 (100.0%) , 12 unresolved %SAMP IMAGE FUNCTION CALLERS 46.4 kernel __mtx_lock_sleep relpbuf:22.3 getpbuf:22.0 xpt_run_devq:0.8 13.3 kernel _mtx_lock_spin_cooki turnstile_trywait 4.3 kernel cpu_search_lowest cpu_search_lowest 2.3 kernel getpbuf physio , and benchmark results confirm it. > Could it be, e.g. some cache line conflicts > which cause the trashing, in fact ? Does it help if you add void *b_pad > before b_freelist instead of adding b_freeslist ? No, this doesn't help. And previously I've tested it also with b_freeslist in place but without other changes -- it didn't help either. >> The present patch is here: >> http://people.freebsd.org/~mav/buf_slist.patch >> >> The question is how to do it better? What is the KPI/KBI policy for >> struct buf? I could replace b_freelist by a union and keep KBI, but >> partially break KPI. Or I could add another field, probably breaking >> KBI, but keeping KPI. Or I could do something handmade with no breakage. >> Or this change is just a bad idea? > The same question about using union for b_freelist/b_freeslist, does the > effect of magically fixing the contention still there if b_freeslist > is on the same offset as the b_freelist ? Yes, it is. > There are no K{B,P}I policy for struct buf in HEAD, just change it as > it fits. Which one would you prefer, the original or http://people.freebsd.org/~mav/buf_slist2.patch ? Thank you. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 13:04:59 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 57224BBF for ; Fri, 28 Jun 2013 13:04:59 +0000 (UTC) (envelope-from david.i.noel@gmail.com) Received: from mail-pb0-x234.google.com (mail-pb0-x234.google.com [IPv6:2607:f8b0:400e:c01::234]) by mx1.freebsd.org (Postfix) with ESMTP id 383561897 for ; Fri, 28 Jun 2013 13:04:59 +0000 (UTC) Received: by mail-pb0-f52.google.com with SMTP id xa12so2237152pbc.11 for ; Fri, 28 Jun 2013 06:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=y0kMIwktXZpKt7vLTZfIJejrH5QiOVHd857ch8U2TIE=; b=sPMiN9Hdw2RUFmzcbqMj+jixvJppdRBYpVK7/T0SD8BNl/6teqo02wdZ+Yto97AKZV QTVQDGW9R7bPqdWBP5GPp6mMiEZmzxsn5N61FLuVhAfLHE2ngjaE9+Kx9z0/NqIC6bQE l1AgZk34HcpsmT7bwh5EPbTwpdSRZqvPBjHJG7FozqngDAqf4FWaqQKoAIDfDYVOGuLC 2DLBV0bnqIYW4YVrj43lZGGv4+r8FIpuptKuHhG4GSixrRkRiICuGSR2NEXb3fepx5gg 906v5usIgasE2BOGey2PwyZIOYb/7NrGBaft3WweEHh7B+F/miqEOf7Fl4vh6tGXCfQ5 BL3Q== MIME-Version: 1.0 X-Received: by 10.68.254.74 with SMTP id ag10mr11360647pbd.81.1372424698856; Fri, 28 Jun 2013 06:04:58 -0700 (PDT) Received: by 10.68.201.201 with HTTP; Fri, 28 Jun 2013 06:04:57 -0700 (PDT) In-Reply-To: References: Date: Fri, 28 Jun 2013 08:04:57 -0500 Message-ID: Subject: Re: bsd boot sequence From: David Noel To: Brian Kim Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: David.I.Noel@gmail.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 13:04:59 -0000 On 6/27/13, Brian Kim wrote: > howdy all, > > As a junior computer engineering major who has dreams of developing an > operating system more ubiquitous than ms windows, I have come to appreciate > the complexity and elegance of both the freebsd os and community. While > achieving proficiency in C programming yet lacking in both UNIX utility > know-how and shell scripting, I would rate my hacker savvy to be on an > intermediate level. By the time I graduate from college, I hope to have a > thorough understanding of the bsd kernel on the lowest level. > > Don't get me wrong though: while my end-game may be to develop my own > operating system, I will forever be a contributor to freebsd. As my > understanding of computers develops, you will undoubtedly see my name > appear more frequently on the freebsd-hacker emails. Unfortunately, I am > just not at the level of understanding just yet. > > To start, I was wondering if someone could briefly explain the operations > and function calls that occur at boot time. I wish to thoroughly examine > the freebsd source code but I'm afraid the sheer volume of code that exists > leaves me with no real starting point for my research. > > Thank you all for your time. > > -- > Best Wishes, > Brian Kim Boot's probably not as bad as it seems. There's boot0, boot1, boot2, loader, and kernel. So those would probably be the best places to start digging. A few hundred k of code if I'm not mistaken. Not entirely unmanageable. Some ideas that might help: - Scan source for comments and function/method names - Use doxygen to generate high-level source documentation - Generate UML diagrams from source (class and sequence diagrams would probably be most useful) - As Warren suggested, read all the documentation you can find - Google - Take notes as you go, publish them on a blog, and share the URL with us! -David From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 14:02:09 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 25AD9E29 for ; Fri, 28 Jun 2013 14:02:09 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward19.mail.yandex.net (forward19.mail.yandex.net [IPv6:2a02:6b8:0:1402::4]) by mx1.freebsd.org (Postfix) with ESMTP id CDA751E89 for ; Fri, 28 Jun 2013 14:02:08 +0000 (UTC) Received: from smtp16.mail.yandex.net (smtp16.mail.yandex.net [95.108.252.16]) by forward19.mail.yandex.net (Yandex) with ESMTP id 042C41120E5A; Fri, 28 Jun 2013 18:02:05 +0400 (MSK) Received: from smtp16.mail.yandex.net (localhost [127.0.0.1]) by smtp16.mail.yandex.net (Yandex) with ESMTP id C4BC46A05F0; Fri, 28 Jun 2013 18:02:05 +0400 (MSK) Received: from v10-166-195.yandex.net (v10-166-195.yandex.net [84.201.166.195]) by smtp16.mail.yandex.net (nwsmtp/Yandex) with ESMTP id MCVsKUk1tt-25rKmKBg; Fri, 28 Jun 2013 18:02:05 +0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1372428125; bh=Di0oq73weh/hmdkmNGx/Enl9EofVAJ5gf4z19WYUsww=; h=Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject: References:In-Reply-To:X-Enigmail-Version:Content-Type: Content-Transfer-Encoding; b=S6Pr39BDZIRoNY5ITfXdXQbLf4grwy+0UnTM1Q6A7zMmslgbMuTs4eYc5owqcQq0q dvnVSGrvFzDGJPL7GzofPiVZdOQVOySIZTijp6gN6mI8DvhpzMfRX2qzC/GMFYs7ge yI1Yb7fhvffm6tRFaDVYB7aqYurr4kjkf2coQ2Sc= Authentication-Results: smtp16.mail.yandex.net; dkim=pass header.i=@yandex.ru Message-ID: <51CD96B0.7000402@yandex.ru> Date: Fri, 28 Jun 2013 17:59:12 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Sreenivasa Honnur Subject: Re: IPv6 - sobind fails with 49 References: In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 14:02:09 -0000 On 28.06.2013 08:15, Sreenivasa Honnur wrote: > if (rv != 0) { > > printf("sock create ipv6 %s failed %d.\n", > > tbuf, rv); > > return NULL; > > } > > Can you try insert here? bzero(&saddr6, sizeof(saddr6)); > saddr6.sin6_family = AF_INET6; -- WBR, Andrey V. Elsukov From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 15:14:43 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 768474BE; Fri, 28 Jun 2013 15:14:43 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x234.google.com (mail-qc0-x234.google.com [IPv6:2607:f8b0:400d:c01::234]) by mx1.freebsd.org (Postfix) with ESMTP id 2D0A01210; Fri, 28 Jun 2013 15:14:43 +0000 (UTC) Received: by mail-qc0-f180.google.com with SMTP id a1so1433179qcx.11 for ; Fri, 28 Jun 2013 08:14:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=0Ja9SSYkQ5OVS0IeF0GxlqvGQs1+THDZaHw3fQODPgU=; b=FfadSADTw10oi8lt14BLf/OAMgX5UTOrb7OdLRxUqVc733YRsuz7EFkTnXAeq5BTuz CSM62MOh4FTXShMj5EOb3UJcTSklgWF7Qim/Ee+5gYY4rIhkKdY3APvYOTjS8VcSBSlh oUKJueuUW5s2HpGSaZDSNaSzjGJe5+hrRe9g+/Gq1BhQkM7yovoKBzgRYajmKBNDFXpH zyFGBMc66q3uJFjH+4oag4mtonYIMBRxd2hDA15Yb6IPq37DlzNzWf7FjPjzvlhtw9Ex 61ip9lf/NLY3WSZ4z5j5WuPIKiFfcoE7gQZk/A3sg4QZm2tEmz+3PUG4jB+3t5c7Q/fk hJVw== MIME-Version: 1.0 X-Received: by 10.49.116.9 with SMTP id js9mr17949963qeb.73.1372432482694; Fri, 28 Jun 2013 08:14:42 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.214.7 with HTTP; Fri, 28 Jun 2013 08:14:42 -0700 (PDT) In-Reply-To: <51CD4FEA.7030605@FreeBSD.org> References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CD4FEA.7030605@FreeBSD.org> Date: Fri, 28 Jun 2013 08:14:42 -0700 X-Google-Sender-Auth: w_WpsgOXBBzicMpBDLpkOZt_BPY Message-ID: Subject: Re: b_freelist TAILQ/SLIST From: Adrian Chadd To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 Cc: Konstantin Belousov , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 15:14:43 -0000 .. i'd rather you narrow down _why_ it's performing better before committing it. Otherwise it may just creep up again after someone does another change in an unrelated part of the kernel. You're using instructions-retired; how about using l1/l2 cache loads, stores, etc? There's a lot more CPU counters available. You have a very cool problem to solve. If I could reproduce it locally I'd give you a hand. Thanks, -adrian From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 15:37:13 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 61B47C08; Fri, 28 Jun 2013 15:37:13 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-x22a.google.com (mail-ee0-x22a.google.com [IPv6:2a00:1450:4013:c00::22a]) by mx1.freebsd.org (Postfix) with ESMTP id C4DB81306; Fri, 28 Jun 2013 15:37:12 +0000 (UTC) Received: by mail-ee0-f42.google.com with SMTP id c4so1101025eek.15 for ; Fri, 28 Jun 2013 08:37:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=n5sRHnBvVfx2gdJKoqkieMouKy2V1G5gGz5zcJyPnA0=; b=coEnr0d5Y5ZWT3UVPO41cy9G0r1ICJH4WXgNuwuW+KVZr64n1/VA7AhghrvkhBX+ga SCygtahgQsFu18SpEBc5YE/SjQAV0wxPBt6G7CiXGDrOtzz2yUcaDwAudiJFGe+0U9b5 tJgeVz6yzFbnwah825O37n1wc9nZa4V+g3F2mW7+YtTlrJbFaeTuzyWOD22fUiGuhkLW J/n8oTcqQGhJ4SXbu7OhQkyd7oPr2RbxXYD6d+eGKyC40dcZs4dbWLZA0Mq4EWSf/lTe YaZ9KLInCNcwcwhaDcfJS+lyigIHq1VVookQodfK0gHt84ZohJJ6ev3VF6NMDUuIV+NG KXmg== X-Received: by 10.15.43.11 with SMTP id w11mr14414589eev.27.1372433831793; Fri, 28 Jun 2013 08:37:11 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPSA id i2sm11226855eeu.4.2013.06.28.08.37.09 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 28 Jun 2013 08:37:10 -0700 (PDT) Sender: Alexander Motin Message-ID: <51CDADA4.9090803@FreeBSD.org> Date: Fri, 28 Jun 2013 18:37:08 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: b_freelist TAILQ/SLIST References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CD4FEA.7030605@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 15:37:13 -0000 On 28.06.2013 18:14, Adrian Chadd wrote: > .. i'd rather you narrow down _why_ it's performing better before committing it. If you have good guesses -- they are welcome. All those functions are so small, that it is hard to imagine how congestion may happen there at all. I have strong feeling that lock spinning there consumes incomparably much more CPU time then the locked region itself could consume. > Otherwise it may just creep up again after someone does another change > in an unrelated part of the kernel. Big win or small, TAILQ is still heavier then STAILQ, while it is not needed there at all. > You're using instructions-retired; how about using l1/l2 cache loads, > stores, etc? There's a lot more CPU counters available. I am using unhalted-cycles, that is more reasonable then instructions-retired. What's about other counters, there are indeed a lot of them, but it is not always easy to get something useful out of them. > You have a very cool problem to solve. If I could reproduce it locally > I'd give you a hand. You'd need a lot of hardware and patches to reproduce it in full. But if you like to see this with your own eyes, I can give you an SSH access to my test machine. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 15:56:54 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AD5A41FF; Fri, 28 Jun 2013 15:56:54 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x230.google.com (mail-qc0-x230.google.com [IPv6:2607:f8b0:400d:c01::230]) by mx1.freebsd.org (Postfix) with ESMTP id 61DD315DC; Fri, 28 Jun 2013 15:56:54 +0000 (UTC) Received: by mail-qc0-f176.google.com with SMTP id z10so1462599qcx.35 for ; Fri, 28 Jun 2013 08:56:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=Hc+Etr5FEvgF5ej2ZHPPvnYAtq5piHO2rRzLoEGSF8s=; b=IDmxnr+bhWPm+T2V+7ZNzGsZiiPQHxq5+DpeEvBRKsKbC7NdUWKVrwcxvx1GNB6fEw c+iycJMDN0JMoXK+X6jX1uP5bAn+20dFp5vQ8rMdD5Uzz/Y8c6sGeoYZLjGHPSVlrE3G dKOvqK8vn4BUu4vccWOIaHDFheelysUwMhGqncm44LPHTalAX7+VM6GJSP9P5Uzp+I1O PTBEzNQuCTOaFxk+OSr1hfek0yEbZ2ggoVwH7nd5mrN4kU4R9Ss1jxs7QRXkOM73SnNx o3BWwrxRu+3Mk9JHKD+ZU4agNbHvvlvgtXwOXpMLUrIeAm/XOS2ytUi8dESCK9DgQtwl 13aQ== MIME-Version: 1.0 X-Received: by 10.224.190.137 with SMTP id di9mr18263598qab.95.1372435013976; Fri, 28 Jun 2013 08:56:53 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.214.7 with HTTP; Fri, 28 Jun 2013 08:56:53 -0700 (PDT) In-Reply-To: <51CDADA4.9090803@FreeBSD.org> References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CD4FEA.7030605@FreeBSD.org> <51CDADA4.9090803@FreeBSD.org> Date: Fri, 28 Jun 2013 08:56:53 -0700 X-Google-Sender-Auth: 4KL2v_vKoRFJuSc_lJGl-jC-SYE Message-ID: Subject: Re: b_freelist TAILQ/SLIST From: Adrian Chadd To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 Cc: Konstantin Belousov , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 15:56:54 -0000 On 28 June 2013 08:37, Alexander Motin wrote: >> Otherwise it may just creep up again after someone does another change >> in an unrelated part of the kernel. > > Big win or small, TAILQ is still heavier then STAILQ, while it is not needed > there at all. You can't make that assumption. I bet that if both pointers are in the _same_ cache line, the overhead of maintaining a double linked list is trivial. You've already bitten the overhead of loading the struct from RAM into L1/L2 cache; once that's done, the second pointer operation should just hit the cache. Same with the store - if they're in the same cache line, the overhead will be very small. The memory controller is likely operating on multiples of L1 cache lines anyway. The only time I can see that being a benefit (on current hardware) is if you're trying to pack more into cache lines; but then if you're doing that, you're likely better off with an array rather than lists. That way you don't waste 4/8 (or 2x that for the double-linked list) bytes for each pointer for each list element, as well as another 4/8 byte pointer to 'self'. But that doesn't -always- give a boost; it also depends upon memory layout and whether you're using all the cache lines or not. So yes - I think you're on to something and I think it's worth digging deeper into what's going on before you commit something. No, don't commit the SLIST change until you absolutely, positively know why it's actually being helpful. It's way way too "voodoo" right now. Thanks, Adrian From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 15:58:02 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2ABFE305; Fri, 28 Jun 2013 15:58:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 951D315F8; Fri, 28 Jun 2013 15:58:01 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r5SFvwb3054500; Fri, 28 Jun 2013 18:57:58 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r5SFvwb3054500 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r5SFvwcj054499; Fri, 28 Jun 2013 18:57:58 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 28 Jun 2013 18:57:58 +0300 From: Konstantin Belousov To: Adrian Chadd Subject: Re: b_freelist TAILQ/SLIST Message-ID: <20130628155757.GS91021@kib.kiev.ua> References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CD4FEA.7030605@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="C6XIXe15e7wzRDAo" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Alexander Motin , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 15:58:02 -0000 --C6XIXe15e7wzRDAo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 28, 2013 at 08:14:42AM -0700, Adrian Chadd wrote: > .. i'd rather you narrow down _why_ it's performing better before committ= ing it. >=20 > Otherwise it may just creep up again after someone does another change > in an unrelated part of the kernel. Or penalize some other set of machines where this is currently not a proble= m. The cause should be identified before any change is committed. >=20 > You're using instructions-retired; how about using l1/l2 cache loads, > stores, etc? There's a lot more CPU counters available. >=20 > You have a very cool problem to solve. If I could reproduce it locally > I'd give you a hand. >=20 > Thanks, >=20 >=20 >=20 > -adrian --C6XIXe15e7wzRDAo Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzbKFAAoJEJDCuSvBvK1BsVIP/i0Q4nw6m5WKkM08yyJnYbTb XHp1e4d5CMwiTZUkkt6gfffJoDTY6q1oyBjzgVja3hdg1WK4dCUk5pUriQMiefWp ZyVvSVeIDRzhLX5cHbUFpsgFfYkNYaMBNcL+zxbetQciVDN7D/yosKKWoRGaUNiI Baio6XTieZnTaIwZItOI0m3Yjci3IrEJMmZ/5IkofCo20VlVFKLSd/ZHMkQRH+wq XQQra04zGkzg4DQA8ay1ErVOx5x0+tPvPGB8ecFLfGckfIzdSUxjiCXPob7hYqhF zvoUmSmAtkcn7hJ5UWYI/Q2va3dQA2qCdiayeqWiR/o79u1+WJGJrWFpLDTfUGz5 uyhIeNv7z+aZ8ld/OP4JJ/GHNjdmtDfNsQYKOiwoF4wz1+Cz/4yKt0F7EDb2NJIU 5r3HKFLe6QDp6it0u9frVDKmXRZkV3pl3d7Pj/x4LiAe7Xp2O6f1h38AeROL5NNl yENrnT+mwYuBB9cNveAbXn/nLL9pErneoXGJ4aa1iJpItfHRJuDyIaqvWr4q22Jj 8Exhb9r/IOLUs0+niG2XL1q5xhCaACaYkIqO7Y0B/VRKv/f69AOyldBtDa+bdRV+ fe2AgUbZKl3WATDIT4yPN5Hd7j1UXk21iSV4mYTEl/UdO9G9p9ITCfqzaycJhrJd N/q5UDbkjRW/wFuv9T9s =NHjw -----END PGP SIGNATURE----- --C6XIXe15e7wzRDAo-- From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 16:18:57 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CA028CF2; Fri, 28 Jun 2013 16:18:57 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-ob0-x22f.google.com (mail-ob0-x22f.google.com [IPv6:2607:f8b0:4003:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 7DA7B1728; Fri, 28 Jun 2013 16:18:57 +0000 (UTC) Received: by mail-ob0-f175.google.com with SMTP id xn12so2168127obc.34 for ; Fri, 28 Jun 2013 09:18:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=cPyPkZdDV2vG2U/bW5gQgBrrXmc+rqSfio5xOgJttBY=; b=hTbNz/ycOki8A47ttFe8uiTOcwozed7zXMPgmoJ//1kVz+BktyZ2sSZIIdfv9Sny0V uHQqsG90+sxePAVlQRK5ldWuiHocqa8xvfTWFsb0x0sbKLktUqzy/Q1V+rck8tQ111oV QTktnIIwm3EXEmCIeryAtreSlaknZ6FjSnEuEN0X2PMEw0TO72T6vytL0fySQNu1CYQH q2RSqM6zsCJ3sT+Fp2ARxbXwAm9Vd3G9jxIll4vKla3/4oTKW7leVpnG326ORL11+prH gslo9HkXKZE8EfMRewwWIzc54jtJEvCbxS5WxxRxDmxxxjsyIzYaJCI5FmF036fK7sFY 8bkw== MIME-Version: 1.0 X-Received: by 10.60.164.33 with SMTP id yn1mr5424742oeb.5.1372436337120; Fri, 28 Jun 2013 09:18:57 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.182.165.162 with HTTP; Fri, 28 Jun 2013 09:18:57 -0700 (PDT) In-Reply-To: References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CD4FEA.7030605@FreeBSD.org> <51CDADA4.9090803@FreeBSD.org> Date: Fri, 28 Jun 2013 09:18:57 -0700 X-Google-Sender-Auth: A593obBiIXHan0gXkMIY9zsv7Tg Message-ID: Subject: Re: b_freelist TAILQ/SLIST From: mdf@FreeBSD.org To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Konstantin Belousov , Alexander Motin , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 16:18:57 -0000 On Fri, Jun 28, 2013 at 8:56 AM, Adrian Chadd wrote: > On 28 June 2013 08:37, Alexander Motin wrote: > >> Otherwise it may just creep up again after someone does another change > >> in an unrelated part of the kernel. > > > > Big win or small, TAILQ is still heavier then STAILQ, while it is not > needed > > there at all. > > You can't make that assumption. I bet that if both pointers are in the > _same_ cache line, the overhead of maintaining a double linked list is > trivial. No, it's not. A singly-linked SLIST only needs to modify the head of the list and the current element. A doubly-linked LIST needs to modify both the head as well as the old first element, which may not be in cache (and may not be in the same TLB, either). I don't recall exactly what [S]TAILQ touches, but the doubly-linked version still has to modify more entries that are not contiguous. Thanks, matthew From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 16:22:20 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8579FDFD; Fri, 28 Jun 2013 16:22:20 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ea0-x22a.google.com (mail-ea0-x22a.google.com [IPv6:2a00:1450:4013:c01::22a]) by mx1.freebsd.org (Postfix) with ESMTP id E522E1742; Fri, 28 Jun 2013 16:22:19 +0000 (UTC) Received: by mail-ea0-f170.google.com with SMTP id h10so1135381eaj.15 for ; Fri, 28 Jun 2013 09:22:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=pmKcIQ0xPrIwS8xiXcW1Dd+q87TAPM3ub11x7eM1iY4=; b=F+hDcONw/Z06/vniRJi3pP6j2adTyQWMagTOlzpcJPKVlYa2ugg+61D5iaSPonK3p2 iixvvQpWEcSu1jGeHKY1mketxAuH0QI+qCFXZ8D2Ijfk/QiT+1jqXFYS69rq677t2w86 ZcRazb7ogYCgzXVJNQ/6OQ/U0nGYfsdb3OM4vxpKLoeiRaawB4Re/OFLrdula0cmyvVC 8qr8OKjOJz+CczW6WpWduB1wUU/8PKqody0lvZzMuGQZemItxtD7M2NH09r10ejlu1Nw eH97Q0/YtUYQS57d8TljynLSo5aLcY7kKJF8wdXFVuaCPv5winDS3aTZL+pQBMGrtkE4 7rkg== X-Received: by 10.15.110.10 with SMTP id cg10mr14610707eeb.57.1372436539109; Fri, 28 Jun 2013 09:22:19 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPSA id m1sm11470382eex.17.2013.06.28.09.22.17 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 28 Jun 2013 09:22:18 -0700 (PDT) Sender: Alexander Motin Message-ID: <51CDB837.7050908@FreeBSD.org> Date: Fri, 28 Jun 2013 19:22:15 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: b_freelist TAILQ/SLIST References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CD4FEA.7030605@FreeBSD.org> <51CDADA4.9090803@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 16:22:20 -0000 On 28.06.2013 18:56, Adrian Chadd wrote: > On 28 June 2013 08:37, Alexander Motin wrote: >>> Otherwise it may just creep up again after someone does another change >>> in an unrelated part of the kernel. >> >> Big win or small, TAILQ is still heavier then STAILQ, while it is not needed >> there at all. > > You can't make that assumption. I bet that if both pointers are in the > _same_ cache line, the overhead of maintaining a double linked list is > trivial. You've already bitten the overhead of loading the struct from > RAM into L1/L2 cache; once that's done, the second pointer operation > should just hit the cache. > > Same with the store - if they're in the same cache line, the overhead > will be very small. The memory controller is likely operating on > multiples of L1 cache lines anyway. > > The only time I can see that being a benefit (on current hardware) is > if you're trying to pack more into cache lines; but then if you're > doing that, you're likely better off with an array rather than lists. > That way you don't waste 4/8 (or 2x that for the double-linked list) > bytes for each pointer for each list element, as well as another 4/8 > byte pointer to 'self'. But that doesn't -always- give a boost; it > also depends upon memory layout and whether you're using all the cache > lines or not. That is true, but still TAILQ_INSERT/_REMOVE modify one more cache line then SLIST and also use branching. > So yes - I think you're on to something and I think it's worth digging > deeper into what's going on before you commit something. No, don't > commit the SLIST change until you absolutely, positively know why it's > actually being helpful. It's way way too "voodoo" right now. OK, I'll dig it more. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 18:16:25 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 31E7381E for ; Fri, 28 Jun 2013 18:16:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 9E0731D06 for ; Fri, 28 Jun 2013 18:16:24 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r5SIGI5g083426; Fri, 28 Jun 2013 21:16:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r5SIGI5g083426 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r5SIGIrP083425; Fri, 28 Jun 2013 21:16:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 28 Jun 2013 21:16:18 +0300 From: Konstantin Belousov To: Chris Torek Subject: Re: expanding amd64 past the 1TB limit Message-ID: <20130628181618.GV91021@kib.kiev.ua> References: <201306261611.r5QGBie4025519@elf.torek.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4DDnoy+zDPJBVbdO" Content-Disposition: inline In-Reply-To: <201306261611.r5QGBie4025519@elf.torek.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 18:16:25 -0000 --4DDnoy+zDPJBVbdO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jun 26, 2013 at 10:11:44AM -0600, Chris Torek wrote: > diff --git a/conf/options.amd64 b/conf/options.amd64 > index 90348b7..f3ce505 100644 > --- a/conf/options.amd64 > +++ b/conf/options.amd64 > @@ -1,6 +1,7 @@ > # $FreeBSD$ > # Options specific to AMD64 platform kernels > =20 > +AMD64_HUGE opt_global.h Is this option needed ? The SMAP is already parsed at the time of pmap_bootstrap() call, so you could determine the amount of physical memory and size the KVA map accordingly ? --4DDnoy+zDPJBVbdO Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzdLxAAoJEJDCuSvBvK1BsMQP/Rk23pZGOCI2UrTYcgVV9XO7 hZQt99IhJwjUVOqBXGd0zrH5//4DADt76wThATOJW8In2HwxB6x4O+0AoeZR0yxm YDbtUiI7AaFRIsCLUbOr5L7l974gFoxQ1mEP9iCJ0mhdMkG8rOELdMnIiY96kOkr X/sxsu80SYZCZD+cJXP/ud6q1G7xpjYRHCs6uWNvxtfYqz5aHXZxhcEMG3APPoNn 3HKmlk8JoRgGTi9I8xD6Q/1MpXqvIL/yh5u5URfYDRPePvJiCFRR4tzceIXubQ6p t2IOvi9UWHth+KfEli1s6+TcPgu8PgqpvG2ka0hveI4ENqHlMPZ2dJvZp1/jJLxn 1hh/ZYPLya2WoxzuIbh3Ufv5hvUxQxCKxhBwU0yTb3jSZ7/ox0SvTdOGEVElaHil MKhY4ErYnJb4ZzqPGopVDlCP+CgWthTbBGzrlfWSVolErdCrfpKcc+y4J1JLndbQ QyYbCYXtjpUab2U0nADNjVjC/tO9O/oJqwaSdw0aB5xtR90+z00drIn2oBiB1fvp 1nbet50aEkehRibJtY0UB+aKAOyKcXXYV3MENC+P3fbOniot0k1JyR4SynKZFBHc VQuWlMK+RsTYvNkas4pgIrkyVFy5aXsaBrxSsdbs+sTDJ05GaMfkT0+I6YncfrBw CXjLfVqLQlSAUaO5g99n =SgO7 -----END PGP SIGNATURE----- --4DDnoy+zDPJBVbdO-- From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 18:21:20 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4E153A67; Fri, 28 Jun 2013 18:21:20 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qa0-x22d.google.com (mail-qa0-x22d.google.com [IPv6:2607:f8b0:400d:c00::22d]) by mx1.freebsd.org (Postfix) with ESMTP id E497D1D49; Fri, 28 Jun 2013 18:21:19 +0000 (UTC) Received: by mail-qa0-f45.google.com with SMTP id ci6so786158qab.18 for ; Fri, 28 Jun 2013 11:21:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=X31Nut++pRjKfKr3U3vdop/p/U6c5dzTb0WwH/ehhfA=; b=lKnx57+WgZaDp4GanL6ROqI5taTRIuDdlWtXaIrmcOUZcjHgTzS56dJYktyMFy4fmH mhvZVYzneHyzGChVLtCnBdWUsvL6XGY18CfQc9x33FK8Yxhkozq++PDDctPVnMVeCmQl FajUy+5ehoiITfiXi1q8yU+Pch9a1ZhcFr5HjE8Ozjm35QQY6UF7eTM61KH55ckAGyzS kjYm+2f3zcQ3HxJcNgWywZUAryr7bLusqJp+77ozxHtpNPpMW1x8lHYOfc7Fw3ZD2hm5 DgQdYmDf2UASG1/3ESxKYWXF1R20MhwvcPMH0HM4ooIJ9huoJVc62oz+vPhRMreXaoI+ F3PA== MIME-Version: 1.0 X-Received: by 10.224.21.130 with SMTP id j2mr19606519qab.112.1372443679474; Fri, 28 Jun 2013 11:21:19 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.214.7 with HTTP; Fri, 28 Jun 2013 11:21:19 -0700 (PDT) In-Reply-To: References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CD4FEA.7030605@FreeBSD.org> <51CDADA4.9090803@FreeBSD.org> Date: Fri, 28 Jun 2013 11:21:19 -0700 X-Google-Sender-Auth: 0ZqxfMs7T42W4PSA8owViIWZx-0 Message-ID: Subject: Re: b_freelist TAILQ/SLIST From: Adrian Chadd To: mdf@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Konstantin Belousov , Alexander Motin , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 18:21:20 -0000 On 28 June 2013 09:18, wrote: >> You can't make that assumption. I bet that if both pointers are in the >> _same_ cache line, the overhead of maintaining a double linked list is >> trivial. > > > No, it's not. A singly-linked SLIST only needs to modify the head of the > list and the current element. A doubly-linked LIST needs to modify both the > head as well as the old first element, which may not be in cache (and may > not be in the same TLB, either). I don't recall exactly what [S]TAILQ > touches, but the doubly-linked version still has to modify more entries that > are not contiguous. Good point. -adrian From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 20:34:02 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D5E9AB6E; Fri, 28 Jun 2013 20:34:02 +0000 (UTC) (envelope-from torek@torek.net) Received: from elf.torek.net (50-73-42-1-utah.hfc.comcastbusiness.net [50.73.42.1]) by mx1.freebsd.org (Postfix) with ESMTP id AF49A1481; Fri, 28 Jun 2013 20:34:01 +0000 (UTC) Received: from elf.torek.net (localhost [127.0.0.1]) by elf.torek.net (8.14.5/8.14.5) with ESMTP id r5SKXtYK053022; Fri, 28 Jun 2013 14:33:55 -0600 (MDT) (envelope-from torek@torek.net) Message-Id: <201306282033.r5SKXtYK053022@elf.torek.net> From: Chris Torek To: Konstantin Belousov Subject: Re: expanding amd64 past the 1TB limit In-reply-to: Your message of "Fri, 28 Jun 2013 11:41:32 +0300." <20130628084132.GM91021@kib.kiev.ua> Date: Fri, 28 Jun 2013 14:33:55 -0600 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (elf.torek.net [127.0.0.1]); Fri, 28 Jun 2013 14:33:55 -0600 (MDT) Cc: alc@freebsd.org, kib@freebsd.org, freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 20:34:02 -0000 [combining two messages and adding kib and alc to cc per Oliver Pinter] >> the CPU's CR4 on entry to the kernel. >It is %cr3. Oops, well, easily fixed. :-) >> (If we used bcopy() to copy the kernel pmap's NKPML4E and NDMPML4E >> entries into the new pmap, the L3 pages would not have to be >> physically contiguous, but the KVA ones would still all have to >> exist. It's free to allocate physically contiguous pages here >> anyway though.) >I do not see how the physical continuity of the allocated page table >pages is relevant there. Not in create_pagetables(), no, but later in pmap_pinit(), which has loops to set pmap->pm_pml4[x] for kernel and and direct-map. And: >Copying the L4 or L3 PTEs would cause serious complications. Perhaps what I wrote was a little fuzzy. Here's the pmap_pinit() code I was referring to, as modified (the original version has only the second loop -- it assumes NKPML4E is always 1 so it just sets pml4[KPML4I]): pmap->pm_pml4 = (pml4_entry_t *)PHYS_TO_DMAP(VM_PAGE_TO_PHYS(pml4pg)); if ((pml4pg->flags & PG_ZERO) == 0) pagezero(pmap->pm_pml4); for (i = 0; i < NKPML4E; i++) { pmap->pm_pml4[KPML4BASE + i] = (KPDPphys + (i << PAGE_SHIFT)) | PG_RW | PG_V | PG_U; } for (i = 0; i < NDMPML4E; i++) { pmap->pm_pml4[DMPML4I + i] = (DMPDPphys + (i << PAGE_SHIFT)) | PG_RW | PG_V | PG_U; } /* install self-referential address mapping entry(s) */ These require that KPDPphys and DMPDPphys both point to the first of n physically-contiguous pages. But suppose we did this (this is deliberately simple for illustration, and furthermore I am assuming here that vmspace0 never acquires any user-level L4 entries): pmap->pm_pml4 = (pml4_entry_t *)PHYS_TO_DMAP(VM_PAGE_TO_PHYS(pml4pg)); /* Clear any junk and wire in kernel global address entries. */ bcopy(vmspace0.vm_pmap.pm_pml4, pmap->pm_pml4, NBPG); /* install self-referential address mapping entry(s) */ Now whatever we set up in create_pagetables() is simply copied to new (user) pmaps, so we could go totally wild if we wanted. :-) >> So, the last NKPML4E slots in KPML4phys point to the following >> page tables, which use all of L3, L2, and L1 style PTEs. (Note >> that we did not need any L1 PTEs for the direct map, which always >> uses 2MB or 1GB super-pages.) >This is not quite true. In the initial state, indeed all PTEs for direct >map are superpages, either 1G or 2M. But Intel states that a situation >when the physical page has mappings with different caching modes causes >undefined behaviour. As result, if a page is remapped with non-write >back caching attributes, the direct map has to demote the superpage and >adjust the mapping attribute of the page frame for the page. Yes, this particular bit of description was restricted to the setup work in create_pagetables(). (Perhaps I should take out "always", or substitute "initially"?) Also, I think I left out a description of the loop where some KPDphys entries are overwritten with 2MB mappings. >> +AMD64_HUGE opt_global.h >Is this option needed ? The SMAP is already parsed at the time of >pmap_bootstrap() call, so you could determine the amount of physical >memory and size the KVA map accordingly ? Mostly I was afraid of the consequences on VM_MIN_KERNEL_ADDRESS, which is #included so widely, and any complaints people might have about: - wasting NKPML4E-2 (i.e., 14) pages on small AMD64 systems (for the new empty L3 pages in KPDphys that will likely not be used); - "wasting" yet another page because dynamic memory will start at the first new L3 page (via KPML4BASE) instead of just using the KMPL4I'th one because VM_MIN_KERNEL_ADDRESS is now at -8TB instead of -.5TB -- with VM_MIN_KERNEL_ADDRESS at -.5TB, all KVAs use the single KMPL4I'th slot; - wasting 30 more pages because NDMPML4E grew from 2 to 32; and - adding a loop to set up NKPML4E entries in every pmap, instead of the single "shove KPDphys into one slot" code that used to be there, and making the pmap_pinit loop run 32 times instead of just 2 for the direct map. Adding these up, the option chews up 45 pages, or 180 kbytes, when compared to the current setup (1 TB direct map, .5 TB kernel VM). 180 kbytes is pretty trivial if you're planning to have a couple of terabytes of RAM, but on a tiny machine ... of course if it's that tiny you could run as i386, in 32 bit mode. :-) If we copied the kernel's L4 table to new pmaps -- or even just put in a new "ndmpdpphys" variable -- we could avoid allocating any pages for DMPDPphys that we know won't actually be used. That would fix the "30 extra" pages above, and even regain one page on many amd64 setups (those with <= 512 GB). We'd be down to just 14 extra pages = 56 kbytes, and the new loop in pmap_pinit(). Here's prototype code for sizing DMPDPphys, for illustration: old: DMPDPphys = allocpages(firstaddr, NDMPML4E); new: ndmpdpphys = howmany(ndmpdp, NPML4EPG); if (ndmpdpphys > NDMPML4E) panic("something or other"); /* or shrink to fit? */ DMPDPphys = allocpages(firstaddr, ndmpdpphys); and then instead of connecting NDMPML4E pages, connect ndmpdpphys of them. Would that break anything? The direct mapped VA range is known in advance; if you get a bad value it will "look" direct- mapped, but now not have an L3 page under it, whereas before it would always have an L3 page, just no L2 page. (Offhand, I think this would only affect pmap_enter(), and calling that for an invalid physical address would be bad anyway.) [I'm also not sure if we might be able to tweak the KPTphys usage slightly to eliminate whole pages full of L1 PTEs, e.g., if the GENERIC kernel occupies about 15 MB, we can map it with 7 2MB big page entries in KPDphys, then just one "regular" PTE-page and 256 "regular" PTEs in the first actually-used page of KPTphys. (This would recover another 7 pages in this particular example.) But this would at least affect pmap_init()'s loop over nkpt entries to initialize the vm page array entries that describe the KPTphys area, so I did not attempt it.] Chris From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 22:15:26 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 467EA2C2; Fri, 28 Jun 2013 22:15:26 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ea0-x22f.google.com (mail-ea0-x22f.google.com [IPv6:2a00:1450:4013:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id A9C9F1945; Fri, 28 Jun 2013 22:15:25 +0000 (UTC) Received: by mail-ea0-f175.google.com with SMTP id z7so1273456eaf.20 for ; Fri, 28 Jun 2013 15:15:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=d0KwWyRhazjX8nas5jCUtCFTbsIGXua5hjGKpYY5LIA=; b=a3nklq2CkqisLxTSZJYgVivL2TSv+Mxf0WMDw+DdKyFlppfOoVceRJwum8wnvCOab2 1nTDqVnTG0E3XIkpjYscKeg9GQTcuvsVPtReV1DpJMXMp/ZDhlTwpAwYq3KQplD30ZBk w44LaxXL1Fnlkw78wGhsfUrsQInd9nnwEzLrj6OsORqIkqw0ul1B6r79h5ilNOjG+ojR Dj+xMyO+Oa3nBsd2DCB750QWgZqMhxv1dwHgI30hIU2+g/qvqXHv1BuHpLWTrSLcteQP cbJzuvffrRyLZXPc9eUm24MeB0Hmt4VI7b7L/KuSMs/PgQsbJDAlqS8okozT2uf+VTqr WEHw== X-Received: by 10.14.100.2 with SMTP id y2mr15556761eef.75.1372457724616; Fri, 28 Jun 2013 15:15:24 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPSA id bj46sm13140611eeb.13.2013.06.28.15.15.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 28 Jun 2013 15:15:23 -0700 (PDT) Sender: Alexander Motin Message-ID: <51CE0AF7.6090906@FreeBSD.org> Date: Sat, 29 Jun 2013 01:15:19 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: b_freelist TAILQ/SLIST References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> In-Reply-To: <20130628065732.GL91021@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Adrian Chadd , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 22:15:26 -0000 On 28.06.2013 09:57, Konstantin Belousov wrote: > On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: >> While doing some profiles of GEOM/CAM IOPS scalability, on some test >> patterns I've noticed serious congestion with spinning on global >> pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is >> already very simple, I've tried to optimize probably the only thing >> possible there: switch bswlist from TAILQ to SLIST. As I can see, >> b_freelist field of struct buf is really used as TAILQ in some other >> places, so I've just added another SLIST_ENTRY field. And result >> appeared to be surprising -- I can no longer reproduce the issue at all. >> May be it was just unlucky synchronization of specific test, but I've >> seen in on two different systems and rechecked results with/without >> patch three times. > This is too unbelievable. Could it be, e.g. some cache line conflicts > which cause the trashing, in fact ? I think it indeed may be a cache trashing. I've made some profiling for getpbuf()/relpbuf() and found interesting results. With patched kernel using SLIST profiling shows mostly one point of RESOURCE_STALLS.ANY in relpbuf() -- first lock acquisition causes 78% of them. Later memory accesses including the lock release are hitting the same cache line and almost free. With "clean" kernel using TAILQ I see RESOURCE_STALLS.ANY spread almost equally between lock acquisition, bswlist access and lock release. It looks like the cache line is constantly erased by something. My guess was that patch somehow changed cache line sharing. But several checks with nm shown that, while memory allocation indeed changed slightly, in both cases content of the cache line in question is absolutely the same, just shifted in memory by 128 bytes. I guess the cache line could be trashed by threads doing adaptive spinning on lock after collision happened. That trashing increases lock hold time and even more increases chance of additional collisions. May be switch from TAILQ to SLIST slightly reduces lock hold time, reducing chance of cumulative effect. The difference is not big, but in this test this global lock acquired 1.5M times per second by 256 threads on 24 CPUs (12xL2 and 2xL3 caches). Another guess was that we have some bad case of false cache line sharing, but I don't know how that can be either checked or avoided. At the last moment mostly for luck I've tried to switch pbuf_mtx from mtx to mtx_padalign on "clean" kernel. For my surprise that also seems fixed the congestion problem, but I can't explain why. RESOURCE_STALLS.ANY still show there is cache trashing, but the lock spinning has gone. Any ideas about what is going on there? -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 29 00:50:30 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A095EF61; Sat, 29 Jun 2013 00:50:30 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qe0-x231.google.com (mail-qe0-x231.google.com [IPv6:2607:f8b0:400d:c02::231]) by mx1.freebsd.org (Postfix) with ESMTP id 56A151E86; Sat, 29 Jun 2013 00:50:30 +0000 (UTC) Received: by mail-qe0-f49.google.com with SMTP id cz11so923292qeb.36 for ; Fri, 28 Jun 2013 17:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=1jAx6GyMpQFeHLD2vvqnJP4c8AjqxOLHoyt25qwYOZU=; b=iSiK9/sfKGMF+BYm9lp07i5Fb6QlWprWtymMfoGthrCAo6iW6bgBFh2/njatI/77Ik OAhn6nqQia+Iw4JOsSLW+8lV83CZvNSo5GeMw99S7MOPFvuaQpDhvGaegdy+3vnAs/Ie NlD9pucvaCm22XmVdltfPSsx5EpC6MAiyYum4DOO75S1tXK0NwsDKAAkQwNQsaEKc3kx sW1H9CUuToloEveQQcnxqSFtGG1Iv/RvPdw0P0k6X8UktwRI+CMGmTosNYBzsSPRUXXX wbjPdzgjZ00/1lfxxOcigKnKy45FtBt4jhB4mn9RWfmWXsT/nkGqe5tMecqN3MEY1QKH 41FQ== MIME-Version: 1.0 X-Received: by 10.224.43.3 with SMTP id u3mr21203342qae.92.1372467029941; Fri, 28 Jun 2013 17:50:29 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.214.7 with HTTP; Fri, 28 Jun 2013 17:50:29 -0700 (PDT) In-Reply-To: <51CE0AF7.6090906@FreeBSD.org> References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CE0AF7.6090906@FreeBSD.org> Date: Fri, 28 Jun 2013 17:50:29 -0700 X-Google-Sender-Auth: g9Cl_A2kEjRwH_V3D9L2Y1fcCR0 Message-ID: Subject: Re: b_freelist TAILQ/SLIST From: Adrian Chadd To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 Cc: Konstantin Belousov , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 00:50:30 -0000 On 28 June 2013 15:15, Alexander Motin wrote: > I think it indeed may be a cache trashing. I've made some profiling for > getpbuf()/relpbuf() and found interesting results. With patched kernel using > SLIST profiling shows mostly one point of RESOURCE_STALLS.ANY in relpbuf() > -- first lock acquisition causes 78% of them. Later memory accesses > including the lock release are hitting the same cache line and almost free. > With "clean" kernel using TAILQ I see RESOURCE_STALLS.ANY spread almost > equally between lock acquisition, bswlist access and lock release. It looks > like the cache line is constantly erased by something. Can you narrow down the resource stall check to each of the sub-types? See which one/ones it is? -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 29 02:35:42 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D2C59EDC; Sat, 29 Jun 2013 02:35:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 45B081153; Sat, 29 Jun 2013 02:35:42 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r5T2ZXSs088897; Sat, 29 Jun 2013 05:35:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r5T2ZXSs088897 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r5T2ZWit088893; Sat, 29 Jun 2013 05:35:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 29 Jun 2013 05:35:32 +0300 From: Konstantin Belousov To: Alexander Motin Subject: Re: b_freelist TAILQ/SLIST Message-ID: <20130629023532.GW91021@kib.kiev.ua> References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CE0AF7.6090906@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="dn4lWQ0qhoFTQ1P3" Content-Disposition: inline In-Reply-To: <51CE0AF7.6090906@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Adrian Chadd , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 02:35:42 -0000 --dn4lWQ0qhoFTQ1P3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jun 29, 2013 at 01:15:19AM +0300, Alexander Motin wrote: > On 28.06.2013 09:57, Konstantin Belousov wrote: > > On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: > >> While doing some profiles of GEOM/CAM IOPS scalability, on some test > >> patterns I've noticed serious congestion with spinning on global > >> pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is > >> already very simple, I've tried to optimize probably the only thing > >> possible there: switch bswlist from TAILQ to SLIST. As I can see, > >> b_freelist field of struct buf is really used as TAILQ in some other > >> places, so I've just added another SLIST_ENTRY field. And result > >> appeared to be surprising -- I can no longer reproduce the issue at al= l. > >> May be it was just unlucky synchronization of specific test, but I've > >> seen in on two different systems and rechecked results with/without > >> patch three times. > > This is too unbelievable. Could it be, e.g. some cache line conflicts > > which cause the trashing, in fact ? >=20 > I think it indeed may be a cache trashing. I've made some profiling for= =20 > getpbuf()/relpbuf() and found interesting results. With patched kernel=20 > using SLIST profiling shows mostly one point of RESOURCE_STALLS.ANY in=20 > relpbuf() -- first lock acquisition causes 78% of them. Later memory=20 > accesses including the lock release are hitting the same cache line and= =20 > almost free. With "clean" kernel using TAILQ I see RESOURCE_STALLS.ANY=20 > spread almost equally between lock acquisition, bswlist access and lock= =20 > release. It looks like the cache line is constantly erased by something. >=20 > My guess was that patch somehow changed cache line sharing. But several= =20 > checks with nm shown that, while memory allocation indeed changed=20 > slightly, in both cases content of the cache line in question is=20 > absolutely the same, just shifted in memory by 128 bytes. >=20 > I guess the cache line could be trashed by threads doing adaptive=20 > spinning on lock after collision happened. That trashing increases lock= =20 > hold time and even more increases chance of additional collisions. May=20 > be switch from TAILQ to SLIST slightly reduces lock hold time, reducing= =20 > chance of cumulative effect. The difference is not big, but in this test= =20 > this global lock acquired 1.5M times per second by 256 threads on 24=20 > CPUs (12xL2 and 2xL3 caches). >=20 > Another guess was that we have some bad case of false cache line=20 > sharing, but I don't know how that can be either checked or avoided. >=20 > At the last moment mostly for luck I've tried to switch pbuf_mtx from=20 > mtx to mtx_padalign on "clean" kernel. For my surprise that also seems=20 > fixed the congestion problem, but I can't explain why.=20 > RESOURCE_STALLS.ANY still show there is cache trashing, but the lock=20 > spinning has gone. >=20 > Any ideas about what is going on there? FWIW, Jeff just changed pbuf_mtx allocation to use padalign, it is a somewhat unrelated change in r252330. Are pbuf_mtx and bswlist are located next to next in your kernel ? If yes, then I would expect that the explanation is how the MESI protocol and atomics work. When performing the locked op, CPU takes the whole cache line into the exclusive ownership. Since our locks try the cmpset as the first operation, and then 'adaptive' loop interleaving cmpset and check for the ownership, false cache line sharing between pbuf_mtx and bswlist should result exactly in such effects. Different cores should bounce the ownership of the cache line, slowing down the accesses. AFAIR Intel exports some MESI events as performance counters, but I was not able to find a reference in the architecturally defined events. It seems to be present in the model-specific part. --dn4lWQ0qhoFTQ1P3 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzkf0AAoJEJDCuSvBvK1BuiIP/ik9OFwcgQ9EwwhCedTG2wik w/dbyY1GvRC5QPKO4PrMN01kV/wY8njulycYqCv/BJ4RnPCyqNmkCpV9l1YPe7Ha HI6JB6G2pSWPMkQeK9T+9jZ4ED9HwqYVxapBWqMgtDV2G8gKSCNyxY16hib8VeXf ARLCr1Z6oDyCTzY1fbZ7WqpRX5cCcZXyAHWOMJMC1oLQMZ3JuHtEUAB3brEZwPFo Upze+7k+aT/pw7TvIF4Lz81a4eLU3IXtW/DgyfXuW2LxLpiEeAs0DuiK4sh0UBFs MyWIHzv37s976avg6KS3yDwkkYEpHBJAN/M9CT86xrTvIDuIchLmK0EOqS5IiFVq xStZiSzT1bM4L0qmZhMfKkUN5qbOSsWa/ptC1kqm6DPWSjASwuUmH2maxI3npgYO e+4RI1LdPfceg+3CIZtZcN4Cue/B+VHd3KL4SnGgbg3L5+kZuFqQRelkBKHphytN RKVElZe/VmeIC9zWIVG/BfKWTAJsfhp/Jgzu7CSfUYW0oiRRE+J+2nwQPs94GLLu z4Dt0cMadAq7v+t9EAQ1reHnZCNrrJ+Q6PzqOxmgTad4ofpHHWzpfYj9jM2/YOdO XXaZMgzvTbIe8bWhSkYgSjeO1kHVnD0UBOiGsuk79615yjSBSFDwRpALFK/ApacU bBn7Ta8HVokib274b/jF =4XcW -----END PGP SIGNATURE----- --dn4lWQ0qhoFTQ1P3-- From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 29 07:06:18 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 12E6A6E7; Sat, 29 Jun 2013 07:06:18 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-we0-x229.google.com (mail-we0-x229.google.com [IPv6:2a00:1450:400c:c03::229]) by mx1.freebsd.org (Postfix) with ESMTP id 7B35D1BD6; Sat, 29 Jun 2013 07:06:17 +0000 (UTC) Received: by mail-we0-f169.google.com with SMTP id n57so1849433wev.0 for ; Sat, 29 Jun 2013 00:06:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=FHVfNA8s1DjfH2tgZEkDU8BPxUXFWYSzK5kTbnNc5Eg=; b=Gg11tmucpSPV+wicOO/OUMqPY57s98cq/uFfDbUhoNd3wfHC4hvD27ty60/8HAEzI6 mutH4HHfb3tctjGyBFLe+GcUvkI0umNEtUuYal3rFFQBB59vUQDJVD2cewB6mNTBPKa9 QLcGYVSmiKzOILxg/o9qBoIk5SrlF9PPC3GKn9jcLL8aZqV90qfhcca9gEFlAzKklPJf AM3Pt+oixRUHCzGImlAS5JBI4p03T/PGfpFAuoQTcy87ASVagncuIb5kHs/HzESlzJKm yJ2GqbwZ0SCiiB7pYH4TvnsEHaN3z3cZtaI5FJv+D7mcDi78y8wclLj7oFmyCU0ttkze GgbQ== X-Received: by 10.180.38.37 with SMTP id d5mr5081863wik.37.1372489576478; Sat, 29 Jun 2013 00:06:16 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPSA id c44sm15269938eeb.8.2013.06.29.00.06.14 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 29 Jun 2013 00:06:15 -0700 (PDT) Sender: Alexander Motin Message-ID: <51CE8763.2090406@FreeBSD.org> Date: Sat, 29 Jun 2013 10:06:11 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: b_freelist TAILQ/SLIST References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CE0AF7.6090906@FreeBSD.org> <20130629023532.GW91021@kib.kiev.ua> In-Reply-To: <20130629023532.GW91021@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Adrian Chadd , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 07:06:18 -0000 On 29.06.2013 05:35, Konstantin Belousov wrote: > On Sat, Jun 29, 2013 at 01:15:19AM +0300, Alexander Motin wrote: >> On 28.06.2013 09:57, Konstantin Belousov wrote: >>> On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: >>>> While doing some profiles of GEOM/CAM IOPS scalability, on some test >>>> patterns I've noticed serious congestion with spinning on global >>>> pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is >>>> already very simple, I've tried to optimize probably the only thing >>>> possible there: switch bswlist from TAILQ to SLIST. As I can see, >>>> b_freelist field of struct buf is really used as TAILQ in some other >>>> places, so I've just added another SLIST_ENTRY field. And result >>>> appeared to be surprising -- I can no longer reproduce the issue at all. >>>> May be it was just unlucky synchronization of specific test, but I've >>>> seen in on two different systems and rechecked results with/without >>>> patch three times. >>> This is too unbelievable. Could it be, e.g. some cache line conflicts >>> which cause the trashing, in fact ? >> >> I think it indeed may be a cache trashing. I've made some profiling for >> getpbuf()/relpbuf() and found interesting results. With patched kernel >> using SLIST profiling shows mostly one point of RESOURCE_STALLS.ANY in >> relpbuf() -- first lock acquisition causes 78% of them. Later memory >> accesses including the lock release are hitting the same cache line and >> almost free. With "clean" kernel using TAILQ I see RESOURCE_STALLS.ANY >> spread almost equally between lock acquisition, bswlist access and lock >> release. It looks like the cache line is constantly erased by something. >> >> My guess was that patch somehow changed cache line sharing. But several >> checks with nm shown that, while memory allocation indeed changed >> slightly, in both cases content of the cache line in question is >> absolutely the same, just shifted in memory by 128 bytes. >> >> I guess the cache line could be trashed by threads doing adaptive >> spinning on lock after collision happened. That trashing increases lock >> hold time and even more increases chance of additional collisions. May >> be switch from TAILQ to SLIST slightly reduces lock hold time, reducing >> chance of cumulative effect. The difference is not big, but in this test >> this global lock acquired 1.5M times per second by 256 threads on 24 >> CPUs (12xL2 and 2xL3 caches). >> >> Another guess was that we have some bad case of false cache line >> sharing, but I don't know how that can be either checked or avoided. >> >> At the last moment mostly for luck I've tried to switch pbuf_mtx from >> mtx to mtx_padalign on "clean" kernel. For my surprise that also seems >> fixed the congestion problem, but I can't explain why. >> RESOURCE_STALLS.ANY still show there is cache trashing, but the lock >> spinning has gone. >> >> Any ideas about what is going on there? > > FWIW, Jeff just changed pbuf_mtx allocation to use padalign, it is a > somewhat unrelated change in r252330. Heh! It was unexpected. I've seen that commit, but haven't look that deep. I'll pick it up and guess case will evaporate. > Are pbuf_mtx and bswlist are located next to next in your kernel ? Yes, as I've tried to say above they are on the same cache line. > If yes, then I would expect that the explanation is how the MESI > protocol and atomics work. When performing the locked op, CPU takes the > whole cache line into the exclusive ownership. Since our locks try the > cmpset as the first operation, and then 'adaptive' loop interleaving > cmpset and check for the ownership, false cache line sharing between > pbuf_mtx and bswlist should result exactly in such effects. Different > cores should bounce the ownership of the cache line, slowing down > the accesses. I understand that lock attempt will steal cache line from lock owner. What I don't very understand is why avoiding it helps performance in this case. Indeed, having mutex on own cache line will not let other cores to steal also bswlist, but it also means that bswlist should be prefetched separately (and profiling shows resource stalls there). Or in this case separate speculative prefetch will be better then forced one which could be stolen? Is there cases when it is not, or the only reason to not pad all global mutexes is only saving memory? -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 29 14:49:08 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0E7C04C1; Sat, 29 Jun 2013 14:49:08 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 931781905; Sat, 29 Jun 2013 14:49:07 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r5TEmx2A040874; Sat, 29 Jun 2013 17:48:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r5TEmx2A040874 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r5TEmxW3040873; Sat, 29 Jun 2013 17:48:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 29 Jun 2013 17:48:59 +0300 From: Konstantin Belousov To: Alexander Motin Subject: Re: b_freelist TAILQ/SLIST Message-ID: <20130629144859.GB91021@kib.kiev.ua> References: <51CCAE14.6040504@FreeBSD.org> <20130628065732.GL91021@kib.kiev.ua> <51CE0AF7.6090906@FreeBSD.org> <20130629023532.GW91021@kib.kiev.ua> <51CE8763.2090406@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="mUmOsk7ZE69Fau6J" Content-Disposition: inline In-Reply-To: <51CE8763.2090406@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Adrian Chadd , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jun 2013 14:49:08 -0000 --mUmOsk7ZE69Fau6J Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jun 29, 2013 at 10:06:11AM +0300, Alexander Motin wrote: > I understand that lock attempt will steal cache line from lock owner.=20 > What I don't very understand is why avoiding it helps performance in=20 > this case. Indeed, having mutex on own cache line will not let other=20 > cores to steal also bswlist, but it also means that bswlist should be=20 > prefetched separately (and profiling shows resource stalls there). Or in= =20 > this case separate speculative prefetch will be better then forced one=20 > which could be stolen? Is there cases when it is not, or the only reason= =20 > to not pad all global mutexes is only saving memory? I can speculate that it is the case when speculative execution helps. If mutex and list head are on the different cache lines, then cpu could speculatively read the head, and then prove that executing the read before the lock acquisition does not break the ordering rules (because lock protects the head, other core indeed cannot modify the head if the lock acquisition was successfull). I think it is very similar reason why locked instructions as barriers are faster then the explicit barriers, cpu could still do the speculative execution after the lock prefix if the ordering is provable consistent. Please see the Intel IA32 architecture optimization manual 8.4.5 for the recommendations (but not much explanation). Yes, I think putting all locks on dedicated cache lines is the waste, only hot locks need this. --mUmOsk7ZE69Fau6J Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzvPbAAoJEJDCuSvBvK1BpdIQAKj4KU/KFAPjIGQWMBi0Mmqs CoItYYEC+okQgZSFpZn6KFWlZmLxb9fB8S0hPL6ytKwdi6XAwNnVdhSuNrITDYZQ 10dylktNHkpeS9/OxmEmxIPe9kvPxlbhd+ffBUHiQqFpbzYgpVJbTVed9ClwrPxI Zp+1pWDugRYnGzzrNz8B4DsD2EkxzlxVG+6bN4Gs/0Hk9FZ2dpbZ0cosESmd8vT2 Jtl2/Mc56pJ4HXOM65Pe3gUwx8Yo1Mj9XQrmC9FroI9iuJL987QhK5aN66r1A+x/ Yhh0koj8+cQ0Vzi6BKHRfVkrd9PU8mV7JKXvsAuaZfQFjYmMpZeNK9WYt3et4Xhy +YO6Cqvy09mJs2JhlsCbpdk/Ytl+BryhjI5WdSMObtw4nuYGOipwIX4xkmNSrn61 IbhKWTnrhrsx5deeARUPQ1Bb9zn5QuBEaXOWO4d+w0yJDFDMTMosj4FlhRRsk3b9 NmdPPOukESP5CgZcgvGvsBn7mCcZlXucoZBwcubOvM6JXBWc3S2DJuA+C39Fs3CN ZVLaQZqna3AwFZCXGhcWFQclsIHlrbYbVgndfsXs8mT2wq15bz2rcdX5A1AL1nAW z66mKxC28ZksEbEC0eZv+IWGVcXnxiOGK2XHXLVpzKq3FEoKG/13QfVRgTOuSvhq PtncUTjUY0V1W/fCJApG =vB/a -----END PGP SIGNATURE----- --mUmOsk7ZE69Fau6J--