Date: Thu, 27 Feb 2014 11:32:47 +0100 From: Julien Charbon <jcharbon@verisign.com> To: freebsd-net@freebsd.org Subject: Re: TCP stack lock contention with short-lived connections Message-ID: <len481$sfv$2@ger.gmane.org> In-Reply-To: <op.w56mamc0ak5tgc@dul1rjacobso-l3.vcorp.ad.vrsn.com> References: <op.w51mxed6ak5tgc@fri2jcharbon-m1.local> <op.w56mamc0ak5tgc@dul1rjacobso-l3.vcorp.ad.vrsn.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------080207040208060206070203 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, On 07/11/13 14:55, Julien Charbon wrote: > On Mon, 04 Nov 2013 22:21:04 +0100, Julien Charbon > <jcharbon@verisign.com> wrote: >> just a follow-up of vBSDCon discussions about FreeBSD TCP >> performances with short-lived connections. In summary: <snip> >> >> I have put technical and how-to-repeat details in below PR: >> >> kern/183659: TCP stack lock contention with short-lived connections >> http://www.freebsd.org/cgi/query-pr.cgi?pr=183659 >> >> We are currently working on this performance improvement effort; it >> will impact only the TCP locking strategy not the TCP stack logic >> itself. We will share on freebsd-net the patches we made for >> reviewing and improvement propositions; anyway this change might also >> require enough eyeballs to avoid tricky race conditions introduction >> in TCP stack. Just a follow-up on this TCP performance improvements task: 1. Our first related patch has been applied in HEAD: Decrease lock contention within the TCP accept case by removing the INP_INFO lock from tcp_usr_accept. http://svnweb.freebsd.org/base?view=revision&revision=261242 Thanks to reviewers. 2. We studied an another lock contention related to INP_INFO when TCP connections in TIME_WAIT state are cleaned-up. The context: This lock contention was found when checking why our Intel 10G was dropping packets in reception even with plenty of free bandwidth. To study this issue we computed the distribution of TCP connection time lengths (i.e. time between the first SYN and the last ACK for a given TCP session) at the maximum rate of TCP connections per second _without_ a single packet drop using FreeBSD 10.0-RELEASE (see joined conntime-bsd10.0-release.pdf) In this graph, in X you have time in second where the SYN was received, and in Y you have TCP session duration in millisecond (i.e. the difference between first SYN received time and last ACK send time). As you can see at some point every 500ms the TCP connection time raises in spikes. This periodicity led us to tcp_slowtimo() that calls tcp_tw_2msl_scan() with the INP_INFO lock taken. Our theory is: Every 500ms there is a competition for the INP_INFO lock between tcp_slowtimo() and tcp_input(), and tcp_input() is indeed directly called by the NIC RX interruption handler. Then, during the whole duration of tcp_tw_2msl_scan() call, packets are no more dequeued from NIC RX rings, and once all RX rings are full the NIC starts to drop packets in reception. The calculus of time needed to filled-up all available RX rings descriptors follows this theory: - 40k TCP connections per second (with 5 packets received per TCP connection) = 200k packets per second - We use 4 RX queues of 2048 descriptors each = 8192 RX descriptors overall Time to filled-up all available NIC descriptors at 200k packet per second: 8192/200000 = 40.96 milliseconds Which is coherent with what we see on the conntime-bsd10.0-release.pdf graph. To confirm this theory, we introduced a new lock (see joined patch tw-lock.patch) to protect the TIME_WAIT global list instead of using INP_INFO, and now the TIME_WAIT states are cleanup one by one in order to prioritize the NIC interruption handler against tcp_tw_2msl_scan(). See joined conntime-bsd10.0-patched.pdf: No more spikes and the maximum TCP connection rate without dropping a single packet becomes: - FreeBSD 10.0: 40k - FreeBSD 10.0 + patch: 53k Obviously, to mitigate this lock contention there are various solutions: - Introduce a new time-wait lock as proposed in joined patch - Call tcp_tw_2msl_scan() more often in case of high workload - Use INP_INFO_TRY_WLOCK() in tcp_tw_2msl_scan() to clean-up time-wait objects only when nobody else handles INP_INFO lock - Etc. The strategy being to prioritize packet reception over time-wait objects cleaned-up as: - we hate dropping packet in reception when the bandwidth is far from being full - the maximum of used time-wait objects is configurable (net.inet.tcp.maxtcptw) - in case of time-wait objects memory exhaustion, the current behavior is already optimal: The oldest time-wait object is recycled and directly reused. We picked the time-wait lock way because it suits well our long-term strategy to completely mitigate the INP_INFO lock contention everywhere in TCP stack. Any thoughts on this particular behavior? -- Julien --------------080207040208060206070203 Content-Type: application/pdf; name="conntime-bsd10.0-release.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="conntime-bsd10.0-release.pdf" JVBERi0xLjMgCjEgMCBvYmoKPDwKL1BhZ2VzIDIgMCBSCi9UeXBlIC9DYXRhbG9nCj4+CmVu ZG9iagoyIDAgb2JqCjw8Ci9UeXBlIC9QYWdlcwovS2lkcyBbIDMgMCBSIF0KL0NvdW50IDEK Pj4KZW5kb2JqCjMgMCBvYmoKPDwKL1R5cGUgL1BhZ2UKL1BhcmVudCAyIDAgUgovUmVzb3Vy Y2VzIDw8Ci9YT2JqZWN0IDw8IC9JbTAgOCAwIFIgPj4KL1Byb2NTZXQgNiAwIFIgPj4KL01l ZGlhQm94IFswIDAgNjQwIDQ4MF0KL0Nyb3BCb3ggWzAgMCA2NDAgNDgwXQovQ29udGVudHMg NCAwIFIKL1RodW1iIDExIDAgUgo+PgplbmRvYmoKNCAwIG9iago8PAovTGVuZ3RoIDUgMCBS Cj4+CnN0cmVhbQpxCjY0MCAwIDAgNDgwIDAgMCBjbQovSW0wIERvClEKZW5kc3RyZWFtCmVu ZG9iago1IDAgb2JqCjMxCmVuZG9iago2IDAgb2JqClsgL1BERiAvVGV4dCAvSW1hZ2VJIF0K ZW5kb2JqCjcgMCBvYmoKPDwKPj4KZW5kb2JqCjggMCBvYmoKPDwKL1R5cGUgL1hPYmplY3QK L1N1YnR5cGUgL0ltYWdlCi9OYW1lIC9JbTAKL0ZpbHRlciBbIC9GbGF0ZURlY29kZSBdCi9X aWR0aCA2NDAKL0hlaWdodCA0ODAKL0NvbG9yU3BhY2UgMTAgMCBSCi9CaXRzUGVyQ29tcG9u ZW50IDgKL0xlbmd0aCA5IDAgUgo+PgpzdHJlYW0KeNrtneu6qyqQRWH7qCDK6e73/9uAcjOJ WRCuyRzfOdkmLhRhWkCBJSEAAAAAAAAAAAAAAAAAAADgQFLNrrYYp6vonR3wYzAqFFJvbGKn snd+wG+x8XOD7/YDgGbsp+IkZSRQIwBN4CunXElPUN33Y7R3fsBvocQnNmX7hOn6MXQAQVP0 0IPsHPoD/VCye9r+Gt/Mv39qCx/4MB+FlSfZqb+n4w/0BkFdhJHdyp/7X6A/UJlVjT923fY+ 8z9Df6AycrfTboxTfpl/g/5AT6A/0BPoD/Skjf4YPWB3u/l2fNtUL2GXDz+fyMOLGa3kuXw7 zqUHWva8PHB7+t8UK7XpLmc99x+n3Fb1dWXBGYI8RA5VatH7VSqT6Ol1JCDXsM++stzjjEgr /TFxLsJ5vZuZNWJmuKSqTtqfN74Gfyo5fVjJcxlXnSfb1OHseUWkP/ubFrN1B1zPeuw/E/BN 5W6nVgZxHiL9mfMyk3hV+9UfbP7ycgUYnUF+V4vVSn/yD7s3nZnDS04O62V+FjS0PKbuY0/S xa9kT6YOF6rucUsbve0ogMezBqc8dSP5aRL3KA+Pf39ejPsnvLyPS4+tmUcZk7b6U5aAcnkM xA+3uNk4d7NAf4z5VPvqj7PpH2NP+tWv7k8m3+mPb0fiZ2cl9uibagCV7dN/a5QoeJQH8yWw 0Wf6836Qu/SXJ4MycEWw6Wb8zKpr2M0uwQ/zrFtvu032ze+IDsKnbJdb648Lpi3Bppop4TaO 3eIoU93aXVPZ48ijMuOZxOu84pFMCqWJN/rTM+LroZzHs57sXKtL9QLUWXajcc6iPOiZJR70 0M7TU3H5xdq/owyiItAG1evP7pJ0Pxpt1aL/Z7cJj3ZE5TijAJuOP5TxMmVkClJbtXODhT1+ dXNTVcAkMmThoeRlJcV1XYUdQBjLc574ye7tyIJt5IOzuv0aNWwwsnNdvW2N86D+W/klgzZT 0S+nRTzKIC6CNdSf3WUUflpkty048TsuB8ke4HSk6fhD2ro51WU3/vcYaNCzC6ftAWUf6I/p g23+vEJedx9jIbpJKa3Q/FnDsZK2Y+YvdPt+tLviqr81sjzP9OeewLH7H4rA68/9ogdHgly2 t/0YNYnHcpyS1u3vUTcH0m7870MH3fT1g4EEOX0a5Lb9Pf7oSLZRRt60v+zB0RKe1R78rF29 ilxnZN0veVDtKQ/XdFzaXzuiltH+oAi8GbUHPncRod0+p4fHbhvvy/nlcpC3LE3qOo0++jss jNsIO+hnf4wFng39k3OI3I0/jj+yAwAq3+hvXY+Ts6dnPRC2drX7Treg1sEXjD+ECF0rl/FH 1Pa7/UEROBvm9Me8+0cNMPzARW/bQ+kvl4PIGY1gD/0RU3VsdxthB307DMfmLZF4bN7+4n8R dL/X3zn0JXrs+nhW4v7I/J02p8alcrgWVaWH/pedXnt71v+yU/FMf74IjFt6P62sSWR3Me4U Ke227v75Hb4cTf9vRs9MF/0dw7XdbZyOZlOKkmu/7eZ7YrH/mbDI/ywP63HxP59uGz2w9v7n TRP078Tm/k4+ntX5n5XCVr6pvuG+u/sgysPhf9kfrjX0Pz/qzxUBMxu6h2cG2dLv0r8wffHK vP53bu+miOyOSznC/r3iOpLQ7qrNb0QTVHq5zjGddvx8GdZ5F5l4+BafTFUTc63lubX68S39 v1PXpvW8ntV0wEwOqZCrGkuveq+Qj3k4xSiup9fXZuffHvXni0BP4+lbQRx+wGCX7uaZKcGd 0v+O7eMy3Y6oHGd0v2D9wVtWJ+3Yz1KS330oB/p7y67sk4gmgIsD/YHXiH3l1DrgqgD9AdAD 6A/0BPoDPYH+QE+gP9AT6A/0BPoDPYH+QE+gP9AT6A/0BPoDPYH+QE+gP9AT6A/0BPoDPYH+ QE+gP9AT6A/0BPoDPYH+QE+gP9AT6A/0BPoDPYH+QE+gP9AT6A/0BPoDPYH+QE+gP9AT6A/0 BPoDPYH+QE+gP9AT6A/0BPoDPYH+QE+gP9AT6A/0BPoDPYH+QE+gP9AT6A/0BPoDPYH+QE+g P9AT6A/0BPoDPYH+QE+gP9AT6K8JS+8MjAr0BxrAqNSfnK4i3gH9gfpIrvXH6CZ2I0QP9Afq sxv98Z2cHx7oD1RHcN3+SsrU9sajXdAfqA5nWn+C6r4fixUH/TXhp8e/20oO/ZkxSNwBhP5A ZaSye9Af6MW6E/K6/dX8+6e28IEP81Faf/QE4w/QA6HYqBDwv4BeMPifQUfc/BvH/BsYCOiv CT/t/7sD+gM9gf5AT6A/UAIpZVY66A98DFuNg3ll6UmhP/AhelXzxhjbVsqTFQj9gc9YA7PH +JqYGvoDn8Fuvr0H+gM9gf5AIbIGwNAf+JyNEabGvxkKhP6a8N3zbztlhHN2WVr1J6A/8DGc EaEkyHh6UugPfAwVygQSpcGMpL3zDuaH74KvRK6pzj8C/YECCEqVCeRUpCeF/voz/+BECr28 PscBA/01YH6B3SAD0lNDf99IS8HTgIzUDXMKvhG98oWu6nOnWH8FDI0b/PVwPO8Y/4IenANf +P9AF/hm/tkw/wF6sNFNSLGh/wf6sOnBb/riewL9NeKrPYAaITImPwj0B/oC/bVhydw3CZj/ GJwv0NhrBMf8B+jHytlBelLorwnLV9u/nIVXNmnvvIP5yfK8HEB//ZneNgrOMP4YmukldgvW X43Od+uPMYw/QF/ywv9BfwMwv3FkHPO/oBvMrH9esf4FdOEMvLFj/d+ozN/E3oH1z6Pz3fMf WP8MemLXP2/pSaG//sxvG8365xz5QX+gCFj/PDRL9s4pYMr0iTVHgdBfEypIbCDVMroSIuH/ m5SBlJQHxr/9mV5EHwD/H+gJ7N/wfPXzb/D/DU8NjY2jW/j/QF/g/xubcWxVHTJfPw39DcD8 2tQPoIs94/VH0N/EDKNbRndGBcP447cYRn/a/0IF/C/j8t3r/7T/Wf0P//O4fLX/BfZveIbR Sg1Uz09QxrD+YE7m1yb8z7/ISLqF/3lovnz9qSHLAw39gc/ZmOoD0hXxr6Zkevun3/zGOeMZ EyDQX3+m1x9nRI1/CYP/5acYRrdUKBOI9c8jM4xWasB3wVci1xfvvxQrPVdIM04vT8lBf/3J 1eYwmhaUKhPIX4Qhl0pzR99Q+6l3Go1SoL/+ZOtoGAFKoUTFXgx/zbQc00IzIoxHKdBfG2o8 /zGM/O6R2iwyZfakmaCLZ4mhv/5MoqN77t8BIrTVE/RQYpSsd77BF7S/5I3+KOVC60+30Czq AEJ/oAi3+hPm5dTQ35h8v/0jeonM8/ZX8++f2sIHPsxHaWUeK2OU2cP4A5RHurf/vngG8whL rj/hf+nG98bfUI3q/fu3tP9Z6CUK8D/343t9fEy+e/+WWOk57ca4GQgHQH8TM7443wL9teF7 7Z8MSE8N/fVnfI3dQQMyUvfO/o/wvfaPBaSnhv76M/fYGO3vzzKEONH+zs4QMsoG7e8MVFn/ N3fDbYD+wGe8nX+7BfprQ66pGj9wwtv5t/vUvbMP6mizGe/n3+6A/vrzBb5BtL+D01wrLU+o w4+j/R2agWxVeThH+zsxVdbftxT8u7X3d0kbZhM8p0r/r6X+1gzDdwL9jc0UzyYJzjD+mJcq zSjGH+CPzO1/Jhh/TEDz939g/PFr5Nf47C+nwfjja8m2cRh//BpVmsMp+oZYfzACzfU3jP3D +oMRyHeV3KQc3/8ibr69B/orxvhSqQLffbdP7qmvYID+ilGnqcy1je1EvdF1E0IKtq3pryCE /gYgU2OjPP8ht2P+g2/pA2Dob2wmadSlEALvHxyYH45/fwv0V4wq/b/sNnYScUJ/xahS45O0 v9lAf8X41fHvR0B/A3CnsS/vHEJ/xfjp9jdn7YEG+ivGWPpril4ALfaM159Df4347vUHdGdU sPTZD+ivIL+7/lS/XZqKy5tl/gb0NzajzLHdotffq/8F1v/9FqPoD/ZvBNq3v6PYRv1eI8oY xfrTnozlDm56ws2sf8kYfkB/jcid453F/ydE3jOY0N8AzK8/xP8bnOYxNhB/A/yVUXSUC+Jv jEAdpcygP8TfGIH2U2Wj2E3E3xiB9v6XUfSH+BtzM7v/GeOPERir/9d0/QHGHzOT7X8eZWyC 8ccIjKKGMzcNs4PxxwiMNf5oeTtg/DE1VWL8If7fr9H8+csPslMWxP8bgXxrVGP9QTPw/t8p aN0BbDb+wPt/Zyf/GY8RrCP9D+3v3FSJMdRMmh84/wj0NwJ1Apc3bH8/Sd0olz9O9vij1inL QZnE+GN0qsTGr5MwFRqQkbpRLn+A+2a0dR+vnf42hvHHCLS3cUMMQND/G4R7MTRf4wL9gT9S xY0M/f0Ydca4w+vvM6C/RmTauK+YON445SY4B+N0jc0l9FeMfPtXRUbjCHCnTGw6OoyOU7TT yEsI/RWjiv6+IAC5NHGxNqpfVUjODwf014ga9m+K4JREmhdkMipPIUZRAqG/YtRpKm80NozC yB/i3++cCDNUZpHkoL9ifENXLZP38e91cHJBT0MY/A79FSN/FV9u2zzK2sD38e8ZXQn0V5cq 9i//oZKG+nsb/3nT8nve/mr+/VNb+Kj6sdzsXfIOutQ4KPmXrr938e83appmjD86UmGObZnE /m22YYb/pRdL+6UqTft/d/HvJV2FMPGh4X+uSfP1L+MMm2/j32/n4lRp5t845t+GY3b7RxD/ fgCq2L+JHs3MegMr9FeMDiGGRmmcN6b6dnTF80fjkj3GHaeX95JdjTw4ZzzjBcDQXzEGW//S EM6IGv+q0UV6UuivGGPN/7bULRXKBBK8f7UvdZ5/m2GJFd8FX4lc1/Sk0F8xsse/U2jsDkGp MoE850kk6K8Y7fU3TL9RCu1cznHAQH+NqKCjZQrjeA/014gqz58P03Aj/tAAVAkVVOPZzNLg /UcjkL0cOX9xdOYJS7Pi/UcDUKXHn7/Gvq3/Lztpu1z+MkuN9afvTtkMjvcfjUBrU5W/qLAw eP/R3FR5jrxl+4v4pyNw28RWMFXDLE1A/NPxqRHjYBT9abJWn0J/JWk+jBhGnEw7ALNGIdBf G6r04gZxv+joBqrxXSna357UWX+afcaW/pdj4fOO9ac96fD+1UEM4Ol/xvrTrnwwHK3hnG4c /4CQm/gvr4H+ilEnGmTuGr+W/b+NbkKK7Sb+1Uugv2I0j1Q/jP/vPv7BLdBfOZq/x3KU8QdB /IMRaP8A3AjvPz+B/7k3H7yqo0Y/rqUAxaqa3zXHAkJ/I5BrxwbpAFr/M55/60qNANAftOnt xGn9z3j+tyv56++zFwBmZqYw8D+PQKVXTA80xngF/M9D0Pw10nUeXEpH0N34n1n6Emjorxzz x9HNhAakJu2d9y8ivzs2+fpTFpCYFPorR/tXTA+lTsTfHZnsh0Oavzg4A8TfnZnZH39D/N0x aP/8xyAB2BB/dwjar38ZpP1F/N0hGOoFmC2DAyL+7hA0D1U6iv8F8XeHoMpzREO5WF6B+Lsj UMP/PMwagzfkxB7SQH/lGOsFIC3RAVDFnuF+gf4KUif+bvPHmtJhdGdUMDz/1pf2MTYG6Ry+ ef/5HdBfOar0/7Iz0zj+gfof/r++VAmVMZSoXwD7NwTZywiGWUeQier5CcoY4l/1pU78jQnm 3xD/YAiqPP82w/iDIP7BCNRRwwTRYU7fs8D8b1fau+qGsH87pVp4ckf8+760fv/bGM9f7nTT K08ZpTvWP3dlLB210qZ2vgi6UZ7VAYT+ylHlBdR1gvMWRDufZd7gl0B/RWltxobwv5hVf7ny g/4KUqfKR38ByKG/3DdgQn/lqDL+HT7+PfQ3CvfNYY0YVyM8/2GivlCG9192J787ljv/cbuv lf7yg78Q6K8kzc3REP6/D4K/EOivFUvrIAcjSPMPQH/laP4c5Sjr/z4A+ivIUO/4mEJ+0F9J mg9HR/D//QVJzdCY8YeXNEB/bWgfm2MgJDf608ukdxo7aaC/glTwv+THeB5icGxg9NCfic92 CdIG/ZXjXg2tY2wMJL+Naf1J83zI5SEl6K8czYcRc3T/pCRGf8JM0rFYcdDfANQJHDOMAInV n7SbHuhvAOrEeIb+fo0q77icPTab5qb91fz7p7bw8emHMmMv9xoTl3Xku4NWuY5a+sP4ozY1 3Cjto6pWgMH/0oD87lhujIO59Af/cz86xKYcTn96/o1j/q0ag61/mQLorxx13uP7BSK7Afor SPsXIN3tm0K40F9BWk+yTTP+eA30V5Aas2FDPGNeD+ivINnzGN8wxs0D+itI85dsTTMB9xLo ryCt11F9wbgZ+ivIWDqaQoDQX0GGWv8yB9BfG6rMDX/B4Bj6K0iV5z8yTziJAKG/0Zkg/v0H QH8Fyfa/NI4NMxDQX0FaV3n+otZhgP7a8MH6v/mdfDdAfwVp/RzR9OqD/opS5VHJ9pPKLYH+ ClJn/d9NcN7eF/w50F8rWo9xYf9+jt98AdxHQH8FyZ3/yD1opYQtgf4K0jzI1RQSuwX6K0i+ qaowjp1Dm9BfQfLtX/PF0aMA/RVkrDhW0N+v0TyO7vTmD/orSX4v7mcXwEB/jfjuZQTZQH8F GcvFN4Woob+CNH/JzPxzI9BfQepI5asfAIb+CpK//v4bgoxnAf0VZCgX8xzShP4K0l4q03cA ob+C1HmPUe4JZ5Af9FeS5sOIOTR2B/RXkDpqyLaNM4gT+itIlXd1zKCifKC/FEbTAtpf0JM5 3n9+B/SXwvLB7hqLA6eQ2C3QXwrv9FdjIUF+p7JRoXwE9FeQ9usPpnfOQH+NqON/nkFit0B/ BRlr/cEU4oT+UnhXo1VegNQ4XVugvxQ+GH9UWmNQ44Qtgf5S+MD+VTnp/AFQob+SNH9X5fSL U6G/FGbq/80B9JdCF/9z9glnUC7014gq72aYH+ivFVWCs00P9DcENWKsTaFb6C+FD6q0ijcE 659BSJWlKtOv8bsD+kvhE/9z65ejT2H+oL8kPml/c/dWWdQ6DNBfCp/4/2o4YND/+y2W+x9q Tv8uN/v+mtsB6aW/BhVXPdePu2u8AO435j+W82L0x+K/L/7uC+35crxQxfx/veLFX/ji/l+i 8ljOz+Uhhf7xWmzLs8RuzxKf6knJHxld3LY993JJuIQJgvwEJRBk7yiChzJYwosLD3y9rPDr 4g7rckniUiRRCYY5jS8hLHx3yWfFDilHepTpYgt3WYKMLv4Cl6g0Flf8JCo6X63XUrb1viyu as7Ki2vgeSEdfxyVoc3xQwp3J8WVZGtgcfpylxCnJbE0Xd0t/hqOcoryFGRs8SqIhPkkg9Hl 2IOeRw515OuHPLte4pS3xPdOkJkh1af1F1k4J8Ez77a2FrIsj+UVVKk3M7ZeIsNjC9dV5eI0 74qOhEmtJKMMLMTtXlxJL67cXZ4CaxfoJahZa9qdsGw9+1vjYlidqYvuHuIvy5WKO+VTox+W tM8i8cW7+MO76wtL9omWfTmHpj76Oqj1U/rzNbZcqyqwcK7yfaF6kxXezaFZsgL1Rw0qOTBq 7rZ10nM1Y08UCphE6byso4oLTZO/V9wluRx7e+H14m4ar48lrMvgMK7kgrsyKKrg+Eusvwf7 H2TQCt3dF+5rbNjdPRSUUqA0b6avbcdA0Khmn5aANylBZVz+KFBfWI9Bqx4IJ5YcCbRkizks XhJWEXFHDQRNSFTtrooC/bkOmc91bFp9ZVu7H1Wuveq46xF0V4KS8SeOO3mh/lwmSJD7oMkJ bxN/D/vCvOgvyPK1/+Tv+TH1Fxj34FqcQsLWLe7bxpYsaHjcQMbXVnDD+3RBVcV9K1/hQX2F lRLeLpHkw0bRt6Bh/yk21aFRjw1rcAWx9XP6iW4CX+FxtyO8f0ItLsFVhLpzN3tYukGnxlr5 QNP+pvWGOdSfa5/H1J/TgavooCcc2ItQhiSu6KAnF7Q6S6BLp51AfUGCqLMXtGhepRfLHCqA +Ep77HSSILthzywwaFeFLMH1hM2mPye5KiU0LmGBBTdUWEihSQvN94XAjPtrjvUX2YzgRgku OmomOsM4XcVVf75Ig75e1FeKfn9oDbwifLVeyydqxOPOvNe4s3LeFvna9bkKrVFgqn1deevi O4VRDQdmKjbYYdfMaemhW7yE5/HnCnR0yW3QNYluVG+TlleQSz/Y5eBB1D7rTw86ggIZ3cRO Zay/BZThTkdvJPZGf6UO2lt/fLcf0F95aumv3EE7y09Spj43Dv1VoYf+0vb9TSbVZmQF1X0/ Fh//qf7Iq5+ifk/QByPP0z10kx43SXToZMjjl2u/Jxi8pJyFPPviu6jhZti5Ct0yl4q33+iL XfEg1u82/9LHLD1oLKyjZwLsrj9p9Bd1AKl3P1x8mw5XSG64SS8jDD8yC37z7tKjpNyZwpGl 7537ky8P2Qt9OWGf/prRy7/RkWOBhGc7av5FsS9R1sKisomiIelNrtx3GhTqc008+dXN0/tb 6aGaroNj+/Xv2gvPVJwX+ssAiZAoneftLwAxtfT3dPwBQCue+V8AaMUz/zMAzWCccvH5YQAA AAAAAAAAAAAAAACAe548EPInZPIsysYp3xLTiDU9zXFZqdmTZuI9dWYyJ39Zc/xyzyg9XeSJ lXtWa64qksmdkJM8NdFOmdhoWhFKVQosZ7o6PXuMCkViIkH35Gsi+jwi9aJWrksvMdGuKndL Koiz3NpN02YuSGA0tYLPpTdpt71ZqcMylgPtyfrLWhS0rvpca3rCjadl71g4l1h60twZe0Ll 2mpttkwlc0GWuj9SGzi5P1n6+jaRSE+jETw5UUo1OUzpZSBTE9qFm0kXZVcb/zmBrdZ2y/Se Lkh9j5Q5qiDaLqVnMeM+5Cw5e3xVHax0VexZ/dP0quWrVCWRZmmt0fxzSdhqzVRFBk8X5P+J LP2x1L6S7q2nr9fZ1vTsUdPBShOgUOJL7/8p0jWrOmaUpjb0XBWdSOuJnPrLVUUqbfXHkktQ 99ZTVaGaD5GePTP0SDTP4uhgJRszkV52a8bozYh2TRuANNdfrqXNyN2WIb8jXdrfr3tu4WV2 sFJPla7YrNEbOVvU5BJo1/7m9zTTSz3ZfaCNX86pcp6ikSzjTDKzntKb3zyls3Sb3nj8kT/S TtZfTj/pKLtUc6H9axsVSd1GYYp8TTyTKbh0a5Y+bM6zf1rnMq3YWVv/S76nMdn/QlfjeE1P tOd4OZJvD+3g3WniUIcZB29q9kTqaUie917plYk1zdPIGvufsx8ISa3g7WwUE/2uK82bCEqf f9tzppxYutMma/hxzF4mn2lXpZfYZLv5NzwmBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AF+Pe4/Uu7XnGWvMw4X9LO95UvDlmOffmPjvf9783fbpc39rZvwX8O385fmj9BCG1wMzvEkP PEXL5Phfh4gU5oNcnlfbdPMp1C/GDLpd9kmz8191BH4+RabDOepwFi5NdgAs8OV4/XEmODcf 5hl4/WT6+TfH49m7MFGR3K7tfKZ3M8/ACnsEQs5fqHRpzgCAAFzx+jMmjB2/HPGCTs2Y6Bnm WXCmdrtdZmPdzn+5T3z+QqVLkxORBfwEXn+SuI84iqP5JlVLKkgQ4NEGJ3AxeNwR3J/YNKRN jCgwIS/0F7pljjAEOsKCDpNnd4lor0oU6M/GADrTEOgPvOCF/lgQg8aFYVEjD+J2Hb9KcWP/ bBoC/YEXPNXfMVxlp9fPxJRi/Lrr6OWtQf/PHsGETduVRbRp0P8DL3iuv2Nsa73Ox/hXv+Fh JX6XHQgfo10WHcGOf880GP+CFzzXX/wOnMP/p/pyJjS/2/Xg/3PWTr+Vxfb/TBr4/0A+WfMf EZj/AB+QM/8bkR6RDwDPhzEWsf4FAAAAAAAAAAAAABTk/wHxWWDbCmVuZHN0cmVhbQplbmRv YmoKOSAwIG9iago3NDA3CmVuZG9iagoxMCAwIG9iagpbIC9JbmRleGVkIC9EZXZpY2VSR0Ig MTA5IDEzIDAgUiBdCmVuZG9iagoxMSAwIG9iago8PAovRmlsdGVyIFsgL0ZsYXRlRGVjb2Rl IF0KL1dpZHRoIDEwNgovSGVpZ2h0IDgwCi9Db2xvclNwYWNlIDEwIDAgUgovQml0c1BlckNv bXBvbmVudCA4Ci9MZW5ndGggMTIgMCBSCj4+CnN0cmVhbQp42u2ci08c1RfH/xT/BpPGaIIa tb/Y2pYaU1OjVaPWGGNqrEl9x1TtQ4sY0lbo01KgUKmEYm0phZbn8lweC+zyfrZQCrsLC7PA sizz+8wOzFx2Z/ntsvXXjHJCpzPnnnvune+cc8+5d+6OLG/Q30Kjo6NtbW12u93r9T7qvpiP 2tvbMzIyrl69Ojw8zOX9+/enpqZ8Pp/f719cXFxYWPCt0GKQfAJxiZh2ibAqozFVJaKecM7i CmkcTU+IZrGtkCqRuheuxLCh8O4FAoFo0BsbG7NYLHV1dZOTk0tLSw0NDTMzM6oSNKh9VikQ JL9AKscZJDqgcgB/YmICJS6Xa3x83O12w5ybm/N4POpz4aiKPQgSwvPz81zSE/qAC1CF6lzO BokTBNCAzPT0NK0gc+/ePcSoSyu0rjatPTuI/qi9RQPnVNE6rMmo92jIAYoozW8pSJxQt7W1 lWP0pkvFa9euXb58ubq6+tatWyUlJWlpaefOneMcZl5e3u3bt8GwpaWlqKiI0u7u7vz8fDj1 9fXnz58/c+YMl52dnTdu3LgWpNra2uvXr5eWlmZmZoItVX7++WfqFhQUXLp0CWGeLw/ijz/+ QDNjDlWKi4sdDgcCmEFZWRmNcllVVfXXX3/Rq1OnTjU1NTU2NlZWVtbU1MChM6vA4TxqrNag daDHY8Ju6TNwJSUlJScnHz58+MKFC3AAJzs7mxvv7e3t7+8Hh8LCQnrO3YEhTKqcPn36zz// BAHQABlw464R4yQ3N5dhhAdx6NAhHgQQgR7MmzdvYktWq7UpSAij3Gaz0RbtZmVlffPNN6hK T08/cuQIJ8eOHaMij+O3337LyclBVUVFxSrHXFiQ/f5Hgl6wcWXQwLk8QcKb8Ck8BUfDrTgi o7obpfgRRx696s5YF36HbyLJUZIkBKjOCfaJWkDGwBiTaQW3RQknVJ8PErWQ5xJJihCAQxXV xzkyAjAacIIYR87hUBQ/Vg8LvQ1SaQO9eGgDvXhoA714aAO9eGgDvXhoA71YiWRJzc3UaY75 0CPXfXQdJrm9ePEiuf3IyAhIkr2ThaoTPTWn1Sh6jsYMuYySE645Ui3+C8zOBny+h969KOe5 TLdBj7nS6OgoVcrLy9VpuzrrV6fnKqnzaI3DicrRBDQOmb/KUVctxFqqWpGjrTmE6EFS46h6 NLWannk4MzNzXu/cStMPq3uRHJAZUEdHhzZbATFmMUxhzOq5TFH/jx222+1Xr15lZh1etIHe /6S6urrjx48PDQ2FF5kSvbm5VejNz8vRLGwyrK1rXaW4uDgzM7Ovry+8yJToEXNFuDiPZsBH hoqxE4MeYWJsbCy8yJTouVyy06lfRml766Xa2trU1NR/DnpANzmpX+KPD2OhOBKBT3Z2tvoO KIRMiR7QeTz65ezs32p7jY2NZ86cGRgYCC8yB3ohQRa4CBwakYlF6v/DcOqenp6CgoIHDx6E F5kDPbonggAmkqRfroEe/LjfXzAX++mnn5hiGKk3A3ohhO2J7ylAMlL/Z2aUEBMfNTc3f/fd d1q+p73NVE/Mhx5uK3ruyIhijYYEP+79EuBWUVExuRKnXC4XGWBRUVFnZycAmg89n28VJliF CKZIWKkYX2InZrWgV1paKq2MFaOjo+pb6YaGBkrNh17IuEcCs640OBrCQ3Nyco4ePTo4OKhy ZmZmwPPu3bvgaUrbAzq3W79cI2qE7BmIfQsByOTn5x86dEhDL6TUfOjhtsEX7svU3x/Rc0Mk EYvdSvHQjIwME2csIRTiuePjETEJl4x9GCRkpKWl3b9/P7zIHOiRs4muGhILKCKOGFIIenfv riOBId/LyspioAsvMgd6gCM6TggmjEjkdRqJ6XGIJPYjOnJ05PV6gW7ByLxNg97o6KpLzE8j ANHyPYICZqmFBmYo4q2RGYo2HB0xzz127JiJVwmAS1wgCsmBcUbRwNaIqtieuDgTHdXX16ek pJh4hQqvmZjQL0NmaliF5o9Ax3kkANcVNWpqampra2dFa18hc6CH7Ykhj8RDRA9/FG1vjWUB HkGMG/nIh2/cuIHtmXiVgO6J+ICe6ICdnbpFMdCBs7YgA5KiseHjYnyJgggZubm5165dmxCM P559y4+A8FycTiNGftESbDa9FNwIwRp6nIgeh+FFyqsjEDM1cFM3r6ocp9N5+/btkpISArE5 Zmrcsri0S8gQLxn3tEgKXF1duvNyX0bjVUzU1dWF82pzDdC7fPnyuXPnGAzNsZfA45kbHNT2 Esy73QuDg8t7CQKBeZttbmhoLriPWuE4ncpOg6BSn9O5dO+eP/gLDi7nXC7f9LQ/uDoX/V4C h8Nx/vz5EUbXIE1OTg4NDXV0dIAntmeCfSyzs4Hh4eVL7GtkZKm1NbAy+gQGBwMeTyAoyb+l kZFA8IcbigCuOja2FBypFAGXa2lmZvky6n0s+CaWZuK5BgmeuMThdCruqRFF2vwLz7Va9Ykb J2J8IWTEOO7JSlDqxMBMnLEwjjHQabZBRieC2damz+NAr79fn18QcHt7dcl1bV0jWz558qS5 VwlE9EgexH0RxF/NwJChVIu5GG3cK1R2u/3ChQvauGc+9LjrlhYdPTAR9xKAnnaJjMOhQ4Sk uKjCZaTVmMhEgMjPzzfxLiDQwD01i2IIEv0IuLQpPOj19OjohSxt4cixz9Tw3IMHDzY0NJgV PXDDc0X0xDl7e7uePCPDpYZeyASZpDr29T08Nz09fVRc5DEXenQPTLRO4n0iCNyXdmug19Sk oxeytIWDiy4fHVmt1qSkJBOvUNE9HEebQZDFifdCBNEcGfSQ1AY3qohRg+AizviiIBLCmzdv Jicnm/i9BpjU1emYkLaJMRenFj03JGqIM2LQi9H2mIz8/vvvubm5M0bLC6ZBr7RUn7ECAiFY I0nSYwFRg1RQu52QCTJFRluh1mw5wCyjr69Py5a9Xu/g4KD6VQdzrBKASXOzjh5OZLHopYAp TiiwLnGEFG2P+Buj54YTaXNeXl5hYeFwcPJosVjcbnf0X3VY92cTQpSs/Q0HlbOshPm+xeKf nlY4fv/C6Ohifb1PbWhpydfd7evr8wXrKpzmZl/wYwiKkvn5xeCPepb1eDwLbvdicPfOur/q QJWuri7mbpOTk+rP5GP9qoN2ucZHEsI5IUrC9YhNr+IsLPgtlsXgzgflPqemAg7Hsh4iw8CA 8hfUpnBsNn/wQxCKktnZwNjYsh7+QHJiQl1PiOerDkFvMM/qqKxk/PqbIGJuTY0+9cAZxaSO jEVbiMZzxRlWyBaOuMk06JGTaDEXEDo79aKQTeDgrKHHfVGk4UwQiXFlfm0yDXokeNpLW076 +3VMMCcxeSZj0UIw+R7ZsjhBFt+PxE2mQQ9j0zyX4Ev6pw3yeK74MyjSEi06k/iJeTUuL/p4 3GQa9Hp7dUywH3HcY84rOrK4MQNn7+nRJeH/S9ALeS3b3a2jhwO2temYMCSKi6V37+pWykBX X69LosFowvVPQ4/7JQEWUwjuWpy9iiCEvGLDSjXbw0oLC3UfR2eMcw1Tosf9MtqL6au4MV4d 9zRsiRHiuMcUWAsNjHtdXaskceR/PHrcL6OZaHvgICYe4lIzAVe0KBxZe/WDj7e3r/Jco9cT caHX0kIGvjyDJn6RXpIjcVQNoLVVYdKfqanlnQyU0geqwKcKkvxxyf3iXNwmDoJnIUxv4fOn JlrqCY6GczH9VBsiJqqboOBziQAWQmlVlaIQSbJftWkGNJIT4iYCNttyQ+ovMugk2oaGFAGG RBDjHEnUVlQoAkiq1dGG+dH/OH7axkRj+asOstyambn48cfygQPy1q1yYqL84YfK+QcfyCUl 8i+/yK++Kh8+LJ84IR8/LmdlyRkZ8sGD8r598v798rffyp9/Ln//vZyWJn/xhcIsKJCzs+XM TPmrr+RPPlGYu3crYklJcnKy/Omn8tmz8pUr8pEj8mefyTSKQpgXLy5rTkmRc3PlS5cUnW+/ Laemynv3ygkJCpOHS5Wvv5bz8uTqarmoSK6tVTTTBI2+9568Z48iic5Tp+Rdu+SPPlLaInPO yZHff1/pPBwwp/NPPKFUjMPd3G53YWGh8lUHWW48enT62WelZ56RNm+Wnn9ePXq3bJlOSHBu 3ep94w1p2zbpueekN9+Udu2SXnvNu23bVEKCa9Mm786dStGLL0qvvipt3y4lJHhfecW1devU Cy940fbUU9KTT0pPPy0lJip/u3crMi+9JL3+urRvn8L5z3+kHTu8O3ZMb9/u3LnT+8470ltv Se++K738slK6ebN3+/apzZtdjz/u3b9fKiuT9uxRmkPPl196f/xxLDHRuWWLlw6npytMNG/a RGe8x4+P7N7tSUxU+nDggHTokNLDHTskepuZ6eH42GPS3r3S5KTk9UpB8nq9TPkNd0mtgZ66 Vt/T1tZ+5469rMze0GBvbLTX19stFs7bSkqsxcUKh8vSUoVfVYWYo6rKVlbWVFLiqKlRiior 7ZzwV1vrqKtrqqiwUYRkRYVSRC1VZ12dcsKxulo5oS2qoLampq2szEoHkIHJHzopslod5eW2 ysomWrRa7a2t9qamZQ0U2WyNZWUt5eWO2lq7zWZvaVGOaEDSZmuwWNoQQycdaG5W+OXldA+x ZjicU7qaGPwNdwus7bnK+cMaQ/99NDs729PTY/jxW5idnZ2G6zZ9fX08rPCihYUFh8PhjLD6 3dbWZlg0NjZGLcOGurq6rFar4crbxMREj1H64fP5bDaby+jdGXfU29sbvqcCPh3o7u7mGBN6 Ho+Htgw/jUg3BgYGDHteX19vEVd3BfSqqqqaiGtGPU9JSQHz8CLuKCMjw220dnTr1q3MzMx5 ox/x0W3D3+/QUKSX1wxx6rdqw++UB3Hnzp3i4uIo10hVmpubwyTmIuyNUb9RGc5vaWnpFfeH rJDf70eb4esn9PAsDCFiDAFVn9FbflR1dHQY3tH4+LghqghjsVNGvyxQb9awFnfKszDEfIM2 6F9ODMuEG1ye4/DwMOMP8YLAROLEJQkD+SdHzuFXVlZSRA4mPdT1TPNSQ0MDw/Kvv/564sSJ 1NTUuiBlZ2efPn36hx9+yMrKunLlytmzZ69fv65+OfnkyZMEwb/pY7+mI8Z5TIshHaMi7mBg mBkAEheam5sZz8GKI2JYaXV1dWNjI4a6YXuRSN0/vIZA8IXtI0jn/wsJ9FgHCmVuZHN0cmVh bQplbmRvYmoKMTIgMCBvYmoKMzg5OAplbmRvYmoKMTMgMCBvYmoKPDwKL0xlbmd0aCAxNCAw IFIKPj4Kc3RyZWFtCv///wAAAKCgoP8AAADAAACA/8AA/wDu7sBAAO7uACAgwP/AIACAQKCA /4BAAP+A/wDAYADAwABggMBggACAAED/gDBggIBgAEBAQECAAAAAgIBgEIBgYIBggAAAwAAA /wBgAOOwwEDAgGCgwGDAAGDAoIAAAIAAgGAggGBgYCAgICBAQCBAgGCAIGCAYGCAgICAQCCA IICAgKCgoKDQ4MAgIACAgMBgAIDA4MBgwMCAAMCAYP9AAP9AQIDA//+AYP+AgMCgAMDAwMD/ wP8AAP8A//+AoMDAoP9gYAD/AP+AAP+gAIDg4KDg4KD/IMAAAMAAwKAgIKAg/4AgAIAgIIBA IIBAgIBgwIBg/4CAAMDAAP+AQP+gQP+gYP+gcP/AwP//AP//gP//wJ+fnx8fHz8/P7+/v39/ f19fX9/f36enp4+Pj4uLixsbGwplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjMzMAplbmRv YmoKMTUgMCBvYmoKPDwKPj4KZW5kb2JqCjE2IDAgb2JqCjMzMAplbmRvYmoKMTcgMCBvYmoK PDwKL1RpdGxlIChjb25udGltZS5wZGYpCi9DcmVhdGlvbkRhdGUgKEQ6MjAxNDAyMjcxMDUw MDApCi9Nb2REYXRlIChEOjIwMTQwMjI3MTA1MDAwKQovUHJvZHVjZXIgKEltYWdlTWFnaWNr IDYuNS40LTcgMjAxNC0wMi0xMCBRMTYgT3Blbk1QIGh0dHA6Ly93d3cuaW1hZ2VtYWdpY2su b3JnKQo+PgplbmRvYmoKeHJlZgowIDE4CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAx MCAwMDAwMCBuIAowMDAwMDAwMDU5IDAwMDAwIG4gCjAwMDAwMDAxMTggMDAwMDAgbiAKMDAw MDAwMDMwMCAwMDAwMCBuIAowMDAwMDAwMzgzIDAwMDAwIG4gCjAwMDAwMDA0MDEgMDAwMDAg biAKMDAwMDAwMDQzOSAwMDAwMCBuIAowMDAwMDAwNDYwIDAwMDAwIG4gCjAwMDAwMDgwNDkg MDAwMDAgbiAKMDAwMDAwODA2OSAwMDAwMCBuIAowMDAwMDA4MTIwIDAwMDAwIG4gCjAwMDAw MTIxNTkgMDAwMDAgbiAKMDAwMDAxMjE4MCAwMDAwMCBuIAowMDAwMDEyNTY1IDAwMDAwIG4g CjAwMDAwMTI1ODUgMDAwMDAgbiAKMDAwMDAxMjYwNyAwMDAwMCBuIAowMDAwMDEyNjI3IDAw MDAwIG4gCnRyYWlsZXIKPDwKL1NpemUgMTgKL0luZm8gMTcgMCBSCi9Sb290IDEgMCBSCj4+ CnN0YXJ0eHJlZgoxMjgxMwolJUVPRgo= --------------080207040208060206070203 Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0"; name="tw-lock.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="tw-lock.patch" diff --git a/sys/netinet/tcp_timer.c b/sys/netinet/tcp_timer.c index bde7503..b45a9ea 100644 --- a/sys/netinet/tcp_timer.c +++ b/sys/netinet/tcp_timer.c @@ -144,9 +144,7 @@ tcp_slowtimo(void) VNET_LIST_RLOCK_NOSLEEP(); VNET_FOREACH(vnet_iter) { CURVNET_SET(vnet_iter); - INP_INFO_WLOCK(&V_tcbinfo); - (void) tcp_tw_2msl_scan(0); - INP_INFO_WUNLOCK(&V_tcbinfo); + tcp_tw_2msl_scan(); CURVNET_RESTORE(); } VNET_LIST_RUNLOCK_NOSLEEP(); diff --git a/sys/netinet/tcp_timer.h b/sys/netinet/tcp_timer.h index 3115fb3..c04723a 100644 --- a/sys/netinet/tcp_timer.h +++ b/sys/netinet/tcp_timer.h @@ -178,7 +178,8 @@ extern int tcp_fast_finwait2_recycle; void tcp_timer_init(void); void tcp_timer_2msl(void *xtp); struct tcptw * - tcp_tw_2msl_scan(int _reuse); /* XXX temporary */ + tcp_tw_2msl_reuse(void); /* XXX temporary? */ +void tcp_tw_2msl_scan(void); void tcp_timer_keep(void *xtp); void tcp_timer_persist(void *xtp); void tcp_timer_rexmt(void *xtp); diff --git a/sys/netinet/tcp_timewait.c b/sys/netinet/tcp_timewait.c index 7e6128b..0230b88 100644 --- a/sys/netinet/tcp_timewait.c +++ b/sys/netinet/tcp_timewait.c @@ -49,6 +49,7 @@ __FBSDID("$FreeBSD$"); #include <sys/socketvar.h> #include <sys/protosw.h> #include <sys/random.h> +#include <sys/refcount.h> #include <vm/uma.h> @@ -98,13 +99,59 @@ static int maxtcptw; * The timed wait queue contains references to each of the TCP sessions * currently in the TIME_WAIT state. The queue pointers, including the * queue pointers in each tcptw structure, are protected using the global - * tcbinfo lock, which must be held over queue iteration and modification. + * timewait lock, which must be held over queue iteration and modification. */ static VNET_DEFINE(TAILQ_HEAD(, tcptw), twq_2msl); #define V_twq_2msl VNET(twq_2msl) -static void tcp_tw_2msl_reset(struct tcptw *, int); -static void tcp_tw_2msl_stop(struct tcptw *); +/* Global timewait lock */ +static VNET_DEFINE(struct rwlock, tw_lock); +#define V_tw_lock VNET(tw_lock) + +#define TW_LOCK_INIT(tw, d) rw_init_flags(&(tw), (d), 0) +#define TW_LOCK_DESTROY(tw) rw_destroy(&(tw)) +#define TW_RLOCK(tw) rw_rlock(&(tw)) +#define TW_WLOCK(tw) rw_wlock(&(tw)) +#define TW_RUNLOCK(tw) rw_runlock(&(tw)) +#define TW_WUNLOCK(tw) rw_wunlock(&(tw)) +#define TW_LOCK_ASSERT(tw) rw_assert(&(tw), RA_LOCKED) +#define TW_RLOCK_ASSERT(tw) rw_assert(&(tw), RA_RLOCKED) +#define TW_WLOCK_ASSERT(tw) rw_assert(&(tw), RA_WLOCKED) +#define TW_UNLOCK_ASSERT(tw) rw_assert(&(tw), RA_UNLOCKED) + +/* + * tw_pcbref() bumps the reference count on an tw in order to maintain + * stability of an tw pointer despite the tw lock being released. + */ +static void +tw_pcbref(struct tcptw *tw) +{ + KASSERT(tw->tw_refcount > 0, ("%s: refcount 0", __func__)); + refcount_acquire(&tw->tw_refcount); +} + +/* + * Drop a refcount on an tw elevated using tw_pcbref(). If it is + * valid, we return with the tw lock held. + */ +static int +tw_pcbrele(struct tcptw *tw) +{ + TW_WLOCK_ASSERT(V_tw_lock); + KASSERT(tw->tw_refcount > 0, ("%s: refcount 0", __func__)); + + if (!refcount_release(&tw->tw_refcount)) { + TW_WUNLOCK(V_tw_lock); + return (0); + } + + uma_zfree(V_tcptw_zone, tw); + TW_WUNLOCK(V_tw_lock); + return (1); +} + +static void tcp_tw_2msl_reset(struct tcptw *, int ream); +static void tcp_tw_2msl_stop(struct tcptw *, int reuse); static int tcptw_auto_size(void) @@ -171,6 +218,7 @@ tcp_tw_init(void) else uma_zone_set_max(V_tcptw_zone, maxtcptw); TAILQ_INIT(&V_twq_2msl); + TW_LOCK_INIT(V_tw_lock, "tcptw"); } #ifdef VIMAGE @@ -179,11 +227,14 @@ tcp_tw_destroy(void) { struct tcptw *tw; - INP_INFO_WLOCK(&V_tcbinfo); - while((tw = TAILQ_FIRST(&V_twq_2msl)) != NULL) - tcp_twclose(tw, 0); - INP_INFO_WUNLOCK(&V_tcbinfo); + TW_WLOCK(V_tw_lock); + while((tw = TAILQ_FIRST(&V_twq_2msl)) != NULL) { + tcp_twclose(tw, 0, 1); + TW_WLOCK(V_tw_lock); + } + TW_WUNLOCK(V_tw_lock); + TW_LOCK_DESTROY(V_tw_lock); uma_zdestroy(V_tcptw_zone); } #endif @@ -204,7 +255,7 @@ tcp_twstart(struct tcpcb *tp) int isipv6 = inp->inp_inc.inc_flags & INC_ISIPV6; #endif - INP_INFO_WLOCK_ASSERT(&V_tcbinfo); /* tcp_tw_2msl_reset(). */ + INP_INFO_WLOCK_ASSERT(&V_tcbinfo); INP_WLOCK_ASSERT(inp); if (V_nolocaltimewait) { @@ -229,7 +280,7 @@ tcp_twstart(struct tcpcb *tp) tw = uma_zalloc(V_tcptw_zone, M_NOWAIT); if (tw == NULL) { - tw = tcp_tw_2msl_scan(1); + tw = tcp_tw_2msl_reuse(); if (tw == NULL) { tp = tcp_close(tp); if (tp != NULL) @@ -238,6 +289,7 @@ tcp_twstart(struct tcpcb *tp) } } tw->tw_inpcb = inp; + refcount_init(&tw->tw_refcount, 1); /* * Recover last window size sent. @@ -356,7 +408,6 @@ tcp_twcheck(struct inpcb *inp, struct tcpopt *to, struct tcphdr *th, int thflags; tcp_seq seq; - /* tcbinfo lock required for tcp_twclose(), tcp_tw_2msl_reset(). */ INP_INFO_WLOCK_ASSERT(&V_tcbinfo); INP_WLOCK_ASSERT(inp); @@ -458,11 +509,11 @@ tcp_twclose(struct tcptw *tw, int reuse) inp = tw->tw_inpcb; KASSERT((inp->inp_flags & INP_TIMEWAIT), ("tcp_twclose: !timewait")); KASSERT(intotw(inp) == tw, ("tcp_twclose: inp_ppcb != tw")); - INP_INFO_WLOCK_ASSERT(&V_tcbinfo); /* tcp_tw_2msl_stop(). */ + INP_INFO_WLOCK_ASSERT(&V_tcbinfo); INP_WLOCK_ASSERT(inp); tw->tw_inpcb = NULL; - tcp_tw_2msl_stop(tw); + tcp_tw_2msl_stop(tw, reuse); inp->inp_ppcb = NULL; in_pcbdrop(inp); @@ -494,11 +545,6 @@ tcp_twclose(struct tcptw *tw, int reuse) } else in_pcbfree(inp); TCPSTAT_INC(tcps_closed); - crfree(tw->tw_cred); - tw->tw_cred = NULL; - if (reuse) - return; - uma_zfree(V_tcptw_zone, tw); } int @@ -616,34 +662,83 @@ tcp_tw_2msl_reset(struct tcptw *tw, int rearm) INP_INFO_WLOCK_ASSERT(&V_tcbinfo); INP_WLOCK_ASSERT(tw->tw_inpcb); + + TW_WLOCK(V_tw_lock); if (rearm) TAILQ_REMOVE(&V_twq_2msl, tw, tw_2msl); tw->tw_time = ticks + 2 * tcp_msl; TAILQ_INSERT_TAIL(&V_twq_2msl, tw, tw_2msl); + TW_WUNLOCK(V_tw_lock); } static void -tcp_tw_2msl_stop(struct tcptw *tw) +tcp_tw_2msl_stop(struct tcptw *tw, int reuse) { INP_INFO_WLOCK_ASSERT(&V_tcbinfo); + + TW_WLOCK(V_tw_lock); TAILQ_REMOVE(&V_twq_2msl, tw, tw_2msl); + crfree(tw->tw_cred); + tw->tw_cred = NULL; + + if (!reuse) { + tw_pcbrele(tw); + return; + } + + TW_WUNLOCK(V_tw_lock); } struct tcptw * -tcp_tw_2msl_scan(int reuse) +tcp_tw_2msl_reuse(void) { - struct tcptw *tw; INP_INFO_WLOCK_ASSERT(&V_tcbinfo); + + struct tcptw *tw; + + TW_WLOCK(V_tw_lock); + tw = TAILQ_FIRST(&V_twq_2msl); + if (tw == NULL) { + TW_WUNLOCK(V_tw_lock); + return NULL; + } + TW_WUNLOCK(V_tw_lock); + + INP_WLOCK(tw->tw_inpcb); + tcp_twclose(tw, 1); + + return (tw); +} + +void +tcp_tw_2msl_scan(void) +{ + + struct tcptw *tw; for (;;) { + TW_RLOCK(V_tw_lock); tw = TAILQ_FIRST(&V_twq_2msl); - if (tw == NULL || (!reuse && (tw->tw_time - ticks) > 0)) + if (tw == NULL || ((tw->tw_time - ticks) > 0)) { + TW_RUNLOCK(V_tw_lock); break; + } + tw_pcbref(tw); + TW_RUNLOCK(V_tw_lock); + + /* Close timewait state */ + INP_INFO_WLOCK(&V_tcbinfo); + + TW_WLOCK(V_tw_lock); + if(tw_pcbrele(tw)) + continue; + + KASSERT(tw->tw_inpcb != NULL, + ("%s: tw->tw_inpcb == NULL", __func__)); + INP_WLOCK(tw->tw_inpcb); - tcp_twclose(tw, reuse); - if (reuse) - return (tw); + tcp_twclose(tw, 0); + INP_INFO_WUNLOCK(&V_tcbinfo); } - return (NULL); } diff --git a/sys/netinet/tcp_var.h b/sys/netinet/tcp_var.h index e3197e5..b44672d 100644 --- a/sys/netinet/tcp_var.h +++ b/sys/netinet/tcp_var.h @@ -355,6 +355,7 @@ struct tcptw { TAILQ_ENTRY(tcptw) tw_2msl; void *tw_pspare; /* TCP_SIGNATURE */ u_int *tw_spare; /* TCP_SIGNATURE */ + u_int tw_refcount; /* refcount */ }; #define intotcpcb(ip) ((struct tcpcb *)(ip)->inp_ppcb) --------------080207040208060206070203 Content-Type: application/pdf; name="conntime-bsd10.0-patched.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="conntime-bsd10.0-patched.pdf" JVBERi0xLjMgCjEgMCBvYmoKPDwKL1BhZ2VzIDIgMCBSCi9UeXBlIC9DYXRhbG9nCj4+CmVu ZG9iagoyIDAgb2JqCjw8Ci9UeXBlIC9QYWdlcwovS2lkcyBbIDMgMCBSIF0KL0NvdW50IDEK Pj4KZW5kb2JqCjMgMCBvYmoKPDwKL1R5cGUgL1BhZ2UKL1BhcmVudCAyIDAgUgovUmVzb3Vy Y2VzIDw8Ci9YT2JqZWN0IDw8IC9JbTAgOCAwIFIgPj4KL1Byb2NTZXQgNiAwIFIgPj4KL01l ZGlhQm94IFswIDAgNjQwIDQ4MF0KL0Nyb3BCb3ggWzAgMCA2NDAgNDgwXQovQ29udGVudHMg NCAwIFIKL1RodW1iIDExIDAgUgo+PgplbmRvYmoKNCAwIG9iago8PAovTGVuZ3RoIDUgMCBS Cj4+CnN0cmVhbQpxCjY0MCAwIDAgNDgwIDAgMCBjbQovSW0wIERvClEKZW5kc3RyZWFtCmVu ZG9iago1IDAgb2JqCjMxCmVuZG9iago2IDAgb2JqClsgL1BERiAvVGV4dCAvSW1hZ2VJIF0K ZW5kb2JqCjcgMCBvYmoKPDwKPj4KZW5kb2JqCjggMCBvYmoKPDwKL1R5cGUgL1hPYmplY3QK L1N1YnR5cGUgL0ltYWdlCi9OYW1lIC9JbTAKL0ZpbHRlciBbIC9GbGF0ZURlY29kZSBdCi9X aWR0aCA2NDAKL0hlaWdodCA0ODAKL0NvbG9yU3BhY2UgMTAgMCBSCi9CaXRzUGVyQ29tcG9u ZW50IDgKL0xlbmd0aCA5IDAgUgo+PgpzdHJlYW0KeNrtne2aoyoaAGG8VBBld+//9wKKgpqk wSTv6TlVz0zaxKCIJV8SVAoAAAAAAAAAAAAAAAAWvI7MYclYPTrp6MC/DKNdwMeFyc3aS8cH /l1Mdl2wc34B+BrzapzXRhU2AnwFO1ptg3pOx7qf0dLxgX8XQT43hbzPpaqfoQIIXyU2PdRs 8Q/kCNpdlr+pb+bPn7DECy/p5c3mebP6d9n+oDYIn8Ul7UZ73f+Cf/BhxtD+mGPZe9X/jH/w Yfycb7sZq+3h/hv+gST4B5LgH0jyNf+MXjDPVttpeTeFisLsTx+v+KUjsxrMU7+zl6N8/Oj3 3V33gsbPzXgMN40hDqNRdeh1l9WmdCauD6FSoMuDaKCId2Q0vRv6B/JF/4xbx+E8Xm3SMLHU Ygpnz+ePJ1sq4a0+Deap303zg338yL/jCTZBnRC1WWcN6ghUm4qRTVEOWwnrwxem/dh6Bawj 6/+qIuuL/vkfrJ5ifJaO8tRTtH7sdJn5pNNfdyZV7/wzuZ7HJn1ubP3R6o236+dzFYHjprYj 2f6Ux3Y/6U7Z86/m6/6FzEBbv7TFl57xtLCuNoV/xuyh5nHfzhQ/rDvT63fTmL7n8h5CaZ5K ylQuTuvuYunp9/0vXxsXa8pKgg/6+TF8105+MdHZKgLpTZFBr1FeLwY/+/3YfJEA1W6d9ov5 uWBPq+IBxK2keOc3ap725XIj9lcWywL+WWdiZjCFksptC8tqtyRrLPCOofJ2/HI+65uJ9btU 0wqVrmnp9pxjaR7zrlgyLrszJg7KSPlZFZFZp/2NRRYz22hXqAKE78xJcGuqCMQ7S7aooS1R 9todD33N/5YEqHdrS//yKq/npdCO8d7ehLptsaJKxN8o4LfbHyHzSsmUkjLmauvCunrJwcL1 rUMaq9I/X27KHwZTVO/8akZUOZq0tgEWb9PuxiUW6ft1RFKlsyopg8xJu62qF3LXKgLh32gP scsxqj5Zc8QlAU673f3Lq5LhuRDY3ji7Lx820t3AEeTb7Q+fT89qV17439LQ0GsVLmYJ2vT5 lze5Z4l+ac1su8v1vCIb2yJyGCMUZU5nNxbuS7nrjv6NVc5z5d/2C5y8/rzbc4x8KGRdDrG9 CW2rvHzYyK9EoPxdTs+Czwv/O9XRU3U/h4qfr90a6lX5W8qYvhqrf5t/RSyW7WwRqUOpvOX1 7MZR5DEW43yIQChPrT0d6Fb+5kZ83fQ57/YcI+Vit0/u4clvYm6+Lh828pLhW+e6ATH/lt6Y baGso6/Vr1xkRuJH6ft5xZP2xyH/C7Ulv21sWeddcba3iFzmfy6f3VipjCVo7uAr2h/OlV0r h/ZHWHVovy7+HXfrTjFKX7a6SLr4plg+bMT/xkxQyL+liWnmbaGso09L3jHZokg9lXCP+18W M5bK+rhWo1L1zW31v7GobeWIpFbLfKz/hY2l7cXafWocu1RNCCe97H+Z9bG2l/tf5rW4PibE abduaUcUMTJ2M9Jvb1xMlLxiT8T1WH8fUv4tLbZ5W1g7mlNCehu7bqel/nfuf86F59rj7JcM pOh/rtu/Xo8utTDDeZxc2dwuWptz3sSp/RsMG+0Uws/zdhFUEVj6X+bTgZb9z2f/qt1OOtb2 UiO7iFH8xMQjj/HOb+ZJ7SsOiUj+94RjSyL2WE37QnWPKo7YWW6nLR8fWnZ7L5k7vUtbXPr/ 8i05k27mRXmiXtPe3biakCMSv19kTnlj2vkxFOFj3Jjz5wism3LHA126HY269q/e7bRUU0dX xShW89JdyBjv9U06yG1FlYi/sfvlrxx/4He5uqjvf4yb13U/yzv59/4o52/0L93/vXFKDznJ HPInV90Afjv493dh3Y1TerrB6uYxFKVr/9tHwD8ACfAPJME/kAT/QBL8A0nwDyTBP5AE/0AS /ANJ8A8kwT+QBP9AEvwDSfAPJME/kAT/QBL8A0nwDyTBP5AE/0AS/ANJ8A8kwT+QBP9AEvwD SfAPJME/kAT/QBL8A0nwDyTBP5AE/0AS/ANJ8A8kwT+QBP9AEvwDSfAPJME/kAT/QBL8A0nw DyTBP5AE/0AS/ANJ8A8kwT+QBP9AEvwDSfAPJME/+AJG+/hq9ejqFfgHn8fb6J/Rk5uTiDv4 B59nTv7ZWa0vO/gHH8fZWP56bcLyZKtV+Acfx5ron9Ox7mdq4/APPs00qsW/1AapK4D4Bx/G h3wP/0CKcVbqcfkb+fMnLPHCS3p5t396hfYHSOACk3aO/heQwtD/DIJs998s99/gHwT+gST4 B5LgH0iCf/AOvPdd4fAPbmPG1ME8mvag+Ac3iaOaJ2PMNGrbbCD+wT3GItszdmwMjX9wD/Pk 3WvwDyTBP3gTXQ1g/IP7TEaZ0P7tMBD/4DazNspacxha9SPwD25jjXJBQWPbg+If3Ea7kAWq 4GBHUOm4w+/Hzs6Oyo+tnX8K/+ANOK1DFmi1aw+Kf3Af7+Lw+p4OGPyDe/iC9tD4B/fQBR2h paMPv5w48kWP4XXWjL8CCcal43mm/QsSrA1f+v9ABDulPxP3P0CCSU/Ou4n6H8gwxcZv++B7 hX/wHpzruPmh8A9kwT94A9z/ADmc5f4HyDFas9AeFP/gNj0Dr3JQ6bjD76er52UB/+A2zhra HyAG469AEmNof4AsfdP/4R+8AWO5/wtimDT+eWT8C4iwTrwxM/4PJGD8M0jC+GeQJI9/ntqD 4h/cJ41/7tEP/+AtMP4Z5DAh63Njj4H4B7cxelTK0/8HMtD+BUno/wNJyP9AEvr/QBT6/0AW +v9Aks7HT+MfvIH4A3Q3dzz+CP/gPkbPRjtD+wNEiP0v2tH/AjLE/ufwn/5nEIH8DyQJNT+n jWH8AchA/zPIQv8zCNPVA41/cJ/JhDqgHpn/CiSIT36z1tiOGyD4B7exRoX2rzL0v4AE2oUs kPHPIISdnR2VHx88/9KNeh0hbaw+/EoO/+A2TuuQBdoH05D74NxSN4z91LOuWin4B/fxLkhl HjR/0205E0VLEtatFPyDD+NjtmhCtufTDbr6LjH+wVt4/gwQF3M9pxcTq2DS8Ya/g6f+aW1d 9C+W0KaqAOIfvIWn/rn0cGr8g4/x6hlck74ufyN//oQlXnhJL+82cxkZE7I92h/wfvz29N8H v8FcpiWPr/S/wNsJherz52/F/mcXhyjQ/wzvx/hXz99yo15vuxmbGsIF+AeS4B/cwxe0h8Y/ uIcu6AgtHX345ZiC9tD4B/eg/AVJKH9BEspf+L3gH9zj5f23p+Af3OPl/bfnoaWjD7+c1/ff noF/8AYof0GOOP045S9IYS3lL8jxauz9s6DScYffz9iR8a3gH9zGWUP7A8Sg/QGS0P4ASWh/ gCS0P0AS2h8gCeMPQBLGH4AY7sm71+Af3MPOe7XPz62PYMA/uMmkx8k578w0tj+CEP/gLn5a 7n/Yqb0BjH/wBrxzjucPwq8D/0AS/ANJ8A8kwT+QBP/gHfSMPYjgH9wnDoB2c8fjz/EP7mP0 bLQz7Xc/8A/eQHy6tHaHJ8v8DPyD28Tx9+G/Y/wfSED+B5LE5xppYzTjT0GEKY1/6Wh+4B+8 Bef6foOJf/AGmP8P5GD+DZCE+TdAEubfAEmYfwMkYf4NkIT2B0hC+wMkof0BktD+AElof4Ak zP8HkjD/H4jB839BEp7/C5Lo/1D+ghw3Ov8U/sFd8A8k0cbT/gAxdEFHaOnowy9HT4b2B4hB /Q8kwT+QBP/g94J/8HEmq22anMNYPdbZJf7Bp5m1cVOcHSbOUzTrqpcQ/+DD+DQv1qTjowrV +rKBf/BhfHpAptF+FbGaJRD/4B28vPc2W+VSU9lUyuEf3Of1/PdxcnKn14yw+Bz/4Dav5783 elT4B5/h5fzPU9TvuvyN/PkTlnjhJb20+/dq/vtJp6KZ9gd8hBf535QLZvpf4BM8n//e69G5 ND80/c/wEZ7Ofz+tg1N9uv9muf8G74f570GYriew4h/cZzKhbqdHfn8EEsyh5WGtsR0PAMY/ uI01KrR/Q+uiPSj+wW20C1mg4vmrIIOdnR2VH8f2oPgHt3FahyzQ9vwSCf/gPt7FzuWeDhj8 A0nwD94A8w+BHDz/CCQZef4RCMLzj0ASy/OPQBCefwSSMP8pSML8pyBN1+hT/IN3YGIHYFcr BP/gNkaPofAdNeUvSLAOfJ4ZfwoSrP3PjD8FEZbJdZ/M//IY/IPbTHpy3k1P5r96CP7BfZ7O f/AU/IN3wPwHIAr9zyCGG0PxO/bkgPgHt8n9z/z+DSTI/c/8/hckoP8ZJKH/GSRxek79z6Z9 CDT+wW10QWtQ6bjD78cUNAbFP3gTzL8LQjD/LgjC/LsgCfPvgiTMvwuSMP8uSML8uyAK8++C LD1zD0XwD+4TJ0B1c0f3C/7BfYyejXaG37+BCC+ef/4M/IPbxIZv+E//H4hA/geShJqf08Yw /xXIwPwHIAvzH4AYa9+z4/4vfJ9Z6yien5n/Hr7PrKc48tRoPTP+Gb5O7HxxetK2qwKIf3CP 2Pns+xq/Cv/gLmnUX69++Ac3WfzrfQIm/sE98A8kSbO+aMPzL0GEG5O/KPyDu9yY/EXhH8iC fyAJ/oEk+Aefx+vUNDb29JAG/IOP423yLw6TnnXdSYN/8GmMXvxL87MdJmnDP/gwIdsz0T+f fh9y+JES/sGH8V4l/1y6SWdq4/APPs/qn8+LO/gHnwf/QJIn5W/kz5+wxAsv6eVT/tH+ACEM /S8giKH/GQQx2/03y/03+AeBfyAJ/oEk+AeS4B9Ign8gCf6BJPgHkuAfSIJ/IAn+gST4B5Lg H0iCfyAJ/oEk+AeS4B9Ign8gCf6BJPgHkuAfSIJ/IAn+gST4B5LgH0iCfyAJ/oEk+AeS4B9I gn8gCf6BJPgHkuAfSIJ/IAn+gST4B5LgH0iCfyAJ/oEk+AeS4B9Ign8gCf6BJPgHkuAfSIJ/ IAn+gST4B5LgH0iCfyAJ/oEk+AeS4B9Ign8gCf6BJPgHkuAfSIJ/IAn+gST4B5LgH0iCfyAJ /l0ybC/wUX6zf0PDp02b6NzYnaP4t7r+z/bv6VkZhudBhstv3z/Rw5OtDC9D/vS7P9n40LSV f2Se/hH/hvYPHunybPtDFWh9V3oWloeHeyjD19IOj4P88Mj7z/Ie420bh6OoU+h17jk8X5G3 IiTmz/2rkna4WjcUSTdcHlPpxsVGh30Le+qW2xmWk7GsHYZiY9Wu960P6/eLbQ2l3EMV723n +VsPr4lLz4ZhO53DvnAOsEWuCFR+b6j2VUajPLrDhVYk0VXcqr0P1SZEKwD6uOsXV/Crg8vZ zlAmcFYkn9Q9HYZ9RZmmqzeLQMVZXRNsWE3NaT3UJ3vdx/q1Ycjv1+2uL5vKw763wtv6rJZh huowNrmHC4/3DRUx3I9I5cjs6bR+YUucYUuhIp22WKxpUR36IQWLYyvSWRWXenk9PC12PuLf o0p8lUfsmdEwFB/VEc5KbOdi2E7QlmvtVpQpvq7b2XKFKrcbNh2HMsSWWWVn9lOy67fpMJTG 5fjssao/LwzNkucYqPJddR3th1ftKce8OvyDQZst+/W3pef27U0/tYu1X+bFR+Xh7y5vEd8i syfshzFWj+7oXyVQkXNtCl4WXNvFXBo7lAe2+1QaWK/aLumhPBtbipYfDMWHpRl7prQHVKdt HDTbPy0vmf0klfHcbVf1MRXXTnHNDNWprzZSJEIRkzKnVsdEPLIfa+HhLnld76h3VCb6nnj7 lvdi5WP66cnN2h/825OmTMiikCsymCqj2jKwfKi1hfmoC2VUuenyDFydzKHeaw6/Xxt1rMsz WZ3YvUwr03kvgCuvtiuryJlqb4sMtv74tLxFodh9UbgWKyoNywB7rl9mnaXU9ekY6u0eolRf 5VU6V/v+FHbOLwf/Dil5GflH764uVnV4f7TxEYdi+PztgwH1mTtmr6reiLrayqngP5TQxYXw 3LOrDOuUXBfGKlV/eE46VUfquHjM3OvjOBc5daiLSH8Mr014nezBv8cpdvbguny9Svdz2j/7 9vONHQQ5xUGdvpaztrNhVwXs1UmoI3fKUC6vsMM5V9WOzvZeC3tQXe3V2cNZ2Ld6Vdg/itQx HSvtU5p9rEfY6Vj3M/X29WVsy/S4WnPIYa5SVFXpf/ze8Uou022raoZlXW3vUdCLYmqoKmRD mdEctlJlFYcibf+CUueDPi1UllSl61G/vaF0TJgHyp7T8To/raN1rDLVn5UF+ZZ+n/TPJ/+q CqCuI3x9aVYN1CdXfnlC9yJyXamr071fb6cEKz/WZ11qUa6SXleF7nXZeIr6fiGWfTMPKgB5 UR+PojyE56l0dWGfFg+Xxt5EvCgWzsHTyT3s5vEB5YT9un8vKmXPz9vllXXOSirTDq2v9WI8 Ka6eqKPKvOL6iiiawXUlUA1PwhUnWOUsU1Uhy0PR24pCu9xLVYYcii9d1U+KvpJT8lbF7rke sP05diKcj1IVCa2UOlcs40dfLn8Baj7l32X7A+BbXPW/AHyLq/5ngK9hrLbu/mYAAAAAAAAA AAAAAJ5z8YOQH+Gb76JMVtupMYwb28Msh9UaPZ9uvLfemeyJX9c9fj93pF5M8saTu57WXiua 6b0h521roFkbN+m2JPQhFUzP7er26BntAo2BnJ6bj0nF/bjWgxptTL3GQHM4uVNTQqzp9r3b tJ0DEoxuPcHr0Ju2yz6N1DEdw4HmZv+6BgWNY9zX2B5wsm3RWwbONaaeT1fG3HBy82n92jCV zgFZ4fpoLeD8fDH09WUg1x4m4mxzoJbTtJFSrwPfGjAP3Gw6qDza+McB8mn93jC9ywGpr/G+ xwoV86X2KHZch9Y0R8+OoYLVbsXcVT9tP7V29CEl2nLanGn+OCXyae20ooPLAfk/oss/01pX irX19vE609gePZ0qWG0CuiBfe/0v0O5sqJhp3VrQ25B0rq0msvrXa0Ur3/XPNKdgrK23WhGK D9cevdT0aMye3VLBas7MXHvajR2ttyTt2NYA+bp/vTltR+ymDv2WcG3fH+fexOusYLXuqt3Y rtabWkvU5hT4XvnbX9NsT/Xm7oOY+fXsqudXNN507Ml3nqf24rfPdNOep3+5/dHf0m72r6ee tKRda3YR+9cm7ZqqjS4l+di4p5Rw7blZe7O5L/+Lnvu2ZDff7X/p72ls7n/RY+p4bQ809/Ry NF8esYN31o1NHZM6eFuj51p3o/p674Ovxo1tPY3my/3P3T8IaT3B01ooNva7jrrvRlD7/be5 55aTae+06Wp+LHcvm/c0h9RrLLK3+2/8TAgAAAAAAAAAAAAAAAAAAAAAAAAAAADgr2d7jtSr secdY8zLgf2m7/ek8JeTfv9m3H/+++J7093f/Y2d87/A385Pfn/UPoXhccOGJ+nBJVGT5X+c ItKlF3X4vdoUi08XPknZ4LYq/9Js/Ru2YNdfkcXpHON0FluY7gmw4C9n988aZ216Sb+Bj79M X7+z/Dx7dmlWpG3VtP6md0q/gXV5C0qtn2i/hVknAAQ4svuXsjCzfLLMF7Q6k2bPSL8FN2H1 tiotjNP61+6B10+038L0zMgC/wp2/7zaXupZHNM7H0pSp4oJHvPkBNscPNsWtq/kMOo7c0TB L+SBf2W3zDINQZxhIU6Tl1e5am0IVPiX5wBawyj8gwc88M8Uc9Bs07CElofaVi2fevck/8th FP7BAy79W5qrZu31S3NKGXtctdTyxqL+l7eQpk2bQ46Yw1D/gwdc+7e0bXOv89L+jU94GNW+ KjeEl9auqbaQ279rGNq/8IBr/+pn4Cz9f6Eul6bm31ad+v+23C4+lSXX/1IY+v+gn677HxXc /4Ab9Nz/rWifkQ9g5+Yci4x/AQAAAAAAAACAN/J/AwdZLAplbmRzdHJlYW0KZW5kb2JqCjkg MCBvYmoKNTg5MAplbmRvYmoKMTAgMCBvYmoKWyAvSW5kZXhlZCAvRGV2aWNlUkdCIDEwOCAx MyAwIFIgXQplbmRvYmoKMTEgMCBvYmoKPDwKL0ZpbHRlciBbIC9GbGF0ZURlY29kZSBdCi9X aWR0aCAxMDYKL0hlaWdodCA4MAovQ29sb3JTcGFjZSAxMCAwIFIKL0JpdHNQZXJDb21wb25l bnQgOAovTGVuZ3RoIDEyIDAgUgo+PgpzdHJlYW0KeNrtXGtPG0cX/jFVVSWt1CZtmkvbJE3U Vq2qSr0oX9pP/R35QBQUlAsRSQgUMGBsY4yBGMrFgLmbi40BY8jaEEMwN3MxtgGvARvIvM8y vCuXXWxjB1VV5tFqNXPOmTNnn52ZnVmvhxCGU4HX6x0fH+c4LhwO/9ux/Pfw8uVLlUpVW1s7 NzeH7NLS0vr6eiQS2d3d3dvbi0ajkf9j7wCRGCALMySoGc7UhgpFJ7F+RMkRt7ES6kd0Iis5 4va48KRuY0tJJWJ4+/v7ybC3vLzc19dntVqDweCbN29sNlsoFKJO4IHGTLF/gN0YUMnW1hYa MJouLYJIVlZW4MTv9+OMNI1qY2MDEmghhPH2AWj8NAEhztQzz/M0DeeIcGdnBza4rZubmxDC cmFhIRAI+Hw++FxbW6PBiPcOQHEaLTzQImLAog0NWFYCKpJsfm8OgATKjo2N4Xyi1vv69WuF QtHY2NjZ2Tk0NFRRUfHkyZP29naNRjMyMmKxWOATt0ar1er1ehj39/f39vaCcLvdPjw8bDKZ CgoKUBz2sGxra2tpaYETjCe4HGgfPXoESV1dXXl5ucFgwP3FjaiurkYVsKmvr4eN0+mEAZpB V1dXTU0NsqiioaEBdel0OrPZ3Nra2t3dPTAwAMno6Og/yEE6aa7iIDX2PB5PR0cHoiorK8MF 3r59Ozs7G6FWVVVBDvYGBwfRPDA4gBO32w3Lnp4esIcELionJycrKwsFcdWo/fHjx/n5+Wq1 GvcCzaaoqCgzMxMMgCLYgP/m5ma0K9ymkQMYjUbcEYfDgbtTWlqKW4AAQLJSqbx37x5NlJSU 4KbgjADgCrX/o2NGo2R3999iTxz60K3QQdCVVldX0dHQ3pBF54IB7jWoQJuhvRJpaNElcYY9 7YN0xIAN/ECIfgrnU1NTaGAYk0X/SMDbzgFgBlfIIgCoaEEUoX0cZ/hEArXADFkkqCR9rt4W ewwUjL10wNhLB4y9dMDYSweMvXTA2DspMFnCTAlzIbrMYeydCJimYo2AhdLi4iKYxOwds1C6 0AP2Y5C8RBQeySYpkXo+rtTphZfkOhfLbbCH1SKWTiiCFRb4BIF0kU6X5xR0HS1KkKAS0UCU YOZPJfT9QGwp8T2AKBHfORzxA0tRQv2IbkU/pxfecR0QyyiXyyWuVugaCksY1nOTAcdxWLBj ZS1VMfYSwmq1Pn36dHZ2Vqpi7CWEyWRSq9XT09NSFWMvITDo4TGxvLwsVTH2EsJiseTl5TH2 UgP40Wq19DegI2DsJcTw8HBhYeHMzIxUxdhLCLfbXVdXt7KyIlUx9hICa7H79+9jiSFVMfYS wm6337lzR5zvib9m0gRjLz7AW09PD5ZmNOv3+zEDbGlpmZiYAIGMvTjAqhbsdXZ28jxPJV6v t7i4GM8Rm80GLWMvDtBDKyoqsrKyPB4PlYRCIfA5Pz8PPlnbiw8wYzAYMjMzRfaOaBl78YEe qlKp2IwlNeCRkZ+fv7S0JFUx9hIC8z2NRoOBTqpi7CVEOBwGddFoVKpi7CUE1rkPHjxgbwlS w+DgYE5ODntDlRoGBgYsFsvW1pZUxdiLD8yHm5qa0PbYW4IUgEeGXq+vr6/3+XyiMM3vlt8d YKUG3ujHq1SytrbW3t7e1taGBzFbqSXE5OQkOq+41gB7Op1OoVBgMGTfEiT8lsDpdBYXFy8u LtJsMBicnZ11uVzgE22PfccS/zsW9E20NLbWSA0TExNoYGzGkhowW87NzWVvCVIDx3GlpaXi uBcLxl5C4AFhMBjYV0CpAT03IyPDZrNJVYy9hEDPVSqVXq9XqmLsJcTQ0NDDhw/ZG6oUgAlh c3NzdnY2+10jBWAxUllZqdfrQ6GQVMvYiw+0PawypqenxdlyOBz2eDx0Vwf2luCkwLS5pqbG aDRiJMTirq+vLxAIJL+rQ8rbJqSwq0NsMKcd3n5yuzqgyOTkJNZuwWAQRaxW60l3dRCzcTZJ kEqOOJH6ia06GcnbDS/5XR3IwQtSwsa99MDYSweMvXTA2EsHjL10wNhLB4y9dMDYSweMvXTw 9tnDeid20r6zQ+gKCOdo9HDzMRxijUiI9pEI2d4+VB3ZpuyIW1GLs+x+XKLxcdudpbENGhYa h7s67O+PdXXtBQLE5yNzc2R2lgSDBFmvl3g8gnBtTThcLtLeTurqSHU16esjVivp7SW1tcRk EtJdXaSpiRiNRK8XDtjYbKS+nqhURKkkDQ2ko4OYzaS5mbx4QfLzBRu4gtZgELQVFUJZh4PU 1JDnz0l/v5D9+29BOzIiVIeyqAsJ+FGrhapRL2ppbBRqgTeLRVChFCSIBD7hqrubVFYKchhw HLHbydgYaW0VtNQV/KAWSGA5MCDcuOQQCASMRqOwqwMhwxkZm7/+yl+9yn/5JX/jBv/99/wf f/A//MD/+CP/009C4uef+W+/5S9f5t97j//ss/CFC5sXLqxfvhw+f16QfPGFYINS778fvn7d d+lS4Px52AiSa9cE7ZUr/Icf8pcu8V99xf/+O//nn/xvv/GffMKfPct//XX46tXQzZvBM2fC sL91i79+nf/mG/7mTf7ixfCVK+s3bvjPng3DEuFBiOPzzwVX587xH3wgRIuYP/4YUcGV4B8V oTocn34q1H7xolAEZriE775DOghXZ87wH30kqH75RdCeOxe+di1469YCuD0Je/RdvdtsfllQ wP31F4dzXp5wFBUJWRyFhYIQByRlZYeHUjleVjYOVWUlp9VyKhWnVnM6HadQOHU6h0YzXlp6 WLC4mCsvF7Q0W1LCVVRwBoNgDBvIcVYoUPsYDGBcXc21tQnaA4fOykqHQjFSVOTU61GpUBcM 4BCW9NBohDPcIkHTMKNu4QSBwRhBolRNjaDSaBxwggSqg08UefFCyJaWjul083L/gozfc4V0 ap2fgZCtrS232y27+S2EExMTsu9tpqen8ayRqqLRqNPpXMM4KYfx8XFZ1fLyMkrJVjQ5OTk0 NCT75s3n8yFyqTwSiTgcDr/fL3tFU1NT0m8qIEcAr169wvlE7G1sbKAu2a0REcbMzIxs5IOD g30YeCUAe729vSMY6uUiz8nJGZMbW3BFKpUKg4lU1draqlard/DglgBhy/5/BxUd9+M1z/N0 r1rpleJGdHR0mEymJN+RUmxvb6NJbB/zuKF7VErlo6OjuGSpfHd3F95kf36CH9wLWYowhoBV XIJUBVcul0v2ilZXV2VZhTFaLN2EU/ZiZUvhSnEvZDlnYHjHgWEZjxt0eZzn5uboFrh4MGHi hOz8/DzmnzgjDbnZbIZqYWFB/GPsOw6bzYZh+fnz58+ePcvLy7MeQKvVFhQU3L17V6PRVFVV FRUVNTY20p2Tc3Nz8RA8pc1+/3PAOI+mhSEdjQrPHTQwNDMQiOeC3W7HeA6ucIYZWml/f//w 8DAaKmt7x4F+PxzHAPP5E/1Q+LbwP7IJmz4KZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoy NTg4CmVuZG9iagoxMyAwIG9iago8PAovTGVuZ3RoIDE0IDAgUgo+PgpzdHJlYW0K////AAAA oKCg/wAAAMAAAID/wAD/AO7uwEAA7u4AICDA/8AgAIBAoID/gEAA/4D/AMBgAMDAAGCAwGCA AIAAQP+AMGCAgGAAQEBAQIAAAACAgGAQgGBggGCAAADAAAD/AGAA47DAQMCAYKDAYMAAYMCg gAAAgACAYCCAYGBgICAgIEBAIECAYIAgYIBgYICAgIBAIIAggICAoKCgoNDgwCAgAICAwGAA gMDgwGDAwIAAwIBg/0AA/0BAgMD//4Bg/4CAwKAAwMDAwP/A/wAA/wD//4CgwMCg/2BgAP8A /4AA/6AAgODgoODgoP8gwAAAwADAoCAgoCD/gCAAgCAggEAggECAgGDAgGD/gIAAwMAA/4BA /6BA/6Bg/6Bw/8DA//8A//+A///An5+fHx8fPz8/v7+/f39/X19f39/fp6enj4+Pi4uLCmVu ZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKMzI3CmVuZG9iagoxNSAwIG9iago8PAo+PgplbmRv YmoKMTYgMCBvYmoKMzI3CmVuZG9iagoxNyAwIG9iago8PAovVGl0bGUgKGNvbm50aW1lLnBk ZikKL0NyZWF0aW9uRGF0ZSAoRDoyMDE0MDIyNzEwNDgyNSkKL01vZERhdGUgKEQ6MjAxNDAy MjcxMDQ4MjUpCi9Qcm9kdWNlciAoSW1hZ2VNYWdpY2sgNi41LjQtNyAyMDE0LTAyLTEwIFEx NiBPcGVuTVAgaHR0cDovL3d3dy5pbWFnZW1hZ2ljay5vcmcpCj4+CmVuZG9iagp4cmVmCjAg MTgKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDEwIDAwMDAwIG4gCjAwMDAwMDAwNTkg MDAwMDAgbiAKMDAwMDAwMDExOCAwMDAwMCBuIAowMDAwMDAwMzAwIDAwMDAwIG4gCjAwMDAw MDAzODMgMDAwMDAgbiAKMDAwMDAwMDQwMSAwMDAwMCBuIAowMDAwMDAwNDM5IDAwMDAwIG4g CjAwMDAwMDA0NjAgMDAwMDAgbiAKMDAwMDAwNjUzMiAwMDAwMCBuIAowMDAwMDA2NTUyIDAw MDAwIG4gCjAwMDAwMDY2MDMgMDAwMDAgbiAKMDAwMDAwOTMzMiAwMDAwMCBuIAowMDAwMDA5 MzUzIDAwMDAwIG4gCjAwMDAwMDk3MzUgMDAwMDAgbiAKMDAwMDAwOTc1NSAwMDAwMCBuIAow MDAwMDA5Nzc3IDAwMDAwIG4gCjAwMDAwMDk3OTcgMDAwMDAgbiAKdHJhaWxlcgo8PAovU2l6 ZSAxOAovSW5mbyAxNyAwIFIKL1Jvb3QgMSAwIFIKPj4Kc3RhcnR4cmVmCjk5ODMKJSVFT0YK --------------080207040208060206070203--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?len481$sfv$2>