Date: Mon, 13 May 2013 19:36:49 +0300 From: Eugen-Andrei Gavriloaie <shiretu@gmail.com> To: Adrian Chadd <adrian@freebsd.org> Cc: freebsd-hackers@freebsd.org Subject: Re: Managing userland data pointers in kqueue/kevent Message-ID: <985C1C3F-3F70-47D2-8F43-F3D6CCA4482C@gmail.com> In-Reply-To: <CAJ-Vmo=CVeXpf9WNOegD3yG9Q0NwUWaLadVrv1RgeyAaHYADiQ@mail.gmail.com> References: <CCE4FFC4-F846-4F81-85EE-776B753C63C6@gmail.com> <CAJ-VmokBfBvdLa_Wf2EajF%2BvecVntLDaxdVeNvhAOiPp6HkjNA@mail.gmail.com> <84DCA050-99D4-4B22-A031-35E0928709E0@gmail.com> <CAJ-Vmo=CVeXpf9WNOegD3yG9Q0NwUWaLadVrv1RgeyAaHYADiQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_59961354-6B33-4E9C-87D4-EB3B97A86EC0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi Adrian, All the tricks, work arounds, paradigms suggested/implemented by us, the = kq users, are greatly simplified by simply adding that thing that Paul = is suggesting. What you are saying here is to basically do = not-so-natural things to overcome a real problem which can be very easy = and non-intrusivly solved at lower levels. Seriously, if you truly = believe that you can put the equal sign between the complexity of the = user space code and the wanted patch in kqueue kernel side, than I = simply shut up. Besides, one of the important points in kq philosophy is simplifying = things. I underline the "one of". It is not the goal, of course. Complex = things are complex things no matter how hard you try to simplify them. = But this is definitely (should) not falling into that category. ------ Eugen-Andrei Gavriloaie Web: http://www.rtmpd.com On May 13, 2013, at 6:47 PM, Adrian Chadd <adrian@freebsd.org> wrote: > ... holy crap. >=20 > On 13 May 2013 08:37, Eugen-Andrei Gavriloaie <shiretu@gmail.com> = wrote: >> Hi, >>=20 >> Well, Paul already asked this question like 3-4 times now. Even = insisting on it. I will also ask it again: >> If user code is responsible of tracking down the data associated with = the signalled entity, what is the point of having user data? >> Is rendered completely useless=85 >=20 > .. why does everything have to have a well defined purpose that is > also suited for use in _all_ situations? That is called perfection. I know we can't achieve it, but I like to = walk in that direction at least. >=20 >> Not to mention, that your suggestion with FD index is a definite = no-go. The FD values are re-used. Especially in MT environments. Imagine = one kqueue call taking place in thread A and another one in thread B. = Both threads waiting for events. >=20 > .. so don't do that. I mean, you're already having to write your code > to _not_ touch FDs in other threads. I've done this before, it isn't > that hard and it doesn't hurt performance. Why not? This is how you achieve natural load balancing for multiple = kevent() calls from multiple threads over the same kq fd. Otherwise, = again, you have to write complex code to manually balance the threads. = That brings locking again=85. Why people always think that locking is cheap? Excessive locking hurts. = A lot! >=20 >> When A does his magic, because of internal business rules, it decides = to close FD number 123. It closes it and it connects somewhere else by = opening a new one. Surprise, we MAY get the value 123 again as a new = socket, we put it on our index, etc. Now, thread B comes in and it has = stale/old events for the old 123 FD. Somethings bad like EOF for the OLD = version of FD number 123 (the one we just closed anyway). Guess what=85 = thread B will deallocate the perfectly good thingy inside the index = associated with 123. >=20 > So you just ensure that nothing at all calls a close(123); but calls > fd_close(123) which will in turn close(123) and free all the state > associated with it. Once threads A and B returned from their kevent() calls, all bets are = off. In between, you get the the behaviour I just described from threads = A and B racing towards FD123 to either close it or create a new one. How = is wrapping close() going to help? Is not like you have any control over = what the socket() function is going to return. (That gave me another = token idea btw=85 I will explain in another email, perhaps you care to = comment) Mathematically speaking, the fd-to-data association is not bijective.=20 >=20 > You have fd_close() either grab a lock, or you ensure that only the > owning thread can call fd_close(123) and if any other thread calls it, > the behaviour is undefined. As I said, that adds up to the user-space code complexity. Just don't = forget that Paul's suggestion solves all this problems in a ridiculously = simple manner. All our ideas of keeping track who is owning who and = indexes are going to be put to rest. kq will notify us when the udata is = out of scope from kq perspective. That is all we ask. >=20 >> And regarding the "thread happiness", that is not happiness at all = IMHO=85 >=20 > Unless you're writing a high connection throughput web server, the > overhead of grabbing a lock in userland during the fd shutdown process > is trivial. Yes, I've written those. It doesn't hurt you that much. That "that much" is subjective. And a streaming server is a few orders = of magnitude more complex than a web server. Remember, a web server is = bound to request/response paradigm. While a streaming server is a full = duplex (not request/response based) animal for most of connections. I = strongly believe that becomes a real problem. (I would love to be wrong = on this one!) >=20 > I'm confused as to why this is still an issue. Sure, fix the kqueue > semantics and do it in a way that doesn't break backwards > compatibility. Than, if someone has time and pleasure, it would be nice to have it. Is = a neat solution. Is one thing saying, hey, we don't have time, do it = yourself. And another thing in trying to offer "better" solutions by = defending such an obvious caveat.=20 > But please don't claim that it's stopping you from > getting real work done. I didn't and I won't. I promise! > I've written network apps with kqueue that > scales to 8+ cores and (back in mid-2000's) gigabit + of small HTTP > transactions. Good for you. How is this relevant to or discussion of simplifying = things? Of course is possible. But let's make things simpler and more = efficient. It really pays off in the long run. Hell, this is how kq was = born in the first place: getting rid of all garbage that one was = supposed to do to achieve what kq does with a few lines of code. Let's = make that even better than it currently is. > This stuff isn't at all problematic. >=20 >=20 > Adrian --Apple-Mail=_59961354-6B33-4E9C-87D4-EB3B97A86EC0 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIO5jCCBJ0w ggOFoAMCAQICEDQ96SusJzT/j8s0lPvMcFQwDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0Ux FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAgTmV0 d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0wNTA2MDcwODA5MTBa Fw0yMDA1MzAxMDQ4MzhaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNh bHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0 dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0 aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsjmF pPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIxB8dOtINknS4p1aJk xIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8om+rWV6lL8/K2m2q L+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHGTPNpsaguG7bUMSAs vIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7NlyP0e03RiqhjKaJMe oYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4H0MIHxMB8GA1UdIwQYMBaAFK29mHo0tCb3 +sQmVO8DveAky1QaMB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNVHQ8BAf8EBAMC AQYwDwYDVR0TAQH/BAUwAwEB/zARBgNVHSAECjAIMAYGBFUdIAAwRAYDVR0fBD0wOzA5oDegNYYz aHR0cDovL2NybC51c2VydHJ1c3QuY29tL0FkZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMDUGCCsG AQUFBwEBBCkwJzAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG 9w0BAQUFAAOCAQEAAbyc42MosPMxAcLfe91ioAGdIzEPnJJzU1HqH0z61p/Eyi9nfngzD3QWuZGH kfWKJvpkcADYHvkLBGJQh5OB1Nr1I9s0u4VWtHA0bniDNx6FHMURFZJfhxe9rGr98cLRzIlfsXzw PlHyNfN87GCYazor4O/fs32G67Ub9VvsonyYE9cAULnRLXPeA3h04QWFMV7LmrmdlMa5lDd1ctxE +2fo8PolHlKn2iXpR+CgxzygTrEKNvt3SJ/vl4r7tP7jlBSog7xcLT/SYHFg7sJxggzpiDbj2iC0 o6BsqpZLuICOdcpJB/Y7FLrf3AXZn9vgsuZNoHgm5+ctbn9fxh6IFTCCBRowggQCoAMCAQICEG0Z 6qcZT2ozIuYiMnqqcd4wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJV VDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29y azEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZp cnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTEwNDI4MDAwMDAwWhcNMjAw NTMwMTA0ODM4WjCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQ MA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENP TU9ETyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBAJKEhFtLV5jUXi+LpOFAyKNTWF9mZfEyTvefMn1V0HhMVbdC lOD5J3EHxcZppLkyxPFAGpDMJ1Zifxe1cWmu5SAb5MtjXmDKokH2auGj/7jfH0htZUOMKi4rYzh3 37EXrMLaggLW1DJq1GdvIBOPXDX65VSAr9hxCh03CgJQU2yVHakQFLSZlVkSMf8JotJM3FLb3uJA AVtIaN3FSrTg7SQfOq9xXwfjrL8UO7AlcWg99A/WF1hGFYE8aIuLgw9teiFX5jSw2zJ+40rhpVJy ZCaRTqWSD//gsWD9Gm9oUZljjRqLpcxCm5t9ImPTqaD8zp6Q30QZ9FxbNboW86eb/8ECAwEAAaOC AUswggFHMB8GA1UdIwQYMBaAFImCZ33EnSZwAEu0UEh83j2uBG59MB0GA1UdDgQWBBR6E04AdFvG eGNkJ8Ev4qBbvHnFezAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIBADARBgNVHSAE CjAIMAYGBFUdIAAwWAYDVR0fBFEwTzBNoEugSYZHaHR0cDovL2NybC51c2VydHJ1c3QuY29tL1VU Ti1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwdAYIKwYBBQUHAQEE aDBmMD0GCCsGAQUFBzAChjFodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVROQWRkVHJ1c3RDbGll bnRfQ0EuY3J0MCUGCCsGAQUFBzABhhlodHRwOi8vb2NzcC51c2VydHJ1c3QuY29tMA0GCSqGSIb3 DQEBBQUAA4IBAQCF1r54V1VtM39EUv5C1QaoAQOAivsNsv1Kv/avQUn1G1rF0q0bc24+6SZ85kyY wTAo38v7QjyhJT4KddbQPTmGZtGhm7VNm2+vKGwdr+XqdFqo2rHA8XV6L566k3nK/uKRHlZ0sviN 0+BDchvtj/1gOSBH+4uvOmVIPJg9pSW/ve9g4EnlFsjrP0OD8ODuDcHTzTNfm9C9YGqzO/761Mk6 PB/tm/+bSTO+Qik5g+4zaS6CnUVNqGnagBsePdIaXXxHmaWbCG0SmYbWXVcHG6cwvktJRLiQfsrR eTjrtDP6oDpdJlieYVUYtCHVmdXgQ0BCML7qpeeU0rD+83X5f27nMIIFIzCCBAugAwIBAgIQY7lP +MJT4QzHh5dD4LWVRzANBgkqhkiG9w0BAQUFADCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdy ZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExp bWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBF bWFpbCBDQTAeFw0xMjA4MjEwMDAwMDBaFw0xMzA4MjEyMzU5NTlaMCIxIDAeBgkqhkiG9w0BCQEW EXNoaXJldHVAZ21haWwuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAziBH0E99 6jx0UrKapQcbxgWLgux1j9/Ec7K3QkkFtzCgxtF9ohpfsiAVOkYuCzVxoxoC0eNKO4qbZPYNbsN6 gtVZMrQnphgG3Wzmf28azn60TtCEQNoFXdW4On+cZgcvUxZS09DaPuTrHB5BuOmYG7b93ZjvJuov KCYAjYNrg0S9TKTQdRJAu95E2OuoVzin+QwIfEVwOA+PSRTPbNnjHvHUMg2G98CXFKIYPy9Pz8Kk rd54q7eSanjNiQK4Z9UYFjjNokx0nhymUAlnzWDAPj8jlh5i6gPrWsr9mYmQ6Y9wm2RHyf7QA6nM BIO43JNPklqjUT5LnB47rxVnLKnYmQIDAQABo4IB4TCCAd0wHwYDVR0jBBgwFoAUehNOAHRbxnhj ZCfBL+KgW7x5xXswHQYDVR0OBBYEFFoYqkq01lUdSiemIAjKHTuJiRj9MA4GA1UdDwEB/wQEAwIF oDAMBgNVHRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgB hvhCAQEEBAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0 cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwVwYDVR0fBFAwTjBMoEqgSIZGaHR0cDovL2NybC5j b21vZG9jYS5jb20vQ09NT0RPQ2xpZW50QXV0aGVudGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNy bDCBiAYIKwYBBQUHAQEEfDB6MFIGCCsGAQUFBzAChkZodHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9D T01PRE9DbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3J0MCQGCCsGAQUFBzAB hhhodHRwOi8vb2NzcC5jb21vZG9jYS5jb20wHAYDVR0RBBUwE4ERc2hpcmV0dUBnbWFpbC5jb20w DQYJKoZIhvcNAQEFBQADggEBAHOQHTMzyiauVSuvCNgN2laY11tdDegdalfDK8UIFpdv08TE1viQ Q9gQfJpGNiO4wxpK8NeB8Qkh+AlL9ygq3jNwuAhk+TXMJ4Jr/I7LkTyJFnW45ADLZs/ufAbc+gdK lxdIxRJBOHmEpA51h0beSlfgHZcvvGshESzwLF290LuEdmLWncIWbhuO4vOJRK/7wJ/KXD8bZdx5 LGvA/QPPi5u95zFoQPRLSDl7lqJailBJZe6YFs7zBK2lm3u0w7exJw/rYeXFjMR0g5BPQCiihMq5 77k2T7uKmQqzapeCqLcd0sFNi94lzgSJxaQDyiDZYeYjWX0lSNVEG0z/fCIVJj4xggOrMIIDpwIB ATCBqDCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE BxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBD bGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIQY7lP+MJT4QzHh5dD4LWV RzAJBgUrDgMCGgUAoIIB1zAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEP Fw0xMzA1MTMxNjM2NDlaMCMGCSqGSIb3DQEJBDEWBBRO5jGaJQbNYkk8pNooWNSHbx4X2TCBuQYJ KwYBBAGCNxAEMYGrMIGoMIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVz dGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDE5MDcGA1UE AxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhBjuU/4 wlPhDMeHl0PgtZVHMIG7BgsqhkiG9w0BCRACCzGBq6CBqDCBkzELMAkGA1UEBhMCR0IxGzAZBgNV BAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RP IENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNl Y3VyZSBFbWFpbCBDQQIQY7lP+MJT4QzHh5dD4LWVRzANBgkqhkiG9w0BAQEFAASCAQCXWkLqmsoD aA6yV9K71t5DJEumwSMWvWs/hxDnb8d19rEBymsXphKLOYSZqM8mPFMKSUlOeHqWVIYeX2frC9sz BiiET7TtG4DUo1q34Tv3IXM9wP+VnYzeFE8I+L2iE8T0d/YAPR282O0ZUxGtFAfSXmin9T15wAXr cY5mnaQv6NoaBXtpPSVe0unY9MiI3yO1KK8embLiAOvv1oQFWbCGBYoD19606IceHeOa0P8hWymA r0coO0uRoAG9eZKNp11z5Bvx31rtMahmrWMbxaxaS+PE4EPmiqFOdnxZjTv4FTdtg82vtxo0urrs +UBOBEfTiZOiM2cCe2ruMOLNg5z0AAAAAAAA --Apple-Mail=_59961354-6B33-4E9C-87D4-EB3B97A86EC0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?985C1C3F-3F70-47D2-8F43-F3D6CCA4482C>