Date: Thu, 5 Dec 2013 14:30:05 +0200 From: Valentin Nechayev <netch@netch.kiev.ua> To: Michael Tuexen <Michael.Tuexen@lurchi.franken.de> Cc: freebsd-net@freebsd.org Subject: Re: SCTP huge connect delays (at amd64) and yet another question Message-ID: <20131205123005.GE71737@netch.kiev.ua> In-Reply-To: <11932BA9-A734-4D4F-BCBB-6A0D926A22A9@lurchi.franken.de> References: <20131205084142.GA31113@netch.kiev.ua> <11932BA9-A734-4D4F-BCBB-6A0D926A22A9@lurchi.franken.de>
next in thread | previous in thread | raw e-mail | index | archive | help
--bCsyhTFzCvuiizWE
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Hi,
Thu, Dec 05, 2013 at 11:32:03, Michael.Tuexen wrote about "Re: SCTP huge connect delays (at amd64) and yet another question":
> > The first discrepancy found is specific for FreeBSD on amd64 and not
> > for i386 version; it's that connection setup lasts 2-4 seconds (!!)
> > Tcpdump shows indication that could be parsed as message miss:
> Hi Valentin,
>
> could you send me the .pcap file instead of the tcpdump output.
> I would like to see the addresses listed in the INIT and INIT-ACK.
I've sent them, thanks.
> > tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 65535 byt
> > es
> > 08:18:34.639422 IP (tos 0x0, ttl 64, id 65094, offset 0, flags [none], proto SCT
> > P (132), length 188, bad cksum 0 (->f274)!)
> > 10.0.0.2.50025 > 127.0.0.1.2500: sctp
> I'm wondering why 10.0.0.2 is the source address and not 127.0.0.1
I've showed the code, it doesn't make any explicit binding or address
suggestion. For this host (9.1/i386), 10.0.0.2 resides on xl0. There
is no routing specifics which forces it to select 10.0.0.2:
$ route -n get 127.0.0.1
route to: 127.0.0.1
destination: 127.0.0.1
interface: lo0
flags: <UP,HOST,DONE,LOCAL>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 16384 1 0
$ telnet 127.0.0.1 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 iv.local ESMTP Sendmail 8.14.5/8.14.5; Thu, 5 Dec 2013 13:48:31 +0200 (EET)
ehlo zzz
250-iv.local Hello netch@localhost [127.0.0.1], pleased to meet you
[...]
At least for TCP and UDP, it's quite straightforward.
> > At 08:18:34.639467, cookie echo was sent but likely ignored. One
> > second later it was resent. Then, yet another strange timeout was
> > invented before HB REQ.
> >
> > Test series show this can spend more than 4 seconds, average value
> > is about 3 seconds. Two 20-times run summary times are 58 to 63
> > seconds, so, I've got 2.9...3.15 average connect time.
> >
> > Neither Linux nor 32-bit FreeBSD shows this.
> FreeBSD should neither... Do you see this on FreeBSD 9.2 amd64?
Yes. A fresh dump has reproduced this.
> > It's definitely better than delay each run, as on other platforms
> > (but the initial delay annoys roughly).
> Without SCTP_NODELAY bundling can happen or not, it depends on timing.
> It would be great, if you can provide a .pcap file for a transfer you
> think shows some buggy behaviour. Then we can figure out what is going on.
> MSG_EOR is nothing you provide at a send() call. The flag is only
> returned by the recvmsg() call.
Yes, I know. This has remained from the code which exposes
SOCK_SEQPACKET specifics over different transport families (e.g.
FreeBSD keeps this flag over AF_UNIX but Linux doesn't). I didn't take
it into account, but, if is needed for sight clarity, I'll remove it:)
> > }
> OK. Here is what I would expect on the wire:
>
> Without SCTP_NODELAY:
>
> > INIT
> < INIT_ACK
> > COOKIE_ECHO
> < COOKIE_ACK
> < DATA(abc)
> > SACK
> < DATA(def);DATA(ghi);DATA(jkl);DATA(mno);DATA(pqr)
> > SACK
> > SHUTDOWN
> < SHUTDOWN_ACK
> > SHUTDOWN_COMPLETE
>
> There should be no substantial delay between any messages above.
>
> With SCTP_NODELAY
> > INIT
> < INIT_ACK
> > COOKIE_ECHO
> < COOKIE_ACK
> < DATA(abc)
> < DATA(def)
> < DATA(ghi)
> < DATA(mno)
> < DATA(pqr)
> > SHUTDOWN
> < SHUTDOWN_ACK
> > SHUTDOWN_COMPLETE
>
> There will be three SACK somewhere between the DATA chunks depending on
> the timing.
>
> There should be no substantial delay between any messages above.
>
> I think if you see anything else, there is a bug. So do you see a different
> behavior on FreeBSD 9.2 (i386/amd64)? If yes, can you provide a .pcap file?
Sorry, I don't have 9.2/i386 yet. The dump from 9.1 is attached. It
has no address mess but the event sequence is following:
> INIT
< INIT_ACK
> COOKIE_ECHO
< COOKIE_ACK
< DATA(abc)
> SACK
< DATA(def)
... delay 200ms...
> SACK
< DATA(ghi); DATA(jkl); DATA(mno); DATA(pqr)
Comparing to your description, it has unexplained waiting after
DATA(def) from the server side, and SACK delay from the client side.
If you think it's fixed in 9.2, we can postpone this part of
discussion until my upgrade to 9.2.
> Do you have any special routing setup?
Just this box (9.1/i386) is trivial, no any routing specifics.
For amd64 boxes, I've sent routing details privately. But it seems
there are also none principally "special" these except multiple
addresses at loopback.
> Please note, that the first SACK is returned without the 200ms delay. This is
> required by the RFC and the above trace seems to show that.
> > But, if server shuts its writing side down ("s" in argv[]), this
> > laziness disappears. Again, the logic is too opaque and confusing.
> What do you mean by this?
At least, removing this delay by shutdown(,SHUT_WR) is unexpected.
-netch-
--bCsyhTFzCvuiizWE
Content-Type: application/octet-stream
Content-Disposition: attachment; filename="dump.blocking.91.i386"
Content-Transfer-Encoding: base64
1MOyoQIABAAAAAAAAAAAAP//AAAAAAAA7GmgUpLSBgCoAAAAqAAAAAIAAABFAACkjqAAAECE
AAB/AAABfwAAAbCOCcQAAAAAAAAAAAEAAITCIlkIABxxxwAKCACDBLlQAAwACAAFAAbABgAI
UExSU4AAAATAAAAEgAgACsGAwIGCDwAAgAIAJJXYsfcXBl4xhHUBiJhYaQrSXyWD9MxR7Ovu
h+UaZfMDgAQACAABAAOAAwAGgMEAAAAFAAjBwcEEAAUACAoAAAEABQAIfwAAAexpoFK/0gYA
8AEAAPABAAACAAAARQAB7PA7AABAhAAAfwAAAX8AAAEJxLCOwiJZCAAAAAACAAHMmtfahgAc
cccACggAjNQE78AGAAhQTFJTgAAABMAAAASACAAKwYDAgYIPAACAAgAk9Jph42rEKuidkZoz
GJX+yxzBaaTvAJD628lchrCAA6yABAAIAAEAA4ADAAaAwQAAAAcBaEtBTUUtQlNEIDEuMQAA
AAAp8nAAfAEEAGDqAAAAAAAAAAAAAMIiWQia19qGfwAAAQAAAAAAAAAAAAAAAAUAAAB/AAAB
AAAAAAAAAAAAAAAABQAAAAAAAACwjgnEAQAAAQEBAAAAAAAAAQAAhMIiWQgAHHHHAAoIAIME
uVAADAAIAAUABsAGAAhQTFJTgAAABMAAAASACAAKwYDAgYIPAACAAgAkldix9xcGXjGEdQGI
mFhpCtJfJYP0zFHs6+6H5Rpl8wOABAAIAAEAA4ADAAaAwQAAAAUACMHBwQQABQAICgAAAQAF
AAh/AAABAgABzJrX2oYAHHHHAAoIAIzUBO/ABgAIUExSU4AAAATAAAAEgAgACsGAwIGCDwAA
gAIAJPSaYeNqxCronZGaMxiV/sscwWmk7wCQ+tvJXIawgAOsgAQACAABAAOAAwAGgMEAABNy
y/c5fNDpoXfpgYzvgMFLDkul7GmgUtrSBgCMAQAAjAEAAAIAAABFAAGIpeUAAECEAAB/AAAB
fwAAAbCOCcSa19qGAAAAAAoAAWhLQU1FLUJTRCAxLjEAAAAAKfJwAHwBBABg6gAAAAAAAAAA
AADCIlkImtfahn8AAAEAAAAAAAAAAAAAAAAFAAAAfwAAAQAAAAAAAAAAAAAAAAUAAAAAAAAA
sI4JxAEAAAEBAQAAAAAAAAEAAITCIlkIABxxxwAKCACDBLlQAAwACAAFAAbABgAIUExSU4AA
AATAAAAEgAgACsGAwIGCDwAAgAIAJJXYsfcXBl4xhHUBiJhYaQrSXyWD9MxR7Ovuh+UaZfMD
gAQACAABAAOAAwAGgMEAAAAFAAjBwcEEAAUACAoAAAEABQAIfwAAAQIAAcya19qGABxxxwAK
CACM1ATvwAYACFBMUlOAAAAEwAAABIAIAArBgMCBgg8AAIACACT0mmHjasQq6J2RmjMYlf7L
HMFppO8AkPrbyVyGsIADrIAEAAgAAQADgAMABoDBAAATcsv3OXzQ6aF36YGM74DBSw5Lpexp
oFIM0wYAKAAAACgAAAACAAAARQAAJHjVQABAhAAAfwAAAX8AAAEJxLCOwiJZCAAAAAALAAAE
7GmgUpbTBgA4AAAAOAAAAAIAAABFAgA0aepAAECEAAB/AAABfwAAAQnEsI7CIlkIAAAAAAAD
ABOM1ATvAAAAAAAAAABhYmMA7GmgUqTTBgA0AAAANAAAAAIAAABFAAAwx1YAAECEAAB/AAAB
fwAAAbCOCcSa19qGAAAAAAMAABCM1ATvABxwxAAAAADsaaBSt9MGADgAAAA4AAAAAgAAAEUC
ADRihEAAQIQAAH8AAAF/AAABCcSwjsIiWQgAAAAAAAMAE4zUBPAAAAABAAAAAGRlZgDsaaBS
e94JADQAAAA0AAAAAgAAAEUAADChDwAAQIQAAH8AAAF/AAABsI4JxJrX2oYAAAAAAwAAEIzU
BPAAHHHHAAAAAOxpoFKZ3gkAdAAAAHQAAAACAAAARQIAcKjqQABAhAAAfwAAAX8AAAEJxLCO
wiJZCAAAAAAAAwATjNQE8QAAAAIAAAAAZ2hpAAADABOM1ATyAAAAAwAAAABqa2wAAAMAE4zU
BPMAAAAEAAAAAG1ubwAAAwATjNQE9AAAAAUAAAAAcHFyAOxpoFIg3wkALAAAACwAAAACAAAA
RQAAKALUQABAhAAAfwAAAX8AAAGwjgnEmtfahgAAAAAHAAAIjNQE9OxpoFIt3wkAKAAAACgA
AAACAAAARQAAJDX4QABAhAAAfwAAAX8AAAEJxLCOwiJZCAAAAAAIAAAE7GmgUjffCQAoAAAA
KAAAAAIAAABFAAAkAjRAAECEAAB/AAABfwAAAbCOCcSa19qGAAAAAA4AAAQ=
--bCsyhTFzCvuiizWE
Content-Type: application/octet-stream
Content-Disposition: attachment; filename="dump.blocking.91.i386.with_shutdown"
Content-Transfer-Encoding: base64
1MOyoQIABAAAAAAAAAAAAP//AAAAAAAAR3GgUo43AQCoAAAAqAAAAAIAAABFAACk7TkAAECE
AAB/AAABfwAAAbqICcQAAAAAAAAAAAEAAIR66MIVABxxxwAKCAD9YvR3AAwACAAFAAbABgAI
UExSU4AAAATAAAAEgAgACsGAwIGCDwAAgAIAJEhbs8Eq+J+5eMsDCDZeYbQkOOToNyfE9mtW
kOtYMkRpgAQACAABAAOAAwAGgMEAAAAFAAjBwcEEAAUACAoAAAEABQAIfwAAAUdxoFLLNwEA
8AEAAPABAAACAAAARQAB7B05AABAhAAAfwAAAX8AAAEJxLqIeujCFQAAAAACAAHM1zt92QAc
cccACggA+kkUysAGAAhQTFJTgAAABMAAAASACAAKwYDAgYIPAACAAgAkf1KiUVKTzCLyCOhR
3JrwxqftPTy4UkCNqSAfMInuC9CABAAIAAEAA4ADAAaAwQAAAAcBaEtBTUUtQlNEIDEuMQAA
AACD+XAAJqoNAGDqAAAAAAAAAAAAAHrowhXXO33ZfwAAAQAAAAAAAAAAAAAAAAUAAAB/AAAB
AAAAAAAAAAAAAAAABQAAAAAAAAC6iAnEAQAAAQEBAAAAAAAAAQAAhHrowhUAHHHHAAoIAP1i
9HcADAAIAAUABsAGAAhQTFJTgAAABMAAAASACAAKwYDAgYIPAACAAgAkSFuzwSr4n7l4ywMI
Nl5htCQ45Og3J8T2a1aQ61gyRGmABAAIAAEAA4ADAAaAwQAAAAUACMHBwQQABQAICgAAAQAF
AAh/AAABAgABzNc7fdkAHHHHAAoIAPpJFMrABgAIUExSU4AAAATAAAAEgAgACsGAwIGCDwAA
gAIAJH9SolFSk8wi8gjoUdya8Man7T08uFJAjakgHzCJ7gvQgAQACAABAAOAAwAGgMEAAPpS
mvBRdTocxYM2w4bfg3L/e5M5R3GgUuo3AQCMAQAAjAEAAAIAAABFAAGIkH4AAECEAAB/AAAB
fwAAAbqICcTXO33ZAAAAAAoAAWhLQU1FLUJTRCAxLjEAAAAAg/lwACaqDQBg6gAAAAAAAAAA
AAB66MIV1zt92X8AAAEAAAAAAAAAAAAAAAAFAAAAfwAAAQAAAAAAAAAAAAAAAAUAAAAAAAAA
uogJxAEAAAEBAQAAAAAAAAEAAIR66MIVABxxxwAKCAD9YvR3AAwACAAFAAbABgAIUExSU4AA
AATAAAAEgAgACsGAwIGCDwAAgAIAJEhbs8Eq+J+5eMsDCDZeYbQkOOToNyfE9mtWkOtYMkRp
gAQACAABAAOAAwAGgMEAAAAFAAjBwcEEAAUACAoAAAEABQAIfwAAAQIAAczXO33ZABxxxwAK
CAD6SRTKwAYACFBMUlOAAAAEwAAABIAIAArBgMCBgg8AAIACACR/UqJRUpPMIvII6FHcmvDG
p+09PLhSQI2pIB8wie4L0IAEAAgAAQADgAMABoDBAAD6UprwUXU6HMWDNsOG34Ny/3uTOUdx
oFIcOAEAKAAAACgAAAACAAAARQAAJJFZQABAhAAAfwAAAX8AAAEJxLqIeujCFQAAAAALAAAE
R3GgUks5AQA4AAAAOAAAAAIAAABFAgA0h0JAAECEAAB/AAABfwAAAQnEuoh66MIVAAAAAAAD
ABP6SRTKAAAAAAAAAABhYmMAR3GgUlo5AQA0AAAANAAAAAIAAABFAAAwEKkAAECEAAB/AAAB
fwAAAbqICcTXO33ZAAAAAAMAABD6SRTKABxwxAAAAABHcaBSszkBADgAAAA4AAAAAgAAAEUC
ADTXKkAAQIQAAH8AAAF/AAABCcS6iHrowhUAAAAAAAMAE/pJFMsAAAABAAAAAGRlZgBHcaBS
yzkBAHQAAAB0AAAAAgAAAEUCAHBuJkAAQIQAAH8AAAF/AAABCcS6iHrowhUAAAAAAAMAE/pJ
FMwAAAACAAAAAGdoaQAAAwAT+kkUzQAAAAMAAAAAamtsAAADABP6SRTOAAAABAAAAABtbm8A
AAMAE/pJFM8AAAAFAAAAAHBxcgBHcaBS1DkBADQAAAA0AAAAAgAAAEUAADC2AwAAQIQAAH8A
AAF/AAABuogJxNc7fdkAAAAAAwAAEPpJFM8AHGu1AAAAAEdxoFLhOQEALAAAACwAAAACAAAA
RQAAKNpBQABAhAAAfwAAAX8AAAEJxLqIeujCFQAAAAAHAAAI/WL0dkdxoFLpOQEAKAAAACgA
AAACAAAARQAAJDaaQABAhAAAfwAAAX8AAAG6iAnE1zt92QAAAAAIAAAER3GgUu85AQAoAAAA
KAAAAAIAAABFAAAkdutAAECEAAB/AAABfwAAAQnEuoh66MIVAAAAAA4AAAQ=
--bCsyhTFzCvuiizWE--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131205123005.GE71737>