Date: Fri, 11 Feb 2005 13:34:21 -0800 (PST) From: Kelly Yancey <kelly@nttmcl.com> To: freebsd-net@freebsd.org Subject: Patch to set TCP_NOPUSH on libfetch HTTP connections Message-ID: <20050211125850.B9541@alicia.nttmcl.com>
next in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.
--0-182495345-1108157661=:9541
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
I recently noticed that libfetch often splits HTTP GET requests across
multiple packets as a result of calling writev() for each line of the
header. A simple request ends up on the wire as:
13:19:42.647461 216.69.71.45.1390 > 216.69.64.149.80: S 1287054177:1287054177(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 87600634 0> (DF)
0x0000 4500 003c a6c1 4000 4006 5bad d845 472d E..<..@.@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e361 0000 0000 .E@..n.PL..a....
0x0020 a002 e000 49b1 0000 0204 05b4 0103 0300 ....I...........
0x0030 0101 080a 0538 adfa 0000 0000 .....8......
13:19:42.653735 216.69.64.149.80 > 216.69.71.45.1390: S 2666070983:2666070983(0) ack 1287054178 win 17376 <mss 1460,nop,wscale 0,nop,nop,timestamp 747421059 87600634> (DF)
0x0000 4500 003c 7647 4000 3c06 9027 d845 4095 E..<vG@.<..'.E@.
0x0010 d845 472d 0050 056e 9ee9 03c7 4cb6 e362 .EG-.P.n....L..b
0x0020 a012 43e0 5900 0000 0204 05b4 0103 0300 ..C.Y...........
0x0030 0101 080a 2c8c bd83 0538 adfa ....,....8..
13:19:42.653800 216.69.71.45.1390 > 216.69.64.149.80: . ack 1 win 57920 <nop,nop,timestamp 87600635 747421059> (DF)
0x0000 4500 0034 a6c2 4000 4006 5bb4 d845 472d E..4..@.@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e362 9ee9 03c8 .E@..n.PL..b....
0x0020 8010 e240 e662 0000 0101 080a 0538 adfb ...@.b.......8..
0x0030 2c8c bd83 ,...
13:19:42.654342 216.69.71.45.1390 > 216.69.64.149.80: P 1:33(32) ack 1 win 57920 <nop,nop,timestamp 87600635 747421059> (DF)
0x0000 4500 0054 a6c3 4000 4006 5b93 d845 472d E..T..@.@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e362 9ee9 03c8 .E@..n.PL..b....
0x0020 8018 e240 e42b 0000 0101 080a 0538 adfb ...@.+.......8..
0x0030 2c8c bd83 4745 5420 2f6e 6f6e 6578 6973 ,...GET./nonexis
0x0040 7465 6e74 2e68 746d 6c20 4854 5450 2f31 tent.html.HTTP/1
0x0050 2e31 0d0a .1..
13:19:42.851075 216.69.64.149.80 > 216.69.71.45.1390: . ack 33 win 17376 <nop,nop,timestamp 747421059 87600635> (DF)
0x0000 4500 0034 640f 4000 3c06 a267 d845 4095 E..4d.@.<..g.E@.
0x0010 d845 472d 0050 056e 9ee9 03c8 4cb6 e382 .EG-.P.n....L...
0x0020 8010 43e0 84a3 0000 0101 080a 2c8c bd83 ..C.........,...
0x0030 0538 adfb .8..
13:19:42.851127 216.69.71.45.1390 > 216.69.64.149.80: P 33:108(75) ack 1 win 57920 <nop,nop,timestamp 87600655 747421059> (DF)
0x0000 4500 007f a6c4 4000 4006 5b67 d845 472d E.....@.@.[g.EG-
0x0010 d845 4095 056e 0050 4cb6 e382 9ee9 03c8 .E@..n.PL.......
0x0020 8018 e240 51b2 0000 0101 080a 0538 ae0f ...@Q........8..
0x0030 2c8c bd83 486f 7374 3a20 7777 772e 6e74 ,...Host:.www.nt
0x0040 746d 636c 2e63 6f6d 0d0a 5573 6572 2d41 tmcl.com..User-A
0x0050 6765 6e74 3a20 6665 7463 6820 6c69 6266 gent:.fetch.libf
0x0060 6574 6368 2f32 2e30 0d0a 436f 6e6e 6563 etch/2.0..Connec
0x0070 7469 6f6e 3a20 636c 6f73 650d 0a0d 0a tion:.close....
13:19:42.857163 216.69.64.149.80 > 216.69.71.45.1390: F 514:514(0) ack 108 win 17376 <nop,nop,timestamp 747421059 87600655> (DF)
0x0000 4500 0034 323a 4000 3c06 d43c d845 4095 E..42:@.<..<.E@.
0x0010 d845 472d 0050 056e 9ee9 05c9 4cb6 e3cd .EG-.P.n....L...
0x0020 8011 43e0 8242 0000 0101 080a 2c8c bd83 ..C..B......,...
0x0030 0538 ae0f .8..
13:19:42.857180 216.69.71.45.1390 > 216.69.64.149.80: . ack 1 win 57920 <nop,nop,timestamp 87600655 747421059> (DF)
0x0000 4500 0034 a6c5 4000 4006 5bb1 d845 472d E..4..@.@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e3cd 9ee9 03c8 .E@..n.PL.......
0x0020 8010 e240 e5e3 0000 0101 080a 0538 ae0f ...@.........8..
0x0030 2c8c bd83 ,...
13:19:42.859754 216.69.64.149.80 > 216.69.71.45.1390: P 1:514(513) ack 108 win 17376 <nop,nop,timestamp 747421059 87600655> (DF)
[ snip file contents ]
13:19:42.859773 216.69.71.45.1390 > 216.69.64.149.80: . ack 515 win 57407 <nop,nop,timestamp 87600655 747421059> (DF)
0x0000 4500 0034 a6c6 4000 4006 5bb0 d845 472d E..4..@.@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e3cd 9ee9 05ca .E@..n.PL.......
0x0020 8010 e03f e5e2 0000 0101 080a 0538 ae0f ...?.........8..
0x0030 2c8c bd83 ,...
13:19:42.860070 216.69.71.45.1390 > 216.69.64.149.80: F 108:108(0) ack 515 win 57920 <nop,nop,timestamp 87600655 747421059> (DF)
0x0000 4500 0034 a6c7 4000 4006 5baf d845 472d E..4..@.@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e3cd 9ee9 05ca .E@..n.PL.......
0x0020 8011 e240 e3e0 0000 0101 080a 0538 ae0f ...@.........8..
0x0030 2c8c bd83 ,...
13:19:42.865699 216.69.64.149.80 > 216.69.71.45.1390: . ack 109 win 17376 <nop,nop,timestamp 747421059 87600655> (DF)
0x0000 4500 0034 55a4 4000 3c06 b0d2 d845 4095 E..4U.@.<....E@.
0x0010 d845 472d 0050 056e 9ee9 05ca 4cb6 e3ce .EG-.P.n....L...
0x0020 8010 43e0 8241 0000 0101 080a 2c8c bd83 ..C..A......,...
0x0030 0538 ae0f .8..
13:19:42.871415 216.69.64.149.80 > 216.69.71.45.1390: . ack 109 win 17376 <nop,nop,timestamp 747421059 87600655> (DF)
0x0000 4500 0034 069a 4000 3c06 ffdc d845 4095 E..4..@.<....E@.
0x0010 d845 472d 0050 056e 9ee9 05ca 4cb6 e3ce .EG-.P.n....L...
0x0020 8010 43e0 8241 0000 0101 080a 2c8c bd83 ..C..A......,...
0x0030 0538 ae0f .8..
13:19:42.871449 216.69.71.45.1390 > 216.69.64.149.80: R 1287054286:1287054286(0) win 0
0x0000 4500 0028 a6c8 0000 4006 9bba d845 472d E..(....@....EG-
0x0010 d845 4095 056e 0050 4cb6 e3ce 0000 0000 .E@..n.PL.......
0x0020 5004 0000 4150 0000 P...AP..
The attached patch sets the TCP_NOPUSH option on the socket and uses
shutdown(conn->sd, SHUT_WR) at the end of the HTTP request in order to
force the entire HTTP request to be coelesced into a minimum number of
packets. With the attached patch applied, the same request shown above
appears on the wire as:
13:17:10.659049 216.69.71.45.2218 > 216.69.64.149.80: S 2067322044:2067322044(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 87585436 0> (DF)
0x0000 4500 003c 9c27 4000 4006 6647 d845 472d E..<.'@.@.fG.EG-
0x0010 d845 4095 08aa 0050 7b38 d4bc 0000 0000 .E@....P{8......
0x0020 a002 e000 61f6 0000 0204 05b4 0103 0300 ....a...........
0x0030 0101 080a 0538 729c 0000 0000 .....8r.....
13:17:10.663461 216.69.64.149.80 > 216.69.71.45.2218: S 3505347452:3505347452(0) ack 2067322045 win 17376 <mss 1460,nop,wscale 0,nop,nop,timestamp 747420755 87585436> (DF)
0x0000 4500 003c da68 4000 3c06 2c06 d845 4095 E..<.h@.<.,..E@.
0x0010 d845 472d 0050 08aa d0ef 5b7c 7b38 d4bd .EG-.P....[|{8..
0x0020 a012 43e0 e8b9 0000 0204 05b4 0103 0300 ..C.............
0x0030 0101 080a 2c8c bc53 0538 729c ....,..S.8r.
13:17:10.663510 216.69.71.45.2218 > 216.69.64.149.80: . ack 1 win 57920 <nop,nop,timestamp 87585436 747420755> (DF)
0x0000 4500 0034 9c28 4000 4006 664e d845 472d E..4.(@.@.fN.EG-
0x0010 d845 4095 08aa 0050 7b38 d4bd d0ef 5b7d .E@....P{8....[}
0x0020 8010 e240 761d 0000 0101 080a 0538 729c ...@v........8r.
0x0030 2c8c bc53 ,..S
13:17:10.664197 216.69.71.45.2218 > 216.69.64.149.80: FP 1:108(107) ack 1 win 57920 <nop,nop,timestamp 87585436 747420755> (DF)
0x0000 4500 009f 9c29 4000 4006 65e2 d845 472d E....)@.@.e..EG-
0x0010 d845 4095 08aa 0050 7b38 d4bd d0ef 5b7d .E@....P{8....[}
0x0020 8019 e240 df70 0000 0101 080a 0538 729c ...@.p.......8r.
0x0030 2c8c bc53 4745 5420 2f6e 6f6e 6578 6973 ,..SGET./nonexis
0x0040 7465 6e74 2e68 746d 6c20 4854 5450 2f31 tent.html.HTTP/1
0x0050 2e31 0d0a 486f 7374 3a20 7777 772e 6e74 .1..Host:.www.nt
0x0060 746d 636c 2e63 6f6d 0d0a 5573 6572 2d41 tmcl.com..User-A
0x0070 6765 6e74 3a20 6665 7463 6820 6c69 6266 gent:.fetch.libf
0x0080 6574 6368 2f32 2e30 0d0a 436f 6e6e 6563 etch/2.0..Connec
0x0090 7469 6f6e 3a20 636c 6f73 650d 0a0d 0a tion:.close....
13:17:10.669275 216.69.64.149.80 > 216.69.71.45.2218: . ack 109 win 17269 <nop,nop,timestamp 747420755 87585436> (DF)
0x0000 4500 0034 8371 4000 3c06 8305 d845 4095 E..4.q@.<....E@.
0x0010 d845 472d 0050 08aa d0ef 5b7d 7b38 d529 .EG-.P....[}{8.)
0x0020 8010 4375 147d 0000 0101 080a 2c8c bc53 ..Cu.}......,..S
0x0030 0538 729c .8r.
13:17:10.670352 216.69.64.149.80 > 216.69.71.45.2218: F 514:514(0) ack 109 win 17376 <nop,nop,timestamp 747420755 87585436> (DF)
0x0000 4500 0034 ebbd 4000 3c06 1ab9 d845 4095 E..4..@.<....E@.
0x0010 d845 472d 0050 08aa d0ef 5d7e 7b38 d529 .EG-.P....]~{8.)
0x0020 8011 43e0 1210 0000 0101 080a 2c8c bc53 ..C.........,..S
0x0030 0538 729c .8r.
13:17:10.670378 216.69.71.45.2218 > 216.69.64.149.80: . ack 1 win 57920 <nop,nop,timestamp 87585437 747420755> (DF)
0x0000 4500 0034 9c2a 4000 4006 664c d845 472d E..4.*@.@.fL.EG-
0x0010 d845 4095 08aa 0050 7b38 d529 d0ef 5b7d .E@....P{8.)..[}
0x0020 8010 e240 75b0 0000 0101 080a 0538 729d ...@u........8r.
0x0030 2c8c bc53 ,..S
13:17:10.672885 216.69.64.149.80 > 216.69.71.45.2218: P 1:514(513) ack 109 win 17376 <nop,nop,timestamp 747420755 87585436> (DF)
[ snip file contents ]
13:17:10.672906 216.69.71.45.2218 > 216.69.64.149.80: . ack 515 win 57407 <nop,nop,timestamp 87585437 747420755> (DF)
0x0000 4500 0034 9c2b 4000 4006 664b d845 472d E..4.+@.@.fK.EG-
0x0010 d845 4095 08aa 0050 7b38 d529 d0ef 5d7f .E@....P{8.)..].
0x0020 8010 e03f 75af 0000 0101 080a 0538 729d ...?u........8r.
0x0030 2c8c bc53
Thus reducing the number of packets on the wire from 14 to 9. Obviously
for larger transfers, the difference gets lost in the noise. Nonetheless,
unless someone spots some undesireable side-effect that may be caused
by the change, I'll commit the attached patch in a few days.
Thanks,
Kelly
--
Kelly Yancey -- kbyanc@{posi.net,FreeBSD.org} -- kelly@nttmcl.com
--0-182495345-1108157661=:9541
Content-Type: TEXT/PLAIN; charset=US-ASCII; name="fetch-nopush.diff"
Content-Transfer-Encoding: BASE64
Content-ID: <20050211133421.H9541@alicia.nttmcl.com>
Content-Description:
Content-Disposition: attachment; filename="fetch-nopush.diff"
SW5kZXg6IGh0dHAuYw0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KUkNTIGZp
bGU6IC9ob21lL25jdnMvc3JjL2xpYi9saWJmZXRjaC9odHRwLmMsdg0KcmV0
cmlldmluZyByZXZpc2lvbiAxLjc0DQpkaWZmIC11IC1wIC1yMS43NCBodHRw
LmMNCi0tLSBodHRwLmMJMjEgU2VwIDIwMDQgMTg6MzU6MjEgLTAwMDAJMS43
NA0KKysrIGh0dHAuYwkxMSBGZWIgMjAwNSAyMTozMDo0MSAtMDAwMA0KQEAg
LTc2LDYgKzc2LDkgQEAgX19GQlNESUQoIiRGcmVlQlNEJCIpOw0KICNpbmNs
dWRlIDx0aW1lLmg+DQogI2luY2x1ZGUgPHVuaXN0ZC5oPg0KIA0KKyNpbmNs
dWRlIDxuZXRpbmV0L2luLmg+DQorI2luY2x1ZGUgPG5ldGluZXQvdGNwLmg+
DQorDQogI2luY2x1ZGUgImZldGNoLmgiDQogI2luY2x1ZGUgImNvbW1vbi5o
Ig0KICNpbmNsdWRlICJodHRwZXJyLmgiDQpAQCAtNjcwLDcgKzY3Myw3IEBA
IF9odHRwX2Nvbm5lY3Qoc3RydWN0IHVybCAqVVJMLCBzdHJ1Y3QgdXINCiB7
DQogCWNvbm5fdCAqY29ubjsNCiAJaW50IHZlcmJvc2U7DQotCWludCBhZjsN
CisJaW50IGFmLCB2YWw7DQogDQogI2lmZGVmIElORVQ2DQogCWFmID0gQUZf
VU5TUEVDOw0KQEAgLTcwNSw2ICs3MDgsMTAgQEAgX2h0dHBfY29ubmVjdChz
dHJ1Y3QgdXJsICpVUkwsIHN0cnVjdCB1cg0KIAkJX2ZldGNoX3N5c2Vycigp
Ow0KIAkJcmV0dXJuIChOVUxMKTsNCiAJfQ0KKw0KKwl2YWwgPSAxOw0KKwlz
ZXRzb2Nrb3B0KGNvbm4tPnNkLCBJUFBST1RPX1RDUCwgVENQX05PUFVTSCwg
JnZhbCwgc2l6ZW9mKHZhbCkpOw0KKw0KIAlyZXR1cm4gKGNvbm4pOw0KIH0N
CiANCkBAIC05MDYsNiArOTEzLDcgQEAgX2h0dHBfcmVxdWVzdChzdHJ1Y3Qg
dXJsICpVUkwsIGNvbnN0IGNoYQ0KIAkJCV9odHRwX2NtZChjb25uLCAiUmFu
Z2U6IGJ5dGVzPSVsbGQtIiwgKGxvbmcgbG9uZyl1cmwtPm9mZnNldCk7DQog
CQlfaHR0cF9jbWQoY29ubiwgIkNvbm5lY3Rpb246IGNsb3NlIik7DQogCQlf
aHR0cF9jbWQoY29ubiwgIiIpOw0KKwkJc2h1dGRvd24oY29ubi0+c2QsIFNI
VVRfV1IpOw0KIA0KIAkJLyogZ2V0IHJlcGx5ICovDQogCQlzd2l0Y2ggKF9o
dHRwX2dldF9yZXBseShjb25uKSkgew0K
--0-182495345-1108157661=:9541--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050211125850.B9541>