Date: Sun, 23 Mar 2014 19:57:09 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Christopher Forgeron <csforgeron@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Garrett Wollman <wollman@freebsd.org>, Jack Vogel <jfvogel@gmail.com>, Markus Gebert <markus.gebert@hostpoint.ch> Subject: Re: 9.2 ixgbe tx queue hang Message-ID: <1936201708.1670348.1395619029045.JavaMail.root@uoguelph.ca> In-Reply-To: <CAB2_NwB%2BL5T1iTaxZtt00SL22_1H0S9X%2BM_LOGfkoJmy_TJokw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_1670344_208375117.1395619029041
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Christopher Forgeron wrote:
>
>
>
>
>
>
> On Sat, Mar 22, 2014 at 6:41 PM, Rick Macklem < rmacklem@uoguelph.ca
> > wrote:
>
>
>
> Christopher Forgeron wrote:
> > #if defined(INET) || defined(INET6)
> > /* Initialize to max value. */
> > if (ifp->if_hw_tsomax == 0)
> > ifp->if_hw_tsomax = IP_MAXPACKET;
> > KASSERT(ifp->if_hw_tsomax <= IP_MAXPACKET &&
> > ifp->if_hw_tsomax >= IP_MAXPACKET / 8,
> > ("%s: tsomax outside of range", __func__));
> > #endif
> >
> >
> > Should this be the location where it's being set rather than in
> > ixgbe? I would assume that other drivers could fall prey to this
> > issue.
> >
> All of this should be prepended with "I'm an NFS guy, not a
> networking
> guy, so I might be wrong".
>
> Other drivers (and ixgbe for the 82598 chip) can handle a packet that
> is in more than 32 mbufs. (I think the 82598 handles 100, grep for
> SCATTER
> in *.h in sys/dev/ixgbe.)
>
>
> [...]
>
>
> Yes, I agree we have to be careful about the limitations of other
> drivers, but I'm thinking setting tso to IP_MAXPACKET is a bad idea,
> unless all of the header subtractions are happening elsewhere. Then
> again, perhaps every other driver (and possibly ixgbe.. i need to
> look more) does a maxtso - various_headers to set a limit for data
> packets.
>
>
> I'm not familiar with the Freebsd network conventions/styles - I'm
> just asking questions, something I have a bad habit for, but I'm in
> charge of code stability issues at my work so it's hard to stop.
>
Well, IP_MAXPACKET is simply the largest # that fits in the 16bit length
field of an IP header (65535). This limit is on the TSO segment (which
is really just a TCP/IP packet greater than the MTU) and does not include
a MAC level (ethernet) header.
Beyond that, it is the specific hardware that limits things, such as
this case, which is limited to 32 mbufs (which happens to imply 64K
total, including ethernet header using 2K mbuf clusters).
(The 64K limit is just a quirk caused by the 32mbuf limit and the fact
that mbuf clusters hold 2K of data each.)
>
>
> Now, since several drivers do have this 32 mbufs limit, I can see an
> argument
> for making the default a little smaller to make these work, since the
> driver can override the default. (About now someone usually jumps in
> and says
> something along the lines of "You can't do that until all the drivers
> that
> can handle IP_MAXPACKET are fixed to set if_hw_tsomax" and since I
> can't fix
> drivers I can't test, that pretty much puts a stop on it.)
>
>
>
>
> Testing is a problem isn't it? I once again offer my stack of network
> cards and systems for some sort of testing.. I still have coax and
> token ring around. :-)
>
>
>
> You see the problem isn't that IP_MAXPACKET is too big, but that the
> hardware
> has a limit of 32 non-contiguous chunks (mbufs)/packet and 32 *
> MCLBYTES = 64K.
> (Hardware/network drivers that can handle 35 or more chunks (they
> like to call
> them transmit segments, although ixgbe uses the term scatter)
> shouldn't have
> any problems.)
>
> I have an untested patch that adds a tsomaxseg count to use along
> with tsomax
> bytes so that a driver could inform tcp_output() it can only handle
> 32 mbufs
> and then tcp_output() would limit a TSO segment using both, but I
> can't test
> it, so who knows when/if that might happen.
>
>
>
>
> I think you give that to me in the next email - if not, please send.
>
>
>
> I also have a patch that modifies NFS to use pagesize clusters
> (reducing the
> mbuf count in the list), but that one causes grief when testing on an
> i386
> (seems to run out of kernel memory to the point where it can't
> allocate something
> called "boundary tags" and pretty well wedges the machine at that
> point.)
> Since I don't know how to fix this (I thought of making the patch
> "amd64 only"),
> I can't really commit this to head, either.
>
>
>
>
> Send me that one too. I love NFS patches.
>
>
>
> As such, I think it's going to be "fix the drivers one at a time" and
> tell
> folks to "disable TSO or limit rsize,wsize to 32K" when they run into
> trouble.
> (As you might have guessed, I'd rather just be "the NFS guy", but
> since NFS
> "triggers the problem" I\m kinda stuck with it;-)
>
>
>
> I know in some circumstances disabling TSO can be a benefit, but in
> general you'd want it on a modern system with heavy data load.
>
>
>
>
> > Also should we not also subtract ETHER_VLAN_ENCAP_LEN from tsomax
> > to
> > make sure VLANs fit?
> >
> No idea. (I wouldn't know a VLAN if it jumped up and tried to
> bite me on the nose.;-) So, I have no idea what does this, but
> if it means the total ethernet header size can be > 14bytes, then I'd
> agree.
>
>
>
> Yeah, you need another 4 bytes for VLAN header if you're not using
> hardware that strips it before the TCP stack gets it. I have a mix
> of hardware and software VLANs running on our backbone, mostly due
> to a mixed FreeBSD/OpenBSD/Windows environment.
>
>
>
>
> > Perhaps there is something in the newer network code that is
> > filling
> > up the frames to the point where they are full - thus a TSO =
> > IP_MAXPACKET is just now causing problems.
> >
> Yea, I have no idea why this didn't bite running 9.1. (Did 9.1 have
> TSO enabled by default?)
>
>
>
> I believe 9.0 has TSO on by default.. I seem to recall it always
> being there, but I can't easily confirm it now. My last 9.0-STABLE
> doesn't have an ixgbe card in it.
>
>
>
>
Ok, I've attached 3 patches:
ixgbe.patch - A slightly updated version of the one that sets if_hw_tsomax,
which subtracts out the additional 4bytes for the VLAN header.
*** If you can test this, it would be nice to know if this gets rid of all
the EFBIG replies, since I think Jack might feel it is ok to commit if
it does do so.
4kmcl.patch - This one modifies NFS to use pagesize mbuf clusters for the
large RPC messages. It is NOT safe to use on a small i386,
but might be ok on a large amd64 box. On a small i386, using
a mix of 2K and 4K mbuf clusters seems to fragment kernel memory
enough that allocation of "boundary tags" (whatever those are?)
fail and this trainwrecks the system.
Using pagesize (4K) clusters reduces the mbuf count for an
IP_MAXPACKET sized TSO segment to 19, avoiding the 32 limit
and any need to call m_defrag() for NFS.
*** Only use on a test system, at your own risk.
tsomaxseg.patch - This one adds support for if_hw_tsomaxseg, which is a limit on
the # of mbufs in an output TSO segment (and defaults to 32).
*** This one HAS NOT BEEN TESTED and probably doesn't even work at this point.
rick
------=_Part_1670344_208375117.1395619029041
Content-Type: text/x-patch; name=ixgbe.patch
Content-Disposition: attachment; filename=ixgbe.patch
Content-Transfer-Encoding: base64
LS0tIGRldi9peGdiZS9peGdiZS5jLnNhdgkyMDE0LTAzLTE5IDE3OjQ0OjM0LjAwMDAwMDAwMCAt
MDQwMAorKysgZGV2L2l4Z2JlL2l4Z2JlLmMJMjAxNC0wMy0yMiAyMjo0NDo1My4wMDAwMDAwMDAg
LTA0MDAKQEAgLTI2MTQsNiArMjYxNCwxMCBAQCBpeGdiZV9zZXR1cF9pbnRlcmZhY2UoZGV2aWNl
X3QgZGV2LCBzdHJ1CiAJaWZwLT5pZl9zbmQuaWZxX2Rydl9tYXhsZW4gPSBhZGFwdGVyLT5udW1f
dHhfZGVzYyAtIDI7CiAJSUZRX1NFVF9SRUFEWSgmaWZwLT5pZl9zbmQpOwogI2VuZGlmCisJaWYg
KChhZGFwdGVyLT5udW1fc2VncyAqIE1DTEJZVEVTIC0gKEVUSEVSX0hEUl9MRU4gKworCSAgICBF
VEhFUl9WTEFOX0VOQ0FQX0xFTikpIDwgSVBfTUFYUEFDS0VUKQorCQlpZnAtPmlmX2h3X3Rzb21h
eCA9IGFkYXB0ZXItPm51bV9zZWdzICogTUNMQllURVMgLQorCQkgICAgKEVUSEVSX0hEUl9MRU4g
KyBFVEhFUl9WTEFOX0VOQ0FQX0xFTik7CiAKIAlldGhlcl9pZmF0dGFjaChpZnAsIGFkYXB0ZXIt
Pmh3Lm1hYy5hZGRyKTsKIAo=
------=_Part_1670344_208375117.1395619029041
Content-Type: text/x-patch; name=4kmcl.patch
Content-Disposition: attachment; filename=4kmcl.patch
Content-Transfer-Encoding: base64
LS0tIGZzL25mc3NlcnZlci9uZnNfbmZzZHBvcnQuYy5zYXYyCTIwMTQtMDEtMjYgMTg6NTQ6Mjku
MDAwMDAwMDAwIC0wNTAwCisrKyBmcy9uZnNzZXJ2ZXIvbmZzX25mc2Rwb3J0LmMJMjAxNC0wMy0x
NiAyMzoyMjo1Ni4wMDAwMDAwMDAgLTA0MDAKQEAgLTU2Niw4ICs1NjYsNyBAQCBuZnN2bm9fcmVh
ZGxpbmsoc3RydWN0IHZub2RlICp2cCwgc3RydWN0CiAJbGVuID0gMDsKIAlpID0gMDsKIAl3aGls
ZSAobGVuIDwgTkZTX01BWFBBVEhMRU4pIHsKLQkJTkZTTUdFVChtcCk7Ci0JCU1DTEdFVChtcCwg
TV9XQUlUT0spOworCQlORlNNQ0xHRVQobXAsIE1fTk9XQUlUKTsKIAkJbXAtPm1fbGVuID0gTkZT
TVNJWihtcCk7CiAJCWlmIChsZW4gPT0gMCkgewogCQkJbXAzID0gbXAyID0gbXA7CkBAIC02MjEs
NyArNjIwLDcgQEAgbmZzdm5vX3JlYWQoc3RydWN0IHZub2RlICp2cCwgb2ZmX3Qgb2ZmLAogICAg
IHN0cnVjdCB0aHJlYWQgKnAsIHN0cnVjdCBtYnVmICoqbXBwLCBzdHJ1Y3QgbWJ1ZiAqKm1wZW5k
cCkKIHsKIAlzdHJ1Y3QgbWJ1ZiAqbTsKLQlpbnQgaTsKKwlpbnQgZG9fcGFnZXNpemUsIGk7CiAJ
c3RydWN0IGlvdmVjICppdjsKIAlzdHJ1Y3QgaW92ZWMgKml2MjsKIAlpbnQgZXJyb3IgPSAwLCBs
ZW4sIGxlZnQsIHNpeiwgdGxlbiwgaW9mbGFnID0gMDsKQEAgLTYzMCwxNCArNjI5LDMzIEBAIG5m
c3Zub19yZWFkKHN0cnVjdCB2bm9kZSAqdnAsIG9mZl90IG9mZiwKIAlzdHJ1Y3QgbmZzaGV1ciAq
bmg7CiAKIAlsZW4gPSBsZWZ0ID0gTkZTTV9STkRVUChjbnQpOworCWRvX3BhZ2VzaXplID0gMDsK
KyNpZiBNSlVNUEFHRVNJWkUgIT0gTUNMQllURVMKKwlpZiAobGVmdCA+IE1DTEJZVEVTKQorCQlk
b19wYWdlc2l6ZSA9IDE7CisjZW5kaWYKIAltMyA9IE5VTEw7CiAJLyoKIAkgKiBHZW5lcmF0ZSB0
aGUgbWJ1ZiBsaXN0IHdpdGggdGhlIHVpb19pb3YgcmVmLiB0byBpdC4KIAkgKi8KIAlpID0gMDsK
IAl3aGlsZSAobGVmdCA+IDApIHsKLQkJTkZTTUdFVChtKTsKLQkJTUNMR0VUKG0sIE1fV0FJVE9L
KTsKKwkJLyoKKwkJICogRm9yIGxhcmdlIHJlYWRzLCB0cnkgYW5kIGFjcXVpcmUgTUpVTVBBR0VT
SVpFIGNsdXN0ZXJzLgorCQkgKiBIb3dldmVyLCBkbyBzbyB3aXRoIE1fTk9XQUlUIHNvIHRoZSB0
aHJlYWQgY2FuJ3QgZ2V0CisJCSAqIHN0dWNrIHNsZWVwaW5nIG9uICJidGFsbG9jIi4KKwkJICog
SWYgdGhpcyBmYWlscywgdXNlIE5GU01DTEdFVCguLk1fTk9XQUlUKSwgd2hpY2ggZG9lcyBhbgor
CQkgKiBNR0VUKC4uTV9XQUlUT0spIGZvbGxvd2VkIGJ5IGEgTUNMR0VUKC4uTV9OT1dBSVQpLiAg
VGhlCisJCSAqIE1DTEdFVCguLk1fTk9XQUlUKSBtYXkgbm90IGdldCBhIGNsdXN0ZXIsIGJ1dCB3
aWxsIGRyYWluCisJCSAqIHRoZSBtYnVmIGNsdXN0ZXIgem9uZSB3aGVuIGl0IGZhaWxzLgorCQkg
KiBBcyBzdWNoLCBhbiBtYnVmIHdpbGwgYWx3YXlzIGJlIGFsbG9jYXRlZCBhbmQgbW9zdCBsaWtl
bHkKKwkJICogaXQgd2lsbCBoYXZlIGEgY2x1c3Rlci4KKwkJICovCisJCW0gPSBOVUxMOworCQlp
ZiAoZG9fcGFnZXNpemUgIT0gMCkKKwkJCW0gPSBtX2dldGpjbChNX05PV0FJVCwgTVRfREFUQSwg
MCwgTUpVTVBBR0VTSVpFKTsKKwkJaWYgKG0gPT0gTlVMTCkKKwkJCU5GU01DTEdFVChtLCBNX05P
V0FJVCk7CiAJCW0tPm1fbGVuID0gMDsKIAkJc2l6ID0gbWluKE1fVFJBSUxJTkdTUEFDRShtKSwg
bGVmdCk7CiAJCWxlZnQgLT0gc2l6OwpAQCAtMTY1MywxMCArMTY3MSwxMCBAQCBhZ2FpbjoKIAlp
ZiAoc2l6ID09IDApIHsKIAkJdnB1dCh2cCk7CiAJCWlmIChuZC0+bmRfZmxhZyAmIE5EX05GU1Yy
KSB7Ci0JCQlORlNNX0JVSUxEKHRsLCB1X2ludDMyX3QgKiwgMiAqIE5GU1hfVU5TSUdORUQpOwor
CQkJTkZTTV9CVUlMRF9QQUdFTUJDTCh0bCwgdV9pbnQzMl90ICosIDIgKiBORlNYX1VOU0lHTkVE
KTsKIAkJfSBlbHNlIHsKIAkJCW5mc3J2X3Bvc3RvcGF0dHIobmQsIGdldHJldCwgJmF0KTsKLQkJ
CU5GU01fQlVJTEQodGwsIHVfaW50MzJfdCAqLCA0ICogTkZTWF9VTlNJR05FRCk7CisJCQlORlNN
X0JVSUxEX1BBR0VNQkNMKHRsLCB1X2ludDMyX3QgKiwgNCAqIE5GU1hfVU5TSUdORUQpOwogCQkJ
dHhkcl9oeXBlcihhdC5uYV9maWxlcmV2LCB0bCk7CiAJCQl0bCArPSAyOwogCQl9CkBAIC0xNzA4
LDcgKzE3MjYsNyBAQCBhZ2FpbjoKIAkgKi8KIAlpZiAobmQtPm5kX2ZsYWcgJiBORF9ORlNWMykg
ewogCQluZnNydl9wb3N0b3BhdHRyKG5kLCBnZXRyZXQsICZhdCk7Ci0JCU5GU01fQlVJTEQodGws
IHVfaW50MzJfdCAqLCAyICogTkZTWF9VTlNJR05FRCk7CisJCU5GU01fQlVJTERfUEFHRU1CQ0wo
dGwsIHVfaW50MzJfdCAqLCAyICogTkZTWF9VTlNJR05FRCk7CiAJCXR4ZHJfaHlwZXIoYXQubmFf
ZmlsZXJldiwgdGwpOwogCQlkaXJsZW4gPSBORlNYX1YzUE9TVE9QQVRUUiArIE5GU1hfVkVSRiAr
IDIgKiBORlNYX1VOU0lHTkVEOwogCX0gZWxzZSB7CkBAIC0xNzM0LDIwICsxNzUyLDI0IEBAIGFn
YWluOgogCQkJICogdGhlIGRpcmVudCBlbnRyeS4KIAkJCSAqLwogCQkJaWYgKG5kLT5uZF9mbGFn
ICYgTkRfTkZTVjMpIHsKLQkJCQlORlNNX0JVSUxEKHRsLCB1X2ludDMyX3QgKiwgMyAqIE5GU1hf
VU5TSUdORUQpOworCQkJCU5GU01fQlVJTERfUEFHRU1CQ0wodGwsIHVfaW50MzJfdCAqLAorCQkJ
CSAgICAzICogTkZTWF9VTlNJR05FRCk7CiAJCQkJKnRsKysgPSBuZXduZnNfdHJ1ZTsKIAkJCQkq
dGwrKyA9IDA7CiAJCQl9IGVsc2UgewotCQkJCU5GU01fQlVJTEQodGwsIHVfaW50MzJfdCAqLCAy
ICogTkZTWF9VTlNJR05FRCk7CisJCQkJTkZTTV9CVUlMRF9QQUdFTUJDTCh0bCwgdV9pbnQzMl90
ICosCisJCQkJICAgIDIgKiBORlNYX1VOU0lHTkVEKTsKIAkJCQkqdGwrKyA9IG5ld25mc190cnVl
OwogCQkJfQogCQkJKnRsID0gdHhkcl91bnNpZ25lZChkcC0+ZF9maWxlbm8pOwogCQkJKHZvaWQp
IG5mc21fc3RydG9tKG5kLCBkcC0+ZF9uYW1lLCBubGVuKTsKIAkJCWlmIChuZC0+bmRfZmxhZyAm
IE5EX05GU1YzKSB7Ci0JCQkJTkZTTV9CVUlMRCh0bCwgdV9pbnQzMl90ICosIDIgKiBORlNYX1VO
U0lHTkVEKTsKKwkJCQlORlNNX0JVSUxEX1BBR0VNQkNMKHRsLCB1X2ludDMyX3QgKiwKKwkJCQkg
ICAgMiAqIE5GU1hfVU5TSUdORUQpOwogCQkJCSp0bCsrID0gMDsKIAkJCX0gZWxzZQotCQkJCU5G
U01fQlVJTEQodGwsIHVfaW50MzJfdCAqLCBORlNYX1VOU0lHTkVEKTsKKwkJCQlORlNNX0JVSUxE
X1BBR0VNQkNMKHRsLCB1X2ludDMyX3QgKiwKKwkJCQkgICAgTkZTWF9VTlNJR05FRCk7CiAJCQkq
dGwgPSB0eGRyX3Vuc2lnbmVkKCpjb29raWVwKTsKIAkJfQogCQljcG9zICs9IGRwLT5kX3JlY2xl
bjsKQEAgLTE3NTcsNyArMTc3OSw3IEBAIGFnYWluOgogCX0KIAlpZiAoY3BvcyA8IGNlbmQpCiAJ
CWVvZmZsYWcgPSAwOwotCU5GU01fQlVJTEQodGwsIHVfaW50MzJfdCAqLCAyICogTkZTWF9VTlNJ
R05FRCk7CisJTkZTTV9CVUlMRF9QQUdFTUJDTCh0bCwgdV9pbnQzMl90ICosIDIgKiBORlNYX1VO
U0lHTkVEKTsKIAkqdGwrKyA9IG5ld25mc19mYWxzZTsKIAlpZiAoZW9mZmxhZykKIAkJKnRsID0g
bmV3bmZzX3RydWU7CkBAIC0xOTI4LDcgKzE5NTAsNyBAQCBhZ2FpbjoKIAkJdnB1dCh2cCk7CiAJ
CWlmIChuZC0+bmRfZmxhZyAmIE5EX05GU1YzKQogCQkJbmZzcnZfcG9zdG9wYXR0cihuZCwgZ2V0
cmV0LCAmYXQpOwotCQlORlNNX0JVSUxEKHRsLCB1X2ludDMyX3QgKiwgNCAqIE5GU1hfVU5TSUdO
RUQpOworCQlORlNNX0JVSUxEX1BBR0VNQkNMKHRsLCB1X2ludDMyX3QgKiwgNCAqIE5GU1hfVU5T
SUdORUQpOwogCQl0eGRyX2h5cGVyKGF0Lm5hX2ZpbGVyZXYsIHRsKTsKIAkJdGwgKz0gMjsKIAkJ
KnRsKysgPSBuZXduZnNfZmFsc2U7CkBAIC0yMDMxLDcgKzIwNTMsNyBAQCBhZ2FpbjoKIAl9IGVs
c2UgewogCQlkaXJsZW4gPSBORlNYX1ZFUkYgKyAyICogTkZTWF9VTlNJR05FRDsKIAl9Ci0JTkZT
TV9CVUlMRCh0bCwgdV9pbnQzMl90ICosIE5GU1hfVkVSRik7CisJTkZTTV9CVUlMRF9QQUdFTUJD
TCh0bCwgdV9pbnQzMl90ICosIE5GU1hfVkVSRik7CiAJdHhkcl9oeXBlcihhdC5uYV9maWxlcmV2
LCB0bCk7CiAKIAkvKgpAQCAtMjE4NiwxMiArMjIwOCwxNCBAQCBhZ2FpbjoKIAkJCSAqIEJ1aWxk
IHRoZSBkaXJlY3RvcnkgcmVjb3JkIHhkcgogCQkJICovCiAJCQlpZiAobmQtPm5kX2ZsYWcgJiBO
RF9ORlNWMykgewotCQkJCU5GU01fQlVJTEQodGwsIHVfaW50MzJfdCAqLCAzICogTkZTWF9VTlNJ
R05FRCk7CisJCQkJTkZTTV9CVUlMRF9QQUdFTUJDTCh0bCwgdV9pbnQzMl90ICosCisJCQkJICAg
IDMgKiBORlNYX1VOU0lHTkVEKTsKIAkJCQkqdGwrKyA9IG5ld25mc190cnVlOwogCQkJCSp0bCsr
ID0gMDsKIAkJCQkqdGwgPSB0eGRyX3Vuc2lnbmVkKGRwLT5kX2ZpbGVubyk7CiAJCQkJZGlybGVu
ICs9IG5mc21fc3RydG9tKG5kLCBkcC0+ZF9uYW1lLCBubGVuKTsKLQkJCQlORlNNX0JVSUxEKHRs
LCB1X2ludDMyX3QgKiwgMiAqIE5GU1hfVU5TSUdORUQpOworCQkJCU5GU01fQlVJTERfUEFHRU1C
Q0wodGwsIHVfaW50MzJfdCAqLAorCQkJCSAgICAyICogTkZTWF9VTlNJR05FRCk7CiAJCQkJKnRs
KysgPSAwOwogCQkJCSp0bCA9IHR4ZHJfdW5zaWduZWQoKmNvb2tpZXApOwogCQkJCW5mc3J2X3Bv
c3RvcGF0dHIobmQsIDAsIG52YXApOwpAQCAtMjIwMCw3ICsyMjI0LDggQEAgYWdhaW46CiAJCQkJ
aWYgKG52cCAhPSBOVUxMKQogCQkJCQl2cHV0KG52cCk7CiAJCQl9IGVsc2UgewotCQkJCU5GU01f
QlVJTEQodGwsIHVfaW50MzJfdCAqLCAzICogTkZTWF9VTlNJR05FRCk7CisJCQkJTkZTTV9CVUlM
RF9QQUdFTUJDTCh0bCwgdV9pbnQzMl90ICosCisJCQkJICAgIDMgKiBORlNYX1VOU0lHTkVEKTsK
IAkJCQkqdGwrKyA9IG5ld25mc190cnVlOwogCQkJCSp0bCsrID0gMDsKIAkJCQkqdGwgPSB0eGRy
X3Vuc2lnbmVkKCpjb29raWVwKTsKQEAgLTIyNjcsNyArMjI5Miw3IEBAIGFnYWluOgogCX0gZWxz
ZSBpZiAoY3BvcyA8IGNlbmQpCiAJCWVvZmZsYWcgPSAwOwogCWlmICghbmQtPm5kX3JlcHN0YXQp
IHsKLQkJTkZTTV9CVUlMRCh0bCwgdV9pbnQzMl90ICosIDIgKiBORlNYX1VOU0lHTkVEKTsKKwkJ
TkZTTV9CVUlMRF9QQUdFTUJDTCh0bCwgdV9pbnQzMl90ICosIDIgKiBORlNYX1VOU0lHTkVEKTsK
IAkJKnRsKysgPSBuZXduZnNfZmFsc2U7CiAJCWlmIChlb2ZmbGFnKQogCQkJKnRsID0gbmV3bmZz
X3RydWU7Ci0tLSBmcy9uZnNjbGllbnQvbmZzX2NsY29tc3Vicy5jLnNhdjIJMjAxNC0wMi0wMSAy
MDo0NzowNy4wMDAwMDAwMDAgLTA1MDAKKysrIGZzL25mc2NsaWVudC9uZnNfY2xjb21zdWJzLmMJ
MjAxNC0wMy0xNiAyMzoyMjowNi4wMDAwMDAwMDAgLTA0MDAKQEAgLTE1NSw3ICsxNTUsNyBAQCBu
ZnNjbF9yZXFzdGFydChzdHJ1Y3QgbmZzcnZfZGVzY3JpcHQgKm5kCiAJICogR2V0IHRoZSBmaXJz
dCBtYnVmIGZvciB0aGUgcmVxdWVzdC4KIAkgKi8KIAlpZiAobmZzX2JpZ3JlcXVlc3RbcHJvY251
bV0pCi0JCU5GU01DTEdFVChtYiwgTV9XQUlUT0spOworCQlORlNNQ0xHRVQobWIsIE1fTk9XQUlU
KTsKIAllbHNlCiAJCU5GU01HRVQobWIpOwogCW1idWZfc2V0bGVuKG1iLCAwKTsKQEAgLTI2Nyw5
ICsyNjcsMjkgQEAgbmZzbV91aW9tYnVmKHN0cnVjdCBuZnNydl9kZXNjcmlwdCAqbmQsIAogCQl3
aGlsZSAobGVmdCA+IDApIHsKIAkJCW1sZW4gPSBNX1RSQUlMSU5HU1BBQ0UobXApOwogCQkJaWYg
KG1sZW4gPT0gMCkgewotCQkJCWlmIChjbGZsZykKLQkJCQkJTkZTTUNMR0VUKG1wLCBNX1dBSVRP
Syk7Ci0JCQkJZWxzZQorCQkJCWlmIChjbGZsZyAhPSAwKSB7CisJCQkJCS8qCisJCQkJCSAqIEZv
ciBsYXJnZSB3cml0ZXMsIHRyeSBhbmQgYWNxdWlyZQorCQkJCQkgKiBNSlVNUEFHRVNJWkUgY2x1
c3RlcnMuCisJCQkJCSAqIEhvd2V2ZXIsIGRvIHNvIHdpdGggTV9OT1dBSVQgc28gdGhlCisJCQkJ
CSAqIHRocmVhZCBjYW4ndCBnZXQgc3R1Y2sgc2xlZXBpbmcgb24KKwkJCQkJICogImJ0YWxsb2Mi
LiAgSWYgdGhpcyBmYWlscywgdXNlCisJCQkJCSAqIE5GU01DTEdFVCguLk1fTk9XQUlUKSwgd2hp
Y2ggZG9lcyBhbgorCQkJCQkgKiBNR0VUKC4uTV9XQUlUT0spIGZvbGxvd2VkIGJ5IGEKKwkJCQkJ
ICogTUNMR0UgVCguLk1fTk9XQUlUKS4gVGhpcyBtYXkgbm90IGdldAorCQkJCQkgKiBhIGNsdXN0
ZXIsIGJ1dCB3aWxsIGRyYWluIHRoZSBtYnVmCisJCQkJCSAqIGNsdXN0ZXIgem9uZSB3aGVuIGl0
IGZhaWxzLgorCQkJCQkgKiBBcyBzdWNoLCBhbiBtYnVmIHdpbGwgYWx3YXlzIGJlCisJCQkJCSAq
IGFsbG9jYXRlZCBhbmQgbW9zdCBsaWtlbHkgaXQgd2lsbAorCQkJCQkgKiBoYXZlIGEgY2x1c3Rl
ci4KKwkJCQkJICovCisjaWYgTUpVTVBBR0VTSVpFICE9IE1DTEJZVEVTCisJCQkJCW1wID0gbV9n
ZXRqY2woTV9OT1dBSVQsIE1UX0RBVEEsIDAsCisJCQkJCSAgICBNSlVNUEFHRVNJWkUpOworCQkJ
CQlpZiAobXAgPT0gTlVMTCkKKyNlbmRpZgorCQkJCQkJTkZTTUNMR0VUKG1wLCBNX05PV0FJVCk7
CisJCQkJfSBlbHNlCiAJCQkJCU5GU01HRVQobXApOwogCQkJCW1idWZfc2V0bGVuKG1wLCAwKTsK
IAkJCQltYnVmX3NldG5leHQobXAyLCBtcCk7Ci0tLSBmcy9uZnMvbmZzbV9zdWJzLmguc2F2Mgky
MDE0LTAyLTAxIDE5OjUxOjEyLjAwMDAwMDAwMCAtMDUwMAorKysgZnMvbmZzL25mc21fc3Vicy5o
CTIwMTQtMDMtMTMgMTg6NTQ6MjcuMDAwMDAwMDAwIC0wNDAwCkBAIC04OSw2ICs4OSwzNyBAQCBu
ZnNtX2J1aWxkKHN0cnVjdCBuZnNydl9kZXNjcmlwdCAqbmQsIGluCiAKICNkZWZpbmUJTkZTTV9C
VUlMRChhLCBjLCBzKQkoKGEpID0gKGMpbmZzbV9idWlsZChuZCwgKHMpKSkKIAorLyoKKyAqIFNh
bWUgYXMgYWJvdmUsIGJ1dCBhbGxvY2F0ZXMgTUpVTVBBR0VTSVpFIG1idWYgY2x1c3RlcnMsIGlm
IHBvc3NpYmxlLgorICovCitzdGF0aWMgX19pbmxpbmUgdm9pZCAqCituZnNtX2J1aWxkX3BhZ2Vt
YmNsKHN0cnVjdCBuZnNydl9kZXNjcmlwdCAqbmQsIGludCBzaXopCit7CisJdm9pZCAqcmV0cDsK
KwlzdHJ1Y3QgbWJ1ZiAqbWIyOworCisJaWYgKHNpeiA+IE1fVFJBSUxJTkdTUEFDRShuZC0+bmRf
bWIpKSB7CisJCW1iMiA9IE5VTEw7CisjaWYgTUpVTVBBR0VTSVpFICE9IE1DTEJZVEVTCisJCW1i
MiA9IG1fZ2V0amNsKE1fTk9XQUlULCBNVF9EQVRBLCAwLCBNSlVNUEFHRVNJWkUpOworI2VuZGlm
CisJCWlmIChtYjIgPT0gTlVMTCkKKwkJCU5GU01DTEdFVChtYjIsIE1fTk9XQUlUKTsKKwkJaWYg
KHNpeiA+IE1MRU4pCisJCQlwYW5pYygiYnVpbGQgPiBNTEVOIik7CisJCW1idWZfc2V0bGVuKG1i
MiwgMCk7CisJCW5kLT5uZF9icG9zID0gTkZTTVRPRChtYjIsIGNhZGRyX3QpOworCQluZC0+bmRf
bWItPm1fbmV4dCA9IG1iMjsKKwkJbmQtPm5kX21iID0gbWIyOworCX0KKwlyZXRwID0gKHZvaWQg
KikobmQtPm5kX2Jwb3MpOworCW5kLT5uZF9tYi0+bV9sZW4gKz0gc2l6OworCW5kLT5uZF9icG9z
ICs9IHNpejsKKwlyZXR1cm4gKHJldHApOworfQorCisjZGVmaW5lCU5GU01fQlVJTERfUEFHRU1C
Q0woYSwgYywgcykJKChhKSA9IChjKW5mc21fYnVpbGRfcGFnZW1iY2wobmQsIChzKSkpCisKIHN0
YXRpYyBfX2lubGluZSB2b2lkICoKIG5mc21fZGlzc2VjdChzdHJ1Y3QgbmZzcnZfZGVzY3JpcHQg
Km5kLCBpbnQgc2l6KQogewotLS0gZnMvbmZzL25mc3BvcnQuaC5zYXYyCTIwMTQtMDItMTMgMTk6
MDM6MjIuMDAwMDAwMDAwIC0wNTAwCisrKyBmcy9uZnMvbmZzcG9ydC5oCTIwMTQtMDItMTMgMTk6
MTQ6MjQuMDAwMDAwMDAwIC0wNTAwCkBAIC0xMzgsNiArMTM4LDggQEAKIAogLyoKICAqIEFsbG9j
YXRlIG1idWZzLiBNdXN0IHN1Y2NlZWQgYW5kIG5ldmVyIHNldCB0aGUgbWJ1ZiBwdHIgdG8gTlVM
TC4KKyAqIE5vdGUgdGhhdCB3aGVuIE5GU01DTEdFVChtLCBNX05PV0FJVCkgaXMgZG9uZSwgaXQg
c3RpbGwgbXVzdCBhbGxvY2F0ZQorICogYW4gbWJ1ZiAoYW5kIGNhbiBzbGVlcCksIGJ1dCBtaWdo
dCBub3QgZ2V0IGEgY2x1c3RlciwgaW4gdGhlIHdvcnN0IGNhc2UuCiAgKi8KICNkZWZpbmUJTkZT
TUdFVChtKQlkbyB7IAkJCQkJXAogCQlNR0VUKChtKSwgTV9XQUlUT0ssIE1UX0RBVEEpOyAJCQlc
Cg==
------=_Part_1670344_208375117.1395619029041
Content-Type: text/x-patch; name=tsomaxseg.patch
Content-Disposition: attachment; filename=tsomaxseg.patch
Content-Transfer-Encoding: base64
LS0tIGtlcm4vdWlwY19zb2NrYnVmLmMuc2F2CTIwMTQtMDEtMzAgMjA6Mjc6MTcuMDAwMDAwMDAw
IC0wNTAwCisrKyBrZXJuL3VpcGNfc29ja2J1Zi5jCTIwMTQtMDEtMzAgMjI6MTI6MDguMDAwMDAw
MDAwIC0wNTAwCkBAIC05NjUsNiArOTY1LDM5IEBAIHNic25kcHRyKHN0cnVjdCBzb2NrYnVmICpz
YiwgdV9pbnQgb2ZmLCAKIH0KIAogLyoKKyAqIFJldHVybiB0aGUgZmlyc3QgbWJ1ZiBmb3IgdGhl
IHByb3ZpZGVkIG9mZnNldC4KKyAqLworc3RydWN0IG1idWYgKgorc2JzbmRtYnVmKHN0cnVjdCBz
b2NrYnVmICpzYiwgdV9pbnQgb2ZmLCBsb25nICpmaXJzdF9sZW4pCit7CisJc3RydWN0IG1idWYg
Km07CisKKwlLQVNTRVJUKHNiLT5zYl9tYiAhPSBOVUxMLCAoIiVzOiBzYl9tYiBpcyBOVUxMIiwg
X19mdW5jX18pKTsKKworCSpmaXJzdF9sZW4gPSAwOworCS8qCisJICogSXMgb2ZmIGJlbG93IHN0
b3JlZCBvZmZzZXQ/IEhhcHBlbnMgb24gcmV0cmFuc21pdHMuCisJICogSWYgc28sIGp1c3QgdXNl
IHNiX21iLgorCSAqLworCWlmIChzYi0+c2Jfc25kcHRyID09IE5VTEwgfHwgc2ItPnNiX3NuZHB0
cm9mZiA+IG9mZikKKwkJbSA9IHNiLT5zYl9tYjsKKwllbHNlIHsKKwkJbSA9IHNiLT5zYl9zbmRw
dHI7CisJCW9mZiAtPSBzYi0+c2Jfc25kcHRyb2ZmOworCX0KKwl3aGlsZSAob2ZmID4gMCAmJiBt
ICE9IE5VTEwpIHsKKwkJaWYgKG9mZiA8IG0tPm1fbGVuKQorCQkJYnJlYWs7CisJCW9mZiAtPSBt
LT5tX2xlbjsKKwkJbSA9IG0tPm1fbmV4dDsKKwl9CisJaWYgKG0gIT0gTlVMTCkKKwkJKmZpcnN0
X2xlbiA9IG0tPm1fbGVuIC0gb2ZmOworCisJcmV0dXJuIChtKTsKK30KKworLyoKICAqIERyb3Ag
YSByZWNvcmQgb2ZmIHRoZSBmcm9udCBvZiBhIHNvY2tidWYgYW5kIG1vdmUgdGhlIG5leHQgcmVj
b3JkIHRvIHRoZQogICogZnJvbnQuCiAgKi8KLS0tIHN5cy9zb2NrYnVmLmguc2F2CTIwMTQtMDEt
MzAgMjA6NDI6MjguMDAwMDAwMDAwIC0wNTAwCisrKyBzeXMvc29ja2J1Zi5oCTIwMTQtMDEtMzAg
MjI6MDg6NDMuMDAwMDAwMDAwIC0wNTAwCkBAIC0xNTMsNiArMTUzLDggQEAgaW50CXNicmVzZXJ2
ZV9sb2NrZWQoc3RydWN0IHNvY2tidWYgKnNiLAogCSAgICBzdHJ1Y3QgdGhyZWFkICp0ZCk7CiBz
dHJ1Y3QgbWJ1ZiAqCiAJc2JzbmRwdHIoc3RydWN0IHNvY2tidWYgKnNiLCB1X2ludCBvZmYsIHVf
aW50IGxlbiwgdV9pbnQgKm1vZmYpOworc3RydWN0IG1idWYgKgorCXNic25kbWJ1ZihzdHJ1Y3Qg
c29ja2J1ZiAqc2IsIHVfaW50IG9mZiwgbG9uZyAqZmlyc3RfbGVuKTsKIHZvaWQJc2J0b3hzb2Nr
YnVmKHN0cnVjdCBzb2NrYnVmICpzYiwgc3RydWN0IHhzb2NrYnVmICp4c2IpOwogaW50CXNid2Fp
dChzdHJ1Y3Qgc29ja2J1ZiAqc2IpOwogaW50CXNibG9jayhzdHJ1Y3Qgc29ja2J1ZiAqc2IsIGlu
dCBmbGFncyk7Ci0tLSBuZXRpbmV0L3RjcF9pbnB1dC5jLnNhdgkyMDE0LTAxLTMwIDE5OjM3OjUy
LjAwMDAwMDAwMCAtMDUwMAorKysgbmV0aW5ldC90Y3BfaW5wdXQuYwkyMDE0LTAxLTMwIDE5OjM5
OjA3LjAwMDAwMDAwMCAtMDUwMApAQCAtMzYyNyw2ICszNjI3LDcgQEAgdGNwX21zcyhzdHJ1Y3Qg
dGNwY2IgKnRwLCBpbnQgb2ZmZXIpCiAJaWYgKGNhcC5pZmNhcCAmIENTVU1fVFNPKSB7CiAJCXRw
LT50X2ZsYWdzIHw9IFRGX1RTTzsKIAkJdHAtPnRfdHNvbWF4ID0gY2FwLnRzb21heDsKKwkJdHAt
PnRfdHNvbWF4c2VncyA9IGNhcC50c29tYXhzZWdzOwogCX0KIH0KIAotLS0gbmV0aW5ldC90Y3Bf
b3V0cHV0LmMuc2F2CTIwMTQtMDEtMzAgMTg6NTU6MTUuMDAwMDAwMDAwIC0wNTAwCisrKyBuZXRp
bmV0L3RjcF9vdXRwdXQuYwkyMDE0LTAxLTMwIDIyOjE4OjU2LjAwMDAwMDAwMCAtMDUwMApAQCAt
MTY2LDggKzE2Niw4IEBAIGludAogdGNwX291dHB1dChzdHJ1Y3QgdGNwY2IgKnRwKQogewogCXN0
cnVjdCBzb2NrZXQgKnNvID0gdHAtPnRfaW5wY2ItPmlucF9zb2NrZXQ7Ci0JbG9uZyBsZW4sIHJl
Y3dpbiwgc2VuZHdpbjsKLQlpbnQgb2ZmLCBmbGFncywgZXJyb3IgPSAwOwkvKiBLZWVwIGNvbXBp
bGVyIGhhcHB5ICovCisJbG9uZyBsZW4sIHJlY3dpbiwgc2VuZHdpbiwgdHNvX3RsZW47CisJaW50
IGNudCwgb2ZmLCBmbGFncywgZXJyb3IgPSAwOwkvKiBLZWVwIGNvbXBpbGVyIGhhcHB5ICovCiAJ
c3RydWN0IG1idWYgKm07CiAJc3RydWN0IGlwICppcCA9IE5VTEw7CiAJc3RydWN0IGlwb3ZseSAq
aXBvdiA9IE5VTEw7CkBAIC03ODAsNiArNzgwLDI0IEBAIHNlbmQ6CiAJCQl9CiAKIAkJCS8qCisJ
CQkgKiBMaW1pdCB0aGUgbnVtYmVyIG9mIFRTTyB0cmFuc21pdCBzZWdtZW50cyAobWJ1ZnMKKwkJ
CSAqIGluIG1idWYgbGlzdCkgdG8gdHAtPnRfdHNvbWF4c2Vncy4KKwkJCSAqLworCQkJY250ID0g
MDsKKwkJCW0gPSBzYnNuZG1idWYoJnNvLT5zb19zbmQsIG9mZiwgJnRzb190bGVuKTsKKwkJCXdo
aWxlIChtICE9IE5VTEwgJiYgY250IDwgdHAtPnRfdHNvbWF4c2VncyAmJgorCQkJICAgIHRzb190
bGVuIDwgbGVuKSB7CisJCQkJaWYgKGNudCA+IDApCisJCQkJCXRzb190bGVuICs9IG0tPm1fbGVu
OworCQkJCWNudCsrOworCQkJCW0gPSBtLT5tX25leHQ7CisJCQl9CisJCQlpZiAobSAhPSBOVUxM
ICYmIHRzb190bGVuIDwgbGVuKSB7CisJCQkJbGVuID0gdHNvX3RsZW47CisJCQkJc2VuZGFsb3Qg
PSAxOworCQkJfQorCisJCQkvKgogCQkJICogUHJldmVudCB0aGUgbGFzdCBzZWdtZW50IGZyb20g
YmVpbmcKIAkJCSAqIGZyYWN0aW9uYWwgdW5sZXNzIHRoZSBzZW5kIHNvY2tidWYgY2FuCiAJCQkg
KiBiZSBlbXB0aWVkLgotLS0gbmV0aW5ldC90Y3Bfc3Vici5jLnNhdgkyMDE0LTAxLTMwIDE5OjQ0
OjM1LjAwMDAwMDAwMCAtMDUwMAorKysgbmV0aW5ldC90Y3Bfc3Vici5jCTIwMTQtMDEtMzAgMjA6
NTY6MTIuMDAwMDAwMDAwIC0wNTAwCkBAIC0xODAwLDYgKzE4MDAsMTIgQEAgdGNwX21heG10dShz
dHJ1Y3QgaW5fY29ubmluZm8gKmluYywgc3RydQogCQkJICAgIGlmcC0+aWZfaHdhc3Npc3QgJiBD
U1VNX1RTTykKIAkJCQljYXAtPmlmY2FwIHw9IENTVU1fVFNPOwogCQkJCWNhcC0+dHNvbWF4ID0g
aWZwLT5pZl9od190c29tYXg7CisjaWZkZWYgbm90eWV0CisJCQkJY2FwLT50c29tYXhzZWdzID0g
aWZwLT5pZl9od190c29tYXhzZWdzOworI2VuZGlmCisJCQkJaWYgKGNhcC0+dHNvbWF4c2VncyA9
PSAwKQorCQkJCQljYXAtPnRzb21heHNlZ3MgPQorCQkJCQkgICAgVENQVFNPX01BWF9UWF9TRUdT
X0RFRkFVTFQ7CiAJCX0KIAkJUlRGUkVFKHNyby5yb19ydCk7CiAJfQotLS0gbmV0aW5ldC90Y3Bf
dmFyLmguc2F2CTIwMTQtMDEtMzAgMTk6Mzk6MjIuMDAwMDAwMDAwIC0wNTAwCisrKyBuZXRpbmV0
L3RjcF92YXIuaAkyMDE0LTAxLTMwIDIwOjUyOjU3LjAwMDAwMDAwMCAtMDUwMApAQCAtMjA5LDYg
KzIwOSw3IEBAIHN0cnVjdCB0Y3BjYiB7CiAJdV9pbnQJdF9rZWVwY250OwkJLyogbnVtYmVyIG9m
IGtlZXBhbGl2ZXMgYmVmb3JlIGNsb3NlICovCiAKIAl1X2ludAl0X3Rzb21heDsJCS8qIHRzbyBi
dXJzdCBsZW5ndGggbGltaXQgKi8KKwl1X2ludAl0X3Rzb21heHNlZ3M7CQkvKiB0c28gYnVyc3Qg
c2VnbWVudCBsaW1pdCAqLwogCiAJdWludDMyX3QgdF9pc3BhcmVbOF07CQkvKiA1IFVUTywgMyBU
QkQgKi8KIAl2b2lkCSp0X3BzcGFyZTJbNF07CQkvKiA0IFRCRCAqLwpAQCAtMjY4LDYgKzI2OSwx
MSBAQCBzdHJ1Y3QgdGNwY2IgewogI2RlZmluZQlUQ1BPT0JfSEFWRURBVEEJMHgwMQogI2RlZmlu
ZQlUQ1BPT0JfSEFEREFUQQkweDAyCiAKKy8qCisgKiBEZWZhdWx0IHZhbHVlIGZvciBUU08gbWF4
aW11bSBudW1iZXIgb2YgdHJhbnNtaXQgc2VnbWVudHMgKGNvdW50IG9mIG1idWZzKS4KKyAqLwor
I2RlZmluZQlUQ1BUU09fTUFYX1RYX1NFR1NfREVGQVVMVAkzMAorCiAjaWZkZWYgVENQX1NJR05B
VFVSRQogLyoKICAqIERlZmluZXMgd2hpY2ggYXJlIG5lZWRlZCBieSB0aGUgeGZvcm1fdGNwIG1v
ZHVsZSBhbmQgdGNwX1tpbnxvdXRdcHV0CkBAIC0zMzMsNiArMzM5LDcgQEAgc3RydWN0IGhjX21l
dHJpY3NfbGl0ZSB7CS8qIG11c3Qgc3RheSBpbgogc3RydWN0IHRjcF9pZmNhcCB7CiAJaW50CWlm
Y2FwOwogCXVfaW50CXRzb21heDsKKwl1X2ludAl0c29tYXhzZWdzOwogfTsKIAogI2lmbmRlZiBf
TkVUSU5FVF9JTl9QQ0JfSF8K
------=_Part_1670344_208375117.1395619029041--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1936201708.1670348.1395619029045.JavaMail.root>