Date: Wed, 8 Oct 2014 17:38:06 +0200 From: Svatopluk Kraus <onwahe@gmail.com> To: Alan Cox <alc@rice.edu> Cc: alc@freebsd.org, FreeBSD Arch <freebsd-arch@freebsd.org> Subject: Re: vm_page_array and VM_PHYSSEG_SPARSE Message-ID: <CAFHCsPWxF0G%2BbqBYgxH=WtV%2BSt_UTWZj%2BY2-PHfoYSLjC_Qpig@mail.gmail.com> In-Reply-To: <5428AF3B.1030906@rice.edu> References: <CAFHCsPWkq09_RRDz7fy3UgsRFv8ZbNKdAH2Ft0x6aVSwLPi6BQ@mail.gmail.com> <CAJUyCcPXBuLu0nvaCqpg8NJ6KzAX9BA1Rt%2BooD%2B3pzq%2BFV%2B%2BTQ@mail.gmail.com> <CAFHCsPWq9WqeFnx1a%2BStfSxj=jwcE9GPyVsoyh0%2Bazr3HmM6vQ@mail.gmail.com> <5428AF3B.1030906@rice.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
--001a1133bf0229b31b0504eb1a08 Content-Type: text/plain; charset=UTF-8 On Mon, Sep 29, 2014 at 3:00 AM, Alan Cox <alc@rice.edu> wrote: > On 09/27/2014 03:51, Svatopluk Kraus wrote: > > > On Fri, Sep 26, 2014 at 8:08 PM, Alan Cox <alan.l.cox@gmail.com> wrote: > >> >> >> On Wed, Sep 24, 2014 at 7:27 AM, Svatopluk Kraus <onwahe@gmail.com> >> wrote: >> >>> Hi, >>> >>> I and Michal are finishing new ARM pmap-v6 code. There is one problem >>> we've >>> dealt with somehow, but now we would like to do it better. It's about >>> physical pages which are allocated before vm subsystem is initialized. >>> While later on these pages could be found in vm_page_array when >>> VM_PHYSSEG_DENSE memory model is used, it's not true for >>> VM_PHYSSEG_SPARSE >>> memory model. And ARM world uses VM_PHYSSEG_SPARSE model. >>> >>> It really would be nice to utilize vm_page_array for such preallocated >>> physical pages even when VM_PHYSSEG_SPARSE memory model is used. Things >>> could be much easier then. In our case, it's about pages which are used >>> for >>> level 2 page tables. In VM_PHYSSEG_SPARSE model, we have two sets of such >>> pages. First ones are preallocated and second ones are allocated after vm >>> subsystem was inited. We must deal with each set differently. So code is >>> more complex and so is debugging. >>> >>> Thus we need some method how to say that some part of physical memory >>> should be included in vm_page_array, but the pages from that region >>> should >>> not be put to free list during initialization. We think that such >>> possibility could be utilized in general. There could be a need for some >>> physical space which: >>> >>> (1) is needed only during boot and later on it can be freed and put to vm >>> subsystem, >>> >>> (2) is needed for something else and vm_page_array code could be used >>> without some kind of its duplication. >>> >>> There is already some code which deals with blacklisted pages in >>> vm_page.c >>> file. So the easiest way how to deal with presented situation is to add >>> some callback to this part of code which will be able to either exclude >>> whole phys_avail[i], phys_avail[i+1] region or single pages. As the >>> biggest >>> phys_avail region is used for vm subsystem allocations, there should be >>> some more coding. (However, blacklisted pages are not dealt with on that >>> part of region.) >>> >>> We would like to know if there is any objection: >>> >>> (1) to deal with presented problem, >>> (2) to deal with the problem presented way. >>> Some help is very appreciated. Thanks >>> >>> >> >> As an experiment, try modifying vm_phys.c to use dump_avail instead of >> phys_avail when sizing vm_page_array. On amd64, where the same problem >> exists, this allowed me to use VM_PHYSSEG_SPARSE. Right now, this is >> probably my preferred solution. The catch being that not all architectures >> implement dump_avail, but my recollection is that arm does. >> > > Frankly, I would prefer this too, but there is one big open question: > > What is dump_avail for? > > > > dump_avail[] is solving a similar problem in the minidump code, hence, the > prefix "dump_" in its name. In other words, the minidump code couldn't use > phys_avail[] either because it didn't describe the full range of physical > addresses that might be included in a minidump, so dump_avail[] was created. > > There is already precedent for what I'm suggesting. dump_avail[] is > already (ab)used outside of the minidump code on x86 to solve this same > problem in x86/x86/nexus.c, and on arm in arm/arm/mem.c. > > > Using it for vm_page_array initialization and segmentation means that > phys_avail must be a subset of it. And this must be stated and be visible > enough. Maybe it should be even checked in code. I like the idea of > thinking about dump_avail as something what desribes all memory in a > system, but it's not how dump_avail is defined in archs now. > > > > When you say "it's not how dump_avail is defined in archs now", I'm not > sure whether you're talking about the code or the comments. In terms of > code, dump_avail[] is a superset of phys_avail[], and I'm not aware of any > code that would have to change. In terms of comments, I did a grep looking > for comments defining what dump_avail[] is, because I couldn't remember > any. I found one ... on arm. So, I don't think it's a onerous task > changing the definition of dump_avail[]. :-) > > Already, as things stand today with dump_avail[] being used outside of the > minidump code, one could reasonably argue that it should be renamed to > something like phys_exists[]. > > > > I will experiment with it on monday then. However, it's not only about how > memory segments are created in vm_phys.c, but it's about how vm_page_array > size is computed in vm_page.c too. > > > > Yes, and there is also a place in vm_reserv.c that needs to change. I've > attached the patch that I developed and tested a long time ago. It still > applies cleanly and runs ok on amd64. > > > Well, I've created and tested minimalistic patch which - I hope - is commitable. It runs ok on pandaboard (arm-v6) and solves presented problem. I would really appreciate if this will be commited. Thanks. BTW, while I was inspecting all archs, I think that maybe it's time to do what was done for busdma not long ago. There are many similar codes across archs which deal with physical memory and could be generalized and put to kern/subr_physmem.c for utilization. All work with physical memory could be simplify to two arrays of regions. phys_present[] ... describes all present physical memory regions phys_exclude[] ... describes various exclusions from phys_present[] Each excluded region will be labeled by flags to say what kind of exclusion it is. The flags like NODUMP, NOALLOC, NOMANAGE, NOBOUNCE, NOMEMRW could be combined. This idea is taken from sys/arm/arm/physmem.c. All other arrays like phys_managed[], phys_avail[], dump_avail[] will be created from these phys_present[] and phys_exclude[]. This way bootstrap codes in archs could be simplified and unified. For example, dealing with either hw.physmem or page with PA 0x00000000 could be transparent. I'm prepared to volunteer if the thing is ripe. However, some tutor will be looked for. Svata --001a1133bf0229b31b0504eb1a08 Content-Type: application/octet-stream; name="phys_managed.patch" Content-Disposition: attachment; filename="phys_managed.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_i10u373a1 SW5kZXg6IHN5cy9hcm0vYXJtL3BtYXAtdjYuYwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMvYXJtL2FybS9w bWFwLXY2LmMJKHJldmlzaW9uIDI3MjY4MikKKysrIHN5cy9hcm0vYXJtL3BtYXAtdjYuYwkod29y a2luZyBjb3B5KQpAQCAtMTM0Myw4ICsxMzQzLDggQEAKIAkvKgogCSAqIENhbGN1bGF0ZSB0aGUg c2l6ZSBvZiB0aGUgcHYgaGVhZCB0YWJsZSBmb3Igc3VwZXJwYWdlcy4KIAkgKi8KLQlmb3IgKGkg PSAwOyBwaHlzX2F2YWlsW2kgKyAxXTsgaSArPSAyKTsKLQlwdl9ucGcgPSByb3VuZF8xbXBhZ2Uo cGh5c19hdmFpbFsoaSAtIDIpICsgMV0pIC8gTkJQRFI7CisJZm9yIChpID0gMDsgcGh5c19tYW5h Z2VkW2kgKyAxXTsgaSArPSAyKTsKKwlwdl9ucGcgPSByb3VuZF8xbXBhZ2UocGh5c19tYW5hZ2Vk WyhpIC0gMikgKyAxXSkgLyBOQlBEUjsKIAogCS8qCiAJICogQWxsb2NhdGUgbWVtb3J5IGZvciB0 aGUgcHYgaGVhZCB0YWJsZSBmb3Igc3VwZXJwYWdlcy4KSW5kZXg6IHN5cy92bS92bV9wYWdlLmMK PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PQotLS0gc3lzL3ZtL3ZtX3BhZ2UuYwkocmV2aXNpb24gMjcyNjgyKQorKysgc3lz L3ZtL3ZtX3BhZ2UuYwkod29ya2luZyBjb3B5KQpAQCAtMTI4LDYgKzEyOCw4IEBACiAKIHN0cnVj dCBtdHhfcGFkYWxpZ24gcGFfbG9ja1tQQV9MT0NLX0NPVU5UXTsKIAordm1fcGFkZHJfdCAqcGh5 c19tYW5hZ2VkOworCiB2bV9wYWdlX3Qgdm1fcGFnZV9hcnJheTsKIGxvbmcgdm1fcGFnZV9hcnJh eV9zaXplOwogbG9uZyBmaXJzdF9wYWdlOwpAQCAtMzAxLDMxICszMDMsNDQgQEAKIAliaWdnZXN0 b25lID0gMDsKIAl2YWRkciA9IHJvdW5kX3BhZ2UodmFkZHIpOwogCi0JZm9yIChpID0gMDsgcGh5 c19hdmFpbFtpICsgMV07IGkgKz0gMikgewotCQlwaHlzX2F2YWlsW2ldID0gcm91bmRfcGFnZShw aHlzX2F2YWlsW2ldKTsKLQkJcGh5c19hdmFpbFtpICsgMV0gPSB0cnVuY19wYWdlKHBoeXNfYXZh aWxbaSArIDFdKTsKKyNpZiBkZWZpbmVkKF9fYXJtX18pCisJcGh5c19tYW5hZ2VkID0gZHVtcF9h dmFpbDsKKyNlbHNlCisJcGh5c19tYW5hZ2VkID0gcGh5c19hdmFpbDsKKyNlbmRpZgkKKworCWZv ciAoaSA9IDA7IHBoeXNfbWFuYWdlZFtpICsgMV07IGkgKz0gMikgeworCQlwaHlzX21hbmFnZWRb aV0gPSByb3VuZF9wYWdlKHBoeXNfbWFuYWdlZFtpXSk7CisJCXBoeXNfbWFuYWdlZFtpICsgMV0g PSB0cnVuY19wYWdlKHBoeXNfbWFuYWdlZFtpICsgMV0pOwogCX0KIAotCWxvd193YXRlciA9IHBo eXNfYXZhaWxbMF07Ci0JaGlnaF93YXRlciA9IHBoeXNfYXZhaWxbMV07CisJbG93X3dhdGVyID0g cGh5c19tYW5hZ2VkWzBdOworCWhpZ2hfd2F0ZXIgPSBwaHlzX21hbmFnZWRbMV07CiAKKwlmb3Ig KGkgPSAwOyBwaHlzX21hbmFnZWRbaSArIDFdOyBpICs9IDIpIHsKKwkJaWYgKHBoeXNfbWFuYWdl ZFtpXSA8IGxvd193YXRlcikKKwkJCWxvd193YXRlciA9IHBoeXNfbWFuYWdlZFtpXTsKKwkJaWYg KHBoeXNfbWFuYWdlZFtpICsgMV0gPiBoaWdoX3dhdGVyKQorCQkJaGlnaF93YXRlciA9IHBoeXNf bWFuYWdlZFtpICsgMV07CisJfQorCisjaWZkZWYgWEVOCisJbG93X3dhdGVyID0gMDsKKyNlbmRp ZgkKKwogCWZvciAoaSA9IDA7IHBoeXNfYXZhaWxbaSArIDFdOyBpICs9IDIpIHsKLQkJdm1fcGFk ZHJfdCBzaXplID0gcGh5c19hdmFpbFtpICsgMV0gLSBwaHlzX2F2YWlsW2ldOworCQl2bV9wYWRk cl90IHNpemU7CiAKKwkJcGh5c19hdmFpbFtpXSA9IHJvdW5kX3BhZ2UocGh5c19hdmFpbFtpXSk7 CisJCXBoeXNfYXZhaWxbaSArIDFdID0gdHJ1bmNfcGFnZShwaHlzX2F2YWlsW2kgKyAxXSk7CisJ CQorCQlzaXplID0gcGh5c19hdmFpbFtpICsgMV0gLSBwaHlzX2F2YWlsW2ldOwogCQlpZiAoc2l6 ZSA+IGJpZ2dlc3RzaXplKSB7CiAJCQliaWdnZXN0b25lID0gaTsKIAkJCWJpZ2dlc3RzaXplID0g c2l6ZTsKIAkJfQotCQlpZiAocGh5c19hdmFpbFtpXSA8IGxvd193YXRlcikKLQkJCWxvd193YXRl ciA9IHBoeXNfYXZhaWxbaV07Ci0JCWlmIChwaHlzX2F2YWlsW2kgKyAxXSA+IGhpZ2hfd2F0ZXIp Ci0JCQloaWdoX3dhdGVyID0gcGh5c19hdmFpbFtpICsgMV07CiAJfQogCi0jaWZkZWYgWEVOCi0J bG93X3dhdGVyID0gMDsKLSNlbmRpZgkKLQogCWVuZCA9IHBoeXNfYXZhaWxbYmlnZ2VzdG9uZSsx XTsKIAogCS8qCkBAIC0zOTMsOCArNDA4LDggQEAKIAlmaXJzdF9wYWdlID0gbG93X3dhdGVyIC8g UEFHRV9TSVpFOwogI2lmZGVmIFZNX1BIWVNTRUdfU1BBUlNFCiAJcGFnZV9yYW5nZSA9IDA7Ci0J Zm9yIChpID0gMDsgcGh5c19hdmFpbFtpICsgMV0gIT0gMDsgaSArPSAyKQotCQlwYWdlX3Jhbmdl ICs9IGF0b3AocGh5c19hdmFpbFtpICsgMV0gLSBwaHlzX2F2YWlsW2ldKTsKKwlmb3IgKGkgPSAw OyBwaHlzX21hbmFnZWRbaSArIDFdICE9IDA7IGkgKz0gMikKKwkJcGFnZV9yYW5nZSArPSBhdG9w KHBoeXNfbWFuYWdlZFtpICsgMV0gLSBwaHlzX21hbmFnZWRbaV0pOwogI2VsaWYgZGVmaW5lZChW TV9QSFlTU0VHX0RFTlNFKQogCXBhZ2VfcmFuZ2UgPSBoaWdoX3dhdGVyIC8gUEFHRV9TSVpFIC0g Zmlyc3RfcGFnZTsKICNlbHNlCkluZGV4OiBzeXMvdm0vdm1fcGFnZS5oCj09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0t IHN5cy92bS92bV9wYWdlLmgJKHJldmlzaW9uIDI3MjY4MikKKysrIHN5cy92bS92bV9wYWdlLmgJ KHdvcmtpbmcgY29weSkKQEAgLTM2OCw2ICszNjgsOCBAQAogCiBleHRlcm4gaW50IHZtX3BhZ2Vf emVyb19jb3VudDsKIAorZXh0ZXJuIHZtX3BhZGRyX3QgKnBoeXNfbWFuYWdlZDsJLyogcGFnZXMg bWFuYWdlZCBieSB2bV9wYWdlX2FycmF5ICovCisKIGV4dGVybiB2bV9wYWdlX3Qgdm1fcGFnZV9h cnJheTsJCS8qIEZpcnN0IHJlc2lkZW50IHBhZ2UgaW4gdGFibGUgKi8KIGV4dGVybiBsb25nIHZt X3BhZ2VfYXJyYXlfc2l6ZTsJCS8qIG51bWJlciBvZiB2bV9wYWdlX3QncyAqLwogZXh0ZXJuIGxv bmcgZmlyc3RfcGFnZTsJCQkvKiBmaXJzdCBwaHlzaWNhbCBwYWdlIG51bWJlciAqLwpJbmRleDog c3lzL3ZtL3ZtX3BoeXMuYwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMvdm0vdm1fcGh5cy5jCShyZXZpc2lv biAyNzI2ODIpCisrKyBzeXMvdm0vdm1fcGh5cy5jCSh3b3JraW5nIGNvcHkpCkBAIC0zNjUsMTcg KzM2NSwxNyBAQAogCXN0cnVjdCB2bV9mcmVlbGlzdCAqZmw7CiAJaW50IGRvbSwgZmxpbmQsIGks IG9pbmQsIHBpbmQ7CiAKLQlmb3IgKGkgPSAwOyBwaHlzX2F2YWlsW2kgKyAxXSAhPSAwOyBpICs9 IDIpIHsKKwlmb3IgKGkgPSAwOyBwaHlzX21hbmFnZWRbaSArIDFdICE9IDA7IGkgKz0gMikgewog I2lmZGVmCVZNX0ZSRUVMSVNUX0lTQURNQQotCQlpZiAocGh5c19hdmFpbFtpXSA8IDE2Nzc3MjE2 KSB7Ci0JCQlpZiAocGh5c19hdmFpbFtpICsgMV0gPiAxNjc3NzIxNikgewotCQkJCXZtX3BoeXNf Y3JlYXRlX3NlZyhwaHlzX2F2YWlsW2ldLCAxNjc3NzIxNiwKKwkJaWYgKHBoeXNfbWFuYWdlZFtp XSA8IDE2Nzc3MjE2KSB7CisJCQlpZiAocGh5c19tYW5hZ2VkW2kgKyAxXSA+IDE2Nzc3MjE2KSB7 CisJCQkJdm1fcGh5c19jcmVhdGVfc2VnKHBoeXNfbWFuYWdlZFtpXSwgMTY3NzcyMTYsCiAJCQkJ ICAgIFZNX0ZSRUVMSVNUX0lTQURNQSk7Ci0JCQkJdm1fcGh5c19jcmVhdGVfc2VnKDE2Nzc3MjE2 LCBwaHlzX2F2YWlsW2kgKyAxXSwKLQkJCQkgICAgVk1fRlJFRUxJU1RfREVGQVVMVCk7CisJCQkJ dm1fcGh5c19jcmVhdGVfc2VnKDE2Nzc3MjE2LAorCQkJCSAgICBwaHlzX21hbmFnZWRbaSArIDFd LCBWTV9GUkVFTElTVF9ERUZBVUxUKTsKIAkJCX0gZWxzZSB7Ci0JCQkJdm1fcGh5c19jcmVhdGVf c2VnKHBoeXNfYXZhaWxbaV0sCi0JCQkJICAgIHBoeXNfYXZhaWxbaSArIDFdLCBWTV9GUkVFTElT VF9JU0FETUEpOworCQkJCXZtX3BoeXNfY3JlYXRlX3NlZyhwaHlzX21hbmFnZWRbaV0sCisJCQkJ ICAgIHBoeXNfbWFuYWdlZFtpICsgMV0sIFZNX0ZSRUVMSVNUX0lTQURNQSk7CiAJCQl9CiAJCQlp ZiAoVk1fRlJFRUxJU1RfSVNBRE1BID49IHZtX25mcmVlbGlzdHMpCiAJCQkJdm1fbmZyZWVsaXN0 cyA9IFZNX0ZSRUVMSVNUX0lTQURNQSArIDE7CkBAIC0zODIsMjEgKzM4MiwyMSBAQAogCQl9IGVs c2UKICNlbmRpZgogI2lmZGVmCVZNX0ZSRUVMSVNUX0hJR0hNRU0KLQkJaWYgKHBoeXNfYXZhaWxb aSArIDFdID4gVk1fSElHSE1FTV9BRERSRVNTKSB7Ci0JCQlpZiAocGh5c19hdmFpbFtpXSA8IFZN X0hJR0hNRU1fQUREUkVTUykgewotCQkJCXZtX3BoeXNfY3JlYXRlX3NlZyhwaHlzX2F2YWlsW2ld LAorCQlpZiAocGh5c19tYW5hZ2VkW2kgKyAxXSA+IFZNX0hJR0hNRU1fQUREUkVTUykgeworCQkJ aWYgKHBoeXNfbWFuYWdlZFtpXSA8IFZNX0hJR0hNRU1fQUREUkVTUykgeworCQkJCXZtX3BoeXNf Y3JlYXRlX3NlZyhwaHlzX21hbmFnZWRbaV0sCiAJCQkJICAgIFZNX0hJR0hNRU1fQUREUkVTUywg Vk1fRlJFRUxJU1RfREVGQVVMVCk7CiAJCQkJdm1fcGh5c19jcmVhdGVfc2VnKFZNX0hJR0hNRU1f QUREUkVTUywKLQkJCQkgICAgcGh5c19hdmFpbFtpICsgMV0sIFZNX0ZSRUVMSVNUX0hJR0hNRU0p OworCQkJCSAgICBwaHlzX21hbmFnZWRbaSArIDFdLCBWTV9GUkVFTElTVF9ISUdITUVNKTsKIAkJ CX0gZWxzZSB7Ci0JCQkJdm1fcGh5c19jcmVhdGVfc2VnKHBoeXNfYXZhaWxbaV0sCi0JCQkJICAg IHBoeXNfYXZhaWxbaSArIDFdLCBWTV9GUkVFTElTVF9ISUdITUVNKTsKKwkJCQl2bV9waHlzX2Ny ZWF0ZV9zZWcocGh5c19tYW5hZ2VkW2ldLAorCQkJCSAgICBwaHlzX21hbmFnZWRbaSArIDFdLCBW TV9GUkVFTElTVF9ISUdITUVNKTsKIAkJCX0KIAkJCWlmIChWTV9GUkVFTElTVF9ISUdITUVNID49 IHZtX25mcmVlbGlzdHMpCiAJCQkJdm1fbmZyZWVsaXN0cyA9IFZNX0ZSRUVMSVNUX0hJR0hNRU0g KyAxOwogCQl9IGVsc2UKICNlbmRpZgotCQl2bV9waHlzX2NyZWF0ZV9zZWcocGh5c19hdmFpbFtp XSwgcGh5c19hdmFpbFtpICsgMV0sCisJCXZtX3BoeXNfY3JlYXRlX3NlZyhwaHlzX21hbmFnZWRb aV0sIHBoeXNfbWFuYWdlZFtpICsgMV0sCiAJCSAgICBWTV9GUkVFTElTVF9ERUZBVUxUKTsKIAl9 CiAJZm9yIChkb20gPSAwOyBkb20gPCB2bV9uZG9tYWluczsgZG9tKyspIHsKSW5kZXg6IHN5cy92 bS92bV9yZXNlcnYuYwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzeXMvdm0vdm1fcmVzZXJ2LmMJKHJldmlzaW9u IDI3MjY4MikKKysrIHN5cy92bS92bV9yZXNlcnYuYwkod29ya2luZyBjb3B5KQpAQCAtODI0LDkg KzgyNCw5IEBACiAJICogSW5pdGlhbGl6ZSB0aGUgcmVzZXJ2YXRpb24gYXJyYXkuICBTcGVjaWZp Y2FsbHksIGluaXRpYWxpemUgdGhlCiAJICogInBhZ2VzIiBmaWVsZCBmb3IgZXZlcnkgZWxlbWVu dCB0aGF0IGhhcyBhbiB1bmRlcmx5aW5nIHN1cGVycGFnZS4KIAkgKi8KLQlmb3IgKGkgPSAwOyBw aHlzX2F2YWlsW2kgKyAxXSAhPSAwOyBpICs9IDIpIHsKLQkJcGFkZHIgPSByb3VuZHVwMihwaHlz X2F2YWlsW2ldLCBWTV9MRVZFTF8wX1NJWkUpOwotCQl3aGlsZSAocGFkZHIgKyBWTV9MRVZFTF8w X1NJWkUgPD0gcGh5c19hdmFpbFtpICsgMV0pIHsKKwlmb3IgKGkgPSAwOyBwaHlzX21hbmFnZWRb aSArIDFdICE9IDA7IGkgKz0gMikgeworCQlwYWRkciA9IHJvdW5kdXAyKHBoeXNfbWFuYWdlZFtp XSwgVk1fTEVWRUxfMF9TSVpFKTsKKwkJd2hpbGUgKHBhZGRyICsgVk1fTEVWRUxfMF9TSVpFIDw9 IHBoeXNfbWFuYWdlZFtpICsgMV0pIHsKIAkJCXZtX3Jlc2Vydl9hcnJheVtwYWRkciA+PiBWTV9M RVZFTF8wX1NISUZUXS5wYWdlcyA9CiAJCQkgICAgUEhZU19UT19WTV9QQUdFKHBhZGRyKTsKIAkJ CXBhZGRyICs9IFZNX0xFVkVMXzBfU0laRTsK --001a1133bf0229b31b0504eb1a08--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFHCsPWxF0G%2BbqBYgxH=WtV%2BSt_UTWZj%2BY2-PHfoYSLjC_Qpig>