Date: Thu, 22 May 2014 01:43:31 +1000 (EST) From: Bruce Evans <brde@optusnet.com.au> To: Gleb Smirnoff <glebius@freebsd.org> Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, Rui Paulo <rpaulo@felyko.com>, Bruce Evans <brde@optusnet.com.au>, Jack F Vogel <jfv@freebsd.org>, Julian Elischer <julian@freebsd.org> Subject: Re: svn commit: r266423 - in head/sys: conf dev/i40e modules/i40e Message-ID: <20140521190353.U974@besplex.bde.org> In-Reply-To: <20140520171613.GM50679@FreeBSD.org> References: <201405190121.s4J1L3qA068339@svn.freebsd.org> <53796149.8060000@freebsd.org> <AF83F052-00D1-40E1-A427-58EDE0853D42@felyko.com> <20140520223516.R2836@besplex.bde.org> <20140520171613.GM50679@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 20 May 2014, Gleb Smirnoff wrote:
> On Tue, May 20, 2014 at 11:40:01PM +1000, Bruce Evans wrote:
> B> Also, verbose names break formatting. E.g., netstat -r has 5 columns
> B> available under Netif for the driver name and device number. netstat
> B> -i has about the same under Name (possibly 1 or 2 not directly under
> B> Name, but reserved for the Name column). systat has 3 columns
> B> available, but with a more flexible format that truncates other info.
> B> All driver name+numbers are broken now on freefall:
>
> We must admit that nowadays 80x25 terminal is not enough :(
Yes, much smaller formats are needed now for small portable screens.
My phone is low-end so it is 29 or 30 wide.
80 is also too wide. Most books use more like 60. Most (?) newspapers
use more like 30, with many columns.
> Would be cool if most of tools (netstat, systat, etc...) could
> determine size of terminal and dynamically widen all their fields.
You mean, determine the size and dynamically narrow all their fields
:-). This is hard to do. Curses-based applications like top and
systat should do a combination of horizontal and vertical scrolling
and variant displays, but this is also hard to do. top and systat
already have some variant displays but are too simple to do much. You
can send raw output to a terminal or window that does the scrolling,
but this doesn't work for repeated output.
> Thus, tool can run w/o any abbreviations when run in a script mode,
> run abbreviated on a small terminal, and run verbose on a wide
> terminal.
>
> This sounds like a generic library providing a special version
> of printf(3), which specifies minimal and maximum sizes for fields
> and when extra terminal width is available it distributes this
> width evenly between all fields. Name it 'elastic printf'.
> Sounds like a nice Google SoC project. Or might be that such
> library already exists.
Try doing that with systat -v output. Even programming tools for
generating fixed forms like the ones used in systat are deficient
in FreeBSD. All screen locations are almost hard-coded (just as
offsets) and hard to change. Some of this is from primitive use
of curses, with no subwindows.
Before doing that, just handle arbitrary columnar output (with headers)
in a filter. This can almost be done in an awk script. Field
separators might be a problem. Consider "vmstat 1 output". It is
messed up by the header being 2 lines with the first line not being
in normal columnar format (some of its keywords are for multiple
columns). A dumb filter can handle headers without knowing that they
are special provided they are in columnar format, but even this minor
complication seems to be difficult to handle without knowing what the
headers mean.
Here is a not so dumb filter using awk. vmstat output is so broken in
-current that even this simple filter improves the formatting significantly.
@BEGIN {
@ columns = 79 # XXX
@}
@
@{
@ # Determine fields and field widths for current line.
@ # awk's field splitting feature turned out to be inadequate,
@ # and this would be even easier in C.
@ n = split($0, a, "")
@ f = 1
@ j = 1
@ while (j <= n) {
@ cpw[f] = 0;
@ cfw[f] = 0;
@ while (j <= n && a[j] == " ") {
@ cpw[f]++ # current (left) padding width
@ j++
@ }
@ while (j <= n && a[j] != " ") {
@ cfw[f]++ # current field width without padding
@ j++
@ }
@ fld[f] = substr($0, j - cfw[f], cfw[f])
@ f++
@ }
@ nf = f # no need to use or trust NF
@
@ if (NR == 1 || NR < 10 && nf != anf) {
@ # Make current field widths active. When NR > 1, this
@ # discards the previous active field widths. Good enough
@ # for vmstat, where the widths from NR == 1 are garbage.
@ anf = nf
@ for (f = 1; f <= anf; f++)
@ afw[f] = cpw[f] + cfw[f]
@
@ # Convert current (unpadded) field widths to minumum (padded)
@ # field widths.
@ mfw[1] = cfw[1]
@ for (f = 2; f <= anf; f++)
@ mfw[f] = 1 + cfw[f]
@ } else if (nf != anf) {
@ # Some non-tabular line after warming up. Probably an ornate
@ # line in the next header. Too hard to handle properly.
@ printf("%.*s\n", columns, $0)
@ fflush(stdout)
@ next
@ } else {
@ # Update and minimum field widths if this line needs
@ # wider fields.
@ if (afw[1] < cfw[1])
@ afw[1] = cfw[1]
@ len = afw[1]
@ if (mfw[1] < cfw[1])
@ mfw[1] = cfw[1]
@ for (f = 2; f <= nf; f++) {
@ if (afw[f] < cfw[f] + 1)
@ afw[f] = cfw[f] + 1
@ len += afw[f]
@ if (mfw[f] < cfw[f] + 1)
@ mfw[f] = cfw[f] + 1
@ }
@
@ # But if the line would be too wide, reset the active field
@ # widths to minimum ones.
@ if (len > columns)
@ for (f = 1; f <= nf; f++)
@ afw[f] = mfw[f]
@ }
@ s = ""
@ for (f = 1; f <= nf; f++)
@ s = s sprintf("%*s", afw[f], fld[f])
@ printf("%.*s\n", columns, s)
@ fflush(stdout)
@}
Examples:
vmstat wasn't broken in FreeBSD-5.2, at least in my version:
ttyv2:bde@besplex:/tmp/s1> vmstat 1
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id
0 0 0 48928 905436 8 0 0 0 6 0 0 0 235 52 596 0 0 100
0 0 0 48928 905436 0 0 0 0 2 0 2 0 234 107 587 0 0 100
0 0 0 48928 905436 0 0 0 0 0 0 0 0 231 107 580 0 0 100
...
0 0 0 48928 905436 0 0 0 0 0 0 0 0 232 107 582 0 0 100
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id
0 0 0 48928 905436 0 0 0 0 0 0 0 0 231 120 581 0 0 100
0 0 0 48928 905436 0 0 0 0 0 0 0 0 231 107 579 0 0 100
0 0 0 48928 905436 0 0 0 0 0 0 0 0 231 115 580 0 0 100
Running this through the filter makes little difference:
ttyv2:bde@besplex:/tmp/s1> vmstat 1 |awk -f z
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id
0 1 0 49408 905232 8 0 0 0 6 0 0 0 235 52 596 0 0 100
0 1 0 49480 905176 14 0 0 0 0 0 0 0 232 118 583 0 0 100
0 1 0 49480 905176 0 0 0 0 0 0 0 0 231 117 582 0 0 100
...
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id
0 1 0 49480 905176 0 0 0 0 0 0 0 0 233 115 587 0 0 100
0 1 0 49480 905176 0 0 0 0 0 0 0 0 232 118 584 1 0 99
1 1 0 49480 905176 0 0 0 0 0 0 0 0 231 109 581 0 0 100
The filter just eventually fixes the misformatting of 100% by right
justifying its header. The formatting is %2.0f under "id", but 100 doesn't
fit. 100 also wouldn't fit under us or sy, but only id is often 100. This
last worked in about 1985 where 100 was impossible for id too, since just
running vmstat every second took more than 1% CPU.
The broken output in -current looks like this in on freefall:
pts/18:bde@freefall:~/s> vmstat 1
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy i
d
0 0 16 7312M 755M 751 2 2 0 1531 158 0 0 1850 1678 251 1 1
98
0 0 16 7312M 755M 46 0 0 0 88 25 0 0 751 610 2456 0 0
100
0 0 16 7384M 752M 551 0 0 0 186 25 0 0 722 1133 2374 0 0
100
...
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy i
d
1 0 16 7541M 744M 2172 0 0 0 1566 28 0 0 1012 76327 18683 1
2 97
1 0 16 7420M 746M 913 0 0 0 1744 27 0 0 1067 104617 20083 1
4 95
1 0 16 7487M 740M 1970 0 0 0 755 28 44 45 1496 124350 23887 1
5 94
I used a vidcontrol screen dump to preserve its full uglyness. Note that
the header formatting has been destroyed even though it is not dynamic:
- the first line of the header was not updated when the second line was
expanded (except "cpu" was moved slightly, but it has an off by 1
error in the move)
- the second header line is unconditionally too wide for an 80 column terminal
Note that avm and fre are now formatted in dehumanized (scientific)
form. My 5.2 system doesn't have much memory, so it doesn't need this.
This makes avm and fre narrow enough to fit, but the formatting is otherwise
broken so they don't. The "M" part never fits under the header. This
might be intentional, but it looks strange. It is actually clearly
unintentional since all the fields after avm and fre, through in, are
similarly misaligned with the header.
Then the bugs are from dynamic values being too large:
- despite expansion of the faults sy, the value is often too large to fit,
so the fields from that point are shifted by 1 or 2 more more relative
to the header.
- the cpu cs has been expanded too, but not enough for all cases
- the worst cases don't all occur on the same line, so the maximum shift
relative to the header is 4.
When the output is not to a terminal, vmstat no longer dehumanizes some
large numbers:
pts/18:bde@freefall:~/s> vmstat 1|cat
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy i
d
0 0 16 7504352 751956 752 2 2 0 1532 157 0 0 1850 1691 259 1 1
98
0 0 16 7500132 751900 118 0 0 0 164 50 1 1 726 476 2423 0 0
100
...
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy i
d
0 0 16 7500132 751924 547 0 0 0 1522 25 0 0 747 1218 2639 0 0
100
0 0 16 7500132 751924 46 0 0 0 88 25 0 0 668 348 2324 0 0
100
Note that the dehumanization didn't even improve the formatting in this case.
The off-by-1 error in the field width and/or alignment gives just enough space
for the large avm.
freefall became less active while I was testing this, so many of the numbers
aren't so large now.
The filter cleans this up to:
pts/18:bde@freefall:~/s> vmstat 1|awk -f ../z
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy
2 0 16 7542196 812772 753 2 2 0 1532 157 0 0 1850 1694 260 1 1 98
0 0 16 7558808 809868 220 0 0 0 4117 27 0 0 6661 1392 18283 2 1 97
0 0 16 7558808 807060 46 0 0 0 3993 27 23 23 6640 1122 19186 2 1 97
1 0 16 7558808 804252 46 0 0 0 3995 27 28 27 6629 1131 19041 2 2 97
1 0 16 7750064 796536 1793 0 0 0 4761 27 0 0 6348 3978 17538 1 2 97
0 0 16 7750064 793648 46 0 0 0 4101 28 0 0 6304 1143 17556 1 1 97
...
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id
1 0 16 7591172 756388 112 0 0 0 5173 28 0 0 6389 1169 17547 2 1 97
0 0 16 7654924 752004 1967 0 0 0 6374 29 0 0 6621 3846 18279 2 2 97
0 0 16 7600364 754668 586 0 0 0 3416 28 54 54 3584 1618 12072 1 1 98
0 0 16 7409108 759696 104 0 0 0 1402 27 0 0 457 488 1428 0 0 100
0 0 16 7600364 754668 1804 0 0 0 934 25 0 0 454 3303 1544 0 0 100
It took a while to stabilize, and significant activity would push it to a
stable setting wanting more than 80 columns. Dehumanized numbers for just
avm would help avoid this.
Note that the filter never shrinks the columns except once to squeeze out
spaces in the header. It is too simple to do that. It also responds too
quickly to transient changes.
netstat -i is also broken in -current (netstat -r in -current shows less
details so it fits easil):
pts/18:bde@freefall:~/s> netstat -i
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs
Coll
igb0 1500 <Link#1> 68:b5:99:b5:2a:02 2821384398 0 0 1778806163
0 0
igb0 - 8.8.178.128/2 freefall 217503944 - - 175357038
- -
igb0 - fe80::6ab5:99 fe80::6ab5:99ff:f 104258 - - 104297 -
-
igb0 - freefall.free 2001:1900:2254:20 2576868711 - - 1641398377
- -
igb0 - 8.8.178.141/3 people 24407600 - - 0 -
-
igb0 - people.freebs 2001:1900:2254:20 1171548 - - 642454 -
-
igb1 1500 <Link#2> 68:b5:99:b5:2a:03 0 0 0 0 0
0
lo0 16384 <Link#3> 2466053 0 0 2466037 0
0
lo0 - localhost ::1 55872 - - 55992 -
-
lo0 - fe80::1%lo0 fe80::1%lo0 0 - - 0 -
-
lo0 - your-net localhost 2402292 - - 2410186 -
-
The filter cleans this up to:
pts/18:bde@freefall:~/s> netstat -i|awk -f ../z
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs
igb0 1500 <Link#1> 68:b5:99:b5:2a:02 2821393981 0 0 1778816441 0
igb0 - 8.8.178.128/2 freefall 217504375 - - 175357732
igb0 - fe80::6ab5:99 fe80::6ab5:99ff:f 104261 - - 104300
igb0 - freefall.free 2001:1900:2254:20 2576877564 - - 1641408119
igb0 - 8.8.178.141/3 people 24407836 - - 0
igb0 - people.freebs 2001:1900:2254:20 1171564 - - 642460
igb1 1500 <Link#2> 68:b5:99:b5:2a:03 0 0 0 0
lo0 16384 <Link#3> 2466061 0 0 2466045 0
lo0 - localhost ::1 55872 - - 55992 -
lo0 - fe80::1%lo0 fe80::1%lo0 0 - - 0 -
lo0 - your-net localhost 2402300 - - 2410194 -
It looks almost readable at first, but further examination shows that the
filter is not working as intended. The Coll field got truncated away fairly
cleanly, and the first line for lo0 is unparseable since it has spaces
instead of "-" for the Address field. The second of these breaks the header
hack. After modifying the hack, it works to pass 2 copies through the filter
and discard the first half:
pts/18:bde@freefall:~/s> cat z
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oer
igb0 1500 <Link#1> 68:b5:99:b5:2a:02 2822276782 0 0 1779538929
igb0 - 8.8.178.128/2 freefall 217533292 - - 175425951
igb0 - fe80::6ab5:99 fe80::6ab5:99ff:f 104397 - - 104436
igb0 - freefall.free 2001:1900:2254:20 2577700440 - - 1642086286
igb0 - 8.8.178.141/3 people 24435004 - - 0
igb0 - people.freebs 2001:1900:2254:20 1173149 - - 643437
igb1 1500 <Link#2> 68:b5:99:b5:2a:03 0 0 0 0
lo0 16384 <Link#3> 2467437 0 0 2467421 0
lo0 - localhost ::1 55880 - - 56000
lo0 - fe80::1%lo0 fe80::1%lo0 0 - - 0
lo0 - your-net localhost 2403657 - - 2411562
Now Oerrs is truncated too (uncleanly), and the first lo0 line remains
unfixable. It accidentally almost lines up with the header, since only
the Opkts field in the header got moved.
Note that the second igb0 field is obviously truncated before it gets to
the filter. The full field can be seen using netstat -rn, and is 8 characters
wider. Many of the other fields are truncated. This is less clear, but
all of the Address fields that use the full field are probably truncated.
netstat -rn uses a separate format with very wide fields for the inet6 case
to avoid this problem. This is ugly in a different way.
Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140521190353.U974>