Date: Mon, 2 Dec 2024 08:37:52 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: J David <j.david.lists@gmail.com> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: NFS 4.2 "RPC struct is bad" revisited (with much more detail) Message-ID: <CAM5tNy4P7RCt0rkV56UMS0hUu2yPHL1=4-YP9UtejNrUA8gmTA@mail.gmail.com> In-Reply-To: <CAM5tNy44wh9h0gvi7ySonkkA3CZQN72a-paSK4Lk_BSsB2M8XA@mail.gmail.com> References: <CABXB=RT7Sb76bPZg3k%2BM5wt3q1Ern56zmzyviN0_zdj27pqQLw@mail.gmail.com> <CAM5tNy44wh9h0gvi7ySonkkA3CZQN72a-paSK4Lk_BSsB2M8XA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Dec 1, 2024 at 5:03=E2=80=AFPM Rick Macklem <rick.macklem@gmail.com=
> wrote:
>
> On Sun, Dec 1, 2024 at 4:20=E2=80=AFPM J David <j.david.lists@gmail.com> =
wrote:
> >
> > We have continued to experience client-side corruption with NFS 4.2 on
> > FreeBSD 14.1 clients running against Debian NFS servers.
> >
> > However, I was recently handed a reproducible way to trigger this
> > issue. With PHP and composer installed, in a writable NFS directory:
> >
> > $ ls -ld .
> > drwxrwxrwx 3 287028 287028 3 Dec 1 23:01 .
> > $ df -h .
> > Filesystem Size Used Avail Capacity Mounted on
> > fs:/deleteme 10G 174M 9.8G 2% /mnt
> >
> > Per "nfsstat -m" the flags are:
> >
> > nfsv4,minorversion=3D2,tcp,resvport,nconnect=3D1,hard,cto,sec=3Dsys,acd=
irmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60,negnameti=
meo=3D60,rsize=3D65536,wsize=3D65536,readdirsize=3D65536,readahead=3D1,wcom=
mitsize=3D8388608,timeout=3D120,retrans=3D2147483647
> >
> > So I run:
> >
> > $ composer create-project drupal/recommended-project:10.3.10 drupal10
> >
> > After quite a lot of output, this finishes successfully.
> >
> > However:
> >
> > $ du -sxk drupal10/
> > du: drupal10/web/core/assets/vendor/ckeditor5/block-quote/translations:
> > RPC struct is bad
> > du: drupal10/web/core/assets/vendor/ckeditor5/list/translations: RPC
> > struct is bad
> > du: drupal10/web/core/assets/vendor/ckeditor5/heading/translations:
> > RPC struct is bad
> > 127246 drupal10/
> >
> > From that client, these directories (and it seems like it's always
> > these three directories) appear to be empty, not even . or .. present:
> >
> > $ ls -la drupal10/web/core/assets/vendor/ckeditor5/block-quote/translat=
ions
> > total 0
> > $ ls -la drupal10/web/core/assets/vendor/ckeditor5/list/translations
> > total 0
> > $ ls -la drupal10/web/core/assets/vendor/ckeditor5/heading/translations
> > total 0
> >
> > I don't know why it's these directories specifically, but it's very
> > consistent. The filenames in all three directories seem to be the
> > same: 65 small javascript files with two-letter country codes for
> > filenames (e.g., , though the actual content of the files differs.
> > E.g., all three directories have a zh.js but the contents of each
> > zh.js file is different.
> >
> > This behavior will survive unmounting & remounting the filesystem. If
> > I mount the same filesystem again at a different location on the same
> > client, it will be the same there. It can even survive reboots of the
> > affected client.
> >
> > At the same time, if I look at these same directories from another
> > FreeBSD client at any point, everything is fine:
> >
> > $ du -sxk drupal10
> > 128109 drupal10
> > $ ls -la drupal10/web/core/assets/vendor/ckeditor5/block-quote/translat=
ions | wc
> > 65 578 2906
> > $ ls -la drupal10/web/core/assets/vendor/ckeditor5/heading/translations=
| wc
> > 65 578 2903
> >
> > The NFS server also gives the correct results:
> >
> > $ du -sxk drupal10/
> > 128109 drupal10/
> > $ ls -la drupal10/web/core/assets/vendor/ckeditor5/block-quote/translat=
ions | wc
> > 65 578 2843
> >
> > This problem does appear to be related to READDIR/listing directory
> > contents. If (using some other client) I get the names of files in the
> > affected directory, I can read that with no problem:
> >
> > $ sha1 drupal10/web/core/assets/vendor/ckeditor5/heading/translations/z=
h.js
> > SHA1 (drupal10/web/core/assets/vendor/ckeditor5/heading/translations/zh=
.js)
> > =3D cb1d75ec267f5673e99b3d0afa963f7c8656cb48
> > $ ls -l drupal10/web/core/assets/vendor/ckeditor5/heading/translations/=
zh.js
> > -rw-rw-r-- 1 deleteme deleteme 467 Nov 22 12:51
> > drupal10/web/core/assets/vendor/ckeditor5/heading/translations/zh.js
> >
> >
> > Under 14.1, if I wait a bit, this will slowly fix itself. After a few
> > minutes, one of the directories will become listable. Then a few
> > minutes later a second one. Then, a few more minutes after that, the
> > third. The correction appears to happen on a per-mount basis. I.e., if
> > I mount the same filesystem three times at three different locations,
> > they will all observe the issue initially, and it will clear at
> > different rates. Sometimes it might take hours.
> >
> > This is a slight change from the behavior under 13.3 where once this
> > issue occurred, it was effectively permanent until the affected
> > directories were removed from another client.
> >
> > This is the part where it makes sense to obtain a pcap showing the
> > traffic that causes the "RPC struct is bad" error. That's frequently
> > cached.
> >
> > Easier said than done. On the client, frequently, after testing this:
> >
> > # tcpdump -vvv -i net1 -n -w rpc-struct-is-bad.pcap host fs
> > tcpdump: listening on net1, link-type EN10MB (Ethernet), snapshot
> > length 262144 bytes
> > ^C0 packets captured
> > 362 packets received by filter
> > 0 packets dropped by kernel
> >
> > So this behavior appears to be cached on the client. After waiting a
> > bit (much more than 60 seconds), it looks like the client did try
> > again, and I think I got it. I'll include the Wireshark output below.
> >
> > Hopefully all of this is enough to shed some light on the issue.
> >
> > Thanks for any feedback!
> >
> > Client request, as decoded by Wireshark:
> >
> > Remote Procedure Call, Type:Call XID:0x3c21500b
> > Fragment header: Last fragment, 220 bytes
> > 1... .... .... .... .... .... .... .... =3D Last Fragment: Yes
> > .000 0000 0000 0000 0000 0000 1101 1100 =3D Fragment Length: 22=
0
> > XID: 0x3c21500b (1008816139)
> > Message Type: Call (0)
> > RPC Version: 2
> > Program: NFS (100003)
> > Program Version: 4
> > Procedure: COMPOUND (1)
> > [The reply to this request is in frame 22]
> > Credentials
> > Flavor: AUTH_UNIX (1)
> > Length: 28
> > Stamp: 0x674ce3c9
> > Machine Name: h3
> > length: 2
> > contents: h3
> > fill bytes: opaque data
> > UID: 287028
> > GID: 287028
> > Auxiliary GIDs (1) [25000]
> > GID: 25000
> > Verifier
> > Flavor: AUTH_NULL (0)
> > Length: 0
> > Network File System, Ops(4): SEQUENCE, PUTFH, READDIR, GETATTR
> > [Program Version: 4]
> > [V4 Procedure: COMPOUND (1)]
> > Tag: Readdir
> > length: 7
> > contents: Readdir
> > fill bytes: opaque data
> > minorversion: 2
> > Operations (count: 4): SEQUENCE, PUTFH, READDIR, GETATTR
> > Opcode: SEQUENCE (53)
> > sessionid: 67e43365b78a01ddd1e5050000000000
> > seqid: 0x000f73d7
> > slot id: 0
> > high slot id: 0
> > cache this?: No
> > Opcode: PUTFH (22)
> > FileHandle
> > length: 32
> > [hash (CRC-32): 0x87cdc24b]
> > FileHandle:
> > 0100060153d075cb002f4a8000000000000000000a00d6bd000000003cf1ea00
> > Opcode: READDIR (26)
> > cookie: 0
> > cookie_verf: 0
> > dircount: 8704
> > maxcount: 8704
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileId)
> > reqd_attr: Type (1)
> > reqd_attr: RDAttr_Error (11)
> > reco_attr: FileId (20)
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > Opcode: GETATTR (9)
> > Attr mask[0]: 0x4010011b (Supported_Attrs, Type, Change,
> > Size, FSID, FileId, MaxRead)
> > reqd_attr: Supported_Attrs (0)
> > reqd_attr: Type (1)
> > reqd_attr: Change (3)
> > reqd_attr: Size (4)
> > reqd_attr: FSID (8)
> > reco_attr: FileId (20)
> > reco_attr: MaxRead (30)
> > Attr mask[1]: 0x0034a23a (Mode, NumLinks, Owner,
> > Owner_Group, RawDev, Space_Used, Time_Access, Time_Create,
> > Time_Metadata, Time_Modify)
> > reco_attr: Mode (33)
> > reco_attr: NumLinks (35)
> > reco_attr: Owner (36)
> > reco_attr: Owner_Group (37)
> > reco_attr: RawDev (41)
> > reco_attr: Space_Used (45)
> > reco_attr: Time_Access (47)
> > reco_attr: Time_Create (50)
> > reco_attr: Time_Metadata (52)
> > reco_attr: Time_Modify (53)
> > [Main Opcode: READDIR (26)]
> >
> > And here's the server's reply (reassembled from two TCP packets):
> >
> > Remote Procedure Call, Type:Reply XID:0x3c21500b
> > Fragment header: Last fragment, 4268 bytes
> > 1... .... .... .... .... .... .... .... =3D Last Fragment: Yes
> > .000 0000 0000 0000 0001 0000 1010 1100 =3D Fragment Length: 42=
68
> > XID: 0x3c21500b (1008816139)
> > Message Type: Reply (1)
> > [Program: NFS (100003)]
> > [Program Version: 4]
> > [Procedure: COMPOUND (1)]
> > Reply State: accepted (0)
> > [This is a reply to a request in frame 19]
> > [Time from request: 0.000216000 seconds]
> > Verifier
> > Flavor: AUTH_NULL (0)
> > Length: 0
> > Accept State: RPC executed successfully (0)
> > Network File System, Ops(4): SEQUENCE PUTFH READDIR GETATTR
> > [Program Version: 4]
> > [V4 Procedure: COMPOUND (1)]
> > Status: NFS4_OK (0)
> > Tag: Readdir
> > length: 7
> > contents: Readdir
> > fill bytes: opaque data
> > Operations (count: 4)
> > Opcode: SEQUENCE (53)
> > Status: NFS4_OK (0)
> > sessionid: 67e43365b78a01ddd1e5050000000000
> > seqid: 0x000f73d7
> > slot id: 0
> > high slot id: 29
> > target high slot id: 29
> > status flags: 0x00000001, SEQ4_STATUS_CB_PATH_DOWN
> > .... .... .... .... .... .... .... ...1 =3D
> > SEQ4_STATUS_CB_PATH_DOWN: Set
> > .... .... .... .... .... .... .... ..0. =3D
> > SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING: Not set
> > .... .... .... .... .... .... .... .0.. =3D
> > SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED: Not set
> > .... .... .... .... .... .... .... 0... =3D
> > SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED: Not set
> > .... .... .... .... .... .... ...0 .... =3D
> > SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED: Not set
> > .... .... .... .... .... .... ..0. .... =3D
> > SEQ4_STATUS_ADMIN_STATE_REVOKED: Not set
> > .... .... .... .... .... .... .0.. .... =3D
> > SEQ4_STATUS_RECALLABLE_STATE_REVOKED: Not set
> > .... .... .... .... .... .... 0... .... =3D
> > SEQ4_STATUS_LEASE_MOVED: Not set
> > .... .... .... .... .... ...0 .... .... =3D
> > SEQ4_STATUS_RESTART_RECLAIM_NEEDED: Not set
> > .... .... .... .... .... ..0. .... .... =3D
> > SEQ4_STATUS_CB_PATH_DOWN_SESSION: Not set
> > .... .... .... .... .... .0.. .... .... =3D
> > SEQ4_STATUS_BACKCHANNEL_FAULT: Not set
> > .... .... .... .... .... 0... .... .... =3D
> > SEQ4_STATUS_DEVID_CHANGED: Not set
> > .... .... .... .... ...0 .... .... .... =3D
> > SEQ4_STATUS_DEVID_DELETED: Not set
> > Opcode: PUTFH (22)
> > Status: NFS4_OK (0)
> > Opcode: READDIR (26)
> > Status: NFS4_OK (0)
> > verifier: 0x0000000000000000
> > Directory Listing
> > Entry: eu.js
> > Value Follows: Yes
> > cookie: 269690263
> > Name: eu.js
> > length: 5
> > contents: eu.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 47869
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bafd
> > Entry: gl.js
> > Value Follows: Yes
> > cookie: 276571741
> > Name: gl.js
> > length: 5
> > contents: gl.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48703
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000be3f
> > Entry: ku.js
> > Value Follows: Yes
> > cookie: 282097910
> > Name: ku.js
> > length: 5
> > contents: ku.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48705
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000be41
> > Entry: nl.js
> > Value Follows: Yes
> > cookie: 283039116
> > Name: nl.js
> > length: 5
> > contents: nl.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48371
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcf3
> > Entry: pt.js
> > Value Follows: Yes
> > cookie: 283739169
> > Name: pt.js
> > length: 5
> > contents: pt.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48609
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bde1
> > Entry: de.js
> > Value Follows: Yes
> > cookie: 286454349
> > Name: de.js
> > length: 5
> > contents: de.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48426
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd2a
> > Entry: pt-br.js
> > Value Follows: Yes
> > cookie: 287836878
> > Name: pt-br.js
> > length: 8
> > contents: pt-br.js
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48217
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bc59
> > Entry: uz.js
> > Value Follows: Yes
> > cookie: 289190542
> > Name: uz.js
> > length: 5
> > contents: uz.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48083
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bbd3
> > Entry: lt.js
> > Value Follows: Yes
> > cookie: 292503267
> > Name: lt.js
> > length: 5
> > contents: lt.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48369
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcf1
> > Entry: cs.js
> > Value Follows: Yes
> > cookie: 292958093
> > Name: cs.js
> > length: 5
> > contents: cs.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48700
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000be3c
> > Entry: sk.js
> > Value Follows: Yes
> > cookie: 293741377
> > Name: sk.js
> > length: 5
> > contents: sk.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48706
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000be42
> > Entry: ms.js
> > Value Follows: Yes
> > cookie: 297104108
> > Name: ms.js
> > length: 5
> > contents: ms.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48081
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bbd1
> > Entry: et.js
> > Value Follows: Yes
> > cookie: 298700082
> > Name: et.js
> > length: 5
> > contents: et.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48429
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd2d
> > Entry: tk.js
> > Value Follows: Yes
> > cookie: 300125681
> > Name: tk.js
> > length: 5
> > contents: tk.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 47955
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bb53
> > Entry: sr-latn.js
> > Value Follows: Yes
> > cookie: 309089285
> > Name: sr-latn.js
> > length: 10
> > contents: sr-latn.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48436
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd34
> > Entry: no.js
> > Value Follows: Yes
> > cookie: 314436706
> > Name: no.js
> > length: 5
> > contents: no.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48372
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcf4
> > Entry: lv.js
> > Value Follows: Yes
> > cookie: 315813800
> > Name: lv.js
> > length: 5
> > contents: lv.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48431
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd2f
> > Entry: fi.js
> > Value Follows: Yes
> > cookie: 320584473
> > Name: fi.js
> > length: 5
> > contents: fi.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48214
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bc56
> > Entry: ro.js
> > Value Follows: Yes
> > cookie: 322655904
> > Name: ro.js
> > length: 5
> > contents: ro.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48433
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd31
> > Entry: zh.js
> > Value Follows: Yes
> > cookie: 323217790
> > Name: zh.js
> > length: 5
> > contents: zh.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48375
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcf7
> > Entry: th.js
> > Value Follows: Yes
> > cookie: 327201823
> > Name: th.js
> > length: 5
> > contents: th.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 47954
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bb52
> > Entry: hi.js
> > Value Follows: Yes
> > cookie: 332981880
> > Name: hi.js
> > length: 5
> > contents: hi.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48430
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd2e
> > Entry: uk.js
> > Value Follows: Yes
> > cookie: 337730183
> > Name: uk.js
> > length: 5
> > contents: uk.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48220
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bc5c
> > Entry: sl.js
> > Value Follows: Yes
> > cookie: 345684024
> > Name: sl.js
> > length: 5
> > contents: sl.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48434
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd32
> > Entry: bs.js
> > Value Follows: Yes
> > cookie: 347404539
> > Name: bs.js
> > length: 5
> > contents: bs.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48602
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bdda
> > Entry: es.js
> > Value Follows: Yes
> > cookie: 349525579
> > Name: es.js
> > length: 5
> > contents: es.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48428
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd2c
> > Entry: az.js
> > Value Follows: Yes
> > cookie: 366619883
> > Name: az.js
> > length: 5
> > contents: az.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 47868
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bafc
> > Entry: ca.js
> > Value Follows: Yes
> > cookie: 369688170
> > Name: ca.js
> > length: 5
> > contents: ca.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48603
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bddb
> > Entry: da.js
> > Value Follows: Yes
> > cookie: 376000730
> > Name: da.js
> > length: 5
> > contents: da.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48604
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bddc
> > Entry: ja.js
> > Value Follows: Yes
> > cookie: 380243387
> > Name: ja.js
> > length: 5
> > contents: ja.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48078
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bbce
> > Entry: bg.js
> > Value Follows: Yes
> > cookie: 386685632
> > Name: bg.js
> > length: 5
> > contents: bg.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48600
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bdd8
> > Entry: hy.js
> > Value Follows: Yes
> > cookie: 387704532
> > Name: hy.js
> > length: 5
> > contents: hy.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48216
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bc58
> > Entry: jv.js
> > Value Follows: Yes
> > cookie: 393320046
> > Name: jv.js
> > length: 5
> > contents: jv.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48079
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bbcf
> > Entry: sq.js
> > Value Follows: Yes
> > cookie: 403421065
> > Name: sq.js
> > length: 5
> > contents: sq.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48435
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd33
> > Entry: bn.js
> > Value Follows: Yes
> > cookie: 405141322
> > Name: bn.js
> > length: 5
> > contents: bn.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48601
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bdd9
> > Entry: en-gb.js
> > Value Follows: Yes
> > cookie: 405890201
> > Name: en-gb.js
> > length: 8
> > contents: en-gb.js
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48701
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000be3d
> > Entry: kn.js
> > Value Follows: Yes
> > cookie: 411501723
> > Name: kn.js
> > length: 5
> > contents: kn.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48368
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcf0
> > Entry: de-ch.js
> > Value Follows: Yes
> > cookie: 417321901
> > Name: de-ch.js
> > length: 8
> > contents: de-ch.js
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48605
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bddd
> > Entry: ko.js
> > Value Follows: Yes
> > cookie: 425323582
> > Name: ko.js
> > length: 5
> > contents: ko.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48704
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000be40
> > Entry: eo.js
> > Value Follows: Yes
> > cookie: 429568351
> > Name: eo.js
> > length: 5
> > contents: eo.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48606
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bdde
> > Entry: vi.js
> > Value Follows: Yes
> > cookie: 433844822
> > Name: vi.js
> > length: 5
> > contents: vi.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48611
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bde3
> > Entry: tr.js
> > Value Follows: Yes
> > cookie: 436805847
> > Name: tr.js
> > length: 5
> > contents: tr.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48610
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bde2
> > Entry: he.js
> > Value Follows: Yes
> > cookie: 441713088
> > Name: he.js
> > length: 5
> > contents: he.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48077
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bbcd
> > Entry: sr.js
> > Value Follows: Yes
> > cookie: 443059815
> > Name: sr.js
> > length: 5
> > contents: sr.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48219
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bc5b
> > Entry: it.js
> > Value Follows: Yes
> > cookie: 445333183
> > Name: it.js
> > length: 5
> > contents: it.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 47953
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bb51
> > Entry: ru.js
> > Value Follows: Yes
> > cookie: 447922792
> > Name: ru.js
> > length: 5
> > contents: ru.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48218
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bc5a
> > Entry: pl.js
> > Value Follows: Yes
> > cookie: 450740642
> > Name: pl.js
> > length: 5
> > contents: pl.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48432
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd30
> > Entry: km.js
> > Value Follows: Yes
> > cookie: 451165557
> > Name: km.js
> > length: 5
> > contents: km.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48080
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bbd0
> > Entry: nb.js
> > Value Follows: Yes
> > cookie: 451928959
> > Name: nb.js
> > length: 5
> > contents: nb.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48608
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bde0
> > Entry: ar.js
> > Value Follows: Yes
> > cookie: 455961028
> > Name: ar.js
> > length: 5
> > contents: ar.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48599
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bdd7
> > Entry: fr.js
> > Value Follows: Yes
> > cookie: 458016628
> > Name: fr.js
> > length: 5
> > contents: fr.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48702
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000be3e
> > Entry: zh-cn.js
> > Value Follows: Yes
> > cookie: 458120880
> > Name: zh-cn.js
> > length: 8
> > contents: zh-cn.js
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48374
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcf6
> > Entry: hr.js
> > Value Follows: Yes
> > cookie: 462395925
> > Name: hr.js
> > length: 5
> > contents: hr.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48607
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bddf
> > Entry: el.js
> > Value Follows: Yes
> > cookie: 469338289
> > Name: el.js
> > length: 5
> > contents: el.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48366
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcee
> > Entry: en-au.js
> > Value Follows: Yes
> > cookie: 474580618
> > Name: en-au.js
> > length: 8
> > contents: en-au.js
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48427
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd2b
> > Entry: sv.js
> > Value Follows: Yes
> > cookie: 487795952
> > Name: sv.js
> > length: 5
> > contents: sv.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48437
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bd35
> > Entry: ug.js
> > Value Follows: Yes
> > cookie: 497925439
> > Name: ug.js
> > length: 5
> > contents: ug.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48373
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcf5
> > Entry: fa.js
> > Value Follows: Yes
> > cookie: 502070838
> > Name: fa.js
> > length: 5
> > contents: fa.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 47870
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bafe
> > Entry: id.js
> > Value Follows: Yes
> > cookie: 508050963
> > Name: id.js
> > length: 5
> > contents: id.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48367
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcef
> > Entry: hu.js
> > Value Follows: Yes
> > cookie: 513212780
> > Name: hu.js
> > length: 5
> > contents: hu.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48215
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bc57
> > Entry: ur.js
> > Value Follows: Yes
> > cookie: 533460897
> > Name: ur.js
> > length: 5
> > contents: ur.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48082
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bbd2
> > Entry: ne.js
> > Value Follows: Yes
> > cookie: 536311814
> > Name: ne.js
> > length: 5
> > contents: ne.js
> > fill bytes: opaque data
> > Attr mask[0]: 0x00100802 (Type, RDAttr_Error, FileI=
d)
> > reqd_attr: Type (1)
> > ftype4: NF4REG (1)
> > reqd_attr: RDAttr_Error (11)
> > Status: NFS4_OK (0)
> > reco_attr: FileId (20)
> > fileid: 48370
> > Attr mask[1]: 0x00800000 (Mounted_on_FileId)
> > reco_attr: Mounted_on_FileId (55)
> > fileid: 0x000000000000bcf2
> > Value Follows: No
> > EOF: Yes
> > Opcode: GETATTR (9)
> > Status: NFS4_OK (0)
> > [Expert Info (Warning/Protocol): Per RFCs 3530 and 5661 an
> > attribute mask is required but was not provided.]
> > [Per RFCs 3530 and 5661 an attribute mask is required
> > but was not provided.]
> Well, this indicates the Debian server is broken. A bitmap and associated
> attribute values are required for a GETATTR reply of NFS4_OK.
> This clearly says they are not there.
>
> That would result in the client saying the RPC is bad.
>
> rick
If you wanted to try and collect more information that the Linux folk
could use to debug this, you could:
1 - Pull your pcap file into a gui wireshark and then...
- Go to this RPC reply and click on it in the top window
- Click on the "RPC ..." line in the middle window
This should result in some part of the hex in the bottom
window being highlighted in blue background.
The first byte should be 80 and then the next three bytes is the
length of the RPC message.
--> Write this value down.
- Now, note the offset of the first byte after the four bytes mentioned
above. (Hex offsets are on the left margin.) This is the start offset
of the actual message.
- Scroll down in the bottom window to the end and figure out what
the offset of the end of message (actually the offset of the byte after
the end of the hex).
Take the difference between the two offsets (end - start).
This should be the same as the value in the 3 bytes after 80 above.
--> If it is not the same, that is where it is broken.
If it is the same, then it did not generate the attribute bitmap and
attribute values for some reason.
On the Debian server, you can:
# cat /dev/null > /var/log/messages
# echo 65535 > /proc/sys/sunrpc/nfsd_debug
--> This will result in a lot of stuff being written to /var/log/messages
Reproduce the failure.
# echo 0 > /proc/sys/sunrpc/nfsd_debug
--> To turn the debugging off.
Now, copy /var/log/messages to some other file. It is text, so you
can read it, but it will be a lot of junk.
(This file might be useful for a Linux NFS server person.)
Note that it would be much better if you did the above when
no other client was active against the server.
Also note that it will slow down the server a lot (and that might
result in the server not failing?).
If you post this to linux-nfs@vger.kernel.org they will probably
tell you to contact Debian first, but you can try.
- They will want to know as much as you can tell them about
the Debian server. (kernel version, what type of file system
is exported,...).
If you are really lucky, they will recognize the bug and tell you
if/when it was fixed.
rick
>
> > [Severity level: Warning]
> > [Group: Protocol]
> > [Main Opcode: READDIR (26)]
> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy4P7RCt0rkV56UMS0hUu2yPHL1=4-YP9UtejNrUA8gmTA>