Date: Fri, 2 Jan 2009 13:58:06 -0200 From: "Fernan Aguero" <fernan.aguero@gmail.com> To: freebsd-perl@freebsd.org Subject: Storable byteorder incompatibilities in different FreeBSD installations Message-ID: <520894aa0901020758o79bb1233teee539f6d599d10a@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
------=_Part_85910_20528024.1230911886862
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Hi,
I've been bitten by binary incompatibilitiess (bytorder, and/or byte
sizes of ints, longs, and whatnot) between two perl installations
(both on FreeBSD, using latest perl5.8 from ports).
Upon investigation, it does not seem clear to me why the two
installations should be different. And it prompted me to ask a few
questions :)
The context:
I have a perl web application running under mod_perl in FreeBSD that
dumps Perl data structures using Storable (binary format) and stores
them, after encoding them with Base64, into a MySQL database. This
application is under production and we have just deployed a test
server for development in another FreeBSD box.
Binary incompatibilities when reading the Storable binary data have
been observed in the past when running the web application from Linux
boxes (Ubuntu, i386 or amd64).
Now, they have been also observed under FreeBSD in the new test
server. This prompted us to do an investigation of the issue, because
up to now we were assuming we would have no problems if we kept our
production servers running FreeBSD.
Below is the description of each box and the output of:
perl -MConfig -le "print @Config{byteorder}"
perl -MConfig -le "print @Config{shortsize}"
perl -MConfig -le "print @Config{intsize}"
perl -MConfig -le "print @Config{longsize}"
perl -MConfig -le "print @Config{longlongsize}"
perl -V (I'm not including the full output, due to its size, only
diffs between selected cases)
1. Production server: FreeBSD-6.3, i386, perl5.8 from ports
byteorder: 12345678
shortsize: 2
intsize: 4
longsize: 4
lonlongsize: 8
2. Development/test server: FreeBSD-7.1-RC1, amd64, perl5.8 from ports
byteorder: 12345678
shortsize: 2
intsize: 4
longsize:8
lonlongsize: 8
3. Yet another server where we did some tests: FreeBSD-6.3, i386,
perl5.8 from ports, perl5.8 standalone build
perl5.8 from ports
byteorder: 12345678
shortsize: 2
intsize: 4
longsize: 4
lonlongsize: 8
perl5.8, standalone build
cd /usr/ports/lang/perl5.8 && make extract
cd work/perl-5.8.8 && ./Configure (answering all questions with the
default values except for the prefix installation)
make && make test && make install (PREFIX is /home/fernan/perl)
byteorder: 1234
shortsize: 2
intsize: 4
longsize: 4
lonlongsize: 8
As you can see, a Perl built outside of the ports system, has a
different binary configuration (and should produce incompatible
Storable images) as the Perl built from ports. As a sidenote, the
binary configuration is similar to the one found in ubuntu
pre-compiled binary perl packages (Linux, see below).
Attached to this message is a diff of the output produced by 'perl -V'
both for the perl built from the lang/perl5.8 port, and from the perl
built from sources, as explained above.
4. Other test boxes: Ubuntu-8.10, i386, perl5.8 and perl5.10
byteorder: 1234
shortsize: 2
intsize: 4
longsize: 4
lonlongsize: 8
Up to now the questions that come to mind are:
i) why is it that the perl built from ports produces binary dumps
(using Storable) with incompatible byteorder with respect to the one
built from Perl sources? (both perls were built on the same platform,
same gcc)
ii) Why does the Perl built from ports have an unusual byteorder?
iii) Why does Perl built from ports in different platforms produce
incompatible binary dumps (this time because of a difference in
longsize)? Is it because it was compiled with a different gcc (3.4.6
in FBSD-6.3 vs 4.2.1 in FBSD-7.1)? Or because it was compiled in a
different platform (amd64 vs i386)?
Reading the Configure script that comes with perl-5.8.8, and regarding
the byteorder, it says that:
"A big-endian machine like a Pyramid or a Motorola 680?0 chip will
come out to 4321. A little-endian machine like a Vax or an Intel 80?86
chip would be 1234. Other machines may have weird orders like 3412. A
Cray will report 87654321,
an Alpha will report 12345678. If the test program works the default is
probably right. I'm now running the test program..."
So why is it that building Perl outside of the ports system makes the
test program report '1234' which is what I would expect from an i386
box, while it gives '12345678' when running from within the ports
system? Why this unusual byteorder?
And now for a pragmatic and urgent question:
iv) how can I use the ports system to build a Perl in different FBSD
boxes that would result in the same binary configuration (byteorder,
shortsize, intsize, etc.) to guarantee it would produce compatible
(i.e. readable from other FBSD boxes) dumps of Perl data structures?
Sorry for the long post, and have yourselves a happy and peaceful 2009!
--
fernan
------=_Part_85910_20528024.1230911886862
Content-Type: text/x-patch; name=perl.gama.diff
Content-Transfer-Encoding: base64
X-Attachment-Id: f_fph0qahy0
Content-Disposition: attachment; filename=perl.gama.diff
LS0tIHBlcmwuc3RhbmRhbG9uZS5nYW1hCTIwMDgtMTItMzEgMTI6NDM6MDQuMDAwMDAwMDAwIC0w
MzAwCisrKyBwZXJsLnBvcnRzLmdhbWEJMjAwOC0xMi0zMSAxMjo0MzoxMi4wMDAwMDAwMDAgLTAz
MDAKQEAgLTEsNDIgKzEsNDYgQEAKIFN1bW1hcnkgb2YgbXkgcGVybDUgKHJldmlzaW9uIDUgdmVy
c2lvbiA4IHN1YnZlcnNpb24gOCkgY29uZmlndXJhdGlvbjoKICAgUGxhdGZvcm06Ci0gICAgb3Nu
YW1lPWZyZWVic2QsIG9zdmVycz02LjMtcmVsZWFzZS1wNSwgYXJjaG5hbWU9aTM4Ni1mcmVlYnNk
Ci0gICAgdW5hbWU9J2ZyZWVic2QgZ2FtYS5paWIudW5zYW0uZWR1LmFyIDYuMy1yZWxlYXNlLXA1
IGZyZWVic2QgNi4zLXJlbGVhc2UtcDUgIzA6IHNhdCBvY3QgMTEgMTM6MTQ6MTQgYXJ0IDIwMDgg
ZmVybmFuQGdhbWEuaWliLnVuc2FtLmVkdS5hcjp1c3JvYmpmcmVlYnNkZnJlZWJzZC02LjNzcmNz
eXNnYW1hIGkzODYgJwotICAgIGNvbmZpZ19hcmdzPScnCisgICAgb3NuYW1lPWZyZWVic2QsIG9z
dmVycz02LjMtcmVsZWFzZS1wMSwgYXJjaG5hbWU9aTM4Ni1mcmVlYnNkLTY0aW50CisgICAgdW5h
bWU9J2ZyZWVic2QgZ2FtYS5paWIudW5zYW0uZWR1LmFyIDYuMy1yZWxlYXNlLXAxIGZyZWVic2Qg
Ni4zLXJlbGVhc2UtcDEgIzA6IHdlZCBhcHIgMiAxODozNzoxNCBhcnQgMjAwOCBmZXJuYW5AZ2Ft
YS5paWIudW5zYW0uZWR1LmFyOnVzcm9iamZyZWVic2RmcmVlYnNkLTYuM3NyY3N5c2dhbWEgaTM4
NiAnCisgICAgY29uZmlnX2FyZ3M9Jy1zZGUgLURwcmVmaXg9L3Vzci9sb2NhbCAtRGFyY2hsaWI9
L3Vzci9sb2NhbC9saWIvcGVybDUvNS44LjgvbWFjaCAtRHByaXZsaWI9L3Vzci9sb2NhbC9saWIv
cGVybDUvNS44LjggLURtYW4zZGlyPS91c3IvbG9jYWwvbGliL3Blcmw1LzUuOC44L3BlcmwvbWFu
L21hbjMgLURtYW4xZGlyPS91c3IvbG9jYWwvbWFuL21hbjEgLURzaXRlYXJjaD0vdXNyL2xvY2Fs
L2xpYi9wZXJsNS9zaXRlX3BlcmwvNS44LjgvbWFjaCAtRHNpdGVsaWI9L3Vzci9sb2NhbC9saWIv
cGVybDUvc2l0ZV9wZXJsLzUuOC44IC1Ec2NyaXB0ZGlyPS91c3IvbG9jYWwvYmluIC1Ec2l0ZW1h
bjNkaXI9L3Vzci9sb2NhbC9saWIvcGVybDUvNS44LjgvbWFuL21hbjMgLURzaXRlbWFuMWRpcj0v
dXNyL2xvY2FsL21hbi9tYW4xIC1VaV9tYWxsb2MgLVVpX2ljb252IC1VaW5zdGFsbHVzcmJpbnBl
cmwgLURjYz1jYyAtRHVzZXNocnBsaWIgLURjY2ZsYWdzPS1EQVBQTExJQl9FWFA9Ii91c3IvbG9j
YWwvbGliL3Blcmw1LzUuOC44L0JTRFBBTiIgLURvcHRpbWl6ZT0tTzIgLWZuby1zdHJpY3QtYWxp
YXNpbmcgLXBpcGUgIC1VZF9kb3N1aWQgLVVpX2dkYm0gLUR1c2V0aHJlYWRzPW4gLUR1c2VteW1h
bGxvYz15IC1EdXNlNjRiaXRpbnQnCiAgICAgaGludD1yZWNvbW1lbmRlZCwgdXNlcG9zaXg9dHJ1
ZSwgZF9zaWdhY3Rpb249ZGVmaW5lCiAgICAgdXNldGhyZWFkcz11bmRlZiB1c2U1MDA1dGhyZWFk
cz11bmRlZiB1c2VpdGhyZWFkcz11bmRlZiB1c2VtdWx0aXBsaWNpdHk9dW5kZWYKICAgICB1c2Vw
ZXJsaW89ZGVmaW5lIGRfc2Zpbz11bmRlZiB1c2VsYXJnZWZpbGVzPWRlZmluZSB1c2Vzb2Nrcz11
bmRlZgotICAgIHVzZTY0Yml0aW50PXVuZGVmIHVzZTY0Yml0YWxsPXVuZGVmIHVzZWxvbmdkb3Vi
bGU9dW5kZWYKLSAgICB1c2VteW1hbGxvYz1uLCBiaW5jb21wYXQ1MDA1PXVuZGVmCisgICAgdXNl
NjRiaXRpbnQ9ZGVmaW5lIHVzZTY0Yml0YWxsPXVuZGVmIHVzZWxvbmdkb3VibGU9dW5kZWYKKyAg
ICB1c2VteW1hbGxvYz15LCBiaW5jb21wYXQ1MDA1PXVuZGVmCiAgIENvbXBpbGVyOgotICAgIGNj
PSdjYycsIGNjZmxhZ3MgPSctREhBU19GUFNFVE1BU0sgLURIQVNfRkxPQVRJTkdQT0lOVF9IIC1m
bm8tc3RyaWN0LWFsaWFzaW5nIC1waXBlIC1XZGVjbGFyYXRpb24tYWZ0ZXItc3RhdGVtZW50IC1J
L3Vzci9sb2NhbC9pbmNsdWRlJywKLSAgICBvcHRpbWl6ZT0nLU8nLAotICAgIGNwcGZsYWdzPSct
REhBU19GUFNFVE1BU0sgLURIQVNfRkxPQVRJTkdQT0lOVF9IIC1mbm8tc3RyaWN0LWFsaWFzaW5n
IC1waXBlIC1XZGVjbGFyYXRpb24tYWZ0ZXItc3RhdGVtZW50IC1JL3Vzci9sb2NhbC9pbmNsdWRl
JworICAgIGNjPSdjYycsIGNjZmxhZ3MgPSctREFQUExMSUJfRVhQPSIvdXNyL2xvY2FsL2xpYi9w
ZXJsNS81LjguOC9CU0RQQU4iIC1ESEFTX0ZQU0VUTUFTSyAtREhBU19GTE9BVElOR1BPSU5UX0gg
LWZuby1zdHJpY3QtYWxpYXNpbmcgLXBpcGUgLVdkZWNsYXJhdGlvbi1hZnRlci1zdGF0ZW1lbnQg
LUkvdXNyL2xvY2FsL2luY2x1ZGUnLAorICAgIG9wdGltaXplPSctTzIgLWZuby1zdHJpY3QtYWxp
YXNpbmcgLXBpcGUgJywKKyAgICBjcHBmbGFncz0nLURBUFBMTElCX0VYUD0iL3Vzci9sb2NhbC9s
aWIvcGVybDUvNS44LjgvQlNEUEFOIiAtREhBU19GUFNFVE1BU0sgLURIQVNfRkxPQVRJTkdQT0lO
VF9IIC1mbm8tc3RyaWN0LWFsaWFzaW5nIC1waXBlIC1XZGVjbGFyYXRpb24tYWZ0ZXItc3RhdGVt
ZW50IC1JL3Vzci9sb2NhbC9pbmNsdWRlJwogICAgIGNjdmVyc2lvbj0nJywgZ2NjdmVyc2lvbj0n
My40LjYgW0ZyZWVCU0RdIDIwMDYwMzA1JywgZ2Njb3NhbmR2ZXJzPScnCi0gICAgaW50c2l6ZT00
LCBsb25nc2l6ZT00LCBwdHJzaXplPTQsIGRvdWJsZXNpemU9OCwgYnl0ZW9yZGVyPTEyMzQKKyAg
ICBpbnRzaXplPTQsIGxvbmdzaXplPTQsIHB0cnNpemU9NCwgZG91Ymxlc2l6ZT04LCBieXRlb3Jk
ZXI9MTIzNDU2NzgKICAgICBkX2xvbmdsb25nPWRlZmluZSwgbG9uZ2xvbmdzaXplPTgsIGRfbG9u
Z2RibD1kZWZpbmUsIGxvbmdkYmxzaXplPTEyCi0gICAgaXZ0eXBlPSdsb25nJywgaXZzaXplPTQs
IG52dHlwZT0nZG91YmxlJywgbnZzaXplPTgsIE9mZl90PSdvZmZfdCcsIGxzZWVrc2l6ZT04Cisg
ICAgaXZ0eXBlPSdsb25nIGxvbmcnLCBpdnNpemU9OCwgbnZ0eXBlPSdkb3VibGUnLCBudnNpemU9
OCwgT2ZmX3Q9J29mZl90JywgbHNlZWtzaXplPTgKICAgICBhbGlnbmJ5dGVzPTQsIHByb3RvdHlw
ZT1kZWZpbmUKICAgTGlua2VyIGFuZCBMaWJyYXJpZXM6Ci0gICAgbGQ9J2NjJywgbGRmbGFncyA9
Jy1XbCwtRSAgLUwvdXNyL2xvY2FsL2xpYicKKyAgICBsZD0nY2MnLCBsZGZsYWdzID0nIC1XbCwt
RSAtTC91c3IvbG9jYWwvbGliJwogICAgIGxpYnB0aD0vdXNyL2xpYiAvdXNyL2xvY2FsL2xpYgot
ICAgIGxpYnM9LWxnZGJtIC1sbSAtbGNyeXB0IC1sdXRpbCAtbGMKLSAgICBwZXJsbGlicz0tbG0g
LWxjcnlwdCAtbHV0aWwgLWxjCi0gICAgbGliYz0sIHNvPXNvLCB1c2VzaHJwbGliPWZhbHNlLCBs
aWJwZXJsPWxpYnBlcmwuYQorICAgIGxpYnM9LWxnZGJtIC1sbSAtbGNyeXB0IC1sdXRpbAorICAg
IHBlcmxsaWJzPS1sbSAtbGNyeXB0IC1sdXRpbAorICAgIGxpYmM9LCBzbz1zbywgdXNlc2hycGxp
Yj10cnVlLCBsaWJwZXJsPWxpYnBlcmwuc28KICAgICBnbnVsaWJjX3ZlcnNpb249JycKICAgRHlu
YW1pYyBMaW5raW5nOgotICAgIGRsc3JjPWRsX2Rsb3Blbi54cywgZGxleHQ9c28sIGRfZGxzeW11
bj11bmRlZiwgY2NkbGZsYWdzPScgJworICAgIGRsc3JjPWRsX2Rsb3Blbi54cywgZGxleHQ9c28s
IGRfZGxzeW11bj11bmRlZiwgY2NkbGZsYWdzPScgIC1XbCwtUi91c3IvbG9jYWwvbGliL3Blcmw1
LzUuOC44L21hY2gvQ09SRScKICAgICBjY2NkbGZsYWdzPSctRFBJQyAtZlBJQycsIGxkZGxmbGFn
cz0nLXNoYXJlZCAgLUwvdXNyL2xvY2FsL2xpYicKIAogCiBDaGFyYWN0ZXJpc3RpY3Mgb2YgdGhp
cyBiaW5hcnkgKGZyb20gbGlicGVybCk6IAotICBDb21waWxlLXRpbWUgb3B0aW9uczogUEVSTF9N
QUxMT0NfV1JBUCBVU0VfTEFSR0VfRklMRVMgVVNFX1BFUkxJTworICBDb21waWxlLXRpbWUgb3B0
aW9uczogTVlNQUxMT0MgUEVSTF9NQUxMT0NfV1JBUCBVU0VfNjRfQklUX0lOVAorICAgICAgICAg
ICAgICAgICAgICAgICAgVVNFX0xBUkdFX0ZJTEVTIFVTRV9QRVJMSU8KKyAgTG9jYWxseSBhcHBs
aWVkIHBhdGNoZXM6CisJZGVmaW5lZC1vcgogICBCdWlsdCB1bmRlciBmcmVlYnNkCi0gIENvbXBp
bGVkIGF0IERlYyAyMyAyMDA4IDIwOjI2OjQzCisgIENvbXBpbGVkIGF0IE1heSAyMCAyMDA4IDEw
OjMzOjA1CiAgIEBJTkM6Ci0gICAgL2hvbWUvZmVybmFuL3BlcmwvbGliLzUuOC44L2kzODYtZnJl
ZWJzZAotICAgIC9ob21lL2Zlcm5hbi9wZXJsL2xpYi81LjguOAotICAgIC9ob21lL2Zlcm5hbi9w
ZXJsL2xpYi9zaXRlX3BlcmwvNS44LjgvaTM4Ni1mcmVlYnNkCi0gICAgL2hvbWUvZmVybmFuL3Bl
cmwvbGliL3NpdGVfcGVybC81LjguOAotICAgIC9ob21lL2Zlcm5hbi9wZXJsL2xpYi9zaXRlX3Bl
cmwKKyAgICAvdXNyL2xvY2FsL2xpYi9wZXJsNS81LjguOC9CU0RQQU4KKyAgICAvdXNyL2xvY2Fs
L2xpYi9wZXJsNS9zaXRlX3BlcmwvNS44LjgvbWFjaAorICAgIC91c3IvbG9jYWwvbGliL3Blcmw1
L3NpdGVfcGVybC81LjguOAorICAgIC91c3IvbG9jYWwvbGliL3Blcmw1L3NpdGVfcGVybAorICAg
IC91c3IvbG9jYWwvbGliL3Blcmw1LzUuOC44L21hY2gKKyAgICAvdXNyL2xvY2FsL2xpYi9wZXJs
NS81LjguOAogICAgIC4K
------=_Part_85910_20528024.1230911886862--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?520894aa0901020758o79bb1233teee539f6d599d10a>