From owner-freebsd-perl@FreeBSD.ORG Fri Jan 2 16:31:05 2009 Return-Path: Delivered-To: freebsd-perl@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67AF41065676 for ; Fri, 2 Jan 2009 16:31:05 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.26]) by mx1.freebsd.org (Postfix) with ESMTP id 170408FC08 for ; Fri, 2 Jan 2009 16:31:04 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: by qw-out-2122.google.com with SMTP id 9so3833058qwb.7 for ; Fri, 02 Jan 2009 08:31:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:mime-version:content-type; bh=xGsrgY6SCZh9G85bPpLhBypmI1lPUBVFL703cU93eTI=; b=vHRaMNHx8iSSBqc3Z9Mo9mMBM5M1AnSLvffea2KcUF52ajKdh0nbDi8bQSA1SiJPUL DMNRHfie1lam8SExN4waBBEpRsLpDB8IQdWJ0lfOoaIeJI+w03AGLHv36u3ZHks43G7Y 2WZ4VpZH9S+QLSaIbhXqW3ywPnurs1Y56bRxQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:mime-version:content-type; b=iXvp14bVH3jrBiHlsR+jJAvpGC6ZT5zCksTkEo12LV49IPEt3qSnWqbmAuqrNoCpuK j3PmL6yLzQR8bdYsuGk+7tIyDu6+GYLqKO8O+cge3HCtFyFEcO5D4cXump0UYNfb007s xXIKkVT3gfv8aKrlNVFYLZmoZesmpux6c/3Gc= Received: by 10.214.80.21 with SMTP id d21mr15006796qab.267.1230911886861; Fri, 02 Jan 2009 07:58:06 -0800 (PST) Received: by 10.214.45.3 with HTTP; Fri, 2 Jan 2009 07:58:06 -0800 (PST) Message-ID: <520894aa0901020758o79bb1233teee539f6d599d10a@mail.gmail.com> Date: Fri, 2 Jan 2009 13:58:06 -0200 From: "Fernan Aguero" To: freebsd-perl@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_85910_20528024.1230911886862" Cc: Subject: Storable byteorder incompatibilities in different FreeBSD installations X-BeenThere: freebsd-perl@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: maintainer of a number of perl-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jan 2009 16:31:06 -0000 ------=_Part_85910_20528024.1230911886862 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi, I've been bitten by binary incompatibilitiess (bytorder, and/or byte sizes of ints, longs, and whatnot) between two perl installations (both on FreeBSD, using latest perl5.8 from ports). Upon investigation, it does not seem clear to me why the two installations should be different. And it prompted me to ask a few questions :) The context: I have a perl web application running under mod_perl in FreeBSD that dumps Perl data structures using Storable (binary format) and stores them, after encoding them with Base64, into a MySQL database. This application is under production and we have just deployed a test server for development in another FreeBSD box. Binary incompatibilities when reading the Storable binary data have been observed in the past when running the web application from Linux boxes (Ubuntu, i386 or amd64). Now, they have been also observed under FreeBSD in the new test server. This prompted us to do an investigation of the issue, because up to now we were assuming we would have no problems if we kept our production servers running FreeBSD. Below is the description of each box and the output of: perl -MConfig -le "print @Config{byteorder}" perl -MConfig -le "print @Config{shortsize}" perl -MConfig -le "print @Config{intsize}" perl -MConfig -le "print @Config{longsize}" perl -MConfig -le "print @Config{longlongsize}" perl -V (I'm not including the full output, due to its size, only diffs between selected cases) 1. Production server: FreeBSD-6.3, i386, perl5.8 from ports byteorder: 12345678 shortsize: 2 intsize: 4 longsize: 4 lonlongsize: 8 2. Development/test server: FreeBSD-7.1-RC1, amd64, perl5.8 from ports byteorder: 12345678 shortsize: 2 intsize: 4 longsize:8 lonlongsize: 8 3. Yet another server where we did some tests: FreeBSD-6.3, i386, perl5.8 from ports, perl5.8 standalone build perl5.8 from ports byteorder: 12345678 shortsize: 2 intsize: 4 longsize: 4 lonlongsize: 8 perl5.8, standalone build cd /usr/ports/lang/perl5.8 && make extract cd work/perl-5.8.8 && ./Configure (answering all questions with the default values except for the prefix installation) make && make test && make install (PREFIX is /home/fernan/perl) byteorder: 1234 shortsize: 2 intsize: 4 longsize: 4 lonlongsize: 8 As you can see, a Perl built outside of the ports system, has a different binary configuration (and should produce incompatible Storable images) as the Perl built from ports. As a sidenote, the binary configuration is similar to the one found in ubuntu pre-compiled binary perl packages (Linux, see below). Attached to this message is a diff of the output produced by 'perl -V' both for the perl built from the lang/perl5.8 port, and from the perl built from sources, as explained above. 4. Other test boxes: Ubuntu-8.10, i386, perl5.8 and perl5.10 byteorder: 1234 shortsize: 2 intsize: 4 longsize: 4 lonlongsize: 8 Up to now the questions that come to mind are: i) why is it that the perl built from ports produces binary dumps (using Storable) with incompatible byteorder with respect to the one built from Perl sources? (both perls were built on the same platform, same gcc) ii) Why does the Perl built from ports have an unusual byteorder? iii) Why does Perl built from ports in different platforms produce incompatible binary dumps (this time because of a difference in longsize)? Is it because it was compiled with a different gcc (3.4.6 in FBSD-6.3 vs 4.2.1 in FBSD-7.1)? Or because it was compiled in a different platform (amd64 vs i386)? Reading the Configure script that comes with perl-5.8.8, and regarding the byteorder, it says that: "A big-endian machine like a Pyramid or a Motorola 680?0 chip will come out to 4321. A little-endian machine like a Vax or an Intel 80?86 chip would be 1234. Other machines may have weird orders like 3412. A Cray will report 87654321, an Alpha will report 12345678. If the test program works the default is probably right. I'm now running the test program..." So why is it that building Perl outside of the ports system makes the test program report '1234' which is what I would expect from an i386 box, while it gives '12345678' when running from within the ports system? Why this unusual byteorder? And now for a pragmatic and urgent question: iv) how can I use the ports system to build a Perl in different FBSD boxes that would result in the same binary configuration (byteorder, shortsize, intsize, etc.) to guarantee it would produce compatible (i.e. readable from other FBSD boxes) dumps of Perl data structures? Sorry for the long post, and have yourselves a happy and peaceful 2009! -- fernan ------=_Part_85910_20528024.1230911886862 Content-Type: text/x-patch; name=perl.gama.diff Content-Transfer-Encoding: base64 X-Attachment-Id: f_fph0qahy0 Content-Disposition: attachment; filename=perl.gama.diff LS0tIHBlcmwuc3RhbmRhbG9uZS5nYW1hCTIwMDgtMTItMzEgMTI6NDM6MDQuMDAwMDAwMDAwIC0w MzAwCisrKyBwZXJsLnBvcnRzLmdhbWEJMjAwOC0xMi0zMSAxMjo0MzoxMi4wMDAwMDAwMDAgLTAz MDAKQEAgLTEsNDIgKzEsNDYgQEAKIFN1bW1hcnkgb2YgbXkgcGVybDUgKHJldmlzaW9uIDUgdmVy c2lvbiA4IHN1YnZlcnNpb24gOCkgY29uZmlndXJhdGlvbjoKICAgUGxhdGZvcm06Ci0gICAgb3Nu YW1lPWZyZWVic2QsIG9zdmVycz02LjMtcmVsZWFzZS1wNSwgYXJjaG5hbWU9aTM4Ni1mcmVlYnNk Ci0gICAgdW5hbWU9J2ZyZWVic2QgZ2FtYS5paWIudW5zYW0uZWR1LmFyIDYuMy1yZWxlYXNlLXA1 IGZyZWVic2QgNi4zLXJlbGVhc2UtcDUgIzA6IHNhdCBvY3QgMTEgMTM6MTQ6MTQgYXJ0IDIwMDgg ZmVybmFuQGdhbWEuaWliLnVuc2FtLmVkdS5hcjp1c3JvYmpmcmVlYnNkZnJlZWJzZC02LjNzcmNz eXNnYW1hIGkzODYgJwotICAgIGNvbmZpZ19hcmdzPScnCisgICAgb3NuYW1lPWZyZWVic2QsIG9z dmVycz02LjMtcmVsZWFzZS1wMSwgYXJjaG5hbWU9aTM4Ni1mcmVlYnNkLTY0aW50CisgICAgdW5h bWU9J2ZyZWVic2QgZ2FtYS5paWIudW5zYW0uZWR1LmFyIDYuMy1yZWxlYXNlLXAxIGZyZWVic2Qg Ni4zLXJlbGVhc2UtcDEgIzA6IHdlZCBhcHIgMiAxODozNzoxNCBhcnQgMjAwOCBmZXJuYW5AZ2Ft YS5paWIudW5zYW0uZWR1LmFyOnVzcm9iamZyZWVic2RmcmVlYnNkLTYuM3NyY3N5c2dhbWEgaTM4 NiAnCisgICAgY29uZmlnX2FyZ3M9Jy1zZGUgLURwcmVmaXg9L3Vzci9sb2NhbCAtRGFyY2hsaWI9 L3Vzci9sb2NhbC9saWIvcGVybDUvNS44LjgvbWFjaCAtRHByaXZsaWI9L3Vzci9sb2NhbC9saWIv cGVybDUvNS44LjggLURtYW4zZGlyPS91c3IvbG9jYWwvbGliL3Blcmw1LzUuOC44L3BlcmwvbWFu L21hbjMgLURtYW4xZGlyPS91c3IvbG9jYWwvbWFuL21hbjEgLURzaXRlYXJjaD0vdXNyL2xvY2Fs L2xpYi9wZXJsNS9zaXRlX3BlcmwvNS44LjgvbWFjaCAtRHNpdGVsaWI9L3Vzci9sb2NhbC9saWIv cGVybDUvc2l0ZV9wZXJsLzUuOC44IC1Ec2NyaXB0ZGlyPS91c3IvbG9jYWwvYmluIC1Ec2l0ZW1h bjNkaXI9L3Vzci9sb2NhbC9saWIvcGVybDUvNS44LjgvbWFuL21hbjMgLURzaXRlbWFuMWRpcj0v dXNyL2xvY2FsL21hbi9tYW4xIC1VaV9tYWxsb2MgLVVpX2ljb252IC1VaW5zdGFsbHVzcmJpbnBl cmwgLURjYz1jYyAtRHVzZXNocnBsaWIgLURjY2ZsYWdzPS1EQVBQTExJQl9FWFA9Ii91c3IvbG9j YWwvbGliL3Blcmw1LzUuOC44L0JTRFBBTiIgLURvcHRpbWl6ZT0tTzIgLWZuby1zdHJpY3QtYWxp YXNpbmcgLXBpcGUgIC1VZF9kb3N1aWQgLVVpX2dkYm0gLUR1c2V0aHJlYWRzPW4gLUR1c2VteW1h bGxvYz15IC1EdXNlNjRiaXRpbnQnCiAgICAgaGludD1yZWNvbW1lbmRlZCwgdXNlcG9zaXg9dHJ1 ZSwgZF9zaWdhY3Rpb249ZGVmaW5lCiAgICAgdXNldGhyZWFkcz11bmRlZiB1c2U1MDA1dGhyZWFk cz11bmRlZiB1c2VpdGhyZWFkcz11bmRlZiB1c2VtdWx0aXBsaWNpdHk9dW5kZWYKICAgICB1c2Vw ZXJsaW89ZGVmaW5lIGRfc2Zpbz11bmRlZiB1c2VsYXJnZWZpbGVzPWRlZmluZSB1c2Vzb2Nrcz11 bmRlZgotICAgIHVzZTY0Yml0aW50PXVuZGVmIHVzZTY0Yml0YWxsPXVuZGVmIHVzZWxvbmdkb3Vi bGU9dW5kZWYKLSAgICB1c2VteW1hbGxvYz1uLCBiaW5jb21wYXQ1MDA1PXVuZGVmCisgICAgdXNl NjRiaXRpbnQ9ZGVmaW5lIHVzZTY0Yml0YWxsPXVuZGVmIHVzZWxvbmdkb3VibGU9dW5kZWYKKyAg ICB1c2VteW1hbGxvYz15LCBiaW5jb21wYXQ1MDA1PXVuZGVmCiAgIENvbXBpbGVyOgotICAgIGNj PSdjYycsIGNjZmxhZ3MgPSctREhBU19GUFNFVE1BU0sgLURIQVNfRkxPQVRJTkdQT0lOVF9IIC1m bm8tc3RyaWN0LWFsaWFzaW5nIC1waXBlIC1XZGVjbGFyYXRpb24tYWZ0ZXItc3RhdGVtZW50IC1J L3Vzci9sb2NhbC9pbmNsdWRlJywKLSAgICBvcHRpbWl6ZT0nLU8nLAotICAgIGNwcGZsYWdzPSct REhBU19GUFNFVE1BU0sgLURIQVNfRkxPQVRJTkdQT0lOVF9IIC1mbm8tc3RyaWN0LWFsaWFzaW5n IC1waXBlIC1XZGVjbGFyYXRpb24tYWZ0ZXItc3RhdGVtZW50IC1JL3Vzci9sb2NhbC9pbmNsdWRl JworICAgIGNjPSdjYycsIGNjZmxhZ3MgPSctREFQUExMSUJfRVhQPSIvdXNyL2xvY2FsL2xpYi9w ZXJsNS81LjguOC9CU0RQQU4iIC1ESEFTX0ZQU0VUTUFTSyAtREhBU19GTE9BVElOR1BPSU5UX0gg LWZuby1zdHJpY3QtYWxpYXNpbmcgLXBpcGUgLVdkZWNsYXJhdGlvbi1hZnRlci1zdGF0ZW1lbnQg LUkvdXNyL2xvY2FsL2luY2x1ZGUnLAorICAgIG9wdGltaXplPSctTzIgLWZuby1zdHJpY3QtYWxp YXNpbmcgLXBpcGUgJywKKyAgICBjcHBmbGFncz0nLURBUFBMTElCX0VYUD0iL3Vzci9sb2NhbC9s aWIvcGVybDUvNS44LjgvQlNEUEFOIiAtREhBU19GUFNFVE1BU0sgLURIQVNfRkxPQVRJTkdQT0lO VF9IIC1mbm8tc3RyaWN0LWFsaWFzaW5nIC1waXBlIC1XZGVjbGFyYXRpb24tYWZ0ZXItc3RhdGVt ZW50IC1JL3Vzci9sb2NhbC9pbmNsdWRlJwogICAgIGNjdmVyc2lvbj0nJywgZ2NjdmVyc2lvbj0n My40LjYgW0ZyZWVCU0RdIDIwMDYwMzA1JywgZ2Njb3NhbmR2ZXJzPScnCi0gICAgaW50c2l6ZT00 LCBsb25nc2l6ZT00LCBwdHJzaXplPTQsIGRvdWJsZXNpemU9OCwgYnl0ZW9yZGVyPTEyMzQKKyAg ICBpbnRzaXplPTQsIGxvbmdzaXplPTQsIHB0cnNpemU9NCwgZG91Ymxlc2l6ZT04LCBieXRlb3Jk ZXI9MTIzNDU2NzgKICAgICBkX2xvbmdsb25nPWRlZmluZSwgbG9uZ2xvbmdzaXplPTgsIGRfbG9u Z2RibD1kZWZpbmUsIGxvbmdkYmxzaXplPTEyCi0gICAgaXZ0eXBlPSdsb25nJywgaXZzaXplPTQs IG52dHlwZT0nZG91YmxlJywgbnZzaXplPTgsIE9mZl90PSdvZmZfdCcsIGxzZWVrc2l6ZT04Cisg ICAgaXZ0eXBlPSdsb25nIGxvbmcnLCBpdnNpemU9OCwgbnZ0eXBlPSdkb3VibGUnLCBudnNpemU9 OCwgT2ZmX3Q9J29mZl90JywgbHNlZWtzaXplPTgKICAgICBhbGlnbmJ5dGVzPTQsIHByb3RvdHlw ZT1kZWZpbmUKICAgTGlua2VyIGFuZCBMaWJyYXJpZXM6Ci0gICAgbGQ9J2NjJywgbGRmbGFncyA9 Jy1XbCwtRSAgLUwvdXNyL2xvY2FsL2xpYicKKyAgICBsZD0nY2MnLCBsZGZsYWdzID0nIC1XbCwt RSAtTC91c3IvbG9jYWwvbGliJwogICAgIGxpYnB0aD0vdXNyL2xpYiAvdXNyL2xvY2FsL2xpYgot ICAgIGxpYnM9LWxnZGJtIC1sbSAtbGNyeXB0IC1sdXRpbCAtbGMKLSAgICBwZXJsbGlicz0tbG0g LWxjcnlwdCAtbHV0aWwgLWxjCi0gICAgbGliYz0sIHNvPXNvLCB1c2VzaHJwbGliPWZhbHNlLCBs aWJwZXJsPWxpYnBlcmwuYQorICAgIGxpYnM9LWxnZGJtIC1sbSAtbGNyeXB0IC1sdXRpbAorICAg IHBlcmxsaWJzPS1sbSAtbGNyeXB0IC1sdXRpbAorICAgIGxpYmM9LCBzbz1zbywgdXNlc2hycGxp Yj10cnVlLCBsaWJwZXJsPWxpYnBlcmwuc28KICAgICBnbnVsaWJjX3ZlcnNpb249JycKICAgRHlu YW1pYyBMaW5raW5nOgotICAgIGRsc3JjPWRsX2Rsb3Blbi54cywgZGxleHQ9c28sIGRfZGxzeW11 bj11bmRlZiwgY2NkbGZsYWdzPScgJworICAgIGRsc3JjPWRsX2Rsb3Blbi54cywgZGxleHQ9c28s IGRfZGxzeW11bj11bmRlZiwgY2NkbGZsYWdzPScgIC1XbCwtUi91c3IvbG9jYWwvbGliL3Blcmw1 LzUuOC44L21hY2gvQ09SRScKICAgICBjY2NkbGZsYWdzPSctRFBJQyAtZlBJQycsIGxkZGxmbGFn cz0nLXNoYXJlZCAgLUwvdXNyL2xvY2FsL2xpYicKIAogCiBDaGFyYWN0ZXJpc3RpY3Mgb2YgdGhp cyBiaW5hcnkgKGZyb20gbGlicGVybCk6IAotICBDb21waWxlLXRpbWUgb3B0aW9uczogUEVSTF9N QUxMT0NfV1JBUCBVU0VfTEFSR0VfRklMRVMgVVNFX1BFUkxJTworICBDb21waWxlLXRpbWUgb3B0 aW9uczogTVlNQUxMT0MgUEVSTF9NQUxMT0NfV1JBUCBVU0VfNjRfQklUX0lOVAorICAgICAgICAg ICAgICAgICAgICAgICAgVVNFX0xBUkdFX0ZJTEVTIFVTRV9QRVJMSU8KKyAgTG9jYWxseSBhcHBs aWVkIHBhdGNoZXM6CisJZGVmaW5lZC1vcgogICBCdWlsdCB1bmRlciBmcmVlYnNkCi0gIENvbXBp bGVkIGF0IERlYyAyMyAyMDA4IDIwOjI2OjQzCisgIENvbXBpbGVkIGF0IE1heSAyMCAyMDA4IDEw OjMzOjA1CiAgIEBJTkM6Ci0gICAgL2hvbWUvZmVybmFuL3BlcmwvbGliLzUuOC44L2kzODYtZnJl ZWJzZAotICAgIC9ob21lL2Zlcm5hbi9wZXJsL2xpYi81LjguOAotICAgIC9ob21lL2Zlcm5hbi9w ZXJsL2xpYi9zaXRlX3BlcmwvNS44LjgvaTM4Ni1mcmVlYnNkCi0gICAgL2hvbWUvZmVybmFuL3Bl cmwvbGliL3NpdGVfcGVybC81LjguOAotICAgIC9ob21lL2Zlcm5hbi9wZXJsL2xpYi9zaXRlX3Bl cmwKKyAgICAvdXNyL2xvY2FsL2xpYi9wZXJsNS81LjguOC9CU0RQQU4KKyAgICAvdXNyL2xvY2Fs L2xpYi9wZXJsNS9zaXRlX3BlcmwvNS44LjgvbWFjaAorICAgIC91c3IvbG9jYWwvbGliL3Blcmw1 L3NpdGVfcGVybC81LjguOAorICAgIC91c3IvbG9jYWwvbGliL3Blcmw1L3NpdGVfcGVybAorICAg IC91c3IvbG9jYWwvbGliL3Blcmw1LzUuOC44L21hY2gKKyAgICAvdXNyL2xvY2FsL2xpYi9wZXJs NS81LjguOAogICAgIC4K ------=_Part_85910_20528024.1230911886862-- From owner-freebsd-perl@FreeBSD.ORG Fri Jan 2 17:00:01 2009 Return-Path: Delivered-To: freebsd-perl@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 862E4106564A for ; Fri, 2 Jan 2009 17:00:01 +0000 (UTC) (envelope-from tobez@tobez.org) Received: from heechee.tobez.org (heechee.tobez.org [194.255.56.42]) by mx1.freebsd.org (Postfix) with ESMTP id 42AEF8FC17 for ; Fri, 2 Jan 2009 17:00:01 +0000 (UTC) (envelope-from tobez@tobez.org) Received: by heechee.tobez.org (Postfix, from userid 1001) id DB0DE6D400; Fri, 2 Jan 2009 17:44:39 +0100 (CET) Date: Fri, 2 Jan 2009 17:44:39 +0100 From: Anton Berezin To: Fernan Aguero Message-ID: <20090102164439.GF40649@heechee.tobez.org> References: <520894aa0901020758o79bb1233teee539f6d599d10a@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <520894aa0901020758o79bb1233teee539f6d599d10a@mail.gmail.com> X-Powered-By: FreeBSD http://www.freebsd.org/ User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-perl@freebsd.org Subject: Re: Storable byteorder incompatibilities in different FreeBSD installations X-BeenThere: freebsd-perl@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: maintainer of a number of perl-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jan 2009 17:00:02 -0000 On Fri, Jan 02, 2009 at 01:58:06PM -0200, Fernan Aguero wrote: > I've been bitten by binary incompatibilitiess (bytorder, and/or byte > sizes of ints, longs, and whatnot) between two perl installations > (both on FreeBSD, using latest perl5.8 from ports). [...] > Binary incompatibilities when reading the Storable binary data have > been observed in the past when running the web application from Linux > boxes (Ubuntu, i386 or amd64). > So why is it that building Perl outside of the ports system makes the > test program report '1234' which is what I would expect from an i386 > box, while it gives '12345678' when running from within the ports > system? Why this unusual byteorder? This is because of PERL_64BITINT (aka -Duse64bitint) is the default with the port. > And now for a pragmatic and urgent question: > iv) how can I use the ports system to build a Perl in different FBSD > boxes that would result in the same binary configuration (byteorder, > shortsize, intsize, etc.) to guarantee it would produce compatible > (i.e. readable from other FBSD boxes) dumps of Perl data structures? Uncheck PERL_64BITINT in the configuration dialog. (make rmconfig first) Apart from that, do you actually use "nstore" variants of Storable functions? If you use it consistently in you application, you might find out that the problem disappears, and it would not matter what platform you use. Cheers, \Anton. -- There is no beauty in entropy. -- Eliezer Yudkowsky From owner-freebsd-perl@FreeBSD.ORG Sat Jan 3 12:33:23 2009 Return-Path: Delivered-To: freebsd-perl@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 50F80106564A for ; Sat, 3 Jan 2009 12:33:23 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.25]) by mx1.freebsd.org (Postfix) with ESMTP id 005BE8FC14 for ; Sat, 3 Jan 2009 12:33:22 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: by qw-out-2122.google.com with SMTP id 9so3980653qwb.7 for ; Sat, 03 Jan 2009 04:33:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=BrkfLdrS6GlCT90bY5LwklSzty/fBqTHEMrWSuWmlNs=; b=STqvaEorS0BrpwzK7ZdlrsPEMXRdG3pKl8DCBTlBm5PGquVckl0o/aYOIVIQ8DVP4+ Z1GVc1nTeCZo2muHBEKPN2HS7YauM8GVYijWmQ/1jySJZ0jfNRW9AtUBEdN9pA6W5AE6 c2VnKULrbMmuAStwpamaHEfxnNG0AkfPrMkMU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=OsKMGLQc0GuuIgvQLGIe8KZmTlwmi2tSfdAu4EXiRowPEwMEn84/imO9srgi+6p8zY v3uaZ7wPbHLU4fPgN5A8uM67WjD06k6FjwDFLSGVbxlPbNsXRP1mfIgafDbP58FZmzUC K/h80ofRIOVBZMoWi+SxatzzjUMdA5n6NoHN4= Received: by 10.215.39.5 with SMTP id r5mr15576959qaj.149.1230986002119; Sat, 03 Jan 2009 04:33:22 -0800 (PST) Received: by 10.214.45.3 with HTTP; Sat, 3 Jan 2009 04:33:22 -0800 (PST) Message-ID: <520894aa0901030433i7a103ba7i4236ae17b83c6eae@mail.gmail.com> Date: Sat, 3 Jan 2009 10:33:22 -0200 From: "Fernan Aguero" To: "Anton Berezin" In-Reply-To: <20090102164439.GF40649@heechee.tobez.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <520894aa0901020758o79bb1233teee539f6d599d10a@mail.gmail.com> <20090102164439.GF40649@heechee.tobez.org> Cc: freebsd-perl@freebsd.org Subject: Re: Storable byteorder incompatibilities in different FreeBSD installations X-BeenThere: freebsd-perl@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: maintainer of a number of perl-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jan 2009 12:33:23 -0000 Hi Anton, On Fri, Jan 2, 2009 at 2:44 PM, Anton Berezin wrote: > On Fri, Jan 02, 2009 at 01:58:06PM -0200, Fernan Aguero wrote: > >> I've been bitten by binary incompatibilitiess (bytorder, and/or byte >> sizes of ints, longs, and whatnot) between two perl installations >> (both on FreeBSD, using latest perl5.8 from ports). > > [...] > >> Binary incompatibilities when reading the Storable binary data have >> been observed in the past when running the web application from Linux >> boxes (Ubuntu, i386 or amd64). > >> So why is it that building Perl outside of the ports system makes the >> test program report '1234' which is what I would expect from an i386 >> box, while it gives '12345678' when running from within the ports >> system? Why this unusual byteorder? > > This is because of PERL_64BITINT (aka -Duse64bitint) is the > default with the port. That is only useful in i386. In amd64, it doesn't change anything. With or Without that define, the outcome is always: perl -MConfig -le "print @Config{byteorder}" 12345678 perl -MConfig -le "print @Config{shortsize}" 2 perl -MConfig -le "print @Config{intsize}" 4 perl -MConfig -le "print @Config{longsize}" 8 perl -MConfig -le "print @Config{longlongsize}" 8 [tested in FreeBSD-7.1 RC1, amd64] In FreeBSD-6.3, i386, unchecking PERL_64BITINT would give: perl -MConfig -le "print @Config{byteorder}" 1234 perl -MConfig -le "print @Config{longsize}" 4 [all others being equal] In FreeBSD-6.3, i386 with PERL_64BITINT: perl -MConfig -le "print @Config{byteorder}" 12345678 perl -MConfig -le "print @Config{longsize}" 4 [all others being equal] Apart from the byteorder issue, the other issue which I raised in my post, was also the incompatibility in 'longsize' between two perl instances built from ports, in different boxes. In this case, the perl built in a FBSD-6.3, i386 box has a longsize of 4, whereas the same perl built in FBSD-7.1, amd64 has a longsize of 8. In both cases they were compiled with PERL_64BITINT (the default for the port), and in both cases the byteorder is the same! How can I build a perl with a byteorder of 12345678, and a longsize of 4, in an amd64 platform? Is that possible at all? Looking at the output of perl -V in these different Perls, in the Compiler section I see (from the diff attached in my previous message, this is from a build in i386) - intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 + intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678 Can we toggle these compiler options? >> And now for a pragmatic and urgent question: >> iv) how can I use the ports system to build a Perl in different FBSD >> boxes that would result in the same binary configuration (byteorder, >> shortsize, intsize, etc.) to guarantee it would produce compatible >> (i.e. readable from other FBSD boxes) dumps of Perl data structures? > > Uncheck PERL_64BITINT in the configuration dialog. (make rmconfig > first) > > Apart from that, do you actually use "nstore" variants of Storable > functions? If you use it consistently in you application, you > might find out that the problem disappears, and it would not > matter what platform you use. I know about nstore, and I am switching my code to use nstore for future releases. But, as it stands now, I've already made the mistake of using plain 'store' in all my production code, which means I have lots of data already written in the DB that I need to be able to read with binary-compatible perl/Storable instances. I can do that currently using the perl instance I have in FBSD-6, i386. But I was wondering if it's possible to make the port build a perl with a different 'binary' configuration. You might want to think about this problem in the following terms: i) you have lots of data dumped from perl in a specific binary format ii) your old hardware is gone/lost iii) you need to build a perl in a new box, perhaps in a different platform, that would read these Storable dumps. Is this possible using the FreeBSD perl port? In my case, this would be translated to: can I build a perl instance with a byteorder of 12345678, and a longsize of 4 in a FreeBSD-7.1, amd64 box? Cheers, Fernan > Cheers, > \Anton. -- fernan From owner-freebsd-perl@FreeBSD.ORG Sat Jan 3 13:57:19 2009 Return-Path: Delivered-To: freebsd-perl@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 420D51065672 for ; Sat, 3 Jan 2009 13:57:19 +0000 (UTC) (envelope-from tobez@tobez.org) Received: from heechee.tobez.org (heechee.tobez.org [194.255.56.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0A1F28FC13 for ; Sat, 3 Jan 2009 13:57:18 +0000 (UTC) (envelope-from tobez@tobez.org) Received: by heechee.tobez.org (Postfix, from userid 1001) id 3F5FA6D401; Sat, 3 Jan 2009 14:57:18 +0100 (CET) Date: Sat, 3 Jan 2009 14:57:18 +0100 From: Anton Berezin To: Fernan Aguero Message-ID: <20090103135718.GC41513@heechee.tobez.org> References: <520894aa0901020758o79bb1233teee539f6d599d10a@mail.gmail.com> <20090102164439.GF40649@heechee.tobez.org> <520894aa0901030433i7a103ba7i4236ae17b83c6eae@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <520894aa0901030433i7a103ba7i4236ae17b83c6eae@mail.gmail.com> X-Powered-By: FreeBSD http://www.freebsd.org/ User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-perl@freebsd.org Subject: Re: Storable byteorder incompatibilities in different FreeBSD installations X-BeenThere: freebsd-perl@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: maintainer of a number of perl-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jan 2009 13:57:19 -0000 Fernan, On Sat, Jan 03, 2009 at 10:33:22AM -0200, Fernan Aguero wrote: > Apart from the byteorder issue, the other issue which I raised in my > post, was also the incompatibility in 'longsize' between two perl > instances built from ports, in different boxes. > > In this case, the perl built in a FBSD-6.3, i386 box has a longsize of > 4, whereas the same perl built in FBSD-7.1, amd64 has a longsize of 8. > In both cases they were compiled with PERL_64BITINT (the default for > the port), and in both cases the byteorder is the same! > > How can I build a perl with a byteorder of 12345678, and a longsize of > 4, in an amd64 platform? Is that possible at all? Not really. Amd64 is a 64-bit platform, meaning sizeof(long) = 8. I am pretty positive that the same is true for x86-64 Linux. That's more or less what "12345678" indicates. At any rate, this is the application design issue. The practical solution in your case would be to take an offline copy of the DB, and to go through every record and convert it into a KNOWN to you binary format. I have not done the relevant testing, so I do not even know whether nstore() will do what you want - after all, it only handles the byteorder issues, not integer size issues. It is dangerous to make assumptions about bytesizes of various structures on different architectures - you've got to do some sort of conversion yourself, "native to known" during the store operation, and "known to native" during the retrieve operation. This is just a common sense design principle. Cheers, \Anton. -- There is no beauty in entropy. -- Eliezer Yudkowsky From owner-freebsd-perl@FreeBSD.ORG Sat Jan 3 14:22:03 2009 Return-Path: Delivered-To: freebsd-perl@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42358106564A for ; Sat, 3 Jan 2009 14:22:03 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.27]) by mx1.freebsd.org (Postfix) with ESMTP id E9F878FC14 for ; Sat, 3 Jan 2009 14:22:02 +0000 (UTC) (envelope-from fernan.aguero@gmail.com) Received: by qw-out-2122.google.com with SMTP id 9so3991829qwb.7 for ; Sat, 03 Jan 2009 06:22:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=bAy2Fo05hweLdGNtivMDRXmEYvzdzZHPZ37wqUGhulo=; b=Cyjc+00bBZ1mUfT4OkwHIZysd+ob2AaXtbr7sU685z7JjWGUvtON/gG7qzMUKJfuIr JVn/QZn6c730KX/QIPQbbyInudyjMFY+SgvteIRYZB5tXYHOxBD61kuVpwOTDgSNuYo2 Wg8L5RfMVRBoMyLXSUKYrE7V01zCF1+Job4jM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=vPVoDOdFJWyDDTOf37/BDacdldgYjk2TQvBjJpL3YnO3djAOx1oQvawv1gLSIrFEAy BKPH08WtGQcU5V9TAVSav6ksn8TUolTSwteGI46R/T+SC6x0YLVHYVg7IhPsWUJd5SMv YGGlYy4jNX6Uvv5MDOI92ReveaJnUePHPp7G8= Received: by 10.214.148.19 with SMTP id v19mr15562045qad.351.1230992522022; Sat, 03 Jan 2009 06:22:02 -0800 (PST) Received: by 10.214.45.3 with HTTP; Sat, 3 Jan 2009 06:22:01 -0800 (PST) Message-ID: <520894aa0901030622k173d9eu2a2792758abbcfee@mail.gmail.com> Date: Sat, 3 Jan 2009 12:22:01 -0200 From: "Fernan Aguero" To: "Anton Berezin" In-Reply-To: <20090103135718.GC41513@heechee.tobez.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <520894aa0901020758o79bb1233teee539f6d599d10a@mail.gmail.com> <20090102164439.GF40649@heechee.tobez.org> <520894aa0901030433i7a103ba7i4236ae17b83c6eae@mail.gmail.com> <20090103135718.GC41513@heechee.tobez.org> Cc: freebsd-perl@freebsd.org Subject: Re: Storable byteorder incompatibilities in different FreeBSD installations X-BeenThere: freebsd-perl@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: maintainer of a number of perl-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jan 2009 14:22:03 -0000 On Sat, Jan 3, 2009 at 11:57 AM, Anton Berezin wrote: > Fernan, > > On Sat, Jan 03, 2009 at 10:33:22AM -0200, Fernan Aguero wrote: > >> Apart from the byteorder issue, the other issue which I raised in my >> post, was also the incompatibility in 'longsize' between two perl >> instances built from ports, in different boxes. >> >> In this case, the perl built in a FBSD-6.3, i386 box has a longsize of >> 4, whereas the same perl built in FBSD-7.1, amd64 has a longsize of 8. >> In both cases they were compiled with PERL_64BITINT (the default for >> the port), and in both cases the byteorder is the same! >> >> How can I build a perl with a byteorder of 12345678, and a longsize of >> 4, in an amd64 platform? Is that possible at all? > > Not really. Amd64 is a 64-bit platform, meaning sizeof(long) = 8. I am > pretty positive that the same is true for x86-64 Linux. That's more or less > what "12345678" indicates. > > At any rate, this is the application design issue. The practical solution > in your case would be to take an offline copy of the DB, and to go through > every record and convert it into a KNOWN to you binary format. I have not > done the relevant testing, so I do not even know whether nstore() will do > what you want - after all, it only handles the byteorder issues, not integer > size issues. I am doing these tests as we speak, and will convert all my data using nstore/nfreeze. Thanks. > Cheers, > \Anton. -- fernan