From owner-svn-src-user@FreeBSD.ORG Sun Sep 28 00:20:12 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 91E91358;
Sun, 28 Sep 2014 00:20:12 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 7D47061B;
Sun, 28 Sep 2014 00:20:12 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s8S0KCJc069055;
Sun, 28 Sep 2014 00:20:12 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s8S0K8lH069033;
Sun, 28 Sep 2014 00:20:08 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201409280020.s8S0K8lH069033@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Sun, 28 Sep 2014 00:20:08 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272231 - in user/marcel/mkimg: . tests
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 28 Sep 2014 00:20:12 -0000
Author: marcel
Date: Sun Sep 28 00:20:08 2014
New Revision: 272231
URL: http://svnweb.freebsd.org/changeset/base/272231
Log:
Sync with ^head/usr.bin/mkimg@272217
Added:
user/marcel/mkimg/qcow.c
- copied unchanged from r272230, head/usr.bin/mkimg/qcow.c
user/marcel/mkimg/tests/
- copied from r272230, head/usr.bin/mkimg/tests/
Modified:
user/marcel/mkimg/Makefile
user/marcel/mkimg/apm.c
user/marcel/mkimg/bsd.c
user/marcel/mkimg/ebr.c
user/marcel/mkimg/gpt.c
user/marcel/mkimg/image.c
user/marcel/mkimg/image.h
user/marcel/mkimg/mbr.c
user/marcel/mkimg/mkimg.1
user/marcel/mkimg/mkimg.c
user/marcel/mkimg/mkimg.h
user/marcel/mkimg/pc98.c
user/marcel/mkimg/scheme.c
user/marcel/mkimg/scheme.h
user/marcel/mkimg/vhd.c
user/marcel/mkimg/vmdk.c
user/marcel/mkimg/vtoc8.c
Directory Properties:
user/marcel/mkimg/ (props changed)
Modified: user/marcel/mkimg/Makefile
==============================================================================
--- user/marcel/mkimg/Makefile Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/Makefile Sun Sep 28 00:20:08 2014 (r272231)
@@ -1,13 +1,18 @@
# $FreeBSD$
+.include <src.opts.mk>
+
PROG= mkimg
SRCS= format.c image.c mkimg.c scheme.c
MAN= mkimg.1
+MKIMG_VERSION=20140927
+CFLAGS+=-DMKIMG_VERSION=${MKIMG_VERSION}
CFLAGS+=-DSPARSE_WRITE
# List of formats to support
SRCS+= \
+ qcow.c \
raw.c \
vhd.c \
vmdk.c
@@ -29,4 +34,8 @@ LDADD= -lutil
WARNS?= 6
+.if ${MK_TESTS} != "no"
+SUBDIR+= tests
+.endif
+
.include <bsd.prog.mk>
Modified: user/marcel/mkimg/apm.c
==============================================================================
--- user/marcel/mkimg/apm.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/apm.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -39,6 +39,9 @@ __FBSDID("$FreeBSD$");
#include "mkimg.h"
#include "scheme.h"
+#ifndef APM_ENT_TYPE_APPLE_BOOT
+#define APM_ENT_TYPE_APPLE_BOOT "Apple_Bootstrap"
+#endif
#ifndef APM_ENT_TYPE_FREEBSD_NANDFS
#define APM_ENT_TYPE_FREEBSD_NANDFS "FreeBSD-nandfs"
#endif
@@ -54,13 +57,12 @@ static struct mkimg_alias apm_aliases[]
{ ALIAS_NONE, 0 }
};
-static u_int
-apm_metadata(u_int where)
+static lba_t
+apm_metadata(u_int where, lba_t blk)
{
- u_int secs;
- secs = (where == SCHEME_META_IMG_START) ? nparts + 2 : 0;
- return (secs);
+ blk += (where == SCHEME_META_IMG_START) ? nparts + 2 : 0;
+ return (round_block(blk));
}
static int
Modified: user/marcel/mkimg/bsd.c
==============================================================================
--- user/marcel/mkimg/bsd.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/bsd.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -52,13 +52,17 @@ static struct mkimg_alias bsd_aliases[]
{ ALIAS_NONE, 0 }
};
-static u_int
-bsd_metadata(u_int where)
+static lba_t
+bsd_metadata(u_int where, lba_t blk)
{
- u_int secs;
- secs = BBSIZE / secsz;
- return ((where == SCHEME_META_IMG_START) ? secs : 0);
+ if (where == SCHEME_META_IMG_START)
+ blk += BBSIZE / secsz;
+ else if (where == SCHEME_META_IMG_END)
+ blk = round_cylinder(blk);
+ else
+ blk = round_block(blk);
+ return (blk);
}
static int
@@ -68,7 +72,7 @@ bsd_write(lba_t imgsz, void *bootcode)
struct disklabel *d;
struct partition *dp;
struct part *part;
- int error, n;
+ int bsdparts, error, n;
uint16_t checksum;
buf = malloc(BBSIZE);
@@ -76,16 +80,13 @@ bsd_write(lba_t imgsz, void *bootcode)
return (ENOMEM);
if (bootcode != NULL) {
memcpy(buf, bootcode, BBSIZE);
- memset(buf + secsz, 0, secsz);
+ memset(buf + secsz, 0, sizeof(struct disklabel));
} else
memset(buf, 0, BBSIZE);
- imgsz = (lba_t)ncyls * nheads * nsecs;
- error = image_set_size(imgsz);
- if (error) {
- free(buf);
- return (error);
- }
+ bsdparts = nparts + 1; /* Account for c partition */
+ if (bsdparts < MAXPARTITIONS)
+ bsdparts = MAXPARTITIONS;
d = (void *)(buf + secsz);
le32enc(&d->d_magic, DISKMAGIC);
@@ -97,7 +98,7 @@ bsd_write(lba_t imgsz, void *bootcode)
le32enc(&d->d_secperunit, imgsz);
le16enc(&d->d_rpm, 3600);
le32enc(&d->d_magic2, DISKMAGIC);
- le16enc(&d->d_npartitions, (8 > nparts + 1) ? 8 : nparts + 1);
+ le16enc(&d->d_npartitions, bsdparts);
le32enc(&d->d_bbsize, BBSIZE);
dp = &d->d_partitions[RAW_PART];
@@ -107,12 +108,15 @@ bsd_write(lba_t imgsz, void *bootcode)
dp = &d->d_partitions[n];
le32enc(&dp->p_size, part->size);
le32enc(&dp->p_offset, part->block);
+ le32enc(&dp->p_fsize, 0);
dp->p_fstype = ALIAS_TYPE2INT(part->type);
+ dp->p_frag = 0;
+ le16enc(&dp->p_cpg, 0);
}
- dp = &d->d_partitions[nparts + 1];
+ dp = &d->d_partitions[bsdparts];
checksum = 0;
- for (p = buf; p < (u_char *)dp; p += 2)
+ for (p = (void *)d; p < (u_char *)dp; p += 2)
checksum ^= le16dec(p);
le16enc(&d->d_checksum, checksum);
Modified: user/marcel/mkimg/ebr.c
==============================================================================
--- user/marcel/mkimg/ebr.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/ebr.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -49,13 +49,12 @@ static struct mkimg_alias ebr_aliases[]
{ ALIAS_NONE, 0 }
};
-static u_int
-ebr_metadata(u_int where)
+static lba_t
+ebr_metadata(u_int where, lba_t blk)
{
- u_int secs;
- secs = (where == SCHEME_META_PART_BEFORE) ? nsecs : 0;
- return (secs);
+ blk += (where == SCHEME_META_PART_BEFORE) ? 1 : 0;
+ return (round_track(blk));
}
static void
Modified: user/marcel/mkimg/gpt.c
==============================================================================
--- user/marcel/mkimg/gpt.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/gpt.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -153,17 +153,15 @@ gpt_tblsz(void)
return ((nparts + ents - 1) / ents);
}
-static u_int
-gpt_metadata(u_int where)
+static lba_t
+gpt_metadata(u_int where, lba_t blk)
{
- u_int secs;
- if (where != SCHEME_META_IMG_START && where != SCHEME_META_IMG_END)
- return (0);
-
- secs = gpt_tblsz();
- secs += (where == SCHEME_META_IMG_START) ? 2 : 1;
- return (secs);
+ if (where == SCHEME_META_IMG_START || where == SCHEME_META_IMG_END) {
+ blk += gpt_tblsz();
+ blk += (where == SCHEME_META_IMG_START) ? 2 : 1;
+ }
+ return (round_block(blk));
}
static int
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/image.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -94,12 +94,19 @@ image_copyin(lba_t blk, int fd, uint64_t
int
image_copyout(int fd)
{
- off_t ofs;
int error;
error = image_copyout_region(fd, 0, image_size);
- if (error)
- return (error);
+ if (!error)
+ error = image_copyout_done(fd);
+ return (error);
+}
+
+int
+image_copyout_done(int fd)
+{
+ off_t ofs;
+ int error;
ofs = lseek(fd, 0L, SEEK_CUR);
if (ofs == -1)
@@ -148,6 +155,33 @@ image_copyout_region(int fd, lba_t blk,
return (error);
}
+int
+image_data(lba_t blk, lba_t size)
+{
+ char *buffer, *p;
+
+ blk *= secsz;
+ if (lseek(image_fd, blk, SEEK_SET) != blk)
+ return (1);
+
+ size *= secsz;
+ buffer = malloc(size);
+ if (buffer == NULL)
+ return (1);
+
+ if (read(image_fd, buffer, size) != (ssize_t)size) {
+ free(buffer);
+ return (1);
+ }
+
+ p = buffer;
+ while (size > 0 && *p == '\0')
+ size--, p++;
+
+ free(buffer);
+ return ((size == 0) ? 0 : 1);
+}
+
lba_t
image_get_size(void)
{
Modified: user/marcel/mkimg/image.h
==============================================================================
--- user/marcel/mkimg/image.h Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/image.h Sun Sep 28 00:20:08 2014 (r272231)
@@ -33,7 +33,9 @@ typedef int64_t lba_t;
int image_copyin(lba_t blk, int fd, uint64_t *sizep);
int image_copyout(int fd);
+int image_copyout_done(int fd);
int image_copyout_region(int fd, lba_t blk, lba_t size);
+int image_data(lba_t blk, lba_t size);
lba_t image_get_size(void);
int image_init(void);
int image_set_size(lba_t blk);
Modified: user/marcel/mkimg/mbr.c
==============================================================================
--- user/marcel/mkimg/mbr.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/mbr.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -50,13 +50,12 @@ static struct mkimg_alias mbr_aliases[]
{ ALIAS_NONE, 0 } /* Keep last! */
};
-static u_int
-mbr_metadata(u_int where)
+static lba_t
+mbr_metadata(u_int where, lba_t blk)
{
- u_int secs;
- secs = (where == SCHEME_META_IMG_START) ? nsecs : 0;
- return (secs);
+ blk += (where == SCHEME_META_IMG_START) ? 1 : 0;
+ return (round_track(blk));
}
static void
Modified: user/marcel/mkimg/mkimg.1
==============================================================================
--- user/marcel/mkimg/mkimg.1 Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/mkimg.1 Sun Sep 28 00:20:08 2014 (r272231)
@@ -24,12 +24,12 @@
.\"
.\" $FreeBSD$
.\"
-.Dd July 2, 2014
+.Dd September 27, 2014
.Dt MKIMG 1
.Os
.Sh NAME
.Nm mkimg
-.Nd "utility to make a disk image"
+.Nd "utility to make disk images"
.Sh SYNOPSIS
.Nm
.Op Fl H Ar heads
@@ -40,9 +40,12 @@
.Op Fl f Ar format
.Op Fl o Ar outfile
.Op Fl v
+.Op Fl y
.Fl s Ar scheme
.Fl p Ar partition
.Op Fl p Ar partition ...
+.Nm
+.Ar --formats | --schemes | --version
.Sh DESCRIPTION
The
.Nm
@@ -111,10 +114,42 @@ option increases the level of output tha
.Nm
utility prints.
.Pp
-For a complete list of supported partitioning schemes or supported output
-format, or for a detailed description of how to specify partitions, run the
+The
+.Op Fl y
+option is used for testing purposes only and is not to be used in production.
+When present, the
+.Nm
+utility will generate predictable values for Universally Unique Identifiers
+(UUIDs) and time stamps so that consecutive runs of the
+.Nm
+utility will create images that are identical.
+.Pp
+A set of long options exist to query about the
+.Nm
+utilty itself.
+Options in this set should be given by themselves because the
+.Nm
+utility exits immediately after providing the requested information.
+The version of the
+.Nm
+utility is printed when the
+.Ar --version
+option is given.
+The list of supported output formats is printed when the
+.Ar --formats
+option is given and the list of supported partitioning schemes is printed
+when the
+.Ar --schemes
+option is given.
+Both the format and scheme lists a space-separated lists for easy handling
+in scripts.
+.Pp
+For a more descriptive list of supported partitioning schemes or supported
+output format, or for a detailed description of how to specify partitions,
+run the
.Nm
utility without any arguments.
+This will print a usage message with all the necessary details.
.Sh ENVIRONMENT
.Bl -tag -width "TMPDIR" -compact
.It Ev TMPDIR
@@ -160,6 +195,25 @@ utility as follows:
.Dl % mkimg -s mbr -b /boot/mbr -p freebsd:-'mkimg -s bsd -b /boot/boot \
-p freebsd-ufs:=root-file-system.ufs -p freebsd-swap::1G' -o mbr-bsd.img
.Pp
+To accomodate the need to have partitions named or numbered in a certain
+way, the
+.Nm
+utility allows for the specification of empty partitions.
+For example, to create an image that is compatible with partition layouts
+found in
+.Pa /etc/disktab ,
+the 'd' partition often needs to be skipped.
+This is accomplished by inserting an unused partition after the first 2
+partition specifications.
+It is worth noting at this time that the BSD scheme will automatically
+skip the 'c' partition by virtue of it referring to the entire disk.
+To create an image that is compatible with the qp120at disk, use the
+.Nm
+utility as follows:
+.Dl % mkimg -s bsd -b /boot/boot -p freebsd-ufs:=root-file-system.ufs \
+-p freebsd-swap::20M -p- -p- -p- -p- -p freebsd-ufs:=usr-file-system.ufs \
+-o bsd.img
+.Pp
For partitioning schemes that feature partition labels, the
.Nm
utility supports assigning labels to the partitions specified.
Modified: user/marcel/mkimg/mkimg.c
==============================================================================
--- user/marcel/mkimg/mkimg.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/mkimg.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -35,6 +35,7 @@ __FBSDID("$FreeBSD$");
#include <errno.h>
#include <err.h>
#include <fcntl.h>
+#include <getopt.h>
#include <libutil.h>
#include <limits.h>
#include <stdio.h>
@@ -48,6 +49,17 @@ __FBSDID("$FreeBSD$");
#include "mkimg.h"
#include "scheme.h"
+#define LONGOPT_FORMATS 0x01000001
+#define LONGOPT_SCHEMES 0x01000002
+#define LONGOPT_VERSION 0x01000003
+
+static struct option longopts[] = {
+ { "formats", no_argument, NULL, LONGOPT_FORMATS },
+ { "schemes", no_argument, NULL, LONGOPT_SCHEMES },
+ { "version", no_argument, NULL, LONGOPT_VERSION },
+ { NULL, 0, NULL, 0 }
+};
+
struct partlisthead partlist = STAILQ_HEAD_INITIALIZER(partlist);
u_int nparts = 0;
@@ -61,44 +73,103 @@ u_int secsz = 512;
u_int blksz = 0;
static void
-usage(const char *why)
+print_formats(int usage)
{
struct mkimg_format *f, **f_iter;
+ const char *sep;
+
+ if (usage) {
+ fprintf(stderr, " formats:\n");
+ SET_FOREACH(f_iter, formats) {
+ f = *f_iter;
+ fprintf(stderr, "\t%s\t- %s\n", f->name,
+ f->description);
+ }
+ } else {
+ sep = "";
+ SET_FOREACH(f_iter, formats) {
+ f = *f_iter;
+ printf("%s%s", sep, f->name);
+ sep = " ";
+ }
+ putchar('\n');
+ }
+}
+
+static void
+print_schemes(int usage)
+{
struct mkimg_scheme *s, **s_iter;
+ const char *sep;
+
+ if (usage) {
+ fprintf(stderr, " schemes:\n");
+ SET_FOREACH(s_iter, schemes) {
+ s = *s_iter;
+ fprintf(stderr, "\t%s\t- %s\n", s->name,
+ s->description);
+ }
+ } else {
+ sep = "";
+ SET_FOREACH(s_iter, schemes) {
+ s = *s_iter;
+ printf("%s%s", sep, s->name);
+ sep = " ";
+ }
+ putchar('\n');
+ }
+}
+
+static void
+print_version(void)
+{
+ u_int width;
+
+#ifdef __LP64__
+ width = 64;
+#else
+ width = 32;
+#endif
+ printf("mkimg %u (%u-bit)\n", MKIMG_VERSION, width);
+}
+
+static void
+usage(const char *why)
+{
warnx("error: %s", why);
- fprintf(stderr, "\nusage: %s <options>\n", getprogname());
+ fputc('\n', stderr);
+ fprintf(stderr, "usage: %s <options>\n", getprogname());
fprintf(stderr, " options:\n");
+ fprintf(stderr, "\t--formats\t- list image formats\n");
+ fprintf(stderr, "\t--schemes\t- list partition schemes\n");
+ fprintf(stderr, "\t--version\t- show version information\n");
+ fputc('\n', stderr);
fprintf(stderr, "\t-b <file>\t- file containing boot code\n");
fprintf(stderr, "\t-f <format>\n");
fprintf(stderr, "\t-o <file>\t- file to write image into\n");
fprintf(stderr, "\t-p <partition>\n");
fprintf(stderr, "\t-s <scheme>\n");
+ fprintf(stderr, "\t-v\t\t- increase verbosity\n");
+ fprintf(stderr, "\t-y\t\t- [developers] enable unit test\n");
fprintf(stderr, "\t-H <num>\t- number of heads to simulate\n");
fprintf(stderr, "\t-P <num>\t- physical sector size\n");
fprintf(stderr, "\t-S <num>\t- logical sector size\n");
fprintf(stderr, "\t-T <num>\t- number of tracks to simulate\n");
-
- fprintf(stderr, "\n formats:\n");
- SET_FOREACH(f_iter, formats) {
- f = *f_iter;
- fprintf(stderr, "\t%s\t- %s\n", f->name, f->description);
- }
-
- fprintf(stderr, "\n schemes:\n");
- SET_FOREACH(s_iter, schemes) {
- s = *s_iter;
- fprintf(stderr, "\t%s\t- %s\n", s->name, s->description);
- }
-
- fprintf(stderr, "\n partition specification:\n");
+ fputc('\n', stderr);
+ print_formats(1);
+ fputc('\n', stderr);
+ print_schemes(1);
+ fputc('\n', stderr);
+ fprintf(stderr, " partition specification:\n");
fprintf(stderr, "\t<t>[/<l>]::<size>\t- empty partition of given "
"size\n");
fprintf(stderr, "\t<t>[/<l>]:=<file>\t- partition content and size "
"are determined\n\t\t\t\t by the named file\n");
fprintf(stderr, "\t<t>[/<l>]:-<cmd>\t- partition content and size "
"are taken from\n\t\t\t\t the output of the command to run\n");
+ fprintf(stderr, "\t-\t\t\t- unused partition entry\n");
fprintf(stderr, "\t where:\n");
fprintf(stderr, "\t\t<t>\t- scheme neutral partition type\n");
fprintf(stderr, "\t\t<l>\t- optional scheme-dependent partition "
@@ -138,6 +209,9 @@ pwr_of_two(u_int nr)
* '-' contents holds a command to run; the output of
* which is the contents of the partition.
* contents the specification of a partition's contents
+ *
+ * A specification that is a single dash indicates an unused partition
+ * entry.
*/
static int
parse_part(const char *spec)
@@ -147,6 +221,11 @@ parse_part(const char *spec)
size_t len;
int error;
+ if (strcmp(spec, "-") == 0) {
+ nparts++;
+ return (0);
+ }
+
part = calloc(1, sizeof(struct part));
if (part == NULL)
return (ENOMEM);
@@ -355,7 +434,8 @@ main(int argc, char *argv[])
bcfd = -1;
outfd = 1; /* Write to stdout by default */
- while ((c = getopt(argc, argv, "b:f:o:p:s:vyH:P:S:T:")) != -1) {
+ while ((c = getopt_long(argc, argv, "b:f:o:p:s:vyH:P:S:T:",
+ longopts, NULL)) != -1) {
switch (c) {
case 'b': /* BOOT CODE */
if (bcfd != -1)
@@ -421,6 +501,18 @@ main(int argc, char *argv[])
if (error)
errc(EX_DATAERR, error, "track size");
break;
+ case LONGOPT_FORMATS:
+ print_formats(0);
+ exit(EX_OK);
+ /*NOTREACHED*/
+ case LONGOPT_SCHEMES:
+ print_schemes(0);
+ exit(EX_OK);
+ /*NOTREACHED*/
+ case LONGOPT_VERSION:
+ print_version();
+ exit(EX_OK);
+ /*NOTREACHED*/
default:
usage("unknown option");
}
Modified: user/marcel/mkimg/mkimg.h
==============================================================================
--- user/marcel/mkimg/mkimg.h Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/mkimg.h Sun Sep 28 00:20:08 2014 (r272231)
@@ -66,6 +66,21 @@ round_block(lba_t n)
return ((n + b - 1) & ~(b - 1));
}
+static inline lba_t
+round_cylinder(lba_t n)
+{
+ u_int cyl = nsecs * nheads;
+ u_int r = n % cyl;
+ return ((r == 0) ? n : n + cyl - r);
+}
+
+static inline lba_t
+round_track(lba_t n)
+{
+ u_int r = n % nsecs;
+ return ((r == 0) ? n : n + nsecs - r);
+}
+
#if !defined(SPARSE_WRITE)
#define sparse_write write
#else
Modified: user/marcel/mkimg/pc98.c
==============================================================================
--- user/marcel/mkimg/pc98.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/pc98.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -59,13 +59,12 @@ static struct mkimg_alias pc98_aliases[]
{ ALIAS_NONE, 0 }
};
-static u_int
-pc98_metadata(u_int where)
+static lba_t
+pc98_metadata(u_int where, lba_t blk)
{
- u_int secs;
-
- secs = PC98_BOOTCODESZ / secsz;
- return ((where == SCHEME_META_IMG_START) ? secs : 0);
+ if (where == SCHEME_META_IMG_START)
+ blk += PC98_BOOTCODESZ / secsz;
+ return (round_track(blk));
}
static void
Copied: user/marcel/mkimg/qcow.c (from r272230, head/usr.bin/mkimg/qcow.c)
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ user/marcel/mkimg/qcow.c Sun Sep 28 00:20:08 2014 (r272231, copy of r272230, head/usr.bin/mkimg/qcow.c)
@@ -0,0 +1,369 @@
+/*-
+ * Copyright (c) 2014 Marcel Moolenaar
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <sys/types.h>
+#include <sys/endian.h>
+#include <sys/errno.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "image.h"
+#include "format.h"
+#include "mkimg.h"
+
+/* Default cluster sizes. */
+#define QCOW1_CLSTR_LOG2SZ 12 /* 4KB */
+#define QCOW2_CLSTR_LOG2SZ 16 /* 64KB */
+
+/* Flag bits in cluster offsets */
+#define QCOW_CLSTR_COMPRESSED (1ULL << 62)
+#define QCOW_CLSTR_COPIED (1ULL << 63)
+
+struct qcow_header {
+ uint32_t magic;
+#define QCOW_MAGIC 0x514649fb
+ uint32_t version;
+#define QCOW_VERSION_1 1
+#define QCOW_VERSION_2 2
+ uint64_t path_offset;
+ uint32_t path_length;
+ uint32_t clstr_log2sz; /* v2 only */
+ uint64_t disk_size;
+ union {
+ struct {
+ uint8_t clstr_log2sz;
+ uint8_t l2_log2sz;
+ uint16_t _pad;
+ uint32_t encryption;
+ uint64_t l1_offset;
+ } v1;
+ struct {
+ uint32_t encryption;
+ uint32_t l1_entries;
+ uint64_t l1_offset;
+ uint64_t refcnt_offset;
+ uint32_t refcnt_entries;
+ uint32_t snapshot_count;
+ uint64_t snapshot_offset;
+ } v2;
+ } u;
+};
+
+static u_int clstr_log2sz;
+
+static uint64_t
+round_clstr(uint64_t ofs)
+{
+ uint64_t clstrsz;
+
+ clstrsz = 1UL << clstr_log2sz;
+ return ((ofs + clstrsz - 1) & ~(clstrsz - 1));
+}
+
+static int
+qcow_resize(lba_t imgsz, u_int version)
+{
+ uint64_t imagesz;
+
+ switch (version) {
+ case QCOW_VERSION_1:
+ clstr_log2sz = QCOW1_CLSTR_LOG2SZ;
+ break;
+ case QCOW_VERSION_2:
+ clstr_log2sz = QCOW2_CLSTR_LOG2SZ;
+ break;
+ default:
+ return (EDOOFUS);
+ }
+
+ imagesz = round_clstr(imgsz * secsz);
+
+ if (verbose)
+ fprintf(stderr, "QCOW: image size = %ju, cluster size = %u\n",
+ (uintmax_t)imagesz, (u_int)(1U << clstr_log2sz));
+
+ return (image_set_size(imagesz / secsz));
+}
+
+static int
+qcow1_resize(lba_t imgsz)
+{
+
+ return (qcow_resize(imgsz, QCOW_VERSION_1));
+}
+
+static int
+qcow2_resize(lba_t imgsz)
+{
+
+ return (qcow_resize(imgsz, QCOW_VERSION_2));
+}
+
+static int
+qcow_write(int fd, u_int version)
+{
+ struct qcow_header *hdr;
+ uint64_t *l1tbl, *l2tbl, *rctbl;
+ uint16_t *rcblk;
+ uint64_t clstr_imgsz, clstr_l2tbls, clstr_l1tblsz;
+ uint64_t clstr_rcblks, clstr_rctblsz;
+ uint64_t n, imagesz, nclstrs, ofs, ofsflags;
+ lba_t blk, blkofs, blk_imgsz;
+ u_int l1clno, l2clno, rcclno;
+ u_int blk_clstrsz;
+ u_int clstrsz, l1idx, l2idx;
+ int error;
+
+ if (clstr_log2sz == 0)
+ return (EDOOFUS);
+
+ clstrsz = 1U << clstr_log2sz;
+ blk_clstrsz = clstrsz / secsz;
+ blk_imgsz = image_get_size();
+ imagesz = blk_imgsz * secsz;
+ clstr_imgsz = imagesz >> clstr_log2sz;
+ clstr_l2tbls = round_clstr(clstr_imgsz * 8) >> clstr_log2sz;
+ clstr_l1tblsz = round_clstr(clstr_l2tbls * 8) >> clstr_log2sz;
+ nclstrs = clstr_imgsz + clstr_l2tbls + clstr_l1tblsz + 1;
+ clstr_rcblks = clstr_rctblsz = 0;
+ do {
+ n = clstr_rcblks + clstr_rctblsz;
+ clstr_rcblks = round_clstr((nclstrs + n) * 2) >> clstr_log2sz;
+ clstr_rctblsz = round_clstr(clstr_rcblks * 8) >> clstr_log2sz;
+ } while (n < (clstr_rcblks + clstr_rctblsz));
+
+ /*
+ * We got all the sizes in clusters. Start the layout.
+ * 0 - header
+ * 1 - L1 table
+ * 2 - RC table (v2 only)
+ * 3 - L2 tables
+ * 4 - RC block (v2 only)
+ * 5 - data
+ */
+
+ l1clno = 1;
+ rcclno = 0;
+ rctbl = l2tbl = l1tbl = NULL;
+ rcblk = NULL;
+
+ hdr = calloc(1, clstrsz);
+ if (hdr == NULL)
+ return (errno);
+
+ be32enc(&hdr->magic, QCOW_MAGIC);
+ be32enc(&hdr->version, version);
+ be64enc(&hdr->disk_size, imagesz);
+ switch (version) {
+ case QCOW_VERSION_1:
+ ofsflags = 0;
+ l2clno = l1clno + clstr_l1tblsz;
+ hdr->u.v1.clstr_log2sz = clstr_log2sz;
+ hdr->u.v1.l2_log2sz = clstr_log2sz - 3;
+ be64enc(&hdr->u.v1.l1_offset, clstrsz * l1clno);
+ break;
+ case QCOW_VERSION_2:
+ ofsflags = QCOW_CLSTR_COPIED;
+ rcclno = l1clno + clstr_l1tblsz;
+ l2clno = rcclno + clstr_rctblsz;
+ be32enc(&hdr->clstr_log2sz, clstr_log2sz);
+ be32enc(&hdr->u.v2.l1_entries, clstr_l2tbls);
+ be64enc(&hdr->u.v2.l1_offset, clstrsz * l1clno);
+ be64enc(&hdr->u.v2.refcnt_offset, clstrsz * rcclno);
+ be32enc(&hdr->u.v2.refcnt_entries, clstr_rcblks);
+ break;
+ default:
+ return (EDOOFUS);
+ }
+
+ if (sparse_write(fd, hdr, clstrsz) < 0) {
+ error = errno;
+ goto out;
+ }
+
+ free(hdr);
+ hdr = NULL;
+
+ ofs = clstrsz * l2clno;
+ nclstrs = 1 + clstr_l1tblsz + clstr_rctblsz;
+
+ l1tbl = calloc(1, clstrsz * clstr_l1tblsz);
+ if (l1tbl == NULL) {
+ error = ENOMEM;
+ goto out;
+ }
+
+ for (n = 0; n < clstr_imgsz; n++) {
+ blk = n * blk_clstrsz;
+ if (image_data(blk, blk_clstrsz)) {
+ nclstrs++;
+ l1idx = n >> (clstr_log2sz - 3);
+ if (l1tbl[l1idx] == 0) {
+ be64enc(l1tbl + l1idx, ofs + ofsflags);
+ ofs += clstrsz;
+ nclstrs++;
+ }
+ }
+ }
+
+ if (sparse_write(fd, l1tbl, clstrsz * clstr_l1tblsz) < 0) {
+ error = errno;
+ goto out;
+ }
+
+ clstr_rcblks = 0;
+ do {
+ n = clstr_rcblks;
+ clstr_rcblks = round_clstr((nclstrs + n) * 2) >> clstr_log2sz;
+ } while (n < clstr_rcblks);
+
+ if (rcclno > 0) {
+ rctbl = calloc(1, clstrsz * clstr_rctblsz);
+ if (rctbl == NULL) {
+ error = ENOMEM;
+ goto out;
+ }
+ for (n = 0; n < clstr_rcblks; n++) {
+ be64enc(rctbl + n, ofs);
+ ofs += clstrsz;
+ nclstrs++;
+ }
+ if (sparse_write(fd, rctbl, clstrsz * clstr_rctblsz) < 0) {
+ error = errno;
+ goto out;
+ }
+ free(rctbl);
+ rctbl = NULL;
+ }
+
+ l2tbl = malloc(clstrsz);
+ if (l2tbl == NULL) {
+ error = ENOMEM;
+ goto out;
+ }
+
+ for (l1idx = 0; l1idx < clstr_l2tbls; l1idx++) {
+ if (l1tbl[l1idx] == 0)
+ continue;
+ memset(l2tbl, 0, clstrsz);
+ blkofs = (lba_t)l1idx * blk_clstrsz * (clstrsz >> 3);
+ for (l2idx = 0; l2idx < (clstrsz >> 3); l2idx++) {
+ blk = blkofs + (lba_t)l2idx * blk_clstrsz;
+ if (blk >= blk_imgsz)
+ break;
+ if (image_data(blk, blk_clstrsz)) {
+ be64enc(l2tbl + l2idx, ofs + ofsflags);
+ ofs += clstrsz;
+ }
+ }
+ if (sparse_write(fd, l2tbl, clstrsz) < 0) {
+ error = errno;
+ goto out;
+ }
+ }
+
+ free(l2tbl);
+ l2tbl = NULL;
+ free(l1tbl);
+ l1tbl = NULL;
+
+ if (rcclno > 0) {
+ rcblk = calloc(1, clstrsz * clstr_rcblks);
+ if (rcblk == NULL) {
+ error = ENOMEM;
+ goto out;
+ }
+ for (n = 0; n < nclstrs; n++)
+ be16enc(rcblk + n, 1);
+ if (sparse_write(fd, rcblk, clstrsz * clstr_rcblks) < 0) {
+ error = errno;
+ goto out;
+ }
+ free(rcblk);
+ rcblk = NULL;
+ }
+
+ error = 0;
+ for (n = 0; n < clstr_imgsz; n++) {
+ blk = n * blk_clstrsz;
+ if (image_data(blk, blk_clstrsz)) {
+ error = image_copyout_region(fd, blk, blk_clstrsz);
+ if (error)
+ break;
+ }
+ }
+ if (!error)
+ error = image_copyout_done(fd);
+
+ out:
+ if (rcblk != NULL)
+ free(rcblk);
+ if (l2tbl != NULL)
+ free(l2tbl);
+ if (rctbl != NULL)
+ free(rctbl);
+ if (l1tbl != NULL)
+ free(l1tbl);
+ if (hdr != NULL)
+ free(hdr);
+ return (error);
+}
+
+static int
+qcow1_write(int fd)
+{
+
+ return (qcow_write(fd, QCOW_VERSION_1));
+}
+
+static int
+qcow2_write(int fd)
+{
+
+ return (qcow_write(fd, QCOW_VERSION_2));
+}
+
+static struct mkimg_format qcow1_format = {
+ .name = "qcow",
+ .description = "QEMU Copy-On-Write, version 1",
+ .resize = qcow1_resize,
+ .write = qcow1_write,
+};
+FORMAT_DEFINE(qcow1_format);
+
+static struct mkimg_format qcow2_format = {
+ .name = "qcow2",
+ .description = "QEMU Copy-On-Write, version 2",
+ .resize = qcow2_resize,
+ .write = qcow2_write,
+};
+FORMAT_DEFINE(qcow2_format);
Modified: user/marcel/mkimg/scheme.c
==============================================================================
--- user/marcel/mkimg/scheme.c Sat Sep 27 23:57:21 2014 (r272230)
+++ user/marcel/mkimg/scheme.c Sun Sep 28 00:20:08 2014 (r272231)
@@ -171,10 +171,8 @@ scheme_max_secsz(void)
*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
From owner-svn-src-user@FreeBSD.ORG Sun Sep 28 00:24:02 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 661BF485;
Sun, 28 Sep 2014 00:24:02 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 537756CB;
Sun, 28 Sep 2014 00:24:02 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s8S0O2jL072884;
Sun, 28 Sep 2014 00:24:02 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s8S0O26B072883;
Sun, 28 Sep 2014 00:24:02 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201409280024.s8S0O26B072883@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Sun, 28 Sep 2014 00:24:02 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272232 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 28 Sep 2014 00:24:02 -0000
Author: marcel
Date: Sun Sep 28 00:24:01 2014
New Revision: 272232
URL: http://svnweb.freebsd.org/changeset/base/272232
Log:
Work around broken make infrastructure. Sigh...
Modified:
user/marcel/mkimg/Makefile
Modified: user/marcel/mkimg/Makefile
==============================================================================
--- user/marcel/mkimg/Makefile Sun Sep 28 00:20:08 2014 (r272231)
+++ user/marcel/mkimg/Makefile Sun Sep 28 00:24:01 2014 (r272232)
@@ -1,6 +1,6 @@
# $FreeBSD$
-.include <src.opts.mk>
+!.include <src.opts.mk>
PROG= mkimg
SRCS= format.c image.c mkimg.c scheme.c
@@ -34,7 +34,7 @@ LDADD= -lutil
WARNS?= 6
-.if ${MK_TESTS} != "no"
+.if ${MK_TESTS:Uno} != "no"
SUBDIR+= tests
.endif
From owner-svn-src-user@FreeBSD.ORG Sun Sep 28 00:43:05 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 2527869C;
Sun, 28 Sep 2014 00:43:05 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 116D48AD;
Sun, 28 Sep 2014 00:43:05 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s8S0h4Ed081855;
Sun, 28 Sep 2014 00:43:04 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s8S0h4Pu081854;
Sun, 28 Sep 2014 00:43:04 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201409280043.s8S0h4Pu081854@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Sun, 28 Sep 2014 00:43:04 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272233 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 28 Sep 2014 00:43:05 -0000
Author: marcel
Date: Sun Sep 28 00:43:04 2014
New Revision: 272233
URL: http://svnweb.freebsd.org/changeset/base/272233
Log:
Start performance optimizations:
Output formats typically need to know whether a sequence of blocks has data
or not. They use this to determine whether to allocate disk space for them
or not. The image_data() function provides that functionality, but is
implemented by reading the amount of blocks from the temporary file and
comparing that with zeroes. The QCOW format needs to go over the image 3
times and every time we read from the temporary file.
We can speed this up by building a "chunk" list in memory while we read the
partition data. Each chunk is a sequence of blocks that is either defined
as a gap (i.e. all zeroes) or defined as containing data. For each chunk we
keep track of the file and the offset in that file where the chunk's data
comes from. This allows us handle regular files and in particular sparse
files more optimally.
For sparse files we can trivially build a chunk for each of the holes in
the parse file by using SEEK_HOLE and SEEK_DATA. The data regions still
need to be read to handle zeroe blocks for block sizes smaller that the
underlying file system's block size. However, we don't have to copy the
data into a temporary file anymore.
For anything but regular files, we still use the temporary file. We call it
a swap file now. With that all data can be mapped and unmapped as we need
to access it.
This commit implements the creation of the chunk list and the swap file
usage for non-regular files (i.e. streams). Mappable files are now handled
like streams, so that needs some work.
The big part that is missing is the use of the chunk list for determining
whether a sequence of blocks has data and all the handling of writing the
image data to the output file.
As such: this commit breaks mkimg and makes it useless. It's a good WIP
to safe thogh -- hence doing it on my branch.
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Sun Sep 28 00:24:01 2014 (r272232)
+++ user/marcel/mkimg/image.c Sun Sep 28 00:43:04 2014 (r272233)
@@ -27,67 +27,303 @@
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
+#include <sys/mman.h>
+#include <sys/queue.h>
+#include <sys/stat.h>
#include <sys/types.h>
#include <assert.h>
#include <errno.h>
#include <limits.h>
#include <paths.h>
+#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <unistd.h>
#include "image.h"
#include "mkimg.h"
-#define BUFFER_SIZE (1024*1024)
+struct chunk {
+ lba_t ch_block; /* Block address in image. */
+ off_t ch_ofs; /* Offset in backing file. */
+ STAILQ_ENTRY(chunk) ch_list;
+ size_t ch_size; /* Size of chunk in bytes. */
+ int ch_fd; /* FD of backing file. */
+ u_int ch_flags;
+#define CH_FLAGS_GAP 1 /* Chunk is a gap (no FD). */
+#define CH_FLAGS_DIRTY 2 /* Data modified/only in memory. */
+};
+
+static STAILQ_HEAD(chunk_head, chunk) image_chunks;
+static u_int image_nchunks;
+
+static char image_swap_file[PATH_MAX];
+static int image_swap_fd = -1;
+static u_int image_swap_pgsz;
+static off_t image_swap_size;
-static char image_tmpfile[PATH_MAX];
-static int image_fd = -1;
static lba_t image_size;
+/*
+ * Swap file handlng.
+ */
+
+static off_t
+image_swap_alloc(size_t size)
+{
+ off_t ofs;
+ size_t unit;
+
+ unit = (secsz > image_swap_pgsz) ? secsz : image_swap_pgsz;
+ assert((unit & (unit - 1)) == 0);
+
+ size = (size + unit - 1) & ~(unit - 1);
+
+ ofs = image_swap_size;
+ image_swap_size += size;
+ if (ftruncate(image_swap_fd, image_swap_size) == -1) {
+ image_swap_size = ofs;
+ ofs = -1LL;
+ }
+ fprintf(stderr, "SWAP: off=%jd, size=%zu\n", (intmax_t)ofs, size);
+ return (ofs);
+}
+
+/*
+ * Image chunk handling.
+ */
+
static void
-cleanup(void)
+image_chunk_dump(void)
{
+ struct chunk *ch;
- if (image_fd != -1)
- close(image_fd);
- unlink(image_tmpfile);
+ fprintf(stderr, "%u chunks:\n", image_nchunks);
+ STAILQ_FOREACH(ch, &image_chunks, ch_list) {
+ fprintf(stderr, "\tblk=%jd, ofs=%jd, fd=%d, sz=%zu, fl=%u\n",
+ (intmax_t)ch->ch_block, (intmax_t)ch->ch_ofs, ch->ch_fd,
+ ch->ch_size, ch->ch_flags);
+ }
}
-int
-image_copyin(lba_t blk, int fd, uint64_t *sizep)
+static size_t
+image_chunk_grow(struct chunk *ch, size_t sz)
+{
+ size_t dsz, newsz;
+
+ newsz = ch->ch_size + sz;
+ if (newsz > ch->ch_size) {
+ ch->ch_size = newsz;
+ return (0);
+ }
+ /* We would overflow -- create new chunk for remainder. */
+ dsz = SIZE_MAX - ch->ch_size;
+ assert(dsz < sz);
+ ch->ch_size = SIZE_MAX;
+ return (sz - dsz);
+}
+
+static int
+image_chunk_skipto(lba_t to)
+{
+ struct chunk *ch;
+ lba_t from;
+ size_t sz;
+
+ ch = STAILQ_LAST(&image_chunks, chunk, ch_list);
+ from = (ch != NULL) ? ch->ch_block + (ch->ch_size / secsz) : 0LL;
+
+ assert(from <= to);
+
+ /* Nothing to do? */
+ if (from == to)
+ return (0);
+ /* Avoid bugs due to overflows. */
+ if ((uintmax_t)(to - from) > (uintmax_t)(SIZE_MAX / secsz))
+ return (EFBIG);
+ sz = (to - from) * secsz;
+ if (ch != NULL && (ch->ch_flags & CH_FLAGS_GAP)) {
+ sz = image_chunk_grow(ch, sz);
+ if (sz == 0)
+ return (0);
+ from = ch->ch_block + (ch->ch_size / secsz);
+ }
+ ch = malloc(sizeof(*ch));
+ if (ch == NULL)
+ return (ENOMEM);
+ memset(ch, 0, sizeof(*ch));
+ ch->ch_block = from;
+ ch->ch_size = sz;
+ ch->ch_fd = -1;
+ ch->ch_flags |= CH_FLAGS_GAP;
+ STAILQ_INSERT_TAIL(&image_chunks, ch, ch_list);
+ image_nchunks++;
+ return (0);
+}
+
+static int
+image_chunk_append(lba_t blk, size_t sz, off_t ofs, int fd)
+{
+ struct chunk *ch;
+
+ ch = STAILQ_LAST(&image_chunks, chunk, ch_list);
+ if (ch != NULL && (ch->ch_flags & CH_FLAGS_GAP) == 0) {
+ if (fd == ch->ch_fd &&
+ blk == (lba_t)(ch->ch_block + (ch->ch_size / secsz)) &&
+ ofs == (off_t)(ch->ch_ofs + ch->ch_size)) {
+ sz = image_chunk_grow(ch, sz);
+ if (sz == 0)
+ return (0);
+ blk = ch->ch_block + (ch->ch_size / secsz);
+ ofs = ch->ch_ofs + ch->ch_size;
+ }
+ }
+ ch = malloc(sizeof(*ch));
+ if (ch == NULL)
+ return (ENOMEM);
+ memset(ch, 0, sizeof(*ch));
+ ch->ch_block = blk;
+ ch->ch_ofs = ofs;
+ ch->ch_size = sz;
+ ch->ch_fd = fd;
+ STAILQ_INSERT_TAIL(&image_chunks, ch, ch_list);
+ image_nchunks++;
+ return (0);
+}
+
+static int
+image_chunk_copyin(lba_t blk, void *buf, size_t sz, off_t ofs, int fd)
+{
+ uint64_t *p = buf;
+ size_t n;
+ int error;
+
+ assert(((uintptr_t)p & 3) == 0);
+
+ error = 0;
+ sz = (sz + secsz - 1) & ~(secsz - 1);
+ while (!error && sz > 0) {
+ n = 0;
+ while (n < (secsz >> 3) && p[n] == 0)
+ n++;
+ if (n == (secsz >> 3))
+ error = image_chunk_skipto(blk + 1);
+ else
+ error = image_chunk_append(blk, secsz, ofs, fd);
+ blk++;
+ p += (secsz >> 3);
+ sz -= secsz;
+ ofs += secsz;
+ }
+ return (error);
+}
+
+/*
+ * File mapping support.
+ */
+
+static void *
+image_file_map(int fd, off_t ofs, size_t sz)
+{
+ void *ptr;
+ size_t unit;
+ int flags, prot;
+
+ unit = (secsz > image_swap_pgsz) ? secsz : image_swap_pgsz;
+ assert((unit & (unit - 1)) == 0);
+
+ flags = MAP_NOCORE | MAP_NOSYNC | MAP_SHARED;
+ /* Allow writing to our swap file only. */
+ prot = PROT_READ | ((fd == image_swap_fd) ? PROT_WRITE : 0);
+ sz = (sz + unit - 1) & ~(unit - 1);
+ ptr = mmap(NULL, sz, prot, flags, fd, ofs);
+ return ((ptr == MAP_FAILED) ? NULL : ptr);
+}
+
+static int
+image_file_unmap(void *buffer, size_t sz)
+{
+ size_t unit;
+
+ unit = (secsz > image_swap_pgsz) ? secsz : image_swap_pgsz;
+ sz = (sz + unit - 1) & ~(unit - 1);
+ munmap(buffer, sz);
+ return (0);
+}
+
+/*
+ * Input/source file handling.
+ */
+
+static int
+image_copyin_stream(lba_t blk, int fd, uint64_t *sizep)
{
char *buffer;
uint64_t bytesize;
- ssize_t bcnt, rdsz;
- int error, partial;
+ off_t swofs;
+ size_t iosz;
+ ssize_t rdsz;
+ int error;
- assert(BUFFER_SIZE % secsz == 0);
+ /*
+ * This makes sure we're doing I/O in multiples of the page
+ * size as well as of the sector size. 2MB is the minimum
+ * by virtue of secsz at least 512 bytes and the page size
+ * at least 4K bytes.
+ */
+ iosz = secsz * image_swap_pgsz;
- buffer = malloc(BUFFER_SIZE);
- if (buffer == NULL)
- return (ENOMEM);
bytesize = 0;
- partial = 0;
- while (1) {
- rdsz = read(fd, buffer, BUFFER_SIZE);
- if (rdsz <= 0) {
- error = (rdsz < 0) ? errno : 0;
- break;
- }
- if (partial)
- abort();
- bytesize += rdsz;
- bcnt = (rdsz + secsz - 1) / secsz;
- error = image_write(blk, buffer, bcnt);
+ do {
+ swofs = image_swap_alloc(iosz);
+ if (swofs == -1LL)
+ return (errno);
+ buffer = image_file_map(image_swap_fd, swofs, iosz);
+ if (buffer == NULL)
+ return (errno);
+ rdsz = read(fd, buffer, iosz);
+ if (rdsz > 0)
+ error = image_chunk_copyin(blk, buffer, rdsz, swofs,
+ image_swap_fd);
+ else if (rdsz < 0)
+ error = errno;
+ else
+ error = 0;
+ image_file_unmap(buffer, iosz);
+ /* XXX should we relinguish unused swap space? */
if (error)
- break;
- blk += bcnt;
- partial = ((ssize_t)(bcnt * secsz) != rdsz) ? 1 : 0;
- }
- free(buffer);
+ return (error);
+
+ bytesize += rdsz;
+ blk += (rdsz + secsz - 1) / secsz;
+ } while (rdsz > 0);
+
if (sizep != NULL)
*sizep = bytesize;
+ return (0);
+}
+
+static int
+image_copyin_mapped(lba_t blk, int fd, uint64_t *sizep)
+{
+
+ return (image_copyin_stream(blk, fd, sizep));
+}
+
+int
+image_copyin(lba_t blk, int fd, uint64_t *sizep)
+{
+ struct stat sb;
+ int error;
+
+ error = image_chunk_skipto(blk);
+ if (!error) {
+ if (fstat(fd, &sb) == -1 || !S_ISREG(sb.st_mode))
+ error = image_copyin_stream(blk, fd, sizep);
+ else
+ error = image_copyin_mapped(blk, fd, sizep);
+ }
return (error);
}
@@ -120,23 +356,25 @@ image_copyout_region(int fd, lba_t blk,
{
char *buffer;
off_t ofs;
- size_t sz;
+ size_t bufsz, sz;
ssize_t rdsz, wrsz;
int error;
+ bufsz = secsz * image_swap_pgsz;
+
ofs = lseek(fd, 0L, SEEK_CUR);
blk *= secsz;
- if (lseek(image_fd, blk, SEEK_SET) != blk)
+ if (lseek(image_swap_fd, blk, SEEK_SET) != blk)
return (errno);
- buffer = malloc(BUFFER_SIZE);
+ buffer = malloc(bufsz);
if (buffer == NULL)
return (errno);
error = 0;
size *= secsz;
while (size > 0) {
- sz = (BUFFER_SIZE < size) ? BUFFER_SIZE : size;
- rdsz = read(image_fd, buffer, sz);
+ sz = ((ssize_t)bufsz < size) ? bufsz : (size_t)size;
+ rdsz = read(image_swap_fd, buffer, sz);
if (rdsz <= 0) {
error = (rdsz < 0) ? errno : 0;
break;
@@ -161,7 +399,7 @@ image_data(lba_t blk, lba_t size)
char *buffer, *p;
blk *= secsz;
- if (lseek(image_fd, blk, SEEK_SET) != blk)
+ if (lseek(image_swap_fd, blk, SEEK_SET) != blk)
return (1);
size *= secsz;
@@ -169,7 +407,7 @@ image_data(lba_t blk, lba_t size)
if (buffer == NULL)
return (1);
- if (read(image_fd, buffer, size) != (ssize_t)size) {
+ if (read(image_swap_fd, buffer, size) != (ssize_t)size) {
free(buffer);
return (1);
}
@@ -185,7 +423,12 @@ image_data(lba_t blk, lba_t size)
lba_t
image_get_size(void)
{
+ static int once = 0;
+ if (once == 0) {
+ once++;
+ image_chunk_dump();
+ }
return (image_size);
}
@@ -193,8 +436,10 @@ int
image_set_size(lba_t blk)
{
+ image_chunk_skipto(blk);
+
image_size = blk;
- if (ftruncate(image_fd, blk * secsz) == -1)
+ if (ftruncate(image_swap_fd, blk * secsz) == -1)
return (errno);
return (0);
}
@@ -204,27 +449,42 @@ image_write(lba_t blk, void *buf, ssize_
{
blk *= secsz;
- if (lseek(image_fd, blk, SEEK_SET) != blk)
+ if (lseek(image_swap_fd, blk, SEEK_SET) != blk)
return (errno);
len *= secsz;
- if (sparse_write(image_fd, buf, len) != len)
+ if (sparse_write(image_swap_fd, buf, len) != len)
return (errno);
return (0);
}
+static void
+image_cleanup(void)
+{
+
+ if (image_swap_fd != -1)
+ close(image_swap_fd);
+ unlink(image_swap_file);
+}
+
int
image_init(void)
{
const char *tmpdir;
- if (atexit(cleanup) == -1)
+ STAILQ_INIT(&image_chunks);
+ image_nchunks = 0;
+
+ image_swap_size = 0;
+ image_swap_pgsz = getpagesize();
+
+ if (atexit(image_cleanup) == -1)
return (errno);
if ((tmpdir = getenv("TMPDIR")) == NULL || *tmpdir == '\0')
tmpdir = _PATH_TMP;
- snprintf(image_tmpfile, sizeof(image_tmpfile), "%s/mkimg-XXXXXX",
+ snprintf(image_swap_file, sizeof(image_swap_file), "%s/mkimg-XXXXXX",
tmpdir);
- image_fd = mkstemp(image_tmpfile);
- if (image_fd == -1)
+ image_swap_fd = mkstemp(image_swap_file);
+ if (image_swap_fd == -1)
return (errno);
return (0);
}
From owner-svn-src-user@FreeBSD.ORG Sun Sep 28 18:34:22 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id EB3603E4;
Sun, 28 Sep 2014 18:34:21 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id D7D24762;
Sun, 28 Sep 2014 18:34:21 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s8SIYLCh091092;
Sun, 28 Sep 2014 18:34:21 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s8SIYLcm091090;
Sun, 28 Sep 2014 18:34:21 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201409281834.s8SIYLcm091090@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Sun, 28 Sep 2014 18:34:21 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272265 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 28 Sep 2014 18:34:22 -0000
Author: marcel
Date: Sun Sep 28 18:34:20 2014
New Revision: 272265
URL: http://svnweb.freebsd.org/changeset/base/272265
Log:
Implement image_copyin_mapped():
1. If we can't seek, then let image_copyin_stream() deal with it.
2. Use SEEK_HOLE and SEEK_DATA to iterate over the regions of a
sparse file.
3. Round or truncate to the sector size based on whether it's a
hold or data. We truncate for holes and round for data. This to
make sure to handle sparse files correctly when the sector size
is larger than the file system's block/fragment size. Since we
detect runts of zeroes for mmap'd data (to handle streams), we
correct ourselves if such is applicable. We never mishandle
data, which is the most important.
I need to check with a file system that doesn't support sparse files
what SEEK_HOLE and SEEK_DATA return. I'm thinking -1, which is what
I don't handle yet. It's easy to do, because everything should be
treated as data and we'll find the holes the hard way.
While here: set DEBUG_FLAGS unconditionally to make sure I can always
debug a core.
Modified:
user/marcel/mkimg/Makefile
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/Makefile
==============================================================================
--- user/marcel/mkimg/Makefile Sun Sep 28 17:53:10 2014 (r272264)
+++ user/marcel/mkimg/Makefile Sun Sep 28 18:34:20 2014 (r272265)
@@ -10,6 +10,8 @@ MKIMG_VERSION=20140927
CFLAGS+=-DMKIMG_VERSION=${MKIMG_VERSION}
CFLAGS+=-DSPARSE_WRITE
+DEBUG_FLAGS=-O0 -gdwarf-2
+
# List of formats to support
SRCS+= \
qcow.c \
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Sun Sep 28 17:53:10 2014 (r272264)
+++ user/marcel/mkimg/image.c Sun Sep 28 18:34:20 2014 (r272265)
@@ -307,8 +307,88 @@ image_copyin_stream(lba_t blk, int fd, u
static int
image_copyin_mapped(lba_t blk, int fd, uint64_t *sizep)
{
+ off_t cur, data, end, hole, pos;
+ void *buf;
+ uint64_t bytesize;
+ size_t sz;
+ int error;
+
+ /*
+ * We'd like to know the size of the file and we must
+ * be able to seek in order to mmap(2). If this isn't
+ * possible, then treat the file as a stream/pipe.
+ */
+ end = lseek(fd, 0L, SEEK_END);
+ if (end == -1L)
+ return (image_copyin_stream(blk, fd, sizep));
+
+ /*
+ * We need the file opened for the duration and our
+ * caller is going to close the file. Make a dup(2)
+ * so that control the faith of the descriptor.
+ */
+ fd = dup(fd);
+ if (fd == -1)
+ return (errno);
- return (image_copyin_stream(blk, fd, sizep));
+ bytesize = 0;
+ cur = pos = 0;
+ error = 0;
+ while (!error && cur < end) {
+ hole = lseek(fd, cur, SEEK_HOLE);
+ data = lseek(fd, cur, SEEK_DATA);
+
+ fprintf(stderr, "XXX: %s: cur=%jd, pos=%jd, hole=%jd, "
+ "data=%jd\n", __func__, (intmax_t)cur, (intmax_t)pos,
+ (intmax_t)hole, (intmax_t)data);
+
+ if (cur == hole && data > hole) {
+ hole = pos;
+ pos = data & ~(secsz - 1);
+
+ fprintf(stderr, "GAP %jd-%jd\n",
+ (intmax_t)hole, (intmax_t)pos);
+
+ blk += (pos - hole) / secsz;
+ error = image_chunk_skipto(blk);
+
+ bytesize += pos - hole;
+ cur = data;
+ } else if (cur == data && hole > data) {
+ data = pos;
+ pos = (hole + secsz - 1) & ~(secsz - 1);
+
+ fprintf(stderr, "DATA %jd-%jd data\n",
+ (intmax_t)data, (intmax_t)pos);
+
+ /* Sloppy... */
+ sz = pos - data;
+ assert((off_t)sz == pos - data);
+
+ buf = image_file_map(fd, data, sz);
+ if (buf != NULL) {
+ error = image_chunk_copyin(blk, buf, sz,
+ data, fd);
+ image_file_unmap(buf, sz);
+ } else
+ error = errno;
+ blk += sz / secsz;
+ bytesize += sz;
+ cur = hole;
+ } else {
+ /*
+ * I don't know what this means or whether it
+ * can happen at all...
+ */
+ error = EDOOFUS;
+ break;
+ }
+ }
+ if (error)
+ close(fd);
+ if (!error && sizep != NULL)
+ *sizep = bytesize;
+ return (error);
}
int
From owner-svn-src-user@FreeBSD.ORG Sun Sep 28 19:55:22 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id A47C9145;
Sun, 28 Sep 2014 19:55:22 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 91311E4A;
Sun, 28 Sep 2014 19:55:22 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s8SJtM37029895;
Sun, 28 Sep 2014 19:55:22 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s8SJtM2C029894;
Sun, 28 Sep 2014 19:55:22 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201409281955.s8SJtM2C029894@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Sun, 28 Sep 2014 19:55:22 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272268 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 28 Sep 2014 19:55:22 -0000
Author: marcel
Date: Sun Sep 28 19:55:21 2014
New Revision: 272268
URL: http://svnweb.freebsd.org/changeset/base/272268
Log:
When SEEK_HOLE and SEEK_DATA return -1, then the entire file is data.
Set hole and data accordingly and before acting on them.
While here: remove some debugging printfs; we're good so far.
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Sun Sep 28 19:17:33 2014 (r272267)
+++ user/marcel/mkimg/image.c Sun Sep 28 19:55:21 2014 (r272268)
@@ -86,7 +86,6 @@ image_swap_alloc(size_t size)
image_swap_size = ofs;
ofs = -1LL;
}
- fprintf(stderr, "SWAP: off=%jd, size=%zu\n", (intmax_t)ofs, size);
return (ofs);
}
@@ -338,17 +337,19 @@ image_copyin_mapped(lba_t blk, int fd, u
hole = lseek(fd, cur, SEEK_HOLE);
data = lseek(fd, cur, SEEK_DATA);
- fprintf(stderr, "XXX: %s: cur=%jd, pos=%jd, hole=%jd, "
- "data=%jd\n", __func__, (intmax_t)cur, (intmax_t)pos,
- (intmax_t)hole, (intmax_t)data);
+ /*
+ * Treat the entire file as data if sparse files
+ * are not supported by the underlying file system.
+ */
+ if (hole == -1 && data == -1) {
+ data = cur;
+ hole = end;
+ }
if (cur == hole && data > hole) {
hole = pos;
pos = data & ~(secsz - 1);
- fprintf(stderr, "GAP %jd-%jd\n",
- (intmax_t)hole, (intmax_t)pos);
-
blk += (pos - hole) / secsz;
error = image_chunk_skipto(blk);
@@ -358,9 +359,6 @@ image_copyin_mapped(lba_t blk, int fd, u
data = pos;
pos = (hole + secsz - 1) & ~(secsz - 1);
- fprintf(stderr, "DATA %jd-%jd data\n",
- (intmax_t)data, (intmax_t)pos);
-
/* Sloppy... */
sz = pos - data;
assert((off_t)sz == pos - data);
From owner-svn-src-user@FreeBSD.ORG Mon Sep 29 16:24:49 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 8B6A22FC;
Mon, 29 Sep 2014 16:24:49 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 6C9AA125;
Mon, 29 Sep 2014 16:24:49 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s8TGOnXB005122;
Mon, 29 Sep 2014 16:24:49 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s8TGOnPJ005121;
Mon, 29 Sep 2014 16:24:49 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201409291624.s8TGOnPJ005121@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Mon, 29 Sep 2014 16:24:49 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272283 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 29 Sep 2014 16:24:49 -0000
Author: marcel
Date: Mon Sep 29 16:24:48 2014
New Revision: 272283
URL: http://svnweb.freebsd.org/changeset/base/272283
Log:
o In image_set_size(), don't truncate the swap file. It isn't holding
the entire image data anymore.
o Prepare for changes to image_write(): partitioning schemes call
image_write() to fill in the sectors they reserved from themselves.
Since we're talking about a handful of sectors, there's no need to
have them in the swap file. Change struct chunk to accomodate for
that. A chunk now has a type. That type can be "zeroes", "file" or
"memory".
o Flesh out the cleanup function: iterate over the chunks and close
or free the corresponding data container. Free the chunk as well.
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Mon Sep 29 15:05:23 2014 (r272282)
+++ user/marcel/mkimg/image.c Mon Sep 29 16:24:48 2014 (r272283)
@@ -45,14 +45,22 @@ __FBSDID("$FreeBSD$");
#include "mkimg.h"
struct chunk {
- lba_t ch_block; /* Block address in image. */
- off_t ch_ofs; /* Offset in backing file. */
STAILQ_ENTRY(chunk) ch_list;
size_t ch_size; /* Size of chunk in bytes. */
- int ch_fd; /* FD of backing file. */
- u_int ch_flags;
-#define CH_FLAGS_GAP 1 /* Chunk is a gap (no FD). */
-#define CH_FLAGS_DIRTY 2 /* Data modified/only in memory. */
+ lba_t ch_block; /* Block address in image. */
+ union {
+ struct {
+ off_t ofs; /* Offset in backing file. */
+ int fd; /* FD of backing file. */
+ } file;
+ struct {
+ void *ptr; /* Pointer to data in memory */
+ } mem;
+ } ch_u;
+ u_int ch_type;
+#define CH_TYPE_ZEROES 0 /* Chunk is a gap (no data). */
+#define CH_TYPE_FILE 1 /* File-backed chunk. */
+#define CH_TYPE_MEMORY 2 /* Memory-backed chunk */
};
static STAILQ_HEAD(chunk_head, chunk) image_chunks;
@@ -100,9 +108,22 @@ image_chunk_dump(void)
fprintf(stderr, "%u chunks:\n", image_nchunks);
STAILQ_FOREACH(ch, &image_chunks, ch_list) {
- fprintf(stderr, "\tblk=%jd, ofs=%jd, fd=%d, sz=%zu, fl=%u\n",
- (intmax_t)ch->ch_block, (intmax_t)ch->ch_ofs, ch->ch_fd,
- ch->ch_size, ch->ch_flags);
+ fprintf(stderr, "\tblk=%jd, sz=%zu, type=%u",
+ (intmax_t)ch->ch_block, ch->ch_size, ch->ch_type);
+ switch (ch->ch_type) {
+ case CH_TYPE_ZEROES:
+ fputc('\n', stderr);
+ break;
+ case CH_TYPE_FILE:
+ fprintf(stderr, "; ofs=%jd, fd=%d\n",
+ (intmax_t)ch->ch_u.file.ofs, ch->ch_u.file.fd);
+ break;
+ case CH_TYPE_MEMORY:
+ fprintf(stderr, "; ptr=%p\n", ch->ch_u.mem.ptr);
+ break;
+ default:
+ abort();
+ }
}
}
@@ -142,7 +163,7 @@ image_chunk_skipto(lba_t to)
if ((uintmax_t)(to - from) > (uintmax_t)(SIZE_MAX / secsz))
return (EFBIG);
sz = (to - from) * secsz;
- if (ch != NULL && (ch->ch_flags & CH_FLAGS_GAP)) {
+ if (ch != NULL && ch->ch_type == CH_TYPE_ZEROES) {
sz = image_chunk_grow(ch, sz);
if (sz == 0)
return (0);
@@ -154,8 +175,7 @@ image_chunk_skipto(lba_t to)
memset(ch, 0, sizeof(*ch));
ch->ch_block = from;
ch->ch_size = sz;
- ch->ch_fd = -1;
- ch->ch_flags |= CH_FLAGS_GAP;
+ ch->ch_type = CH_TYPE_ZEROES;
STAILQ_INSERT_TAIL(&image_chunks, ch, ch_list);
image_nchunks++;
return (0);
@@ -167,15 +187,15 @@ image_chunk_append(lba_t blk, size_t sz,
struct chunk *ch;
ch = STAILQ_LAST(&image_chunks, chunk, ch_list);
- if (ch != NULL && (ch->ch_flags & CH_FLAGS_GAP) == 0) {
- if (fd == ch->ch_fd &&
+ if (ch != NULL && ch->ch_type == CH_TYPE_FILE) {
+ if (fd == ch->ch_u.file.fd &&
blk == (lba_t)(ch->ch_block + (ch->ch_size / secsz)) &&
- ofs == (off_t)(ch->ch_ofs + ch->ch_size)) {
+ ofs == (off_t)(ch->ch_u.file.ofs + ch->ch_size)) {
sz = image_chunk_grow(ch, sz);
if (sz == 0)
return (0);
blk = ch->ch_block + (ch->ch_size / secsz);
- ofs = ch->ch_ofs + ch->ch_size;
+ ofs = ch->ch_u.file.ofs + ch->ch_size;
}
}
ch = malloc(sizeof(*ch));
@@ -183,9 +203,10 @@ image_chunk_append(lba_t blk, size_t sz,
return (ENOMEM);
memset(ch, 0, sizeof(*ch));
ch->ch_block = blk;
- ch->ch_ofs = ofs;
ch->ch_size = sz;
- ch->ch_fd = fd;
+ ch->ch_type = CH_TYPE_FILE;
+ ch->ch_u.file.ofs = ofs;
+ ch->ch_u.file.fd = fd;
STAILQ_INSERT_TAIL(&image_chunks, ch, ch_list);
image_nchunks++;
return (0);
@@ -513,13 +534,12 @@ image_get_size(void)
int
image_set_size(lba_t blk)
{
+ int error;
- image_chunk_skipto(blk);
-
- image_size = blk;
- if (ftruncate(image_swap_fd, blk * secsz) == -1)
- return (errno);
- return (0);
+ error = image_chunk_skipto(blk);
+ if (!error)
+ image_size = blk;
+ return (error);
}
int
@@ -538,7 +558,24 @@ image_write(lba_t blk, void *buf, ssize_
static void
image_cleanup(void)
{
+ struct chunk *ch;
+ while ((ch = STAILQ_FIRST(&image_chunks)) != NULL) {
+ switch (ch->ch_type) {
+ case CH_TYPE_FILE:
+ /* We may be closing the same file multiple times. */
+ if (ch->ch_u.file.fd != -1)
+ close(ch->ch_u.file.fd);
+ break;
+ case CH_TYPE_MEMORY:
+ free(ch->ch_u.mem.ptr);
+ break;
+ default:
+ break;
+ }
+ STAILQ_REMOVE_HEAD(&image_chunks, ch_list);
+ free(ch);
+ }
if (image_swap_fd != -1)
close(image_swap_fd);
unlink(image_swap_file);
From owner-svn-src-user@FreeBSD.ORG Tue Sep 30 23:16:03 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id CEB03803;
Tue, 30 Sep 2014 23:16:03 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id AF90625A;
Tue, 30 Sep 2014 23:16:03 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s8UNG3Bp082312;
Tue, 30 Sep 2014 23:16:03 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s8UNG3rb082311;
Tue, 30 Sep 2014 23:16:03 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201409302316.s8UNG3rb082311@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Tue, 30 Sep 2014 23:16:03 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272335 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 30 Sep 2014 23:16:03 -0000
Author: marcel
Date: Tue Sep 30 23:16:03 2014
New Revision: 272335
URL: http://svnweb.freebsd.org/changeset/base/272335
Log:
Implement image_write():
For every sector in the buffer, find the chunk that contains it
and break it up (if it's not already a memory chunk) so that we
have have a new chunk for every sector being written. We don't
write to chunks for type file, because we may not be able to
write and we don't want to clobber partition data.
We don't try to combine adjacent chunks of type memory into a
single one. It doesn't seem to have any significant upsides
other than reducing syscall overhead. The downsides includes
"excessive" copying and memory fragmantation. Without any kind
of benchmarking, simplicity wins.
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Tue Sep 30 23:01:11 2014 (r272334)
+++ user/marcel/mkimg/image.c Tue Sep 30 23:16:03 2014 (r272335)
@@ -73,6 +73,22 @@ static off_t image_swap_size;
static lba_t image_size;
+static int
+is_empty_sector(void *buf)
+{
+ uint64_t *p = buf;
+ size_t n, max;
+
+ assert(((uintptr_t)p & 3) == 0);
+
+ max = secsz / sizeof(uint64_t);
+ for (n = 0; n < max; n++) {
+ if (p[n] != 0UL)
+ return (0);
+ }
+ return (1);
+}
+
/*
* Swap file handlng.
*/
@@ -102,11 +118,11 @@ image_swap_alloc(size_t size)
*/
static void
-image_chunk_dump(void)
+image_chunk_dump(int count)
{
struct chunk *ch;
- fprintf(stderr, "%u chunks:\n", image_nchunks);
+ fprintf(stderr, "Dump %d: %u chunks:\n", count, image_nchunks);
STAILQ_FOREACH(ch, &image_chunks, ch_list) {
fprintf(stderr, "\tblk=%jd, sz=%zu, type=%u",
(intmax_t)ch->ch_block, ch->ch_size, ch->ch_type);
@@ -127,6 +143,25 @@ image_chunk_dump(void)
}
}
+static struct chunk *
+image_chunk_find(lba_t blk)
+{
+ static struct chunk *last = NULL;
+ struct chunk *ch;
+
+ ch = (last != NULL && last->ch_block <= blk)
+ ? last : STAILQ_FIRST(&image_chunks);
+ while (ch != NULL) {
+ if (ch->ch_block <= blk &&
+ (lba_t)(ch->ch_block + (ch->ch_size / secsz)) > blk) {
+ last = ch;
+ break;
+ }
+ ch = STAILQ_NEXT(ch, ch_list);
+ }
+ return (ch);
+}
+
static size_t
image_chunk_grow(struct chunk *ch, size_t sz)
{
@@ -144,6 +179,50 @@ image_chunk_grow(struct chunk *ch, size_
return (sz - dsz);
}
+static struct chunk *
+image_chunk_memory(struct chunk *ch, lba_t blk)
+{
+ struct chunk *new;
+ void *ptr;
+
+ ptr = calloc(1, secsz);
+ if (ptr == NULL)
+ return (NULL);
+
+ if (ch->ch_block < blk) {
+ new = malloc(sizeof(*new));
+ if (new == NULL) {
+ free(ptr);
+ return (NULL);
+ }
+ memcpy(new, ch, sizeof(*new));
+ ch->ch_size = (blk - ch->ch_block) * secsz;
+ new->ch_block = blk;
+ new->ch_size -= ch->ch_size;
+ STAILQ_INSERT_AFTER(&image_chunks, ch, new, ch_list);
+ image_nchunks++;
+ ch = new;
+ }
+
+ if (ch->ch_size > secsz) {
+ new = malloc(sizeof(*new));
+ if (new == NULL) {
+ free(ptr);
+ return (NULL);
+ }
+ memcpy(new, ch, sizeof(*new));
+ ch->ch_size = secsz;
+ new->ch_block++;
+ new->ch_size -= secsz;
+ STAILQ_INSERT_AFTER(&image_chunks, ch, new, ch_list);
+ image_nchunks++;
+ }
+
+ ch->ch_type = CH_TYPE_MEMORY;
+ ch->ch_u.mem.ptr = ptr;
+ return (ch);
+}
+
static int
image_chunk_skipto(lba_t to)
{
@@ -215,24 +294,18 @@ image_chunk_append(lba_t blk, size_t sz,
static int
image_chunk_copyin(lba_t blk, void *buf, size_t sz, off_t ofs, int fd)
{
- uint64_t *p = buf;
- size_t n;
+ uint8_t *p = buf;
int error;
- assert(((uintptr_t)p & 3) == 0);
-
error = 0;
sz = (sz + secsz - 1) & ~(secsz - 1);
while (!error && sz > 0) {
- n = 0;
- while (n < (secsz >> 3) && p[n] == 0)
- n++;
- if (n == (secsz >> 3))
+ if (is_empty_sector(p))
error = image_chunk_skipto(blk + 1);
else
error = image_chunk_append(blk, secsz, ofs, fd);
blk++;
- p += (secsz >> 3);
+ p += secsz;
sz -= secsz;
ofs += secsz;
}
@@ -522,12 +595,9 @@ image_data(lba_t blk, lba_t size)
lba_t
image_get_size(void)
{
- static int once = 0;
+ static int count = 0;
- if (once == 0) {
- once++;
- image_chunk_dump();
- }
+ image_chunk_dump(count++);
return (image_size);
}
@@ -545,13 +615,28 @@ image_set_size(lba_t blk)
int
image_write(lba_t blk, void *buf, ssize_t len)
{
+ struct chunk *ch;
- blk *= secsz;
- if (lseek(image_swap_fd, blk, SEEK_SET) != blk)
- return (errno);
- len *= secsz;
- if (sparse_write(image_swap_fd, buf, len) != len)
- return (errno);
+ while (len > 0) {
+ if (!is_empty_sector(buf)) {
+ ch = image_chunk_find(blk);
+ if (ch == NULL)
+ return (ENXIO);
+ /* We may not be able to write to files. */
+ if (ch->ch_type == CH_TYPE_FILE)
+ return (EINVAL);
+ if (ch->ch_type == CH_TYPE_ZEROES) {
+ ch = image_chunk_memory(ch, blk);
+ if (ch == NULL)
+ return (ENOMEM);
+ }
+ assert(ch->ch_type == CH_TYPE_MEMORY);
+ memcpy(ch->ch_u.mem.ptr, buf, secsz);
+ }
+ blk++;
+ buf = (char *)buf + secsz;
+ len--;
+ }
return (0);
}
From owner-svn-src-user@FreeBSD.ORG Tue Sep 30 23:29:07 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id A70FEB59;
Tue, 30 Sep 2014 23:29:07 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 93CF33A1;
Tue, 30 Sep 2014 23:29:07 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s8UNT78O087312;
Tue, 30 Sep 2014 23:29:07 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s8UNT7tn087311;
Tue, 30 Sep 2014 23:29:07 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201409302329.s8UNT7tn087311@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Tue, 30 Sep 2014 23:29:07 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272337 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 30 Sep 2014 23:29:07 -0000
Author: marcel
Date: Tue Sep 30 23:29:06 2014
New Revision: 272337
URL: http://svnweb.freebsd.org/changeset/base/272337
Log:
Implement image_data(): we now simply call image_chunk_find() to
get the chunk that contains the block and we effectively have the
answer right away. Account for the as-of-yet non-existent case of
having multiple chunks of type zeroes adjacent to each other and
having to handle chunk-crossing -- hence the loop.
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Tue Sep 30 23:16:26 2014 (r272336)
+++ user/marcel/mkimg/image.c Tue Sep 30 23:29:06 2014 (r272337)
@@ -568,28 +568,22 @@ image_copyout_region(int fd, lba_t blk,
int
image_data(lba_t blk, lba_t size)
{
- char *buffer, *p;
+ struct chunk *ch;
+ lba_t lim;
- blk *= secsz;
- if (lseek(image_swap_fd, blk, SEEK_SET) != blk)
- return (1);
-
- size *= secsz;
- buffer = malloc(size);
- if (buffer == NULL)
- return (1);
-
- if (read(image_swap_fd, buffer, size) != (ssize_t)size) {
- free(buffer);
- return (1);
+ while (1) {
+ ch = image_chunk_find(blk);
+ if (ch == NULL)
+ return (0);
+ if (ch->ch_type != CH_TYPE_ZEROES)
+ return (1);
+ lim = ch->ch_block + (ch->ch_size / secsz);
+ if (lim >= blk + size)
+ return (0);
+ size -= lim - blk;
+ blk = lim;
}
-
- p = buffer;
- while (size > 0 && *p == '\0')
- size--, p++;
-
- free(buffer);
- return ((size == 0) ? 0 : 1);
+ /*NOTREACHED*/
}
lba_t
From owner-svn-src-user@FreeBSD.ORG Wed Oct 1 01:18:26 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 2087FD2F;
Wed, 1 Oct 2014 01:18:26 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 0D883FFD;
Wed, 1 Oct 2014 01:18:26 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s911IPCn039777;
Wed, 1 Oct 2014 01:18:25 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s911IPb0039776;
Wed, 1 Oct 2014 01:18:25 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201410010118.s911IPb0039776@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Wed, 1 Oct 2014 01:18:25 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272338 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 01:18:26 -0000
Author: marcel
Date: Wed Oct 1 01:18:25 2014
New Revision: 272338
URL: http://svnweb.freebsd.org/changeset/base/272338
Log:
Implement image_copyout_region(): This is the one function that is
responsible for writing to the output file. For eack chunk that
covers all or parts of the region to write, call a corresponding
function that knows how to write a chunk of that type:
image_copyout_memory(): Use write(2) to write the buffer.
image_copyout_zeroes(): Try and seek forward to create a sparse
file. If that fails, scribble zeroes to the output file to
support writing to stdout. Use image_copyout_memoryIO for
that.
image_copyout_file(): Iteratively map bits of the input file and
use image_copyout_memory() to write the bits to the output
file.
With this commit all unit tests pass again.
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Tue Sep 30 23:29:06 2014 (r272337)
+++ user/marcel/mkimg/image.c Wed Oct 1 01:18:25 2014 (r272338)
@@ -499,6 +499,10 @@ image_copyin(lba_t blk, int fd, uint64_t
return (error);
}
+/*
+ * Output/sink file handling.
+ */
+
int
image_copyout(int fd)
{
@@ -523,46 +527,103 @@ image_copyout_done(int fd)
return (error);
}
-int
-image_copyout_region(int fd, lba_t blk, lba_t size)
+static int
+image_copyout_memory(int fd, size_t size, void *ptr)
{
- char *buffer;
- off_t ofs;
- size_t bufsz, sz;
- ssize_t rdsz, wrsz;
+
+ if (write(fd, ptr, size) == -1)
+ return (errno);
+ return (0);
+}
+
+static int
+image_copyout_zeroes(int fd, size_t size)
+{
+ static uint8_t *zeroes = NULL;
+ size_t sz;
int error;
- bufsz = secsz * image_swap_pgsz;
+ if (lseek(fd, (off_t)size, SEEK_CUR) != -1)
+ return (0);
- ofs = lseek(fd, 0L, SEEK_CUR);
+ /*
+ * If we can't seek, we must write.
+ */
+
+ if (zeroes == NULL) {
+ zeroes = calloc(1, secsz);
+ if (zeroes == NULL)
+ return (ENOMEM);
+ }
+
+ while (size > 0) {
+ sz = (size > secsz) ? secsz : size;
+ error = image_copyout_memory(fd, sz, zeroes);
+ if (error)
+ return (error);
+ size -= sz;
+ }
+ return (0);
+}
+
+static int
+image_copyout_file(int fd, size_t size, int ifd, off_t iofs)
+{
+ void *buf;
+ size_t iosz, sz;
+ int error;
+
+ iosz = secsz * image_swap_pgsz;
+
+ while (size > 0) {
+ sz = (size > iosz) ? iosz : size;
+ buf = image_file_map(ifd, iofs, sz);
+ if (buf == NULL)
+ return (errno);
+ error = image_copyout_memory(fd, sz, buf);
+ image_file_unmap(buf, sz);
+ if (error)
+ return (error);
+ size -= sz;
+ iofs += sz;
+ }
+ return (0);
+}
+
+int
+image_copyout_region(int fd, lba_t blk, lba_t size)
+{
+ struct chunk *ch;
+ size_t ofs, sz;
+ int error;
- blk *= secsz;
- if (lseek(image_swap_fd, blk, SEEK_SET) != blk)
- return (errno);
- buffer = malloc(bufsz);
- if (buffer == NULL)
- return (errno);
- error = 0;
size *= secsz;
+
while (size > 0) {
- sz = ((ssize_t)bufsz < size) ? bufsz : (size_t)size;
- rdsz = read(image_swap_fd, buffer, sz);
- if (rdsz <= 0) {
- error = (rdsz < 0) ? errno : 0;
+ ch = image_chunk_find(blk);
+ if (ch == NULL)
+ return (EINVAL);
+ ofs = (blk - ch->ch_block) * secsz;
+ sz = ch->ch_size - ofs;
+ sz = ((lba_t)sz < size) ? sz : (size_t)size;
+ switch (ch->ch_type) {
+ case CH_TYPE_ZEROES:
+ error = image_copyout_zeroes(fd, sz);
break;
- }
- wrsz = (ofs == -1) ?
- write(fd, buffer, rdsz) :
- sparse_write(fd, buffer, rdsz);
- if (wrsz < 0) {
- error = errno;
+ case CH_TYPE_FILE:
+ error = image_copyout_file(fd, sz, ch->ch_u.file.fd,
+ ch->ch_u.file.ofs + ofs);
break;
+ case CH_TYPE_MEMORY:
+ error = image_copyout_memory(fd, sz, ch->ch_u.mem.ptr);
+ break;
+ default:
+ return (EDOOFUS);
}
- assert(wrsz == rdsz);
- size -= rdsz;
+ size -= sz;
+ blk += sz / secsz;
}
- free(buffer);
- return (error);
+ return (0);
}
int
From owner-svn-src-user@FreeBSD.ORG Wed Oct 1 01:50:24 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 9385C528;
Wed, 1 Oct 2014 01:50:24 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 8033E3EC;
Wed, 1 Oct 2014 01:50:24 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s911oOK8056243;
Wed, 1 Oct 2014 01:50:24 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s911oO1g056242;
Wed, 1 Oct 2014 01:50:24 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201410010150.s911oO1g056242@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Wed, 1 Oct 2014 01:50:24 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272339 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 01:50:24 -0000
Author: marcel
Date: Wed Oct 1 01:50:23 2014
New Revision: 272339
URL: http://svnweb.freebsd.org/changeset/base/272339
Log:
Remove image_chunk_dump() and its call-site. I used it for debugging
in the beginning, but not at the end. So, its value is marginal.
Below a performance comparison while running the unit test:
x old
+ new
N Min Max Median Avg Stddev
x 5 57.48 58.28 57.58 57.736 0.33886576
+ 5 46.82 47.61 46.93 47.034 0.32791767
Difference at 95.0% confidence
-10.702 +/- 0.486298
-18.5361% +/- 0.842279%
(Student's t, pooled s = 0.333437)
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Wed Oct 1 01:18:25 2014 (r272338)
+++ user/marcel/mkimg/image.c Wed Oct 1 01:50:23 2014 (r272339)
@@ -117,32 +117,6 @@ image_swap_alloc(size_t size)
* Image chunk handling.
*/
-static void
-image_chunk_dump(int count)
-{
- struct chunk *ch;
-
- fprintf(stderr, "Dump %d: %u chunks:\n", count, image_nchunks);
- STAILQ_FOREACH(ch, &image_chunks, ch_list) {
- fprintf(stderr, "\tblk=%jd, sz=%zu, type=%u",
- (intmax_t)ch->ch_block, ch->ch_size, ch->ch_type);
- switch (ch->ch_type) {
- case CH_TYPE_ZEROES:
- fputc('\n', stderr);
- break;
- case CH_TYPE_FILE:
- fprintf(stderr, "; ofs=%jd, fd=%d\n",
- (intmax_t)ch->ch_u.file.ofs, ch->ch_u.file.fd);
- break;
- case CH_TYPE_MEMORY:
- fprintf(stderr, "; ptr=%p\n", ch->ch_u.mem.ptr);
- break;
- default:
- abort();
- }
- }
-}
-
static struct chunk *
image_chunk_find(lba_t blk)
{
@@ -650,9 +624,7 @@ image_data(lba_t blk, lba_t size)
lba_t
image_get_size(void)
{
- static int count = 0;
- image_chunk_dump(count++);
return (image_size);
}
From owner-svn-src-user@FreeBSD.ORG Wed Oct 1 15:22:12 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 66BE3667;
Wed, 1 Oct 2014 15:22:12 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 5406A7A9;
Wed, 1 Oct 2014 15:22:12 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s91FMCBp046079;
Wed, 1 Oct 2014 15:22:12 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s91FMCmG046078;
Wed, 1 Oct 2014 15:22:12 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201410011522.s91FMCmG046078@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Wed, 1 Oct 2014 15:22:12 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272364 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 15:22:12 -0000
Author: marcel
Date: Wed Oct 1 15:22:11 2014
New Revision: 272364
URL: https://svnweb.freebsd.org/changeset/base/272364
Log:
Remove DEBUG_FLAGS again.
Modified:
user/marcel/mkimg/Makefile
Modified: user/marcel/mkimg/Makefile
==============================================================================
--- user/marcel/mkimg/Makefile Wed Oct 1 15:02:37 2014 (r272363)
+++ user/marcel/mkimg/Makefile Wed Oct 1 15:22:11 2014 (r272364)
@@ -10,8 +10,6 @@ MKIMG_VERSION=20140927
CFLAGS+=-DMKIMG_VERSION=${MKIMG_VERSION}
CFLAGS+=-DSPARSE_WRITE
-DEBUG_FLAGS=-O0 -gdwarf-2
-
# List of formats to support
SRCS+= \
qcow.c \
From owner-svn-src-user@FreeBSD.ORG Wed Oct 1 15:23:23 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id EEC3A7FF;
Wed, 1 Oct 2014 15:23:23 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id DC46B7C0;
Wed, 1 Oct 2014 15:23:23 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s91FNNdO046286;
Wed, 1 Oct 2014 15:23:23 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s91FNNDl046285;
Wed, 1 Oct 2014 15:23:23 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201410011523.s91FNNDl046285@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Wed, 1 Oct 2014 15:23:23 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272365 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 15:23:24 -0000
Author: marcel
Date: Wed Oct 1 15:23:23 2014
New Revision: 272365
URL: https://svnweb.freebsd.org/changeset/base/272365
Log:
Bump the version to 20141001.
Modified:
user/marcel/mkimg/Makefile
Modified: user/marcel/mkimg/Makefile
==============================================================================
--- user/marcel/mkimg/Makefile Wed Oct 1 15:22:11 2014 (r272364)
+++ user/marcel/mkimg/Makefile Wed Oct 1 15:23:23 2014 (r272365)
@@ -6,7 +6,7 @@ PROG= mkimg
SRCS= format.c image.c mkimg.c scheme.c
MAN= mkimg.1
-MKIMG_VERSION=20140927
+MKIMG_VERSION=20141001
CFLAGS+=-DMKIMG_VERSION=${MKIMG_VERSION}
CFLAGS+=-DSPARSE_WRITE
From owner-svn-src-user@FreeBSD.ORG Wed Oct 1 16:09:12 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 69A3EB3;
Wed, 1 Oct 2014 16:09:12 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 568EFDEE;
Wed, 1 Oct 2014 16:09:12 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s91G9CW9066527;
Wed, 1 Oct 2014 16:09:12 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s91G9CkA066526;
Wed, 1 Oct 2014 16:09:12 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201410011609.s91G9CkA066526@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Wed, 1 Oct 2014 16:09:12 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272370 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 16:09:12 -0000
Author: marcel
Date: Wed Oct 1 16:09:11 2014
New Revision: 272370
URL: https://svnweb.freebsd.org/changeset/base/272370
Log:
Fix sloppy code (that was commented as such): Map the file with
partition data using smaller pieces at at time. It's rather hard
to map a 10G partition in memory on a 32-bit machine...
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Wed Oct 1 16:08:19 2014 (r272369)
+++ user/marcel/mkimg/image.c Wed Oct 1 16:09:11 2014 (r272370)
@@ -377,7 +377,7 @@ image_copyin_mapped(lba_t blk, int fd, u
off_t cur, data, end, hole, pos;
void *buf;
uint64_t bytesize;
- size_t sz;
+ size_t iosz, sz;
int error;
/*
@@ -398,6 +398,8 @@ image_copyin_mapped(lba_t blk, int fd, u
if (fd == -1)
return (errno);
+ iosz = secsz * image_swap_pgsz;
+
bytesize = 0;
cur = pos = 0;
error = 0;
@@ -427,19 +429,22 @@ image_copyin_mapped(lba_t blk, int fd, u
data = pos;
pos = (hole + secsz - 1) & ~(secsz - 1);
- /* Sloppy... */
- sz = pos - data;
- assert((off_t)sz == pos - data);
-
- buf = image_file_map(fd, data, sz);
- if (buf != NULL) {
- error = image_chunk_copyin(blk, buf, sz,
- data, fd);
- image_file_unmap(buf, sz);
- } else
- error = errno;
- blk += sz / secsz;
- bytesize += sz;
+ while (data < pos) {
+ sz = (pos - data > (off_t)iosz)
+ ? iosz : (size_t)(pos - data);
+
+ buf = image_file_map(fd, data, sz);
+ if (buf != NULL) {
+ error = image_chunk_copyin(blk, buf,
+ sz, data, fd);
+ image_file_unmap(buf, sz);
+ } else
+ error = errno;
+
+ blk += sz / secsz;
+ bytesize += sz;
+ data += sz;
+ }
cur = hole;
} else {
/*
From owner-svn-src-user@FreeBSD.ORG Wed Oct 1 16:47:14 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id D92C6F72;
Wed, 1 Oct 2014 16:47:14 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id C66202C9;
Wed, 1 Oct 2014 16:47:14 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s91GlECg085124;
Wed, 1 Oct 2014 16:47:14 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s91GlE1C085123;
Wed, 1 Oct 2014 16:47:14 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201410011647.s91GlE1C085123@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Wed, 1 Oct 2014 16:47:14 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272374 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 16:47:14 -0000
Author: marcel
Date: Wed Oct 1 16:47:14 2014
New Revision: 272374
URL: https://svnweb.freebsd.org/changeset/base/272374
Log:
Make sure to widen secsz from size_t to off_t when using it as a mask.
On 32-bit machines this would other wise mask off the upper 32-bits
of the 64-bit file offset.
Modified:
user/marcel/mkimg/image.c
Modified: user/marcel/mkimg/image.c
==============================================================================
--- user/marcel/mkimg/image.c Wed Oct 1 16:18:59 2014 (r272373)
+++ user/marcel/mkimg/image.c Wed Oct 1 16:47:14 2014 (r272374)
@@ -418,7 +418,7 @@ image_copyin_mapped(lba_t blk, int fd, u
if (cur == hole && data > hole) {
hole = pos;
- pos = data & ~(secsz - 1);
+ pos = data & ~((uint64_t)secsz - 1);
blk += (pos - hole) / secsz;
error = image_chunk_skipto(blk);
@@ -427,7 +427,7 @@ image_copyin_mapped(lba_t blk, int fd, u
cur = data;
} else if (cur == data && hole > data) {
data = pos;
- pos = (hole + secsz - 1) & ~(secsz - 1);
+ pos = (hole + secsz - 1) & ~((uint64_t)secsz - 1);
while (data < pos) {
sz = (pos - data > (off_t)iosz)
From owner-svn-src-user@FreeBSD.ORG Wed Oct 1 16:48:55 2014
Return-Path: <owner-svn-src-user@FreeBSD.ORG>
Delivered-To: svn-src-user@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by hub.freebsd.org (Postfix) with ESMTPS id 94E2E102;
Wed, 1 Oct 2014 16:48:55 +0000 (UTC)
Received: from svn.freebsd.org (svn.freebsd.org
[IPv6:2001:1900:2254:2068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id 820312E1;
Wed, 1 Oct 2014 16:48:55 +0000 (UTC)
Received: from svn.freebsd.org ([127.0.1.70])
by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id s91GmtGa085372;
Wed, 1 Oct 2014 16:48:55 GMT (envelope-from marcel@FreeBSD.org)
Received: (from marcel@localhost)
by svn.freebsd.org (8.14.9/8.14.9/Submit) id s91Gmtsj085371;
Wed, 1 Oct 2014 16:48:55 GMT (envelope-from marcel@FreeBSD.org)
Message-Id: <201410011648.s91Gmtsj085371@svn.freebsd.org>
X-Authentication-Warning: svn.freebsd.org: marcel set sender to
marcel@FreeBSD.org using -f
From: Marcel Moolenaar <marcel@FreeBSD.org>
Date: Wed, 1 Oct 2014 16:48:55 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-user@freebsd.org
Subject: svn commit: r272375 - user/marcel/mkimg
X-SVN-Group: user
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-user@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: "SVN commit messages for the experimental " user"
src tree" <svn-src-user.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-user/>
List-Post: <mailto:svn-src-user@freebsd.org>
List-Help: <mailto:svn-src-user-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/svn-src-user>,
<mailto:svn-src-user-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Oct 2014 16:48:55 -0000
Author: marcel
Date: Wed Oct 1 16:48:54 2014
New Revision: 272375
URL: https://svnweb.freebsd.org/changeset/base/272375
Log:
Make sure the cookies are constants of appropriate width.
Modified:
user/marcel/mkimg/vhd.c
Modified: user/marcel/mkimg/vhd.c
==============================================================================
--- user/marcel/mkimg/vhd.c Wed Oct 1 16:47:14 2014 (r272374)
+++ user/marcel/mkimg/vhd.c Wed Oct 1 16:48:54 2014 (r272375)
@@ -64,7 +64,7 @@ __FBSDID("$FreeBSD$");
struct vhd_footer {
uint64_t cookie;
-#define VHD_FOOTER_COOKIE 0x636f6e6563746978
+#define VHD_FOOTER_COOKIE 0x636f6e6563746978ULL
uint32_t features;
#define VHD_FEATURES_TEMPORARY 0x01
#define VHD_FEATURES_RESERVED 0x02
@@ -236,7 +236,7 @@ vhd_resize(lba_t imgsz)
struct vhd_dyn_header {
uint64_t cookie;
-#define VHD_HEADER_COOKIE 0x6378737061727365
+#define VHD_HEADER_COOKIE 0x6378737061727365ULL
uint64_t data_offset;
uint64_t table_offset;
uint32_t version;