Date: Sun, 07 Jan 2007 22:15:05 +0100 From: Ivan Voras <ivoras@fer.hr> To: freebsd-questions@freebsd.org Subject: Re: Tuning PostgreSQL for bulk imports Message-ID: <enrnt5$9h6$1@sea.gmane.org> In-Reply-To: <200701071301.27423.kirk@strauser.com> References: <200701071301.27423.kirk@strauser.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB3C3CDB3E1E0BB55F6F7B7E5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Kirk Strauser wrote: > I have an hourly job that converts our legacy Foxpro database into > PostgreSQL tables so that our web applications, etc. can run reports of= f > the data in a reasonable amount of time. Believe it or not, this has b= een > running perfectly in production for over a year. The only problem I'd > still like to solve is that loading the data pegs the filesystem at 100= % > for many minutes at a time. There's a dedicated mailing list for PostgreSQL performance: pgsql-performance/at/postgresql.org, which can give you really good advice, but here's some tips: - have you tuned pgsql away from the (very conservative) defaults? increased shared_buffers, effective_cache_size, temp_buffers and work_mem= ? - What might help you is to keep the WAL (write-ahead-log, i.e. journal) files on a completely separate (and fast) drive from the rest of the database, to allow parallelism and speed. For best results, format it with 32k blocks/8k fragments. - If you don't specifically need the atomicity of transactions, you might divide your import into many small transactions, for example one for every 100,000 rows instead of doing 8 million at once. If you want to be somewhat adventurous (but still within data safety limits), you can try fiddling with increasing wal_buffers, commit_delay and checkpoint_timeout. Also, what version of PostgreSQL are you using? As a general rule, the newer the version, the faster it is. This is especially true if you're using 7.x - go to 8.1.5 immediately (but don't use 8.2 until 8.2.1 gets out). --------------enigB3C3CDB3E1E0BB55F6F7B7E5 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFoWLjldnAQVacBcgRAnSgAJ9/TvTR4yP4vY8+LZrgfiprLHj6GACgztWr G8ZBbUw0HnsHZot3lz6v6dg= =96dd -----END PGP SIGNATURE----- --------------enigB3C3CDB3E1E0BB55F6F7B7E5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?enrnt5$9h6$1>