perl DBI vs c libpq PostgreSQL ingestion

I'm planning to ingest ~1B rows into a PostgreSQL data base.

There are some interesting storage requirements here, but in the first instance I've been working on getting ingestion throughput up.

For my trial ingestion of 5M rows, i've found libPQ to be roughly twice as fast as Perl DBI.

Some notes:

• indexing is omitted during ingestion.


• libpq included data type hints, my Perl code, using DBI, didn't


• each row ingested included a timestamp, and a few (<5) numeric values


• running multiple ingestion process improved throughput, although less than linearly (for <4 cpus). I.e. 2 CPUs ~ doubled throughput, but it tailed off after that.

making books from a pdf

I needed to produce a book - or booklet more like from a PDF file I created. I wanted to print it on a duplex printer and fold the pages together (a signature). This is a common way of producing books.

I used pdftk (apt-get install pdftk) and pdfnup (apt-get install pdfjam).

The first thing to do was split my file into individual pages using:

 pdftk ./book.pdf burst

which produced a file for each page named 'pg_dddd.pdf' where d = digit.
Assuming my book has a page number which is a multiple of 4 (56 in the case below), I used the following script to concatenate these pages together in the correct order:

NOTE: This code will over write the file 'booksig.pdf' with a file of reorganised pages.

#!/usr/bin/perl -w

use strict;

my $totalPages = 56;

my $processedPages = 0;

my $pageOrderStr = "";
while ($processedPages < $totalPages/2) {     $pageOrderStr .= sprintf ("pg_%04d.pdf pg_%04d.pdf pg_%04d.pdf pg_%04d.pdf ",      $totalPages - $processedPages,     $processedPages + 1,      $processedPages + 2,      $totalPages - $processedPages - 1);     $processedPages += 2; }  my $catCMD = "pdftk $pageOrderStr cat output booksig.pdf"; print `$catCMD`; 

this resulted in a pdf call 'booksig.pdf'.
The final step was to use pdfnup to write these pages back to back:

pdfnup --nup 2x1 booksig.pdf

Which produced a file 'booksig-2x1.pdf' which I sent to my duplex printer and then bound together.

To it printed correctly, I chose 'Flip on Short Edge' two-sided printing. Which is non-standard. Apparently.

UPDATE: Julian spotted an error which is now fixed.

Java floats and doubles

I'm attempting to write some high performance Java. There is some limited evidence that you might expect a small performance improvement using floats over doubles.

There is a good case why memory usage will be reduced.

However, why is it such a pain to use floats? I am constantly finding methods, in the Graphics2D classes, which only support one or the other (generally better support for Doubles).

I think this post sums up reasons not to use floats, and makes a good point: why have floats at all?

chains vs args

I've been using chains for endpoints for a bit. They are quite useful if I want one simple filter criteria.

Recently, I have a requirement for two or more criteria. This could be, for example: filetype=10 and project=8. I'm imagining that I could construct the required end point with chains, but how could I do this without implementing two controllers: one for ./filetype/10/project/8 and the other for /project/8/filetype/10.


In this case, I think it would be nice for the order of criteria to be arbitrary. I'm not sure how to achieve this, so instead I've used: $c->req->query_parameters and get the GET hash and then used that to construct my filter criteria.

I've seen a post, dating from 2006 I think, on this topic. I feel like there should be a pattern, or best policy which should improve on my crude solution. I look forward to it.

Gmane save the day again

I had a problem where I wanted to perform authorization without authentication. After alot of scratching of my head, Gmane came to the rescue:
http://article.gmane.org/gmane.comp.web.catalyst.general/17180

I really like Gmane, there is a handy search plugin for it as well:
http://article.gmane.org/gmane.discuss/8737

many_to_many torture

I have just spent the best part of the last week wrestling with many_to_many configuration with DBIx in Catalyst.

My problem manifested it self as an error when calling $newApp->add_to_directory($directories);

DBIx seeming was unable to do the sensible thing and put the primary key from my apps table into the join table.

I also got errors like:
ManyToMany::__ANON__(): No such column

Now, after a week, I have discovered that if I update_or_insert my $newApp (which I made with $c->model('Proj::App')->new({}); first, then add_to_$rel works.

This wasn't obvious to me in the documentation, but hopefully you have stumgled across this post as you search for help with the same problem.

Adding body events to a single catalyst page

I wanted to include some javascript on a single page which was identical to the rest of the site, except it had 'onLoad()' and 'onResize()' events in the page body.

To do this, I created a new ./lib/MyApp/View/NewView.pm, which was identical to ./lib/MyApp/View/HTML.pm except it pointed to a different wrapper: 'site/jsPageWrapper'.

My 'site/jsPage' was identical to 'site/wrapper' except it called the WRAPPER 'site/jsPageHTML'

In turn, 'site/jsPageHTML' was identical to 'site/html' but with the relevant event handlers on the body tags.

In my controller, I needed to forward to my new view: $c->forward('MyApp::View::NewView').

and that was that.