Fedora, sqlplus: error while loading shared libraries

My sqlplus stopped working one day with the error:

sqlplus: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directory

this is a classic shared object where the shared object cannot be found.

I don't know how it happened, but my sqlplus install had become detached. The solution is simple as root: search for the .so file in your file system. You can do this using your expert knowledge, or find:

find /usr -name 'libsqlplus.so'

Once you have this location, stick the folder which contains this library (i.e. /usr/lib/oracle/11.2/client/lib/) in a file for linker to find:

/etc/ld.so.conf.d/oracle.conf

The linker is call 'ld' and on Fedora 11 uses /etc/ld... for it's configuration files. The final step is to re-configure the dynamic linker run time bindings which is done with:

ldconfig

Building a self contained Jar with Maven on Netbeans

Sometimes I need to build a Jar for distribution. A typically ideal type of Jar is one which can be executed from the command line:

java -jar ./myApp.jar

I have in the past used One-Jar which works nicely for this purpose, but now I'm using Maven as standard, this should be able to do this for me. And do it without IDE dependent plugins.

Today I'm going to build a jar with Eclipse. The project which I'm building is dependent on another local Maven project, and they both are dependent on locally and repository managed Jars.

Once all the dependencies are in the Maven Classpath Container, it is a matter of editing the pom.xml file which dictates the build process.

to this I added a plugin - in the section of the node:

  <artifactid>maven-assembly-plugin</artifactid>
  <configuration>
    <descriptorRefs>
      <descriptorRef>jar-with-dependencies</descriptorRef>
    </descriptorRefs>
    <archive>
      <manifest>
        <mainClass>net.crisppacket.myApp</mainClass>
      </manifest>
    </archive>
  </configuration>

so now, issuing (from the command line):

mvn assembly:assembly

will create a Jar for you. I got this far with the assistance of the excellent KSB's Maven Notes.

Now, in Netbeans, you can add a Custom Goal. Right click on the project you want to build, select 'Custom->Goals...' and then in the Goal field type:

assembly:assembly

OK. Now you can select this Goal (using the same mouse navigation) and it will build you a Jar. Hopefully.

reserved words in MVC

There is no other words to express my disappointment with object relational mapping framworks than: woeful

I have played with three MVC frameworks (Catalyst/perl, rails/ruby, grails/groovy) and by one metric, Rails is pittyful. That metric is narrow but significant (in my opinion) and is described below.

The object-relational mapping frameworks in question are:

Catalyst/DBIC
Rails/ActiveRecord
Grails/Hibernate

My issue is that all these frameworks have implicit 'reserved words'. These typically throw an obscure error if you use in your Models because they collide with either framework or DB reserved words.


Let me draw up a table:

Framework	# reserved words	notes
DBIC	0 + db dependent	DB reserved words are not permitted.
Hibernate	0 + db dependent	DB reserved words are not permitted.
ActiveRecord	7 + db dependent	based on Reserved words in Rails blog post from 2007

So, there is an important uncertainty here: DB reserved words.

Irrespective of db reserved words, the frameworks differed on DB reserved words - so your models aren't portable across DBs. Am I alone in feeling this is a real lost opportunity for relational mapping frameworks?

I have still not seen a reason why these 'reserved words' restrictions exist
All (?) decent DBs will (I think) allow you arbetary table names with by quoting db names...
From an architecture perspective, having reserved words for something which is supposed by be abstract is just perverse.

I await the end of 'reserved words'.

UPDATE: This post is poorly researched. I discovered an additional resource in a blog post by Daniel Butler. However, my experience still points to Catalyst being an good choice of MVC framework when dealing with large legacy databases.

postgresql search_path with ruby on rails

I studied (amongh other things) electrical engineering at Univsrity. One of the lectures said that you only need one forumal to master the subject: V=IR. However, the lecture warned, you have to know that forumla inside out.

Indeed, you could argue that V=IR is a simplification of Maxwells law, so V=IR not only covers you for EE, but a large chunk of physics (astro, plasma for example) as well.

Some times, I think MVC frameworks are similar.

Anyways... .I'm finding myself using Rails today. I've used it before with some success but didn't like the way it wanted to dominate the DB. My current thinking is that this approach is alright unless you want to integrate a legacy DB.

I'm integrating a legacy DB. It's in PostgreSQL and it uses schema namespaces. Namespaces are really worth the effort. They look great if you use PgAdminIII.

I struggled for a hour or so with my namespaces and then I discovered that I could include a search_path in ./config/database.yml. Now everything is working well again.

discovering namespaces in postgresql

I've been working on a project with a friend. We're using PostgreSQL for our db...

I thoughtfully prepared the DB locally. I then did a complete dump of it and gave it to my chum...

However, I hadn't anticipated a few things:

1. I didn't tell my friend the username I had used when creating the database. The PostgreSQL dump had helpfully included a bunch of 'ALTER TABLE, OWNER TO' commands. If you haven't anticipated this - and the user doesn't exist - you will end up with a bunch of problems when ingesting the dump.
   Proposed solutions:
   
   
   1. Create a user with the name recorded in the dump file before ingestion
   2. when doing a pg_dump, add the --no-owner flag.
      
   
   
   
2.  I didn't tell my friend that I have used schema in the db. Once they had got over the first problem, they were greeted with an apprently empty database! If you don't set your search_path to add the relevant schema names, \dt will tell you 'no relations found'
   Proposed solutions:
   
   
   
   1. Tell my friend the names of the schema that i've used - maybe even in the form of a SET search_path=newschema,"$user",public; command.
   2. Let my friend grep the dump file for 'search_path'.
      
   3. Let my friend figure out the relevant names of the schema by having them do a SELECT * FROM pg_namespaces;
      
   
   
   

I think I could improve on the namespace discovery query, possibably by just suggesting that my friend does a SELECT * FROM pg_tables; which seems to reveal more useful information.

UPDATE: after some more playing around, I've discovered that I'll will save myself alot of time by editing the dump file to remove all the 'alter to' commands. One it is ingested, the resulting db requires alot of effort to manipulate all the permissions.

maven, fedora.

I've been tipped that Maven is worth the learning curve.

I had a go, on Fedora 11. I can't wait for Fedora 12.

My first tip, is download and install locally: maven 2.2.1.
Install this into /usr/local/maven...

Use that in your Eclipse/NetBeans.

I'm on holiday for a few days... but I will post on how to get Eclipse and NetBeans building a distributable (all dependencies included) jar from the IDE when I return.

In conclusion: Maven is worth the learning curve.

UPDATE: After reflecting on actual use of Maven, I can add the following:

1. Because it manages the build process, the IDE does not manage the build process. This means that you reduce your dependence on a given IDE. The project I'm working on is now developed by NetBeans and Eclipse folks. At the same time.
2. Because it manages the build process, you are not dependent on IDE specific tools to build your distribution Jar. I like One-Jar but it is an extra dependency.
3. By allowing you to collect all dependent code in your own repository, Maven enables a new level of technical management to be applied across a project, group, division etc.
   

vlc, jackd no audio

I've been playing with Jack.

I'm feeling like I should always have it running since it works pretty well on the machine i'm using (xubuntu 9.10) and I'm going to want to use some audio production applications in future.

But, today I wanted to play a movie in VLC.

The sound didn't work and patchage didn't show 'vlc'. A brief search and I discovered that there is a jack plugin for vlc.

apt-get install vlc-plugin-jack

fixes that... Then within VLC, you need to switch to jack output:

tools->preferences->audio output label 'jack audio output'

then, I had to switch to 'show settings' all.
Open the audio->output modules->jack tree
and check the 'automatically connect to writable clients'

This made it work. I expect I'm going to experience some problems when I have more than one 'writable client', but I'll burn that bridge when I get to it.

basic http auth with Shiro in Grails

A quick note:

I found a blog nice simple recipe for basic http auth using Shiro in Grails. However, since this was published, JSecurity has been renamed Apache Shiro and so some minor changes need to be made at the end of the blog.

I posted them as a comment to that page but will repeat them here for good measure:

the last additions to Config.groovy when using Shiro should be:
security.shiro.filter.config = """
[filters]
authcBasic = org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter
authcBasic.applicationName = Demo app name
 
[urls]
/rest/** = authcBasic
"""

Grails, Shiro, addToPermissions() error

I've been playing with Grails with an eye to using it on a new project.

Apache Shiro looks like a suitable security framework for me so I thought I'd use it right from the off. There is even a plugin for Grails to simplify installation.

I discovered that you still have to create your own shiro.jar from the shiro source - but that was relatively straight forward....

[... once I had downloaded a new version of Maven (Fedora 11 packaged version wasn't recent enough)....]

So, once I had installed shiro.jar, and the plugin, I decided to run:
'grails create-db-realm' and
'grails create-authcontroller'
and added bootstrap code according to the plugin advice

ASIDE: for the purposes of just playing around. A better choice for 'playing around' is to do:
'grails quick-start'

If you followed the same route as me, you may have experienced:

Caused by: groovy.lang.MissingMethodException: No signature of method: static groovy.lang.MissingMethodException.addToPermissions() is applicable for argument types: (java.lang.String) values: [*:*]

The fix I arrived at is to add the line:
static hasMany = [ roles: ShiroRole, permissions: String ]
to ShiroUser.groovy so it becomes:

class ShiroUser {
String username
String passwordHash

static hasMany = [ roles: ShiroRole, permissions: String ]

static constraints = {
username(nullable: false, blank: false)
}
}

This got me past this error, however since I feel that I still don't properly 'get it', I expect more issues (and possibably blog posts) in the future!

karmic ubuntu on hp mini 1000

I've spent the last couple of days trying to get Karmic (beta) on a HP Mini 1000 (1030NR).

This is simple enough with the help of unetbootin. My problems started when I tried to get wireless working with the proprietary drivers. I have never enjoyed satisfaction with proprietary drivers and this occation was the same.

In the 'Live mode' the installed (jockey) happily identified the hardware and swiftly installed ssb broadcom drivers. Once installed jockey didn't seem to be `all there'. I couln't install the drivers.

What finally worked was to go to synaptic package manager and search for 'broadcom' and force a reinstall of the relevant drivers:

bcmwl-kernel-source and bcmwl-modaliases

After this, it all worked perfectly.

Tomcat6 on Fedora 11

I'm contemplating using Tomcat for my next project. I'm going to be developing on my Fedora 11 desktop.

I am new to this whole app server business and have spent this morning getting everything working. It shouldn't have taken that long, but it did.

My aim was to have some thing running on localhost which I could connect to and click about in. Compressed down in to a couple of steps I needed to do:

1. yum install tomcat6 tomcat6-webapps tomcat6-admin-webapps
2. edit /etc/tomcat6/tomcat-users.xml
   add the line:

   <user name="manager" password="tomcat_manager_secret" roles="manager"/>

3. service tomcat6 restart

tomcat6 is a basic package. to get something which you can click about with when you point your browser at localhost:8080, you need tomcat6-webapps. To get the manager component, you need tomcat6-admin-webapps.

This sounds obvious enough now that I've done it.

plotting for cairo

I've been using cairo quite a bit recently. Inevitably, i've found myself wanting to do some graphs. Instead of doing an exhaustive search on the internet, I quickly decided that there was nothing suitable out there and I should do my plots manually for my application.

Some weeks (months?) on, I've noticed that this is not a sensible approach. I've spent a little more time on the internet looking for solutions, and had some helpful suggestions on email lists.

The primary candidate is:

• PLplot
• 
  

PLplot that provides an 'extcairo' driver. This driver will allow you to construct a cairo context in your app, pass it to PLplot, call PLplotting commands, and provide you with a nice graph plotted on your context.

An example is provided: http://plplot.sourcearchive.com/documentation/5.9.2/ext-cairo-test_8c-source.html

Up until now, I've been a heavy user of gnuplot. I'm currently interested in extending gnuplot to support this type of work-flow.

firefox 3.5 landscape printing orientation icon

I have recently upgraded to Firefox 3.5. I am experiencing pretty poor performance from it, but this is no different to 3.0 I was running previously.

I suspect that it is either: the extensions i'm using (adblock, noscript, cookiemonster?) or my machine setup: openSuse 11.1.

Today i'm posting on something which I have noticed: I think the 'landscape' icon in firefox printing dialogues is broken. This icon appears in 'page setup' and 'print preview' and I believe it should show the picture of a person rotated through 90deg anti-clockwise to be correct. Here is an annotated screenshot capturing the offending icon.
The same criticism could be leveled at the 'reverse landscape' icon.

UPDATE: it would appear that these icons are provided by GTK, not firefox.

creating a custom widget with pyGTK

I'm working on a desktop app for Gnome.

I thought I'd write my own widget to display the results, and since I had made rapid progress with python, I figured I'd stick with it.

After a hour or so looking around the web, I found an excellent tutorial on extending GTK.Widget with Python:
http://www.learningpython.com/2006/07/25/writing-a-custom-widget-using-pygtk/
and subsequently found a link to some demo code on the gnome website:
http://git.gnome.org/cgit/pygtk/tree/examples/gtk/widget.py

So, now I'm armed with these resources, I can get hacking!
Next post: the results!

Hamster time tracker on opensuse

I've got interested in trying to identify what I spend my time in front of my computer doing. I knocked up a python script to watch 'wnck' and 'dbus' for desktop events (dbus for screensaver) which is available on my website.

While I was thinking of alerting the gnome community to the embryonic work, I discovered the 'Hamster' applet, which apparently is aimed at fulfiling the same need as my scratchy program.

So, I visited the hamster site and cloned the git repository onto my machine. Then I went about configuring it.

For OpenSuse, this turned out to require a bunch of dependencies to be fulfilled, so I thought I'd record them here so others may benifit.

zypper install autoconf automake libtool glib2-devel libgconf2-dev intltool gconf2-devel python-devel
zypper install gtk2-devel python-gtk-devel python-gobject2-devel python-gnome-devel

I might have missed some out while going through my .history, but that should get you close.

I'm not sure how much this is an additional OpenSuse specific feature, but I needed to set $PKG_CONFIG_PATH which I set to: /usr/lib64/pkgconfig/ (i'm on 64bit).

PostgreSQL row overhead

As mentioned previously, I'm in currently planning a +1 billion row ingestion. I've begun to examine the on-disk storage requirements. Memory will be treated in a future post.

One of the properties of the dataset which I'm ingesting is that it is time stamped. Infact, there is not too much else other than timestamps. With one obvious representation, total cost of the data in a row is ~ 8+4+4+4+4 = 22 bytes.

However, what I didn't appreciate was that the overhead which PostgreSQL imposes on a row is ~34bytes! This figure has improved slightly in 8.3, but not significantly enough.

So a couple of options occur to me: use a data base with a smaller row overhead (MySQL?) or redesign the tables.

I've plumped for the latter. Basically I'm working the tables into a form where they are much squarer. Reducing the row count and increasing the amount of data per row. I'm using arrays to increase the amount of data.

This redesign has a cost, and without exact requirements from our users it's too early to tell if this decision will return to haunt me...

perl DBI vs c libpq PostgreSQL ingestion

I'm planning to ingest ~1B rows into a PostgreSQL data base.

There are some interesting storage requirements here, but in the first instance I've been working on getting ingestion throughput up.

For my trial ingestion of 5M rows, i've found libPQ to be roughly twice as fast as Perl DBI.

Some notes:

• indexing is omitted during ingestion.


• libpq included data type hints, my Perl code, using DBI, didn't


• each row ingested included a timestamp, and a few (<5) numeric values


• running multiple ingestion process improved throughput, although less than linearly (for <4 cpus). I.e. 2 CPUs ~ doubled throughput, but it tailed off after that.

making books from a pdf

I needed to produce a book - or booklet more like from a PDF file I created. I wanted to print it on a duplex printer and fold the pages together (a signature). This is a common way of producing books.

I used pdftk (apt-get install pdftk) and pdfnup (apt-get install pdfjam).

The first thing to do was split my file into individual pages using:

 pdftk ./book.pdf burst

which produced a file for each page named 'pg_dddd.pdf' where d = digit.
Assuming my book has a page number which is a multiple of 4 (56 in the case below), I used the following script to concatenate these pages together in the correct order:

NOTE: This code will over write the file 'booksig.pdf' with a file of reorganised pages.

#!/usr/bin/perl -w

use strict;

my $totalPages = 56;

my $processedPages = 0;

my $pageOrderStr = "";
while ($processedPages < $totalPages/2) {     $pageOrderStr .= sprintf ("pg_%04d.pdf pg_%04d.pdf pg_%04d.pdf pg_%04d.pdf ",      $totalPages - $processedPages,     $processedPages + 1,      $processedPages + 2,      $totalPages - $processedPages - 1);     $processedPages += 2; }  my $catCMD = "pdftk $pageOrderStr cat output booksig.pdf"; print `$catCMD`; 

this resulted in a pdf call 'booksig.pdf'.
The final step was to use pdfnup to write these pages back to back:

pdfnup --nup 2x1 booksig.pdf

Which produced a file 'booksig-2x1.pdf' which I sent to my duplex printer and then bound together.

To it printed correctly, I chose 'Flip on Short Edge' two-sided printing. Which is non-standard. Apparently.

UPDATE: Julian spotted an error which is now fixed.

Java floats and doubles

I'm attempting to write some high performance Java. There is some limited evidence that you might expect a small performance improvement using floats over doubles.

There is a good case why memory usage will be reduced.

However, why is it such a pain to use floats? I am constantly finding methods, in the Graphics2D classes, which only support one or the other (generally better support for Doubles).

I think this post sums up reasons not to use floats, and makes a good point: why have floats at all?

chains vs args

I've been using chains for endpoints for a bit. They are quite useful if I want one simple filter criteria.

Recently, I have a requirement for two or more criteria. This could be, for example: filetype=10 and project=8. I'm imagining that I could construct the required end point with chains, but how could I do this without implementing two controllers: one for ./filetype/10/project/8 and the other for /project/8/filetype/10.


In this case, I think it would be nice for the order of criteria to be arbitrary. I'm not sure how to achieve this, so instead I've used: $c->req->query_parameters and get the GET hash and then used that to construct my filter criteria.

I've seen a post, dating from 2006 I think, on this topic. I feel like there should be a pattern, or best policy which should improve on my crude solution. I look forward to it.

Gmane save the day again

I had a problem where I wanted to perform authorization without authentication. After alot of scratching of my head, Gmane came to the rescue:
http://article.gmane.org/gmane.comp.web.catalyst.general/17180

I really like Gmane, there is a handy search plugin for it as well:
http://article.gmane.org/gmane.discuss/8737

many_to_many torture

I have just spent the best part of the last week wrestling with many_to_many configuration with DBIx in Catalyst.

My problem manifested it self as an error when calling $newApp->add_to_directory($directories);

DBIx seeming was unable to do the sensible thing and put the primary key from my apps table into the join table.

I also got errors like:
ManyToMany::__ANON__(): No such column

Now, after a week, I have discovered that if I update_or_insert my $newApp (which I made with $c->model('Proj::App')->new({}); first, then add_to_$rel works.

This wasn't obvious to me in the documentation, but hopefully you have stumgled across this post as you search for help with the same problem.