Crisp Packet Data Centre

routing real-time audio to the internet

2011-10-20T20:36:00.000-07:00

I've been working on a server-side audio tool. My app works well on the desktop - it is written in Java and the audio from SourceDataLine just works. So, I moved the app to a remove server.

Pulseaudio was installed on the remote server, and I thought it should be pretty easy to route the output back over the internet to myself.

In the end it was:

parec -r | oggenc -r -o - - | ./sighttpd -f ./oggout.conf

I called my java code with 'padsp', I'm not sure that is necessary but it probably is. This method avoids using Gstreamer (that should be a good solution if you want to scale: It can plug your audio into icecast/shoutcast.)

SourceDataLine provided the audio stream in my Java app.

NOTE: I spent a whole bunch of time playing with a file output pipe from pulse audio. The excellent support from pulse audio mailing list and irc channel steered me away from that.

understanding the decision copyright in EU

2011-09-10T22:15:00.000-07:00

If you are struggling to understand the sense of the copyright extension granted by the EU, I invite you to exercise the following idea against the decision:

The tradition of dead generations hang like a nightmare in the minds of the living.

In this case, Cliff et al have conjured a spectre that law makers have felt confident cowering to. The instantaneous value of the copyright in question has become crippling to law makers, trapped in the headlights of old intellectual capital. The alternative, to welcome a continuous introduction of content to the chaos of the public domain ecosystem, lost this round.

It doesn't take a great deal of imagination to think that: as Marx provides a compelling critique of modern political economy, Marx may provide a compelling critique of modern political economy.

For the record, IMHO Cliff's music is utterly worthless. And (IMHO) people that enjoy his music are idiots. And that goes for every other artist that has become so alienated from their are that they need lawyers to provide critical appraisal of their (past) actions.

sighttpd ogg only

2011-07-18T08:47:00.000-07:00

sighttpd is an excellent light weight streaming tool. Version 1.2 is the current master from git: https://github.com/kfish/sighttpd

One issue I had with this version is that it has a run-time dependency on shcodecs-record. The symptoms of noting having shcodecs-record available on the path is that sighttpd fails with:

execvp() failed: No such file or directory

I didn't need the stuff that shcodecs provides (MPEG-4 and H.264 encoding and decoding on SH-Mobile processor series.) To remove this dependency, a cheap hack was employed: comment out ./src/main.c:165

// shrecord_run();

Then, recompile. I have a post on making a static sighttpd if you're interested in that.

UPDATE:
The shrecord dependency has been removed in the recent git master, so get that and ignore this.

static build of sighttpd

2011-07-10T14:23:00.000-07:00

sighttpd is a handy lightweight tool for streaming media over the internet. It is a lightweight alternative to icecast/shoutcast.

I need a static build of this tool, so I followed the instructions and added LDFLAGS="-all-static" to the make command. The compete build chain was:

   autoreconf -vif
   ./configure
   make LDFLAGS="-all-static"

scp ./src/sighttpd richard@myserver:tools/

NOTE: you'll need to have shcodecs-record is in search path.

one-liner to get useful info on OpenNebula VMs

2011-06-26T12:43:00.000-07:00

I've been playing with OpenNebula. I've got a few dozen VMs which are being provided with context dynamically. It would be handy to list all the machines MAC and IP addresses, along with their names. For this purpose, I use the following one-liner:

 for ID in `sqlite3 /var/lib/one/one.db 'select oid from vm_pool where state = 3 order by name;'` ; do onevm show $ID | grep "NAME\|IP\|MAC" | grep -v VIRTUAL | grep -v ' : ' | sed 's/REQUIREMENTS=/  /'; done

koan error on Scientific Linux 6

2011-05-03T08:46:00.000-07:00

I'm doing some virtualisation stuff, using cobbler and koan. On a new install of Scientific Linux 6 (as a proxy for RHEL6) I got the error:

...

  if ret == -1: raise libvirtError ('virConnectGetVersion() failed', conn=self)

it seems that the package: qemu-kvm was not installed. So, to fix:

yum install qemu-kvm

resetting password for user in authlogic

2011-03-10T23:00:00.000-08:00

I needed to quickly reset a user's password in an Rails application that used authlogic. I didn't have the nice 'request reminder' stuff set up, so I used the console instead:

././script/console production

...

>> forgetfulUser = User.find(:first, :conditions => {:login => 'richard'})

...

...

>> forgetfulUser.password = 'newPassw0rd'

>> forgetfulUser.password_confirmation = 'newPassw0rd'

>> forgetfulUser.save

and it was done.

encoding files to watch on Samsung TV

2011-02-17T16:44:00.000-08:00

I have access to a nice Samsung TV that supports media on USB drives.

It has a bunch of codecs including x264 and MP3. My Ubuntu has some codecs on it so I needed to encode my media files into a suitable format to play them.

I received the error:

Error while opening encoder for output stream #0.0 - maybe incorrect parameters such as bit_rate, rate, width or height

And finally arrived at the following:

ffmpeg -i ./test.flv -acodec libmp3lame -vcodec libx264 -vpre medium test.avi

You can find helpful details on encoding on the blog:
http://rob.opendot.cl/index.php/useful-stuff/ffmpeg-x264-encoding-guide/
including a handy list of encoding quality vs speed presets.

Update

It may be worth running ffmpeg -i test.

If either of the streams are mp3 or h264 then you can save time by passing 'copy' as the -acodec or -vcodec respectivly.

Drupal pretty URLs on Ubuntu

2011-01-11T13:30:00.000-08:00

Ubuntu helpfully provides Drupal in its repositories. This installed fine for me but during install.php (to configure drupal) pretty urls wouldn't work.

To fix this, I had to edit

/etc/drupal/6/htaccess

find the line at the bottom:

#  RewriteBase /drupal

remove the '#' and add a '6' onto the end to give:

RewriteBase /drupal6

getting workspace with gnome visal effects

2010-12-29T08:48:00.000-08:00

I maintain linux-desktop-audit software. My Ubuntu machine recently allowed me to enable 'visual effects' to make nice looking transitions between workspaces.

However, this broke linux-desktop-audit. So, instead of listening to 'active_workspace_changed' the code should be listening to 'viewports_changed'. Once this event fires, to get the current workspace number I use the following code:

def viewport_change (self, screen):
        self.current_workspace = screen.get_active_workspace()
        v_x = self.current_workspace.get_viewport_x()
        v_y = self.current_workspace.get_viewport_y()
        v_w = screen.get_width()
        v_h = screen.get_height()
        v_r = self.current_workspace.get_width()
        v_c = self.current_workspace.get_height()
        total_cols = v_r/v_w
        total_rows = v_c/v_h
        desk_col = v_x/v_w
        desk_row = v_y/v_h
        ws_num = desk_col + (desk_row * total_cols)

NUnit-console-runner, FileLoadException, access is denied

2010-11-23T02:23:00.000-08:00

I was using NUnit, and careless stuffed it into svn from my Linux box.
When I checked it out from svn onto my Windows environment I got this error when trying to run: ./nunit-console.exe:

Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'nunit-console-runner, Version=2.4.7.0, Culture=neutral, PublicKeyToken=96d09a1eb7f44a77' or one of its dependencies. Access is denied.
File name: 'nunit-console-runner, Version=2.4.7.0, Culture=neutral, PublicKeyToken=96d09a1eb7f44a77' ---> System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

at NUnit.ConsoleRunner.Class1.Main(String[] args)

I had created a problem for myself because various dll's didn't have 'execute' flag enabled.

To fix this in a brutal way, from the cmd.com, I used:

cacls nunit.core.dll /p /e Everyone:F

handy java property to dump http traffic.

2010-08-06T03:47:00.001-07:00

I find myself needing this command and unable to remember it or find it on Google easily. So I'm going to make a note of it here to save my precious memory for other more important information:

-Dcom.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.dump=true 

Right, now I can try and remember where I live...

svn Id keyword in Perl

2010-08-03T02:39:00.000-07:00

My Perl program wants to report it's SVN Id keyword. Setting this up is simple enough if you follow one of the many excellent examples on the web. However I found that it was beyond me to get the value of my Id printed out by my code.
The problem is that the completed SVN Id keyword contains @ and $ symbols.

So my code resorts to greping it's self to get the value from the source code.

# $Id$

my $id_str = `grep -m1 '\$Id:' $0`;

print $id_str;

It feels like a crime against programming, but it works - and provided that there is an $Id$ before the grep command, it should work reliabably.

Exit code idiom in Java

2010-07-29T11:55:00.000-07:00

I have been looking into employing C 'enum' like exit codes in Java. For example, if the application you are writing will be called by a script so it is handy for the script to know when there is an exit that is OK, and when it is a FAIL.

I looked to the web for inspiration, and couldn't find anything Java specific, so, I have arrived at the following:

class MyClass {

  private static enum EXIT {
        OK(0), BAD_COMMANDLINE(1), INPUT_ERROR(10);
       
        EXIT (int code) {
            this.code = code;
        }
        private int code;
        public int getCode() {
            return code;
        }
    }

   public static void main (String[] args) {

        if (args.length() < 1) {

            System.exit(EXIT.BAD_COMMANDLINE.getCode());

}

         System.exit(EXIT.OK.getCode());

}

}

backing up mysql database without mysql root password

2010-07-09T06:28:00.001-07:00

This is a quickie, I needed to back up a mysql database without knowing the MySQL root user password.

I had sudo, so the solution was:

sudo /etc/init.d/mysql stop

sudo mysqld_safe --skip-grant-tables &

mysqldump --all-databases | gzip > /tmp/alldb.gz

sudo /etc/init.d/mysql stop

sudo /etc/init.d/mysql start

there, I told you it was a quickie.

associationtypemismatch expected got array

2010-07-07T11:43:00.000-07:00

I'm building an app a bit at a time. Sufficient time elapses between development effort that I forget some basic stuff.

Something that I haven't forgotten is how often I come across this error when I've made some changes to my models and edit forms:

ActiveRecord::AssociationTypeMismatch (MyModelName(#-613518908) expected, got Array(#-608606448)):

This typically means that I've forgotten:

accepts_nested_attributes_for 

in my model.

JQuery automatic linking

2010-06-21T13:23:00.000-07:00

I have some values on my webpage, they are URLs and I have wrapped them in the form:

<span class="url">google.com</span>

<span class="url">http://lwn.net</span>

What I would like is to reneder anything that is of class 'url' as a link. And I would also like to stick a http:// infront of links that are currently missing them.

For this I use JQuery (I should get around to doing something sensible in the database as well but not just now).

So the JQuery looks like:

  $(".url:not(:contains('http://'))").prepend("http://");
  $(".url").each(function() {
    $(this) .replaceWith("" + $(this).text() + "");
   });

That's it.

building kernel for joggler

2010-04-13T01:21:00.001-07:00

The excellent Playing the Joggler blog contains all the information you need to build a 2.6.33.2 kernel for your joggler. I thought I would collect together what I did so interested others can complete to process too:

I cloned the git 'baseline' branch of
http://gitorious.org/mer-meego/mer-meego-kernel
then applied the patches.
[the 'joggler' branch of this repo now has the patches applied - so cloning that will avoid adding the patches].
The patches can be found here:
http://adqmisc.googlecode.com/svn/trunk/joggler/

I then took the .config file from
http://jogglerhacks.blogspot.com/2010/04/kernel-config-for-2633-for-joggler.html

I did:
1. make
2. make modules
3. make modules_install INSTALL_MOD_DIR=./initfs (I think)
4. make bzImage - (and renamed the image vmlinuz)

I took the initrd from efi-nokeyboard.tar.gz and put the modules (that I built with line 3 above) into a new initrd. Instructions for rebuilding a initrd can also be found on the excellent Playing the Joggler blog.

There is a great wiki that contains information about running alternative OS's on the Joggler - and it has a page on building the kernel which is worth watching.

rails ferret in production

2010-03-27T01:32:00.000-07:00

I've had a few problems as i've moved from dev to production. My task has been complicated by not always maintaining my migrations. However the problem I've had recently has been with ferret not working.

When I've started './console/script procution' and try and rebuild indexes manually I get:

Mymodel1.rebuild_index()
=> false

It turns out that the problem was that ferret has an expectation that in production, there will be a ferret server running. This is asserted in the file:

./config/ferret_server.yml

I commented out the production server and it works.

gnuplot point types

2010-03-26T11:23:00.000-07:00

I was using gnuplot today and wanted a list of the different point types which are available so I could choose a suitable one.

After visiting the internet, I couldn't find a list so I made my own:

echo plot `perl -e 'for ($i = 0; $i < 30; $i++) {print " \"<echo $i 1\" u 1:2 pt $i ps 2 notitle, ";}'` | sed 's/,$//' | gnuplot -persist

echo set term postscript enhanced\; set output \'./points.eps\'\; plot `perl -e 'for ($i = 0; $i < 90; $i++) {print " \"<echo $i 1\" u 1:2 pt $i ps 1 notitle, ";}'` | sed 's/,$//' | gnuplot

rails won't run in production

2010-03-25T12:36:00.000-07:00

I've been working on a Rails app (2.3.5).

I'm hosting on modest hardware and chose Nginx and Thin to hopefully get the most performance out of my app.

My problem was that my app wouldn't run in production. It worked fine in development, but not in production.

There were a couple of problems:

I was using a legacy database, and like non-pluralised table names, development picked up 'ActiveRecord::Base.pluralize_table_names = false' in my config/environment.rb, but I didn't seem to propagate to production. I added in 'ActiveRecord::Base.pluralize_table_names = false' into ./config/environments/production.rb
Thin wouldn't start. Starting thin with -D allowed me to start it not daemonized, and then I could see the problem: 'Missing the Rails gem. Please `gem install -v= rails`, update your RAILS_GEM_VERSION'.
This is Thin reacting against rack version 1.1.0 apparently.
gem uninstall --version '> 1.0.1' rack fixed that.

One less thing on my todo list.

Building TOra with Oracle support on Ubuntu

2010-02-08T02:11:00.000-08:00

I'm doing some Oracle development, so I wanted to use the TOra db browser for convenience.

TOra is Free Software. There is a license conflict between TOra and the license that Oracle use on their client libraries. As a result TOra is not distributed with Oracle support: if you want to use TOra with Oracle, you must download the separate components, TOra source and Oracle client, and build TOra yourself.

This process is documented on the web, notably at pythian blog. However the instructions are for Ubuntu 9.04, and I'm using 9.10, so I thought there is an opportunity to document my experience as well.

So, on Ubuntu 9.10, I have installed Oracle XE (a resource-limited 10G installation), into it's default directory: /usr/lib/oracle/xe/

I'm also assuming that you, like me, have the three Oracle packages oracle-instantclient-basic-10.2.0.4-1.i386.rpm oracle-instantclient-devel-10.2.0.4-1.i386.rpm oracle-instantclient-sqlplus-10.2.0.4-1.i386.rpm installed in their default location: /usr/lib/oracle/10.2.0.4/client/

NOTE: I chose the '10.x' client packages - I'm not sure if this makes a difference.

It seems that the Ubuntu source for TOra have migrated to git, so when I tried to pull down the source using apt-get source I didn't have any joy. So I cloned the TOra git repository and attempted to modify the debian/rules file to include the relevant Oracle libraries. This wasn't possible - there were no configure rules in there...

So instead I configure using 'cmake'.
First I need to configure: ORACLE_HOME and LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.4/client/lib
export ORACLE_HOME=/usr/lib/oracle/10.2.0.4/client/

I also need to install the old libstdc++5 which is available from the Debian repository: http://packages.debian.org/lenny/i386/libstdc++5/download

Then a command line to configure build:

 fakeroot cmake -DORACLE_PATH_LIB=/usr/lib/oracle/10.2.0.4/client/lib -DORACLE_LIBRARY_CLNTSH=/usr/lib/oracle/10.2.0.4/client/lib/libclntsh.so.10.1 -DORACLE_LIBRARY_OCCI=/usr/lib/oracle/10.2.0.4/client/lib/libocci.so.10.1 -DORACLE_INCLUDES=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/rdbms/public 

NOTE: I'm calling in the INCLUDES from the XE install.
...and another to build:

make

And you should know what to do from here...

FormFu, DBIC, model_config and resultset.

2010-01-23T12:27:00.000-08:00

I have found myself back in Catalyst development work. This wouldn't be such a bad think if I hadn't just spent a whole day trying to fix up a form to edit a 'many_to_many' relationship which is stored in my Schema.

I've been following this post which says it should be about as simple as it sounds. i.e. not as simple as Rails, but definitely doable with finite effort. So, FormFu syntax not withstanding, I following these instructions but ended up with some peculiar error talking about 'result_source'. I tried playing with 'nested_name', 'model_config' etc... I began to think about building my Select in the controller.

Finally, I discovered my error: I had not configured Controller::HTML::FormFu which is shown nicely in this post:

http://lists.scsys.co.uk/pipermail/html-formfu/2009-March/001932.html

That is a day of my life gone, but at least I will be able to rise from sleep tomorrow knowing that I can move onto doing some new development.

UPDATE: One additional comment to the post above from 2009-March. I found that I needed to use the many_to_many relationship as the nested_name, not the has_many as stated.

Skype on a Ubuntu HP Mini 1000

2010-01-08T07:35:00.000-08:00

I have been playing with Ubuntu 9.10 on a HP Mini for a while - on and off.

I have a request to get Skype working. I struggled with this for a bit, but I couldn't get it to work with the duplex sound card: when I pluged a microphone in, the speakers stopped working for example.

Something changed - it could have been an kernel update - anyway, I can now use Skype with minimal effort - including the web-cam.

The vital thing I had to do was disable the checkbox (options->audio settings or something):

'Allow Skype to automatically adjust my mixer levels'

Then, I could twiddle the 'pulse audio' options (right click on the volume icon) to increase the microphone input level. It seemed that one problem was Skype wouldn't let me increase the microphone input level while I was letting it 'automatically adjust my mixer levels'.

Fedora, sqlplus: error while loading shared libraries

2009-12-16T04:16:00.000-08:00

My sqlplus stopped working one day with the error:

sqlplus: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directory

this is a classic shared object where the shared object cannot be found.

I don't know how it happened, but my sqlplus install had become detached. The solution is simple as root: search for the .so file in your file system. You can do this using your expert knowledge, or find:

find /usr -name 'libsqlplus.so'

Once you have this location, stick the folder which contains this library (i.e. /usr/lib/oracle/11.2/client/lib/) in a file for linker to find:

/etc/ld.so.conf.d/oracle.conf

The linker is call 'ld' and on Fedora 11 uses /etc/ld... for it's configuration files. The final step is to re-configure the dynamic linker run time bindings which is done with:

ldconfig

Building a self contained Jar with Maven on Netbeans

2009-12-16T02:16:00.000-08:00

Sometimes I need to build a Jar for distribution. A typically ideal type of Jar is one which can be executed from the command line:

java -jar ./myApp.jar

I have in the past used One-Jar which works nicely for this purpose, but now I'm using Maven as standard, this should be able to do this for me. And do it without IDE dependent plugins.

Today I'm going to build a jar with Eclipse. The project which I'm building is dependent on another local Maven project, and they both are dependent on locally and repository managed Jars.

Once all the dependencies are in the Maven Classpath Container, it is a matter of editing the pom.xml file which dictates the build process.

to this I added a plugin - in the section of the node:

  <artifactid>maven-assembly-plugin</artifactid>
  <configuration>
    <descriptorRefs>
      <descriptorRef>jar-with-dependencies</descriptorRef>
    </descriptorRefs>
    <archive>
      <manifest>
        <mainClass>net.crisppacket.myApp</mainClass>
      </manifest>
    </archive>
  </configuration>

so now, issuing (from the command line):

mvn assembly:assembly

will create a Jar for you. I got this far with the assistance of the excellent KSB's Maven Notes.

Now, in Netbeans, you can add a Custom Goal. Right click on the project you want to build, select 'Custom->Goals...' and then in the Goal field type:

assembly:assembly

OK. Now you can select this Goal (using the same mouse navigation) and it will build you a Jar. Hopefully.

reserved words in MVC

2009-12-10T12:09:00.000-08:00

There is no other words to express my disappointment with object relational mapping framworks than: woeful

I have played with three MVC frameworks (Catalyst/perl, rails/ruby, grails/groovy) and by one metric, Rails is pittyful. That metric is narrow but significant (in my opinion) and is described below.

The object-relational mapping frameworks in question are:

Catalyst/DBIC
Rails/ActiveRecord
Grails/Hibernate

My issue is that all these frameworks have implicit 'reserved words'. These typically throw an obscure error if you use in your Models because they collide with either framework or DB reserved words.

Let me draw up a table:

Framework	# reserved words	notes
DBIC	0 + db dependent	DB reserved words are not permitted.
Hibernate	0 + db dependent	DB reserved words are not permitted.
ActiveRecord	7 + db dependent	based on Reserved words in Rails blog post from 2007

So, there is an important uncertainty here: DB reserved words.

Irrespective of db reserved words, the frameworks differed on DB reserved words - so your models aren't portable across DBs. Am I alone in feeling this is a real lost opportunity for relational mapping frameworks?

I have still not seen a reason why these 'reserved words' restrictions exist
All (?) decent DBs will (I think) allow you arbetary table names with by quoting db names...
From an architecture perspective, having reserved words for something which is supposed by be abstract is just perverse.

I await the end of 'reserved words'.

UPDATE: This post is poorly researched. I discovered an additional resource in a blog post by Daniel Butler. However, my experience still points to Catalyst being an good choice of MVC framework when dealing with large legacy databases.

postgresql search_path with ruby on rails

2009-12-06T10:31:00.000-08:00

I studied (amongh other things) electrical engineering at Univsrity. One of the lectures said that you only need one forumal to master the subject: V=IR. However, the lecture warned, you have to know that forumla inside out.

Indeed, you could argue that V=IR is a simplification of Maxwells law, so V=IR not only covers you for EE, but a large chunk of physics (astro, plasma for example) as well.

Some times, I think MVC frameworks are similar.

Anyways... .I'm finding myself using Rails today. I've used it before with some success but didn't like the way it wanted to dominate the DB. My current thinking is that this approach is alright unless you want to integrate a legacy DB.

I'm integrating a legacy DB. It's in PostgreSQL and it uses schema namespaces. Namespaces are really worth the effort. They look great if you use PgAdminIII.

I struggled for a hour or so with my namespaces and then I discovered that I could include a search_path in ./config/database.yml. Now everything is working well again.

discovering namespaces in postgresql

2009-12-05T09:41:00.000-08:00

I've been working on a project with a friend. We're using PostgreSQL for our db...

I thoughtfully prepared the DB locally. I then did a complete dump of it and gave it to my chum...

However, I hadn't anticipated a few things:

I didn't tell my friend the username I had used when creating the database. The PostgreSQL dump had helpfully included a bunch of 'ALTER TABLE, OWNER TO' commands. If you haven't anticipated this - and the user doesn't exist - you will end up with a bunch of problems when ingesting the dump.
Proposed solutions:
1. Create a user with the name recorded in the dump file before ingestion
2. when doing a pg_dump, add the --no-owner flag.
I didn't tell my friend that I have used schema in the db. Once they had got over the first problem, they were greeted with an apprently empty database! If you don't set your search_path to add the relevant schema names, \dt will tell you 'no relations found'
Proposed solutions:
1. Tell my friend the names of the schema that i've used - maybe even in the form of a SET search_path=newschema,"$user",public; command.
2. Let my friend grep the dump file for 'search_path'.
3. Let my friend figure out the relevant names of the schema by having them do a SELECT * FROM pg_namespaces;

I think I could improve on the namespace discovery query, possibably by just suggesting that my friend does a SELECT * FROM pg_tables; which seems to reveal more useful information.

UPDATE: after some more playing around, I've discovered that I'll will save myself alot of time by editing the dump file to remove all the 'alter to' commands. One it is ingested, the resulting db requires alot of effort to manipulate all the permissions.

maven, fedora.

2009-11-25T12:58:00.000-08:00

I've been tipped that Maven is worth the learning curve.

I had a go, on Fedora 11. I can't wait for Fedora 12.

My first tip, is download and install locally: maven 2.2.1.
Install this into /usr/local/maven...

Use that in your Eclipse/NetBeans.

I'm on holiday for a few days... but I will post on how to get Eclipse and NetBeans building a distributable (all dependencies included) jar from the IDE when I return.

In conclusion: Maven is worth the learning curve.

UPDATE: After reflecting on actual use of Maven, I can add the following:

Because it manages the build process, the IDE does not manage the build process. This means that you reduce your dependence on a given IDE. The project I'm working on is now developed by NetBeans and Eclipse folks. At the same time.
Because it manages the build process, you are not dependent on IDE specific tools to build your distribution Jar. I like One-Jar but it is an extra dependency.
By allowing you to collect all dependent code in your own repository, Maven enables a new level of technical management to be applied across a project, group, division etc.

vlc, jackd no audio

2009-11-19T10:33:00.000-08:00

I've been playing with Jack.

I'm feeling like I should always have it running since it works pretty well on the machine i'm using (xubuntu 9.10) and I'm going to want to use some audio production applications in future.

But, today I wanted to play a movie in VLC.

The sound didn't work and patchage didn't show 'vlc'. A brief search and I discovered that there is a jack plugin for vlc.

apt-get install vlc-plugin-jack

fixes that... Then within VLC, you need to switch to jack output:

tools->preferences->audio output label 'jack audio output'

then, I had to switch to 'show settings' all.
Open the audio->output modules->jack tree
and check the 'automatically connect to writable clients'

This made it work. I expect I'm going to experience some problems when I have more than one 'writable client', but I'll burn that bridge when I get to it.

basic http auth with Shiro in Grails

2009-11-04T02:00:00.000-08:00

A quick note:

I found a blog nice simple recipe for basic http auth using Shiro in Grails. However, since this was published, JSecurity has been renamed Apache Shiro and so some minor changes need to be made at the end of the blog.

I posted them as a comment to that page but will repeat them here for good measure:

the last additions to Config.groovy when using Shiro should be:
security.shiro.filter.config = """
[filters]
authcBasic = org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter
authcBasic.applicationName = Demo app name

[urls]
/rest/** = authcBasic
"""

Grails, Shiro, addToPermissions() error

2009-11-02T04:11:00.000-08:00

I've been playing with Grails with an eye to using it on a new project.

Apache Shiro looks like a suitable security framework for me so I thought I'd use it right from the off. There is even a plugin for Grails to simplify installation.

I discovered that you still have to create your own shiro.jar from the shiro source - but that was relatively straight forward....

[... once I had downloaded a new version of Maven (Fedora 11 packaged version wasn't recent enough)....]

So, once I had installed shiro.jar, and the plugin, I decided to run:
'grails create-db-realm' and
'grails create-authcontroller'
and added bootstrap code according to the plugin advice

ASIDE: for the purposes of just playing around. A better choice for 'playing around' is to do:
'grails quick-start'

If you followed the same route as me, you may have experienced:

Caused by: groovy.lang.MissingMethodException: No signature of method: static groovy.lang.MissingMethodException.addToPermissions() is applicable for argument types: (java.lang.String) values: [*:*]

The fix I arrived at is to add the line:
static hasMany = [ roles: ShiroRole, permissions: String ]
to ShiroUser.groovy so it becomes:

class ShiroUser {
String username
String passwordHash

static hasMany = [ roles: ShiroRole, permissions: String ]

static constraints = {
username(nullable: false, blank: false)
}
}

This got me past this error, however since I feel that I still don't properly 'get it', I expect more issues (and possibably blog posts) in the future!

karmic ubuntu on hp mini 1000

2009-10-15T15:10:00.000-07:00

I've spent the last couple of days trying to get Karmic (beta) on a HP Mini 1000 (1030NR).

This is simple enough with the help of unetbootin. My problems started when I tried to get wireless working with the proprietary drivers. I have never enjoyed satisfaction with proprietary drivers and this occation was the same.

In the 'Live mode' the installed (jockey) happily identified the hardware and swiftly installed ssb broadcom drivers. Once installed jockey didn't seem to be `all there'. I couln't install the drivers.

What finally worked was to go to synaptic package manager and search for 'broadcom' and force a reinstall of the relevant drivers:

bcmwl-kernel-source and bcmwl-modaliases

After this, it all worked perfectly.

Tomcat6 on Fedora 11

2009-10-12T04:04:00.000-07:00

I'm contemplating using Tomcat for my next project. I'm going to be developing on my Fedora 11 desktop.

I am new to this whole app server business and have spent this morning getting everything working. It shouldn't have taken that long, but it did.

My aim was to have some thing running on localhost which I could connect to and click about in. Compressed down in to a couple of steps I needed to do:

yum install tomcat6 tomcat6-webapps tomcat6-admin-webapps

edit /etc/tomcat6/tomcat-users.xml
add the line:

<user name="manager" password="tomcat_manager_secret" roles="manager"/>

service tomcat6 restart

tomcat6 is a basic package. to get something which you can click about with when you point your browser at localhost:8080, you need tomcat6-webapps. To get the manager component, you need tomcat6-admin-webapps.

This sounds obvious enough now that I've done it.

plotting for cairo

2009-09-10T02:19:00.000-07:00

I've been using cairo quite a bit recently. Inevitably, i've found myself wanting to do some graphs. Instead of doing an exhaustive search on the internet, I quickly decided that there was nothing suitable out there and I should do my plots manually for my application.

Some weeks (months?) on, I've noticed that this is not a sensible approach. I've spent a little more time on the internet looking for solutions, and had some helpful suggestions on email lists.

The primary candidate is:

PLplot

PLplot that provides an 'extcairo' driver. This driver will allow you to construct a cairo context in your app, pass it to PLplot, call PLplotting commands, and provide you with a nice graph plotted on your context.

An example is provided: http://plplot.sourcearchive.com/documentation/5.9.2/ext-cairo-test_8c-source.html

Up until now, I've been a heavy user of gnuplot. I'm currently interested in extending gnuplot to support this type of work-flow.

firefox 3.5 landscape printing orientation icon

2009-08-21T02:27:00.000-07:00

I have recently upgraded to Firefox 3.5. I am experiencing pretty poor performance from it, but this is no different to 3.0 I was running previously.

I suspect that it is either: the extensions i'm using (adblock, noscript, cookiemonster?) or my machine setup: openSuse 11.1.

Today i'm posting on something which I have noticed: I think the 'landscape' icon in firefox printing dialogues is broken. This icon appears in 'page setup' and 'print preview' and I believe it should show the picture of a person rotated through 90deg anti-clockwise to be correct. Here is an annotated screenshot capturing the offending icon.
The same criticism could be leveled at the 'reverse landscape' icon.

UPDATE: it would appear that these icons are provided by GTK, not firefox.

creating a custom widget with pyGTK

2009-07-09T04:08:00.000-07:00

I'm working on a desktop app for Gnome.

I thought I'd write my own widget to display the results, and since I had made rapid progress with python, I figured I'd stick with it.

After a hour or so looking around the web, I found an excellent tutorial on extending GTK.Widget with Python:
http://www.learningpython.com/2006/07/25/writing-a-custom-widget-using-pygtk/
and subsequently found a link to some demo code on the gnome website:
http://git.gnome.org/cgit/pygtk/tree/examples/gtk/widget.py

So, now I'm armed with these resources, I can get hacking!
Next post: the results!

Hamster time tracker on opensuse

2009-06-18T04:10:00.000-07:00

I've got interested in trying to identify what I spend my time in front of my computer doing. I knocked up a python script to watch 'wnck' and 'dbus' for desktop events (dbus for screensaver) which is available on my website.

While I was thinking of alerting the gnome community to the embryonic work, I discovered the 'Hamster' applet, which apparently is aimed at fulfiling the same need as my scratchy program.

So, I visited the hamster site and cloned the git repository onto my machine. Then I went about configuring it.

For OpenSuse, this turned out to require a bunch of dependencies to be fulfilled, so I thought I'd record them here so others may benifit.


zypper install autoconf automake libtool glib2-devel libgconf2-dev intltool gconf2-devel python-devel
zypper install gtk2-devel python-gtk-devel python-gobject2-devel python-gnome-devel

I might have missed some out while going through my .history, but that should get you close.

I'm not sure how much this is an additional OpenSuse specific feature, but I needed to set $PKG_CONFIG_PATH which I set to: /usr/lib64/pkgconfig/ (i'm on 64bit).

PostgreSQL row overhead

2009-06-05T07:00:00.000-07:00

As mentioned previously, I'm in currently planning a +1 billion row ingestion. I've begun to examine the on-disk storage requirements. Memory will be treated in a future post.

One of the properties of the dataset which I'm ingesting is that it is time stamped. Infact, there is not too much else other than timestamps. With one obvious representation, total cost of the data in a row is ~ 8+4+4+4+4 = 22 bytes.

However, what I didn't appreciate was that the overhead which PostgreSQL imposes on a row is ~34bytes! This figure has improved slightly in 8.3, but not significantly enough.

So a couple of options occur to me: use a data base with a smaller row overhead (MySQL?) or redesign the tables.

I've plumped for the latter. Basically I'm working the tables into a form where they are much squarer. Reducing the row count and increasing the amount of data per row. I'm using arrays to increase the amount of data.

This redesign has a cost, and without exact requirements from our users it's too early to tell if this decision will return to haunt me...

perl DBI vs c libpq PostgreSQL ingestion

2009-06-02T05:06:00.000-07:00

I'm planning to ingest ~1B rows into a PostgreSQL data base.

There are some interesting storage requirements here, but in the first instance I've been working on getting ingestion throughput up.

For my trial ingestion of 5M rows, i've found libPQ to be roughly twice as fast as Perl DBI.

Some notes:

indexing is omitted during ingestion.

libpq included data type hints, my Perl code, using DBI, didn't

each row ingested included a timestamp, and a few (<5) numeric values

running multiple ingestion process improved throughput, although less than linearly (for <4 cpus). I.e. 2 CPUs ~ doubled throughput, but it tailed off after that.

making books from a pdf

2009-05-06T14:13:00.000-07:00

I needed to produce a book - or booklet more like from a PDF file I created. I wanted to print it on a duplex printer and fold the pages together (a signature). This is a common way of producing books.

I used pdftk (apt-get install pdftk) and pdfnup (apt-get install pdfjam).

The first thing to do was split my file into individual pages using:

 pdftk ./book.pdf burst

which produced a file for each page named 'pg_dddd.pdf' where d = digit.
Assuming my book has a page number which is a multiple of 4 (56 in the case below), I used the following script to concatenate these pages together in the correct order:

NOTE: This code will over write the file 'booksig.pdf' with a file of reorganised pages.


#!/usr/bin/perl -w

use strict;

my $totalPages = 56;

my $processedPages = 0;

my $pageOrderStr = "";
while ($processedPages < $totalPages/2) {     $pageOrderStr .= sprintf ("pg_%04d.pdf pg_%04d.pdf pg_%04d.pdf pg_%04d.pdf ",      $totalPages - $processedPages,     $processedPages + 1,      $processedPages + 2,      $totalPages - $processedPages - 1);     $processedPages += 2; }  my $catCMD = "pdftk $pageOrderStr cat output booksig.pdf"; print `$catCMD`;

this resulted in a pdf call 'booksig.pdf'.
The final step was to use pdfnup to write these pages back to back:

pdfnup --nup 2x1 booksig.pdf

Which produced a file 'booksig-2x1.pdf' which I sent to my duplex printer and then bound together.

To it printed correctly, I chose 'Flip on Short Edge' two-sided printing. Which is non-standard. Apparently.

UPDATE: Julian spotted an error which is now fixed.

Java floats and doubles

2009-05-05T13:46:00.000-07:00

I'm attempting to write some high performance Java. There is some limited evidence that you might expect a small performance improvement using floats over doubles.

There is a good case why memory usage will be reduced.

However, why is it such a pain to use floats? I am constantly finding methods, in the Graphics2D classes, which only support one or the other (generally better support for Doubles).

I think this post sums up reasons not to use floats, and makes a good point: why have floats at all?

chains vs args

2009-04-19T13:49:00.000-07:00

I've been using chains for endpoints for a bit. They are quite useful if I want one simple filter criteria.

Recently, I have a requirement for two or more criteria. This could be, for example: filetype=10 and project=8. I'm imagining that I could construct the required end point with chains, but how could I do this without implementing two controllers: one for ./filetype/10/project/8 and the other for /project/8/filetype/10.

In this case, I think it would be nice for the order of criteria to be arbitrary. I'm not sure how to achieve this, so instead I've used: $c->req->query_parameters and get the GET hash and then used that to construct my filter criteria.

I've seen a post, dating from 2006 I think, on this topic. I feel like there should be a pattern, or best policy which should improve on my crude solution. I look forward to it.

Gmane save the day again

2009-02-22T08:11:00.000-08:00

I had a problem where I wanted to perform authorization without authentication. After alot of scratching of my head, Gmane came to the rescue:
http://article.gmane.org/gmane.comp.web.catalyst.general/17180

I really like Gmane, there is a handy search plugin for it as well:
http://article.gmane.org/gmane.discuss/8737

many_to_many torture

2009-02-20T05:59:00.000-08:00

I have just spent the best part of the last week wrestling with many_to_many configuration with DBIx in Catalyst.

My problem manifested it self as an error when calling $newApp->add_to_directory($directories);

DBIx seeming was unable to do the sensible thing and put the primary key from my apps table into the join table.

I also got errors like:
ManyToMany::__ANON__(): No such column

Now, after a week, I have discovered that if I update_or_insert my $newApp (which I made with $c->model('Proj::App')->new({}); first, then add_to_$rel works.

This wasn't obvious to me in the documentation, but hopefully you have stumgled across this post as you search for help with the same problem.

Adding body events to a single catalyst page

2008-11-05T02:19:00.000-08:00

I wanted to include some javascript on a single page which was identical to the rest of the site, except it had 'onLoad()' and 'onResize()' events in the page body.

To do this, I created a new ./lib/MyApp/View/NewView.pm, which was identical to ./lib/MyApp/View/HTML.pm except it pointed to a different wrapper: 'site/jsPageWrapper'.

My 'site/jsPage' was identical to 'site/wrapper' except it called the WRAPPER 'site/jsPageHTML'

In turn, 'site/jsPageHTML' was identical to 'site/html' but with the relevant event handlers on the body tags.

In my controller, I needed to forward to my new view: $c->forward('MyApp::View::NewView').

and that was that.

2008-10-05T05:56:00.000-07:00

Thanks to mst and edenc on irc, I managed to reuse a configuration for connect_info among a number of models. And then I wrote it up for the wiki:

http://dev.catalystframework.org/wiki/gettingstarted/howtos
/ModelConfigSharing.highlight

Template toolkit rendering a hash

2008-10-02T04:50:00.001-07:00

I wanted to render a which I generated within my app; it is a calendar basically:
$calendar{$year}{$month}{$day} = daynumber; which I add stash as:
$c->stash->{calendar} = $calendar;

the code I used to get this working is as follows:

[% FOREACH year IN calendar.keys -%]
[% FOREACH month IN calendar.$year -%]
[% SET m = month.key %]
[% FOREACH day IN calendar.$year.$m -%]
[% SET d = day.key %]
values =
[% year %],
[% month.key %],
[% day.key %],
day number = [% calendar.$year.$m.$d %]

[% END %]
[% END %]
[% END %]

another day another incantation

2008-09-03T08:16:00.000-07:00

I want to use namespaces within my PostgreSQL database (schema in PostgreSQL world). However, they are not currently supported within the Catalyst scripts which will automatically build catalyst::schema automatically.

So the solution is to call Schema::Loader 'manually' with this incantation:

perl -MDBIx::Class::Schema::Loader=make_schema_at,dump_to_dir:./lib -e 'make_schema_at("AddressBook::Addresses", { debug => 1, db_schema => "addressStuff" }, [ "dbi:Pg:host=localhost;dbname=testdb", "testuser", "123" ])'

so there you have it: thanks to the irc crew for sorting this out... and they pointed out that cross schema querying will require some jiggery pokery when I get to that...

LDAP, catalyst, roles and groups

2008-08-22T03:48:00.001-07:00

After a day or so of effort, I've got LDAP authorization working for groups/roles. I'm posting my config files here, since they worked for me and someone else may stumble across them and find them useful:

my ldif file is as follows:


dn: o=example.org
objectclass: top
objectclass: organization
o: example.org

dn: cn=admin,o=example.org
objectClass: simpleSecurityObject
objectclass: organizationalRole
cn: admin
description: LDAP Administrator
userPassword: 123

dn: ou=groups,o=example.org
objectClass: organizationalUnit
ou: groups
description: generic groups branch

dn: ou=people,o=example.org
ou: people
description: All people in organisation
objectclass: organizationalunit

dn: cn=researchers,ou=groups,o=example.org
objectClass: groupofnames
cn: researchers
member: uid=richard,ou=people,o=example.org

dn: uid=lionel,ou=people,o=example.org
objectClass: top
objectClass: organizationalPerson
objectClass: inetOrgPerson
cn: Lionel Porcheron
sn: Porcheron
userPassword: password
mail: l@a.b
title: Just a person
initials: LP
ou: people

dn: uid=richard,ou=people,o=example.org
objectClass: top
objectClass: organizationalPerson
objectClass: inetOrgPerson
cn: Richard Richard
sn: Richard
userPassword: password
mail: r@a.b
title: Researcher and person
initials: R
ou: people
ou: researchers

This LDIF file creates a group: researchers and a couple of people: richard and lionel. Richard is added to the group of researchers.


# Config for Store::LDAP
authentication:
    default_realm: ldap
    realms:
        ldap:
            credential:
                class: Password
                password_field: password
                password_type:  self_check
            store:
                class: LDAP
                ldap_server: localhost
                ldap_server_options:
                    timeout: 10
                binddn: anonymous
                bindpw: dontcare
                start_tls: 0
                start_tls_options:
                    verify: none
                user_basedn: o=example.org
                user_filter: (&(objectClass=organizationalPerson)(uid=%s))
                user_scope: sub
                user_field: uid
                use_roles: 1
                role_basedn: ou=groups,o=example.org
                role_filter: (member=%s)
                role_scope: one
                role_field: cn
                role_value: dn
                role_search_options:
                    deref: always

This is my config file for catalyst.


sub list : Local {
    my ($self, $c) = @_;
    $c->assert_user_roles( qw/researchers/ ); # only researchers and view a list.
    my $people : Stashed = $c->model('AddressDB::People');
    $c->stash->{template} = 'person/list.tt2';
}

and this is a chunk of text from my controller.

Hope this is helpful - and saves you some time.

ldap thoughts

2008-08-22T03:36:00.001-07:00

I have spent the last day or so trying to get my head around LDAP/Catalyst role authorization. For this project, authentication, authorization and auditing are important considerations.

LDAP seems like a suitable choice for the first two, there is someone else who is worrying about provenance and auditing - all I have to do is make sure the hooks are in place.

My struggle with LDAP and Catalyst was metaphorically like building of the channel tunnel: working from two foreign lands and meeting in the middle. The benefit of this process is that I have a much better understanding of LDAP, and ultimatly, the system works.

LDAP is powerful and it is not obvious that this is the right tool for this deployment.

PROS:

It is mature, stable and well documented.
It is scalable and can be integrated into existing infrastructure.
Many tools support it.

CONS:

It is complicated to administer.

LDAP will allow this project to relate to exsting web applications which have been deployed already. There is a considerable problem with 'account' creep. Each new application requires the same users to create a new account: username/password - and this is a pain. LDAP provides a foundation apon which this can be centralised and ultimatly - single-sign-on can be implemented.

The mature, stable and widly deployed nature of LDAP means that the administration can be centralised.

centralised configuration

2008-08-15T07:55:00.000-07:00

I should be obvious, but today's lesson for me is to centralise configuration at all cost.

And Catalyst should make this easy; and it does.

Next week I should have some machines in the data centre to start building the system on.

LDAP + catalyst.

2008-08-14T07:10:00.000-07:00

LDAP in Catalyst is as easy as it should be... however, just because I'm using Catalyst, dosen't mean I'm doing it right.

On average thought, I think Catalyst will keep things modular so that someone will be able to pick up the work once I've left.

This brings me onto the point of this work: prepare for departure.

Nothing much else to report.

LDAP, there must be an upside

2008-08-13T04:05:00.000-07:00

LDAP is a fairly natural choice for a organisation who will be distributed over a number of sites and have users with differing roles.

So I thought I'd have a play with it to get a feel for how it would work for us.

That play took a while (two days) but I'm beginning to get a feel for it. While I normally like the command line for alot of sys-admin stuff, I really wanted a graphical browser to surf around entries and make changes.

It took half a day to track down and configure jxplorer, but this did what I wanted and my search ended there. Other candidates which I tried out included:

directory_administrator old school X11 interface, does not currently work with OpenLDAP 2.2 or newer - which was a problem for me.
lat which was a prone to crashing at my hand: I think due to my incompetence.
luma which I couldn't get configured - probably for the same reason I had problems with lat.

after all this tinkering with LDAP, i'm starting to see the benifits though: using tools which support it. The first of these will likely be an ftp server.

The upside: One directory of users with roles, all the services use this directory. Centralised and manageable.

enhancement to the code

2008-08-07T07:15:00.000-07:00

Well, I wondered if this should be part of the catalyst book errerta, but in the end I decided that the code in the book worked, so it wasn't in error.

I have been working through the book using Pg instead of SQLite. The big difference with Pg is that a user/pass are needed. I've navigated a few hurdles on the way, and this afternoon I found another one.

When including a script within your catalyst app, it is The Book has an excellent example which properly integrates your script with the existing config files.

If you construct a DSN in the following way, you can also use the user/pass conbination from your config file!

my @dsn = @{$config->{'Model::AddressDB'}->{connect_info}};

my $HOME = dir($Bin, '..');
$dsn[0] =~ s/__HOME__/$HOME/;

my $schema = AddressBook::Schema::AddressDB->connect(@dsn)
or die "Failed to connect to database at $dsn[0]";

config files

2008-08-07T03:32:00.000-07:00

I'm using a config file as I work through the tutorial. This is an obvious thing to do and I completely agree with it. However, when I introduce this structure, I don't really want it to 'over-ride' existing values - I want to delete all the values which are now to be stored only in the config file.

I had to play around a bit. My model previously contained database connection information. When I introduce the config file, the values were taken from the file. However, when I deleted all of the relevant section from the Model, it broke the app. I needed to leave in a minimal stub:

__PACKAGE__->config(
schema_class => 'AddressBook::Schema::AddressDB');

day two, hurdle navigated

2008-08-01T08:19:00.000-07:00

Well this is my second day working through the JROCK Catalyst book.

It has been a little frustrating - but this has reinforced some of the paradigms behind MVC and Catalyst so it can be thought of as useful.

Todays TIP: find_or_new does not return and empty model if it fails. The new it returns has the primary key which it failed to find. This could be 'undef' or 'null', and this object is not suitable for a subsequent: update_or_insert. Instead, do a find, if that fails, create a new object.

Catalyst, day one

2008-07-31T08:28:00.001-07:00

I've previously played with Ruby on Rails. My experience was ok with that. However, I considered 'no composite keys' a limitation.

So I've been watching the mailing lists of both CakePHP and Catalyst. I couldn't convince myself that CakePHP would allow for composite keys. In order to evaluate Catalyst, I have bought the book by Jonathan Rockaway and am working thought that.

This is the end of the first day at that and I thought note a couple my initial impressions.

On the book: It is worth visiting the errata. I went thought this and pencilled it into the book before I started reading.

On Catalyst: It seems a little complicated for my tastes. I'm expect that 'complicated' will become 'powerful' as I gain experience in it. We will see.

TIP for today: when creating a model in catalyst, and using a database with a user and password, you'll need to use something like: ...create=static dbi:Pg:dbname=testdatabase testuser testpassword if you want to include a username and password.

brief introduction

2008-07-31T08:23:00.001-07:00

The purpose of this blog is to record my experiences while setting up a data centre for some folks.

This is mainly focusing on technology and tools which are used in the work.

Expect frameworks (catalyst), database, linux, perl, security.

if I manage a post every day I spend working on the technology behind the datacentre I'll be pleased.

first post!

2008-07-31T08:16:00.000-07:00

Get the difficult first post out the way.