Shortlink

I ♥ Sublime Text 2

There are tons of text-editors and IDEs out there and I tried my fair share of them but there is nothing for me that beats Sublime Text while working on non-Java based web projects or Python scripts.

It has tons of features that I love. Speed, saving the state of the open files without actually saving them and very easy extensibility (through the Package Control extension).

For example today I was writing a Soy-template when I thought: it would be nice to have syntax highlighting. A quick search on Google returned a GitHub-repo with syntax highlighting. I added it as a repository to Package Control, went to install packages and within 1 minute I had it up and running. Easy as that.

You are missing out if you are not using Sublimte Text 2 yet. Try it out. The trial isn’t even irritating or limiting. I bought it anyway beacuse I gladly support this kind of amazing products!

Shortlink

Launching Informatica Powercenter 9 workflows from the command-line with PMCMD

During my last year working as an ETL developer I wrote some scripts to automate the work that I was asked to do. This work mostly related to loading files with the use of Informatica Powercenter 9. I’m putting my gathered knowledge up here as this is information that might be useful for anyone. Please note: it is possible to do a lot more things with PMCMD then just start workflows (ie. schedule workflow, start/stop tasks) but that’s outside of the scope of this post.

Launching PMCMD

To get the PMCMD command working it is required to have 4 environment variables setup:

  • INFA_HOME: absolute path to the directory of the Informatica client
  • INFA_DOMAINS_FILE: absolute path to the domains.infa file
  • INFA_USER: username to connect to Informatica. Might be named differently.
  • INFA_PASS: password to go with the username in INFA_USER. Might be named differently.

Tip: set these environment variables inside your script so that you don’t require each user to setup it’s environment variables to use the script.

The PMCMD executable for Windows is found here: INFA_HOME\clients\PowerCenterClient\CommandLineUtilities\PC\server\bin\pmcmd.exe

To start a workflow use the following command.
Note: New lines added for clarity of reading, should not be in real command. Domain_MY_DOMAIN, Service_MY_SERVICE, FOLDER_OF_WORKFLOW and WORKFLOW_NAME should be replaced by your own values.

pmcmd.exe startworkflow
 -d Domain_MY_DOMAIN -sv Service_MY_IS_SERVICE_NAME
 -uv INFA_USER -pv INFA_PASS -f FOLDER_OF_WORKFLOW
 WORKFLOW_NAME

We reference in the command to the environment variables INFA_USER and INFA_PASS. You might change this to any other name if you make sure the environment contains values for the variables with the new name. It is also possible to use PMCMD by specifying the username and password directly with the -u and -p command-line options. Don’t do this as your username and password will be readable in the processlist!

Useful other command-line options

To make the command-line more useful there are some options that you might want to add:

-localparamfile PATH_TO_PARAM_FILE

This option allows you to override the parameter file that is specified within the workflow. A use-case for this could be that you want to create a parameter file per file that you want to load. You can then run the same workflow in parallel, one for each file (You will need to enable Concurrent Execution in the General Tab of the Edit Workflow screen).

-wait

By default the PMCMD command will return immediately after the workflow has been started. By issuing the -wait command-line option the command will not return until the workflow has finished. This option is very useful to organize files that have been processed by the workflows. Yes, this can be done by using the command-task in your workflow but I like to put only ETL logic in my workflows so it can be easily migrated to different environments.

After the workflow has finished the result can be determined by checking the value of the return code. An overview of the return codes can be found below and in the Informatica Powercenter Help -> Command Reference -> pmcmd Command Reference -> Using pmcmd -> Running Commands in Command Line Mode -> pmcmd Return Codes.

Code  Description 
0 For all commands, a return value of zero indicates that the command ran successfully. You can issue the following commands in the wait or nowait mode: starttask, startworkflow, aborttask, and abortworkflow. If you issue a command in the wait mode, a return value of zero indicates the command ran successfully. If you issue a command in the nowait mode, a return value of zero indicates that the request was successfully transmitted to the Integration Service, and it acknowledged the request.
1 Integration Service is not available, or pmcmd cannot connect to the Integration Service. There is a problem with the TCP/IP host name or port number or with the network.
2 Task name, workflow name, or folder name does not exist.
3 An error occurred starting or running the workflow or task.
4 Usage error. You passed the wrong options to pmcmd.
5 An internal pmcmd error occurred. Contact Informatica Global Customer Support.
7 You used an invalid user name or password.
8 You do not have the appropriate permissions or privileges to perform this task.
9 Connection to the Integration Service timed out while sending the request.
12 Integration Service cannot start recovery because the session or workflow is scheduled, waiting for an event, waiting, initializing, aborting, stopping, disabled, or running.
13 User name environment variable is set to an empty value.
14 Password environment variable is set to an empty value.
15 User name environment variable is missing.
16 Password environment variable is missing.
17 Parameter file does not exist.
18 Integration Service found the parameter file, but it did not have the initial values for the session parameters, such as $input or $output.
19 Integration Service cannot resume the session because the workflow is configured to run continuously.
20 A repository error has occurred. Make sure that the Repository Service and the database are running and the number of connections to the database is not exceeded.
21 Integration Service is shutting down and it is not accepting new requests.
22 Integration Service cannot find a unique instance of the workflow/session you specified. Enter the command again with the folder name and workflow name.
23 There is no data available for the request.
24 Out of memory.
25 Command is cancelled.
Shortlink

Creating 1 euro per dag: lessons learned

After months of work we finally released the website for our charity 1 euro per dag at the beginning of this month. The idea is to have a monthly voting between two charities and the winner gets 1 euro per day of the participating people. Simple yet efficient method of making donating money to charities fun. Participation is doing quite well. Without any publicity besides posting it on Facebook/Twitter/Google+ we have managed to get 16 participators.

From the beginning on we had the focus on keeping the website as simple and lean as possible. Thinking twice about every page that we would add. We achieved this by hiding a lot of unnecessary content behind buttons (but not in an annoying way).

I was the developer and since we were starting the project from scratch it was a nice opportunity to play with jQuery and Less CSS. We had to stick to PHP to keep hosting costs to a minimum but for the rest I could go all out.

Less CSS saves time and helps in keeping things clean and lean

This is the first time I release a project built with Less and I must say that using it is a blessing. I used to have projects where the designer had his classes too loosely defined ending up in weird style errors all over the project. Nesting in Less CSS solves this problem once and for all. Will definitely use this again next project.

A good framework saves a lot of time, pick a good one

Picking a good framework that supports all your requirements will help you save a lot of time. After some testing I went for CodeIgniter 2 based XTA 2 which includes login for Facebook, Yahoo, Google and Twitter. It was only once we started testing intensively that we found out that XTA 2 had quite some shortcomings which required a lot of development to get it all fixed but I managed: a user account system that allows the user to link multiple external authentication providers to its account and saves the original target url throughout login and registration process.

Working with different account providers is the source of a lot of headaches. Example: if a user creates an account through Twitter we do not have an e-mail address or password. If we require the user to enter his password to delete his account he first has to setup an e-mail address, verify it, create a password and only then the user is able to delete his account. This means the user has to jump through a lot of hoops to delete his account and give even more information to us in the process. In the end we chose to let the user delete his account without a password when he has not setup a password. A minor security issue but we avoid a big inconvenience for the user.

jQuery rocks for adding small interactions to the site.

I have mainly worked on projects that serve whole apps written in JavaScript (GWT, Closure Tools). For adding small interactions jQuery was really a blessing as a requirement for reusability is nonexistent. The only thing that I made for reusability is the small piece of code that I wrote for tooltips:

Screenshot tooltip

The code is only a few lines of jQuery, Less CSS, a sprite for the arrows and utilizes the data-* attributes introduced by HTML5. And since sharing = caring. Here is the code, free to use for whatever you like.

Example of an element with a tooltip (adding the attribute data-tooltip-position=”below” will show the tooltip below instead of above the element)
This <span data-tooltip="I'm a tooltip!">text</span> has a tooltip!
The JavaScript:

function setupTooltips() {
  $("[data-tooltip]")
    .unbind('mouseenter')
    .unbind('mouseleave')
    .mouseenter(function(e) {
    var el = $(this);
    var offset = el.offset();

    if(typeof tooltip === "undefined") {
      $("body").append("<div id='tooltip'><div class='arrow-up'></div><div class='content'></div><div class='arrow-down'></div></div>");
      tooltip = $("#tooltip");
    }

    // show tooltip, we need this to calculate height
    tooltip.show();

    tooltip.find('.content').html(el.attr("data-tooltip"));

    if(el.attr("data-tooltip-position") == "below") {
      tooltip.find('.arrow-up').show();
      tooltip.find('.arrow-down').hide();

      // position 3 pixels between bottom element and tip arrow
      var top = offset.top + el.height() + 3;
    } else {
      tooltip.find('.arrow-up').hide();
      tooltip.find('.arrow-down').show();

      // position 3 pixels between top element and tip arrow
      var top = offset.top-tooltip.height()-3;
    }

    // center the tooltip above the center of the element
    var left = offset.left + el.width()/2 - tooltip.width()/2;

    // show tooltip and set position
    tooltip.offset({ top: top, left: left});
  }).mouseleave(function() {
    tooltip.hide();
  });  
}

The Less CSS

#tooltip {
  display: none;
  position: fixed;
  top: 0px;
  left: 0px;
  z-index: @zIndexTooltip;

  .content {
    padding: 1px 7px;
    background-color: @orange;
    text-shadow: 0 -1px 0 rgba(0, 0, 0, 0.25);
    color: white;
    font-weight: bold;
    text-align: center;
    -box-shadow:        3px 3px 0 #ddd;
    -webkit-box-shadow: 3px 3px 0 #ddd;
    -moz-box-shadow:    3px 3px 0 #ddd;
  }

  .arrow-up {
    height: 7px;
    background-image: url(img/sprite-tooltip.png);
    background-position: center top;
    background-repeat: no-repeat;
  }  

  .arrow-down {
    height: 10px;
    background-image: url(img/sprite-tooltip.png);
    background-position: center bottom;
    background-repeat: no-repeat;
  }  
}

The sprite image for the tooltip: 

Using sprites to limit number of connections

To minimize the number of requests to your server to cope with the limitation of the HTTP 2-connection-per-server constraint it is advised to bundle your images in 1 image. While using Google Web Toolkit I never had to care about doing this manually, it is done automagically. Doing this manually can be a big pain if it wasn’t for this solid tutorial that includes anything you need to know to get started using sprites quick and easy. It uses a Photoshop script to make creating the images an easy process and offers a couple of Less-functions to integrate it easily in to your site.

Small note on the sprite Photoshop script: make sure you have set your default measurement units to pixels or it won’t work.

Shortlink

Java component JMapViewer with offline OpenStreetMap support

For a prototype that is running in an isolated environment I was in need of an offline OpenStreetMap component for Java. Of all the awesomeness that OpenStreetMap does provide this was not one of them.

Luckily I’m not the person that is afraid of some quick hack-n-slash work. After inspecting the code of JMapViewer which is released under GPL by OpenStreetMap I quickly found out to add support for OpenStreetMap by creating an extension of AbstractOsmTileSource. A stunning 29 lines of code later and offline support was ready. I put it up on GitHub in case anyone is interested in it as well: https://github.com/balloob/JMapViewer

I put up a full copy of their SVN-repository with my offline-class included. Just the class can be found here.

How to use it:

JMapViewer map = new JMapViewer(); 
map.setTileSource(new OfflineOsmTileSource("file:///path/to/tiles/",1,9));

Downloading tiles
To download the Open Streetmap tiles to your computer you can use jTileDownloader.

Note that jTileDownloader 0.6 requires Java 7. It will not throw an error when an older JRE is used untill you click the download button.

As of this writing (August 2, 2012) Apple has not released Java 7 for OS X yet.

Shortlink

Empty strings treated as null value in Oracle

Why o why, dear Oracle, do you treat empty strings as a null value in SQL-queries? And, dear Oracle, if you do so, why did you implement it sloppy?

SELECT * FROM table_x WHERE varchar2_column_y <> '';

This query takes a few minutes to grind through ~70 million rows and then returns 0 results. This takes way longer then other queries and the result count is wrong.

Initially I started to doubt my own ability to write valid SQL. After an hour and many tests I decided to see if maybe there is something in Oracle that can create this kind of behavior and I found this page on the world wide web: Oracle/PLSQL: Difference between an empty string and a null value. It still doesn’t help me to understand why the query would return 0 results after running for a few minutes but at least it made me understand how to formulate the correct query:

SELECT * FROM table_x WHERE varchar2_column_y is not null;

Sigh.

Shortlink

Open courseware on Open Standards

For a while I have been involved in a project within the University of Twente to work together with the Dutch government to develop free courseware in Dutch about open standards for use by and within governments. We finalised and released the courseware last week.

The courses are developed for people working at the government, intermediary organisations involved with the government and students studying at the University or Higher Professional Education.

The courseware exists out of 3 modules (Open Standards, Costs and Benefits, Implementation and Organisation) existing out of 3/4 components each. Each component takes half a day to teach.

The courseware can be found here.

Shortlink

Hacking the Wii: playing games stored on USB-devices

I love being lazy, extremely lazy. So when I read online that I am able to play games on my Wii from an external drive so I do not have to get of the couch to change games I jumped on the bandwagen straight away.

As the Wii-community is a bit unstructured it took a while to sort out all the right stuff but after some hours fiddling I got it all right.

 

Preparing the SD-Card: Get BootMii and the Homebrew Channel installer

My Wii was running the System Menu version 4.3E and thus I was able to use a fairly new hack to get the Homebrew Channel running: the LetterBomb. Contrary to earlier hacks this exploits a bug in the Wii system software making us independent of buggy games which make them go for a lot on the second hand market.

  1. Get yourself an SD-card
  2. Start your Wii, go into system settings, internet settings and write down the MAC address of your Wii
  3. Go to http://please.hackmii.com/, follow the instructions, download letterbomb and extract it to the root of your SD card

Preparing the SD-Card: Get IOS236

  1. Download it here
  2. Extract it to your SD-card

Preparing the SD-Card: Get d2x IOS

  1. Download the installer and map236.xml here
  2. Extract the archive to the apps-folder on your SD-card
  3. Move map236.xml to the d2x-folder that you just extracted to the apps folder and rename it to ciosmaps.xml

Preparing the SD-Card: Get Usb Loader

  1. Download the installer here
  2. Extract the archive to your SD-card
  3. Download meta.xml and put it in /apps/USBLoader on your SD-card replacing the existing meta.xml

Preparing the USB-device: Get WBFS files

  1. Make sure your USB-device is formatted in FAT32
  2. Create a folder in the root called “wbfs”
  3. Put WSBF-files of your games inside it

 

Now you’re ready to mess up your Wii

  1. Plug your SD-card into your Wii
  2. Plug your USB device in the USB port that is closest to the edge of the Wii

 

Hacking the Wii: installing BootMii and the Homebrew Channel

  1. Go to your messageboard, scroll to yesterday/2 days ago (depends on timezone) and you’ll see an envelope containing a bomb.
  2. Click on the envelope and follow the instructions

Hacking the Wii: installing IOS236

  1. Follow these instructions

Hacking the Wii: installing d2x

  1. Follow these instructions

 

Done, you can now launch the USB Loader from the Homebrew Channel!

Shortlink

CampusVirus

One of my little projects to try to make the world a better place: CampusVirus. A virus scanner for the Samba-based network of our campus.

I live on the campus of the University of Twente where all computers are connected to the same network. This results in a lot of Samba-shares with sometimes virusses in the files offered. A bit too often in my opinion.

So to gather some statistics I made up a little project in Ruby to gather the info: CampusVirus. It uses a special XML-feed from the CampusSearch search-engine that returns all executables found on Campusnet, scans them with a combination of SmbNetFS+ClamAV and saves the result to a MySQL-database.

In my first test-run I scanned a total of 13,479 files resulting in 149 files (1.1%!) to be found infected.

The great plans I had with the project was to integrate it with CampusSearch to warn people if they were browsing CampusSearch with a computer sharing infected files. But, as with so many projects, it never got to a release and ended up eating dust on my harddrive.

But maybe there is someone out there who would like to take this task upon him to complete the project and make the world a tiny bit better. And for that person I am making the source code of CampusVirus available. To be used freely as one sees fit.

Download CampusVirus

Shortlink

Better Interpol: Interpol Wanted-list goes Social

While I was browsing the Interpol Wanted-list to search for a new arch-enemy for the USA I became fed up with the ugly layout of Interpol. Wouldn’t it be better if all those profiles would be in a layout that is recognizable and usable? Since we had to travel today 520 miles from Vegas to Sillicon Valley (on our way to SF for Google I/O!) I decided to brew up some magic with GreaseMonkey for Firefox.

I call the result Better Interpol: It adds a Facebook-sauce to the search result page and the profile pages of wanted-people:

 

Download Better Interpol (Firefox + GreaseMonkey only)

Shortlink

Managing my digital life: information flow

Some notes:

  • Mail accounts are imported to GMail via POP3-import
  • SMS are imported to GMail via SMS Backup
  • Google Talk has a feature to save chat logs to GMail
  • Description of what’s happening inside GMail is here