Why time booking isn’t just for grown-ups

According to the testimony of friends who work as lawyers, the worst part of the job (apart from putting paper into colour-coded files) is the hassle of accounting for every minute of every day. They do this so their practice can charge its outrageous fees to the right clients – it seems that anything not being directly charged for isn’t worth doing (unless you can get someone commanding a lower hourly rate to do it). I feel palpable relief that I have never been subject to such officious management.

It has therefore been something of a shock that, over the last six months, I’ve become a convert to time booking. This transformation has been so complete that I book time to all the projects I work on, even if I’m not billing a customer. The reason I do this is simple – far apart from the homicidal minute-by-minute accounting of the lawyer, keeping a record of how I spend time brings a number of benefits, the most pronounced of which is a change of behaviour to spend time doing what I know I want to be doing. This might sound frankly un-earth-shattering, but as I hope to make clearer, knowing you’re doing what you want to be doing feels good, especially if you can get there without it being burdensome or annoying.

Why I book time

This breaks down into three explanations, which build on top of each other: knowing what I’m doing, knowing I’m doing what I want to be doing and doing what I’ve said I’m going to do.

Knowing what I’m doing

I recently carried out an experiment on myself, where I kept a diary of everything I spent time on for three months and then analysed the data to see where my time was going (I’ve yet to write this up, although the sister experiment on what I spend money on is here). In advance, I’d predicted what I spend time on – things like sleeping, eating, hanging out with friends – to see how accurate a picture I had of my own doings. The outcome showed that, although I’d been able to make a decent gut estimate of how I spent my time on habitual activities (such as sleeping), there was a wide gap of 30 or so hours (nearly 20% of the week) where I had no idea in advance what I was going to do.

It is not such a terrible thing not to be able to say what you’ve been doing, but it feels like we are under a lot of pressure to increase efficiencylive life fastersleep when you’re dead, etc. etc. Even if you don’t buy into this nonsense, it’s likely you’ve wished there were more hours in the day at least once. How can you improve something if you can’t measure it? Writing down what I’m up to obviously means I have a much better idea of what I’m up to.

Knowing I’m doing what I want to be doing

I don’t think I’m alone in finding it easy to get to the end of a day and think “very good, martini time. Now then, what exactly have I achieved today?”. It’s a somewhat more pronounced feeling if I get to the end of a week and think similar thoughts. It’s easy to brush off the question and determine to be more memorably constructive the next week, but at some point it starts to feel like there’s a lack of control over your own time.

I do think it’s important to feel that you’re doing what you want to be doing, rather than just reacting to things, bouncing about. Having a written record of where my hours have gone means I can’t just kid myself into thinking I’ve been terribly productive when I haven’t – initially this record is an uncomfortable kind of crutch. However, I think the discomfort is worth it, as after a short time, making myself aware of what I’m doing means I change my behaviour to be happy with that record.

Another thing worth noting is that once you feel confident that you’re doing things you want to be doing, all the other things that previously seemed so pressing now seem less so. In other words, you’ll end up choosing how you spend time.

Doing what I’ve said I’m going to do

Diets, fitness programmes, personal development plans, business development – many things take more than a week to come to fruition, but since it seems natural to think of time in a rhythm of the week and the weekend, it’s easy to become dispirited if looked-for effects take longer than a single week to surface.

One way of making it harder to stop doing things you’ve previously decided you want to be doing is to have the fact that you’re doing them written down – it feels way harder to stop something I’ve started when I can look back and see what progress I’ve made. It’s also easier to succumb to continuity and copy across activities from an earlier week than it is to commit to new ones: if I can see I’m on the way to finishing something, that makes it harder to start something shiny and new. Which brings me on to a bonus benefit…

Avoiding kittening

My habit of writing down what I’m doing means that I often have to expend effort to change what I’m doing. I like to avoid effort, so I’ve tended to want to avoid changing what I’m doing – my behaviour has drifted naturally towards spending longer periods of time on one thing rather than multi-tasking fruitlessly.

How I book time

Now you’re totally convinced of the need to do some time booking of your own, you will be worrying that it is going to take you forever to keep up to date. Fear not! Even with a mild tendency towards data-OCD, it takes me somewhere between ten minutes and an hour per week. As I have said, I like to avoid effort, so I don’t go much detail unless there’s a good reason to (such as wanting a detailed record for billing).

At the start of the week, I take the time to write down what I want to be doing that week and an estimate of how long I think each thing will take. At the end of the week, I write down whether each thing has been finished or not and compare my estimate of how long something would take with how long it actually did.

At the moment, I’m using two methods to book time – a broad estimate of time-spent vs. an hour-by-hour account.


At the end of a day, if I know I’ve spend a chunk of time on something, I’ll write that next to the thing, with a brief note of what I did. This is pretty rough, and I’m only bothered about the accuracy being to the nearest 30 minutes or even hour. I think this probably takes ten minutes in total over a week.


If I’m doing something that I want to keep detailed notes for, I’ll write down each day when I start spending time on it and what I’m doing as I go along. When I change to something else, I’ll record the end of that period. The accuracy is to the nearest five minutes. I usually have one or two projects running like this at any one time and over a week it might take closer to an hour than ten minutes to make the record.

What time booking is not for

I do not advocate the measurement of time-spent as some sort of performance metric. It is not right to reward yourself or someone else just because a lot of time has been spent on something: reward outcomes, not efforts.

However, the ability to accurately forecast how long things will take you and to do what you say you will are both measurable traits you ought to encourage.


Maemo Summit 2009, Amsterdam – a bit of an open-source eye-opener

Last month, Tager Communications (a customer) took me out to the Nokia-sponsored Maemo Summit in Amsterdam to help figure out whether Maemo could play a part in the national message Nokia’s marketing division puts out about its new N9xx series.

The most striking thing about being a visitor at the summit was how large the community was – at least 300 people turned up. It was later made doubly striking when I found out that almost all of those people don’t get paid to work on or with Maemo – in fact, the N900 pre-production loan we were treated to was only available to people not on Nokia’s payroll.

This was my first experience of a cohesive open source community outside of the TiddlyWiki community – it gave me much more faith in the ability of large numbers of people to work together to produce something, without needing a centrally-enforced power structure coercing them to do so. Everyone we spoke to had a very different reason for participating in the project, often because their participation gave them experience or tools that helped them sell services to other companies. Others contributed simply because they found it interesting (or earnt them wuffie).

The whole community seemed to be attached to Nokia through a single employee – Quim Gil – and, almost surprisingly, there was ’nuff luv felt for the man. It really seems as if Nokia have done a good job “stewarding” the Maemo community, filling such gaps as needed filling (graphic design, workshop hosting) and ultimately, being rewarded with a piece of software that represents an entirely different understanding of what it means to be a smart phone OS.

And that is killer with Maemo – Nokia’s N900 is obviously a competitor to the iPhone, but the experience of using it is nothing short of using a tiny computer. The “app” metaphor is nowhere near as prominent as on the iPhone or on Google Android – applications can also surface in deep integrations with other system software. As an example, the N900 ships with Skype, although you’d never know it – there is no app icon to click on. To make a Skype call you first add your account as a “VOIP and IM” account and then choose “Skype call” instead of “Cellular call” from the dialer. (Cellular? I’m sure there will be a UK localisation by the time the device is released in Blighty.)

Another departure from the standards set by both the iPhone and Android is the acceptance of web technologies as the building blocks for Maemo applications. Launching at some point in November, Nokia Web Runtime will allow packages, suspiciously similar to W3C widgets, to be installed on the phone. HTML and CSS control an application’s appearance and JavaScript controls the behaviour and taps into the native abilities of the device, such as the camera and accelerometer. The fact that these applications can also make it into Nokia’s Ovi Store will mean that the huge financial appeal of Apple’s App Store is extended to thousands times more people than the Objective-C massive.

[Update: Palm took a big move toward the web app model with the release of the Palm Pre and webOS]

The N900 isn’t going to convert many Mac fan-boys (self included), but it’s a damn fine start at a different approach. Keep an eye out for the N910.

Testing Adobe AIR applications with QUnit

I’d like to say I always write tests before I write any JavaScript code… Those times when I do write tests, I use the QUnit framework, which is used to test jQuery, so pretty well-tested itself.

I’ve recently been putting together an Adobe AIR application using HTML, CSS and JavaScript (aka Web Standards), which is very empowering and everything, but turned out to be a little tricky to write tests for. All the interesting stuff I wanted to do through AIR’s window.runtime object, such as opening native windows and messing with dock icons, is missing from the browser if you’re trying to test the JavaScript components in isolation. The ideal would be able to run your QUnit tests within the AIR runtime itself.

Fortunately, it’s straightforward to create a test app that wraps the QUnit test runner in the AIR runtime and exercises your code. Here’s how I got going:

Setting up the file structure

The folder structure I am using for the AIR app is like this:


The myApp-app.xml file is the “application descriptor” that AIR uses to create the .air package when you build the app.

I added two folders at the same level as /myApp, to contain the test framework and test code; the complete structure looks like:




Configuring the test app

The application descriptor for myAppTests is not very different from the descriptor for myApp; here they are:

<?xml version="1.0" encoding="UTF-8"?>
<application xmlns="http://ns.adobe.com/air/application/1.5">

<?xml version="1.0" encoding="UTF-8"?>
<application xmlns="http://ns.adobe.com/air/application/1.5">

The main structural difference is that myApp-app.xml has an initialWindow.content property set to myApp.html, whereas myAppTests-app.xml refers to the test runner from a directory above: myAppTests/runner.html. This is needed because the test app needs to have its working directory set to the top-level of the file structure, otherwise runner.html won’t be able to get at all the files it needs in other directories (more below).

Putting the correct files in runner.html

Here’s a runner.html that works with the folder structure described above:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
		<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
		<title>Test Suite</title>
		<link rel="stylesheet" type="text/css" href="../qunit/testsuite.css">
		<script src="../myApp/js/AIRAliases.js" type="text/javascript"></script>
		<script src="../myApp/js/jquery-1.3.2.min.js" type="text/javascript"></script>
		<script src="../qunit/testrunner.js" type="text/javascript"></script>
		<script src="../myApp/js/myApp.js"></script>
		<script src="tests.js" type="text/javascript"></script>
		<h2 id="banner"></h2>
		<h2 id="userAgent"></h2>
		<ol id="tests"></ol>
		<div id="main"></div>

Running the test suite in debug mode

The easiest way to run the test suite is to use the built-in Adobe Debugger, which lets you test your app without having to build it into an AIR package:

adl myAppTests/myAppTests-app.xml .

Don’t forget the ‘.’ character on the end – this tells the debugger to run with the top-level directory as the working directory – without it, the default is the same directory as XML file.

A short jog with Eddie Izzard

About 1:45 yesterday afternoon, Nick and I stood in the rain near Stratford station, waiting to see if Eddie Izzard would run round the corner, so we could join him for the last 10 miles of a marathon finishing in Trafalgar Square. A marathon is never something to be sniffed at, but in this case, it was Eddie’s 43rd marathon in just 51 days, and the final leg of more than 1100 miles running around the British Isles for Sport Relief. (It’s not too late to donate! £200k so far…)

I had my Flip camera with me and took some videos as we ran round. Some of them are a little shakey (some of them are nausea-inducing). Nevertheless, I’m happy to have a record of Eddie’s gutsy sprint down the Mall for a 5-hour personal best, knowing literally how far he’d come but only guessing what must have been going through his mind.

Meeting Eddie

As we waited in the rain for Eddie to get to Stratford, I couldn’t help but notice that he has over 1 million Twitter followers.

Pit stop

You run 1000 miles and your crew can’t even keep up…

Eddie meets a fan

It was amazing how many people shouted encouragement or beeped their horns as we ran through London. One guy actually leaned out of his van to give Eddie a donation.

The rickshaw

The other side of the lens, this guy pedaled the camera crew smoothly through London traffic, displaying some fine maneuvers weaving between cones.

Catching up

Quite early on, I ran in the opposite direction to tell the producer we were moving on. I didn’t realise Eddie was going to set such a punishing pace all the way from the Olympic stadium to Aldgate East, where I finally caught them.

A chat with a doctor

Eddie’s put up pictures of his battered feet (I’m not providing a link!) to show the physical effects of all his running, but this doctor thought his shins must be taking a serious beating. Eddie did admit to all the muscles being “badly bruised”. Ouch.

Legging it

Eddie had an unnerving habit of picking up the pace just as you were getting comfortable. You wouldn’t know he’d been running for 50 days…

Over London Bridge

The traffic was blocked off from our side of the road and a helicopter flew overhead – it felt a lot like we were in a 5-man London Marathon.

Approaching St. Paul’s

Could seeing St. Paul’s and knowing there was only three miles left to go failed to have had an uplifting effect?

A new supporter

Danny joined at St. Paul’s. I was really expecting there to be a lot of people running in support, but there was just the three of us. I guess the torrential rain made a difference…


Pit stop – the camera-crew needed directions once again.

South Bank

Eddie ran past the crowds by the London Eye and straight into a tourist, providing them with a pretty good story to tell back home.

Parliament Square

More speed from Eddie as St. James’ Park swung into view and we neared the last mile.

Minor drama

You can’t run 1100 miles and not have your sprint-finish captured on film, right? So where are the crew?

Buckingham Palace

Crew located, we sped on – just one straight, long road to go.

Down The Mall

This is where the temptation to attach “Chariots of Fire” gets very strong. I haven’t, so if you are that way inclined, please imagine it playing in your head.

Eddie’s Ices

Free ice-cream is never a bad thing.

Gracious End

Personally, I would have collapsed at this point, but Eddie had a nice long chat with the press and then hobbled down the steps of Trafalgar Square signing autographs and talking to people. He posed for some photographs with us. 🙂

Baselining my life (part 1, money)


I began to seriously contemplate the idea of going freelance back in May, and realised I wasn’t very prepared. For one thing, I didn’t really know how much money I spent each month or what I spent it on. Nor did I have a good handle on how I spent my time. I wanted to see how I was behaving financially, so as to know how much I’d need to earn as a freelancer to maintain my lifestyle. I also wanted to know what made up this lifestyle to see how much time I’d have available for doing this freelance thing. I did go freelance at the end of June and carried on measuring things for a while so I’d have enough data for a meaningful result.

On the 6th June, I started keeping a record of my spending; on the 1st July, I began to keep a diary of what I was doing all day. At the end of August, I stopped measuring. This means I have four months of records of how I’ve spent money and three of how I’ve spent time.

Having this information is very useful, I think, so I’ve published all the data and analysis spreadsheets for you to play with. If you wanted to do something similar, they might be helpful. (The data are all published as Google Spreadsheets – see the links at the bottom of this post.)

Of course, it would be remiss of me, as a one-time scientist, if I were to look at data and deduce hypotheses after the fact. So, without prejudicing my judgement too much by figuring out a collection of perhaps-useful things in advance, I’ve written down some questions that I want answering and hypotheses to test for each.


  1. How much money do I spend in a month?
    • Hypothesis: I think I spend £2400 a month
  2. What am I spending most of my money on?
    • I think the majority is being spent on rent, dining and boozing
  3. How much do I spend on booze?
    • I think I spend £400 a month on booze


  1. How much time do I spend sleeping?
    • I think I spend 1/3 of my time sleeping
  2. How much time do I have free in a week after I’ve done all the habitual activities?
    • I estimated I have 31 hours a week to use (estimate recorded in Habits)
  3. What is the most expensive activity per time spent on it?
    • I think it is boozing (although dining will be a close second)

This post is going to cover the questions about spending money.


  1. The Hawthorne effect – will I change my behaviour by measuring it?
  2. Not splitting money and time spending when on holiday – it just goes down as ‘holiday’

Measuring money

Answering the questions

How much money do I spend in a month?

Hypothesis: I think I spend £2400 a month
The data shows: Between May and August 2009, I spent an average of £2492.65 each month

What am I spending most of my money on?

Hypothesis: I think the majority is being spent on rent, dining and boozing
The data shows: rent 21.7%, holiday 20.6%, dining 10%, boozing 7.7%

How much do I spend on booze?

Hypothesis: I think I spend £100 a week or £400 a month on booze
The data shows: Between May and August 2009, I spent an average of £191.20 on booze each month

The data illuminates

The images below are generated from the summary spreadsheet, which contains live graphs you can play with – clicking on pie segments, for example, reveals the percentage and value attached to each.


Collecting data – iPod Touch & Notes

Online banking is a boon to the would-be data collector, as you can generally download a transaction history in csv format. Unfortunately, plenty of entries for “cash” creates a big unknown in the results. I decided to keep a record of when I spent cash during the day, what I spent it on and how much I spent. I didn’t start monitoring expenditure until the 6th May, so approximately £140 cash is unaccounted for over the four month period (I haven’t been worried sufficiently to alter any calculations to take this into account, as my monthly spending has a standard deviation of more than this).

I’d done something like this before, when I was having a crisis about where all my money was going shortly after I started working for BT in 2005. Back then, I used a pencil and notepad, and the prospect of copying all the records into a computer was so daunting that I never did. This time, I had the benefit of an iPod Touch (thanks Dad) and its “Notes” application. You can mail yourself a note, which I did every month or so.

Dealing with the data after it fell into my inbox was a matter of some search-and-replace in a text editor, to make the fields tab-separated so that they could be copy-and-pasted into Google Spreadsheets.

Analysing data – Google Spreadsheet

The first thing to do was enhance the data by tagging each expense with a chunky category like “rent” or “boozing”.

Using Google Spreadsheets to analyse an amount of information is a task with a fairly steep learning curve, since the examples for using the more complicated functions are not always that enlightening (I imagine if you are already an Excel functions master, it won’t be so hard). Nevertheless, the Google Docs forum is replete with the writings of two characters in particular – Otávio Alves Ribeiro, the Spreadsheet Ninja from Brazil, and “ahab“, the mysterious Google Docs Guru.

The performance of a Google Spreadsheet is generally quite good, although there is often slowdown in processing. I found, mainly through trial and error, that a good way to understand what is going on and get decent performance out of Google Spreadsheets is to separate worksheets between data collection and analysis; then perform any inter-spreadsheet aggregation by importing the plain data from each spreadsheet into a single worksheet and make all your calculations based on that.

Further work

I ended up with just over 20 different tags; if I were to do this again, I would split the ‘holiday’ tag into its constituent parts and I would split “dining” into “breakfast”, “lunch” and “dinner”.

It will be interesting to compare my monthly spend (which is the simplest data to measure) to the relatively steady pattern I have discovered over these four months to see how much effect measuring my spending had on dampening it; indeed, I could measure the historical monthly spends and perhaps see an effect that way.


All the data for this experiment is published on various Google Spreadsheets. See the links below.

Coloured-in jaytoon

Nick‘s been playing around with Illustrator and is pretty bowled-over by how easy it is to manipulate images. He did me this to show off his skills. I’m feeling most appreciative.

Here’s the original cartoon: Celebration

How-to log into HSBC online banking from the command-line

This is a description of the HTTP calls you need to make to achieve the log-in and account information retrieval I showed in my recent video.

Note that the content of cookies and session variables will be different each time, but examples are included here to illustrate what it going on.

Please let me know if you have similar information about a different online bank or find that this doesn’t work for you for some reason!

Getting to the login page

You can skip the HSBC homepage altogether and make a GET to:


even without any cookies set. The response sets these cookies:

Set-Cookie: HSBC_COOKIEMI=e7f408d0-75f9-11de-8a43-000408000305;Domain=;Expires=Sun, 20-Jul-2014 13:25:06 GMT;Path=/
Set-Cookie: CAMToken=GCPyfe4/i1TFoVS3L/z4qRhzbNs=;Path=/1; Secure
Set-Cookie: piba=73472940.21248.0000; path=/

Submitting your Internet Banking number

Now POST to:


Note the jsessionid=... on the end of the URL, which is different for each session; the POSTed content:


The response is a 302 to (note the doubled jsessionid):


that sets this cookie:

Set-Cookie: CAMToken=ItaH8ksgEuKY6vv6YC5nRJSTiEg=;Path=/1; Secure

The bits of your PIN that the system wants are contained in this structure in the content:

<div class="extPibRow hsbcRow">
<div class="logonPageAlignment">

         <span class="hsbcTextHighlight">&#160;<strong>FIRST</strong></span>
         <span class="hsbcTextHighlight">&#160;<strong>NEXT TO LAST</strong></span>
         &nbsp; <strong>and</strong>
         <span class="hsbcTextHighlight">&#160;<strong>LAST</strong></span>

Submitting login details

POST to:


with content:


memorableAnswer is date-of-birth formatted ddmmyy
The response is a 302 with this location:


which sets these cookies:

Set-Cookie: CAMToken=1HOPXe7zOn2qQ/dp28qb1yeozS0=;Path=/1; Secure

Continuing with login

An additional step happens due to the JavaScript in the page. There is a GET to:


The response is a 200.

What you can get once you’re logged in – example – accounts’ balance

The account summary information on the left-hand column is in a structure like this:

<div id="jsAccountDetails" class="hsbcDivletBoxContent">
<div class="hsbcDivletBoxTitle" id="jsHideAccounts" style="display: none;">
<div class="hsbcDivletBoxRow">
			<a title="Hide my balances summary" onkeypress="this.onclick" onclick="hideAccounts(); return false;" href="#" class="hsbcDivletBoxRowText">
				<span class="hsbcDivletBoxRowImage">
					<img border="0" title="Hide my balances summary" src="/1/themes/html/hsbc_ukpersonal_ib/icon_balances.gif"/>
				<span>Hide balances</span>
<div class="hsbcSeparator"/></div>
<div class="hsbcDivletBoxRow hsbcDivletBoxRowNoLink">
			<span class="hsbcDivletBoxRowText">
				<strong>GRAD PLUS</strong>
<div class="hsbcDivletBoxRow hsbcDivletBoxRowNoLink">
			<span class="hsbcDivletBoxRowText">LISTER JR</span></div>
<div class="hsbcDivletBoxRow hsbcDivletBoxRowNoLink">
			<span class="hsbcDivletBoxRowText">			40-27-16 91466410			</span></div>
<div class="hsbcDivletBoxRow hsbcDivletBoxRowNoLink">
			<span class="hsbcDivletBoxRowText">
				<strong> £ xx.xx  C</strong>
<div class="hsbcSeparator"></div>

Online banking with JavaScript – first working thing

I recorded this at Islington Hackspace tonight, figured it would be cool to share visually rather than have a big blag-rant, (see my last post for one of those).

It ought to be self-explanatory. I’m thinking about wrapping this up in an iPhone app so I can pay people for free. Rather than use PayPal or something.

Sorry it’s a bit blurry.

Source (hopefully changing lots) on github.

How to build a DIY TwitPic (without any coding skillz)

Accounts you need

  1. A blog you can upload photos to from your phone (e.g. Blogger account on my Sony Ericsson K800i)
  2. Notify.me (or other email notification service)
  3. Gmail (or other email account with mail filters and auto-forwarding)
  4. Tarpipe
  5. Yahoo! Pipes
  6. Twitter

Coding skillz you need

  1. None

What is TwitPic?

In a nutshell, TwitPic lets you upload pictures and automatically tweet a link to them. You send photos by emailing them to TwitPic.

Why would I want to build my own TwitPic?

For me, it comes down to having an existing place that I post pictures to from my phone. In my case, I have a Sony Ericsson K800i, which offers me the option to ‘Blog this’ when I take a photo. If I choose to, the photo is uploaded to a Blogger blog. This is neat, but I’d really like to be able send out tweet with a link to new photos, as I generally take photos of things I’d like to show to people.

Another reason to want your own auto-tweeting photo-blog is that you are not able to use TwitPic because you can’t send email with photo attachments from your phone.

How to build your own TwitPic (without any coding skillz)

I’m going to run through how to make a complex system without writing any code. This involves mashing together existing services to get the desired cause and effect you’re looking for. The first thing to understand is the flow of data through the services we’re going to use. After that, I’ll step through how to set up each bit.

The flow

The flow looks like this: you take a photograph on your phone and choose to send it to your mobile blog (this is the point when you lose control); the blog system updates the RSS feed for your blog; Notify.me (or another notification service) notices the change and sends an email message to you with the body and title of the RSS item; Gmail (or another mail service) filters the incoming email and forwards it to a Tarpipe workflow; Tarpipe posts the email to a Yahoo! Pipe; Yahoo! Pipes extracts the URL of the new blog post and formats the text of the tweet, before returning the information to Tarpipe; Tarpipe tweets on your Twitter account.

Tabulated, the flow looks like this:

this thing does this to this
Person sends photo to Blog
Blog updates RSS feed
Notify.me reads change from RSS feed and emails Gmail
Gmail forwards email to Tarpipe
Tarpipe posts email to Yahoo! Pipes
Yahoo! Pipe extracts blog post URL and sends back to Tarpipe
Tarpipe posts a tweet to Twitter

Step 1 – sending a photo to your blog

In my case, the option to ‘blog this’ was already available on my phone’s camera menu, which posts a photo to a Blogger blog. It is not unusual for blog services to allow you to send new messages as email, so you could email photos to your blog as attachments.

Step 2 – updating the blog RSS feed

If your blogging service doesn’t already do this, it’s time to get a new blogging service…

Step 3 – getting an email about the change

Both Notify.me and Notifixio.us can email you if an RSS feed changes. They can take up to an hour to respond to changes, so this is the bottleneck in the flow. On the other hand, it is very simple to set one of these services up. There are other notification services, but they tend to require the writing of some code to work with them.

Assuming you’re using Notify.me, the subject of the email will be the title of the blog post prepended by “notify.me” and a space, and the body of the email will be the body of the blog post appended by a load of garbage advertising Notify.me.

Step 4 – responding to the incoming email

I expect that your email client allows you to set up filters to respond to incoming emails automatically. If it doesn’t, as above, it is time to get a new email client… Assuming you are using Notify.me as the notification service, you can filter out the incoming mail by creating a filter that selects email with a ‘FROM’ email address including ‘@notify.me’. Of these, you can get to the emails about your photo blog by selecting those whose subject includes your blog title. Set up the mail filter to forward to the email address you’ll get for the Tarpipe workflow we’re about to create.

Step 5 – the Tarpipe workflow

Tarpipe is a service for setting up workflows that respond to emails or API calls. Generally, the output will be an update to Twitter, Facebook or some other service that receives messages. Because a Tarpipe workflow can be kicked-off using email, and because you can post the email to any web service, it has opened the door to creating zero-code systems that are able to do interesting things. In short, hurray for Tarpipe.

We want to create a Tarpipe workflow that receives an email, posts it to Yahoo! Pipes, takes the response and tweets it. This is a simple chain of the “MailDecoder”, “REST” (with a “TextInput” connector to supply the URL to post to) and “TwitterUpdater” connectors. When you save the workflow for the first time, you’ll get the email address to use in the email filter from the previous step.

At the moment, Tarpipe has a shortcoming whereby the REST connector, which is what is used to send messages to any web service of your choosing, encodes the information in a structure that is very difficult to use directly with Yahoo! Pipes, despite Pipes’ obvious utility providing the string manipulation capabilities that Tarpipe lacks. You can use an intermediate service, such as this proxy, to munge the data into a more suitable format. The proxy takes a parameter in the URL to specify the ID of the Yahoo! Pipe to use. To get that ID, we’ll need to create a Yahoo! Pipe.

(That proxy URL in full: http://tarpipe-yahoopipes-proxy.smart.joyent.com)

Step 6 – getting at the blog post URL with a Yahoo! Pipe

Pipes has been around for a while now, and I believe it is mainly used to grab remote RSS feeds and other data, perform some operations such as selecting items from a feed, sorting them, turning them into Japanese and sending on the resulting feed. However, you can also use Pipes as a text manipulation machine, posting in some strings of text rather than grabbing them from somewhere else.

We’re going to create a Pipe to take the large and messy email body, delicately extract the URL of the blog post with our photo on, and append that to the subject of the email. We’ll also remove the string “Fwd: notify.me” (plus space) from the subject of the email.

You can get access to any of a list of parameters, posted as text to a Pipe, by adding a “Text Input” module from the “User inputs” list, one for each named parameter being sent, with the ‘name’ property of each module set to the name of the parameter. Using the proxy linked to in the previous step, the subject of the email will be posted to the Pipe as ‘title’ and the body will be posted as ‘description’. Set up your text input modules as shown below (only the ‘name’ property matters).

Now we have access to the strings, we have to extract the relevant strings from them. String extraction can be achieved directly by using the ‘string regex’ module, which stands for ‘string regular expression’. “Regex“, or “Regular expressions” is a language for describing patterns, such as “four letters, followed by a space, followed by a letter or a number”.

The string regex module doesn’t actually extract strings, it replaces any matches of a pattern with another string. However, we can make this a string extractor by replacing a pattern of “anything, followed by the pattern we want, followed by anything” with “the pattern we want”.

In regex, that looks like this:

replace: (?s).*(<the pattern we want>).*
with: $1

An explanation of the above runs like this: –
(?s) means “treat the string as a single-line” – without it, the pattern matching would stop at a line-break;
.* means “match zero-or-more of any character”;
(<the pattern we want>) puts <the pattern we want> into a “group”, to be used in…;
$1 means “the first group from the pattern”

The title is the simpler case, where the pattern we want to extract is [My Mobile Blog] - <the rest of the title>, from a string that looks like “Fwd: notify.me [My Mobile Blog] – super new photo”. To get what we want, connect a string regex module to the text input module named ‘title’ and put the following in the ‘replace’ and ‘with’ properties:

replace: (?s).*Fwd: notify\.me (\[My Mobile Blog\] - .*)
with: $1

The places where there is a \ preceding another character are examples of “escaping” characters that would otherwise have special meaning in regex. For example, [ would normally mean “start a set of characters to choose from”, so \[ just means “match [“.

The more complicated extraction is getting the URL from the email body. Here we want to get a URL of the form http://c.notify.me/F6d3HG out of a much longer string of text. This URL points to an address on Notify.me’s URL-shortening service, that will direct you to the blog post’s much longer URL. Connect a new string regex module to the text input module named ‘description’ and put the following in the ‘replace’ and ‘with’ properties:

replace: (?s).*(http:\/\/c\.notify\.me/\w{6}).*
with: $1

The \w{6} pattern means “6 word-characters”. A “word-character” is something you would expect to find in a word, rather than punctuation or a space. We don’t need to worry too much about what is and what isn’t a word-character, as we only expect to find letters, numbers and “_” in the 6 characters making up the end of the URL, which are definitely word-characters.

(If this has piqued your curiosity, there is a lot of information on regular expressions at http://regular-expressions.info.)

We’re nearly done. Now we have two strings that we want to join together as our tweet. To do this, use the ‘string builder’ module and assemble them as “<title>” plus ” – ” plus “<URL>”. Before outputting this string, we’re going to create a new ‘item’, which is the unit of output from a Pipe – use the “item builder” module, setting the ‘title’ of the new item as the tweet string.

With that all done, we can save the Pipe. By clicking on the ‘run this pipe’ link, you can find the ID of the pipe in the URL of the page you’re taken to, named “_id”. To use this ID in the Tarpipe workflow, in the text input module which is connected to the REST connector, add the ID to the URL as a query-string parameter named “pipeId”: http://<proxy>/?pipeId=... . (Note the / between the domain of the proxy and the ?.)

Step 7 – tweeting from Tarpipe

If you have set up your pipe as above, the text to use for the tweet will be available from the ‘title’ terminal on the right-hand side of the REST connector. Wire this to the ‘title’ terminal of aTwitterUpdater connector.

You need to set up your Twitter account in the ‘accounts’ tab so Tarpipe can tweet with it. At the moment, you need to give out your Twitter username and password, which is the dreaded anti-pattern, and since Twitter now supports OAuth, it makes sense for Tarpipe to use that to get the appropriate level of access to your account. For the purposes of this example, you could set up a new twitter account to link to Tarpipe.

Save the workflow, and you’re done! If everything is configured correctly, sending a photo to your mobile blog will cause a tweet to appear on your timeline, linking back to your new blog post.

Testing the bugger

With a total of 5 remote systems (6 with the Tarpipe / Yahoo! Pipes proxy) between you and the intended tweet, there’s a lot of scope for things to go wrong. The weakest parts of the chain are going to be those you set up yourself, particularly the regex-heavy Yahoo! Pipe, however errors can happen at any point in the system, and problems caused by delays, caching or rate-limits can manifest themselves in various ways.

It’s a good idea to set up test accounts for any services you are hooking into that you use on a day-to-day basis, especially if they publish information to your friends or the public. If you don’t, as I didn’t for my Twitter account, you could end up with something like this mess:

The trick to understanding what is going on in these remote systems is to log activity whenever you get the chance. One useful service for this is PostBin, which lets you create a logger that records anything you post to it.

In the flow above, opportunities to post to this log are:

  1. Adding another REST connector to post what is received by the Tarpipe workflow
  2. Adding a ‘web service’ module to post what is received by the Yahoo! Pipe
  3. Adding a ‘web service’ module to post what comes out of the string extraction steps
  4. Adding another REST connector to post what is returned to Tarpipe from the Yahoo! Pipe

When using the Tarpipe / Yahoo! Pipes proxy, adding &postBinId=<id> to the URL in the text input which connects to the REST connector will cause the proxy to log at these four points:

  1. What comes into the proxy
  2. What is sends Yahoo! Pipes
  3. What comes back from Yahoo! Pipes
  4. What it sends back to Tarpipe

Here’s an example of the PostBin logs you get from the proxy:

As well as logging, some of the services mentioned provide help with the testing and debugging process:

  • Gmail puts mail it forwards due to filters into your ‘sent items’; the exact text sent can be found by selecting ‘show original’ from the ‘reply’ drop-down in an email
  • Tarpipe provides an ‘activity’ tab that shows what came in and went out (note that sometimes you have to click on ‘all’ to see workflow activity, particularly when workflows only contain REST connectors)
  • Yahoo! Pipes has an in-built debugger that allows you to examine the effect of your modules on the text you put into the ‘debug’ property of a user input
  • RegexPal lets you test out a regex on some text (such as the body text of your email)

Despite all these tools, debugging a chain of systems you have no control over can be a painful process. Be particularly careful when using Yahoo! Pipes, as it is not uncommon for results of a Pipe to be cached for 30 minutes. I have experienced this happening even if you change the input data. Lastly, note that should an error actually occur in your flow, it’s likely you’re not going to be told about it…

Good luck!

Notes on migrating an application from AppJet to Smart Platform

I’ve been wingeing quite a lot recently about AppJet’s demise and the shoddy state of potential replacements, and got quite a nice response from Jim Pick at Joyent. I decided today to have a play with their new Smart Platform and had a go at porting one of my AppJet applications over to Smart. Here’s my notes, fresh from the lab…

It’s not as simple as hitting an edit button yet, but it’s still worth having a play. The use of Sammy to handle and dispatch incoming requests, along with a single “bootstrap.js” file that controls all the SSJS, means that it ought to be fairly easy to port over single-page AppJet apps to Smart Platform. Here’s some notes I’ve gathered whilst porting over an application that acts as a gateway between tarpipe’s REST connector and a Yahoo! Pipe, allowing you to send data to a Yahoo! Pipe for processing.

Request object

In AppJet, you get the POSTed variables and the query-string parameters both in request.params. In Smart, the query-string parameters are in request.query with the raw string in request.queryString; POSTed variables are in request.body with the raw datastring in request.content.


In AppJet, you have the option to define different functions for handling different types of request e.g. GETs, POSTs and requests to different resources e.g. /this, /that. Smart uses Sammy, which gives you a similar mechanism to choose which functions respond to different incoming requests, although you also have the option to use a single function main(request) {...} to handle all responses. A concrete example is:

    function get_resource() { ... }
    GET("/resource", function() { });

Writing a response

AppJet provides a default chrome that wraps an application, which you can remove by setting page.setMode('plain'); then you use functions like print(response) to send the respons. Smart does it slightly differently, where whatever you return from a function is sent as the response. It doesn’t seem possible to send HTTP header information.

Making HTTP requests

AppJet provides functions like wpost and wget to make HTTP calls. Smart provides the system.http.request function to do the same sort of thing. The difference in the returned response is that AppJet gave you a raw string of the response body, whereas Smart returns an object containing headers and content properties.


The lack of an in-browser edit experience has been my complaint of late, but doing the Git bit to deploy my app to Smart was straightforward, and now it’s up at http://f7e3dbf.smart.joyent.com.

Try it out with:

curl -d 'pipeId=lAUEoB1R3hGTqt6hggSecQ&data={"items":[{"title":"A title","description":"Any description","link":"http:\/\/example.com"}]}' http://f7e3dbf.smart.joyent.com


For the sake of completion, here’s the source code. I’ve gone from this:

/* appjet:version 0.1 */

var p = request.params;
var result = "";
var output = ""; 

function serialize(obj) {
    var str = "";
    for(var i in obj) {
        str += "&"+encodeURIComponent(i)+"="+encodeURIComponent(obj[i]);
    return str.substring(1);
if(p && p.data && p.pipeId) {
    var data = JSON.parse(p.data);
    var pipeId = p.pipeId;
    var items = data.items;
    if(items) {
        var url = "http://pipes.yahoo.com/pipes/pipe.run?_id="+pipeId+"&_render=json";
        // tarpipe only POSTs 1 item
        var result = wpost(url,{title:items[0].title,description:items[0].description});
        if(result) {
            result = JSON.parse(result);
            if(result) {
                output = result.value;
                if(!output.items[0].title) {
                    output.items[0].title = "returned from Yahoo! Pipe at: "+pipeId;
                output = JSON.stringify(output);
            } else {
                result = "bag result from Yahoo! Pipe at: "+pipe;
        } else {
            result = "no result returned";
    } else {
        result = "no items in data input";
} else {
    result = "no data or no pipe input";

output ? print(output) : print(result);

To this:


function serialize(obj) {
    var str = "";
    for(var i in obj) {
        str += "&"+encodeURIComponent(i)+"="+encodeURIComponent(obj[i]);
    return str.substring(1);

function main(request) {
	var p = request.content!="" ? request.body : request.query;
	var result = "";
	var output = "";
	/* for debug input
	for (var i in p) {
		output += i+", "+p[i]+"\n";
	return output;*/
	if(p && p.data && p.pipeId) {
	    var data = JSON.parse(p.data);
	    var pipeId = p.pipeId;
	    var items = data.items;
	    if(items) {
	        var url = "http://pipes.yahoo.com/pipes/pipe.run?_id="+pipeId+"&_render=json";
	        // tarpipe only POSTs 1 item
	        var toPost = "title="+items[0].title+"&description="+items[0].description;
	        var result = system.http.request("POST",url,null,toPost);
	        if(result && result.content) {
	            result = JSON.parse(result.content);
	            if(result) {
	                output = result.value;
	                if(!output.items[0].title) {
	                    output.items[0].title = "returned from Yahoo! Pipe at: "+pipeId;
	                output = JSON.stringify(output);
	            } else {
	                result = "bag result from Yahoo! Pipe at: "+pipe;
	        } else {
	            result = "no result returned";
	    } else {
	        result = "no items in data input";
	} else {
	    result = "no data or no pipe input";
	return output || result;