Adapting WireIt to work with independent data stores

I’ve been working during the last fortnight on the “Streams” project, which LShift, an IT consultancy in Shoreditch, are building in conjunction with the BBC. My part in this project has been to prototype the interface that a non-technical person would use to edit and create Streams, which are box-and-wire visual workflows similar to those found in Yahoo! Pipes or Tarpipe.

(The most interesting difference is that Streams combines Tarpipe’s event-driven model with Yahoo! Pipe’s modular data processing model. For more on this project and its ambition to open-sourcing, see LShift’s and the BBC’s blog posts.)

I have been using WireIt, an open-source JavaScript library, to aid with the interface prototyping. Fortunately for me, WireIt’s latest version (v0.4), announced shortly before I started this project, includes a fully-featured “Wiring Editor”, which gets you up and running with the familiar Yahoo! Pipes-style editor.

The brief for the prototype was to be able to complete a round-trip of loading up a Stream (stored as a document in CouchDB), edit it, save it (back to CouchDB) and be able to load it up again.

I thought it would be helpful to bring to your attention the things you need to be aware of if you want to use WireIt’s Wiring Editor and adapt it to work with an independent source of module and workflow data.

Stage 1 – different data formats

It is hardly worth saying that there is no point trying to persuade your data store owners to change their data format so your job is made easier. Apart from this obvious need to convert from one data format to another, you also have to deal with the fact that WireIt only keeps hold of the data it needs to display and configure its modules.

The best place I could find to hook into the loading process was by overriding the function called after the load button is pressed:

WireIt.WiringEditor.prototype._onLoad = WireIt.WiringEditor.prototype.onLoad;
WireIt.WiringEditor.prototype.onLoad = function() {
    this.service.listWirings({group:true}, {
        success: function(result) {
            var responseText = result.responseText;
            if(responseText) {
	        this.pipes = convertFromStreams(responseText);
                this.pipesByName = {};
        failure:function() {
            console.log('failure when loading, args:',arguments);
        scope: this

The data format of a combination of blocks and wires in WireIt looks like this:

var modules = [
		name: ,
		working: {}
where <em>working</em> is defined as:
working = {

The convertStreams function changes each stream document available in CouchDB into the format above.

WireIt doesn’t keep hold of any properties you attach to blocks that aren’t used explicitly. This means you need to keep hold of any properties you need to save back to your data store. I did this by keeping a globally available variable, WireIt.customFields:

if(node.type) {
    WireIt.customFields.nodes[i].type = node.type;
} else {
    WireIt.customFields.nodes[i].terminal = node.terminal;

Stage 2 – loading the list of blocks from your data store

As well as loading up descriptions of a full combination of blocks and wires, I wanted to load all the available blocks in CouchDB into WireIt so they’d be available for making workflows with.

The WireIt Wiring Editor doesn’t load its blocks from a remote system, you have to provide all the definitions in a JSON “language” file when you create the Wiring Editor. I hooked into the setup of the Wiring Editor after it had finished loading its backend-describing SMD file (see below for explanation of this SMD malarkey):

WireIt.WiringEditor.prototype.onSMDsuccess = function() {
	WireIt.customFields = {};

The populateModules function is responsible for turning the description of the block from CouchDB into the format WireIt uses for defining available blocks, which looks like this (note it is different to the format used to describe a block that is part of a workflow, described above):

var modules = [
		"name": "detect_iplayer",
		container: {
			xtype: "WireIt.FormContainer",
			type: "regexp_replace",
			title: "Detect iplayer and remove",
			icon: "images/detect_iplayer.png",
			fields: [
				{"inputParams": {"name": "caseinsensitive", "label": "case insensitive?", "value":true}, "type": "boolean"},
				{"inputParams": {"name": "regexp", "label": "regular expression", "value": "(iplayer)"}},
				{"inputParams": {"name": "dotall", "label": "dot matches all?", "value": false}, "type": "boolean"},
				{"inputParams": {"name": "multiline", "label": "multiline?", "value": false}, "type": "boolean"},
				{"inputParams": {"name": "replacement", "label": "replacement", "value": "\\1"}}
			terminals: [
				{"name": "input", direction: [-1,0], offsetPosition: {left: 110, top: -15}},
				{"name": "positive", direction: [1,0], offsetPosition: {left: 60, bottom: -15}},
				{"name": "negative", direction: [1,0], offsetPosition: {right: 60, bottom: -15}}

Stage 3 – adding default layout

The Streams data store doesn’t contain any information about how the blocks in a stream should be laid out, as this is not important to the execution of the streams. However, for loading a stream into WireIt, I wanted to give the blocks a default layout so they didn’t end up on top of each other.

I picked a layout that put the blocks equally spaced around a ring. This algorithm gives you the coordinates of a set of count blocks. It assumes the blocks are going to be square, and sets the radius of the ring so that the blocks would just touch if they were square.

function calculateLayout(count,width) {
	var coords = [];
	var n = count || 4;
	var h = width || 100;
	var radius = (n/Math.PI + 1.5)*h/Math.sqrt(2);
	var theta = 0;
	for (var i=0; i<n; i++) {
		theta = 2*Math.PI*i/n;
		coords&#91;i&#93; = &#91;&#93;;
		coords&#91;i&#93;&#91;0&#93; = radius*(1+Math.cos(theta)); // x
		coords&#91;i&#93;&#91;1&#93; = radius*(1-Math.sin(theta)); // y
	return coords;
<h2>Stage 4 - different load/save API (and the limitation of yui-rpc)</h2>
The Wiring Editor is hard-wired to use an implementation of <a title="Service Mapping Description Proposal - JSON Schema" href="" target="_blank">JSON-Schema</a> called "YUI-RPC", along with a defining Service Mapping Description (SMD - think WSDL for JavaScript). The down-side to this is that the implementation does not currently support HTTP verbs other than GET and POST, nor is it obvious how to extend the implementation in line with the spec. If your API requires the use of PUT (as CouchDB does to update or create named documents), then you will need to do something about this.

YUI-RPC supported what I needed for loading, so supporting CouchDB was only as hard as changing one of the method specifications:

"listWirings": {
    "transport": "GET",
    "envelope": "URL",
    "target": "couchdb/_design/feeds/_view/join",
    "parameters": [
        {"name":"group", "type":"boolean"}
    "description": "Get a list of streams"

To support PUT for saving, I overrode WireIt.WiringEditor.prototype.saveModule and, deciding to go with what I know, stuck in a block of jQuery that I’d had hanging around from a test working with CouchDB’s API.

General comments on prototyping with single-page applications

I made the prototype as an application that could just be served as a single HTML page from a file: URI. I strongly support this over trying to make your prototype work with some web framework or other, as you will lose valuable time setting up the framework in the first place.

Working with HTML files loaded as files gives you a couple of special properties: you can read and write from the file system and make cross-domain XMLHttpRequests (on many browsers). I had to modify YUI’s Connect object a bit to support the cross-domain XHR requests. For more information on the special properties of HTML pages viewed over file: URI’s, see these sections of a W3C position paper about TiddlyWiki written by Osmosoftonian Paul Downey.



  1. Posted July 7, 2009 at 8:05 am | Permalink


    Great post jay ! Thanks from all WireIt users 🙂

    I replied on the datastores issue in the forum:

    Layouts in WireIt is under development and should be part of the next release.

  2. Posted February 24, 2011 at 11:07 pm | Permalink

    Hey! I had started work on adapting WireIt to meet CloudCrowd’s internal workflow needs and took pretty much the exact same approach you did.

    One thing I realized is that you don’t need to get as hacky as you are, setting up your global custom fields. I hooked into setOptions method on the container just like you hooked into the load event and, in that hook, added additional loading of my custom shiz into the container object.

    From there I could define a custom getValue method off the module definition for the container so that WireIt’s serializer just ‘did the right thing’ sucking in my custom fields when it loops over each of the containers.

  3. sandeep
    Posted March 27, 2012 at 9:26 am | Permalink

    how can i use wire it in give me a sample example