Using Yahoo! Pipes as bots – automating data in and data out

One of the things I’ve always thought was a weakness in Yahoo! Pipes was that you needed a human to come along and plonk in any input parameters and then stick around to deal with the output. It turns out that I’ve overlooked two old features that allow you to do just that. [Thanks to @psd for opening my eyes!]

Why would you want to do this?

The default mode of operation for a Yahoo! Pipe is to suck data in from one or more data feeds, process it, and then spit the resulting data out. Generally, you expect there to be human interaction at one or both of the input or output stages – for example, a pipe outputting to an iGoogle page requires you to visit your page.

The main reason why I’d want to run a pipe without human interaction is so I can use it as a BOT – that is, it automates processes.

Examples for following along

For the two steps described below, you can see an example skeleton pipe that illustrates both steps here:

Also, this page will let you POST and GET that pipe to try it out:

Getting data in

An increasing number of web apps are getting into the “web hooks” idea. To explain, take the example of Github. You can setup a URL which Github will POST a data packet to when you make a commit, containing information about the commit. Why is this good? It means you can connect services to Github that respond to changes in the repository without the service having to constantly poll Github, asking “Has anything changed? Has anything changed? Has anything changed?!”

What does this have to do with Pipes? If you are using a service that will make a call to a URL when something interesting happens, you can use your pipe to pick that up. How?

Adding a User Input module to your pipe opens up a parameter that you could use to manually configure the pipe before running it. However, if you run your pipe and go through “More options” to “Get as RSS”, you’ll see that your parameters appear as parameters in the query string. This is great if you want a service to run your pipe using a GET, but what if your service is going to make a POST? It turns out that the pipe will work identically if you POST the parameters, rather than GETting them.

Getting data out

Now we’re able to push data into our pipe, we need some way of getting it out after the pipe has done what it’s going to do. The Web Service module does the job for us, as you can specify a url for the pipe output to be POSTed to. If you want to, you can parameterise this url by connecting a User Input module to the Web Service module, in a similar way to the step above.