How to build a DIY TwitPic (without any coding skillz)

Accounts you need

  1. A blog you can upload photos to from your phone (e.g. Blogger account on my Sony Ericsson K800i)
  2. Notify.me (or other email notification service)
  3. Gmail (or other email account with mail filters and auto-forwarding)
  4. Tarpipe
  5. Yahoo! Pipes
  6. Twitter

Coding skillz you need

  1. None

What is TwitPic?

In a nutshell, TwitPic lets you upload pictures and automatically tweet a link to them. You send photos by emailing them to TwitPic.

Why would I want to build my own TwitPic?

For me, it comes down to having an existing place that I post pictures to from my phone. In my case, I have a Sony Ericsson K800i, which offers me the option to ‘Blog this’ when I take a photo. If I choose to, the photo is uploaded to a Blogger blog. This is neat, but I’d really like to be able send out tweet with a link to new photos, as I generally take photos of things I’d like to show to people.

Another reason to want your own auto-tweeting photo-blog is that you are not able to use TwitPic because you can’t send email with photo attachments from your phone.

How to build your own TwitPic (without any coding skillz)

I’m going to run through how to make a complex system without writing any code. This involves mashing together existing services to get the desired cause and effect you’re looking for. The first thing to understand is the flow of data through the services we’re going to use. After that, I’ll step through how to set up each bit.

The flow

The flow looks like this: you take a photograph on your phone and choose to send it to your mobile blog (this is the point when you lose control); the blog system updates the RSS feed for your blog; Notify.me (or another notification service) notices the change and sends an email message to you with the body and title of the RSS item; Gmail (or another mail service) filters the incoming email and forwards it to a Tarpipe workflow; Tarpipe posts the email to a Yahoo! Pipe; Yahoo! Pipes extracts the URL of the new blog post and formats the text of the tweet, before returning the information to Tarpipe; Tarpipe tweets on your Twitter account.

Tabulated, the flow looks like this:

this thing does this to this
Person sends photo to Blog
Blog updates RSS feed
Notify.me reads change from RSS feed and emails Gmail
Gmail forwards email to Tarpipe
Tarpipe posts email to Yahoo! Pipes
Yahoo! Pipe extracts blog post URL and sends back to Tarpipe
Tarpipe posts a tweet to Twitter

Step 1 – sending a photo to your blog

In my case, the option to ‘blog this’ was already available on my phone’s camera menu, which posts a photo to a Blogger blog. It is not unusual for blog services to allow you to send new messages as email, so you could email photos to your blog as attachments.

Step 2 – updating the blog RSS feed

If your blogging service doesn’t already do this, it’s time to get a new blogging service…

Step 3 – getting an email about the change

Both Notify.me and Notifixio.us can email you if an RSS feed changes. They can take up to an hour to respond to changes, so this is the bottleneck in the flow. On the other hand, it is very simple to set one of these services up. There are other notification services, but they tend to require the writing of some code to work with them.

Assuming you’re using Notify.me, the subject of the email will be the title of the blog post prepended by “notify.me” and a space, and the body of the email will be the body of the blog post appended by a load of garbage advertising Notify.me.

Step 4 – responding to the incoming email

I expect that your email client allows you to set up filters to respond to incoming emails automatically. If it doesn’t, as above, it is time to get a new email client… Assuming you are using Notify.me as the notification service, you can filter out the incoming mail by creating a filter that selects email with a ‘FROM’ email address including ‘@notify.me’. Of these, you can get to the emails about your photo blog by selecting those whose subject includes your blog title. Set up the mail filter to forward to the email address you’ll get for the Tarpipe workflow we’re about to create.

Step 5 – the Tarpipe workflow

Tarpipe is a service for setting up workflows that respond to emails or API calls. Generally, the output will be an update to Twitter, Facebook or some other service that receives messages. Because a Tarpipe workflow can be kicked-off using email, and because you can post the email to any web service, it has opened the door to creating zero-code systems that are able to do interesting things. In short, hurray for Tarpipe.

We want to create a Tarpipe workflow that receives an email, posts it to Yahoo! Pipes, takes the response and tweets it. This is a simple chain of the “MailDecoder”, “REST” (with a “TextInput” connector to supply the URL to post to) and “TwitterUpdater” connectors. When you save the workflow for the first time, you’ll get the email address to use in the email filter from the previous step.

At the moment, Tarpipe has a shortcoming whereby the REST connector, which is what is used to send messages to any web service of your choosing, encodes the information in a structure that is very difficult to use directly with Yahoo! Pipes, despite Pipes’ obvious utility providing the string manipulation capabilities that Tarpipe lacks. You can use an intermediate service, such as this proxy, to munge the data into a more suitable format. The proxy takes a parameter in the URL to specify the ID of the Yahoo! Pipe to use. To get that ID, we’ll need to create a Yahoo! Pipe.

(That proxy URL in full: http://tarpipe-yahoopipes-proxy.smart.joyent.com)

Step 6 – getting at the blog post URL with a Yahoo! Pipe

Pipes has been around for a while now, and I believe it is mainly used to grab remote RSS feeds and other data, perform some operations such as selecting items from a feed, sorting them, turning them into Japanese and sending on the resulting feed. However, you can also use Pipes as a text manipulation machine, posting in some strings of text rather than grabbing them from somewhere else.

We’re going to create a Pipe to take the large and messy email body, delicately extract the URL of the blog post with our photo on, and append that to the subject of the email. We’ll also remove the string “Fwd: notify.me” (plus space) from the subject of the email.

You can get access to any of a list of parameters, posted as text to a Pipe, by adding a “Text Input” module from the “User inputs” list, one for each named parameter being sent, with the ‘name’ property of each module set to the name of the parameter. Using the proxy linked to in the previous step, the subject of the email will be posted to the Pipe as ‘title’ and the body will be posted as ‘description’. Set up your text input modules as shown below (only the ‘name’ property matters).

Now we have access to the strings, we have to extract the relevant strings from them. String extraction can be achieved directly by using the ‘string regex’ module, which stands for ‘string regular expression’. “Regex“, or “Regular expressions” is a language for describing patterns, such as “four letters, followed by a space, followed by a letter or a number”.

The string regex module doesn’t actually extract strings, it replaces any matches of a pattern with another string. However, we can make this a string extractor by replacing a pattern of “anything, followed by the pattern we want, followed by anything” with “the pattern we want”.

In regex, that looks like this:

replace: (?s).*(<the pattern we want>).*
with: $1

An explanation of the above runs like this: –
(?s) means “treat the string as a single-line” – without it, the pattern matching would stop at a line-break;
.* means “match zero-or-more of any character”;
(<the pattern we want>) puts <the pattern we want> into a “group”, to be used in…;
$1 means “the first group from the pattern”

The title is the simpler case, where the pattern we want to extract is [My Mobile Blog] - <the rest of the title>, from a string that looks like “Fwd: notify.me [My Mobile Blog] – super new photo”. To get what we want, connect a string regex module to the text input module named ‘title’ and put the following in the ‘replace’ and ‘with’ properties:

replace: (?s).*Fwd: notify\.me (\[My Mobile Blog\] - .*)
with: $1

The places where there is a \ preceding another character are examples of “escaping” characters that would otherwise have special meaning in regex. For example, [ would normally mean “start a set of characters to choose from”, so \[ just means “match [“.

The more complicated extraction is getting the URL from the email body. Here we want to get a URL of the form http://c.notify.me/F6d3HG out of a much longer string of text. This URL points to an address on Notify.me’s URL-shortening service, that will direct you to the blog post’s much longer URL. Connect a new string regex module to the text input module named ‘description’ and put the following in the ‘replace’ and ‘with’ properties:

replace: (?s).*(http:\/\/c\.notify\.me/\w{6}).*
with: $1

The \w{6} pattern means “6 word-characters”. A “word-character” is something you would expect to find in a word, rather than punctuation or a space. We don’t need to worry too much about what is and what isn’t a word-character, as we only expect to find letters, numbers and “_” in the 6 characters making up the end of the URL, which are definitely word-characters.

(If this has piqued your curiosity, there is a lot of information on regular expressions at http://regular-expressions.info.)

We’re nearly done. Now we have two strings that we want to join together as our tweet. To do this, use the ‘string builder’ module and assemble them as “<title>” plus ” – ” plus “<URL>”. Before outputting this string, we’re going to create a new ‘item’, which is the unit of output from a Pipe – use the “item builder” module, setting the ‘title’ of the new item as the tweet string.

With that all done, we can save the Pipe. By clicking on the ‘run this pipe’ link, you can find the ID of the pipe in the URL of the page you’re taken to, named “_id”. To use this ID in the Tarpipe workflow, in the text input module which is connected to the REST connector, add the ID to the URL as a query-string parameter named “pipeId”: http://<proxy>/?pipeId=... . (Note the / between the domain of the proxy and the ?.)

Step 7 – tweeting from Tarpipe

If you have set up your pipe as above, the text to use for the tweet will be available from the ‘title’ terminal on the right-hand side of the REST connector. Wire this to the ‘title’ terminal of aTwitterUpdater connector.

You need to set up your Twitter account in the ‘accounts’ tab so Tarpipe can tweet with it. At the moment, you need to give out your Twitter username and password, which is the dreaded anti-pattern, and since Twitter now supports OAuth, it makes sense for Tarpipe to use that to get the appropriate level of access to your account. For the purposes of this example, you could set up a new twitter account to link to Tarpipe.

Save the workflow, and you’re done! If everything is configured correctly, sending a photo to your mobile blog will cause a tweet to appear on your timeline, linking back to your new blog post.

Testing the bugger

With a total of 5 remote systems (6 with the Tarpipe / Yahoo! Pipes proxy) between you and the intended tweet, there’s a lot of scope for things to go wrong. The weakest parts of the chain are going to be those you set up yourself, particularly the regex-heavy Yahoo! Pipe, however errors can happen at any point in the system, and problems caused by delays, caching or rate-limits can manifest themselves in various ways.

It’s a good idea to set up test accounts for any services you are hooking into that you use on a day-to-day basis, especially if they publish information to your friends or the public. If you don’t, as I didn’t for my Twitter account, you could end up with something like this mess:

The trick to understanding what is going on in these remote systems is to log activity whenever you get the chance. One useful service for this is PostBin, which lets you create a logger that records anything you post to it.

In the flow above, opportunities to post to this log are:

  1. Adding another REST connector to post what is received by the Tarpipe workflow
  2. Adding a ‘web service’ module to post what is received by the Yahoo! Pipe
  3. Adding a ‘web service’ module to post what comes out of the string extraction steps
  4. Adding another REST connector to post what is returned to Tarpipe from the Yahoo! Pipe

When using the Tarpipe / Yahoo! Pipes proxy, adding &postBinId=<id> to the URL in the text input which connects to the REST connector will cause the proxy to log at these four points:

  1. What comes into the proxy
  2. What is sends Yahoo! Pipes
  3. What comes back from Yahoo! Pipes
  4. What it sends back to Tarpipe

Here’s an example of the PostBin logs you get from the proxy:

As well as logging, some of the services mentioned provide help with the testing and debugging process:

  • Gmail puts mail it forwards due to filters into your ‘sent items’; the exact text sent can be found by selecting ‘show original’ from the ‘reply’ drop-down in an email
  • Tarpipe provides an ‘activity’ tab that shows what came in and went out (note that sometimes you have to click on ‘all’ to see workflow activity, particularly when workflows only contain REST connectors)
  • Yahoo! Pipes has an in-built debugger that allows you to examine the effect of your modules on the text you put into the ‘debug’ property of a user input
  • RegexPal lets you test out a regex on some text (such as the body text of your email)

Despite all these tools, debugging a chain of systems you have no control over can be a painful process. Be particularly careful when using Yahoo! Pipes, as it is not uncommon for results of a Pipe to be cached for 30 minutes. I have experienced this happening even if you change the input data. Lastly, note that should an error actually occur in your flow, it’s likely you’re not going to be told about it…

Good luck!

Advertisements

5 Comments

  1. Posted July 30, 2009 at 12:29 pm | Permalink

    while this is brilliant and inspired saying that it’s a no coding solution and then hacking regular expressions is a bit much for my string munged brain.

  2. Posted July 30, 2009 at 3:05 pm | Permalink

    This is amazing! Gonna have to try it myself.

  3. Posted July 30, 2009 at 3:13 pm | Permalink

    Nicely done, Jay. I like to call this sort of thing a “nothing-but-net” solution, since you don’t have to own any actual server space to make it work.

  4. Simon
    Posted July 31, 2009 at 12:23 pm | Permalink

    Can pipes read an rss feed? This might shorten your line of domino

    • Posted August 5, 2009 at 10:51 am | Permalink

      What do you have in mind? The problem is that something needs to trigger the Pipe.


One Trackback/Pingback

  1. […] Jay by Jay Fresh Right-on slack-jawed yokel-type tech-farmer « How to build a DIY TwitPic (without any coding skillz) […]