Remixing Feeds with Yahoo! Pipes

Yahoo! Pipes ( is a “an interactive data aggregator and manipulator that lets you mash up your favorite online data sources.” Yahoo! Pipes is focused on enabling end users to filter and combine feeds into new feeds. You construct pipes through dragging and dropping graphical widgets (called modules), entering parameters, and describing data flows through wiring these widgets together. Yahoo! Pipes is arguably more accessible to nonprogrammers because it does not involve typing code in a text editor. You’ll see in practice whether the masses will be making mashups with Yahoo! Pipes.


I will say that as a programmer, Yahoo! Pipes does make it easier to remix feeds in many instances and got me to create feeds that I could have created programmatically but was not inspired to do so without the Yahoo! Pipes environment.

In this section, I describe how I built a series of pipes to solve a specific problem. In doing so, I hope to shed light on how to think about Yahoo! Pipes, specifically how to construct increasingly more complicated structures. For the basics of Yahoo! Pipes, please consult the official documentation:

especially the documentation of the modules available in Yahoo! Pipes:

The problem I address with Yahoo! Pipes is creating a single feed from diverse news sources, unified around a single topic or search term. In constructing my pipes, I had a concrete scenario in mind. I wanted a feed that enables one to follow the latest news about the aftermath of Hurricane Katrina. Though I generalized my Yahoo! Pipes where I could easily do so, I am not attempting here to develop a comprehensive solution.

The solution I devised was to synthesize a feed out of the following four sources:

This range of new sources enables me to illustrate how to overcome some of the challenges you’ll likely face when using Yahoo! Pipes.

A Simple First Pipe with Yahoo! News

The first step I took was to build a pipe to handle the first source, Yahoo! News. I exploited the fact that you can generate an RSS Yahoo! News for a search term with this:{search-term}

I built two versions of a pipe to return a feed for a given search term. The first version—called “Yahoo! News by Search Term (First Version)”—is here:

You can run it and view the source if you are logged in. You can run a pipe to get an RSS 2.0 feed for a given search term here:{pipe-id}&_render=rss&search_term={search-term}

For example, to search for Hurricane Katrina, go here:

The pipe uses three widgets to enable a user to pass in a search term and return a feed of Yahoo! News (see Figure 4-3):

  • A Text Input module that takes the search term from the user and is wired to feed this term to the URL Builder described next

  • A URL Builder module that has a Base parameter of and a query parameter of p

  • A Fetch Feed that fetches the feed at the URL coming from the URL Builder

The second news source that I need to pull in is Google News, which returns an Atom feed for a given search term here:{search-term}&output=atom

I use output=atom instead of output=rss to show that Yahoo! Pipes can handle Atom feeds.

One way to build a module to handle Google News is to clone the one for Yahoo! News and change the parameters in the URL Builder module. Instead, because I figured that there are plenty of feeds with URLs that consist of a single parameter and search term, I decided to build a utility pipe that would return feeds at URLs in the following form:


Constructing such a pipe is equivalent to writing a reusable function. By contrast, cloning a pipe is analogous to copying and pasting code. When you use pipes seriously, you begin to see patterns that can be captured and reused in a pipe.

The pipe I constructed for that purpose (called “Feed from a URL Constructed from One Variable Parameter”) is located here:

Note the intermixing of the URL Builder and String Builder modules to concatenate parameters and build a URL that can actually be fed to Fetch Feed module.

I used that pipe and the fact you can write the URLs to retrieve feeds from Yahoo! News and Google News in terms of these four parameters (see Table 4-5).

Table 4.5. Parameters for Pipe Called “Feed from a URL Constructed from One Variable Parameter”
base-URL parameter_name parameter_value URL_suffix{search-term} &output=atom {search-term}

The pipe that accesses Yahoo! News using the utility pipe (entitled “Yahoo! News by Search Term”) is as follows:

The “Google News by Search Term” pipe is here:

Wikinews and NY Times: Filtering Feeds

Now we come to the third and fourth sources: Wikinews, which has a single feed:

and the New York Times National News (I select the National News feed because of the focus on Hurricane Katrina):

In contrast to the Yahoo! News and Google News for which I can generate a feed for a given search term by constructing the appropriate URL, I use search-term to filter the feed. To that end, I use the Filter module in the Filter for Given Term in Feed Description and Title pipe here:

to pass along only those items in the feed whose description or title contains the search term.

With that pipe as a foundation, I construct the “Wikinews Filtered by Search Term” pipe here:

and the NY Times National News Filtered by a Search Term pipe here:

Pulling the Feeds Together

With a pipe each for my four news sources, each of which takes a search term as input, I then create a pipe here:

that does the following:

  • Takes a search term as input and passes it to the individual news source pipes

  • Concatenates the individual feeds with the Union module

  • Gets rid of feed items that have the same link with the Unique module

You’re done. You can search for Hurricane Katrina in the four news sources here: _id=qlUkcn1A3BGeWlNQjknRlg&_run=1

The corresponding feed is available here:

Obviously, the pipes I created could be refined. Indeed, you can do so right now by going to the list of pipes I created for this chapter and cloning and modifying them for your own use:

This section used feeds that are relatively easy to access. If you start to use tools such as to screen-­scrape new feeds, you have even more combinatorial possibilities for your mashups. Note that other tools that I discuss later in the book (such as the Google Mashup Editor) are able to consume feeds, such as those generated by Yahoo! Pipes and Feedburner.