Remixing Feeds with Yahoo! Pipes

Yahoo! Pipes (http://pipes.yahoo.com/pipes/) is a “an interactive data aggregator and manipulator that lets you mash up your favorite online data sources.” Yahoo! Pipes is focused on enabling end users to filter and combine feeds into new feeds. You construct pipes through dragging and dropping graphical widgets (called modules), entering parameters, and describing data flows through wiring these widgets together. Yahoo! Pipes is arguably more accessible to nonprogrammers because it does not involve typing code in a text editor. You’ll see in practice whether the masses will be making mashups with Yahoo! Pipes.

[Note]Note

I will say that as a programmer, Yahoo! Pipes does make it easier to remix feeds in many instances and got me to create feeds that I could have created programmatically but was not inspired to do so without the Yahoo! Pipes environment.

In this section, I describe how I built a series of pipes to solve a specific problem. In doing so, I hope to shed light on how to think about Yahoo! Pipes, specifically how to construct increasingly more complicated structures. For the basics of Yahoo! Pipes, please consult the official documentation:

http://pipes.yahoo.com/pipes/docs

especially the documentation of the modules available in Yahoo! Pipes:

http://pipes.yahoo.com/pipes/docs?doc=modules

The problem I address with Yahoo! Pipes is creating a single feed from diverse news sources, unified around a single topic or search term. In constructing my pipes, I had a concrete scenario in mind. I wanted a feed that enables one to follow the latest news about the aftermath of Hurricane Katrina. Though I generalized my Yahoo! Pipes where I could easily do so, I am not attempting here to develop a comprehensive solution.

The solution I devised was to synthesize a feed out of the following four sources:

This range of new sources enables me to illustrate how to overcome some of the challenges you’ll likely face when using Yahoo! Pipes.

A Simple First Pipe with Yahoo! News

The first step I took was to build a pipe to handle the first source, Yahoo! News. I exploited the fact that you can generate an RSS Yahoo! News for a search term with this:

http://news.search.yahoo.com/news/rss?p={search-term}

I built two versions of a pipe to return a feed for a given search term. The first version—called “Yahoo! News by Search Term (First Version)”—is here:

http://pipes.yahoo.com/pipes/pipe.info?_id=Rg_rh3NA3BGdECIel7okhQ

You can run it and view the source if you are logged in. You can run a pipe to get an RSS 2.0 feed for a given search term here:

http://pipes.yahoo.com/pipes/pipe.run?_id={pipe-id}&_render=rss&search_term={search-term}

For example, to search for Hurricane Katrina, go here:

http://pipes.yahoo.com/pipes/pipe.run?_id=Rg_rh3NA3BGdECIel7okhQ&_render=rss&search_term=Hurricane+Katrina

The pipe uses three widgets to enable a user to pass in a search term and return a feed of Yahoo! News (see Figure 4-3):

  • A Text Input module that takes the search term from the user and is wired to feed this term to the URL Builder described next

  • A URL Builder module that has a Base parameter of http://news.search.yahoo.com/news/rss and a query parameter of p

  • A Fetch Feed that fetches the feed at the URL coming from the URL Builder


The second news source that I need to pull in is Google News, which returns an Atom feed for a given search term here:

http://news.google.com/news?q={search-term}&output=atom

I use output=atom instead of output=rss to show that Yahoo! Pipes can handle Atom feeds.

One way to build a module to handle Google News is to clone the one for Yahoo! News and change the parameters in the URL Builder module. Instead, because I figured that there are plenty of feeds with URLs that consist of a single parameter and search term, I decided to build a utility pipe that would return feeds at URLs in the following form:

{base-URL}?{parameter_name}={parameter_value}{URL_suffix}

Constructing such a pipe is equivalent to writing a reusable function. By contrast, cloning a pipe is analogous to copying and pasting code. When you use pipes seriously, you begin to see patterns that can be captured and reused in a pipe.

The pipe I constructed for that purpose (called “Feed from a URL Constructed from One Variable Parameter”) is located here:

http://pipes.yahoo.com/pipes/pipe.info?_id=VoLceXZA3BGkqcJZJxOy0Q

Note the intermixing of the URL Builder and String Builder modules to concatenate parameters and build a URL that can actually be fed to Fetch Feed module.

I used that pipe and the fact you can write the URLs to retrieve feeds from Yahoo! News and Google News in terms of these four parameters (see Table 4-5).

Table 4.5. Parameters for Pipe Called “Feed from a URL Constructed from One Variable Parameter”
base-URL parameter_name parameter_value URL_suffix
http://news.google.com/newsq{search-term} &output=atom
http://news.search.yahoo.com/news/rssp {search-term}

The pipe that accesses Yahoo! News using the utility pipe (entitled “Yahoo! News by Search Term”) is as follows:

http://pipes.yahoo.com/pipes/pipe.info?_id=5NhmMndA3BGg5zQ5n0artA

The “Google News by Search Term” pipe is here:

http://pipes.yahoo.com/pipes/pipe.info?_id=OKWv6nNA3BGkPtA8qWIyXQ

Wikinews and NY Times: Filtering Feeds

Now we come to the third and fourth sources: Wikinews, which has a single feed:

http://feeds.feedburner.com/WikinewsLatestNews

and the New York Times National News (I select the National News feed because of the focus on Hurricane Katrina):

http://www.nytimes.com/services/xml/rss/nyt/National.xml

In contrast to the Yahoo! News and Google News for which I can generate a feed for a given search term by constructing the appropriate URL, I use search-term to filter the feed. To that end, I use the Filter module in the Filter for Given Term in Feed Description and Title pipe here:

http://pipes.yahoo.com/pipes/pipe.info?_id=KIYSv3pA3BGgloVbCB2yXQ

to pass along only those items in the feed whose description or title contains the search term.

With that pipe as a foundation, I construct the “Wikinews Filtered by Search Term” pipe here:

http://pipes.yahoo.com/pipes/pipe.info?_id=PA7iqHpA3BGb0AiVX0sBXw

and the NY Times National News Filtered by a Search Term pipe here:

http://pipes.yahoo.com/pipes/pipe.info?_id=yhBh7HxA3BGu_YRj1vC6Jw

Pulling the Feeds Together

With a pipe each for my four news sources, each of which takes a search term as input, I then create a pipe here:

http://pipes.yahoo.com/pipes/pipe.info?_id=qlUkcn1A3BGeWlNQjknRlg

that does the following:

  • Takes a search term as input and passes it to the individual news source pipes

  • Concatenates the individual feeds with the Union module

  • Gets rid of feed items that have the same link with the Unique module

You’re done. You can search for Hurricane Katrina in the four news sources here:

http://pipes.yahoo.com/pipes/pipe.info?Search_term=Hurricane+Katrina&_cmd=Run+Pipe& _id=qlUkcn1A3BGeWlNQjknRlg&_run=1

The corresponding feed is available here:

http://pipes.yahoo.com/pipes/pipe.run?Search_term=Hurricane+Katrina&_id=qlUkcn1A3BGeWlNQjknRlg&_render=rss

Obviously, the pipes I created could be refined. Indeed, you can do so right now by going to the list of pipes I created for this chapter and cloning and modifying them for your own use:

http://pipes.yahoo.com/pipes/tag.info?namespace=user&tag=mashupguide

This section used feeds that are relatively easy to access. If you start to use tools such as Openkapow.com to screen-­scrape new feeds, you have even more combinatorial possibilities for your mashups. Note that other tools that I discuss later in the book (such as the Google Mashup Editor) are able to consume feeds, such as those generated by Yahoo! Pipes and Feedburner.