Yahoo! Pipes (http://pipes.yahoo.com/pipes/
) is a “an interactive data aggregator and manipulator that lets you mash up your favorite online data sources.” Yahoo! Pipes is focused on enabling end users to filter and combine feeds into new feeds. You construct pipes through dragging and dropping graphical widgets (called modules), entering parameters, and describing data flows through wiring these widgets together. Yahoo! Pipes is arguably more accessible to nonprogrammers because it does not involve typing code in a text editor. You’ll see in practice whether the masses will be making mashups with Yahoo! Pipes.
![]() | Note |
---|---|
I will say that as a programmer, Yahoo! Pipes does make it easier to remix feeds in many instances and got me to create feeds that I could have created programmatically but was not inspired to do so without the Yahoo! Pipes environment. |
In this section, I describe how I built a series of pipes to solve a specific problem. In doing so, I hope to shed light on how to think about Yahoo! Pipes, specifically how to construct increasingly more complicated structures. For the basics of Yahoo! Pipes, please consult the official documentation:
http://pipes.yahoo.com/pipes/docs
especially the documentation of the modules available in Yahoo! Pipes:
http://pipes.yahoo.com/pipes/docs?doc=modules
The problem I address with Yahoo! Pipes is creating a single feed from diverse news sources, unified around a single topic or search term. In constructing my pipes, I had a concrete scenario in mind. I wanted a feed that enables one to follow the latest news about the aftermath of Hurricane Katrina. Though I generalized my Yahoo! Pipes where I could easily do so, I am not attempting here to develop a comprehensive solution.
The solution I devised was to synthesize a feed out of the following four sources:
Yahoo! News
Google News
Wikinews
The national section of The New York Times
This range of new sources enables me to illustrate how to overcome some of the challenges you’ll likely face when using Yahoo! Pipes.
The first step I took was to build a pipe to handle the first source, Yahoo! News. I exploited the fact that you can generate an RSS Yahoo! News for a search term with this:
http://news.search.yahoo.com/news/rss?p={search-term}
I built two versions of a pipe to return a feed for a given search term. The first version—called “Yahoo! News by Search Term (First Version)”—is here:
http://pipes.yahoo.com/pipes/pipe.info?_id=Rg_rh3NA3BGdECIel7okhQ
You can run it and view the source if you are logged in. You can run a pipe to get an RSS 2.0 feed for a given search term here:
http://pipes.yahoo.com/pipes/pipe.run?_id={pipe-id}&_render=rss&search_term={search-term}
For example, to search for Hurricane Katrina, go here:
The pipe uses three widgets to enable a user to pass in a search term and return a feed of Yahoo! News (see Figure 4-3):
A Text Input
module that takes the search term from the user and is wired to feed this term to the URL Builder
described next
A URL Builder
module that has a Base
parameter of http://news.search.yahoo.com/news/rss
and a query parameter of p
A Fetch Feed
that fetches the feed at the URL coming from the URL Builder
Figure 4.3. Figure 4-3.Pipe for “Yahoo! News by Search Term (First Version)”. (Reproduced with permission of Yahoo! Inc. ® 2007 by Yahoo! Inc. YAHOO! and the YAHOO! logo are trademarks of Yahoo! Inc.)
The second news source that I need to pull in is Google News, which returns an Atom feed for a given search term here:
http://news.google.com/news?q={search-term}&output=atom
I use output=atom
instead of output=rss
to show that Yahoo! Pipes can handle Atom feeds.
One way to build a module to handle Google News is to clone the one for Yahoo! News and change the parameters in the URL Builder
module. Instead, because I figured that there are plenty of feeds with URLs that consist of a single parameter and search term, I decided to build a utility pipe that would return feeds at URLs in the following form:
{base-URL}?{parameter_name}={parameter_value}{URL_suffix}
Constructing such a pipe is equivalent to writing a reusable function. By contrast, cloning a pipe is analogous to copying and pasting code. When you use pipes seriously, you begin to see patterns that can be captured and reused in a pipe.
The pipe I constructed for that purpose (called “Feed from a URL Constructed from One Variable Parameter”) is located here:
http://pipes.yahoo.com/pipes/pipe.info?_id=VoLceXZA3BGkqcJZJxOy0Q
Note the intermixing of the URL Builder
and String Builder
modules to concatenate parameters and build a URL that can actually be fed to Fetch Feed
module.
I used that pipe and the fact you can write the URLs to retrieve feeds from Yahoo! News and Google News in terms of these four parameters (see Table 4-5).
base-URL | parameter_name | parameter_value | URL_suffix |
http://news.google.com/news | q | {search-term} | &output=atom |
http://news.search.yahoo.com/news/rss | p | {search-term} |
The pipe that accesses Yahoo! News using the utility pipe (entitled “Yahoo! News by Search Term”) is as follows:
http://pipes.yahoo.com/pipes/pipe.info?_id=5NhmMndA3BGg5zQ5n0artA
The “Google News by Search Term” pipe is here:
http://pipes.yahoo.com/pipes/pipe.info?_id=OKWv6nNA3BGkPtA8qWIyXQ
Now we come to the third and fourth sources: Wikinews, which has a single feed:
http://feeds.feedburner.com/WikinewsLatestNews
and the New York Times National News (I select the National News feed because of the focus on Hurricane Katrina):
http://www.nytimes.com/services/xml/rss/nyt/National.xml
In contrast to the Yahoo! News and Google News for which I can generate a feed for a given search term by constructing the appropriate URL, I use search-term
to filter the feed. To that end, I use the Filter
module in the Filter for Given Term in Feed Description and Title pipe here:
http://pipes.yahoo.com/pipes/pipe.info?_id=KIYSv3pA3BGgloVbCB2yXQ
to pass along only those items in the feed whose description or title contains the search term.
With that pipe as a foundation, I construct the “Wikinews Filtered by Search Term” pipe here:
http://pipes.yahoo.com/pipes/pipe.info?_id=PA7iqHpA3BGb0AiVX0sBXw
and the NY Times National News Filtered by a Search Term pipe here:
http://pipes.yahoo.com/pipes/pipe.info?_id=yhBh7HxA3BGu_YRj1vC6Jw
With a pipe each for my four news sources, each of which takes a search term as input, I then create a pipe here:
http://pipes.yahoo.com/pipes/pipe.info?_id=qlUkcn1A3BGeWlNQjknRlg
that does the following:
Takes a search term as input and passes it to the individual news source pipes
Concatenates the individual feeds with the Union
module
Gets rid of feed items that have the same link with the Unique
module
You’re done. You can search for Hurricane Katrina in the four news sources here:
The corresponding feed is available here:
Obviously, the pipes I created could be refined. Indeed, you can do so right now by going to the list of pipes I created for this chapter and cloning and modifying them for your own use:
http://pipes.yahoo.com/pipes/tag.info?namespace=user&tag=mashupguide
This section used feeds that are relatively easy to access. If you start to use tools such as Openkapow.com to screen-scrape new feeds, you have even more combinatorial possibilities for your mashups. Note that other tools that I discuss later in the book (such as the Google Mashup Editor) are able to consume feeds, such as those generated by Yahoo! Pipes and Feedburner.