Learning About Specific Web APIs

In the previous section, I showed you how to call web APIs that use XML-­RPC and SOAP. That still leaves many APIs that fall under the name of REST—ones that look a lot like the Flickr REST interface. These APIs take some things we are familiar with from web browsers, such as going to a specific URL to get back some results and submitting HTML forms to make a query, but they have one important difference: instead of sending mostly HTML (which is directed at human consumption), you send primarily XML, a lingua franca of computer data exchange. For that reason, you should remind yourself of what you’ve learned from the previous chapters as you embark on a study of other REST APIs.

In the following sections, I’ll make sense of the world of web APIs, covering how to find out what APIs are available and then how to use a particular API. I’ll start my discussion by introducing perhaps the single most useful website about web APIs: Programmableweb.com. There’s a lot of information that is both readily apparent and waiting to be discovered in this treasure trove of data.

[Note]Note

Other directories of web services that are worth exploring are http://www.xmethods.net/, which lists publicly available SOAP services, and http://strikeiron.com/, a provider of commercial web services that you can try for free.

Programmableweb.com

Programmableweb.com, started and maintained by Jon Musser, is an excellent resource for learning about what APIs are available, the basic parameters for the APIs, and the mashups that use any given API. Some noteworthy sections are as follows:

  • http://www.programmableweb.com/apis is the “API dashboard” that lists the latest APIs to be registered and the most popular APIs being used in mashups.

  • ht tp://www.programmableweb.com/apilist/bycat lists APIs by categories. Understanding the various categories that have emerged is helpful for understanding for which fields of endeavor people are making APIs.

  • http://www.programmableweb.com/apilist/bymashups lists APIs by how many times they are used in the mashups registered at Programmableweb.com.

I highly recommend a close and periodic study of Programmableweb.com for anybody wanting to learn about web APIs and mashups. Let me show some of the things you can learn from the website, based both on what is directly presented on the site and on data that John Musser has sent me. Although web APIs and corresponding mashups are rapidly changing, the data (and derived results), accurate for August 11, 2007, demonstrates some trends that I think will hold for a while yet.

The first thing to know is that of the 494 web APIs in the database, we get the distribution of number of APIs by protocol supported shown in Table 7-2. Note that some APIs are multiply represented.

Table 7.2. Table 7-2. Number of APIs vs. Protocol in Programmableweb.com
Protocol Number of APIs with Support
REST 255
SOAP 131
XML-RPC 19
JavaScript 30
Other 16

Some other observations drawn from the database are as follows:

  • Ninety-three APIs have WSDL files associated with them.

  • Of the 131 APIs that support SOAP, 42 also support REST—leaving 89 that support SOAP but not REST. Eighty-­eight APIs support only SOAP.

  • XML-RPC is the only choice for nine APIs.

  • JavaScript is listed as the exclusive protocol for 25 APIs.

The following are my conclusions based on this data:

  • REST is the dominant mode of presenting web APIs, but a significant number of APIs exist where your only choice is SOAP.

  • There are a relatively small number of APIs listing XML-­RPC as the only choice of protocol.

It’s therefore useful to know how to use SOAP and XML-­RPC, even if they are not your first choice.

[Note]Note

 A large number of APIs list JavaScript as a protocol. I’ll cover such APIs in the next chapter.

Table 7-3 lists the top 20 APIs on Programmableweb.com by mashup count and also lists the type of protocols supported by the API.

Table 7.3. Table 7-3. Top 21 APIs by Mashup Count
API Name Number of Mashups Protocols Support
Google Maps 1110 JavaScript
Flickr 243 REST, SOAP, XML-­RPC
Amazon E-­Commerce Service 174 REST, SOAP
YouTube 149 REST, XML-­RPC
Microsoft Virtual Earth 97 JavaScript
Yahoo! Maps 95 REST, JavaScript, Flash
411Sync 89 RSS input over HTTP, SOAP
eBay 89 SOAP, REST
del.icio.us 83 REST
Google Search 79 SOAP
Yahoo! Search 78 REST
Yahoo! Geocoding 66 REST
Technorati 40 REST
Yahoo! Image Search 31 REST
Yahoo! Local Search 30 REST
Last.fm 28 REST
Google home page 27 JavaScript
Google Ajax Search 24 JavaScript
Upcoming.org 21 REST
Windows Live Search 21 SOAP
Feedburner 21 REST

What can you do with this information? To learn about popular APIs, one approach would be to go down the list systematically to figure out how each works. Indeed, through the rest of the book, I’ll cover many of the APIs in the table. The Flickr API is the second most used API in mashups and is a main subject throughout this book. I’ll cover the JavaScript-­based maps (first and foremost Google Maps but also Yahoo! Maps and Virtual Earth) first in Chapter 8 and then in depth in Chapter 13. I’ll cover the Yahoo! Geocoding API extensively also in Chapter 13. I’ll cover various search APIs (Google Search, Yahoo! Search, Yahoo! Image Search, and Windows Live Search) in Chapter 19. Finally, I’ll cover the del.icio.us API in Chapter 14 on social bookmarking. Indeed, the fact that I cover many APIs clustered by subject matter indicates that it is a natural way to think about APIs.

YouTube

YouTube is probably the most famous video-­sharing site on the Web—and it also uses tagging as one way of organizing content. The YouTube API is documented at http://www.youtube.com/dev.

The YouTube API supports both a REST interface and an XML-­RPC interface. The examples I give in this section use the REST interface. You can find a list of methods at http://?www.youtube.com/dev_docs.

To use the API, you need to set up your own development profile; see http://www.youtube.com/ ?my_profile_dev.

An interesting feature of the registration process is that you enter your own secret (instead of having one set by YouTube). When you submit your profile information, you then get a “developer ID.” The following are some sample calls. To get the user profile for a user (for example, rdhyee), you do an HTTP GET on the following:

http://www.youtube.com/api2_rest?method=youtube.users.get_profile&dev_id={youtube-key}&user=rdhyee
         

YouTube will send you a response something like this:

<?xml version="1.0" encoding="utf-8"?>
<ut_response status="ok">
  <user_profile>
    <first_name>Raymond</first_name>
    <last_name/>
    <about_me/>
    <age>40</age>
    <video_upload_count>2</video_upload_count>
    <video_watch_count>102</video_watch_count>
    [....]
  </user_profile>
</ut_response>
         

To get the list of rdhyee’s favorite videos, use this:

http://www.youtube.com/api2_rest?method=youtube.users.list_favorite_videos&dev_id={youtube-key}&user=rdhyee
         

To get details of a video with an ID of XHnE4umovw4, use this:

http://www.youtube.com/api2_rest?method=youtube.videos.get_details&dev_id={youtube-key}&video_id=XHnE4umovw4
         

To get videos for the tag HolidayWeekend, use this:

http://www.youtube.com/api2_rest?method=youtube.videos.list_by_tag&dev_id={youtube-key}&tag=HolidayWeekend&page=1&per_page=100
         

There’s more to the API, but you can get a feel for how it works through these examples.

[Caution]Caution

 Expect the YouTube API to evolve into something more like the rest of Google’s APIs: http://?code.google.com/apis/youtube/overview.html.

GData and the Blogger API

The Atom Publishing Protocol (APP), a companion to the Atom Syndication Format (Atom 1.0) described in Chapter 2, represents the next generation of the blogging APIs. APP is currently a draft IETF proposal:

http://tools.ietf.org/wg/atompub/draft-ietf-atompub-protocol/

which is linked from here:

http://tools.ietf.org/wg/atompub/

One of APP’s biggest supporters thus far has been Google, which has implemented GData, which is based on Atom 1.0 and RSS 2.0 feeds, combined with APP. GData, which incorporates Google-­specific extensions, is the foundation of the APIs for many of its services, including Google Base, Blogger, Google Calendar, Google Code Search, and Google Notebook:

http://code.google.com/apis/gdata/index.html

The API for Blogger is documented here:

http://code.google.com/apis/blogger/developers_guide_protocol.html

In the following sections, you’ll learn the basics of the API for Blogger as a way of understanding GData and APP in general.

Obtaining an Authorization Token

The first thing you need to have is a Google account to use Blogger. If you don’t have one, go to the following location to create one:

https://www.google.com/accounts/NewAccount

Next, with a Google account, you obtain an authorization token. One way to do so is to follow the procedure for ClientLogin (one of two Google authorization mechanisms) detailed here:

http://code.google.com/apis/blogger/developers_guide_protocol.html#client_login

Specifically, you make an HTTP POST request to the following location:

https://www.google.com/accounts/ClientLogin

The body must contain the following parameters (using the application/? x-www-form-urlencoded content type):

Email: Your Google email (for example, raymond.yee@gmail.com)

Password: Your Google password

source: A string of the form companyName-applicationName-versionID to identify your program (for example, mashupguide.net-Chap7-v1)

service: The name of the Google service, which in this case is blogger

Using the example parameters listed here, you can package up the authorization request as the following curl invocation:

curl -v -X POST -d "Passwd={passwd}&source=mashupguide.net-Chap7-v1&Email=raymond.yee%40gmail.com&service=blogger" https://www.google.com/accounts/ClientLogin
            

If this call succeeds, you will get in the body of the response an Auth token (of the form Auth=[AUTH-TOKEN]). Retain the AUTH-TOKEN for your next calls.

Figuring Out Your Blogger User ID

If you don’t have a blog on Blogger.com, create one here:

http://www.blogger.com/create-blog.g

Now figure out your Blogger user ID by going to and noting the URL associated with the View link (beside the Edit Profile link):

http://www.blogger.com/home

Your View link will be of the following form:

http://www.blogger.com/profile/{userID}

For example, my blog profile is as follows:

http://www.blogger.com/profile/13847941708302188690

So, my user ID is 13847941708302188690.

Getting a List of a User’s Blogs and a Blog’s Posts

Note that Blogger lists user blogs in a user’s profile:

http://www.blogger.com/profile/{userID}

From an API point of view, you can retrieve an Atom feed of a user’s blog here:

http://www.blogger.com/feeds/{userID}/blogs

That the list of blogs is an Atom feed and not some custom-­purpose XML (such as that coming out of the Flickr API) is useful. You can look at the feed of your blogs to pull out the blog ID for one of your blogs. For instance, one of my blogs is entitled “Hypotyposis Redux” and is listed in the feed with the following tag:

<id>tag:blogger.com,1999:user-354598769533.blog-5586336</id>

From this you can determine its blog ID of 5586336. With this blogID, you can now send HTTP GET requests to retrieve an Atom feed of posts here:

http://www.blogger.com/feeds/{blogID}/posts/default

For example:

http://www.blogger.com/feeds/5586336/posts/default

Creating a New Blog Entry

Let’s now look at how to create a new post. A central design idea behind the Atom Publishing Protocol and hence its derivatives—GData generally and the Blogger API specifically—is the notion of a uniform interface based on the standard HTTP methods. At this point, it’s useful to refer to the “Methods Definition” of the HTTP 1.1 specification (http://www.w3.org/Protocols/ ?rfc2616/rfc2616-sec9.html), specifically the definition for POST:

The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-­URI in the Request-­Line.

You may be surprised to read this definition of the POST method, considering, for instance, how POST is used for every single SOAP 1.1 call bound to HTTP—whether the call is for retrieving a simple piece of information, creating a new resource, or deleting it.

Let’s see how to create an HTTP POST request to create a new blog entry and confirm how the process follows the earlier definition:

  1. Create a blog entry formatted as an <entry> Atom element, something like this:

    <entry xmlns='http://www.w3.org/2005/Atom'>
      <title type='text'>Using Blogger to demo APP</title>
      <content type='xhtml'>
        <div xmlns='http://www.w3.org/1999/xhtml'>
          <p>This message is being created from invoking the blogger 
    APP-based API.</p>
          <p>This process is documented at <a href="http://code.google.com/
    apis/blogger/developers_guide_protocol.html#CreatingPosts">Blogger Data 
    API -- Creating Posts</a></p>
        </div>
      </content>
      <author>
        <name>Raymond Yee</name>
        <email>raymond.yee@gmail.com</email>
      </author>
    </entry>
                      
  2. Save the file, say with the filename blogger.message.1.xml.

  3. Issue the following curl invocation to POST the contents of your file to http://? www.blogger.com/feeds/{blogID}/posts/default—which is the feed of all your entries for the blog—to create a new entry for the blog (a “new subordinate of the resource identified by the Request-­URI”).

    curl -X POST -v --data-binary "@blogger.message.1.xml" -H "Content-Type:application/atom+xml " -H "Authorization: GoogleLogin  auth=[AUTH-TOKEN]" http://www.blogger.com/feeds/{blogID}/posts/default
                      
  4. If things go fine, you’ll get an HTTP 201 Created code and an <entry> holding the new post. This <entry> tells you things such as the post ID of your new entry. The response will look like the following:

    HTTP/1.1 201 Created
    Content-Type: application/atom+xml; charset=UTF-8
    Cache-Control: ?max-?age=0, ?must-?revalidate, private
    Location: http://www.blogger.com/feeds/5586336/posts/default/409227349217351
    7704
    Content-Location: http://www.blogger.com/feeds/5586336/posts/default/4092273
    492173517704
    Transfer-Encoding: chunked
    Date: Sat, 25 Aug 2007 14:38:49 GMT
    Server: GFE/1.3
    <?xml version='1.0' encoding='UTF-8'?><?xml-stylesheet href="http://www.b
    logger.com/styles/atom.css" type="text/css"?><entryxmlns='http://www.w3.o
    rg/2005/Atom'><id>tag:blogger.com,1999:blog-5586336.post-4092273492173517
    704</id><published>2007-08-25T07:38:00.001-07:00</published><updated>2007
    -08-25T07:38:49.607-07:00</updated><title type='text'>Using Blogger to de
    mo APP</title><content type='html'>&lt;br /&gt;    &lt;div xmlns='http://
    www.w3.org/1999/xhtml'&gt;&lt;br /&gt;
       &lt;p&gt;This message is being created from invoking the blogger APP-
    based API.&lt;/p&gt;&lt;br /&gt;      &lt;p&gt;This process is documented 
    at &lt;a href='http://code.google.com/apis/blogger/developers_guide_proto
    col.html#CreatingPosts'&gt;Blogger Data API -- Creating Posts&lt;/a&gt;&l
    t;/p&gt;&lt;br /&gt;    &lt;/div&gt;&lt;br /&gt;  </content><link rel='al
    ternate' type='text/html' href='http://hypotyposis.blogspot.com/2007_08_0
    1_archive.html#4092273492173517704' title='Using Blogger to demo APP'/><l
    ink rel='replies' type='applicati* Connection #0to host www.blogger.com l
    eft intact* Closing connection #0
                      
  5. In this example, the POST request created a new blog entry with a post ID of 40922734921735177.

Updating the Blog Entry

You can update your blog entry using an HTTP PUT request, in accordance to the HTTP 1.1 specification that states the following:

The PUT method requests that the enclosed entity be stored under the supplied Request-­URI. If the Request-­URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server.

Let’s package this request for curl after first creating an updated message in the blogger.message.2.xml file:

curl -X PUT -v --data-binary "@blogger.message.2.xml" -H "Content-Type:application/atom+xml " -H "Authorization: GoogleLogin  auth=[AUTH-TOKEN]" http://www.blogger.com/feeds/{blogID}/posts/default/{postID}
            

If you are unfamiliar with using the HTTP PUT method, you’re hardly alone. As mentioned in Chapter 6, there is little support for it. (Remember, for instance, that the HTML forms define the GET and POST methods.) Recognizing that PUT might not be supported by the client doing the entry update (or that firewalls might block PUT requests), you can tunnel the PUT request through a POST request like so:

curl -X POST -v --data-binary "@blogger.message.2.xml" -H "X-HTTP-Method-Override:PUT"  -H "Content-Type: application/atom+xml " -H "Authorization: GoogleLogin auth=[AUTH-TOKEN]" http://www.blogger.com/feeds/{blogID}/posts/default/{postID}
            

Deleting a Blog Entry

You can use the HTTP DELETE method to delete an entry but send that request to the URL of the entry itself. As a curl invocation, the request looks like this:

curl -X DELETE -v -H "Content-Type: application/atom+xml " -H "Authorization: GoogleLogin  auth=[AUTH-TOKEN]" http://www.blogger.com/feeds/{blogID}/posts/default/{postID}
            

As with updating a blog entry, you can tunnel a DELETE request through an HTTP POST request using an “X-HTTP-Method-Override: DELETE” request header.

Using the Blogger API As a Uniform Interface Based on HTTP Methods

Now that you have seen how to use the Blogger API to retrieve feeds of blogs and blog entries, create new blog entries, update an entry, and delete an entry, you should notice how all these actions are performed while hewing closely to HTTP methods as they are actually defined in the HTTP specification. This pattern of using the HTTP methods as the fundamental methods of the API, in fact, repeats itself in all the APIs that are based on the Atom Publishing Protocol and therefore GData. Thus, the uniform interface of GData is the full collection of standard HTTP methods.