Inspecting a Webpage's Traffic

Bypassing the Browser

We just covered a lot of steps, but let's make sure we have the big picture here. The network panel didn't reveal anything we couldn't have found with a thorough inspection of the raw HTML and Javascript files. But we're not HTML/JS experts, and even if we were, the inspector's network panel gives us an easy point-and-click interface to cut through the clutter.

One particular nicety of the network panel is that it lets us see data requests to and from the browser in real time.

Twitter's infinite scroll

For example, Twitter webpages have a feature called infinite scroll. Every time you scroll to the bottom of the page, more tweets appear. You don't even have to push a button or click a link.

If you're logged into Twitter on your browser, log out for now. Now visit a Twitter user's page and keep scrolling to the bottom of their list of tweets. Check out how your network panel updates accordingly:

Each time new tweets are appended to the list, the network panel lists a new request for the URL:

http://api.twitter.com/1/statuses/user_timeline.json?max_id=124092248651612160&count=21&include_entities=1&include_available_features=1&contributor_details=true&pc=false&include_rts=true&user_id=14074515

Does that URL look familiar (I've bolded the important part)? It's the API call we made in the introductory chapter to tweet-fetching. Go ahead and visit the link in your browser.

The contents of the text file at that visited URL are the same as in the "Response/Content" tab of your network panel:

Getting direct to the source with Ruby

How does using the network panel fit in with Ruby? Think of the panel as just a way of scouting the target. In the Wikipedia searchbox example, we were able to quickly suss out the URL of the relevant script and the needed parameters. The panel also reveals that it is a simple GET request – something you can visit in-browser, as we've seen.

We have all we need to perform a scrape. If we were interested in collecting what Wikipedia considers the most relevant suggested search terms for each letter of the alphabet, then we could run this simple script:

require 'rubygems'
require 'restclient'
require 'crack'
WURL = 'http://en.wikipedia.org/w/api.php?action=opensearch&namespace=0&suggest=&search='

('A'..'Z').to_a.each do |letter|
  resp = RestClient.get("#{WURL}#{letter}", 'User-Agent' => 'Ruby')
  arr = Crack::JSON.parse(resp)
  puts arr.join(', ')
  sleep 0.5
end

Which nets us this:

A, Animal, Association football, Australia, African American (U.S. Census), Allmusic, Arthropod, Album, Actor, Austria, American football
B, Brazil, BBC, Bass guitar, Belgium, Billboard (magazine), Boston, Basketball, Berlin, Bird, Billboard Hot 100
C, Canada, Countries of the world, China, California, Census, Chordate, Catholic Church, Communes of France, Chicago, Central European Time

resp = RestClient.get("#{WURL}#{letter}", 'User-Agent' => 'Ruby')

The Recovery.gov Flash map

Recovery.gov is a U.S. federal website that provides information on the progress and returns from the American Recovery and Reinvestment Act of 2009, i.e. the stimulus program. On the front page, the user is presented with an attractive Flash map of the U.S. The map displays a popup of summary data when you hover over a state with your mouse.

Recovery.gov homepage

A closeup view of the map:

Recovery.gov map detail

This is a case where structured data has been transformed into an attractive view. But to do a state-by-state comparison, it's better to just read the raw data directly from the structured datafile itself (again, as earlier noted, the site provides a prominent link to a text-display of the data; we're just practicing).

So open up your network panel and reload the page. You don't have to filter the results. But sort the listing by file type:

Examining Recovery.gov with Google Chrome

Lo and behold, the file is GetXmlData.aspx and it weighs in less than 6KB

It's easy to narrow down the candidates for the target data file. As I've mentioned before, XML and JSON are the most common data formats, and they are considered text files. When you're trying to find datafiles, it's best to start there first. Another clue will be the file size. Text files are usually pretty small. But if the Flash app contains a lot of data, then adjust your prediction accordingly. In this case, 50 entries of state summary data is not much text at all.

Click on the GetXmlData.aspx entry in your web inspector. You should see the data displayed as XML. You can also visit the actual URL in your browser's address bar and see the same thing:

Examining GetXmlData.aspx with Google Chrome

This inspection tactic works for many Flash-based data tables (I cover a real-world example dealing with corporate public data on ProPublica's data blog). However, it will not work in cases in which the data has been embedded directly into the application; in such cases, there is no external data file to catch with the inspector because the Flash app never needs to retrieve an external file.

These cases are rare, however, as data files are usually kept separate from the Flash apps so that non-Flash designers can update the data files without having to muddle with the Flash app source code itself. In cases where the data file is part of the app, it's usually the mark of a bad Flash programmer or a deliberate attempt to keep the data from being easily copied.

Parse the Recovery.gov map data with Ruby

Parsing the Recovery.gov XML file is easy, using methods we've used in the past:

require 'rubygems'
require 'crack'
require 'open-uri'

URL = 'http://www.recovery.gov/pages/GetXmlData.aspx?data=recipientHomeMap'

Crack::XML.parse(open(URL).read)['totals']['state'].each do |state|
  puts ['id', 'awarded', 'received', 'jobs'].map{|f| state[f]}.join(', ')
end

The output:

AL, 3693.77, 2683.89, 8373.58
AK, 2348.02, 1536, 2272.23
AZ, 7744.18, 4128.23, 6945.34
AR, 2341.72, 1622.7, 6001.51
CA, 35883.13, 20929.25, 28255.03
CO, 5607.56, 3644.37, 5637.33
CT, 2566.82, 1861.74, 4380.12
#...and so forth