Tweet Fetching

Read from files

The following code reads another page from Wikipedia, saves it to disk, reopens that saved file, and prints its contents to your screen.

require "open-uri"

remote_base_url = "http://en.wikipedia.org/wiki"
remote_page_name = "Steve_Wozniak"
remote_full_url = remote_base_url + "/" + remote_page_name

somedata = open(remote_full_url).read

my_local_filename = "my_copy_of-" + remote_page_name + ".html"
my_local_file = open(my_local_filename, "w")
    my_local_file.write(somedata)    
my_local_file.close


file_for_reading = open(my_local_filename, "r")
    puts file_for_reading.read
file_for_reading.close

The open method

By now you should've noticed that we're using the same word – open – to refer to three seemingly different methods:

somedata = open(remote_full_url)

This retrieves a webpage from the Internet address in remote_full_url

open(my_local_filename, "w")

This opens a file – a file on your hard drive with a filename that is equal to the variable my_local_filename – for writing into.

open(my_local_filename, "r")

Same as above, but the file is now opened for reading, not writing.

The latter two methods are easier to explain. We've given both method calls each a second string (the values passed to a method are called arguments) – either a "r" or "w". These set the kind of open operation: either the file is being opened to read from or to be written into.

The use of open with remote_full_url actually uses the open method supplied by the OpenURI library. The OpenURI library is not included for use by default, so that's why the first line of our programs so far has been:

require "open-uri"

How Ruby chooses between the open for local files and the open for Internet addresses is a subject for future study. The only thing you need to know now, as a beginner, is that without the require "open-uri", you won't be able to use open for Internet addresses, such as "http://wikipedia.org"

Note: By default, the open method goes into read mode. This is why we don't bother supplying "r" in the line:

somedata = open(remote_full_url).read    

Read and write, twice-over

This script will:

1. Read two different webpages
2. Write both to your hard drive
3. Reopen those saved files
4. Write the combined contents of those files into a third file

require "open-uri"

remote_base_url = "http://en.wikipedia.org/wiki"
r1 = "Steve_Wozniak"
r2 = "Steve_Jobs"
f1 = "my_copy_of-" + r1 + ".html"
f2 = "my_copy_of-" + r2 + ".html"

# read the first url
remote_full_url = remote_base_url + "/" + r1
rpage = open(remote_full_url).read

# write the first file to disk
file = open(f1, "w")
file.write(rpage)
file.close

# read the second url
remote_full_url = remote_base_url + "/" + r2
rpage = open(remote_full_url).read

# write the second file to disk
file = open(f2, "w")
file.write(rpage)
file.close

# open a new file:
compiled_file = open("apple-guys.html", "w")

# reopen the first and second files again
k1 = open(f1, "r")
k2 = open(f2, "r")

compiled_file.write(k1.read)
compiled_file.write(k2.read)

k1.close
k2.close
compiled_file.close

You should now have a file in your working directory called apple-guys.html. This file should contain the combined contents of the local files my_copy_of-Steve_Jobs.html and my_copy_of-Steve_Wozniak.html

We didn't use any new methods or techniques to combine the two files. Just different variable names that I chose at my pleasure. Note how I reuse the variable file twice:

• To open, write to, and close a file with the name pointed to by f1
• To open, write to, and close a file with the name pointed to by f2

This is totally kosher. However, by reassigning file to file = open(f2, "w"), we lose the pointer to the first file. So when reassigning variables, be sure you're done with whatever data it once pointed to.

# this code prints "hello world"
puts "hello world"

Counting from 1 to 10

Here's a simple collection:

1..10

It's a collection of the numbers from 1 through 10; in Ruby, this data object is known as a Range.

All collections, including ranges, have a method called each.

When we use a single dot operator (.), we signify that the method each belongs to a particular collection

This is how we use each with the range 1..10:

(1..10).each do |a_number|
    puts a_number
end

Running that code should get you this:

1
2
3
4
5
6
7
8
9
10

Combining strings and numbers

We've seen how to add strings together with + (the technical term is concatenate). But this is the first time we've really dealt with numbers (See? You don't have to be a math genius to program). Numbers can be added together as you've done in arithmetic class:

1 + 8
#=> 9
   
50 * 50 + (100/2)   * 42
#=> 4600

However, strings and numbers cannot be added together. They are two different data types, as far as Ruby is concerned. The following code will throw an error:

puts( 10 + "42")    

The "42" may look like a number to us humans. But again, Ruby considers anything surrounded by quotation marks as a string. So "42" has more in common with "forty-two" than with 42.

However, we can use the to_s method to convert the number into a string:

puts( 10.to_s + "42")    

The answer (remember, you're adding two strings together):

1042

Back to our file operations, try:

remote_base_url = "http://en.wikipedia.org/wiki"
(1..3).each do |some_number|
    r_url = remote_base_url + "/" + some_number.to_s
    puts r_url
end    

The result:

http://en.wikipedia.org/wiki/1
http://en.wikipedia.org/wiki/2
http://en.wikipedia.org/wiki/3

All together now

Visit one of the URLs generated from our numerical loop, such as http://en.wikipedia.org/wiki/1. It's the Wikipedia entry for Year 1.

Try to see if you can write the following program based on what you've learned so far:

• Specify two numbers, representing a start and end year
• Use that range to create a loop
• Retrieve the Wikipedia entry that corresponds to each iteration of the loop
• Save that Wikipedia page to a corresponding file on your hard drive
• In a second loop, combine all those year entries into one file, with the name of "start_year-end_year.html"

This consists of nothing more than a combination of previously written code.

TextWrangler's Run in Terminal

In TextWrangler – and this may apply to other text editors – the output of the following program won't appear until everything is finished. In the previous code snippet, you'll have a long period of no feedback, even though there is a puts command to output after each loop iteration.

To see the output from the puts commands in real time, choose Run in Terminal as opposed to just Run:

require "open-uri"
remote_base_url = "http://en.wikipedia.org/wiki"

start_year = 1900
end_year = 1906

(start_year..end_year).each do |yr|
    rpage = open(remote_base_url + "/" + yr.to_s)
    
    local_fname = "copy-of-" + yr.to_s + ".html"
    local_file = open(local_fname, "w")
    local_file.write(rpage.read)
    local_file.close
   # Optional output line:
   puts "Wrote file " + local_fname
   sleep 1
end

# Write to the compiled file now:
compiled_file = open(start_year.to_s + "-" + end_year.to_s + ".html",  "w")
(start_year..end_year).each do |yr|
    local_fname = "copy-of-" + yr.to_s + ".html"
    local_file = open(local_fname, "r")
    
    compiled_file.write(local_file.read)
    local_file.close
end

compiled_file.close

If you copied and ran the above code exactly as is, you should have files for years 1900 to 1906, and a file named 1900-1906.html that is a combination of all those files:

Here's an additional challenge: write the above program using just one each loop.

Ask yourself: why not perform the actions in the second loop inside of the first loop?

require "open-uri"
remote_base_url = "http://en.wikipedia.org/wiki"

start_year = 1900
end_year = 1906
compiled_file = open(start_year.to_s + "-" + end_year.to_s + ".html",  "w")

(start_year..end_year).each do |yr|
    rpage = open(remote_base_url + "/" + yr.to_s)
    rpage_data = rpage.read
   
    compiled_file.write(rpage_data)
    
    local_fname = "copy-of-" + yr.to_s + ".html"
    local_file = open(local_fname, "w")
    local_file.write(rpage_data)
    local_file.close

    puts "Wrote file " + local_fname
    sleep 1
end
compiled_file.close

There's one subtlety here that you wouldn't know about before studying the chapter on file reading/writing. I declare a new variable in this one-loop version:

rpage = open(remote_base_url + "/" + yr.to_s)
rpage_data = rpage.read
...
compiled_file.write(rpage_data)
local_file.write(rpage_data)

Previously, the equivalent code was:

rpage = open(remote_base_url + "/" + yr.to_s)   
...
local_file.write(rpage.read)

For reasons that won't be clear until the chapter on file reading/writing, you can't call the read method twice on the same File object (in this example, rpage).

However, we can take the output of read – a string of all the raw HTML – and assign it to a different variable. We can access that variable – rpage_data – as many times as we like.

Bonus: Each time you invoke a method, it returns some kind of value. As I mentioned above, rpage.read returns a String. But what is rpage itself? It is the result of this operation:

   
rpage = open(remote_base_url + "/" + yr.to_s)
rpage_data = rpage.read

This can all be done in one line:

rpage_data = open(remote_base_url + "/" + yr.to_s).read

Take a crack at redoing the one-loop downloader with that change. It should be just a copy-and-paste job with a slight alteration:

require "open-uri"
remote_base_url = "http://en.wikipedia.org/wiki"

start_year = 1900
end_year = 1906
compiled_file = open(start_year.to_s + "-" + end_year.to_s + ".html",  "w")

(start_year..end_year).each do |yr|
    rpage_data = open(remote_base_url + "/" + yr.to_s).read

    compiled_file.write(rpage_data)

    local_fname = "copy-of-" + yr.to_s + ".html"
    puts local_fname
    local_file = open(local_fname, "w")
    local_file.write(rpage_data)
    local_file.close
    sleep 1
end
compiled_file.close

What you've learned so far

Whether you understood anything or not, by running through this Wikipedia-scraping script, step-by-step, you've covered most of practical concepts in most of the fundamentals section of this book, including:

• Code commenting
• Numbers
• Strings
• Variables
• Methods
• Loops
• Collections
• File reading/writing

Introduction to the Hash

What crack does so nicely for us is convert XML content into the data structure known as a Hash, which is a type of collection. So this:

<?xml version="1.0" encoding="UTF-8"?>
<person>
  <first_name>Tim</first_name>
  <last_name>Berners-Lee</last_name>  
</person>

Becomes this:

{"person"=> {
    "last_name"=>"Berners-Lee", "first_name"=>"Tim"}
}

I cover this in detail in the collections chapter. You just need to be acquainted with how to access values from the Hash data structure:

data = {"person"=> {"last_name"=>"Berners-Lee", "first_name"=>"Tim"} }

puts data["person"]["last_name"]   
#=> Berners-Lee

Introduction to the Array

The crack gem, when parsing XML, also produces the Ruby collection-type known as the Array. Let's look at a more complicated XML document:

<?xml version="1.0" encoding="UTF-8"?>
<people>
    <person>
      <first_name>Tim</first_name>
      <last_name>Berners-Lee</last_name>  
    </person>  
    <person>
      <first_name>Robert</first_name>
      <last_name>Cailliau</last_name>  
    </person>  
</people>

Besides the new <people> tag and another level of nesting, do you see what the major structural difference is? There are two elements with the person tag. How do we specify one from the other?

Let's try to parse it out as we did in the previous example:

require "rubygems"
require "crack"
require "pp"

xml = '
<?xml version="1.0" encoding="UTF-8"?>
<people>
    <person>
      <first_name>Tim</first_name>
      <last_name>Berners-Lee</last_name>  
    </person>  
    <person>
      <first_name>Robert</first_name>
      <last_name>Cailliau</last_name>  
    </person>  
</people>
'

parsed_xml = Crack::XML.parse(xml)
puts parsed_xml['people']['person']['first_name']

Ruby will throw an error at that last line because there are multiple <person> tags. The structure for handling tags of the same name at the same level is not a hash, but an array.

Replace the final line of the previous code with:

puts parsed_xml['people']['person'][0]['first_name']
puts parsed_xml['people']['person'][1]['first_name']

The output should be the first_name elements of the first and second person elements, represented by the indices of 0 and 1, respectively:

Tim
Robert

Parse user info from the tweets data

Here's some code to open one of the previously downloaded tweet files, parse the XML, and print out:

• The user's Twitter name
• The user's real name
• The user's account creation date
• The user's total tweet count
• The date of the first tweet in the file
• The text of the first tweet in the file

Note: You'll have to change the tweet_filename variable to point to whatever you named one of your own downloaded tweet files:

require "rubygems"
require "crack"

tweet_filename = "USAGov-tweets-page-1.xml"
tweets_file = File.open(tweet_filename)
parsed_xml = Crack::XML.parse( tweets_file.read )
tweets_file.close


tweets = parsed_xml["statuses"]

first_tweet = tweets[0]
user = first_tweet["user"]

puts user["screen_name"]
puts user["name"]
puts user["created_at"]
puts user["statuses_count"]

puts first_tweet["created_at"]
puts first_tweet["text"]

You should see output similar to this format:

USAgov
USA.gov
Mon Mar 03 21:06:45 +0000 2008
1560
Sun Oct 23 18:01:06 +0000 2011
Celebrate Pink Ribbon Sunday to raise awareness about breast cancer. Learn how you can participate at http://t.co/5dgvtnOh

In your text editor, Open up any one of the tweet files you've downloaded and examine the XML.

The general structure goes like this:

<?xml version="1.0" encoding="UTF-8"?>
<statuses type="array">
    <status>
        <user>
            fields for username, id, profile pic, date started, etc
        </user>
        fields for content of the tweet, date, number of times retweeted, etc.
    </status>
    <status>
        <user></user>
    </status>
    <status>
        <user></user>
    </status>
    etc...
</statuses>  

The XML consists of:

• An overarching <statuses> element
• <statuses> contains a set of <status> elements
• Each <status> contains the metadata and content of a single tweet.
• Also inside each <status> is a <user> element
• Each <user> contains a set of elements with the user's information. The <user> is the same in every <status> element (i.e tweet)

Here's the XML for a single tweet:

<status>
  <created_at>Sun Mar 20 17:22:56 +0000 2011</created_at>
  <id>49521314461007872</id>
  <text>Spring is here! Today marks the Vernal Equinox, the official start of the spring season.</text>
  <source><a href="http://app.measuredvoice.com/" rel="nofollow">Measured Voice</a></source>
  <truncated>false</truncated>
  <favorited>false</favorited>
  <in_reply_to_status_id></in_reply_to_status_id>
  <in_reply_to_user_id></in_reply_to_user_id>
  <in_reply_to_screen_name></in_reply_to_screen_name>
  <retweet_count>38</retweet_count>
  <retweeted>false</retweeted>
  <user>
    <id>14074515</id>
    <name>USA.gov</name>
    <screen_name>USAgov</screen_name>
    <location>Washington, DC</location>
    <description>Follow us to stay up to date on the latest official government news and information.</description>
    <profile_image_url>http://a0.twimg.com/profile_images/1155238675/usagov_normal.jpg</profile_image_url>
    <profile_image_url_https>https://si0.twimg.com/profile_images/1155238675/usagov_normal.jpg</profile_image_url_https>
    <url>http://USA.gov</url>
    <protected>false</protected>
    <followers_count>70372</followers_count>
    <profile_background_color>f7f7f7</profile_background_color>
    <profile_text_color>6c6c6c</profile_text_color>
    <profile_link_color>022945</profile_link_color>
    <profile_sidebar_fill_color>ffffff</profile_sidebar_fill_color>
    <profile_sidebar_border_color>d2d2d2</profile_sidebar_border_color>
    <friends_count>288</friends_count>
    <created_at>Mon Mar 03 21:06:45 +0000 2008</created_at>
    <favourites_count>0</favourites_count>
    <utc_offset>-21600</utc_offset>
    <time_zone>Central Time (US & Canada)</time_zone>
    <profile_background_image_url>http://a3.twimg.com/profile_background_images/64898326/twitter-bg-usagov.gif</profile_background_image_url>
    <profile_background_image_url_https>https://si0.twimg.com/profile_background_images/64898326/twitter-bg-usagov.gif</profile_background_image_url_https>
    <profile_background_tile>false</profile_background_tile>
    <profile_use_background_image>false</profile_use_background_image>
    <notifications></notifications>
    <geo_enabled>false</geo_enabled>
    <verified>true</verified>
    <following></following>
    <statuses_count>1560</statuses_count>
    <lang>en</lang>
    <contributors_enabled>false</contributors_enabled>
    <follow_request_sent></follow_request_sent>
    <listed_count>3233</listed_count>
    <show_all_inline_media>false</show_all_inline_media>
    <default_profile>false</default_profile>
    <default_profile_image>false</default_profile_image>
    <is_translator>false</is_translator>
  </user>
  <geo/>
  <coordinates/>
  <place/>
  <contributors/>
</status>

Print out the content of all tweets in a file

Here's some code that loops through and parses each tweet in one of the files we've previously downloaded.

require "rubygems"
require "crack"

tweet_filename = "USAGov-tweets-page-1.xml"
tweets_file = File.open(tweet_filename)
parsed_xml = Crack::XML.parse( tweets_file.read )

tweets = parsed_xml["statuses"]
tweets.each do |tweet_xml|
  puts "Created at: " + tweet_xml['created_at']
  puts "Text: " + tweet_xml['text']
  puts "Retweet count: "+ tweet_xml['retweet_count']
  puts " - "
end

tweets_file.close

Think of the tweets variable as holding a collection of tweet XML objects. This is why we use .each to loop through each tweet.

The output should look something like this:

Created at: Tue Sep 13 17:01:04 +0000 2011
Text: September is National Preparedness Month. Find out how you can prepare for emergencies at http://t.co/uHj5fNF
Retweet count: 15
 - 
Created at: Mon Sep 12 22:55:12 +0000 2011
Text: September is National Ovarian Cancer Awareness Month. Learn about the risk factors and stay informed: http://t.co/83pfLOE
Retweet count: 19
 - 
Created at: Mon Sep 12 18:01:08 +0000 2011
Text: Hospital patients can now choose their own visitors and who can make end of life decisions. Learn more at http://t.co/JqORLBw
Retweet count: 18
 - 
Created at: Mon Sep 12 17:01:06 +0000 2011
Text: Create a video to help answer common questions about government benefits and services and you could win $1,000: http://t.co/rWWCGel
Retweet count: 21

Loop through all the downloaded tweets

This uses the same technique for looping through the files you downloaded and then using the parsing code from the previous script.

You should end up with a program that contains two each loops, one inside the other:

require "rubygems"
require "crack"

tweet_basename = "USAGov-tweets-page-"
first_page = 1
last_page = 10
(first_page..last_page).each do |pg_num|

   tweet_filename = tweet_basename + pg_num.to_s + ".xml"
   
   tweets_file = File.open(tweet_filename)
   parsed_xml = Crack::XML.parse( tweets_file.read )

   tweets = parsed_xml["statuses"]
   tweets.each do |tweet_xml|
     puts "Created at: " + tweet_xml['created_at']
     puts "Text: " + tweet_xml['text']
     puts "Retweet count: "+ tweet_xml['retweet_count']
     puts " - "
   end

   tweets_file.close

end

The result should be the parsed content of 1,000 tweets (or however many pages you downloaded).

Tweet Analysis

This next section contains some ways to count our collected data. I'm keeping the methods basic and just wanted to demonstrate how easy it is to loop through everything and perform any kind of analysis we want.

Daily tweeting rate, based on one page

In our requests to the Twitter API to download tweets, we set the tweets-per-page parameter to 100:

http://api.twitter.com/1/statuses/user_timeline.xml?count=100&screen_name=SOMEUSERNAME

But we can programmatically count the tweets by parsing the XML and getting an array of tweets, and then calling that array's length method:

require "rubygems"
require "crack"

tweet_filename = "USAGov-tweets-page-1.xml"
tweets_file = File.open(tweet_filename)
parsed_xml = Crack::XML.parse( tweets_file.read )

tweets = parsed_xml["statuses"]
puts tweets.length

The result:

100

Each tweet has a created_at field. So to find the tweeting rate over time – for a single page, at least –, we extract the created_at timestamp for the first and last tweets, convert them to time units (i.e. seconds), and divide the number of tweets by the difference in time between the first and last tweets:

require "rubygems"
require "crack"

tweet_filename = "USAGov-tweets-page-1.xml"
tweets_file = File.open(tweet_filename)
parsed_xml = Crack::XML.parse( tweets_file.read )

tweets = parsed_xml["statuses"]
total_tweet_count = tweets.length

# tweets are listed in chronological order, most recent first
most_recent_tweet = parsed_xml["statuses"].first  # same as array[0]
earliest_tweet = parsed_xml["statuses"].last  

most_recent_time = Time.parse(most_recent_tweet['created_at'])
earliest_time  = Time.parse(earliest_tweet['created_at'])

puts "most recent time: " + most_recent_time.to_s
puts "earliest time: " + earliest_time.to_s

tweets_per_second = total_tweet_count / (most_recent_time - earliest_time) 
tweets_per_day = tweets_per_second * 60 * 60 * 24

puts "Tweets per day: " + tweets_per_day.to_s

The results:

most recent time: Sun Oct 23 14:01:06 -0400 2011
earliest time: Fri Sep 09 13:01:50 -0400 2011
Tweets per day: 2.27060336028273

Daily tweeting rate based on all downloaded tweets

#Same thing, keep a counter of total tweets

require "rubygems"
require "crack"

tweet_basename = "USAGov-tweets-page-"
first_page = 1
last_page = 10

total_tweet_count = 0
earliest_tweet = 0
most_recent_tweet = 0

(first_page..last_page).each do |pg_num|
   tweet_filename = tweet_basename + pg_num.to_s + ".xml"
  tweets_file = File.open(tweet_filename)
  parsed_xml = Crack::XML.parse( tweets_file.read )
  tweets = parsed_xml["statuses"]
  total_tweet_count = total_tweet_count + tweets.length

  if pg_num == first_page
    most_recent_tweet = tweets.first
  end

  if pg_num == last_page
    earliest_tweet = tweets.last
  end

end

puts "Total tweets collected: " + total_tweet_count.to_s

most_recent_time = Time.parse(most_recent_tweet['created_at'])
earliest_time  = Time.parse(earliest_tweet['created_at'])

puts "most recent time: " + most_recent_time.to_s
puts "earliest time: " + earliest_time.to_s

tweets_per_second = total_tweet_count / (most_recent_time - earliest_time) 
tweets_per_day = tweets_per_second * 60 * 60 * 24
puts "Tweets per day: " + tweets_per_day.to_s

The results (in my case, it was fewer than 1,000 because we didn't download the retweets):

Total tweets collected: 994
most recent time: Sun Oct 23 14:01:06 -0400 2011
earliest time: Tue Sep 29 15:56:13 -0400 2009
Tweets per day: 1.31844217383975

Number of replies to other users

Interested in something else besides number of tweets? Once you've figured out how to parse a data source, it's trivial to change what you're looking for.

In the following script, we check the <in_reply_to_screen_name> node of each tweet. If there is something in it, we push the name onto an Array. At the end of the script, we print out the number of names collected (using the array's length method), the number of unique names, and then just the entire list of names:

require "rubygems"
require "crack"

tweet_basename = "USAGov-tweets-page-"
first_page = 1
last_page = 10

tweeters_replied_to = []

(first_page..last_page).each do |pg_num|
   tweet_filename = tweet_basename + pg_num.to_s + ".xml"
  tweets_file = File.open(tweet_filename)
  parsed_xml = Crack::XML.parse( tweets_file.read )
  tweets = parsed_xml["statuses"]
  puts "On page: " + pg_num.to_s

  tweets.each do |tweet|
    nm = tweet['in_reply_to_screen_name']
    if nm.nil?
      # do nothing, the field is empty
    else  
      tweeters_replied_to.push(nm)
    end
  end
end

unique_tweeters_replied_to = tweeters_replied_to.uniq
puts "There were replies to " + tweeters_replied_to.length.to_s + " Tweeters"
puts "The number of unique names is: " + unique_tweeters_replied_to.length.to_s
puts "--"
puts unique_tweeters_replied_to

The results:

There were replies to 49 Tweeters
The number of unique names is: 46

Tweet text analysis

For the purposes of this novice introduction, I'm only demonstrating the simplest ways to examine the tweets data we've collected. Using the same core of code from previous exercises, here are other simple aggregations we can do over the dataset:

• The total number of characters used
• A list, including the number of occurrences, of every word used (we can use regular expressions to count only strings of alphabetical characters used in a sentence, and not things like hashtags)
• The most frequently-used words
• The most frequently-used words having more than 4 characters
• The longest word used

The code is not any more conceptually complex than what we've already run through. But I'm not going to explain the details at length and I expect much of it to be unfamiliar to you. If you've come this far, it's a better use of your time for you to move on to the explanatory lessons to learn what's going on at a deeper level, rather than have me try to come up with oversimplified explanations.

So again, you're not expected to understand it if you're new to programming. I include it just to let you experiment with for fun. You'll probably recognize the same patterns we've been using in our code.

require 'rubygems'
require 'crack'

tweet_basename = "USAGov-tweets-page-"
first_page = 1
last_page = 10

total_tweet_count = 0
total_chars_in_tweets = 0
word_list = {}

(first_page..last_page).each do |pg_num|
   tweet_filename = tweet_basename + pg_num.to_s + ".xml"
  tweets_file = File.open(tweet_filename)
  parsed_xml = Crack::XML.parse( tweets_file.read )
  tweets = parsed_xml["statuses"]
  puts "On page: " + pg_num.to_s

  # A running total of tweets
  total_tweet_count += tweets.length 
  
  tweets.each do |tweet|
    # The tweet text, in lowercase letters
    txt = tweet['text'].downcase
    
    # A running total of total tweet length in characters
    total_chars_in_tweets += txt.length
    
    # Regular expression to select any string of consecutive
    # alphabetical letters (with optional apostrophes and hypens)
    # that are surrounded by whitespace or end with a punctuation mark 
    words = txt.scan(/(?:^|\s)([a-z'\-]+)(?:$|\s|[.!,?:])/).flatten.select{|w| w.length > 1}

    # Use of the word_list Hash to keep list of different words
    words.each do |word|
      word_list[word] ||= 0
      word_list[word] += 1
    end

  end
end

puts "\nTotal tweets:"
puts total_tweet_count

puts "\nTotal characters used:"
puts total_chars_in_tweets

puts "\nAverage tweet length in chars:"  
puts(total_chars_in_tweets/total_tweet_count)

puts "\nNumber of different words:"
puts word_list.length

puts "\nNumber of total words used:"
puts word_list.inject(0){|sum, w| sum += w[1]}

# sort word_list by frequency of word, descending order
word_list = word_list.sort_by{|w| w[1]}.reverse

puts "\nTop 20 most frequent words"
word_list[0..19].each do |w|
  puts "#{w[0]}:\t#{w[1]}"
end

puts "\nTop 20 most frequent words more than 4 characters long"
word_list.select{|w| w[0].length > 4}[0..19].each do |w|
  puts "#{w[0]}:\t#{w[1]}"
end


puts "\nLongest word:"
puts word_list.max_by{|w| w[0].length}[0]

The output of the analysis:

Total tweets:
994

Total characters used:
116128

Average tweet length in chars:
116

Number of different words:
1781

Number of total words used:
9080

Top 20 most frequent words
http:   774
the:   367
to:   277
learn:   276
at:   193
and:   187
you:   183
find:   164
of:   154
for:   143
your:   142
if:   95
from:   94
is:   88
in:   86
on:   64
how:   62
get:   61
this:   58
today:   57

Top 20 most frequent words more than 4 characters long
learn:   276
today:   57
about:   52
national:   50
these:   35
summer:   30
happy:   28
check:   27
government:   24
there:   23
federal:   19
follow:   18
information:   18
credit:   18
video:   18
school:   17
health:   17
season:   17
create:   16
people:   16

Longest word:
annualcreditreport

Granted, it's nothing too exciting. But I tried to use the most basic methods for this introductory chapter. Once you have a decent grasp on programming concepts, it's easy to do much more useful calculations.