Na-Rae Han (naraehan@pitt.edu), 5/30/2017, CMU DH Summer Workshop

# Preparation¶

Jupyter tips:

• Shift+ENTER to run cell, go to next cell
• Alt+ENTER to run cell, create a new cell below

# The very basics¶

### First code¶

• Printing a string, using print().
In [ ]:
print("hello, world!")


### The string type¶

• String type objects are enclosed in quotation marks.
• + is a concatenation operator.
• Below, greet is a variable name assigned to a string value; note the absence of quotation marks.
In [ ]:
greet = "Hello, world!"
greet + " I come in peace."

• String methods such as .upper(), .lower() transform a string.
In [ ]:
greet.upper()

• len() returns the length of a string in the # of characters.
In [ ]:
len(greet)


### Numbers¶

• Integers and floats are written without quotes.
• You can use algebraic operations such as +, -, * and / with numbers.
In [ ]:
num1 = 5678
num2 = 3.141592
result = num1 / num2
print(num1, "divided by", str(num2), "is", result)


### Lists¶

• Lists are enclosed in [ ], with elements separated with commas. Lists can have strings, numbers, and more.
• Like with string, you can use len() to get the size of a list.
• Like with string, you can use in to see if an element is in a list.
• A list can be indexed through li[i]. Python indexes starts with 0.
• A list can be sliced: li[3:5] returns a sub-list beginning with index 3 up to and not including index 5.
In [ ]:
li = ['red', 'blue', 'green', 'black', 'white', 'pink']
len(li)

In [ ]:
'mauve' not in li

In [ ]:
# Try [0], [2], [-1], [3:5], [3:], [:5]
li[0]


### for loop¶

• Using for loop, you can loop through a list of items, applying the same set of operations to each element.
• The embedded code block is marked with indentation.
In [ ]:
for x in li :
print(x, len(x))
print("Done!")


### List comprehension¶

• List comprehension builds a new list from an existing list.
• You can filter in only certain elements, and you can apply transformation in the process.
• Try: .upper(), len(), +'ish'
In [ ]:
[x for x in li if x.endswith('e')]

In [ ]:
[x+'ish' for x in li]

In [ ]:
[len(x) for x in li]


### Dictionaries¶

• Dictionaries hold key:value mappings.
• len() on dictionary returns the number of keys.
In [ ]:
di = {'Homer':35, 'Marge':35, 'Bart':10, 'Lisa':8}
di['Homer']

In [ ]:
len(di)


# Using NLTK¶

• NLTK is an external module; you can start using it after importing it.

• nltk.word_tokenize() is a handy tokenizing function out of literally tons of functions it provides.

• It turns a text (a single string) into a list tokenized words.

In [ ]:
import nltk

In [ ]:
nltk.word_tokenize(greet)

In [ ]:
help(nltk.word_tokenize)

In [ ]:
sent = "You haven't seen Star Wars...?"
nltk.word_tokenize(sent)

• nltk.FreqDist() is is another useful NLTK function.
• It builds a frequency dictionary from a list.
In [ ]:
# First "Rose" is capitalized. How to lowercase?
sent = 'Rose is a rose is a rose is a rose.'
toks = nltk.word_tokenize(sent)
print(toks)

In [ ]:
freq = nltk.FreqDist(toks)
freq

In [ ]:
freq.most_common(3)

In [ ]:
freq['rose']

In [ ]:
len(freq)


# Processing a single text file¶

### Reading in a text file¶

• open(filename).read() reads in the content of a text file as a single string.
In [ ]:
myfile = 'C:/Users/zoso/Desktop/inaugural/1789-Washington.txt'  # Mac users should leave out C:
print(wtxt)

In [ ]:
len(wtxt)     # Number of characters in text

In [ ]:
'fellow citizens'.lower() in wtxt.lower()  # phrase as a substring

In [ ]:
'Americans' in wtxt


### Tokenize text, compile frequency count¶

In [ ]:
# Turn off/on pretty printing (prints too many lines)
%pprint

In [ ]:
# Tokenize text
nltk.word_tokenize(wtxt)

In [ ]:
wtokens = nltk.word_tokenize(wtxt)
len(wtokens)     # Number of words in text

In [ ]:
# Build a dictionary of frequency count
wfreq = nltk.FreqDist(wtokens)
wfreq['citizens']

In [ ]:
wfreq['the']

In [ ]:
len(wfreq)      # Number of unique words in text

In [ ]:
wfreq.most_common(40)     # 40 most common words


### Average sentence length, frequency of long words¶

In [ ]:
sentcount = wfreq['.'] + wfreq['?'] + wfreq['!']  # Assuming every sentence ends with ., ! or
print(sentcount)

In [ ]:
# Tokens include symbols and punctuation. First 50 tokens:
wtokens[:50]

In [ ]:
wtokens_nosym = [t for t in wtokens if t.isalnum()]    # alpha-numeric tokens only
len(wtokens_nosym)

In [ ]:
# First 50 tokens, alpha-numeric tokens only:
wtokens_nosym[:50]

In [ ]:
len(wtokens_nosym)/sentcount     # Average sentence length in number of words

In [ ]:
[w for w in wfreq if len(w) >= 13]       # all 13+ character words

In [ ]:
long = [w for w in wfreq if len(w) >= 13]
for w in long :
print(w, len(w), wfreq[w])               # long words tend to be less frequent


# Processing a corpus¶

• NLTK can read in an entire corpus from a directory (the 'root' directory).
• As it reads in a corpus, it applies word tokenization (shown below) and sentence tokenization (not shown here).
In [ ]:
from nltk.corpus import PlaintextCorpusReader
corpus_root = 'C:/Users/zoso/Desktop/inaugural'  # Mac users should leave out C:
inaug = PlaintextCorpusReader(corpus_root, '.*txt')  # all files ending in 'txt'

In [ ]:
# .txt file names as file IDs
inaug.fileids()

In [ ]:
# NLTK automatically tokenizes the corpus. First 50 words:
print(inaug.words()[:50])

In [ ]:
# You can also specify individual file ID. First 50 words from Obama 2009:
print(inaug.words('2009-Obama.txt')[50:])

In [ ]:
# NLTK automatically segments sentences too, which are accessed through .sents()
print(inaug.sents('2009-Obama.txt')[0])   # first sentence
print(inaug.sents('2009-Obama.txt')[1])   # 2nd sentence

In [ ]:
# How long are these speeches in terms of word and sentence count?
print('Washington 1789:', len(inaug.words('1789-Washington.txt')), len(inaug.sents('1789-Washington.txt')))
print('Obama 2009:', len(inaug.words('2009-Obama.txt')), len(inaug.sents('2009-Obama.txt')))

In [ ]:
# for-loop through file IDs and print out word count.
for f in inaug.fileids():
print(len(inaug.words(f)), f)


### Trouble shooting¶

• Unfortunately, 2005 Bush file produces Unicode encoding error.
• Let's make a new text file from http://www.presidency.ucsb.edu/inaugurals.php
• Copy text and paste in Notepad (Windows). Make sure to choose UTF-8 encoding and not ANSI.
• The text files are locked; We will need to save, halt and then re-start the Python notebook.
In [ ]:
# Corpus size in number of words
print(len(inaug.words()))

In [ ]:
# Building word frequency distribution for the entire corpus
inaug_freq = nltk.FreqDist(inaug.words())
inaug_freq.most_common(100)


# What next?¶

Take a Python course!