I never met with a disposition more truly amiable.
But Queequeg, do you see, was a creature in the transition stage -- neither caterpillar nor butterfly.
I'll bet you can, but can a computer algorithm?
Let's find out. We will build a classifier which, given a sentence as input, predicts which of the two authors wrote it. We will be using Austen's Emma and Melville's Moby Dick as our corpus data, included in NLTK's Gutenberg corpus. There are two parts in this experiment: [A] developing a classifier, and [B] analysis and write-up.
You may submit a Jupyter Notebook file (.ipynb) for this homework. Start with this TEMPLATE jupyter notebook; please stick with the overall organization.
Before you begin
For this homework, we will be using a couple of classifier methods that do not come with NLTK's original implementation of Naive Bayes. You will therefore need to update your NLTK's NaiveBayesClassifier module. Follow the instructions below.
Find your local copy of naivebayes.py, found in the classify directory in your NLTK directory. Rename it naivebayes.py.ORIGINAL just in case you need it later. (NOTE: You may run into permission issues. Mac users: Follow the instructions in this FAQ.)
Download this new naivebayes.py file: naivebayes.py and copy it into the directory. Supply administrator password if necessary.
Quit and restart your Python. Import NLTK.
Verify that the update was successful by checking if these two new commands are recognized:
>>> import nltk
>>> help(nltk.classify.NaiveBayesClassifier.feature_weights)
Help on function feature_weights in module nltk.classify.naivebayes:
feature_weights(self, fname, fval)
Displays per label the probability of the feature+value.
Added By Na-Rae Han, October 2023
>>> help(nltk.classify.NaiveBayesClassifier.show_most_informative_feats_all)
Help on function show_most_informative_feats_all in module nltk.classify.naivebayes:
show_most_informative_feats_all(self, n=10)
Displays top n most informative features.
Unlike show_most_informative_features, includes odds ratio of
[PART A] Developing a Naive Bayes Classifier
In this part, you will be developing a Naive Bayes classifier. Start with this template script: hw4_whosaid.TEMPLATE.py. Before you begin editing, execute the script and see what it prints out. Your job is to modify the script so it proceeds through the following steps:
Load each text as a list of sentences (already COMPLETE in template script)
From the two author-segregated lists, build two new lists where each sentence is labeled with its author, while weeding out sentences that are too short (1-2 words in length, e.g., 'Chapter XIII') to be useful
Join the two labeled author-segregated lists into a single list
Print out how many Austen, Melville, and total sentences there are (COMPLETE)
Shuffle the labeled sentence list you made in STEP 3
Partition sentences into three data sets: testing (first 1,000 sentences), development-testing (next 1,000), and training (the rest); and then, print out the size of each set.
Define a feature-generator function (COMPLETE)
Using the feature-generator function, convert the three data sets into their corresponding feature representations
Using the training set, construct a Naive Bayes classifier named whosaid
Using the test set, evaluate the classifier's performance
From the development test set, create 4 subsets based on the real author vs. the classifier's prediction
Print out sample correct and incorrect predictions from the 4 sub-divided sets
Finally, print out 40 most informative features. Use two methods: first the usual .show_most_informative_features(), and then .show_most_informative_feats_all() which is new.
As always, switch back and forth between your Python script and shell: execute your script, try out some commands in the shell, update your script with successful code, and repeat.
That concludes your classifier development. Note that every time you run your script you will get slightly different performance scores: that's because you are randomly shuffling your data set every time, resulting in a different "test -- development test -- training set" partition.
That brings us to one last step I want you to take. Replace your 5. sentence-shuffling command with the following line:
random.Random(10).shuffle(sents)
and re-run the script. This shuffles the sentence list based on a fixed random seed. The result is a list of sentences that have been randomly shuffled but nevertheless *in the same sequential order for all of us*, which will then lead to identical classifier models for everyone! This effectively freezes your/our model, and it allows us to share the same reference points for the next part of the homework, which centers on analysis. (And, obviously, it will make grading feasible at all...)
[PART B] Analysis and Write-Up
In this part, you will explore the classifier you built in PART A in order to gain an understanding of its inner workings. Instructions:
You should primarily work in a Python shell environment immediately following an execution of your script.
Answer the following questions in this MS Word document: HW4 Who Said It - PART B Analysis.docx. For most questions, you will be entering two things: (1) a screenshot of your IDLE shell showing relevant code bits and output, and (2) your written analysis accompanying the code. Don't just point to the code output and skip your commentary! YOUR ANALYSIS is the focus here, not the numbers or calculations. See this example to get a sense of what's expected.
As usual, when you're done save your IDLE shell session as a text file: hw4_shell.txt. You'll be submitting it as well. Feel free to post-edit to remove long, unnecessary bits.
Let's get to it!
Classifier accuracy What is the system's accuracy? Is it lower or higher than you expected?
Features
Examine the gen_feats() function. What sort of features is used in this classifier model?
Examine the first most informative features list and make observations. Do you notice any patterns? Any surprising entries?
Examine the second most informative features list. How is this list different from the first one? Any additional observations? (NOTE: don't get confused! You are still dealing with the SAME model, it's just that the two methods show a slightly different set of top features.) (NOTE 2: If you are unclear about this one, revisit after you've worked on Q10 below.)
Main character names Some of you are probably thinking: the classifier must be getting a lot of help from the main character names such as Emma, Ahab and Queequeg. Let's see how well it does without them.
The script already contains a switch that you can turn on to "neutralize" the top 35 most common character names and place names in the two novels by turning them all into 'MontyPython'. Edit the file and set the value of noCharNames to True.
Re-run the script. How is the new classifier's performance? Did it degrade as much as you expected? Why do you think that is? How is the top feature list affected?
When you're done, set noCharNames back to False and re-build your classifier by running the script again. For the rest of this homework, USE THIS ORIGINAL SETTING.
Trying out sentences
Test the classifier on the two sentences below. Sent1 is actually by Jane Austen, taken from Persuasion. Sent2 is from Alice's Adventures in Wonderland by Lewis Caroll.
(Sent1) Anne was to leave them on the morrow, an event which they all dreaded.
(Sent2) So Alice began telling them her adventures from the time when she first saw the White Rabbit.
You'll have to tokenize the sentences first. Use nltk.word_tokenize(). Then, convert them to their feature representation, which can then be fed into your classifier's .classify method.
What label did the classifier give to Sent1 and Sent2? Did it match your expectation?
Label probabilities for a sentence
Labeling judgments aside, how likely does your model thinks that Sent1 is Austen? That is essentially P(austen|Sent1). To find out, we need to use the .prob_classify method instead of the usual .classify. Below demonstrates how to find the probability estimates assigned to either label for the sentence 'Hello, world'. whosaid thinks it's 72% Melville and 28% Austen:
Try it with Sent1. What is P(austen|Sent1)? That is, given Sent1, how likely is it to be Austen? What is P(melville|Sent1)?
How about Sent2 -- P(austen|Sent2) and P(melville|Sent2)?
From a. and b., how "confident" is your classifier about Sent1 being Austen? Is it equally confident on Sent2 being Melville?
Trying out made-up sentences
Now, test the classifier on the following made-up sentences:
(Sent3) He knows the truth
(Sent4) She knows the truth
(Sent5) blahblahblah blahblah
What labels did the classifier give to Sent3 and Sent4, and with what probabilities? Any thoughts?
What about Sent5? Given that neither "word" appeared in the training data, why do you think the classifier made the prediction it did?
Base probabilities (=priors) Not knowing anything about a sentence, is it more likely to be Austen or Melville? We can answer this question by establishing the base probabilities, i.e. priors. In your training data (i.e., train_sents or train_feats):
How many sentences are there?
How many of them are Austen?
How many of them are Melville?
From the above, what are P(austen) and P(melville)?
How is your answer to d. related to the classifier's prediction on Sent5 above?
train_sents is a list of pairs, where each pair consists of a tokenized sentence and its label:
Using this pattern, you should use list comprehension to extract items with either author.
Calculating odds ratio Would the word 'very' be more indicative of Austen or Melville, and how strongly so? Let's answer this by calculating its odds ratio. Find out the following, again in the training data:
How many Austen sentences contain 'very'? Make sure to count 'Very' as well.
How about Melville sentences?
What is P(very|austen)? That is, given an Austen sentence, how likely is it to contain 'very'?
What is P(very|melville)? That is, given a Melville sentence, how likely is it to contain 'very'?
What is Austen-to-Melville odds ratio of 'very'?
Feature weights in model P(very|austen) and P(very|melville) are indeed the 'weights' your model assigns to the feature 'contains-very':1. Let's confirm this by probing your model. Use the .feature_weights() method:
This means that 1 out of 5 Austen sentences contain 'she'. For Melville, the likelihood is much lower: 1 out of 100 sentences. (For fun, try with 'he'.)
What are the weights of 'very'?
Do they match up with what you calculated in 8.c and 8.d above? (They should. The small differences are effects of smoothing, which may be more pronounced in other cases.)
Zero counts and feature weights
In order to accommodate features and feature-value pairs never encountered in the training data, a machine learning algorithm will adopt a couple of strategies, including smoothing.
Look up the feature weights of 'contains-whale' and also 'contains-ahab'. What do you notice?
The notation in, say, 5.7829483345e-05 is something called 'scientific notation', also known as 'exponential notation', and it is used for a very small number. In this case, 5.7829483345e-05 is the same as 0.000057829483345, because e-05 is equal to 0.00001. As a rule of thumb, e-05 means you move up the decimal point by 5 spots in order to get the equivalent non-exponential notation.
This time, look up the feature weights for words 'housekeeper' and 'Emma'. Anything noticeable?
Don't forget to lowercase 'Emma' in the feature name!
Find a word that occurs in Austen's work only, and another that occurs only in Melville, and then look up their feature weights. You should have a theory by now -- sum up what is going on with these words and their feature weights.
As a comparison point, the word 'enchanting' occurs exactly once in the Austen training sentences, and likewise in the Melville training sentences. How do its feature weights compare against the ones you saw above?
Now, try 'contains-internet'. What happens this time?
Then, using the .prob_classify method, find out the likelihood of 'She hates the internet' being an Austen sentence. Then try 'She hates the'. What can you conclude about the classifier's handling of features it never encountered in the training data?
Combining feature weights
(**NOTE: Do not use rounding for the calculations below! Numbers will fail to match up if you do.)
So, how is an overall probability estimation for a given sentence being, say, Austen obtained from individual feature weights? That is, how is P(austen|sent) obtained? Recall from class that:
P(label|sent) = P(sent,label) / P(sent)
Let's focus on the label being Austen for now:
P(austen|sent) = P(sent,austen) / P(sent)
①
Note that P(sent, austen) + P(sent, melville) equals the probability of the sentence occurring at all, i.e., P(sent). Therefore:
So this means we need to know P(sent,austen) and P(sent,melville). A corollary of ① is P(sent|austen) = P(sent,austen) / P(austen). (Note that P(sent,austen) = P(austen,sent).) Therefore, we can estimate P(sent, austen) as follows:
That is, the probability of a sentence occurring and it being Austen equals the base probability of the label P(austen) multiplied by the probability of each word feature in the sentence occurring in an Austen sentence. Then, the probability of Sent3 'He knows the truth' occurring as an Austen sentence is:
You already calculated the Austen prior: P(austen) in 7 d. What is it?
P(he|austen) can be found through whosaid.feature_weights('contains-he', 1). Likewise with the rest of the words.
From a. and b., calculate P(Sent3, austen).
Similarly, calculate P(Sent3, melville).
Now, calculate P(Sent3) as P(Sent3, austen) + P(Sent3, melville).
Ultimately, the probability question we like to answer is: "Given the sentence He knows the truth, how likely is it to be Austen"? That is, what is P(austen|Sent3)? Use formula ① above to calculate this.
Does the figure match up with the classifier's estimation from 6.a above? (It should.)
Performance on the development-test data
Work with the four lists aa, mm, am, ma to answer the following questions.
Of the 1,000 development-test set, how many of them did the classifier correctly label?
What is whosaid's accuracy on the development test data? Ideally, it should be close to its performance on the test data -- is it?
What % of the sentences did the classifier label as 'austen'? How about 'melville'? Why do you think it is not 50-50?
What % of the classifier's 'austen' rulings is correct? That is, when the classifier labels a sentence as 'Austen', what is the likelihood of this prediction to be correct?
Likewise, when the classifier labels a sentence as 'melville', what is the likelihood of this prediction being correct?
Error analysis
The list am contains all sentences from the dev-test set that are in fact Austen but were mis-labeled as Melville by whosaid. Let's take a look at these errors.
Print out all mis-classified Austen sentences by:
>>> for x in am: print(' '.join(x[2]))
What do you think of these sentences? Do they sound Melville-like to you?
When a classifier mislabels, it is hoped that it at least did so with low confidence (say, 55%) than high (98%). Pick some sentence from the list and see what likelihood whosaid assigned to them for being Melville. Of all sentences you tried, which was judged Melville with the lowest confidence? Which one was it most sure about being Melville?
This is a big assignment with 60 total points (Part A: 20, Part B: 40), so you have a full week to finish. But to help you NOT procrastinate, I am building in a checkpoint. The homework submission link is configured to accept multiple submissions, so use the same link for both.
10/3 (Thu) Check your PART A output
In the text box, report your classifier's accuracy (Question 10) after you fixed the random seed. Don't round up the number! After your submission, you will be able to see if your figure is correct. If incorrect, there's something wrong with your script. Rework your script and submit a new answer.
That's it! The point was to make sure you are on the right track. No need to submit your script at this time.
10/8 (Tue) SUBMIT completed homework
For PART A, Upload your finished script: "hw4_whosaid.py"
For PART B, Upload:
Analysis answer sheet: "HW4 Who Said It - PART B Analysis.docx"
Your saved IDLE shell (you can edit to trim errors, etc.): "hw4_shell.txt"
Advanced Pythonians: You may submit a single Jupyter Notebook file (.ipynb) instead of all three. See the "ADVANCED PYTHONIANS" box on top for a template.