Word2Vec through gensim¶
4/10/2018 PyLing meeting
- Tutorial at: https://machinelearningmastery.com/develop-word-embeddings-python-gensim/
- The code below is a near-exact copy of everything in this tutorial. See the tutorial page for detailed explanation of what's going on.
- Also, read SLP3 chapters:
- Ch.15 Vector semantics https://web.stanford.edu/~jurafsky/slp3/15.pdf
- Ch.16 Semantics with dense vectors https://web.stanford.edu/~jurafsky/slp3/16.pdf
Setting up your python environment on CRC¶
- ssh into
h2p.crc.pitt.edu - Your default python is 2.7.5. Confirm by:
python --version - See which versions are available on system:
module spider python - Switch over to the appropriate version of python:
module load python/anaconda3.5-4.2.0 - Confirm you are now working with this version of python:
python --version - gensim package is not installed system-wide. You should install your own copy:
pip intall gensim --user
Using Jupyter Notebook on CRC¶
- Point your browser to https://hub.crc.pitt.edu/
- Log in with your Pitt ID (will probably have to 2-factor-authenticate)
- For this demo session, use "Host process" as job profile (less prone to network overload)
- When you are running a real job, you should choose "SMP - 1 core, 1 hour"!!!
- When done using Jupyter Notebook, make sure to properly close your session by logging out.
In [1]:
# Turns on/off pretty printing
%pprint
# Every returned Out[] is displayed, not just the last one.
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
In [2]:
# Import Word2Vec from gensim.
# gensim is not installed system-wide. Install for yourself: pip install gensim --user
from gensim.models import Word2Vec
In [4]:
from nltk import word_tokenize
mary = """Mary had a little lamb,
His fleece was white as snow,
And everywhere that Mary went,
The lamb was sure to go.
"Why does the lamb love Mary so?"
The eager children cry.
"Why, Mary loves the lamb, you know."
The teacher did reply."""
sentences = [word_tokenize(s) for s in mary.lower().split('\n')]
sentences
Out[4]:
In [5]:
# train model
model = Word2Vec(sentences, min_count=1)
# summarize the loaded model
print(model)
In [6]:
# summarize vocabulary
words = list(model.wv.vocab)
print(words)
In [7]:
# access vector for one word
#print(model['sentence'])
print(model['lamb'])
In [8]:
# save model
model.save('model.bin')
In [9]:
# load model
new_model = Word2Vec.load('model.bin')
print(new_model)
In [10]:
dir(model)
Out[10]:
In [11]:
model.wv.vocab
Out[11]:
Visualization through PCA (principal component analysis)¶
In [12]:
from sklearn.decomposition import PCA
from matplotlib import pyplot
In [13]:
# fit a 2d PCA model to the vectors
X = model[model.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model.wv.vocab)
for i, word in enumerate(words):
pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Out[13]:
Load Google's pre-trained word2vec embedding¶
- Good news! I have already downloaded the model file and saved it in our
shared_datafolder on h2p.
In [14]:
from gensim.models import KeyedVectors
filename = '/ihome/pyling/shared_data/word_vectors/GoogleNews-vectors-negative300.bin'
model = KeyedVectors.load_word2vec_format(filename, binary=True)
# calculate: (king - man) + woman = ?
result = model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)
print(result)
# works!
In [15]:
result = model.most_similar(positive=['woman', 'actor'], negative=['man'], topn=1)
print(result)
# works again!
In [16]:
result = model.most_similar(positive=['dog', 'kitten'], negative=['baby'], topn=1)
print(result)
# Hmm. Not sure about this one. 100 billion words of news data might not feature enough kittens and puppies.
Load Stanford's pre-trained GloVe embedding¶
- Again, the model files are already downloaded in the
shared_datafolder.
In [17]:
# GloVe file format must be first convertd to Word2Vec format
# Something that needs to be done ONCE: you guys don't need to do this!
from gensim.scripts.glove2word2vec import glove2word2vec
vdir = '/ihome/pyling/shared_data/word_vectors/'
glove_input_file = vdir + 'glove.6B.100d.txt'
word2vec_output_file = vdir + 'glove.6B.100d.txt.word2vec'
glove2word2vec(glove_input_file, word2vec_output_file)
Out[17]:
In [18]:
from gensim.models import KeyedVectors
# load the Stanford GloVe model
filename = vdir + 'glove.6B.100d.txt.word2vec'
model = KeyedVectors.load_word2vec_format(filename, binary=False)
# calculate: (king - man) + woman = ?
result = model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)
print(result)
In [19]:
result = model.most_similar(positive=['dog', 'kitten'], negative=['baby'], topn=1)
print(result)
# 6 billion words of Wikipedia --> more kittens and puppies possibly.
In [20]:
result = model.most_similar(positive=['pittsburgh', 'eagles'], negative=['philadelphia'], topn=5)
print(result)
# Beautiful!
In [21]:
result = model.most_similar(positive=['turtle', 'snail'], negative=['shell'], topn=5)
print(result)
# Hmm...
In [22]:
result = model.most_similar(positive=['cow', 'pork'], negative=['pig'], topn=5)
print(result)
# Yayy!
In [23]:
result = model.most_similar(positive=['michelle', 'barack'], negative=['bush'], topn=5)
print(result)
# Huh! Is it because Laura Bush is still a Bush?
# But George is not a very distinctive first name, and we can't do George W.
In [ ]: