Pages covering this lab session: here and here.
Managing multiple files with *
- Move into your gutenberg/ directory.
- We learned that wc -w carroll-alice.txt command displays the total number of words contained in the carroll-alice.txt. (It's 26,443 words.) But how do we find out the size of the entire corpus? You can specify all text files as arguments:
wc -w austen-emma.txt austen-persuasion.txt austen-sense.txt bible-kjv.txt (... and so on.)
But a better way is to utilize this handy wildcard * notation:
wc -w *
which prints out the following result:
The wildcard * matches every file name in the directory (except for those starting with a period, such as .bash_history); it even prints out the total at the bottom.
- Now the problem is, the README file, which is not a corpus text file, is included in this tally. To exclude this file and include only the ones ending with .txt:
wc -w *.txt
You might want to save that information into a file for future reference (but probably better to place that file in your home directory, so as not to disturb your corpus directory):
wc -w *.txt > ~/corpus-size.txt
- If you are interested in the size of the Shakespeare subcorpus, you can do:
wc -w sha*.txt
- Now move up to the parent directory which contains both abc/ and gutenberg/ directories. What is the size (in # of words) of the two corpora combined? The total number of lines?
Searching for words and patterns using grep
- grep is a popular unix command for pattern-based searching. Say you're interested in the word 'cause'. The following command:
grep cause carroll-alice.txt
prints out every line in carroll-alice.txt that matches (== contains) "cause". If there are a lot of lines, you can "pipe" the result into less or more so you can scroll through, or use head to just print the first few lines:
grep what carroll-alice.txt | less
grep what carroll-alice.txt | head -15
- grep provides many useful option switches, including --color. Try:
grep --color cause carroll-alice.txt
A list of grep command options and syntax:
|grep pattern file(s) ||prints out all lines in file(s) that match pattern|
|grep -i pattern file(s) ||does the same, but ignores case (so 'the' and 'The' are both matched)|
|grep -w ||restricts the search to whole words only|
|grep -n ||precedes each line with the line number|
|grep -h ||stops preceding each line with the file name (searching multiple files)|
|grep -l ||displays a list of files that contain the string (actual lines are not shown)|
|grep -v ||prints the lines that do NOT match pattern|
|grep --color ||prints the matched portion in color (extremely handy!)|
|grep -iw --color pattern file(s) ||'-' options can be strung together; '--' options cannot|
|grep "word1 word2" file(s) ||pattern must be in quotes if it contains space|
- The first command below looks up all instances of "of course" in the Gutenberg corpus and counts the number of lines. The second one saves the matching lines into a file.
grep -i "of course" *.txt | wc -l
grep -i "of course" *.txt > search_result.text (Why am I using '.text' instead of '.txt'?)
- Now for some exercise:
- We are interested in thou, the archaic second person singular pronoun. In how many corpus text files does this word occur?
- How many lines of this corpus contain this word?
- Is that number the same as thou's total # of occurrence in the corpus? Why not?
- Among the lines with thou, how many of them also contain shalt? (hint: use "|")
- Among the lines with thou and shalt, how many of them do NOT contain not?
- Lastly, it is also possible to get grep print out additional context lines before and after the matching lines. Use this option -C n where n is 1, 2, 3... and so on. This command:
grep --color -C 1 cause carroll-alice.txt
gets you this result:
little histories about children who had got burnt, and eaten up by wild
Note that the -C switch takes its own argument n; it does not "mesh" with other simple option switches. The syntax:
beasts and other unpleasant things, all because they WOULD not remember
the simple rules their friends had taught them: such as, that a red-hot
about two feet high, and was going on shrinking rapidly: she soon found
out that the cause of this was the fan she was holding, and she dropped
it hastily, just in time to avoid shrinking away altogether.
is the driest thing I know. Silence all round, if you please! "William
the Conqueror, whose cause was favoured by the pope, was soon submitted
to by the English, who wanted leaders, and had been of late much
|grep -C n ||prints out n lines before and after each matching line|
|grep -iw -C n --color pattern file(s) ||only simple '-' options can be strung together|