Python 3 Notes         [ HOME | LING 1330/2330 ]

Python & NLTK Frequently Asked Questions

(Updated 6/1/2018) Python Accessing command history: How do I quickly bring up a previously entered command? [link] IDLE's default key bindings for command history are Ctrl-p (Atl-p in Windows) for a previously entered command and Ctrl-n (Alt-p in Windows) for the next command in the command history. You might find that these keys are not exactly handy -- see the next FAQ for how to assign different keyboard shortcuts. I dislike using "Ctrl-p/n" (or "Alt-p/n") keys for command history. Can I use ⇧ Up Arrow and ⇩ Down Arrow instead like in most other shell environments? [link] Frankly, Ctrl-p/n and Alt-n/p are a pain to use. But you can customize keyboard shortcuts in IDLE -- follow the steps below to assign UpArrow and DownArrow to "previous command" and "next command" instead: Open up IDLE, from the menu go Options --> Configure IDLE. Click Keys tab. Scroll down and click on the line starting with "history-next". Click button "Get New Keys for Selection". Scroll down to find "Down Arrow" and click on it. The new key is now set to "<Key-Down>". Press OK. [screenshot] You are prompted to name your custom key scheme. Give any name. Now repeat above process for "history-previous". But this time select "Up Arrow". [screenshot] How do I save my IDLE shell session into a file? [link] From the menu, choose File -> Save As, and then give the file name. Make sure to use the .txt extension and not .py: a saved IDLE session should be a text file and not a Python script file. Where is my Python installed? [link] To find out, in IDLE shell execute two commands: import os, sys os.path.dirname(sys.executable) So in this case, the python application is installed in C:\Program Files (x86)\Python38-32. Windows: if you followed these installation directions, then it is C:\Users\noamchomsky\AppData\Local\Programs\Python\Python39. Alternatively, it could be C:\Program Files\Python39 or C:\Program Files (x86)\Python39-32. If you have Anaconda Python, it could be C:\Users\noamchomsky\Anaconda3 or C:\ProgramData\Anaconda3. OS-X: If you followed these installation directions, then it is /Library/Frameworks/Python.framework/Versions/3.9. If you have Anaconda Python, it could be /Applications/anaconda3. (Mac) IDLE prints a WARNING message about the Tcl/Tk version. How can I fix this? [link] If you are getting a WARNING message like this, then your Python IDLE is using a wrong version of Tcl/Tk. This makes your IDLE prone to crashes. See this page for an explanation. You will need to uninstall your Python and then install the latest version (Python version 3.6.5/3.7/later, Anaconda3 5.2.0/later). If you have an older, pre-10.9 version of Mac OS, see the next FAQ. (Mac) I have an older version (10.6, 10.7, 10.8) of Mac OS. How do I install Python? [link] If you have a pre-10.9 version of Mac OS, you should first install Tcl/Tk, followed by an appropriate version of Python. (See this page for an explanation.) As of June 2018, that means: Install TCL version 8.5.18.0 from this ActiveState page. It has to be THIS EXACT VERSION!! Then, back on the Python download page, download and install "macOS 64bit/32-bit installer, for Mac OS X 10.6 or later". See this screenshot. Open up your IDLE to verify. If you do not see a warning message like this, you are set. If you do, then you should uninstall your Python and start over. (Win) IDLE wants to save my scripts in a weird system directory. How do I change it to use a folder of my choice? [link] The "Start in" parameter of an application, also known as "current working directory", determines this behavior. You should get Python IDLE to "start in" a folder you want. Follow the instructions on the "Some Basic Configurations" page. (Mac) IDLE "starts in" the 'Documents' folder by default and prompts to save all my scripts there. But I want to organize my Python scripts into a subfolder. How do I change this setting? [link] Changing IDLE's default directory is a great idea. The default directory is also where your Python looks for modules and other files, so that's an added bonus beyond having your script files neatly organized. Follow the steps below. [screenshot] Create a subfolder under 'Documents'. Preferably, give it a one-word name (without a space), such as "pythoncode". Move all your existing scripts there. Login from an administrator account. Open a Terminal. It is under the "Utilities" directory in "Applications", in Finder. [screenshot] Move into the IDLE app setting directory, using the cd ('change directory') command: cd /Applications/Python\ 3.8/IDLE.app/Contents/Resources Now, you have to edit idlemain.py using a program called nano. But you have to do it as a "super user". The sudo command ('super user do') lets you execute a command as a super user. Try: sudo nano idlemain.py It will prompt for the admin password. [screenshot] A file editing window will then open up. Move to the 8th line using the arrow key, which is: os.chdir(os.path.expanduser('~/Documents')). Edit the line to add your subfolder name, e.g., : os.chdir(os.path.expanduser('~/Documents/pythoncode')). Exit the edit window using Ctrl+X. It will ask "Save modified buffer?" Type Y and enter. When you are done, open up your Python IDLE. From the menu, File --> Open. It should open right up in your own Python code directory. (Win) My file names show up without the extensions (.txt, .py, .pdf, etc.). How do I make them visible? [link] Unless you changed the default setting, your system will display file names without the proper extension, e.g., you see "myscript" instead of "myscript.py". This leads to a lot of confusion while programming. See this video tutorial for step-by-step instructions. I am trying out these commands that I found on a Python tutorial site, but I keep getting an error. What is wrong? [link] It is likely that your Python tutorial site is based on an older version of Python: Python 2. Many changes were introduced in Python 3.X.X, most notable of which is the print command and handling of Unicode. How do I refer to a file in Python? [link] OS-X/Linux file paths look like this: /Users/yourname/Desktop/foo.txt. They start from the root "/", and slashes are used to separate directories. Windows file paths look like this: C:\Users\yourname\Desktop\foo.txt. They start with the disk label "C:", and backslashes are used to separate directories. In Python, Windows files can be referred to in multiple ways: Python simply lets you use OS-X/Linux style slashes "/" even in Windows. Therefore, you can refer to the file as 'C:/Users/yourname/Desktop/foo.txt'. If using backslash, because it is a special character in Python, you must remember to escape every instance: 'C:\\Users\\yourname\\Desktop\\foo.txt' Alternatively, you can prefix the entire file name string with the rawstring marker "r": r'C:\Users\yourname\Desktop\foo.txt'. That way, everything in the string is interpreted as a literal character, and you don't have to escape every backslash. I have this file sitting in my Desktop/My Documents ... area. How do I find its full file path and name? [link] Windows: Right-click on the file icon to show the "Properties" tab In this example, the file name is "out.txt", and the location "C:\Users\yourname\Documents\scripts" The full Python file path is therefore 'C:/Users/yourname/Documents/scripts/out.txt'. Mac: Right click on the file and select "Get Info". Alternatively, "Command + i" summons up the Get Info panel. In this example, the file name is "mary-short.txt" and the location "/Users/student/Desktop" The full Python file path is therefore '/Users/student/Desktop/mary-short.txt'. I am having trouble reading/writing/loading a particular file. What's wrong? [link] If you are getting a "No such file or directory" error (example here), that is because the file name your specified is incorrect/insufficient and as a result Python failed to locate your file. The most straightforward solution is to refer to the file by its full path and file name. See the two FAQs above for how to do that. OK, so using the full file path and name always works. But I've seen files being referred to by the file name only. How is it done? [link] The concept of Current Working Directory (CWD) is crucial here. Basically, referring to a file without specifying its path ('myfile.txt') works only when the file is in your CWD. To complicate the matter, your Python has different initial CWD settings depending on whether you are working with a Python script or in a shell environment. In a Python script: When you execute your script, your CWD is set to the directory where your script is. Therefore, you can refer to a file in a script by its name only provided that the file and the script are in the same directory. In Python shell: In your shell, the initial CWD setting varies by system. You have two options: Change your CWD to the file's directory, or Copy or move your file to your CWD. (Not recommended, since your shell's CWD may change.) See this screen shot and and the next FAQ for how to work with your CWD setting in Python shell. A final note: if you are working in IDLE, bear in mind that executing a script changes your IDLE shell's CWD to your script's directory. How do I work with CWD (current working directory) in Python shell? [link] Python module os provides utilities for that. Below illustrates how to find your CWD (.getcwd()) and change it into a different directory (.chdir()). Below is an example for the windows OS:   >>> import os >>> os.getcwd() 'D:\\Lab' >>> os.chdir(r'scripts\gutenberg') # relative path: scripts dir is under Lab >>> os.getcwd() 'D:\\Lab\\scripts\\gutenberg' >>> os.chdir(r'D:\Corpora\corpus_samples') # absolute path >>> os.getcwd() 'D:\\Corpora\\corpus_samples' On a Mac, your file path should look like '/Users/yourname/Desktop/'. (Win) My text file shows up with no line break. What gives? [link] First, you must understand that the end-of-line (EOL, also called "line break") is encoded differently depending on the OS: Unix, Linux, Mac OS X: \n (called "line feed") Old Mac OS up to 9: \r ("carriage return") Windows: \r\n Because of this discrepancy, if you open a unix-style text file in Windows with Notepad, you will see the text in one line. But you can use other text viewers such as Wordpad, Notepad++, and even Chrome and Firefox browsers, and they will display the unix-style line breaks just fine. Python, on any OS, represents the end-of-line as \n, but it has what's called universal newline support. Details: Regardless of your OS, within Python all newline types are converted to \n. When you write out to a file, Python will replace \n with the OS-appropriate newline marker. For more information, read https://docs.python.org/3/library/functions.html#open. NLTK How do I install NLTK 3 (Mac/Linux)? [link] The procedure is more or less the same as what's on the "Installing NLTK" page, with some extra steps. First, install NLTK: In a Terminal window (what is Terminal?), run pip3 install -U nltk. Depending on your system, you might need to use sudo with the command: sudo pip3 install -U nltk. This automatically downloads and installs the most current version of NLTK. pip3 uses the Python 3 version of pip (Macs ship with Python 2 by default, and pip defaults to Python 2.) Afterwards, in IDLE shell try importing nltk: import nltk. If no error message, NLTK was installed successfully. And then, there are some additional packages to install: numpy: run pip3 install -U numpy matplotlib: run pip3 install -U matplotlib BeautifulSoup: run pip3 install -U beautifulsoup4 Afterwards, in IDLE shell try importing the three: >>> import numpy >>> import matplotlib >>> import bs4 >>> If no error message, the packages have been installed successfully. How do I install NLTK 3 (Windows)? [link] Start by installing NLTK: Open up cmd (command prompt) as an administrator. You can do this by right-clicking cmd and choosing "Run as administrator" (screenshot). Install NLTK by running this command: pip install nltk. If cmd complains about not finding pip, try pip3 instead. If still no go, then your Python was not added to PATH during installation. Consult your instructor. When it finishes, in IDLE shell try importing nltk: import nltk. (screenshot) If no error message, NLTK was installed successfully. And then, there are three additional packages to install: numpy: pip install numpy matplotlib: pip install matplotlib beautifulsoup: pip install beautifulsoup4 Afterwards, in IDLE shell try importing the three (screenshot): >>> import numpy >>> import matplotlib >>> import bs4 >>> If no error message, the packages have been installed successfully. You will now need to download NLTK data. See the next FAQ. I finished installing NLTK through pip, but import nltk still gives an error. What's happening? [link] This happens when you have multiple versions of Python on your system, which means multiple versions of pip / pip3. Your system's default pip3 belongs to the wrong copy of Python, therefore NLTK was installed in a wrong spot. The most common scenario is when you have both Anaconda and Python.org's python distributions. How to use the correct copy of pip3, then? Let's first look up all copies that exist on your system. In your terminal, Mac/Linux users should type in which -a pip3: and Windows users where pip3 in their command prompt: In both examples above, you can see Anaconda's version of pip3 as well as Python.org's. Then, instead of simply referencing pip3, you will need to use the full file path when executing the NLTK installation command. So, in the Mac example above, you will run this command to install NLTK for python.org's python: /Library/Frameworks/Python.framework/Versions/3.12/bin/pip3 install -U nltk You'll do something similar in Windows, but there's one more wrinkle: because of the space in the file path, you need to enclose the full path in "". So: "C:\Program Files\Python311\Scripts\pip3" install -U nltk That doubles up as a mini lesson on how your OS internally organizes various executable commands. Useful stuff! I get a "Protocol error" (screenshot) while trying to install NLTK. Help! (Mac) [link] This problem occurs on Mac systems. Here are the steps to remedy the issue (screenshot): Open Preferences, select WiFi, hit Advanced.. Click on TCP/IP tab and change Configure IPv6 to Link-local only. After that, pip installation will work. For details, see this page. How do I download corpora and other data for NLTK? [link] You can do that using the nltk.download() utility. Follow the steps in this NLTK book section. You should download either "All packages", or "Everything used in the NLTK book". As for the "Download Directory", the default location will be some place deeply buried in your directory hierarchy, so I recommend choosing a more accessible directory, such as "C:\nltk_data" or "C:\Users\yourname\Documents\nltk_data". Bear in mind that once you download the data, moving the directory will likely cause NLTK to lose track of it. Once downloading finishes, you should confirm success by trying the following command (screenshot): >>> nltk.corpus.brown.words() ['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...] >>> If you do not want to bother with the graphical interface, you can do the downloading of appropriate packages by specifying them as an argument. 'book' downloads everything referenced in the NLTK book:   >>> nltk.download('book') [nltk_data] Downloading collection 'book' [nltk_data] | Downloading package abc to D:\Lab\nltk_data... [nltk_data] | Downloading package brown to D:\Lab\nltk_data... I get an "SSL error" (screenshot) while trying to download NLTK data. Help! (Mac) [link] This happens on a Mac when Python installation failed to configure an appropriate SSL certificate. There is a handy command you can run (inside the Terminal app): /Applications/Python\ 3.9/Install\ Certificates.command You can run the command graphically (track it down in Finder and double-click) or via command-line shown above. Substitute '3.9' with your appropriate version, such as '3.10'. See this page for more information. Where do I begin with NLTK? [link] First of all, import the NLTK module: this should be the very first thing you do at every session. And then confirm that NLTK can locate data by looking up the Brown Corpus.   >>> import nltk >>> nltk.corpus.brown.words() ['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...] >>> If the system returns an error message, either you haven't downloaded the Brown Corpus yet or NLTK cannot locate it (see the previous and next FAQs). Where is NLTK installed? [link] You can look up your NLTK's installation path by issuing the following command (note double underscores "__"): >>> import nltk >>> nltk.__path__ ['C:\\Program Files\\Python311\\Lib\\site-packages\\nltk'] The following are typical OS-specific installation paths. Windows "all-users, 64-bit" installation: C:\Program Files\Python311\Lib\site-packages\nltk Windows default (per-user) installation: C:\Users\NoamChomsky\AppData\Local\Programs\Python\Python311\lib\site-packages\nltk NOTE: Windows hides AppData folder by default. Once you're in the parent folder, you should check "Show Hidden items" folder option to unhide it. Anaconda3 (Windows): C:\ProgramData\anaconda3\lib\site-packages\nltk or something similar. OS-X: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/nltk Where is my NLTK data located on my computer? [link] It depends on where you set the destination folder when you download the data using nltk.download(). On Windows 10, the default destination is either C:\Users\yourname\nltk_data or C:\Users\yourname\AppData\Roaming\nltk_data, but you can specify a different directory before downloading. Internally, NLTK keeps a list of places where it looks for its data: nltk.data.path.   >>> nltk.data.path ['C:\\Users\\yourname/nltk_data', 'C:\\nltk_data', 'D:\\nltk_data', 'E:\\nltk_data', 'D:\\Lab\\nltk_data', 'C:\\Program Files (x86)\\Python38-32\\nltk_data', 'C:\\Program Files (x86)\\Python38-32\\lib\\nltk_data', 'C:\\Users\\yourname\\AppData\\Roaming\\nltk_data'] If your NLTK data directory is not listed in the result, NLTK won't be able to find your data. You have two options: (1) move your NLTK data directory to one of the locations already in the data path, or (2) edit NLTK's data.py file to add your data's location to NLTK's data directory path (see this FAQ). But those are many places NLTK looks for your data... which is actually the data folder? There is no straightforward solution, but assuming you have the Brown corpus downloaded as part of your data package you can look up its root directory:   >>> nltk.corpus.brown.root FileSystemPathPointer('D:\\Lab\\nltk_data\\corpora\\brown') From this, I can deduce my NLTK data files are located in 'D:\Lab\nltk_data'. The system gives me an error when I try to access Brown/Gutenberg/Inaugural/... corpus. How to fix? [link] If you have already downloaded NLTK data, then NLTK is having trouble locating your nltk_data directory. See this FAQ above and then change NLTK's data path, as shown in this FAQ. How do I add my NLTK data's location to NLTK's data directory path? (NLTK 3) [link] You will need to edit NLTK's data.py file. Procedure: First, find the data.py file and make a backup copy. It is found under your NLTK directory. Open the file in a text editor. IDLE editor also works. Then find the lines below (NOTE: example is for Windows OS). Use Ctrl+F to search. if sys.platform.startswith('win'): # Common locations on Windows: path += [ str(r'C:\nltk_data'), str(r'D:\nltk_data'), str(r'E:\nltk_data'), Now edit the second line to append your NLTK data's directory. In this example, it is the highlighted portion. Don't forget to add the comma ',' at the end: str(r'C:\nltk_data'), str(r'D:\nltk_data'), str(r'E:\nltk_data'), str(r'F:\Lab\nltk_data'), Save the file. If you get a "permission error" and blocked from saving, you have to use the following get-around. First, save the data.py file on your DESKTOP area. And then, move the file over by dragging. You will get "Overwrite file?" and/or "Administrator Permission Needed" error boxes -- press OK on them. Now, your original data.py file has been overwritten by the new data.py file. Verify your setting change is successful. First, check the nltk.data.path variable (see this FAQ) and make sure your new directory shows up in the list. Now, try loading up Brown Corpus again. Open up IDLE and type:   >>> import nltk >>> nltk.corpus.brown.words() ['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...] >>> How do I modify an NLTK module? [link] The process is as simple as replacing the original .py module file with a new one, restarting Python and importing NLTK. You might have to, however, get around some file permission issues. If you are a PC user, the above process is straightforward. You might need an administrator privilege. If you are a Mac user, an attempt to update files in the normal Finder view will run into a permission problem. To get around this, follow the instructions below. Login from an administrator account. Open your Terminal. It is under the "Utilities" directory in Finder. Move into the directory where the Python module is located using the cd ('change directory') command (substitute 3.11 with your version): cd /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/nltk/classify (NOTE: the entire thing is a single command and not two!) If that command didn't work, your NLTK is installed in a different place. To find out exactly where it is, open up your IDLE shell and try the following commands (note double underscores "__"): >>> import nltk >>> nltk.__path__ ['/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/nltk/'] and then, adjust your cd command accordingly. Make sure you include the last /classify folder while cd-ing. Now, mv ('move') is the command to use for renaming the original Python file, but you will need to do so as a "super user". The sudo command ('super user do') lets you execute a command as a super user. Try: sudo mv naivebayes.py naivebayes.py.ORIGINAL It will prompt for a password. Supply the administrator password. Next, copy or move the new Python module file (say, naivebayes.py) into the module directory. You can now do this in a normal Finder window: open Finder, navigate to the directory, and then drag-and-drop the file. System will prompt for an admin password. Alternatively, you can use the copy command below. This will work if (1) your previous cd command worked and you are currently inside the classify directory, and (2) your new naivebayes.py file is in your Desktop area. cp ~/Desktop/naivebayes.py . Relaunch your Python, and import NLTK. Your module will be rebuilt upon importing NLTK. General Computing Tips How do I take a screen shot? [link] Windows: Method 1: Alt-PrtScn (copies the active screen onto clipboard) and then Ctrl+V (pastes onto, say, a Word Document). Method 2: In Windows 7, you can use "Snipping Tool" under "Accessories" folder. It gives an option to capture a particular portion of a window and also to save image as a file. Method 3: Use an excellent free application called Greenshot. It lets you annotate your image. Mac OS-X: Method 1: To capture a portion of the desktop, press Command-Shift-4. A cross-hair cursor will appear and you can click and drag to select the area you wish to capture. When you release the mouse button, the screen shot will be automatically saved as a PNG file on your desktop. (The file is saved as PDF in Mac OS 10.3 and earlier.) [source] Method 2: To capture a specific application window, press Command-Shift-4, then press the Spacebar. The cursor will change to a camera, and you can move it around the screen. As you move the cursor over an application window, the window will be highlighted. The entire window does not need to be visible for you to capture it. When you have the cursor over a window you want to capture, just click the mouse button and the screen shot will be saved as a PNG file on your desktop. (The file is saved as PDF in Mac OS 10.3 and earlier.) [source] Method 3: Add Control to the two shortcuts above to place the screen shot on the clipboard instead of saving it to the desktop. [source] What text editing programs should I use? [link] For Windows, Notepad++ and Atom are highly recommended. For Mac, try BBEdit or Atom.