PATH VARIABLES & PTH FILES

PATH VARIABLES & PTH FILES (or, how to fix your python installation without throwing your computer out the window)

imagine that you've installed a python package (eg using pip), you've uninstalled & re-installed python a thousand times, but you still can't import your package into python. if this sounds familiar, these instructions may help you out.

I. UNDERSTANDING YOUR SYSTEM PATH

first of all, type this at your command line (without the first dollar sign...that's just meant to signify your command prompt. but make sure you include the second dollar sign!):

$ echo $PATH

this should show you a list of ":" delimited paths, eg:

/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

this string is your "system path". the entries in this string (again, separated by ":") are the paths (or equivalently, directories) in your filesystem where your operating system will look for programs when you type them at the command line.

you can see this in action by using the "which" command:

$ which ls

"which" is a unix program that takes another program as its argument & returns the path to that program. to verify this, look at the man page for which by typing:

$ man which

the first line of the output says:

which -- locate a program file in the user's path

when i type "which ls" at my command line, the output is "/bin/ls". this tells me 2 things: first, that the ls program lives in the /bin directory, and second, that it explicitly does not live in /usr/local/bin, /usr/local/sbin, or /usr/bin. how do i know this? because the operating system finds programs by looking at the entries in the path variable in order.

as a consequence, if you have two versions of the same program on your machine, invoking it at the command line (without specifying the path explicitly) will always launch the version that lives in the leftmost directory in your system path. take a second to mull this over if it seems mysterious.

here's another example...try typing "which python" at your command line. this will tell you exactly where the python program that you're running when you type "python" lives.

if you suspect that you have multiple competing versions of python on your machine, try typing "python" at the command line, but now hit tab instead of enter (unix supports tab completion). this will show you all the programs on your machine that look like "python*" (eg, all the programs whose names begin with the letters p-y-t-h-o-n).

II. UNDERSTANDING YOUR PYTHON PATH

okay...so we've seen that the operating system uses the system path to determine where programs live. similarly, python uses a variable called the "python path" to determine where python packages live.

to see this, open up the python shell (by typing "python" at the command line) and type the following (don't type ">>>"...this just signifies the python prompt):

>>> import sys
>>> for k in sys.path: print k

the sys module is a python built-in module, so you should be able to import this without difficulty. the sys.path variable is an array of paths which plays the same role for python that the system path plays for the operating system (note that although python calls this variable sys.path, it contains your python path and NOT your system path!)

when i type the above commands in my python shell, i see the following output:

/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/distribute-0.6.45-py2.7.egg /usr/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg /usr/local/lib/python2.7/site-packages/distribute-0.6.45-py2.7.egg /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python27.zip /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7 /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PIL /Library/Python/2.7/site-packages /usr/local/lib/python2.7/site-packages /usr/local/lib/python2.7/site-packages/PIL

this is the list of directories that python will look into (not recursively) when i try to import a third-party library, like numpy. i can see where numpy lives by typing

>> import numpy
>> help(numpy)

when i enter these commands, the output looks like this:

NAME
    numpy

FILE
    /usr/local/lib/python2.7/site-packages/numpy/__init__.py

this tells me that python looked through the entries in my python path (again, in order) and found numpy in the directory listed here (as before, it also tells me that numpy wasn't found in any paths above this in my python path).

III. THE PIP OF THE ICEBERG

so, thus far our story has assumed that everything went according to plan...but what if you install numpy with pip and you still can't import it? the first thing to do is to make sure you've installed it correctly.

typing "pip -h" at the command line will bring up the help for pip (pip is the python package manager). you can see from the output (among other things) that "pip list" or "pip freeze" will show you which packages you've installed. i installed numpy 1.8 with pip, so i see the following:

$ pip freeze | grep numpy
numpy==1.8.0

now suppose you've installed numpy, and you can see numpy in the list of installed pip packages as above. typing "pip show numpy" will show you where your numpy installation landed:

Name: numpy
Version: 1.8.0
Location: /some/crazy/dir
Requires:

this output is also useful to see what dependencies a package may have, if any. so in this version of the story, pip has installed numpy to /some/crazy/dir, but python can't see it. should i give up? certainly not.

IV. PTH FILES TO THE RESCUE

barring a comprehensive python-ectomy (which may still leave old config details behind, causing angst and confusion even after reinstallation), i can patch this situation up with a .pth file.

in the simplest case, a .pth file is just a text file containing one path per line. each of these paths will be appended (eg, added to the bottom) of your python path. a .pth file must end with the ".pth" extension, and in order to work it must be placed in a directory that's already on your python path (the most common location is your site-packages directory, which is where python usually puts third-party libraries).

so returning to our example, suppose my numpy installation has landed in /some/crazy/dir, which is not in my python path, but my python path does contain the site-packages directory

/usr/local/lib/python2.7/site-packages

so here's what i want to do:

1) navigate to the site-packages directory
	$ cd /usr/local/lib/python2.7/site-packages

2) create a pth file with /some/crazy/dir on line 1
	$ cat > my.pth
	/some/crazy/dir		(note this line is input, eg you have to type it)

(btw if you create a file with cat like this, you can save & stop editing by closing the cat program with ctrl-C)

3) verify that you created the .pth file correctly
	$ cat my.pth
	/some/crazy/dir		(note this line is output, eg it's now saved in my.pth)

3) navigate back to your home directory (why? because your working directory is on your python path by default...a potential gotcha!)
	$ cd

4) fire up python & look upon your works
	>>> import sys
	>>> for k in sys.path: print k
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/distribute-0.6.45-py2.7.egg
	/usr/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg
	/usr/local/lib/python2.7/site-packages/distribute-0.6.45-py2.7.egg
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
	/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PIL
	/Library/Python/2.7/site-packages
	/usr/local/lib/python2.7/site-packages
	/usr/local/lib/python2.7/site-packages/PIL
	/some/crazy/dir

as you can see, /some/crazy/dir shows up!

5) verify that you can now import your package	
	>>> import numpy
	(no complaints!)

6) now you are good to go!

one final point to keep in mind: if the directory you specify in your .pth file does not exist in your filesystem, it won't appear in your python path...sounds obvious, but good to know.

i hope some of you find this helpful...as usual please feel free to reach out with questions/comments/etc!

ps - i highly encourage you to read the docs (sound advice for life as a data scientist/developer in general): http://docs.python.org/2/install/#modifying-python-s-search-path

as a general rule of thumb: if there's something you can't figure out, google it...chances are someone else had the same problem & saw fit to tell the internet