1. Overview
While developing code with the Python language, it’s simple to use the built-in modules. We just import the module and then use the functions and variables in the module.
However, if we use custom Python modules, importing them in the calling script may not be successful because they may not be in the default module search path. An example of a custom module is a Python script developed by us but isn’t in the same directory as our calling script.
In this tutorial, we’ll discuss how to augment the default module search path for finding Python modules.
2. Introduction to the Problem
Let’s start with the following Python files, my_module.py, and main.py, in the /home/alice/project and /home/alice/project/modules directories, respectively:
$ pwd
/home/alice/project
$ ls
main.py modules
$ ls modules/
my_module.py
Here’s the content of my_module.py:
#!/usr/bin/python3
def hello_world():
print("Hello World")
This file has only one function named hello_world(). It just prints Hello World using the built-in print() function. This file is a module in Python jargon. We can use the functions and variables defined in a module from other Python files.
Let’s look at the content of main.py:
#!/usr/bin/python3
from my_module import hello_world
hello_world()
First, we import the function hello_world() in the module my_module using the statement from my_module import hello_world. Then, we call the imported function using the statement hello_world().
Now, we’ll run the Python script main.py:
$ pwd
/home/alice/project
$ ./main.py
Traceback (most recent call last):
File "./main.py", line 3, in <module>
from my_module import hello_world
ModuleNotFoundError: No module named 'my_module'
We were unsuccessful in running the script because the script can’t import the module, my_module. In other words, it can’t find where my_module.py is.
In the following sections, we’ll discuss several methods for finding a custom Python module.
3. Methods Updating Python Scripts
We’ll first discuss augmenting the default module search path within a Python script.
3.1. Using the sys Module With Hard Coded Paths
One way of finding other modules is explicitly specifying their path in the calling script.
Let’s inspect the modified version of the script main.py, namely main_sys.py:
#!/usr/bin/python3
import sys
sys.path.append('./modules')
from my_module import hello_world
hello_world()
The script has two additional statements. First, we import the built-in sys module in Python using the import sys statement. This module provides several functions and variables for configuring the Python runtime environment.
Then, we append the modules directory in the current directory to the directories Python searches while importing modules. We do it using the sys.path.append(‘./modules’) statement. path is a built-in variable in the sys module, which contains a list of directories. Its type is the Python list. We use the append() method of the list to add the ./modules directory to the end of the list.
If we print the value of the path variable of the sys module using print(sys.path) after appending the ./modules directory, we’ll obtain a list like the following:
['home/alice/project', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/usr/local/lib64/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages', './modules']
Of course, the paths in this output may be slightly different in other systems, but we see that the directory we added using sys.path.append(‘./modules’) is at the end of the list. Therefore, it’s now in the module search path.
Now, let’s run the script, main_sys.py:
$ pwd
/home/alice/project
$ ./main_sys.py
Hello World
As it’s apparent from the output, we were successful in calling hello_world() this time.
However, if we decide to move the module hello_world.py to another directory, we must update the path passed to the sys.path.append() method accordingly. This is the downside of passing hard-coded paths to sys.path.append().
3.2. Using the sys Module With an Environment Variable
Instead of using hard-coded paths, we can obtain the path using an environment variable.
Let’s define an environment variable, MODULE_PATH, that points to the path of the custom Python modules:
$ export MODULE_PATH=/home/alice/project/modules
Now, we can use a slightly modified version of main_sys.py, namely main_sys_env.py:
#!/usr/bin/python3
import os
import sys
module_path=os.getenv('MODULE_PATH')
sys.path.append(module_path)
from my_module import hello_world
hello_world()
We get the value of the environment variable, MODULE_PATH, using the getenv() method in the os module. This module is one of the built-in modules in Python, like sys. Then, we add this path to the search path as before using sys.path.append(module_path).
Let’s run it:
$ ./main_sys_env.py
Hello World
Running the script was successful.
Therefore, if we move the modules directory to another directory and update the path in the environment variable accordingly, the script will still be successful. We don’t need to update the search path passed to sys.path.append() in the script.
4. Methods Without Updating Python Scripts
We used the modified versions of the original main.py script in the previous section to augment the module search path. In this section, we’ll discuss two methods that don’t require the modification of Python scripts.
4.1. Using the site-packages Directories
The first method we’ll discuss is using the site-packages directories. As we saw earlier in the previous section, the path list returned by printing the sys.path variable contains several directories named site-packages. For example, /usr/lib64/python3.6/site-packages is one of them.
When we download third-party Python libraries, these libraries are installed in those directories. We can use path configuration files for augmenting the search paths. A path configuration file is a file that contains additional paths to be added to sys.path. Its name must be in the form of name.pth where name is a valid string. This file must be in one of the existing site-packages directories.
We’ll use the following my_module.pth file:
$ cat my_module.pth
/home/alice/project/modules
After copying this file to one of the site-packages directories, for example, to /usr/lib64/python3.6/site-packages, let’s run the original main.py script:
$ ./main.py
Hello World
We ran the script successfully without making any changes in the script.
We generally need to have root privileges to copy a path configuration file to one of the site-packages directories. This is the downside of this method.
Sometimes, we may use a framework to use the classes and their member functions written in another language from Python. The Boost Python library is an example that interfaces C++ and Python. The framework generates a shared library that contains the exposed classes and functions. This library is visible to Python. Besides the Python modules, we must also copy this shared library to one of the site-packages directories.
4.2. Using the PYTHONPATH Environment Variable
We can expand the module search path using the PYTHONPATH environment variable.
Let’s try it using main.py:
$ pwd
/home/alice/project
$ export PYTHONPATH=$PWD/modules
$ echo $PYTHONPATH
/home/alice/project/modules
$ ./main.py
Hello World
First, we set the environment variable PYTHONPATH, using export PYTHONPATH=$PWD/modules. Its value is /home/alice/project/modules as we set it in the directory /home/alice/project. Then, we ran the script main.py. This time, we were successful in running main.py.
If we use a framework like the Boost Python library, the path of the shared library that interfaces C++ and Python must also be added to the PYTHONPATH environment variable.
5. Conclusion
In this article, we discussed how to augment the default module search path for Python.
Firstly, we saw a simple example that illustrates the problem.
Secondly, we discussed methods that need the update of the calling Python script. We used the path variable in the sys module to expand the default search path for finding custom modules. We can use hard-coded paths, but we must update the path in the Python script if the path of the module changes. Therefore, we saw that getting the path from an environment variable is better.
Finally, we discussed methods that don’t need to modify a calling Python script. The first method used site-packages directories. It’s enough to copy the path configuration file of the custom module to one of the site-packages directories available. However, copying the path configuration file needs root privileges. The second method used the PYTHONPATH environment variable.