2. Virtual environment, Package and Distribution

2.1. Anadconda virtual environment

2.1.1. Installation

  • Go to Anaconda webpage and download the latest Python3 version for your OS (i.e. Linux or Windows).
  • Windows: Double click the downloaded .exe file to install the Anaconda. During installation, it will ask to include the Anaconda in the Environment variable, Press “YES” there.
  • Linux: Do the below for installation
$ chmod 777 <downloaded .sh file>
$ ./<downloaded .sh file>
(When asked, press yes to include the Anaconda in the Path)
  • Verify installation : Open terminal and run the ‘python’ command. It should show the Install version of Python e.g. Python 3.6.5 in the below result
C:\Users\meherp> python
    Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) ...
    >>>exit() # to exit from Python Shell

2.1.2. Creating virtual environment

  • Python applications will often use packages and modules that don’t come as part of the standard library. Applications will sometimes need a specific version of a library, because the application may require that a particular bug has been fixed or the application may be written using an obsolete version of the library’s interface.
  • This means it may not be possible for one Python installation to meet the requirements of every application. If application A needs version 1.0 of a particular module but application B needs version 2.0, then the requirements are in conflict and installing either version 1.0 or 2.0 will leave one application unable to run.
  • The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.
  • Creating virtual environment is very easy in Anaconda as shown below,
(create env: "conda create -n <name of env> python=x.x" where x.x is Python version)
$ conda create -n myenv python=3.5


[...]
Proceed ([y]/n)? y


Downloading and Extracting Packages
wincertstore-0.2     | 13 KB     | ############################################################################ | 100%
pip-10.0.1           | 1.8 MB    | ############################################################################ | 100%
wheel-0.31.1         | 81 KB     | ############################################################################ | 100%
certifi-2018.8.24    | 140 KB    | ############################################################################ | 100%
python-3.5.6         | 18.2 MB   | ############################################################################ | 100%
setuptools-40.2.0    | 597 KB    | ############################################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > activate myenv
#
# To deactivate an active environment, use:
# > deactivate
#
# * for power-users using bash, you must source
#
  • We need to activate this env to use it. You can see the activate command in above result as well.
$ activate myenv
(if activated successfully, then name of the env will be show as below i.e. (myenv))
(myenv) C:\Users\meherp>deactivate

Note

(see activate command in above result, sometimes below result are shown in different OS/version)

$ source activate myenv
$ conda activate myenv
  • Now, we can install various libraries to this environment without affecting our original instillation (or other environments); i.e. the changes will remain within this environment only. In the below result we can see that the ‘wget’ library is not available in the current environment,
(myenv) C:\Users\meherp>python
Python 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 16:05:27) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import wget
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'wget'
>>> exit()
  • We can use ‘conda install’ or ‘pip install’ option to install the libraries in the environment as shown below,
(myenv) C:\Users\meherp>pip install wget

Collecting wget
  Downloading https://files.pythonhosted.org/packages/47/6a/62e288da7bcda82b935ff0c6cfe542970f04e29c756b0e147251b2fb251f/wget-3.2.zip
Building wheels for collected packages: wget
  Running setup.py bdist_wheel for wget ... done
  Stored in directory: C:\Users\meherp\AppData\Local\pip\Cache\wheels\40\15\30\7d8f7cea2902b4db79e3fea550d7d7b85ecb27ef992b618f3f
Successfully built wget
Installing collected packages: wget
Successfully installed wget-3.2
You are using pip version 10.0.1, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
  • Now, we can see that the ‘wget’ library is installed in the environment,
(myenv) C:\Users\meherp>python
Python 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 16:05:27) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import wget
>>> exit()
  • Deactivate the environment
(myenv) C:\Users\meherp>deactivate

C:\Users\meherp>

Note

Use below based on the message which is shown while creating the environment

$ source deactivate
$ conda deactivate

2.1.3. Other commands

Below are some other useful command related to virtual environment,

  • List of environments
(show the list of enviroments)
$ conda env list


# conda environments:
#
base * C:\Users\meherp\AppData\Local\Continuum\anaconda3
audio C:\Users\meherp\AppData\Local\Continuum\anaconda3\envs\audio
caffe C:\Users\meherp\AppData\Local\Continuum\anaconda3\envs\caffe
myenv C:\Users\meherp\AppData\Local\Continuum\anaconda3\envs\myenv
tensorflow_env C:\Users\meherp\AppData\Local\Continuum\anaconda3\envs\tensorflow_env
wts C:\Users\meherp\AppData\Local\Continuum\anaconda3\envs\wts
  • Install libraries using .txt file
  • Let’s create a text file ‘myEnv.txt’ at the desired location e.g. Desktop with below content
flask==1.0.0
django
  • Install the packages using this file as below. Note that, all the dependent libraries will be downloaded & installed automatically e.g. Django depends on ‘pytz’ and ‘sqlparse’ library
(myenv) C:\Users\meherp>pip install -r C:\Users\meherp\Desktop\myEnv.txt
[...]
Installing collected packages: itsdangerous, MarkupSafe, Jinja2, click, Werkzeug, flask, sqlparse, pytz, django
Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 Werkzeug-0.15.4 click-7.0 django-2.2.2 flask-1.0 itsdangerous-1.1.0 pytz-2019.1 sqlparse-0.3.0
  • See the list of installed libraries: ‘freeze’ command is used for it,
(myenv) C:\Users\meherp>pip freeze

certifi==2018.8.24
Click==7.0
Django==2.2.2
Flask==1.0
itsdangerous==1.1.0
Jinja2==2.10.1
MarkupSafe==1.1.1
pytz==2019.1
sqlparse==0.3.0
Werkzeug==0.15.4
wget==3.2
wincertstore==0.2
  • Export the list of install libraries to a file,
(C:\Users\meherp\Desktop\copyMyEnv.txt is the saved location)
(myenv) C:\Users\meherp>pip freeze > C:\Users\meherp\Desktop\copyMyEnv.txt
  • ‘copyMyEnv.txt’ will be created at Desktop with below content.
certifi==2018.8.24
Click==7.0
Django==2.2.2
Flask==1.0
itsdangerous==1.1.0
Jinja2==2.10.1
MarkupSafe==1.1.1
pytz==2019.1
sqlparse==0.3.0
Werkzeug==0.15.4
wget==3.2
wincertstore==0.2
  • Deactivate and remove the environment
conda deactivate

(delete the environment)
conda remove --name myenv --all

2.2. Virtual environment using virtualenv and virtualenvw

2.2.1. Install virtualenv and virtualenvwrapper

  • Install virtualenv and virtualenvwrapper
$ pip install virtualenv virtualenvwrapper

2.2.2. Modify .bashrc

  • Add below line at the end of .bashrc file if virtual environment is installed in the Anaconda environment
# replace /home/meherp/anaconda2 with correct installation location

# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/home/meherp/anaconda2/bin/python
source /home/meherp/anaconda2/bin/virtualenvwrapper.sh
  • Add below at the end of .bashrc file if virtualenv environment is created outside the Anaconda environment.
# replace python3 with correct version i.e. python3 or python2

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3

2.2.3. Create virtualenv

Test the installation by creating a virtualenv and installing numpy to it.

$ source ~/.bashrc
$ mkvirtualenv vtest -p python3
$ pip install numpy
  • activate the environment,
$ workon vtest
  • Deactivate the environment
$ deactivate
  • Delete the environment
$ rmvirtualenv vtest

2.3. Packages

In this section, we will learn to create the ‘packages’, which are nothing but the collection of modules under the same name.

2.3.1. Location of installed packages

Before creating our own package, let’s see the location of the installed packages. The installed packages (using pip command) are saved in the folder ‘lib/python3.6/site-packages’, as shown below,

(for environment 'pythondsp')
pythondsp/lib/python3.6/site-packages

(without environment)
/home/anaconda3/lib/python3.6/site-packages

Apart from current working directory, the Python look for the files in the folder ‘site-packages’. We will understand this in Section 2.3.4. The complete list of the directories can be seen using ‘sys.path’ command.

(pythondsp) $ python

>>> import sys

>>> sys.path
[
    [...]
    '/home/pythondsp/lib/python3.6/site-packages',
    [...]
]

2.3.2. Create files

In this section, we will write some Python codes, and then in next section, we will convert these Python modules into a package.

  • Activate the environment, and create a folder with any name e.g. ‘bucket’ at desired location
(pythondsp) $ mkdir -p ~/Desktop/bucket
(pythondsp) $ cd ~/Desktop/bucket

Now create two files inside the folder with following contents,

# bucket/my_calc.py

def sum2Num(x, y):
    return (x+y)

def diff2Num(x, y):
    return (x-y)
# bucket/my_work.py

from my_calc import sum2Num, diff2Num

x = 10
y = 5

print("{0} + {1} = {2}".format(x, y, sum2Num(x, y)))
print("{0} - {1} = {2}".format(x, y, diff2Num(x, y)))

Next, check execute ‘my_work.py’ to check the setup,

Note

First ‘import my_work’ will import everything from the file, therefore ‘print’ statement will be executed. But the second import will import the values from the ‘cache’ (not from the file), therefore ‘print’ statements from the file will not be executed.

(pythondsp) $ python my_work.py
10 + 5 = 15
10 - 5 = 5

Also, check the below commands,

(pythondsp) $ python

>>> from my_calc import sum2Num
>>> sum2Num(2, 4)
6

>>> import my_work
10 + 5 = 15
10 - 5 = 5
>>> import my_work
>>>

2.3.3. Packages

In this section, we will convert the Python modules into the package.

Note

  • Package name should be unique, so that it will not collide with other package names.
  • Create another folder inside the folder ‘bucket’ with any desired name e.g. ‘wolfpack’, and move the ‘python files’ inside it. After running below commands, we will have following folder structure (after excluding the folder ‘__pycache__),

Note

The folder (i.e. ‘wolfpack’) inside the ‘root folder (i.e. bucket)’ is called the ‘package’ and needs special settings to use it with root-folder.

bucket/
└── wolfpack
    ├── my_calc.py
    └── my_work.py
(pythondsp) $ mkdir wolfpack
(pythondsp) $ mv my_work.py my_calc.py wolfpack
  • Now, run the shell commands again as shown below. Note that the command at Line 4 is working fine, but command at Line 6 is generating error.
Listing 2.1 Import package and error
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
(pythondsp) $ python

>>> from wolfpack.my_calc import sum2Num
>>> sum2Num(10, 2)
12
>>> from wolfpack import my_work
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/meher/Desktop/bucket/wolfpack/my_work.py", line 3, in <module>
    from my_calc import sum2Num, diff2Num
ModuleNotFoundError: No module named 'my_calc'

Note

The error at Line 6 is ‘No module named ‘my_calc’, as we are running the command from the ‘root directory (i.e. bucket)’, therefore the line ‘import my_calc’ in ‘my_work.py’ will look for it in the root directory only (not inside the ‘wolfpack’). The error can be removed by modifying the code as below,

In the below code, we used ‘from .my_calc import sum2Num, diff2Num’. The ‘dot operator’ tells python to look in the current directory (i.e. wolfpack), not in the ‘root directory’.

# bucket/wolfpack/my_work.py

from .my_calc import sum2Num, diff2Num

x = 10
y = 5

print("{0} + {1} = {2}".format(x, y, sum2Num(x, y)))
print("{0} - {1} = {2}".format(x, y, diff2Num(x, y)))
  • Close the Python terminal and open it again. Then execute the commands and it will work fine,
(pythondsp) $ python

>>> from wolfpack import my_work
10 + 5 = 15
10 - 5 = 5

Warning

  • Now, the folder ‘wolfpack’ is a ‘package’ and the files inside it can not be executed directly, as these files have ‘dot’ operators in the ‘import’ statement. Following error will be generated if we run the ‘package-module’ directly.
(pythondsp) $ cd wolfpack/

(pythondsp) $ ls
my_calc.py  my_work.py

(pythondsp) $ python my_work.py
Traceback (most recent call last):
  File "my_work.py", line 3, in <module>
    from .my_calc import sum2Num, diff2Num
ModuleNotFoundError: No module named '__main__.my_calc';
  • The files in the package module can be executed through root-folders (i.e. bucket) by importing the modules.

  • Also, the folder ‘wolfpack’ can be called as a python-file in the codes i.e. ‘import wolfpack’

  • We will use following terms for the two folders,

    • Root directory : ‘bucket’
    • Package directory : ‘wolfpack’

2.3.4. Globally available package

If we ‘cut and paste’ the package folder (i.e. wolfpack) inside the folder ‘site-packages’, then it will be available globally in the environment, i.e. it can be imported into any project at any location.

Note

Do not cut and paste the folder now. We will create the package in Section 2.4 and use ‘setup.py’ command to install the package.

2.3.5. __init__ file

In Listing 2.1, we import the function ‘sum2Num’ using following command .

from wolfpack.my_calc import sum2Num'

Currently, we have only two files in the package therefore it’s easy to import ‘function’ like this. But, if we have 100 files with 10 folders in a package, then it will be difficult/inconvenient to remember the import-location of the ‘methods’. A better approach is to to use the ‘__init__.py’ file as shown below,

  • First go to package folder and create an __init__.py with following content,
# wolfpack/__init__.py

# import functions from my_calc
from .my_calc import sum2Num, diff2Num
  • Now, we can import the commands directly without knowing the file structure,
>>> # run from root-folder 'bucket'
>>> from wolfpack import sum2Num, diff2Num
>>> sum2Num(3, 12)
15

2.3.6. __all__ in __init__ file

If we want to allow ‘import *’ option for our package, then we need to add the magic keyword ‘__all__’ in the __init__.py file,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# wolfpack/__init__.py

# import sum2Num from my_calc
from .my_calc import sum2Num

# import 'my_calc' and 'my_calc' for 'import *'
__all__ = [ "sum2Num",
            "my_calc",
            "my_work"
            ]

Now, use the ‘import *’ command in the Python shell as below. Following items will be imported with * command,

  • module ‘my_calc’
  • module ‘my_work’
  • function ‘sum2Num’

Warning

  • In Line 4 of above code, only ‘sum2Num’ is imported. If we do not include sum2Num in the ‘__all__ (Line 7)’, then ‘__all__’ will overwrite the Line 4 and ‘sum2Num’ will not be available. And error (in below code) similar to ‘diff2Num(10, 10)’ will be shown for sum2Num.
(run from the root-folder 'bucket')
(pythondsp) $ python

>>> from wolfpack import *
10 + 5 = 15
10 - 5 = 5
>>>
>>> my_work.x
10
>>> sum2Num(10, 10)
20
>>>
>>> my_calc.diff2Num(25, 5)
20
>>>
>>> diff2Num(10, 10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'diff2Num' is not defined

2.4. Distribute the package

In this section, we will convert our package to ‘distributable package’, which can be used by others as well,

Note

We need to create the setup.py file inside the root-folder ‘bucket’ to generate the ‘distribution’. Suppose, if we have the following ‘directory structure’.

bucket/
├── documentation.txt   # document the package here.
├── my_exec.py          # executable python file
├── my_file1.py         # python file
├── my_file2.py         # python file
├── readme.txt          # add tips here....
├── setup.py            # needs to create a 'distribution'
└── wolfpack            # package
    ├── __init__.py
    ├── item1.py
    └── item2.py

Then our setup.py file will be as follows,

# setup.py

from distutils.core import setup

setup(name = "wolf",                       # choose any name
    version = "1.0",                       # give version number
    py_modules = ['my_file1', 'my_file2'], # python files
    packages = ['wolfpack'],               # add package here
    scripts = ['my_exec.py'],              # executable
)
  • Since our project have only ‘package’ in it, therefore we will use below ‘setup.py’ configuration,
# setup.py

from distutils.core import setup

setup(name = "wolfpack",     # choose any name
    version = "1.0",         # give version number
    packages = ['wolfpack'], # add package here
)
  • Next, run the setup.py as below and it will create a ‘distribution’. It will show some warning as we did not include the ‘README.txt’ and ‘MANIFEST.in’ etc.
(pythondsp) $ python setup.py sdist

running sdist
running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list)

warning: sdist: standard file not found: should have one of README, README.txt

writing manifest file 'MANIFEST'
creating wolfpack-1.0
creating wolfpack-1.0/wolfpack
making hard links in wolfpack-1.0...
hard linking setup.py -> wolfpack-1.0
hard linking wolfpack/__init__.py -> wolfpack-1.0/wolfpack
hard linking wolfpack/my_calc.py -> wolfpack-1.0/wolfpack
hard linking wolfpack/my_work.py -> wolfpack-1.0/wolfpack
creating dist
Creating tar archive
removing 'wolfpack-1.0' (and everything under it)
  • The resultant distribution will be saved as ‘wolfpack-1.0.tar.gz’ inside the ‘dist’ folder.
  • Extract the ‘zipped’ file. Then execute the setup.py file in the unzipped folder as below,
$ cd dist/wolfpack-1.0
$ python setup.py install
  • Next, see the ‘site-packages’ folder for ‘wolfpack’ package; or use below command to check the list of packages.
(pythondsp) $ pip list

[...]
wolfpack (1.0)
  • Finally, check the working of the package. Create a folder at any location and run the below commands,
(pythondsp) $ python

>>> from wolfpack import *
10 + 5 = 15
10 - 5 = 5
>>> sum2Num(10, 10)
20
>>> my_calc.diff2Num(3,1)
2

2.5. Conclusion

In this chapter, we learn about the virtual environment and packages. Also, we saw the method by which we can convert the ‘package’ into the ‘distribution’.