Bundle Large Python Application using PyInstaller

Objective

Simple python scripts can be converted into executable using pyinstaller from command line. A project can have other dependencies like data files or run-time libraries (dll or so files). These dependencies need to handle separately by set of instructions defined inside a spec file.

Note: Please follow prerequisite section of this article to install required programs and libraries.

Spec File Operations

The spec file is actually executable Python code. PyInstaller builds the app by executing the contents of the spec file. The statements in a spec file create instances of four classes are Analysis, PYZ, EXE and COLLECT.

• A new instance of class Analysis takes a list of script names as input. It analyzes all imports and other dependencies. The resulting object contains lists of dependencies in class members named:
  • datas: non-binary files included in the app (like .txt or .jpg files)

    added_files = [( 'src/README.txt', '.' ), # README.txt is inside output folder
                         ( '/mygame/sfx/*.mp3', 'sfx' ) # all mp3 files will be inside sfx subfolder 
                        ]
    a = Analysis(...
         datas = added_files,
         ...
         )
    

  • binaries: non-python modules needed by the scripts (like .dll or .so files)
  • hiddenimport: An optional list of additional (hidden) modules to include
  • pathex: An optional list of paths to be searched before sys.path
  • excludes: An optional list of module or package names (their Python names, not path names) that will be ignored (as though they were not found).
  • Sometimes a module doesn't get packed correctly by pyinstaller (can miss submodules, binaries or data files). Below method can be used to pack them correctly.

# If a python module not packed correctly then below mentioned hooks can be used
hiddens = []
data = []
bins = []

from PyInstaller.utils.hooks import collect_submodules
hiddens += collect_submodules('scipy') # ensures all hidden modules included

from PyInstaller.utils.hooks import collect_data_files
data += collect_data_files('scipy') # ensures all data files included

from PyInstaller.utils.hooks import collect_dynamic_libs
bins += collect_dynamic_libs('scipy') # ensures all binaries included

a = Analysis(...
                binaries = bins,
                datas = data,
                hiddenimports = hiddens,
                ...)

• An instance of class PYZ is a .pyz archive which contains all the Python modules from a.pure.
• An instance of EXE is built from the analyzed scripts and the PYZ archive. This object creates the executable file.
• An instance of COLLECT class creates output folder from all other parts.

Create an Executable

To give you hands on experience, I have created a python project. You can clone the project from the github link. Please follow README.md to install required software and libraries. The cloned python program lists all image files inside program/images folder and allows you to switch by using dropdown option.

Create Executable with Supporting Files (Recommended)

• You can create a basic spec file using command pyi-makespec --specpath folder_to_store_generated_spec_file pythonFile.py. You can run createSpec.bat file to generate spec file inside executable forlder. I have created mainOriginal.spec file (please keep it for your reference). You can create your own spec file and edit path of spec file inside buildExec.bat file to build executable.
• The spec file contains instructions to include jpg image files required by the python program.
• We can build executable with command pyinstaller path_to_spec_file. I have included buildExec.bat file that builds executable inside executable folder. If you have different spec file then please edit path of spec file inside it.
• After building the python project, you will find many files along with main.exe inside executable/dist folder.
• You can run main.exe. It will ask for images folder path, just press ENTER key to select default path.

Note: We will learn how to create an installer by combining all the supporting files generated by pyinstaller in the next article.

Create Single Executable

• In this section, we will learn how to create a single executable that contains all the supporting files inside it.
• You can create a basic spec file using command pyi-makespec --specpath folder_to_store_generated_spec_file --onefile pythonFile.py. You can run createSpecOnefile.bat file to generate spec file inside executableOnefile forlder. I have created mainOnefileOriginal.spec file (please keep it for your reference).
• I have included buildExecOnefile.bat file that builds single executable inside executableOnefile folder. If you have different spec file then please edit path of spec file inside it.
• After building the python project, you will find single mainOnefile.exe inside executableOnefile/dist folder.
• Please run main.exe file from CMD. After running exe, you will file a folder with name _MEIxxxxx (Example: _MEI93842)temporary folder created. This is an important folder as all the supporting files inside the executable are extracted inside this temporary folder.
• In mainOnefileOriginal.spec, I have included runtime_tmpdir value as current folder. So temporary folder is created in current folder.
• The program is asking for path to images folder so please press ENTER to select default path. Now, the images folder is taken inside temporary folder as default path(Example: ./_MEI93842/images/).
• There are few drawbacks to create single executable
  • As single executable extracts everything in a temporary folder, so all the relative paths mentioned in the program are invalid.
  • Everytime we run this exe file, it adds extra overhead due to extraction.

Reference

1. Official documentation of PyInstaller spec file
2. PyInstaller documentation in PDF