PyInstaller+Tensorflow+Sonnet

Derek_Magee · November 14, 2021, 4:42pm

I’m having issues building tensorflow+sonnet applications with pyinstaller. I’ve built keras ones fine, but sonnet ones have an issue as below.

Sonnet doesn’t seem to work with pyinstaller. This can be reproduced with a very simple example as below:

import sonnet
print(“Hello world”)

I used a spec file:

-- mode: python ; coding: utf-8 --

The following fixes a different pyinstaller issue due to recursion

import sys
sys.setrecursionlimit(5000)

block_cipher = None

a = Analysis([‘pi_test.py’],
pathex=[‘c:\Software\ICServer\ICServer-src\Python\GraphNet\minimal_pyinstaller_test’],
binaries=,
datas=,
hiddenimports=,
hookspath=,
runtime_hooks=,
excludes=,
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
,
exclude_binaries=True,
name=‘pi_test’,
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=,
name=‘pi_test’)

The error is:

2021-11-08 19:42:43.507236: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Traceback (most recent call last):
File “site-packages\tensorflow\python\autograph\pyct\parser.py”, line 150, in parse_entity
File “site-packages\tensorflow\python\autograph\pyct\inspect_utils.py”, line 147, in getimmediatesource
File “inspect.py”, line 798, in findsource
OSError: could not get source code

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “site-packages\tensorflow\python\autograph\impl\api.py”, line 749, in to_graph
File “site-packages\tensorflow\python\autograph\impl\api.py”, line 284, in _convert_actual
File “site-packages\tensorflow\python\autograph\pyct\transpiler.py”, line 286, in transform
File “site-packages\tensorflow\python\autograph\pyct\transpiler.py”, line 470, in transform_function
File “site-packages\tensorflow\python\autograph\pyct\transpiler.py”, line 346, in transform_function
File “site-packages\tensorflow\python\autograph\pyct\parser.py”, line 152, in parse_entity
ValueError: Unable to locate the source code of <function BaseBatchNorm.call at 0x000002BF2C3C84C0>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “C:\Software\ICServer\ICServer-src\Python\GraphNet\minimal_pyinstaller_test2\pi_test.py”, line 1, in
import sonnet
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in load_unlocked
File “c:\python38\lib\site-packages\PyInstaller\loader\pyimod03_importers.py”, line 623, in exec_module
exec(bytecode, module.dict)
File "site-packages\sonnet_init.py", line 21, in
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “c:\python38\lib\site-packages\PyInstaller\loader\pyimod03_importers.py”, line 623, in exec_module
exec(bytecode, module.dict)
File “site-packages\sonnet\distribute.py”, line 21, in
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “c:\python38\lib\site-packages\PyInstaller\loader\pyimod03_importers.py”, line 623, in exec_module
exec(bytecode, module.dict)
File “site-packages\sonnet\src\distribute\batch_norm.py”, line 22, in
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “c:\python38\lib\site-packages\PyInstaller\loader\pyimod03_importers.py”, line 623, in exec_module
exec(bytecode, module.dict)
File “site-packages\sonnet\src\batch_norm.py”, line 34, in
File “site-packages\sonnet\src\batch_norm.py”, line 132, in BaseBatchNorm
File “site-packages\sonnet\src\utils.py”, line 180, in smart_autograph
File “site-packages\tensorflow\python\autograph\impl\api.py”, line 752, in to_graph
tensorflow.python.autograph.impl.api.ConversionError: converting <function BaseBatchNorm.call at 0x000002BF2C3C84C0>: ValueError: Unable to locate the source code of <function BaseBatchNorm.call at 0x000001EE45EA81F0>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code

I reported this to the sonnet developers and the closed the issue straight away saying it wasn’t their problem and I should report it to tensorflow developers

https://github.com/deepmind/sonnet/issues/222

Sonnet is written in pure python, and the method in question (in src/batch_norm.py) is decorated with @utils.smart_autograph. Can anyone suggest how to get this working?

Thanks

Derek

Bhack · November 15, 2021, 4:12pm

Is this a gast version issue?

https://github.com/deepmind/sonnet/issues/143

Derek_Magee · November 15, 2021, 5:25pm

The gast error looks like a historic thing (and not quite the same error) “On TensorFlow 2.2 or higher, the gast bug has been fixed and gast 0.3+ would be required.”. I’m using tensorflow 2.4 and gast 0.3.3. If I attempt to update gast to the latest version it breaks tensorflow even from the python interpreter.

The error I see is only with pyinstaller, and relates to autograph not being able to access the BaseBatchNorm call (or possibly code within this?) method once packaged.

Bhack · November 16, 2021, 12:31am

Have you tried to use the pyinstaller Tensorflow hooks with a supported version?

github.com

pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-tensorflow.py

# ------------------------------------------------------------------
# Copyright (c) 2020 PyInstaller Development Team.
#
# This file is distributed under the terms of the GNU General Public
# License (version 2.0 or later).
#
# The full license is available in LICENSE, distributed with
# this software.
#
# SPDX-License-Identifier: GPL-2.0-or-later
# ------------------------------------------------------------------

from _pyinstaller_hooks_contrib.compat import importlib_metadata
from packaging.version import Version

from PyInstaller.compat import is_linux
from PyInstaller.utils.hooks import (
    collect_data_files,
    collect_dynamic_libs,
    collect_submodules,

This file has been truncated. show original

Derek_Magee · November 16, 2021, 9:51am

Thanks, I got it working with that hook. Two futher changes were also required:

i) I had to change the line in the hook:

datas = collect_data_files('tensorflow', excludes=data_excludes)

to:
datas = collect_data_files(‘tensorflow’)

as excludes dosn’t seem to be in the API (at least for versions I have Python: 3.8.3, Pyinstaller 3.6)

ii) I had to add the following import to my python file:

import tensorflow.python.keras.engine.base_layer_v1

Thanks a lot!

D.