error running install script from the gpu branch

[joao-bapdm]

Hello,

I am trying this procedure in a Ubuntu 19.04 machine with cuda-10.1 installed.

I tried the following command:

python3 firedrake-install --slepc --disable-ssh --with-parmetis --venv-name firedrake-gpu --install pyadjoint --cuda --package-branch firedrake gpu --package-branch tsfc gpu --package-branch PyOP2 cuda -v

And got the following log file:

firedrake-install.log

In particular, the installation of PyOP2 fails:

Installing collected packages: PyOP2
Running setup.py develop for PyOP2
Running command /home/joao/envs/firedrake-gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/joao/envs/firedrake-gpu/src/PyOP2/setup.py'"'"'; file='"'"'/home/joao/envs/firedrake-gpu/src/PyOP2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps
running develop
running egg_info
writing PyOP2.egg-info/PKG-INFO
writing dependency_links to PyOP2.egg-info/dependency_links.txt
writing requirements to PyOP2.egg-info/requires.txt
writing top-level names to PyOP2.egg-info/top_level.txt
reading manifest file 'PyOP2.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
/usr/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'test_requires'
warnings.warn(msg)
warning: no files found matching '*.c' under directory 'pyop2'
writing manifest file 'PyOP2.egg-info/SOURCES.txt'
running build_ext
cythoning pyop2/sparsity.pyx to pyop2/sparsity.c
building 'pyop2.sparsity' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/pyop2
mpicc -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Ipyop2 -I/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/numpy/core/include -I/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/petsc4py/include -I/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/petsc/include -I/home/joao/envs/firedrake-gpu/include -I/usr/include/python3.7m -c pyop2/sparsity.c -o build/temp.linux-x86_64-3.7/pyop2/sparsity.o
In file included from /home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830,
from /home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from pyop2/sparsity.c:598:
/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it with "
^~~~~~~
In file included from /home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/petsc/include/petsc/private/matimpl.h:7,
from pyop2/sparsity.c:601:
/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/petsc/include/petsc/private/petscimpl.h:20:10: fatal error: cuda.h: No such file or directory
#include <cuda.h>
^~~~~~~~
compilation terminated.
/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/joao/envs/firedrake-gpu/src/PyOP2/pyop2/sparsity.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
error: command 'mpicc' failed with exit status 1
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-ettixjnp'
ERROR: Command errored out with exit status 1: /home/joao/envs/firedrake-gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/joao/envs/firedrake-gpu/src/PyOP2/setup.py'"'"'; file='"'"'/home/joao/envs/firedrake-gpu/src/PyOP2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
Exception information:
Traceback (most recent call last):
File "/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
status = self.run(options, args)
File "/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 407, in run
use_user_site=options.use_user_site,
File "/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/pip/_internal/req/init.py", line 58, in install_given_reqs
**kwargs
File "/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 918, in install
install_options, global_options, prefix=prefix,
File "/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 774, in install_editable
cwd=self.setup_py_dir,
File "/home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 931, in call_subprocess
raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/joao/envs/firedrake-gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/joao/envs/firedrake-gpu/src/PyOP2/setup.py'"'"'; file='"'"'/home/joao/envs/firedrake-gpu/src/PyOP2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

2019-10-09 09:40:54,216 INFO

Install log saved in /home/joao/envs/firedrake-install.log

In bold the fatal error message, and the failed command a little above it. mpicc was successfully executed before, but using the option -I/usr/local/cuda/include, where cuda.h is stored.

How could I proceed with the installation?

Best regards,

João Moreira

[wence-]

I will tag in @kaushikcfd who has been doing more work on this. Please note, these branches are very work-in-progress right now.

I think the problem is likely that we don't have a good way of saying in the install script what extra flags (if any) should be provided to compilers.

[joao-bapdm]

The installation worked after adding my cuda library path to the includes list in the PyOP2 setup.py file.

Running the test suit gives the following:

Building extension modules
Linting firedrake codebase
firedrake/functionspace.py:142:17: E128 continuation line under-indented for visual indent
firedrake/functionspace.py:143:17: E128 continuation line under-indented for visual indent
firedrake/functionspace.py:185:13: E128 continuation line under-indented for visual indent
firedrake/functionspaceimpl.py:268:13: E128 continuation line under-indented for visual indent
firedrake/functionspaceimpl.py:277:17: E128 continuation line under-indented for visual indent
firedrake/functionspaceimpl.py:278:17: E128 continuation line under-indented for visual indent
firedrake/mesh.py:652:13: E128 continuation line under-indented for visual indent
firedrake/mesh.py:660:33: E128 continuation line under-indented for visual indent
firedrake/functionspacedata.py:101:9: E128 continuation line under-indented for visual indent
firedrake/functionspacedata.py:113:13: E128 continuation line under-indented for visual indent
firedrake/functionspacedata.py:137:9: E128 continuation line under-indented for visual indent
firedrake/functionspacedata.py:408:13: E128 continuation line under-indented for visual indent
firedrake/functionspacedata.py:433:9: E303 too many blank lines (2)
firedrake/functionspacedata.py:564:9: E128 continuation line under-indented for visual indent
firedrake/functionspacedata.py:580:13: E128 continuation line under-indented for visual indent
firedrake/functionspacedata.py:581:13: E128 continuation line under-indented for visual indent
make: *** [Makefile:9: lint] Error 1

The helmholtz.py example also fails:

Traceback (most recent call last):
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspacedata.py", line 51, in cached
return cache[key]
KeyError: (((0, (0,), (1,), (2,)), (1, (), (), ()), (2, ())), False)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "helmholtz.py", line 10, in
V = FunctionSpace(mesh, "CG", 1)
File "</home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/decorator.py:decorator-gen-49>", line 2, in FunctionSpace
File "/home/joao/envs/firedrake-gpu/src/PyOP2/pyop2/profiling.py", line 60, in wrapper
return f(*args, **kwargs)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspace.py", line 118, in FunctionSpace
element = make_scalar_element(mesh, family, degree, vfamily, vdegree)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspace.py", line 40, in make_scalar_element
mesh.init()
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/mesh.py", line 1063, in init
self._callback(self)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/mesh.py", line 1424, in callback
should_reorder=False)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspace.py", line 185, in VectorFunctionSpace
should_reorder=should_reorder)
File "</home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/decorator.py:decorator-gen-49>", line 2, in FunctionSpace
File "/home/joao/envs/firedrake-gpu/src/PyOP2/pyop2/profiling.py", line 60, in wrapper
return f(*args, **kwargs)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspace.py", line 143, in FunctionSpace
should_reorder=should_reorder)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspaceimpl.py", line 278, in init
should_reorder=should_reorder)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspacedata.py", line 581, in get_shared_data
should_reorder=should_reorder)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspacedata.py", line 428, in init
self.entity_node_lists, self.old_to_new_ordering = get_entity_node_lists(mesh, (edofs_key, real_tensorproduct), entity_dofs, global_numbering, self.offset, should_reorder)
File "</home/joao/envs/firedrake-gpu/lib/python3.7/site-packages/decorator.py:decorator-gen-35>", line 2, in get_entity_node_lists
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspacedata.py", line 53, in cached
result = f(mesh, key, *args, **kwargs)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspacedata.py", line 151, in get_entity_node_lists
cell_node_list, old_to_new_ordering = get_cell_node_list(mesh, entity_dofs, global_numbering, offsets, should_reorder)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/functionspacedata.py", line 113, in get_cell_node_list
should_reorder=should_reorder)
File "/home/joao/envs/firedrake-gpu/src/firedrake/firedrake/mesh.py", line 668, in make_cell_node_list
assert ncells % batch_size == 0
AssertionError

So it seems that the installation worked, but for my machine configuration the software as it is didn't..

[kaushikcfd]

So it seems that the installation worked, but for my machine configuration the software as it is didn't..

I have pushed some fixes to the packages firedrake@gpu and PyOP2@cuda(the problem was: there were some data layout optimizations which were being tried whose implementation wasn't general enough, so I've reverted them for now). You would need to pull the corresponding branches and from the directory /home/joao/envs/firedrake-gpu/src the command python -m pip install --no-binary mpi4py --no-deps -vvv -e firedrake/ must be run after activating the firedrake-gpu environment.

The above helmholtz test would still fail as we only support matrix free assembly of the forms.

[joao-bapdm]

I've updated the files and ran the four examples from https://firedrakeproject.org/matrix-free.html, getting a similar result for all of them:

poisson.log
stokes.log
navier_stokes.log
rayleigh-benard.log

[connorjward]

Closing as per our policy of closing old installation issues after >6 months. Note also that the GPU branches are now very stale.