Generate code for Enzyme autodiff for functions with Pointer/array arguments

[Nirhar]

For functions of type
double myfunction(double* arr){...};
OR double myfunction(double arr[3]){...};

we generate derivative code of type:

        void d_myfunction(double* arr, clad::array_ref<double> _d_arr){
		double* d_arr = _d_arr.ptr();
		__enzyme_autodiff_myfunction(myfunction, arr, d_arr);
	}

Further on enzyme will handle the function differentiation

This Commit also Downloads and Installs Enzyme in the Clad Directory, if
the flag "-DENZYME=On" is passed to cmake during project configuration.

This Commit also contains tests for the above feature in test/Gradient/Enzyme.C

[Nirhar]

@vgvassilev @parth-07
The clad code in this pull request throws the error

"cannot initialize a parameter of type 'double (double *)' with an lvalue of type 'double (double *)'"

during compilation of the below code:

#include "clad/Differentiator/Differentiator.h"
double f(double *arr) { return arr[0] * arr[1]; }
int main() {
    auto f_grad=clad::gradient<clad::opts::use_enzyme>(f);
}

Any idea how to resolve this?

[parth-07]

@vgvassilev @parth-07 The clad code in this pull request throws the error

"cannot initialize a parameter of type 'double (double *)' with an lvalue of type 'double (double *)'"

during compilation of the below code:

#include "clad/Differentiator/Differentiator.h"
double f(double *arr) { return arr[0] * arr[1]; }
int main() {
    auto f_grad=clad::gradient<clad::opts::use_enzyme>(f);
}

Any idea how to resolve this?

Are you able to print generated derived function? If yes, can you please paste that in a comment here?

[Nirhar]

@parth-07 No, I am not able to print the generated function. The same error is thrown when I attempt to print it.

[parth-07]

@parth-07 No, I am not able to print the generated function. The same error is thrown when I attempt to print it.

Have you tried printing the derived function using the flag -fdump-derived-fn?

[parth-07]

"cannot initialize a parameter of type 'double (double *)' with an lvalue of type 'double (double *)'"

The error suggests that you are using an incorrect value category (https://en.cppreference.com/w/cpp/language/value_category) for the function argument of the call to __enzyme_autodiff_. Correctly generating clang::DeclRefExpr node may fix the issue. You can use, Sema::BuildDeclRefExpr or VisitorBase::BuildDeclRef for correctly generating it.

[Nirhar]

@parth-07 No, I am not able to print the generated function. The same error is thrown when I attempt to print it.

Have you tried printing the derived function using the flag -fdump-derived-fn?

Yes I tried that, the generated function does not contain the __enzyme_autodiff function call

[Nirhar]

"cannot initialize a parameter of type 'double (double *)' with an lvalue of type 'double (double *)'"

The error suggests that you are using an incorrect value category (https://en.cppreference.com/w/cpp/language/value_category) for the function argument of the call to __enzyme_autodiff_. Correctly generating clang::DeclRefExpr node may fix the issue. You can use, Sema::BuildDeclRefExpr or VisitorBase::BuildDeclRef for correctly generating it.

Okay let me try this out

[Nirhar]

This code runs on my machine, but here the tests throw the error fatal: could not create work tree dir 'Enzyme': Permission denied. How do I deal with this @vgvassilev @parth-07 ?

[Nirhar]

Now I am getting the below error in the CI tests:

CMake Error: Could not open file for write in copy operation /usr/local/EnzymeAD/tmp/Enzyme-mkdirs.cmake.tmp

I am not getting this error on my machine. How do I resolve this @vgvassilev @parth-07?

[parth-07]

Now I am getting the below error in the CI tests:

CMake Error: Could not open file for write in copy operation /usr/local/EnzymeAD/tmp/Enzyme-mkdirs.cmake.tmp

I am not getting this error on my machine. How do I resolve this @vgvassilev @parth-07?

It seems to be a permission issue. Can you try to install it in a non-root directory? And also, check if we are missing the right permissions somewhere.

[Nirhar]

Now I am getting the below error in the CI tests:

CMake Error: Could not open file for write in copy operation /usr/local/EnzymeAD/tmp/Enzyme-mkdirs.cmake.tmp

I am not getting this error on my machine. How do I resolve this @vgvassilev @parth-07?

It seems to be a permission issue. Can you try to install it in a non-root directory? And also, check if we are missing the right permissions somewhere.

Currently I am installing Enzyme in the "CMAKE_INSTALL_PREFIX". I think in the CI machines this is in some root directory. Where do I install it, is the build_dir a good place?

[parth-07]

Now I am getting the below error in the CI tests:

CMake Error: Could not open file for write in copy operation /usr/local/EnzymeAD/tmp/Enzyme-mkdirs.cmake.tmp

I am not getting this error on my machine. How do I resolve this @vgvassilev @parth-07?

It seems to be a permission issue. Can you try to install it in a non-root directory? And also, check if we are missing the right permissions somewhere.

Currently I am installing Enzyme in the "CMAKE_INSTALL_PREFIX". I think in the CI machines this is in some root directory. Where do I install it, is the build_dir a good place?

You can just specify a non-root directory for the CMAKE_INSTALL_PREFIX option in the CMake configure command.

[parth-07]

Where do I install it, is the build_dir a good place?

Any non-root directory should work for the debugging purpose right now. We would like things to work as expected when the user is installing in root locations as well. Thus the main goal is to find the underlying issue that's causing the CMake configuration to fail.

[Nirhar]

Its mentioned in the Enzyme website that Enzyme has been tested against LLVM 7,8,9,10,11,12. I personally tested it against LLVM 13 and 14 and it works. Maybe the lower versions of LLVM/clang are causing a problem? As I have seen, most failures in the CI are because the Enzyme build has failed.

[parth-07]

Its mentioned in the Enzyme website that Enzyme has been tested against LLVM 7,8,9,10,11,12. I personally tested it against LLVM 13 and 14 and it works. Maybe the lower versions of LLVM/clang are causing a problem? As I have seen, most failures in the CI are because the Enzyme build has failed.

Can you test enzyme integration with Clad when lower versions of LLVM are used in your local system? If you are getting the same/similar errors in the local system as well, then it will be easier to debug.

[vgvassilev]

LGTM! Great work @Nirhar!

The codecov coverage reports are mostly due to code using the new pass manager. Unfortunately enzyme has not yet moved to the new pass manager. It's registration happens via the old way. However, I'd like to keep the new way around and they move we will do little work to reuse the already put in place infrastructure.