Skip to content

WIP Indexing autograd and tests, new loss functions #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

WIP Indexing autograd and tests, new loss functions #47

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0410583
adds googletest framework for testing
syurkevi Sep 24, 2018
54fd74f
adds indexing and softmax autograd functions
syurkevi Oct 1, 2018
88bee36
adds CrossEntropyLoss and SVMLoss
syurkevi Oct 1, 2018
eaa986e
remove autograd example, now part of tests
syurkevi Oct 1, 2018
eea9572
change names to assign/lookup, remove unnecessary copies
syurkevi Oct 2, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "extern/googletest"]
path = extern/googletest
url = https://github.com/google/googletest.git
6 changes: 6 additions & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,9 @@ set_target_properties(afml
CXX_STANDARD 11)

add_subdirectory(examples)

option(PACKAGE_TESTS "Build tests" ON)
if(PACKAGE_TESTS)
enable_testing()
add_subdirectory(tests)
endif()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can avoid creating a separate option PACKAGE_TESTS. gtest automatically addes BUILD_TESTING which you can use.

189 changes: 0 additions & 189 deletions examples/autograd.cpp
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions extern/googletest
Open in desktop
Submodule googletest added at ed6e84
3 changes: 3 additions & 0 deletions include/af/autograd/Functions.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ namespace af {
Variable operator <=(const Variable &lhs, const double &rhs);

Variable operator !(const Variable &input);
Variable select_index(const Variable &input, const Variable &idx);
Variable set_index(const Variable &input, const Variable &idx, const Variable &vals);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think lookup and assign are better. Or go withgather vs scatter like tensorflow.


Variable negate(const Variable &input);
Variable reciprocal(const Variable &input);
Expand All @@ -54,6 +56,7 @@ namespace af {
Variable cos(const Variable &input);
Variable tanh(const Variable &input);
Variable sigmoid(const Variable &input);
Variable softmax(const Variable &input);

Variable max(const Variable &lhs, const Variable &rhs);
Variable max(const Variable &lhs, const double &rhs);
Expand Down
27 changes: 27 additions & 0 deletions include/af/nn/Modules/Loss.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,36 @@ namespace af
const autograd::Variable &weights);
};

class CrossEntropyLoss : public Loss
{
public:
CrossEntropyLoss() {}

autograd::Variable forward(const autograd::Variable &inputs,
const autograd::Variable &targets);

autograd::Variable forward(const autograd::Variable &inputs,
const autograd::Variable &targets,
const autograd::Variable &weights);
};

class MultiMarginLoss : public Loss
{
public:
MultiMarginLoss() {}

autograd::Variable forward(const autograd::Variable &inputs,
const autograd::Variable &targets);

autograd::Variable forward(const autograd::Variable &inputs,
const autograd::Variable &targets,
const autograd::Variable &weights);
};

typedef MeanSquaredError MSE;
typedef MeanAbsoluteError MAE;
typedef MeanAbsoluteError L1Loss;
typedef BinaryCrossEntropyLoss BCELoss;
typedef CrossEntropyLoss CELoss;
}
}
49 changes: 48 additions & 1 deletion src/autograd/Functions.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,35 @@ namespace af {
return Variable(result, false);
}

Variable select_index(const Variable &input, const Variable &idx)
{
af::array result = input.array()(idx.array());
af::array mask = af::constant(0, input.dims());
mask(idx.array()) = 1;

auto grad_func = [](std::vector<Variable> &inputs, const Variable &grad_output) {
auto grad = inputs[2].array();
auto grad_mask = af::where(grad);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you converting idx into a mask and converting it back into an index again? This is making unnecessary copies.

grad(grad_mask) *= grad_output.array();

inputs[0].addGrad(Variable(grad, false));
};
return Variable(result, {input, idx, Variable(mask, false)}, grad_func);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You only need idx or mask. Not both.

}

Variable set_index(const Variable &input, const Variable &idx, const Variable &vals)
{
af::array result = input.array();
result(idx.array()) = vals.array();
af::array mask = af::constant(1, input.dims(), s32);
mask(idx.array()) = 0;

auto grad_func = [](std::vector<Variable> &inputs, const Variable &grad_output) {
inputs[0].addGrad(inputs[3] * grad_output);
};
return Variable(result, {input, idx, vals, Variable(mask, false)}, grad_func);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is better to use just index and do the masking inside the grad_func. No reason to do the masking op until it is necessary.

}

Variable max(const Variable &lhs, const Variable &rhs)
{
auto mask = lhs > rhs;
Expand Down Expand Up @@ -241,6 +270,24 @@ namespace af {
return Variable(result, {input}, grad_func);
}

Variable softmax(const Variable &input)
{
//todo: add axis to apply?
auto exps = exp(input.array());
auto result = exps / tile(sum(exps, 0), exps.dims(0));
auto grad_func = [](std::vector<Variable> &inputs, const Variable &grad_output) {
auto exps = exp(inputs[0]);
auto tmp = exps / tileAs(sum(exps, {0}), exps);

auto ps_j = tile(tmp, { 1, (int)tmp.dims()[0] } );
auto ps_i = transpose(tile(tmp, {1,(int)tmp.dims()[0] } ));
Variable I(identity((int)tmp.dims()[0], (int)tmp.dims()[0]), false);
auto jac = (sum(ps_i * (I - ps_j), { 1 }));
inputs[0].addGrad(grad_output * jac);
};
return Variable(result, {input}, grad_func);
}

Variable transpose(const Variable &input)
{
auto result = transpose(input.array());
Expand Down Expand Up @@ -281,7 +328,7 @@ namespace af {

Variable tile(const Variable &input, const std::vector<int> &repeats)
{
dim4 dims;
dim4 dims(0);
for (size_t i = 0; i < repeats.size(); i++) {
dims[i] = repeats[i];
}
Expand Down
Loading