Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create some rule to initialize models #5

Open
muammar opened this issue Jan 16, 2019 · 2 comments
Open

Create some rule to initialize models #5

muammar opened this issue Jan 16, 2019 · 2 comments

Comments

@muammar
Copy link
Owner

muammar commented Jan 16, 2019

The biases are not correctly initialized. I should create some kind of trick to have them starting to values that make sense for the problem in question.
muammar added a commit that referenced this issue Jan 17, 2019
@muammar
Working on bias initialization
e3d9a1e 
This is related to issue #5. If biases are not initialized using some
information from the data set, then the neural network's loss function
starts with a very large value. That translate in more epochs needed to
optimize the models.
@muammar muammar changed the title Create some rule to initialize biases Create some rule to initialize models Jan 17, 2019
@muammar
Copy link
Owner Author

muammar commented Jan 22, 2019

Fixed in 9bddd07.

@muammar muammar closed this as completed Jan 22, 2019
@muammar
Copy link
Owner Author

muammar commented Jan 26, 2019

I am reopening this issue after reading this article in medium:

https://medium.com/usf-msds/deep-learning-best-practices-1-weight-initialization-14e5c0295b94

Depending on the activation function used the weights have to be initialized differently.

@muammar muammar reopened this Jan 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@muammar