17. Embeddings

Jump to bottom

Antonio Erdeljac edited this page Apr 22, 2019 · 1 revision

Embeddings

Topic: Embeddings

Course: GMLC

Date: 16 April 2019

Professor: Not specified

Resources

Key Points

Collaborative filtering
- Task of making predictions for an user based on predictions of other users
Categorical data
- refers to input features that can represent one or more items from a finite number of choices
Embeddings
- used to translate large sparse-vectors into low dimensional space while preserving semantic relationships
Sparse input data problems can be solved by translating into lower dimensional space
Shrinking the network
- The aim is to have enough space for rich semantic relations, but also not too much that it slows down the mode training
Embeddings as lookup tables
- To receive dense vectors (multiple items) from a matrix, we retrieve individual embedding then add them together
Embedding lookup as matrix multiplication
- equivalent to matrix multiplication
Obtaining embeddings
- Principal component analysis (PCA)
  - Used for word embeddings
  - Finds highly correlated dimensions that can be collapsed into a single dimension
- Word2Vec
  - Algorithm invented by Google for training word embeddings
  - Uses distributional hypothesis to map semantically similar word to geometrically close embedding vectors
Training an embedding as part of a model
- Can be used as a layer in neural networks

Check your understanding

What are embeddings useful for?
What are ways of obtaining embeddings?

Summary of Notes

Embeddings are used to translate large sparse-vectors to a lower dimension while preserving their semantic relationships
Embeddings can be trained in a neural network as an embedding layer