Skip to content

YNNEKUW/KrabbyPatty_Pytorch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

97b2ff2 Β· Feb 4, 2021

History

20 Commits
Feb 3, 2021
Jan 28, 2021
Jan 24, 2021
Feb 4, 2021
Feb 4, 2021

Repository files navigation

KrabbyPatty_Pytorch πŸ”

Introduction

ICLR 2021 paper Is Attention Better Than Matrix Decomposition? Pytorch implementation. I haved tested this on IWSLT for the correctness and the efficacy. The hamburger-pytorch is not correct.

Usage

import torch
from krabbypatty_pytorch import KrabbyPatty

x = torch.randn(42,64,512)  # [Sequence Length, Batch Size, Hidden Dimension]
krabbypatty = KrabbyPatty(input_dim=512)
output = krabbypatty(x) + x

Citations

@inproceedings{
    title={Is Attention Better Than Matrix Decomposition?},
    author={Geng, Zhengyang and Guo, Meng-Hao and Chen, Hongxu and Li, Xia and Wei, Ke and Lin Zhouchen}
    year={2021},
    url={https://openreview.net/forum?id=1FvkSpWosOl},
    note={ICLR 2021}
}

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages