PyTorch笔记 - Position Embedding (Transformer/ViT/Swin/MAE)_position embding

欢迎关注我的CSDN：https://blog.csdn.net/caroline_wendy
本文地址：https://blog.csdn.net/caroline_wendy/article/details/128447794

Position Embedding(位置编码)

• Transformer
  • 1d absolute
  • sin/cos constant
• Vision Transformer
  • 1d absolute
  • trainable
• Swin Transformer
  • 2d relative bias
  • trainable
• Masked AutoEncoder
  • 2d absolute
  • sin/cos constant

Paper:

• Transformer - Attention Is All You Need
• ViT - An Image is Worth 16x16 Words Transformers for Image Recognition at Scale
• SwinTransformer - Hierarchical Vision Transformer using Shifte