🧬TRANSFORMER Course materials

Выложили с коллегами материалы продвинутого Сколтеховского курса по трансформерам.

Вы репозитории вы найдете материалы 15 лекций и практических занятий к ним, которые покрывают основные темы по трансформерам для работы с текстами и применение трансформеров в других областях.

Warning: курс на английском языке!

🔻Программа:
1. The Transformer: motivation, original architecture, and attention mechanism.
2. Transformer-based Encoders. Masked language models based on the Transformer architecture. BERT and related models.
3. Classification and sequence tagging with Transformers. Using encoders to generate feature representation for various NLU tasks.
4. Transformer-based Decoders. Generation of text using Transformers. GPT and related decoders.
5. Prompt and instruction tuning. Reinforcement Learning from Human Feedback (RLHF), ChatGPT, and related models.
6. Sequence to sequence tasks: machine translation, text detoxification, question answering, dialogue. Technical tricks for training and inference.
7. Multilingual language models based on the Transformer architecture.
8. Uncertainty estimation for Transformers and NLP.
9. Efficient Transformers.
10. Compression of Transformer models and low-rank approaches.
11. Network encoders with Transformers
12. Multimodal and Vision Transformers.
13. Transformers for event sequences.
14. Transformers for tabular data.
15. Final lecture.

@mashkka_ds

#nlp #transformers