Here're several fully-detailed notes compiled by myself about deep learning, covering topics from basic deep learning concepts to cutting-edge deep learning techniques.
Deep learning is a quickly growing field and any static note will soon be out-of-date. So I put these notes on Github and keep updating them. You can discuss with me about these details or directly contribute what you know!
To see the math formulas correctly, install this chrome plugin first: github-mathjax.
- Deep Learning Basics. This chapter introduces the motivation and basic concepts of deep learning.
- Loss Functions. This chapter introduces the loss functions used in deep learning, from basic classfication/regression loss to some fancy losses designed for ranking, rating prediction, object detection, etc.
- Activation Functions. This chapter introduces the activation functions used in deep learning, thier variaints and comparisons.
- Optimization Techniques. This chapter first talks about the details of the basic gradient descent, and then discuss some key aspects of deep learning model training, including parameter initialization, optimization algorithms, learning rate scheduling, normalization and regularization.
- Model Architectures. This is a collection of "design patterns" for deep learning, especially for natural language processing. It covers the design of projection layers, embedding layers, sequence encoding layers, sequence aggregation layers, the alignment of two sequences and so on. All these architectures are paired with the reference to the original paper or the open-source code. You can treat this as an (incomplete) roadmap to the existing building blocks for deep learning models.