.. _ENOT Tutorials: ######### Tutorials ######### .. _E1 ref: `1. Tutorial - getting started`_ ================================ .. _1. Tutorial - getting started: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/1.%20Tutorial%20-%20getting%20started.ipynb Describes the basic steps you need to optimize an architecture: - Create your model and move it into SearchSpace; - Pretrain search space on target task (Imagenette image classification in this example); - Search the best architecture for your task in search space; - Tune this model. .. _E2 ref: `2. Tutorial - search space autogeneration`_ ============================================ .. _2. Tutorial - search space autogeneration: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/2.%20Tutorial%20-%20search%20space%20autogeneration.ipynb This notebook describes how to automatically generate search space from your model. .. _E3 ref: `3. Tutorial - custom model`_ ============================= .. _3. Tutorial - custom model: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/3.%20Tutorial%20-%20custom%20model.ipynb This notebook describes the ways to implement your own model. Typical use cases: - Creating models which can not be built by "block models builders" from ENOT framework; - You already have your model, and you don't want to rewrite your code. .. _E4 ref: `4. Tutorial - using latency optimization`_ =========================================== .. _4. Tutorial - using latency optimization: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/4.%20Tutorial%20-%20using%20latency%20optimization.ipynb This notebook describes the additional steps required to enable latency optimization for custom models. .. _E5 ref: `5. Tutorial - latency calculation`_ ==================================== .. _5. Tutorial - latency calculation: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/5.%20Tutorial%20-%20latency%20calculation.ipynb This notebook describes how to calculate latency using ENOT framework. Main chapters of this notebook: - Initialize latency of search space (SearchSpaceModel); - Calculate latency of an arbitrary model/module. .. _E6 ref: `6. Tutorial - search with the specified latency`_ ================================================== .. _6. Tutorial - search with the specified latency: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/6.%20Tutorial%20-%20search%20with%20the%20specified%20latency.ipynb This notebook describes how to search only architectures with latency strictly lower than the specified value. .. _E7 ref: `7. Tutorial - resolution search for image classification`_ =========================================================== .. _7. Tutorial - resolution search for image classification: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/7.%20Tutorial%20-%20resolution%20search%20for%20image%20classification.ipynb This notebook shows how to search optimal image resolution for computer vision task. In this tutorial, we search model and resolution with the best accuracy and fixed maximum latency. .. _E8 ref: `8. Tutorial - search space autogeneration (EfficientNet-V2 S)`_ ================================================================ .. _8. Tutorial - search space autogeneration (EfficientNet-V2 S): https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/8.%20Tutorial%20-%20search%20space%20autogeneration%20(EfficientNet-V2%20S).ipynb This notebook describes how to generate search space automatically from EfficientNet-V2 Small model. This example uses `pytorch-image-models`_ EfficientNet-V2 implementation. .. _pytorch-image-models: https://github.com/rwightman/pytorch-image-models .. _E9 ref: `9. Tutorial - metric learning`_ ================================ .. _9. Tutorial - metric learning: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/9.%20Tutorial%20-%20metric%20learning.ipynb This notebook shows an advanced example usage of ENOT framework. We will show how to use angular margin loss for metric learning task. .. _E10 ref: `10. Tutorial - adding custom operations for model builder`_ ============================================================ .. _10. Tutorial - adding custom operations for model builder: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/10.%20Tutorial%20-%20adding%20custom%20operations%20for%20model%20builder.ipynb This notebook describes the ways to implement your own operations, and how to use them with the model builder. .. _E11 ref: `11. Experimental - Tutorial - automatic quantization for enot-lite`_ ===================================================================== .. _11. Experimental - Tutorial - automatic quantization for enot-lite: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/11.%20Experimental%20-%20Tutorial%20-%20automatic%20quantization%20for%20enot-lite.ipynb This experimental notebook shows how to apply enot-autodl framework for automatic quantization to create quantized model for enot-lite framework. .. _E12 ref: `12. Experimental - Tutorial - automatic pruning`_ ================================================== .. _12. Experimental - Tutorial - automatic pruning: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/12.%20Experimental%20-%20Tutorial%20-%20automatic%20pruning.ipynb This experimental notebook shows how to apply enot-autodl framework for automatic network pruning and fine-tuning. .. _E13 ref: `13. Experimental - Tutorial - automatic pruning (manual)`_ =========================================================== .. _13. Experimental - Tutorial - automatic pruning (manual): https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/13.%20Experimental%20-%20Tutorial%20-%20automatic%20pruning%20(manual).ipynb This experimental notebook shows how to apply enot-autodl framework for automatic network pruning and fine-tuning. Gradient accumulation is performed manually. .. _Distributed (multi-gpu / multi-node) pretrain example reference: `Distributed (multi-gpu / multi-node) pretrain example`_ ======================================================== .. _Distributed (multi-gpu / multi-node) pretrain example: https://github.com/ENOT-AutoDL/ENOT_Tutorials/tree/v2.9.0/multigpu_pretrain .. _.sh script: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/multigpu_pretrain/run_multigpu_pretrain.sh .. _.py script: https://github.com/ENOT-AutoDL/ENOT_Tutorials/blob/v2.9.0/multigpu_pretrain/run_multigpu_pretrain.sh In this folder you can find `.sh script`_ for running multi-gpu pretrain. You can change it's configuration to run on single GPU, on multiple GPU within a single node, or on multiple compute nodes with multiple GPUs. The second file in this folder is a `.py script`_ which is launched by .sh script. This script uses functions from other tutorials, but is adapted to run in distributed manner. This script should be viewed as a reference point for user-defined distributed pretrain scripts. Distributed search is not recommended as it is under development. Moreover, the search procedure is usually relatively fast. At the tuning stage, you will have a regular model without any of the ENOT specifics, so it is your responsibility to write correct distributed code (probably by wrapping found model with `DistributedDataParallel module `_).