########################################
Autogeneration transformation parameters
########################################

**Short summary:** *To automatically generate a new model with search variant
containers, you should use*
:class:`enot.autogeneration.TransformationParameters` *class. There are two
available autogeneration properties to the date:*

1. *Hidden width multiplier,* ``width_mult``
2. *Convolutional layer kernel size,* ``kernel_size``
3. *Bottlenecks hidden depth multiplier,* ``depth_mult``

*To use both options with our autogeneration mechanism, simply do the
following:*

.. code-block:: python

    import torch.nn as nn
    from enot.autogeneration import generate_pruned_search_variants_model
    from enot.autogeneration import TransformationParameters

    ...

    your_model: nn.Module = ...

    generated_model = generate_pruned_search_variants_model(
        your_model,
        search_variant_descriptors=(
            TransformationParameters(width_mult=1.0, kernel_size=3),
            TransformationParameters(width_mult=1.0, kernel_size=5),
            TransformationParameters(width_mult=0.5, kernel_size=3),
            TransformationParameters(width_mult=0.5, kernel_size=5),
            TransformationParameters(width_mult=0.0),
        ),
    )

    # move model with search variants to the search space
    search_space = SearchSpaceModel(generated_model).cuda()

    # pretrain, search, extract sub-model, ...
    ...

To simplify search space creation, we provide a tool to automatically generate
new model with search variants from the user-defined model, see more in
:doc:`autogeneration`.

To define which search variants will be inserted into user-defined model, you
should create a tuple with
:class:`enot.autogeneration.TransformationParameters` instances and pass it to
the autogeneration function. :class:`.TransformationParameters` instance
specifies single transformation configuration for matched patterns in the user
model. Each transformation will be applied once to each matched pattern, and
transformed sub-model will be inserted into :class:`.SearchVariantsContainer`.

This procedure ensures that each :class:`.SearchVariantsContainer` instance in
the model will have the same number of choice options.

:class:`.TransformationParameters` class supports two auto-generated properties
for the time being:

1. *Hidden width multiplier,* ``width_mult``
2. *Convolutional layer kernel size,* ``kernel_size``
3. *Bottlenecks hidden depth multiplier,* ``depth_mult``

.. autoclass:: enot.autogeneration.TransformationParameters
    :members:

Let's see the most basic example:

.. code-block:: python

    search_variant_descriptors = (
        TransformationParameters(width_mult=1.00),
        TransformationParameters(width_mult=0.75),
        TransformationParameters(width_mult=0.50),
        TransformationParameters(width_mult=0.25),
    )

For example, if your model contains
`ResNet <https://arxiv.org/abs/1512.03385>`_ blocks - they will be
replaced with :class:`.SearchVariantsContainer` instances, each with 4
operations which were created by reducing hidden (inner) channel number in the
replaced block. Channel counts will be reduced with factors 1.0, 0.75, 0.5,
0.25 (1.0 factor means no channel count reduction). Weights in search variants
are initialized from the weights from the original block.

The same applies for `EfficientNet <https://arxiv.org/abs/1905.11946>`_ blocks,
`MobileNetV2 <https://arxiv.org/abs/1801.04381>`_ blocks, and others.

Let's look at another interesting case:

.. code-block:: python

    search_variant_descriptors = (
        TransformationParameters(width_mult=1.0),
        TransformationParameters(width_mult=0.5),
        TransformationParameters(width_mult=0.0),
    )

The main difference here from the previous example is that we have zero width
multiplier, which replaces original layer with search variant with single
`Conv1x1 <https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html>`_ of
appropriate stride and channel sizes, followed by
`BatchNorm2d <https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html>`_
layer.

:green:`Our default autogeneration parameters are set as following:`

.. code-block:: python

    DEFAULT_SPEED_UP_SEARCH_SPACE = (
        TransformationParameters(width_mult=1.00),
        TransformationParameters(width_mult=0.75),
        TransformationParameters(width_mult=0.50),
        TransformationParameters(width_mult=0.25),
        TransformationParameters(width_mult=0.00),
    )

We can also search for kernel size in convolutional layer:

.. code-block:: python

    search_variant_descriptors = (
        TransformationParameters(kernel_size=3),
        TransformationParameters(kernel_size=5),
        TransformationParameters(kernel_size=7),
    )

In known patterns, we replace convolutions with kernels larger than 1 with the
convolution, which kernel is equal to the ``kernel_size`` argument.

We can also combine these transformations together to search patterns with
different inner channel counts and kernel sizes:

.. code-block:: python

    search_variant_descriptors = (
        TransformationParameters(width_mult=1.0, kernel_size=3),
        TransformationParameters(width_mult=1.0, kernel_size=5),
        TransformationParameters(width_mult=1.0, kernel_size=7),
        TransformationParameters(width_mult=0.5, kernel_size=3),
        TransformationParameters(width_mult=0.5, kernel_size=5),
        TransformationParameters(width_mult=0.5, kernel_size=7),
        TransformationParameters(width_mult=0.0),
    )

This is the default configuration to search MobileNetV2-like nets, as it allows
to search different kernel sizes (depthwise convolutions with large kernels are
quite fast on CPUs), different expansion ratios (inner MobileNetV2 block
channel multiplier), and even "skip" unnecessary operations (by replacing them
with cheap 1x1 convolution).

As we use Conv1x1 when ``width_mult`` is equal to 0, ``kernel_size`` argument
becomes redundant, and should be set to None.

Another one interesting example for `YoloV5-like models <https://github.com/ultralytics/yolov5>`_.
For this type of models you can not choose kernel size, but you can choose number
of bottlenecks in Yolo C3 block('depth') with ``depth_mult`` argument, by default ``depth_mult`` equals 1.0
keeps number of blocks in the created model unchanged (like in original model).

.. code-block:: python

    search_variant_descriptors = (
        TransformationParameters(width_mult=1.0, depth_mult=1.0),
        TransformationParameters(width_mult=1.0, depth_mult=0.5),
        TransformationParameters(width_mult=1.0, depth_mult=0.33),
        TransformationParameters(width_mult=0.5, depth_mult=1.0),
        TransformationParameters(width_mult=0.5, depth_mult=0.5),
        TransformationParameters(width_mult=0.5, depth_mult=0.66),
    )