Main Content

patchEmbeddingLayer

Patch embedding layer

Since R2023b

expand all in page

    Description

    A patch embedding layer maps patches of pixels to vectors. Use this layer in vision transformer neural networks to encode information about patches in images.

    Creation

    Syntax

    layer = patchEmbeddingLayer(patchSize,outputSize)
    layer = patchEmbeddingLayer(patchSize,outputSize,Name=Value)

    Description

    example

    layer = patchEmbeddingLayer(patchSize,outputSize) creates a patch embedding layer and sets the PatchSize and OutputSize properties.

    This feature requires a Deep Learning Toolbox™ license.

    example

    layer = patchEmbeddingLayer(patchSize,outputSize,Name=Value) sets additional properties using one or more name-value arguments.

    Properties

    expand all

    Patch Embedding

    This property is read-only.

    Size of patches to split input images into, specified as a positive integer or row vector of positive integers.

    If PatchSize is a vector, then each element of PatchSize is the size of the patch in the corresponding spatial dimension of the input. If PatchSize is a scalar, then the layer uses the same value for all spatial dimensions of the input.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Mode for flattening the output of the convolution operation, specified as "column-major" or "row-major".

    If SpatialFlattenMode is "column-major", then the flatten operation outputs the data in its column-major representation. For example, consider the input:

    A = [
   1 2 3
   4 5 6
   7 8 9];
    The output in column-major representation is:
    AFlat = [1 4 7 2 5 8 3 6 9];

    If SpatialFlattenMode is "row-major", then the flatten operation outputs the data in its row-major representation. For example, consider the input:

    A = [
   1 2 3
   4 5 6
   7 8 9];
    The output in row-major representation is:
    AFlat = [1 2 3 4 5 6 7 8 9];

    Set this option when creating or importing models that require this representation.

    This property is read-only.

    Size of output vectors, specified as a positive integer.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    This property is read-only.

    Number of input channels, specified as one of these values:

    • "auto" — Automatically determine the number of input channels at training time.

    • Positive integer — Configure the layer for the specified number of input channels. InputSize and the number of channels in the layer input data must match. For example, if the input is an RGB image, then InputSize must be 3.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string

    Parameters and Initialization

    Function to initialize the weights, specified as one of these values:

    • "glorot" — Initialize the weights with the Glorot initializer [1] [2] (also known as Xavier initializer). The Glorot initializer independently samples from a uniform distribution with zero mean and a variance of 2/(numIn + numOut), where numIn and numOut are the values of the InputSize and OutputSize properties, respectively.

    • "he" – Initialize the weights with the He initializer [3]. The He initializer samples from a normal distribution with zero mean and a variance of 2/numIn, where numIn is the value of the InputSize property.

    • "narrow-normal" — Initialize the weights by independently sampling from a normal distribution with zero mean and a standard deviation of 0.01.

    • "zeros" — Initialize the weights with zeros.

    • "ones" — Initialize the weights with ones.

    • Function handle — Initialize the weights with a custom function. If you specify a function handle, then the function must have the form weights = func(sz), where sz is the size of the weights.

    The layer initializes the weights only when the Weights property is empty.

    Data Types: char | string | function_handle

    Function to initialize the biases, specified as one of these values:

    • "zeros" — Initialize the biases with zeros.

    • "ones" — Initialize the biases with ones.

    • "narrow-normal" — Initialize the biases by independently sampling from a normal distribution with a mean of zero and a standard deviation of 0.01.

    • Function handle — Initialize the biases with a custom function. If you specify a function handle, then the function must have the form bias = func(sz), where sz is the size of the biases.

    The layer initializes the biases only when the Bias property is empty.

    Data Types: char | string | function_handle

    Learnable weights.

    If PatchSize is a positive integer, then Weights is an PatchSize-by-...-by-PatchSize-by-InputSize-by-OutputSize numeric array or [], where the number of dimensions of size PatchSize is the number of spatial dimensions of the input.

    If PatchSize is a vector, then Weights is an PatchSize(1)-by-...-by-PatchSize(K)-by-InputSize-by-OutputSize numeric array or [], where K is the number of spatial dimensions of the input.

    The layer weights are learnable parameters. You can specify the initial value of the weights directly using the Weights property of the layer. When you train a network, if the Weights property of the layer is nonempty, then the trainnet (Deep Learning Toolbox) and trainNetwork (Deep Learning Toolbox) functions use the Weights property as the initial value. If the Weights property is empty, then the software uses the initializer specified by the WeightsInitializer property of the layer.

    Data Types: single | double

    Layer biases, specified as a numeric column vector of length OutputSize or [].

    The layer biases are learnable parameters. When you train a neural network, if Bias is nonempty, then the trainnet (Deep Learning Toolbox) and trainNetwork (Deep Learning Toolbox) functions use the Bias property as the initial value. If Bias is empty, then software uses the initializer specified by BiasInitializer.

    Data Types: single | double

    Learning Rate and Regularization

    Learning rate factor for the weights, specified as a nonnegative scalar.

    The software multiplies this factor by the global learning rate to determine the learning rate for the weights in this layer. For example, if WeightLearnRateFactor is 2, then the learning rate for the weights in this layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using the trainingOptions (Deep Learning Toolbox) function.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Learning rate factor for the biases, specified as a nonnegative scalar.

    The software multiplies this factor by the global learning rate to determine the learning rate for the biases in this layer. For example, if BiasLearnRateFactor is 2, then the learning rate for the biases in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using the trainingOptions (Deep Learning Toolbox) function.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    L2 regularization factor for the weights, specified as a nonnegative scalar.

    The software multiplies this factor by the global L2 regularization factor to determine the L2 regularization for the weights in this layer. For example, if WeightL2Factor is 2, then the L2 regularization for the weights in this layer is twice the global L2 regularization factor. You can specify the global L2 regularization factor using the trainingOptions (Deep Learning Toolbox) function.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    L2 regularization factor for the biases, specified as a nonnegative scalar.

    The software multiplies this factor by the global L2 regularization factor to determine the L2 regularization for the biases in this layer. For example, if BiasL2Factor is 2, then the L2 regularization for the biases in this layer is twice the global L2 regularization factor. The software determines the global L2 regularization factor based on the settings you specify using the trainingOptions (Deep Learning Toolbox) function.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Layer

    Layer name, specified as a character vector or a string scalar. For Layer array input, the trainnet (Deep Learning Toolbox) and dlnetwork (Deep Learning Toolbox) functions automatically assign names to layers with the name "".

    The PatchEmbeddingLayer object stores this property as a character vector.

    Data Types: char | string

    This property is read-only.

    Number of inputs to the layer, returned as 1. This layer accepts a single input only.

    Data Types: double

    This property is read-only.

    Input names, returned as {'in'}. This layer accepts a single input only.

    Data Types: cell

    This property is read-only.

    Number of outputs from the layer, returned as 1. This layer has a single output only.

    Data Types: double

    This property is read-only.

    Output names, returned as {'out'}. This layer has a single output only.

    Data Types: cell

    Examples

    collapse all

    This example uses:

    Open Live Script

    Create a patch embedding layer that embeds patches of size 16 with an output size of 768.

    patchSize = 16;
embeddingOutputSize = 768;
layer = patchEmbeddingLayer(patchSize,embeddingOutputSize)
    layer = 
  PatchEmbeddingLayer with properties:

                     Name: ''
                PatchSize: 16
                InputSize: 'auto'
               OutputSize: 768
       SpatialFlattenMode: 'column-major'
       WeightsInitializer: 'glorot'
          BiasInitializer: 'zeros'
    WeightLearnRateFactor: 1
      BiasLearnRateFactor: 1
           WeightL2Factor: 1
             BiasL2Factor: 1

   Learnable Parameters
                  Weights: []
                     Bias: []

   State Parameters
    No properties.

Use properties method to see a list of all properties.

    Create a dlnetwork object.

    net = dlnetwork;

    Specify layers of the network, including a patch embedding layer.

    inputSize = [384 384 3];

maxPosition = (inputSize(1)/patchSize)^2 + 1;

numHeads = 4;
numKeyChannels = 4*embeddingOutputSize;

numClasses = 1000;

layers = [ 
    imageInputLayer(inputSize)
    patchEmbeddingLayer(patchSize,embeddingOutputSize,Name="patch-emb")
    embeddingConcatenationLayer(Name="emb-cat")
    positionEmbeddingLayer(embeddingOutputSize,maxPosition,Name="pos-emb");
    additionLayer(2,Name="add")
    selfAttentionLayer(numHeads,numKeyChannels,AttentionMask="causal")
    indexing1dLayer(Name="idx-first")
    fullyConnectedLayer(numClasses)
    softmaxLayer];
net = addLayers(net,layers);

    Connect the embedding concatenation layer with the "in2" input of the addition layer.

    net = connectLayers(net,"emb-cat","add/in2");

    View the neural network architecture.

    plot(net)

    Algorithms

    expand all

    References

    [1] Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani et al. "An Image is Worth 16x16 words: Transformers for Image Recognition at Scale." Preprint, submitted June 3, 2021. https://doi.org/10.48550/arXiv.2010.11929.

    [2] Glorot, Xavier, and Yoshua Bengio. "Understanding the Difficulty of Training Deep Feedforward Neural Networks." In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 249–356. Sardinia, Italy: AISTATS, 2010. https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

    [3] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." In 2015 IEEE International Conference on Computer Vision (ICCV), 1026–34. Santiago, Chile: IEEE, 2015. https://doi.org/10.1109/ICCV.2015.123

    Extended Capabilities

    Version History

    Introduced in R2023b

    expand all

    See Also

    | (Deep Learning Toolbox) | (Deep Learning Toolbox) | (Deep Learning Toolbox) | (Deep Learning Toolbox) | (Deep Learning Toolbox) | (Deep Learning Toolbox)

    Topics