[NeoML] CDnnHeadAdapterLayer

[daniyalaliev]

No description provided.

[favorart]

What if the first adapter will be removed from the dnn?

[favorart]

The special test should be where the CDnnHeadAdapterLayer is inside the CRecursiveLayer.

[favorart]

Please, add one more test with the exact scenario suggested by the user of this feature

    IMathEngine& mathEngine = MathEngine();
    CRandom random( 42 );
    CDnn dnn( random, mathEngine );

    // To be able run the same set of layers 3 times
    // on 3 different inputs to get 3 different outputs links
    CPtr<CDnnHead> head = new CDnnHead<
        CFullyConnectedLayer,
        CGELULayer,
        CFullyConnectedLayer,
        CReLULayer,
        CDropoutLayer,
        CFullyConnectedLayer
    >(
        random, mathEngine,
        FullyConnected( 128 ),
        Gelu(),
        FullyConnected( 64 ),
        Relu(),
        Dropout( 0.5f ),
        FullyConnected( 1 )
    );

    CBaseLayer* x = Source( dnn, "srcX" );
    x = FullyConnected( 512 )( x );
    x = Gelu()( x );
    x = DnnHeadAdapter( head )( x ); // #1 head dnn call

    CBaseLayer* y = Source( dnn, "srcY" );
    y = FullyConnected( 512 )( y );
    y = Gelu()( y );
    y = DnnHeadAdapter( head )( y ); // #2 head dnn call

    CBaseLayer* z = Source( dnn, "srcZ" );
    z = FullyConnected( 512 )( z );
    z = Gelu()( z );
    z = DnnHeadAdapter( head )( z ); // #3 head dnn call

    CBaseLayer* out = ConcatChannels()( x, y, z );

    CBaseLayer* labels = Source( dnn, "labels" );
    BinaryCrossEntropyLoss()( out, labels );

[favorart]

If there is a only one adapter for this exact sub-network, this adapter should be removed and sub-net reconnected directly, as it works for a composite in the optimization::UnpackComposites.