Hi,

How can I implement this architecture in image bellow.

I really appreciate your help;

Thank you

Hi @youb, Once you have defined your 2 models. For example

```
import tensorflow as tf
vgg_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
resnet = tf.keras.applications.ResNet50(include_top=False, weights='imagenet', input_shape=(224, 224, 3))
model1=tf.keras.Sequential([
vgg_model,
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(1024, activation='relu'),
tf.keras.layers.Dense(512, activation='relu')
])
model2=tf.keras.Sequential([
resnet,
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation='relu')
])
```

Then to do average of the last layers of 2 models and adding the remaining layers, you can do it by

```
merged_model=tf.keras.layers.Average()([model1.output, model2.output])
merged_model=tf.keras.layers.Dense(512,activation='relu')(merged_model)
merged_model=tf.keras.layers.Dropout(0.5)(merged_model)
merged_model=tf.keras.layers.Dense(3,activation='relu')(merged_model)
model = tf.keras.models.Model(inputs=[model1.output, model2.output], outputs=merged_model)
```

Please refer to this gist for working code example. Thank You.

1 Like

Hi @Kiran_Sai_Ramineni

thank you for code;

please how to train this model; is I should train each model alone, I mean model1, then model2 and after merged_model;

I have tried to compile and train the merged_model; I got an error

Thank you

I have tried this but I’m not sure:

import tensorflow as tf

from tensorflow.keras.layers import Dense, Dropout, Input, GlobalAveragePooling2D, Average, TimeDistributed, Average, Flatten

from tensorflow.keras.models import Model

from tensorflow.keras.applications.resnet50 import ResNet50

import tensorflow_hub as hub

resnet = ResNet50(weights=“imagenet”, include_top=False, input_tensor=Input(shape=(224, 224, 3)))

resnet.trainable = False

vit16 = hub.KerasLayer(“Sayak | vision_transformer | Kaggle”, trainable=False)

inputs = Input(shape=(10, 224, 224, 3))

resnet_features = TimeDistributed(resnet)(inputs)

resnet_features = TimeDistributed(GlobalAveragePooling2D())(resnet_features)

resnet_features = Flatten()(resnet_features)

resnet_features = Dense(1024)(resnet_features)

resnet_features = Dense(512)(resnet_features)

vit_features = TimeDistributed(vit16)(inputs)

#vit_features = TimeDistributed(tf.keras.layers.LayerNormalization())(vit_features)

vit_features = Flatten()(vit_features)

vit_features = Dense(512)(vit_features)

averaged_features = Average()([resnet_features, vit_features])

x = Dense(512, activation=‘relu’)(averaged_features)

x = Dropout(0.5)(x)

outputs = Dense(5, activation=‘sigmoid’)(x)

model = Model(inputs=inputs, outputs=outputs)

model.compile(optimizer=‘adam’, loss=‘mse’, metrics=[‘mae’])

model.summary()

Hi @youb, first you have to train the vgg16 and resnet50 model then you have to pass the last layers outputs of both models to the merged model for training. You are getting the error because you are passing data to the merged model with shape (224,224,3) but the merged model input shape will be last layer output of vgg and resnet models which is (None,512). Thank You,

1 Like

@Kiran_Sai_Ramineni

Thank you so much

Remember to freeze everything but the last Dense(3) and Dense(512) when you train the combined model otherwise you will lose any pretraining of the two models and the stretch after them. You can unfreeze and fine-tune once the last two dense layers have converged.

Yes I did; freeze all pretrained layers.

Thank you so much.