Hi,
How can I implement this architecture in image bellow.
I really appreciate your help;
Thank you
Hi @youb, Once you have defined your 2 models. For example
import tensorflow as tf
vgg_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
resnet = tf.keras.applications.ResNet50(include_top=False, weights='imagenet', input_shape=(224, 224, 3))
model1=tf.keras.Sequential([
vgg_model,
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(1024, activation='relu'),
tf.keras.layers.Dense(512, activation='relu')
])
model2=tf.keras.Sequential([
resnet,
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation='relu')
])
Then to do average of the last layers of 2 models and adding the remaining layers, you can do it by
merged_model=tf.keras.layers.Average()([model1.output, model2.output])
merged_model=tf.keras.layers.Dense(512,activation='relu')(merged_model)
merged_model=tf.keras.layers.Dropout(0.5)(merged_model)
merged_model=tf.keras.layers.Dense(3,activation='relu')(merged_model)
model = tf.keras.models.Model(inputs=[model1.output, model2.output], outputs=merged_model)
Please refer to this gist for working code example. Thank You.
1 Like
Hi @Kiran_Sai_Ramineni
thank you for code;
please how to train this model; is I should train each model alone, I mean model1, then model2 and after merged_model;
I have tried to compile and train the merged_model; I got an error
Thank you
I have tried this but I’m not sure:
import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout, Input, GlobalAveragePooling2D, Average, TimeDistributed, Average, Flatten
from tensorflow.keras.models import Model
from tensorflow.keras.applications.resnet50 import ResNet50
import tensorflow_hub as hub
Load the ResNet50 model with pre-trained ImageNet weights
resnet = ResNet50(weights=“imagenet”, include_top=False, input_tensor=Input(shape=(224, 224, 3)))
resnet.trainable = False
Load the ViT-B16 model
vit16 = hub.KerasLayer(“Sayak | vision_transformer | Kaggle”, trainable=False)
Define the inputs
inputs = Input(shape=(10, 224, 224, 3))
Extract features from ResNet50 and ViT-B16 models
resnet_features = TimeDistributed(resnet)(inputs)
resnet_features = TimeDistributed(GlobalAveragePooling2D())(resnet_features)
resnet_features = Flatten()(resnet_features)
resnet_features = Dense(1024)(resnet_features)
resnet_features = Dense(512)(resnet_features)
vit_features = TimeDistributed(vit16)(inputs)
#vit_features = TimeDistributed(tf.keras.layers.LayerNormalization())(vit_features)
vit_features = Flatten()(vit_features)
vit_features = Dense(512)(vit_features)
Take the average of the features
averaged_features = Average()([resnet_features, vit_features])
Add Dense layer with Dropout
x = Dense(512, activation=‘relu’)(averaged_features)
x = Dropout(0.5)(x)
Add final Dense layer with sigmoid activation function
outputs = Dense(5, activation=‘sigmoid’)(x)
Define the model
model = Model(inputs=inputs, outputs=outputs)
Compile the model
model.compile(optimizer=‘adam’, loss=‘mse’, metrics=[‘mae’])
model.summary()
Hi @youb, first you have to train the vgg16 and resnet50 model then you have to pass the last layers outputs of both models to the merged model for training. You are getting the error because you are passing data to the merged model with shape (224,224,3) but the merged model input shape will be last layer output of vgg and resnet models which is (None,512). Thank You,
1 Like
@Kiran_Sai_Ramineni
Thank you so much
Remember to freeze everything but the last Dense(3) and Dense(512) when you train the combined model otherwise you will lose any pretraining of the two models and the stretch after them. You can unfreeze and fine-tune once the last two dense layers have converged.
Yes I did; freeze all pretrained layers.
Thank you so much.