How to update Embedding vector in keras with custom operator

TensorFlow General Discussion
, ,
1

Suppose I have two features f1, f2 with corresponding embedding vector e1,e2 built by keras Embeding layer, how could I update each vector with some custom operation instead of only using gradient?
For example , I would like to update e1 as follows:
e1 = 0.9 * (e1 + learning_rate * d e1) + 0.1 * e2

3

Hi @Bean_Dog, Once you get the embedding vector for the features f1, f2. you can update those embedding by passing those embedding vectors to the custom operation defined.

For example,

import tensorflow as tf

text = "This is a first sample text."
text2= 'This is a second sample text'

tokenizer = tf.keras.preprocessing.text.Tokenizer()

tokenizer.fit_on_texts([text])
tokenizer.fit_on_texts([text2])

f1 = tokenizer.texts_to_sequences([text])
f2 = tokenizer.texts_to_sequences([text2])

print('tokenized text for f1:',f1)
print('\n')
print('tokenized text for f2:',f2)
print('\n')


embedding_layer = tf.keras.layers.Embedding(input_dim=8,output_dim=10)

e1 = embedding_layer(tf.constant(f1,dtype=tf.int32))
e2 = embedding_layer(tf.constant(f2,dtype=tf.int32))

print('before customization e1:',e1)
print('\n')
print('before customization e2:',e2)
print('\n')

learning_rate = 0.1
with tf.GradientTape() as tape:
    tape.watch(e1)
    gradients = tape.gradient(e1, e1)
    
def custom_operation(e1, e2):
  return 0.9 * (e1 + learning_rate * gradients) + 0.1 * e2

e1 = custom_operation(e1, e2)
e2 = custom_operation(e2, e1)

print('after customization e1:',e1)
print('\n')
print('after customization e2:',e2)

please refer to this gist for working code example. Thank You.