# Masking with Attention layer

**URL:** <https://discuss.ai.google.dev/t/masking-with-attention-layer/30576>\
**Category:** Keras\
**Tags:** keras, api, transformers\
**Created:** [January 12, 2023, 2:18am UTC](https://discuss.ai.google.dev/t/masking-with-attention-layer/30576 "2023-01-12T02:18:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kevin\_Guimard](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/kevin_guimard/32/15414_2.png) [@Kevin\_Guimard](https://discuss.ai.google.dev/u/Kevin_Guimard)\
**Post date:** [January 12, 2023, 2:18am UTC](https://discuss.ai.google.dev/t/masking-with-attention-layer/30576/1 "2023-01-12T02:18:33Z")

</div>

Hello everyone,

I’ve been running an implementation of Transformers with keras functional API and I got an issue on this line:

> attention = Attention()([query, value, key], mask=[query.\_keras\_mask, value.\_keras\_mask])

It triggers the following error:  
`TypeError: Cannot convert a symbolic Keras input/output to a numpy array.`  
After a look at the documentation, it seems that the passed tensors are of correct format. What’s the problem with this?

Thanks in advance,  
Kevin

---

<div class="post-metadata">

**Author:** ![chunduriv](https://avatars.discourse-cdn.com/v4/letter/c/bbe5ce/32.png) [@chunduriv](https://discuss.ai.google.dev/u/chunduriv)\
**Post date:** [January 16, 2023, 11:46am UTC](https://discuss.ai.google.dev/t/masking-with-attention-layer/30576/3 "2023-01-16T11:46:14Z")

</div>

@Kevin_Guimard

Welcome to the Tensorflow Forum!

Can you try to pass the inputs to the Attention layer as shown below and let us know?

`attention = Attention()(query, value, key, mask=[query._keras_mask, value._keras_mask])`

Thank you!

---

<div class="post-metadata">

**Author:** ![Kevin\_Guimard](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/kevin_guimard/32/15414_2.png) [@Kevin\_Guimard](https://discuss.ai.google.dev/u/Kevin_Guimard)\
**Post date:** [January 16, 2023, 7:08pm UTC](https://discuss.ai.google.dev/t/masking-with-attention-layer/30576/5 "2023-01-16T19:08:00Z")

</div>

For an unknown reason, I now get this error:

> AttributeError: ‘Tensor’ object has no attribute ‘\_keras\_mask’

Though, the query and the value objects are Dense layers applied to inputs, they should be KerasTensors that hold the \_keras\_mask field.

---

<div class="post-metadata">

**Author:** ![chunduriv](https://avatars.discourse-cdn.com/v4/letter/c/bbe5ce/32.png) [@chunduriv](https://discuss.ai.google.dev/u/chunduriv)\
**Post date:** [January 24, 2023, 3:24pm UTC](https://discuss.ai.google.dev/t/masking-with-attention-layer/30576/7 "2023-01-24T15:24:39Z")

</div>

Could you please share the standalone code to debug your issue?