Inpainting with Imagen

Gemini API
, ,
1

I am trying to inpaint using Imagen via API with Python. I am able to successfully generate images from a text prompt this way but the inpainting is failing. Specifically, the failure is that while an image is created, it is creating an image from scratch rather than inpainting into the masked area. In my code, I am specifying the base_img (which is a JPG file) and mask_img (which is a PNG file).

I am trying to follow this tutorial: Add image content using mask-based inpainting with Imagen | Generative AI on Vertex AI | Google Cloud

I have uploaded the base image and mask file, as well as a sample output that I get after running the code below. I am new to both Imagen and Python, so any pointers greatly appreciated!

# Based on https://cloud.google.com/vertex-ai/generative-ai/docs/samples/generativeaionvertexai-imagen-edit-image-inpainting-insert-mask?hl=en#generativeaionvertexai_imagen_edit_image_inpainting_insert_mask-python

# Setup environment
output_dir = "Z:\\Temp\\"

# Setup Vertex AI and Imagen 3 model
import vertexai
from vertexai.preview.vision_models import Image, ImageGenerationModel
project_id = "123456789012"
vertexai.init(project=project_id, location="us-west4")
model = ImageGenerationModel.from_pretrained("imagegeneration@006")

# Define base image file and inpaint area
input_file = "testimage.jpg"
mask_file = "testimagemask.png"
base_img = Image.load_from_file(location=output_dir + input_file)
mask_img = Image.load_from_file(location=output_dir + mask_file)

# Define prompt
prompt = "glass vase with purple flowers"
negative_prompt = "people, animals"

#Define parameters
images = model.edit_image(
	base_image=base_img,
    mask=mask_img,
	prompt=prompt,
	negative_prompt=negative_prompt,
	edit_mode="inpainting-insert",
	number_of_images=2,
	language="en",
	safety_filter_level="block_low_and_above",
	person_generation="dont_allow",
)

# Loop through each image and save it with a dynamic filename
for idx, image in enumerate(images):
	# Initialize timestamp, location & filename protocol
	output_file = f"{output_dir}gen_img_{int(idx)+1}.png"

	# Save the image with the dynamic filename
	image.save(location=output_file, include_generation_parameters=False)

testimage
testimage900×600 124 KB

testimagemask
testimagemask900×600 247 Bytes

gen_img_1
gen_img_1900×600 39.1 KB

1 Like
3

Hi @gurs ,

This usually happens if the base image and mask image don’t match properly. Make sure both images are PNG and have the same dimensions. Convert your base JPG to PNG (RGBA). The mask image should be black (keep) and white (inpaint). Ensure you’re using the right format in the API call as shown in the tutorial.
If the format or size is off, Imagen will ignore the mask and generate from scratch.