For a machine learning task I have coded with support of the internet a configurable PoinNet Version in tensorflow keras.
The keras example from PointNet I got here: Point cloud classification with PointNet
I am using tensorflow = "2.11.0"
What do I want?
- Implemented an easy version of PointNet (see code below)
- I have some stl files which I load with
open3d = "^0.18.0"to convert it to PointClouds - I want to use
tf.data.Datasetpipe to load the data and pre- and postprocess it - I want to predict a certain curve/values on the end, depending on the input (but this is not the problem)
- For simplicity and clarity I implemented two options: 1. Load all pointcloud data in one numpy array before training AND 2.Load pointcloud data while training via
tf.py_function
My Problem
- When using option 2 (load pointcloud data while training via
tf.py_function): regardless of the number of points in a point cloud, the data is passed through the neural network, although it does not match theINPUT_SHAPEfrom the neural net - When I use option 1 (load all pointclouds before training and then build the
tf.data.Datasetpipe): everything works as expected. If the pointcloud shape does not match theINPUT_SHAPEof the neural net, an error occurs. - Why is there the strange behavior? Is it because of the usage of
tf.py_function?
My code block for the PointNet
import os
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import open3d as o3d
import pandas as pd
import random
##############################################################################################
# define constants
NUM_POINTS = 20
INPUT_SHAPE = [20, 3]
BATCHSIZE = 6
NUM_CLASSES = 10
NUM_SAMPLES = 6
WHICH_LOADING = "all" # all, single_py_function
PATH_FOLDER = "data/few_examples/stl_examples/"
##############################################################################################
# define loading functions
def load_all_pointclouds(list_filepaths):
list_arrays = []
for path in list_filepaths:
mesh = o3d.io.read_triangle_mesh(path)
pointcloud = mesh.sample_points_poisson_disk(
number_of_points=NUM_POINTS, init_factor=5
)
array_pointcloud = np.asarray(pointcloud.points)
list_arrays.append(array_pointcloud)
print("array_pointcloud.shape: ", array_pointcloud.shape)
arrays_pointcloud = np.array(list_arrays)
return arrays_pointcloud
def load_single_pointcloud(filepath):
filepath = filepath.numpy().decode("utf-8")
mesh = o3d.io.read_triangle_mesh(filepath)
pointcloud = mesh.sample_points_poisson_disk(
number_of_points=NUM_POINTS, init_factor=5
)
array_pointcloud = np.asarray(pointcloud.points)
array_pointcloud = np.float32(array_pointcloud)
array_pointcloud = tf.ensure_shape(array_pointcloud, INPUT_SHAPE)
print("array_pointcloud.shape: ", array_pointcloud.shape)
return array_pointcloud
##############################################################################################
### create filepaths
list_files = os.listdir(PATH_FOLDER)
list_filepaths = [PATH_FOLDER + x for x in list_files]
### create y data
df_y = pd.DataFrame(np.random.rand(NUM_SAMPLES, NUM_CLASSES))
ds_y = tf.data.Dataset.from_tensor_slices(df_y)
### logic for loading while train or preload all data
if WHICH_LOADING == "single_py_function":
df_3d = pd.DataFrame(list_filepaths)
ds_3d = tf.data.Dataset.from_tensor_slices(df_3d)
ds_3d = ds_3d.map(
lambda filepath: tf.py_function(
load_single_pointcloud, [filepath[0]], Tout=tf.float32
)
)
if WHICH_LOADING == "all":
arr_3d = load_all_pointclouds(list_filepaths)
ds_3d = tf.data.Dataset.from_tensor_slices(arr_3d)
### zip datasets and postprocess data
ds = tf.data.Dataset.zip((ds_3d, ds_y))
ds = ds.shuffle(buffer_size=1000, seed=42).batch(BATCHSIZE).prefetch(tf.data.AUTOTUNE)
### check ds data
for i, element in enumerate(ds.as_numpy_iterator()):
print("array_pointcloud.type: ", type(element[0]))
print("array_pointcloud.shape: ", element[0].shape)
print("y.shape", element[1].shape)
### Build a model
def conv_bn(x, filters):
x = layers.Conv1D(filters, kernel_size=1, padding="valid")(x)
x = layers.BatchNormalization(momentum=0.0)(x)
return layers.Activation("relu")(x)
def dense_bn(x, filters):
x = layers.Dense(filters)(x)
x = layers.BatchNormalization(momentum=0.0)(x)
return layers.Activation("relu")(x)
class OrthogonalRegularizer(keras.regularizers.Regularizer):
def __init__(self, num_features, l2reg=0.001):
self.num_features = num_features
self.l2reg = l2reg
self.eye = tf.eye(num_features)
def __call__(self, x):
x = tf.reshape(x, (-1, self.num_features, self.num_features))
xxt = tf.tensordot(x, x, axes=(2, 2))
xxt = tf.reshape(xxt, (-1, self.num_features, self.num_features))
return tf.reduce_sum(self.l2reg * tf.square(xxt - self.eye))
def tnet(inputs, num_features):
# Initalise bias as the indentity matrix
bias = keras.initializers.Constant(np.eye(num_features).flatten())
reg = OrthogonalRegularizer(num_features)
x = conv_bn(inputs, 32)
x = conv_bn(x, 64)
x = conv_bn(x, 512)
x = layers.GlobalMaxPooling1D()(x)
x = dense_bn(x, 256)
x = dense_bn(x, 128)
x = layers.Dense(
num_features * num_features,
kernel_initializer="zeros",
bias_initializer=bias,
activity_regularizer=reg,
)(x)
feat_T = layers.Reshape((num_features, num_features))(x)
return layers.Dot(axes=(2, 1))([inputs, feat_T])
inputs = keras.Input(shape=INPUT_SHAPE)
x = tnet(inputs, 3)
x = conv_bn(x, 32)
x = conv_bn(x, 32)
x = tnet(x, 32)
x = conv_bn(x, 32)
x = conv_bn(x, 64)
x = conv_bn(x, 512)
x = layers.GlobalMaxPooling1D()(x)
x = dense_bn(x, 256)
x = layers.Dropout(0.3)(x)
x = dense_bn(x, 128)
x = layers.Dropout(0.3)(x)
outputs = layers.Dense(NUM_CLASSES, activation="softmax")(x)
model = keras.Model(inputs=inputs, outputs=outputs, name="pointnet")
### compile and fit data
# model.summary()
model.compile(
loss="mse",
optimizer=keras.optimizers.Adam(learning_rate=0.001),
)
model.fit(ds, epochs=2)
Print Results on different configurations
Configuration 1 - loading all before train works fine when shapes match
NUM_POINTS = 20
INPUT_SHAPE = [20, 3]
BATCHSIZE = 6
WHICH_LOADING = "all_before"
Print Result for Config 1
array_pointcloud.shape in load_all_pointclouds function: (20, 3)
array_pointcloud.shape in load_all_pointclouds function: (20, 3)
array_pointcloud.shape in load_all_pointclouds function: (20, 3)
array_pointcloud.shape in load_all_pointclouds function: (20, 3)
array_pointcloud.shape in load_all_pointclouds function: (20, 3)
array_pointcloud.shape in load_all_pointclouds function: (20, 3)
array_pointcloud.type in dataset loop: <class 'numpy.ndarray'>
array_pointcloud.shape in dataset loop: (6, 20, 3)
y.shape in dataset loop (6, 10)
INPUT_SHAPE neural net: [20, 3]
NUM_POINTS predefined: 20
BATCHSIZE predefined: 6
Epoch 1/2
1/1 [==============================] - 9s 9s/step - loss: 0.7237
Epoch 2/2
1/1 [==============================] - 0s 16ms/step - loss: 0.6924
Configuration 2 - loading all before train throws error as expected when shapes do not match
NUM_POINTS = 21
INPUT_SHAPE = [20, 3]
BATCHSIZE = 6
WHICH_LOADING = "all_before"
Print Result for Config 2
array_pointcloud.shape in load_all_pointclouds function: (21, 3)
array_pointcloud.shape in load_all_pointclouds function: (21, 3)
array_pointcloud.shape in load_all_pointclouds function: (21, 3)
array_pointcloud.shape in load_all_pointclouds function: (21, 3)
array_pointcloud.shape in load_all_pointclouds function: (21, 3)
array_pointcloud.shape in load_all_pointclouds function: (21, 3)
array_pointcloud.type in dataset loop: <class 'numpy.ndarray'>
array_pointcloud.shape in dataset loop: (6, 21, 3)
y.shape in dataset loop (6, 10)
INPUT_SHAPE neural net: [20, 3]
NUM_POINTS predefined: 21
BATCHSIZE predefined: 6
Error message for Config 2
ValueError: Input 0 of layer "pointnet" is incompatible with the layer: expected shape=(None, 20, 3), found shape=(None, 21, 3)
Configuration 3 - loading while train works fine when shapes match
NUM_POINTS = 20
INPUT_SHAPE = [20, 3]
BATCHSIZE = 6
WHICH_LOADING = "single_py_function_while_train"
Print Result for Config 3
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.type in dataset loop: <class 'numpy.ndarray'>
array_pointcloud.shape in dataset loop: (6, 20, 3)
y.shape in dataset loop (6, 10)
INPUT_SHAPE neural net: [20, 3]
NUM_POINTS predefined: 20
BATCHSIZE predefined: 6
Epoch 1/2
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
1/1 [==============================] - 16s 16s/step - loss: 0.5778
Epoch 2/2
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
array_pointcloud.shape in load_single_pointcloud function: (20, 3)
1/1 [==============================] - 0s 193ms/step - loss: 0.5620
Configuration 4 - loading while train does not work as expected if shapes do not match. It should throw an error. But it does not.
NUM_POINTS = 21
INPUT_SHAPE = [20, 3]
BATCHSIZE = 6
WHICH_LOADING = "single_py_function_while_train"
Print Result for Config 3
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.type in dataset loop: <class 'numpy.ndarray'>
array_pointcloud.shape in dataset loop: (6, 21, 3)
y.shape in dataset loop (6, 10)
INPUT_SHAPE neural net: [20, 3]
NUM_POINTS predefined: 21
BATCHSIZE predefined: 6
Epoch 1/2
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
1/1 [==============================] - 13s 13s/step - loss: 0.6657
Epoch 2/2
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
array_pointcloud.shape in load_single_pointcloud function: (21, 3)
1/1 [==============================] - 0s 64ms/step - loss: 0.6431
Conclusion
So, how is it possible that a tf neural net can be successful trained, although the data shape and the input shape does not match when loading and feeding data while train? Is it because of the usage of tf.py_function?
Thanks for help.