How to use converted tensorflowjs model

TensorFlow General Discussion
1

I have converted a saved_model tensorflow model into tensorflowjs using the converter.
Link to tfjsexport tfjsexport - Google Drive. Link to the saved_model saved_model - Google Drive. The label map used to train the model is as follows

item {
id: 1
name: “uk_drivers_license”
}

item {
id: 2
name: “passport”
}

I am tyring to work with this react repo which can be found on github nicknochnack/TFODApp which draws boxes around the detected object if the object is found. When running the app no boxes are drawn around presented passport or driver licenses.

I’m using TensorFlow.js to perform object detection, and I’m having some trouble understanding the output tensors returned by executeAsync(). After logging the tensors, I received the following data structure, and I’m not sure which tensor corresponds to the bounding boxes, scores, and class labels. Additionally, my program isn’t functioning as expected.

Here is the relevant part of my code:

// TODO - Make Detections
const img = tf.browser.fromPixels(video);
const resized = tf.image.resizeBilinear(img, [640, 480]);
const casted = resized.cast("int32");
const expanded = casted.expandDims(0);
const obj = await net.executeAsync(expanded);

const tensors = await Promise.all(obj.map((t) => t.array()));

console.log(tensors);

The logged output of the tensors array contains the following structure:

0: [Array(12804)]
1: [Array(100)]
2: [Array(100)]
3: [Array(12804)]
4: [Array(100)]
5: 100
6: [Array(100)]
7: [Array(100)]
length: 8

The first item has the shape:

Array(1)
0: Array(12804)
[0 … 9999]
[0 … 99]
0: (4) [-0.030606862157583237, -0.03885898366570473, 0.07085278630256653, 0.06487758457660675]

The second item looks like this:

Array(1)
0: Array(100)
0: Array(4)
    0: 0.36487695574760437
    1: 0.2668212950229645
    2: 0.9987643957138062
    3: 0.9981945753097534
length: 4

And so on for the remaining items:

  • Third tensor:

    Array(1)
0: Array(100)
0: Array(3)
    0: 0.002725705737248063
    1: 0.031245484948158264
    2: 0.1295188069343567

  • Fourth tensor:

    Array(1)
0: Array(12804)
[0 … 9999]
[0 … 99]
0: Array(3)
    0: 0.005913852714002132
    1: 0.006506113335490227
    2: 0.006477345712482929

  • Fifth tensor:

    Array(1)
0: Array(100)
    0: 0.1295188069343567

  • Sixth tensor:

    Array(1)
0: 100

  • Seventh tensor:

    Array(1)
0: Array(100)
    0: 12457

  • Eighth tensor:

    Array(1)
0: Array(100)
    0: 2
    1: 2
    2: 1
    3: 1

In the TFOD repo

  const boxes = await obj[4].array();
  const classes = await obj[5].array();
  const scores = await obj[6].array();

Are the constant send a drawRect function which is responsible for drawing the boxes around items. I have limited knowledge but can understand classes can not be he sixth tensor obj[5] as its just the number 100 it should look like eight tensor which is an array of 1 and 2’s which represent the numbers provided in the label map used for training.

My question:

  1. What does each tensor represent? Specifically, which ones correspond to the bounding boxes, scores, and class labels?
  2. Given that my program is not working as expected, am I correctly interpreting the outputs, or should I make any adjustments to how I’m handling these tensors?
  3. If all above fails how can I verify my model was converted correctly as I know what my model worked correctly before being exported.

Thank you for any clarification you can provide!