Calling addRNNv2 with multiple batches

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi all,

I am encountering incorrect results when calling RNNv2 with multiple batches ( ie. multiple observations).

For a single observation, I get the right results but the results are incorrect for batchSize greater than 1. What might I be doing wrong?

I am currently creating the input to RNN as follows:

[b]    auto iSequenceInputITensor = network->addInput(
        "SequenceInputData", DataType::kFLOAT, Dims2(SEQUENCE_LENGTH, EMBEDDING_LENGTH);[/b]

Further, I construct the add RNNv2Layer as follows:

[b]    // construct weights
    Weights hidden_state_weights{DataType::kFLOAT, HiddenState,  num_hidden_units};
    Weights cell_state_weights{DataType::kFLOAT, CellState, num_hidden_units};

    // add constant layers
    auto HiddenStateTensor = network->addConstant(Dims2{1, num_hidden_units}, hidden_state_weights);
    auto CellStateTensor = network->addConstant(Dims2{1, num_hidden_units}, cell_state_weights);


    const int layerCount = 1;

    // add RNN Layer
    auto RNNLayer = network->addRNNv2(*prevLayerTensor, layerCount, num_hidden_units, seqLength, RNNOperation::kLSTM);
    
    // set Initial Hidden and CellState
    RNNLayer->setHiddenState(*HiddenStateTensor->getOutput(0));
    RNNLayer->setCellState(*CellStateTensor->getOutput(0));[/b]

BatchSize is populated in the call to enqueue and builder’s property:

builder->setMaxBatchSize(batchSize);
 context->enqueue(batchSize, m_buffers, stream, nullptr);

I have experimented with Dims3 objects whose format is Nbatches x SeqLen x EmbeddingSize, but this did not yield the right result as well.

The documentation mentions “The input ITensor should contain zero or more index dimensions {N1, …, Np}”

What do N1,… to Np represent here?

My sample test parameters are as follows:

Sequence length = 4
Embedding size = 3
batches/observations = 2
num hidden units = 10

Thanks,