Error: 12 UNIMPLEMENTED

Please provide the following information when requesting support.

Hardware - GPU (V100)
Hardware - CPU -
Operating System (Ubuntu)
Riva Version (1.7.0 and 1.8.0)
TLT Version (if relevant)
How to reproduce the issue? (This is for errors. Please share the command and the detailed log here)

We are trying to use riva-contact as starter example;

  1. Starting Riva 1.7 or 1.8 (both have this issue) backend (served over 8001)
  2. Starting Riva client 1.7 or 1.8 for running npm start run (server over 6006)

Sometimes npm start run works well and we are able to use it, but once we interrupt the npm server and try to restart it falls into UNIMPLEMENTED 12. After this, we need to restart the machine and try many times to get it to work again. It seems rather we are receiving this error almost randomly.

Client at ::ffff:xx.xx.xx.14 disconnected. Reason:  client namespace disconnect
Client connected from ::ffff:xx.xx.xx.14
Initializing Riva ASR
Error via streamingRecognize callback
Error: 12 UNIMPLEMENTED: 
    at Object.exports.createStatusError (/workspace/lc/node_modules/grpc/src/common.js:91:15)
    at ClientDuplexStream._emitStatusIfDone (/workspace/lc/node_modules/grpc/src/client.js:233:26)
    at ClientDuplexStream._receiveStatus (/workspace/lc/node_modules/grpc/src/client.js:211:8)
    at Object.onReceiveStatus (/workspace/lc/node_modules/grpc/src/client_interceptors.js:1311:15)
    at InterceptingListener._callNext (/workspace/lc/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/workspace/lc/node_modules/grpc/src/client_interceptors.js:618:8)
    at /workspace/lc/node_modules/grpc/src/client_interceptors.js:1127:18 {
  code: 12,
  metadata: Metadata { _internal_repr: {}, flags: 0 },
  details: ''
}
/workspace/lc/node_modules/express-socket.io-session/lib/hash.js:12
                JSON.stringify(sess, function(key, val) {
                     ^

TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'ClientDuplexStream'
    |     property '_writableState' -> object with constructor 'WritableState'
    |     property 'afterWriteTickInfo' -> object with constructor 'Object'
    --- property 'stream' closes the circle
    at JSON.stringify (<anonymous>)
    at hash (/workspace/lc/node_modules/express-socket.io-session/lib/hash.js:12:8)
    at isSaved (/workspace/lc/node_modules/express-socket.io-session/index.js:96:51)
    at shouldSave (/workspace/lc/node_modules/express-socket.io-session/index.js:112:8)
    at /workspace/lc/node_modules/express-socket.io-session/index.js:66:10
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! riva-contact@1.0.0 start: `node server.js "run"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the riva-contact@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-01-07T14_39_38_068Z-debug.log

Hello,

Is there any news on this issue? I am getting the same error on Riva 1.8-beta.

Thank you.

We believe this may be a bug in the grpc module in node.js. We’re trying to find a reliable way to reproduce and fix.

Are you able to share any logs from the Riva server from this crash? Once you start getting the UNIMPLEMENTED error, run docker logs riva-speech. We are looking for something out of the ordinary that may be causing Riva to respond like this.

Thanks!

@selim , @mertcan.alici18

Another possibility: we’ve seen similar UNIMPLEMENTED behaviour on occasion after an nvidia driver update. If your drivers get updated (sometimes automatically), the Riva models may need to get rebuilt. You might try clearing the model repo with riva_clean.sh, then running riva_init.sh again.