Examples of v0.5.0 pipeline definition files

Are there any current examples of a v0.5.0 pipeline definition file around?

I’ve just upgraded my deployment of Clara to 0.7.1 and have tried simply changing the API version from 0.4.0 to 0.5.0 for a very simple pipeline definition. However, the platform API isn’t happy about something and won’t create the pipeline, it’s fine when I revert back to specifying 0.4.0.

I’m curious to know if there are any syntax changes etc with the new version causing my pipeline to break.

Regards,

Darren

v0.5.0 pipeline definitions added support for shared memory data transfer between pipeline-job operators. That said, to the best of my recollection any v0.4.0 pipeline definition should be supported in v0.5.0.

Could you provide the exact command you’re using and error you’re receiving? That should help me to better understand the issue and provide more useful guidance.

Hi @jwyman,

Yes, I’m very keen to tryout the new shared memory functions as I have been using the FastIO shared memory features in my own operators.

I’ve now tried some more things and have isolated the problem down to the input and output definitions, particularly the use of “path”. After re-reading through the docs I’m a little bit unsure exactly how these should now be specified.

Below is on of my simple pipeline definitions:

api-version: 0.4.0
orchestrator: Clara
name: csiroct-raw-correction-pipeline
parameters:
  JSON_PIPELINE: test
pull-secrets:
  - ngc-clara
  - regcred
operators:
  - name: raw-correction
    description: Raw correction and output to file
    container:
      image: docker-registry.it.csiro.au/csiroct/csiroct-clara-operator-python
      tag: 0.7.1-2008.1
    variables:
      JSON_PIPELINE: ${{JSON_PIPELINE}}
      OPERATOR_TYPE: raw_correction
    input:
    - path: /input
    output:
    - name: out
      path: /output 

Using the above works fine when the pipeline is created via the “pipelines” “create” GPRS API call.

However, if I simply change to API version to “0.5.0” and do the same create call I get the following in the response.

['Failed to store pipeline.', 'Unable to complete Pipelines::Create, failed to load pipeline definition. Nvidia.Clara.Platform.Server.Models.PipelineLoadException: "Invalid Pipeline Syntax;\nerror: .operators[0].output[0].type must be one of \'array\', \'stream\', \'float16, f16, float32, f32, float64, f64, int8, i8, int16, i16, int32, i32, int64, i64, uint8, u8, uint16, u16, uint32, u32, uint64, u64, string\' (Operator: raw-correction).\nerror: .operators[0].input[0].type must be one of \'array\', \'stream\', \'float16, f16, float32, f32, float64, f64, int8, i8, int16, i16, int32, i32, int64, i64, uint8, u8, uint16, u16, uint32, u32, uint64, u64, string\' (Operator: raw-correction).\n".']

Looking at the message above and the docs again, is the problem due to be needing to either 1) omit the “path” property in 0.5.0 for input and output or 2) include an additional “type” property? I’m not really sure how this should now be specified.

I did test simply removing both “input” and “output” sections and the pipeline was created successfully, however I’m not sure if this is correct. I also tried removing both “path” properties but this didn’t work.

Regards,

Darren

Oh, one more thing. In our last discussion a month or so ago you have me access to the Clara R6 GRPC proto files, I was wondering if there have been any updates to these in the latest Clara release? If so, would you mind sending them to me.

Regards,

Darren