Morpheus_22.11.sif Command-Line message-meta Issue

I am part of a student group attempting to get Morpheus running on our HPC at school.

I have been working on getting a command line example similar to this example which worked until mid-December on the morpheus_22.09.sif, until this bug started to occur. Updating to the morpheus_22.11.sif file did not fix this issue.

How to get the issue:
We are running Morpheus using a conda environment. I then run this command below:

morpheus run pipeline-fil from-file --filename email_with_addresses.jsonlines serialize deserialize to-file --output.jsonlines --overwrite

Note that email_with_addresses.jsonlines is located inside of the git repo Morpheus/examples/data/email_with_addresses.jsonlines, and I am using hard coded directories for the file locations.

The issue that we receive is

Configuring Pipeline via CLI
Starting pipeline via CLI... Ctrl+C to Quit
W20230121 17:28:20.330015 2114319 topology.cpp:149] user_cpuset was not a subset of topo_cpuset; 70 cpus were dropped
E20230121 17:28:20.336977 2114492 controller.cpp:62] exception caught while performing update - this is fatal - issuing kill
E20230121 17:28:20.337110 2114492 context.cpp:125] rank: 0; size: 1; tid: 140318725318400; fid: 0x7f9e78041000: 
set_exception issued; issuing kill to current runnable. Exception msg: 
RuntimeError: The serialize stage cannot handle input of <class 'morpheus.messages.message_meta.MessageMeta'>. 
Accepted input types: (<class 'morpheus.messages.multi_message.MultiMessage'>,) 

At:
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/single_port_stage.py(66):_pre_build
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py(322): build
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py(346): build
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py(186): inner_build
E20230121 17:28:20.337141 2114492 context.cpp:125] rank: 0; size: 1; tid: 140318725318400; fid: 0x7f9e78041000: 
set_exception issued; issuing kill to current runnable. Exception msg: 
RuntimeError: The serialize stage cannot handle input of <class 'morpheus.messages.message_meta.MessageMeta'>.
 Accepted input types: (<class 'morpheus.messages.multi_message.MultiMessage'>,) 

At:
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/single_port_stage.py(66): _pre_build
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py(322): build
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py(346): build
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py(186): inner_build
E20230121 17:28:20.337234 2114492 manager.cpp:86] error detected on controller
E20230121 17:28:20.337292 2114319 runner.cpp:190] Runner::await_join - an exception was caught while awaiting on one or more contexts/instances - rethrowing
Exception occurred in pipeline. Rethrowing
Traceback (most recent call last):
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 251, in join
    await self._srf_executor.join_async()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 186, in inner_build
    stage.build(builder)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py", line 346, in build
    dep.build(builder, do_propagate=do_propagate)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py", line 322, in build
    in_ports_pairs = self._pre_build()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/single_port_stage.py", line 66, in 
_pre_build
    raise RuntimeError("The {} stage cannot handle input of {}. Accepted input types: {}".format(
RuntimeError: The serialize stage cannot handle input of <class 'morpheus.messages.message_meta.MessageMeta'>. Acc
epted input types: (<class 'morpheus.messages.multi_message.MultiMessage'>,)

Traceback (most recent call last):
  File "/opt/conda/envs/morpheus/bin/morpheus", line 11, in <module>
    sys.exit(run_cli())
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/cli/run.py", line 20, in run_cli
    cli(obj={}, auto_envvar_prefix='MORPHEUS', show_default=True, prog_name="morpheus")
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1689, in invoke
    return _process_result(rv)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1626, in _process_result
    value = ctx.invoke(self._result_callback, value, **ctx.params)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/cli/commands.py", line 613, in post_pipeline
    pipeline.run()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 517, in run
    asyncio.run(self._do_run())
  File "/opt/conda/envs/morpheus/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/envs/morpheus/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 495, in _do_run
    await self.join()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 251, in join
    await self._srf_executor.join_async()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 186, in inner_build
    stage.build(builder)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py", line 346, in build
    dep.build(builder, do_propagate=do_propagate)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py", line 322, in build
    in_ports_pairs = self._pre_build()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/single_port_stage.py", line 66, in _pre_build
    raise RuntimeError("The {} stage cannot handle input of {}. Accepted input types: {}".format(
RuntimeError: The serialize stage cannot handle input of <class 'morpheus.messages.message_meta.MessageMeta'>. 
Accepted input types: (<class 'morpheus.messages.multi_message.MultiMessage'>,)

Any help would be greatly appreciated, this would allow us to continue working with Morpheus.

The order of your pipeline is wrong. You should deserialize from the input file and serialize to the output file.

Thank you for the help!