I am part of a student group attempting to get Morpheus running on our HPC at school.
I have been working on getting a command line example similar to this example which worked until mid-December on the morpheus_22.09.sif, until this bug started to occur. Updating to the morpheus_22.11.sif file did not fix this issue.
How to get the issue:
We are running Morpheus using a conda environment. I then run this command below:
morpheus run pipeline-fil from-file --filename email_with_addresses.jsonlines serialize deserialize to-file --output.jsonlines --overwrite
Note that email_with_addresses.jsonlines is located inside of the git repo Morpheus/examples/data/email_with_addresses.jsonlines, and I am using hard coded directories for the file locations.
The issue that we receive is
Configuring Pipeline via CLI
Starting pipeline via CLI... Ctrl+C to Quit
W20230121 17:28:20.330015 2114319 topology.cpp:149] user_cpuset was not a subset of topo_cpuset; 70 cpus were dropped
E20230121 17:28:20.336977 2114492 controller.cpp:62] exception caught while performing update - this is fatal - issuing kill
E20230121 17:28:20.337110 2114492 context.cpp:125] rank: 0; size: 1; tid: 140318725318400; fid: 0x7f9e78041000:
set_exception issued; issuing kill to current runnable. Exception msg:
RuntimeError: The serialize stage cannot handle input of <class 'morpheus.messages.message_meta.MessageMeta'>.
Accepted input types: (<class 'morpheus.messages.multi_message.MultiMessage'>,)
At:
/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/single_port_stage.py(66):_pre_build
/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py(322): build
/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py(346): build
/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py(186): inner_build
E20230121 17:28:20.337141 2114492 context.cpp:125] rank: 0; size: 1; tid: 140318725318400; fid: 0x7f9e78041000:
set_exception issued; issuing kill to current runnable. Exception msg:
RuntimeError: The serialize stage cannot handle input of <class 'morpheus.messages.message_meta.MessageMeta'>.
Accepted input types: (<class 'morpheus.messages.multi_message.MultiMessage'>,)
At:
/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/single_port_stage.py(66): _pre_build
/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py(322): build
/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py(346): build
/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py(186): inner_build
E20230121 17:28:20.337234 2114492 manager.cpp:86] error detected on controller
E20230121 17:28:20.337292 2114319 runner.cpp:190] Runner::await_join - an exception was caught while awaiting on one or more contexts/instances - rethrowing
Exception occurred in pipeline. Rethrowing
Traceback (most recent call last):
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 251, in join
await self._srf_executor.join_async()
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 186, in inner_build
stage.build(builder)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py", line 346, in build
dep.build(builder, do_propagate=do_propagate)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py", line 322, in build
in_ports_pairs = self._pre_build()
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/single_port_stage.py", line 66, in
_pre_build
raise RuntimeError("The {} stage cannot handle input of {}. Accepted input types: {}".format(
RuntimeError: The serialize stage cannot handle input of <class 'morpheus.messages.message_meta.MessageMeta'>. Acc
epted input types: (<class 'morpheus.messages.multi_message.MultiMessage'>,)
Traceback (most recent call last):
File "/opt/conda/envs/morpheus/bin/morpheus", line 11, in <module>
sys.exit(run_cli())
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/cli/run.py", line 20, in run_cli
cli(obj={}, auto_envvar_prefix='MORPHEUS', show_default=True, prog_name="morpheus")
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1689, in invoke
return _process_result(rv)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1626, in _process_result
value = ctx.invoke(self._result_callback, value, **ctx.params)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/cli/commands.py", line 613, in post_pipeline
pipeline.run()
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 517, in run
asyncio.run(self._do_run())
File "/opt/conda/envs/morpheus/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/conda/envs/morpheus/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 495, in _do_run
await self.join()
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 251, in join
await self._srf_executor.join_async()
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 186, in inner_build
stage.build(builder)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py", line 346, in build
dep.build(builder, do_propagate=do_propagate)
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/stream_wrapper.py", line 322, in build
in_ports_pairs = self._pre_build()
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/single_port_stage.py", line 66, in _pre_build
raise RuntimeError("The {} stage cannot handle input of {}. Accepted input types: {}".format(
RuntimeError: The serialize stage cannot handle input of <class 'morpheus.messages.message_meta.MessageMeta'>.
Accepted input types: (<class 'morpheus.messages.multi_message.MultiMessage'>,)
Any help would be greatly appreciated, this would allow us to continue working with Morpheus.