I am part of a student group attempting to get Morpheus running on our HPC at our school.
I have been attempting to create the basic pass-through example from this link. One thing I noticed is that in the “Putting the stage together” section it has the line import srf
which I am confident is supposed to be import mrc
now.
I have pulled the most recent Morpheus version from github into /data/sdp/cybersecurity_ai/mrc_test/Morpheus
. I have a directory Morpheus/passthru
that contains my run_passthru.py
file as well as my pass_thru.py
file.
My run_passthru,py:
import logging
import os
import sys
sys.path.insert(0, '/data/sdp/cybersecurity_ai/mrc_test/Morpheus')
from morpheus.config import Config
#from ../../Morpheus/morpheus.config import Config
from morpheus.pipeline import LinearPipeline
from morpheus.stages.general.monitor_stage import MonitorStage
from morpheus.stages.input.file_source_stage import FileSourceStage
from morpheus.utils.logger import configure_logging
from pass_thru import PassThruStage
def run_pipeline():
#print("DEBUG: " + MonitorStage.__path__)
# Enable the Morpheus logger
configure_logging(log_level=logging.DEBUG) # fixme?
root_dir = os.environ['MORPHEUS_ROOT'] # fixme?
# input_file = os.path.join(root_dir, 'examples/data/email_with_addresses.jsonlines')
input_file = "/data/sdp/cybersecurity_ai/mrc_test/pass_thru/email_with_addresses.jsonlines"
config = Config()
# Create a linear pipeline object
pipeline = LinearPipeline(config)
# Set source stage
pipeline.set_source(FileSourceStage(config, filename=input_file, iterative=False))
# Add our own stage
pipeline.add_stage(PassThruStage(config))
# Add monitor to record the performance of our new stage
pipeline.add_stage(MonitorStage(config))
# Run the pipeline
pipeline.run()
if __name__ == "__main__":
run_pipeline()
My pass_thru.py file:
import typing
#import srf
import mrc
from morpheus.cli.register_stage import register_stage
from morpheus.pipeline.single_port_stage import SinglePortStage
from morpheus.pipeline.stream_pair import StreamPair
@register_stage("pass-thru")
class PassThruStage(SinglePortStage):
"""
A Simple Pass Through Stage
"""
@property
def name(self) -> str:
return "pass-thru"
def accepted_types(self) -> typing.Tuple:
return (typing.Any, )
def supports_cpp_node(self) -> bool:
return False
def on_data(self, message: typing.Any):
# Return the message for the next stage
return message
def _build_single(self, builder: srf.Builder, input_stream: StreamPair) -> StreamPair:
node = builder.make_node(self.unique_name, self.on_data)
builder.make_edge(input_stream[0], node)
return node, input_stream[1]
For context this is being ran on MSOE’s HPC.
Steps to get the issue I am getting:
- I first reserve a job on one of the DGX nodes of the HPC and then ssh into it.
- I run the command
srun --gpus=1 --pty singularity exec --nv -B /data/:/data /data/sdp/cybersecurity_ai/sif/morpheus_22.11-runtime.sif bash
to get into the sif. - I cd to
/data/sdp/cybersecurity_ai/mrc_test/Morpheus
- I run
python3 passthru/run_passthru.py
Once I do this I get this error:
Singularity> python3 run_passthru.py
installing mrc
Traceback (most recent call last):
File "run_passthru.py", line 10, in <module>
from morpheus.pipeline import LinearPipeline
File "/data/sdp/cybersecurity_ai/mrc_test/Morpheus/morpheus/pipeline/__init__.py", line 21, in <module>
from morpheus.pipeline.stream_pair import StreamPair
File "/data/sdp/cybersecurity_ai/mrc_test/Morpheus/morpheus/pipeline/stream_pair.py", line 20, in <module>
import mrc
ModuleNotFoundError: No module named 'mrc'
To my understanding the MRC module should already be installed with the runtime environment but I could be mistaken.
Any tips on how to resolve this problem would be greatly appreciated as we are hoping that solving the python problem would allow us to create custom stages.