Error while importing modules

FireWorks
1

Hello everybody,

I am currently on a project where we intend to use fireworks as our workflow manager. I have installed everything but I am having “Module not find” errors when trying to do a rlaunch rapidfire. This is my example code:

from fireworks import Firework, Workflow, LaunchPad, PyTask
import numpy
from matrix_operations import multiply_by_two, divide_by_two, subtract_half

initial_matrix = numpy.ones((30, 30))

fw1 = Firework(PyTask(func='matrix_operations.multiply_by_two', args=[initial_matrix, 2], stored_data_varname='result'), name="multiply_by_two")
fw2 = Firework(PyTask(func='matrix_operations.divide_by_two', stored_data_varname='result', args=[initial_matrix, 2]), name="divide_by_two")
fw3 = Firework(PyTask(func='matrix_operations.subtract_half', stored_data_varname='result', args=[initial_matrix, 0.5]), name="subtract_half")

# Define dependencies between the FireWorks
dependencies = {fw1: [fw2], fw2: [fw3]}

# Create the workflow
wf = Workflow([fw1, fw2, fw3], dependencies)

# Connect to the FireWorks database and add the workflow
lp = LaunchPad()
lp.add_wf(wf)

And this is the matrix_operations:

# matrix_operations.py
import numpy as np


def multiply_by_two(matrix):
    return matrix * 2


def divide_by_two(matrix):
    return matrix / 2


def subtract_half(matrix):
    return matrix - 0.5


def save_result(matrix, filename="final_matrix.txt"):
    np.savetxt(filename, matrix)
    print(f"Matrix saved to {filename}")

Both files are in the same working directory. I know I could use numpy (and in this case, the import works) for these operations, but I wanted an example like this, since I will be using our in-house functions for the operations.

Below is the error message:

2024-03-14 16:37:43,911 INFO Hostname/IP lookup (this will take a few seconds)
2024-03-14 16:37:43,966 INFO Created new dir C:\Users\marinhoa\PycharmProjects\fireworksTest\launcher_2024-03-14-15-37-43-964949
2024-03-14 16:37:43,967 INFO Launching Rocket
2024-03-14 16:37:43,981 INFO RUNNING fw_id: 3 in directory: C:\Users\marinhoa\PycharmProjects\fireworksTest\launcher_2024-03-14-15-37-43-964949
2024-03-14 16:37:43,992 INFO Task started: PyTask.
Traceback (most recent call last):
File “C:\Users\marinhoa\AppData\Local\anaconda3\envs\fireworks_env\lib\site-packages\fireworks\core\rocket.py”, line 261, in run
m_action = t.run_task(my_spec)
File “C:\Users\marinhoa\AppData\Local\anaconda3\envs\fireworks_env\lib\site-packages\fireworks\user_objects\firetasks\script_task.py”, line 170, in run_task
mod = import(modname, globals(), locals(), [str(funcname)], 0)
ModuleNotFoundError: No module named ‘matrix_operations’
2024-03-14 16:37:44,011 INFO Rocket finished

Thanks in advance for any tip that might get me further on this problem.

2

Hey @marinhoa,

To use an external module with fireworks, it has to be in your PYTHONPATH. Try to extend your PYTHONPATH in your local environment, or in the job submission script (in the pre_rocket section of my_qadapter.yaml) like so:
export PYTHONPATH=$PYTHONPATH:<path to matrix_operations.py>

2 Likes
3

Hey @Aaron_Kaplan

Thank your for the fast response. I am using PyCharm as IDE and adding more paths to the PYTHONPATH structure did not work. However, this command on the PowerShell worked.

$Env:PYTHONPATH = $Env:PYTHONPATH + ";C:\Users\marinhoa\PycharmProjects\MyModules"

Since I wanna keep everything in a self-made environment, I think the best option will be creating a package and adding it by pip install in the env I am using. Thanks a lot for your contribution.

Cheers,

André