Hi Anubhav,
I don’t think I’m in this case : I do not open files, just do a lot of file movement with some FileTransferTasks. I use some subprocesses, but not with Popen as I don’t need the output, just the return code.
I use the old API call() func (subprocess — Subprocess management — Python 3.12.4 documentation). I guess it closes the file descriptor after execution …
The script which has caused the error (too many open files) is just a dummy script task echoing that the workflow execution has completed :
def get_dummy_end_fw():
# get caller module name
caller = inspect.currentframe().f_back
caller_name = caller.f_globals['__name__']
# create dummy end fw
ft = ScriptTask.from_str("echo \"ending %s workflow\"" % caller_name)
task_name = "dummy end: %s" % caller_name
fw_end = Firework([ft], name=task_name)
return fw_end
The only part in the code where I do an explicit open is into a with construct :
with open(param_path, “w”) as paramfile:
paramfile.write("%s" % param_content)
I’m an experienced programmer (C, C++, Fortran & Perl mainly), but pretty new to Python…
Maybe have I missed something … for example, I use the psycopg2 lib to connect to a PgSQL DB
So when I create a cursor like this in a funct :
cursor = db_connector.cursor()
I suppose it is destroyed at the end of the function, since no more reference points to this object …
the number of fireworks in all the workflows in the launchpad when it crashed was greater than 1024 (more than 1300 actually).
I will launch further testes and check with lsof command which process open which file. Not this week, because I’m in travel, but next week I will tell you more.
Best,
David
···
Le mardi 13 septembre 2016 02:28:24 UTC+2, Anubhav Jain a écrit :
Hi David
Sorry for the late reply, somehow I did not get the latest FWS tickets in my email
Can you tell me some details of your script? I want to be sure that the issue with too many open files pertains to FireWorks and not what is going on inside the script (e.g., if the script is opening a file and not closing it). e.g. see this:
http://stackoverflow.com/questions/16526783/python-subprocess-too-many-open-files
http://stackoverflow.com/questions/24682167/too-many-open-files-error-with-popen-of-subprocess
Best,
Anubhav
On Friday, September 2, 2016 at 1:47:26 AM UTC-7, [email protected] wrote:
Hi,
I ran in a problem of too many open files during execution of big workflows (several thousands of fireworks). the system is currently under development so it is not a problem, but it will, because the final system is intended to run many worflows of several thousand of fireworks on a dedicated cluster.
You can find here after the state of a simple scriptTask FIZZLED and the call stack which lead to the problem.
My system limit was 1024 opened files (ulimit -n). I have increased this limit to 4096 and everything runs fine now, but I wonder how I can reduce the number of files open by firework :
- should I reduce the number of fireworks and increase the number of tasks inside them or something like this ?
- Run the script-task with useShell = False ?
- … every advice will be appreciated
And last but not least : thanks a lot for the good work and for this very nice tool that makes my life easier
Cheers,
David
“stored_data”: {
“_exception”: {
- “_details”: null,
- “_failed_task_n”: 0,
- “_stacktrace”: “Traceback (most recent call last):\n File "/usr/lib/python3.4/site-packages/fireworks/core/rocket.py", line 211, in run\n m_action = t.run_task(my_spec)\n File "/usr/lib/python3.4/site-packages/fireworks/user_objects/firetasks/script_task.py", line 37, in run_task\n return self._run_task_internal(fw_spec, stdin)\n File "/usr/lib/python3.4/site-packages/fireworks/user_objects/firetasks/script_task.py", line 48, in _run_task_internal\n shell=self.use_shell)\n File "/usr/lib64/python3.4/subprocess.py", line 859, in init\n restore_signals, start_new_session)\n File "/usr/lib64/python3.4/subprocess.py", line 1359, in _execute_child\n errpipe_read, errpipe_write = os.pipe()\nOSError: [Errno 24] Too many open files\n”
},
- “_message”: “runtime error during task”,
-
“_task”: {
- “_fw_name”: “ScriptTask”,
-
"script": [
- "echo \"ending correl_S2 workflow\""
],
