Hi,
what is the most memory saving way to pass a numpy array between fireworks? I am running into the 16mb limit of MongoDB when trying to update a spec with a large matrix.
Is there any way to circumvent that problem? Is it better to pass the information through files in that case?
Cheers
Marc
Hi Marc,
If your matrices are sparse, you might look into the matrices in scipy.sparse, which maybe are a more memory friendly way to hold big matrices.
Otherwise, I’d probably recommend just using files to hold them, and having the spec entry be the path to the file. Your Firetask should then load the file from this path.
A third option is storing your matrices in GridFS with mongo. Your Firetask could then look up the GridFS entry and get the matrices from that.
Thanks,
Alex
On Fri, Mar 1, 2019 at 8:15 AM Marc Jäger [email protected] wrote:
Hi,
what is the most memory saving way to pass a numpy array between fireworks? I am running into the 16mb limit of MongoDB when trying to update a spec with a large matrix.
Is there any way to circumvent that problem? Is it better to pass the information through files in that case?
Cheers
Marc
–
You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
For more options, visit https://groups.google.com/d/optout.