Dumping data files without headers ( just data in binary form)

LAMMPS LAMMPS Mailing List Mirror
1

Hi Axel,

I am not concern at all about saving because of the headers. The problem
is that my files are really big ( 8 Gb or more), and it takes so much time
to enter the file to delete those lines.

​that is *definitely* the wrong approach and flat out the *least* smart
solution. what keeps you from just changing the code reading the data to
skip those lines. that is trivial to do. your wording seems to imply that
you manually remove those in a text editor. if that is the case, then you
seriously should talk to somebody that knows a little bit about using
stream editing with tools sed, awk, perl, python and in general reading
files from a pipe.

you definitely should consider compression as well, since the extra compute
effort for compression is usually less than the additional time spent on
waiting to write and transfer the uncompressed files.

axel.

2

Carlos,

Have you tried using the following command on the lammps files?

“awk ‘$0 !~/[A-Za-z]/ {print $0}’ starting_lammps_file > wanted_modified_file”

Don’t know if it may help get rid of the unwanted text, if that was one of your main concerns.

respectfully,

3

Thanks Alex,

I completely understand your point. Sorry for my wording. I do use stream editing with sed.

Carlos

​

4

Hi Daniel,

Yes, I do something similar. I just was wondering if there was a way of not having the need to do that.

Thanks

Carlos

5

Hi Daniel,

Yes, I do something similar. I just was wondering if there was a way of
not having the need to do that.

if you do it right, plain text processing with sed via a pipe should be
pretty low overhead. what is time consuming is writing and particularly
reading formatted files. thus it would be best to dump out you data in
binary and then use a reader code similar to that in tools/binary2txt.cpp

in general, writing out unstructured data is almost always a very bad idea
and the better solution is always to make a "reader" smarter than making a
"writer" dumber.

axel.

6

Alex,

I really like the idea you suggest. I am going to look at the binary2txt.cpp file and read it in that format.

​

Thanks again for your help.

Carlos