Jim,
You should be able to use more than 1 CPU core per GPU with Kokkos. However, it may be slower. The reason is that Kokkos tries to run everything possible on the GPU, including fixes like the time integrator, and avoids transferring atom positions and forces between the GPU and CPU if possible. Unless your input script has many non-Kokkos fixes or a non-Kokkos pair style, virtually nothing is running on the CPU, so adding more CPUs won’t necessarily help. The exception is KSpace, where there is an experimental mode to overlap CPU and GPU computation by running KSpace + Bonds on the CPU and the Pair computation on the GPU. This won’t apply to ReaxFF though.
This in in contrast to the GPU package, which only runs the pair style and a few other computations on the GPU, and everything else is run on the CPU, so using more than 1 CPU core per GPU will often speed up the GPU package.
With the GPU package, atom positions and forces are transferred back and forth between the GPU every timestep, something that Kokkos tries to avoid if possible.
Stan
Ray and Stan,
Thanks. I don’t think the GPU package supports ReaxFF so I can’t use that. However, as Ray has pointed out, I can create KOKKOS Lammps with CUDA and OpenMP. I’ll try that.
Jim
James Kress Ph.D., President
The KressWorks® Institute
An IRS Approved 501 ©(3) Charitable, Nonprofit Corporation
“Engineering The Cure” ©
(248) 573-5499
Learn More and Donate At:
Website: http://www.kressworks.org
Facebook: https://www.facebook.com/KressWorks.Institute/
Twitter: @KressWorksFnd
Confidentiality Notice | This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, immediately contact the sender by reply e-mail and destroy all copies of the original message.
However, as Ray has pointed out, I can create KOKKOS Lammps with CUDA and OpenMP.
Jim, this probably won’t help with ReaxFF. All of the kernels in Kokkos ReaxFF are running on the GPU. If you run them on the CPU with OpenMP, they won’t benefit from GPU acceleration.
If you do want to mix both CUDA and OpenMP you would also have to explicitly specify which parts run on the CPU by using the “/kk/host” suffix. Simply compiling with both CUDA and OpenMP support isn’t enough.
Stan
“If you do want to mix both CUDA and OpenMP you would also have to explicitly specify which parts run on the CPU by using the “/kk/host” suffix.”
Is this documented somewhere? I searched through the lammps documentation and could not find this suffix.
Thanks.
Jim
James Kress Ph.D., President
The KressWorks® Institute
An IRS Approved 501 ©(3) Charitable, Nonprofit Corporation
“Engineering The Cure” ©
(248) 573-5499
Learn More and Donate At:
Website: http://www.kressworks.org
Facebook: https://www.facebook.com/KressWorks.Institute/
Twitter: @KressWorksFnd
Confidentiality Notice | This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, immediately contact the sender by reply e-mail and destroy all copies of the original message.
It isn’t yet documented–it is still somewhat experimental. Here are the basics: The suffix “/kk” is equivalent to “/kk/device”, and for Kokkos CUDA, using the “-sf kk” in the command line gives you the default CUDA version everywhere. However, if you explicitly add the “/kk/host” suffix to a specific style in your input script, you will instead get the Kokkos OpenMP CPU version for that specific style. Conversely, if you use the “-sf kk/host” at the command line and then use the “/kk” or “/kk/device” suffix for a specific style in your input script, that specific style will run on the GPU while everything else will run on the CPU in OpenMP mode.
Stan
Stan,
Thanks. I’ll give it a try.
Jim
James Kress Ph.D., President
The KressWorks® Institute
An IRS Approved 501 ©(3) Charitable, Nonprofit Corporation
“Engineering The Cure” ©
(248) 573-5499
Learn More and Donate At:
Website: http://www.kressworks.org
Facebook: https://www.facebook.com/KressWorks.Institute/
Twitter: @KressWorksFnd
Confidentiality Notice | This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, immediately contact the sender by reply e-mail and destroy all copies of the original message.
I am making lammps with kokkos using the kokkos_cuda_openmpi Makefile. Is it normal to get a continuing series of warnings like these:
Overloaded virtual function xxxxx is only partially overridden in class yyyy
?
James Kress Ph.D., President
The KressWorks® Institute
An IRS Approved 501 ©(3) Charitable, Nonprofit Corporation
“Engineering The Cure” ©
(248) 573-5499
Learn More and Donate At:
Website: http://www.kressworks.org
Facebook: https://www.facebook.com/KressWorks.Institute/
Twitter: @KressWorksFnd
Yes, that is normal. I’d like to clean that up but haven’t yet.
Stan
Stan,
Thanks. The executable seems to work fine. How does one log GPU time/ function in the log.lammps file?
BTW, I am using lammps v 21Sep2016 that I downloaded via git.
Jim
James Kress Ph.D., President
The KressWorks® Institute
An IRS Approved 501 ©(3) Charitable, Nonprofit Corporation
“Engineering The Cure” ©
(248) 573-5499
Learn More and Donate At:
Website: http://www.kressworks.org
Facebook: https://www.facebook.com/KressWorks.Institute/
Twitter: @KressWorksFnd
Confidentiality Notice | This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, immediately contact the sender by reply e-mail and destroy all copies of the original message.
Hi Stan,
Would you please give me an example command line for this? Also, how would I modify an example, e.g. in.lg, to implement what you suggested?
Thanks.
Jim
James Kress Ph.D., President
The KressWorks® Institute
An IRS Approved 501 ©(3) Charitable, Nonprofit Corporation
“Engineering The Cure” ©
(248) 573-5499
Learn More and Donate At:
Website: http://www.kressworks.org
Facebook: https://www.facebook.com/KressWorks.Institute/
Twitter: @KressWorksFnd
Confidentiality Notice | This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, immediately contact the sender by reply e-mail and destroy all copies of the original message.