pair_style hybrid/overlay is purely additive; it does not do splining, or rather splicing, of two potentials as a function of radial distance. This is not something that can be done in a general way in LAMMPS, because the splicing function must be carefully constructed to match the forms of the inner and outer functions. One specific case where this has been done is for Tersoff and ZBL:
It might be possible to code up a version of hybrid/overlay that splices together two strictly twobody pair styles using the same 1/(1+exp(-k(r-rc)) splicing function. Making that work for manybody potentials like EAM would still present some complications.
In order to address your specific problem, simpler solutions are:
- Eliminate the discontinuity in the EAM potential itself.
- Prevent the system from access the “very small separations” perhaps by using ZBL and pair hybrid/overlay to raise the energy of the system near the discontinuity.