Too expensive computational cost for the large basis set

[nesquik91]

Hello!

I'm trying to use DerivOnce module but found its computational cost grows extremely large w.r.t. the basis set size.
In the case of NaCl (2.5 Ang.), the computational cost for aug-cc-pvtz basis set is quite reasonable but, when I use the aug-cc-pvqz, it never ends..
Any tips or ideas for this? (Seems like, more than 120 basis set are suffered from this problem.)
Below is what I tried.

Thank you very much!

from pyscf import gto, dft, lib
from pyxdh.DerivOnce import GradXDH, DipoleXDH, DipoleMP2
import numpy as np
np.set_printoptions(7, suppress=True, linewidth=120)

lib.num_threads(4)

mol = gto.M(
verbose = 4,
atom = '''
na 0 0 0
cl 0 0 2.5
''',
charge = 0,
spin = 0,
basis = 'aug-cc-pvqz',
max_memory = 10000,
)
mol.build()

grids = dft.Grids(mol)
grids.atom_grid = (75, 302)

scf_eng = dft.RKS(mol)
scf_eng.xc = 'b3lypg'
scf_eng.grids = grids

nc_eng = dft.RKS(mol)
nc_eng.xc = '0.8033HF - 0.0140LDA + 0.2107B88, 0.6789LYP'
nc_eng.grids = grids
config = {
'scf_eng': scf_eng,
'nc_eng': nc_eng,
'cc' : 0.3211,
'cphf_tol': 1e-12
}

dip = DipoleXDH(config)
print (dip.E_1*2.541765)

[ajz34]

This problem could be resolved by using this statement in bash then run python script:

$ export MAXMEM=6

Value larger than 6 (GB) should be also okay. This value should be set to even larger if using more basis functions.

---

Following is some explanation.
Any value smaller than 6 (GB) could make program stuck at ERI AO->MO transformation at O(N^8) complexity (instead of O(N^5)), since storing ERI AO or ERI MO tensor could consume up 5.94 GB memory if aug-cc-pVQZ:

Py_xDH/pyxdh/DerivOnce/deriv_once_scf.py

Line 515 in bd4940c

return np.einsum("uvkl, up, vq, kr, ls -> pqrs", self.eri0_ao, self.C, self.C, self.C, self.C)

Although this environment is intended in pyxdh, actually it introduces some inconvenience ...
It defines how much memory space left for np.einsum to store intermediate tensor:

Py_xDH/pyxdh/DerivOnce/deriv_once_scf.py

Line 15 in bd4940c

np.einsum = partial(np.einsum, optimize=["greedy", 1024 ** 3 * MAXMEM / 8])

If memory is set to be too small, np.einsum just make naive tensor contraction without parallel.

I have to say that this program is not optimized for CPU or memory efficiency, especially memory 😥
Hope you are not running out of memory if you make even larger-scale calculation.

[nesquik91]

Thank you very much for the kind and detailed explanation! Now it runs well :D