BioSimCodes_6Nov03

Opportunities for Biological Consortia on HPCx Code Capabilities and PerformanceHPCx and CCP Staff http://www.ccp.ac.uk/ http://www.hpcx.ac.uk/ Welcome to the Meeting ‡ Background ± HPCx ‡ Objectives ± to consider whether there is a case to bid ‡ Agenda ± ± ± ± ± Introduction to the HPCx service Overview of Code Performance Contributed Presentations Invited Presentation Discussion HPCx/Biology Discussions 2 Royal Institution, 6th November 2003 Outline ‡ Overview of Code Capabilities and Performance ± Macromolecular simulation ‡ DL_POLY, AMBER, CHARMM, NAMD ± Localised basis molecular codes ‡ Gaussian, GAMESS-UK, NWChem ± Local basis periodic code ‡ CRYSTAL ± Plane wave periodic codes ‡ CASTEP ‡ CPMD (Alessandro Curioni talk) ‡ Note - consortium activity is not limited to these codes. HPCx/Biology Discussions 3 Royal Institution, 6th November 2003 The DL_POLY Molecular Dynamics Simulation Package Bill Smith . 000.000 atoms ‡ Full force field and molecular description ± DL_POLY_3 ‡ Domain Decomposition. HPCx/Biology Discussions 5 Royal Institution.DL_POLY Background ‡ General purpose parallel MD code ‡ Developed at Daresbury Laboratory for CCP5 1994-today ‡ Available free of charge (under licence) to University researchers worldwide ‡ DL_POLY versions: ± DL_POLY_2 ‡ Replicated Data. up to 1. 6th November 2003 . up to 30.000 atoms ‡ Full force field but no rigid body description. Adiabatic shell model.DL_POLY Force Field ‡ Intermolecular forces ± ± ± ± ± All common van de Waals potentials Sutton Chen many-body potential 3-body angle forces (SiO2) 4-body inversion forces (BO3) Tersoff potential -> Brenner ‡ Intramolecular forces ± Bonds. inversions ‡ Coulombic forces ± Ewald* & SPME (3D). angle. 6th November 2003 . Reaction field.shear field.electric field. ‡ Externally applied field ± Walled cells. dihedrals. Truncated Coulombic. etc * Not in DL_POLY_3 HPCx/Biology Discussions 6 Royal Institution. Neutral groups*. HK Ewald* (2D). 6th November 2003 .y periodic. isolated macromolecules) Cubic periodic boundaries Orthorhombic periodic boundaries Parallelepiped periodic boundaries Truncated octahedral periodic boundaries* Rhombic dodecahedral periodic boundaries* Slabs (i.e.Boundary Conditions ‡ ‡ ‡ ‡ ‡ ‡ ‡ None (e. x. z nonperiodic) HPCx/Biology Discussions 7 Royal Institution.g. 6th November 2003 .Algorithms and Ensembles Algorithms ‡ ‡ ‡ ‡ ‡ * Ensembles ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ NVE Berendsen NVT Hoover NVT Evans NVT Berendsen NPT Hoover NPT Berendsen NWT Hoover NWT Verlet leapfrog RD-SHAKE Euler-Quaternion* QSHAKE* [All combinations] Not in DL_POLY_3 HPCx/Biology Discussions 8 Royal Institution. 6th November 2003 .128x128x128) C D An alternative FFT algorithm has been designed to reduce communication costs HPCx/Biology Discussions 9 Royal Institution.Migration from Replicated to Distributed data DL_POLY-3 : Domain Decomposition ‡ Distribute atoms. forces across the nodes ± More memory efficient. can address much larger cases (105-107) ‡ Shake and short-ranges forces require only neighbour communication ± communications scale linearly with number of nodes A B ‡ Coulombic energy remains global ± strategy depends on problem and machine characteristics ± Adopt Smooth Particle Mesh Ewald scheme ‡ includes Fourier transform smoothed charge density (reciprocal space grid typically 64x64x64 . each involving communications only between blocks in a given column ± More data is transferred. An alternative FFT algorithm has been designed to reduce communication costs.Migration from Replicated to Distributed data DL_POLY-3: Coulomb Energy Evaluation ‡ ‡ Plane Block Conventional routines (e. the communications are columnwise only ‡ HPCx/Biology Discussions 10 Royal Institution. ± the 3D FFT are performed as a series of 1D FFTs. 6th November 2003 . but in far fewer messages ± Rather than all-to-all. fftw) assume plane or column distributions A global transpose of the data is required to complete the 3D FFT and additional costs are incurred reorganising the data from the natural block domain decomposition.g. 6th November 2003 .DL_POLY_2 & 3 Differences ‡ ‡ ‡ ‡ ‡ ‡ ‡ Rigid bodies not in _3 MSD not in _3 Tethered atoms not in _3 Standard Ewald not in _3 HK_Ewald not in _3 DL_POLY_2 I/O files work in _3 but NOT vice versa No multiple timestep in _3 HPCx/Biology Discussions 11 Royal Institution. Symplectic versions 2/3 DL_POLY .Distributed multipoles DL_PIMD .DL_POLY_2 Developments ‡ ‡ ‡ ‡ ‡ ‡ DL_MULTI . 6th November 2003 .Rare event simulation DL_POLY .Multiple timestep DL_POLY .F90 re-vamp HPCx/Biology Discussions 12 Royal Institution.Path integral (ionics) DL_HYPE . 300Dt) ± NaKSi2O5 . 15625 LC) ± 32-512 processors (4-64 nodes) HPCx/Biology Discussions 13 Royal Institution.disilicate glass ± SPME (1283grid)+3 body terms.DL_POLY_3 on HPCx ‡ Test case 1 (552960 atoms. 6th November 2003 . DL_POLY_3 on HPCx ‡ Test case 2 (792960 atoms. 10Dt) ± 64xGramicidin(354)+256768 H2O ± SHAKE+SPME(2563 grid).14812 LC ± 16-256 processors (2-32 nodes) HPCx/Biology Discussions 14 Royal Institution. 6th November 2003 . [email protected] ‡ Maurice Leslie DL_MULTI ± m. Graphics.shtml ± W. C.Molecular Simulation (2002). 385 HPCx/Biology Discussions 15 Royal Institution.ac. Rodger.uk ‡ Ilian Todorov DL_POLY_3 ± i. Smith.R.leslie@dl. P.ac.DL_POLY People ‡ Bill Smith DL_POLY_2 & _3 & GUI ± w.uk/msi/software/DL_POLY/index.uk ‡ Further Information: ± W. Molec.ac.M.t.ac. Smith and T. J. 136 ± http://www. 14.smith@dl. 6th November 2003 .cse.W. Forester. (1996).clrc. Yong. 28. DL_POLY V2: Replicated Data Performance Relative to the Cray T3E/1200E 12 10 8 6 4 2 0 16 32 64 128 SGI Origin 3800/R14k-500 Compaq AlphaServer SC ES45/1000 IBM SP/Regatta-H Macromolecular Simulations Bench 7: Gramicidin in water. NaCl. rigid bonds and SHAKE. 12. CHARMM. Ewald. AMBER 16 Number of CPUs Royal Institution. 500 time steps Performance Relative to the Cray T3E/1200E 7 6 5 4 3 2 1 0 16 32 64 SGI Origin 3800/R14k-500 Compaq AlphaServer SC ES45/1000 IBM SP/Regatta-H Number of CPUs Bench 4.390 atoms.000 ions. Cutoff=24Å Ionic Simulations HPCx/Biology Discussions . 75 time steps.. 6th November 2003 . 27. DL_POLY3 Macromolecular Simulations Measured Time (seconds) 661 Gramicidin in water; rigid bonds + SHAKE: Speedup 256 792,960 ions, 50 time steps 600 SGI Origin 3800/R14k-500 A Server SC ES45/1000 Speedup Linear SGI Origin 3800/R14k-500 AlphaServer SC ES45/1000 IBM SP/Regatta-H 149.5 145.0 215.8 224 192 160 IBM SP/Regatta400 370 349 273 174.1 200 186 189 140 109 114 97 128 96 64 68 77 76.7 98.0 59.1 0 32 64 128 256 32 32 64 96 128 160 192 224 256 Number of CPUs Number of CPUs HPCx/Biology Discussions 17 Royal Institution, 6th November 2003 AMBER, NAMD and Gaussian Lorna Smith and Joachim Hein AMBER ‡ AMBER (Assisted Model Building with Energy Refinement) ± A molecular dynamics program, particularly for biomolecules ± Weiner and Kollman, University of California, 1981. ‡ Current version ± AMBER7 ‡ Widely used suite of programs ± Sander, Gibbs, Roar ‡ Main program for molecular dynamics: Sander ± Basic energy minimiser and molecular dynamics ± Shared memory version ± only for SGI and Cray ± MPI version: master / slave, replicated data model HPCx/Biology Discussions 19 Royal Institution, 6th November 2003 6th November 2003 .Initial Scaling 12 10 8 Speed-up 6 4 2 0 0 16 32 48 64 80 96 112 128 144 No of Processors ‡ Factor IX protein with Ca++ ions ² 90906 atoms HPCx/Biology Discussions 20 Royal Institution.AMBER . removed multiple copies of routines.« ± Likely to be incorporated into AMBER8 ‡ We are looking at optimising the collective communications ± the reduction / scatter HPCx/Biology Discussions 21 Royal Institution. 6th November 2003 .AMBER ‡ Bob Duke ± Developed a new version of Sander on HPCx ± Originally called AMD (Amber Molecular Dynamics) ± Renamed PMEMD (Particle Mesh Ewald Molecular Dynamics) ‡ Substantial rewrite of the code ± Converted to Fortran90.Current developments . Optimisation ± PMEMD 300 250 PMEMD Sander7 Tim (s conds) 200 150 100 50 0 0 50 100 150 200 250 300 No o ro ors HPCx/Biology Discussions 22 Royal Institution. 6th November 2003 . 2. ± Theoretical and Computational Biophysics Group. 6th November 2003 .5b and 2. ‡ Versions 2.4.NAMD ‡ NAMD ± molecular dynamics code designed for high-performance simulation of large biomolecular systems. University of Illinois at Urbana-Champaign.5 available on HPCx ‡ One of the first codes to be awarded a capability incentive rating ± bronze HPCx/Biology Discussions 23 Royal Institution. 6th November 2003 .NAMD Performance ‡Benchmarks from Prof Peter Coveney ‡TCR-peptideMHC system HPCx/Biology Discussions 24 Royal Institution. 6th November 2003 .NAMD Performance HPCx/Biology Discussions 25 Royal Institution. object-oriented MD code High-performance simulation of large biomolecular systems ‡ Scales to 100¶s of processors on highend parallel platforms Speedup 512 Li ear I M / egattaaq Al haServer S45/1000 384 256 ‡ standard NAMD ApoA-I benchmark.edu/Research/namd/ ‡ ‡ Parallel.NAMD Scaling http://www. a system comprising 92. ‡ scalability improves with larger simulations speedup of 778 on 1024 CPUs of TCS-1 in a 327K particle simulation of F1-ATPase. with 12Å cutoff and PME every 4 time steps. HPCx/Biology Discussions 26 128 0 0 128 256 384 512 Number of CPUs Royal Institution.ks.uiuc.Molecular Simulation . 6th November 2003 .442 atoms. CHARMM and NAMD ‡ See: http://www.edu/brooks/Benchmarks/ ‡ Benchmark ± dihydrofolate reductase protein in an explicit water bath with cubic periodic boundary conditions. 6th November 2003 . ± 23.scripps.Performance Comparison ‡ Performance comparison between AMBER.558 atoms HPCx/Biology Discussions 27 Royal Institution. 6th November 2003 .Performance HPCx/Biology Discussions 28 Royal Institution. www.Gaussian ‡ Gaussian 03 ± Performs semi-empirical and ab initio molecular orbital calulcations.gaussian. 6th November 2003 .com ‡ Shared memory version available on HPCx ± Limited to the size of a logical partition (8 processors) ± Phase 2 upgrade will allow access to 32 processors ‡ Task farming option HPCx/Biology Discussions 29 Royal Institution. ± Gaussian Inc. CRYSTAL and CASTEP Ian Bush and Martin Plummer . Crystal ‡ ‡ ‡ ‡ ‡ Electronic structure and related properties of periodic systems All electron. DFT and Hartree-Fock Under continuous development since 1974 Distributed to over 500 sites world wide Developed jointly by Daresbury and the University of Turin HPCx/Biology Discussions 31 Royal Institution. 6th November 2003 . local Gaussian basis set. E. 6th November 2003 . RHF) ± DFT (LSDA. GGA) ± Hybrid funcs (B3LYP) ‡ Techniques ± Replicated data parallel ± Distributed data parallel ‡ ‡ ‡ Forces ± Structural optimization Direct SCF Visualisation ± AVS GUI (DLV) HPCx/Biology Discussions 32 Royal Institution. EFG classical) Fermi contact (NMR) EMD (Compton.Gaussians Properties Energy Structure Vibrations (phonons) Elastic tensor Ferroelectric polarisation Piezoelectric constants X-ray structure factors Density of States / Bands Charge/Spin Densities Magnetic Coupling Electrostatics (V.Crystal Functionality ‡ ‡ ‡ Basis Set ± LCAO . e-2e) All electron or pseudopotential Hamiltonian ± Hartree-Fock (UHF. 6th November 2003 .Benchmark Runs on Crambin ‡ Very small protein from Crambe Abyssinica .1284 atoms per unit cell Initial studies using STO3G (3948 basis functions) Improved to 6-31G * * (12354 functions) All calculations Hartree-Fock As far as we know the largest HF calculation ever converged ‡ ‡ ‡ ‡ HPCx/Biology Discussions 33 Royal Institution. 948 GTOs) ‡ ‡ 99. 6th November 2003 .Crambin .95% parallel Speed-up 128 Number of Processors 0 0 256 512 768 1024 HPCx/Biology Discussions 34 Royal Institution.194 GTOs) STO-3G (3.Parallel Performance ‡ Fit measured data to Amdahl¶s law to obtain estimate of speed up 1024 ‡ 896 Increasing the basis set size increases the scalability 768 640 About 700 speed up on 1024 processors for 6-31G * * 512 384 Takes about 3 hours instead of about 3 months 256 ‡ Linear 6-31G* (12.354 GTOs) 6-31G (7. 6th November 2003 .Results ± Electrostatic Potential ‡ Charge density isosurface coloured according to potential ‡ Useful to determine possible chemically active groups HPCx/Biology Discussions 35 Royal Institution. Hasnain (DL) we want to calculate redox potentials for rusticyanin and associated mutants HPCx/Biology Discussions 36 Royal Institution.Futures .Rusticyanin ‡ Rusticyanin (Thiobacillus Ferrooxidans) has 6284 atoms and is involved in redox processes ‡ We have just started calculations using over 33000 basis functions ‡ In collaboration with S. 6th November 2003 . chemical reactions ± molecular dynamics ‡ Method (direct minimization) ± plane wave expansion of valence electrons ± pseudopotentials for core electrons HPCx/Biology Discussions 37 Royal Institution.What is Castep? ‡ First principles (DFT) materials simulation code ± ± ± ± electronic energy geometry optimization surface interactions vibrational spectra ‡ materials under pressure. 6th November 2003 . 6th November 2003 .HPCx: biological applications ‡ Examples currently include: ± NMR of proteins ± hydroxyapatite (major component of bone) ± chemical processes following stroke ‡ Possibility of treating systems with a few hundred atoms on HPCx ‡ May be used in conjunction with classical codes (eg DL_POLY) for detailed QM treatment of µfeatures of interest¶ HPCx/Biology Discussions 38 Royal Institution. Castep 2003 HPCx performance gain Al2O3 120 atom cell. 5 k-points 000 000 6000 000 000 3000 2000 000 0 Job time an 03 Current 'Best' 0 60 2 0 320 Total number of processors HPCx/Biology Discussions 39 Royal Institution. 6th November 2003 . 2 k-points 6000 4000 2000 0000 000 6000 4000 2000 0 Job Time an 03 Current 'Best' 2 2 6 2 Total number of processors HPCx/Biology Discussions 40 Royal Institution. 6th November 2003 .Castep 2003 HPCx performance gain Al2O3 270 atom cell. V Milman. 6th November 2003 .uk) HPCx/Biology Discussions 41 Royal Institution. ± µEasy¶ to understand top-level code. B Montanari. K Refson. M Payne. P Lindan.HPCx: biological applications ‡ Castep (version 2) is written by: ± M Segall. ‡ Castep is fully maintained and supported on HPCx ‡ Castep is distributed by Accelrys Ltd ‡ Castep is licensed free to UK academics by the UKCP consortium (contact ukcp@dl. P Hasnip.ac. M Probert C Pickard. S Clark. NWChem and GAMESS-UK Paul Sherwood .CHARMM. hared data tru ture ‡ Tools ± Global arrays: ‡ portable distributed data tool: Physi ally distributed data ‡ Parallel programming model ± non-uniform memory access. « ‡ Used by CCP1 groups (e. ‡ guaranteed orthogonality of eigenvectors ± Wide range of parallel functionality essential for HPCx HPCx/Biology Discussions 43 Royal Institution. 6th November 2003 .NWChem ‡ Objectives ± Highly efficient and portable MPP computational chemistry package ± Distributed Data .Scalable with respect to chemical system size as well as MPP hardware size ± Extensible Architecture ‡ Object-oriented design ± abstraction. RTDB. MA. data hiding. APIs Single. MOLPRO) ± PeIGS: ‡ parallel eigensolver. global arrays ‡ Infrastructure ± GA.g. handles. Parallel I/O. 6th November 2003 . P the density matrix and F the Fock or Hamiltonian matrix FVW Integrals V XC V Coul V 1e If Converged MOAO PQR Sequential guess orbitals ga_dgemm PeIGS FVW Integrals VXC VCoul V 1e If Converged Distributed Data HPCx/Biology Discussions 44 Royal Institution.Distributed Data SCF MOAO PQR dgemm guess orbitals Sequential Eigensol ver Pictorial representation of the iterative SCF process in (i) a sequential process. and (ii) a distributed data parallel process: MOAO represents the molecular orbitals. ± Coupled cluster. ± Semi-direct and RI-based MP2 calculations for RHF and UHF wave functions using up to 3.NWChem NWChem Capabilities (Direct. ± Classical molecular dynamics and free energy simulations with the forces obtainable from a variety of sources HPCx/Biology Discussions 45 Royal Institution. numerical 1st and 2nd derivatives of the CC energy. CCSD and CCSD(T) using up to 3. 6th November 2003 . analytic 1st derivatives and numerical 2nd derivatives. analytic 1st and numerical 2nd derivatives. analytic 1st and 2nd derivatives.000 basis functions.000 basis functions. ± CASSCF. using up to 10.000 basis functions. UHF. analytic 1st and 2nd derivatives. ± DFT with a wide variety of local and non-local XC potentials. ROHF using up to 10. Semi-direct and conventional): ± RHF.000 basis functions. 6th November 2003 . Si DGAUSSDGAUSS-A2 .O.H Si26O37H36 1199/2818 ‡ NWChem & GAMESS-UK GAMESSBoth codes use auxiliary fitting basis for coulomb energy.O.Zeolite Fragments ‡ DFT Calculations with Coulomb Si8O7H18 347/832 Fitting Basis (Godbout et al. with 3 centre 2 electron integrals held in core.Case Studies .) Si8O25H18 617/1444 DZVP .H Fitting Basis: DGAUSSDGAUSS-A1 . Si DZVP2 . Si28O67H30 1687/3928 HPCx/Biology Discussions 46 Royal Institution. NWChem Si26O37H36 Measured Time (seconds) 2500 2000 1500 1147 2388 2414 1199/2818 Si28O67H30 Measured Time (seconds) 1687/3928 CS7 AMD K7/1000 + SCI CS9 P4/2000 + Myri e 2k CS2 QSNe Al a Clus er/667 S I ri i 3800/R14k-500 IBM SP/ 690 Al aServer SC ES45/1000 1271 6000 4682 5507 CS7 AMD K7/1000 + SCI CS9 P4/2000 + Myri e 2k CS2 QSNe Al a Clus er S I ri i 3800/R14k-500 IBM SP/ 690 Al aServer SC ES45/1000 4000 3008 2424 2053 3050 1000 500 0 32 951 907 2000 517 502 490 404 390 2351 1580 1617 1504 1418 1182 880 834 611 303 0 Number of CPUs 64 128 32 64 128 Number of CPUs 47 Royal Institution.DFT Coulomb Fit . 6th November 2003 HPCx/Biology Discussions . 184 seconds 128 CPUs= 3.Memory-driven Approaches: NWChem . In core = 100% 2eHPCx/Biology Discussions .54 X 10 9 ‡ % 3c 2e-ints.242 seconds 128 CPUs= 3.00 X 10 12 2e‡ Schwarz screening = 6. 6th November 2003 ‡ 3-centre 2e-integrals = 1.DFT (LDA): Performance on the IBM SP/p690 Zeolite ZSM-5 ZSM‡ DZVP Basis (DZV_A2) and Dgauss A1_DFT Fitting basis: AO basis: CD basis: ‡ IBM SP/p690) 3554 12713 Wall time (13 SCF iterations): 64 CPUs = 9.966 seconds MIPS R14k-500 CPUs (Teras) R14kWall time (13 SCF iterations): 64 CPUs = 5.451 seconds 48 Royal Institution. K. and ICI plc.dl. Sherwood. Other major sources that have assisted in the on-going development and support of the program include various academic funding agencies in the Netherlands. M. Handy. Dupuis. together with a variety of techniques for post Hartree Fock calculations. P. and has been extensively modified and enhanced over the past decade.D.and DFT-gradient calculations. Harrison.J. Knowles. Hillier. R. Schoffel & P.J.uk/CFS Guest. V. 6th November 2003 . with contributions from R. A. van Dam. R.J. ± This work has included contributions from numerous authors‚.H.P.ac. Amos. J. A.J.C. H.GAMESS-UK ‡ GAMESS-UK is the general purpose ab initio molecular electronic structure program for performing SCF-.F. Tozer.J. N. and has been conducted largely at the CCLRC Daresbury Laboratory. obtained from Michel Dupuis in 1981 (then at the National Resource for Computational Chemistry. Bonacic-Koutecky van Lenthe. I. MCSCF. V.J. Additional information on the code may be found from links at: http://www. ± The program is derived from the original GAMESS code. Kendrick. HPCx/Biology Discussions 49 Royal Institution. under the auspices of the UK s Collaborative Computational Project No. ‡ ‚ M. NRCC). J. W. Stone and D.. Rendell. von Niessen. Buenker.H. Saunders.R. 1 (CCP1). MP3 Energies MCSCF and CASSCF Energies. B97. BLYP. gradients and frequencies. Multi-reference MP2. gradients and numerical 2nd derivatives MR-DCI Energies. conventional and direct including Dunlap fit ‡ B3LYP. direct SCF-Frequencies: numerical and analytic 2nd derivatives Restricted. unrestricted open shell SCF and GVB. in-core. Valence bond (Turtle) 50 Royal Institution.GAMESS-UK features 1. RPA gradients Full-CI Energies Green s functions calculations of IPs. ± Hartree Fock: ‡ ‡ ‡ ‡ Segmented/ GC + spherical harmonic basis sets SCF-Energies and Gradients: conventional. ± Density Functional Theory ‡ Energies + gradients. HCTH. 6th November 2003 HPCx/Biology Discussions . properties and transition moments (semi-direct module) CCSD and CCSD(T) Energies RPA (direct) and MCLR excitation energies / oscillator strengths. B97-1. BP86. FT97 & LDA functionals ‡ Numerical 2nd derivatives (analytic implementation in testing) ± Electron Correlation: ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ MP2 energies. Multipole Moments Natural Bond Orbital (NBO) + Bader Analysis IR and Raman Intensities. Polarizabilities & Hyperpolarizabilities Solvation and Embedding Effects (DRF) Relativistic Effects (ZORA) ± Pseudopotentials: ‡ Local and non-local ECPs.GAMESS-UK features 2. Morokuma Analysis. gradients. frequencies MP2 energies and gradients Direct RPA 51 Royal Institution. AM1. ± ± ± ± Visualisation: tools include CCP1 GUI Hybrid QM/MM (ChemShell + CHARMM QM/MM) Semi-empirical : MNDO. Electrostatic Potential-Derived Charges Distributed Multipole Analysis. 6th November 2003 HPCx/Biology Discussions . ± Molecular Properties: ‡ ‡ ‡ ‡ ‡ ‡ Mulliken and Lowdin population analysis. and PM3 hamiltonians Parallel Capabilities: ‡ ‡ ‡ ‡ MPP and SMP implementations (GA tools) SCF/DFT energies. 6th November 2003 .Parallel Implementation of GAMESS-UK ‡ Extensive use of Global Array (GA) Tools and Parallel Linear Algebra from NWChem Project (EMSL) ‡ SCF and DFT ± ± ± ± Replicated data. Inversion of 2c-2e matrix) ‡ SCF and DFT second derivatives ± Distribution of <vvoo> and <vovo> integrals via GAs ‡ MP2 gradients ± Distribution of <vvoo> and <vovo> integrals via Gas ‡ Direct RPA Excited States ± Replicated data with parallelisation of direct integral evaluation HPCx/Biology Discussions 52 Royal Institution. but « GA Tools for caching of I/O for restart and checkpoint files Storage of 2-centre 2-e integrals in DFT Jfit Linear Algebra (via PeIGs. DIIS/MMOs. GAMESS-UK: DFT Calculations Elapsed Time (seconds) 5000 4731 Speedup 128 ¥ ¤ 3750 2614 2504 2838 96 2500 1681 1584 1867 1281 1100 64 1250 32 32 £ ¢ 64 0 32 64 128 2 Number of CPUs ¡ Cyclosporin (DFT B3LYP): Basis: 6-31G* 6(1855 GTOs) 6 Linear SGI Origin 3800/ 14k 500 I SP/ egatta H lphaSer er ES45/1000 6 32 HPCx/Biology Discussions 53 £ ¢ ¡ 32 6 ¥ © ¤   ¨ §¦   SGI Origin 3800/ 14k 500 I SP/ egatta H lphaSer er ES45/1000 Linear SGI Origin 3800/ 14k 500 I SP/ egatta H lphaSer er ES45/1000 81 Valinomycin (DFT HCTH): Basis: DZVP2_A2 (Dgauss) (1620 GTOs) 96 128 12000 11053 G Or gi 3800 R14k 500 B P Regatta H A a erver ES45 1000 104 102 92 9000 5557 5823 5846 6000 3109 3081 3388 1940 1825 3000 0 6 2 32 Number of CPUs 64 128 Royal Institution. 6th November 2003 . HP/Compaq SC ES45/1000 and SGI O3800 (C6H4(CF3))2: Basis 6-31G (196 GTO) Elapsed Time (seconds) 3000 2500 2000 1500 989 1614 CS14 III/1000 + y in t 1 CPU) SGI i in3800/ 14k-500 .DFT Analytic 2nd Derivatives Performance IBM SP/p690. 6th November 2003 .CTH Alph Server S45/1000 .3LYP Alph Server S45/1000 . Fock matrices).3LYP I SP/p690 .3LYP I SP/p690 .HCTH 1937 1073 743 569 1175 1000 500 0 32 HPCx/Biology Discussions 470 354 307 Terms from MO 2e-integrals in GA storage (CPHF & pert. Calculation dominated by CPHF: 64 CPUs 128 54 Royal Institution. 4 (1983) 187-217 ‡ Parallel Benchmark .CHARMM ‡ CHARMM (Chemistry at HARvard Macromolecular Mechanics) is a general purpose molecular mechanics. 6th November 2003 . lipids etc.) ‡ Supports energy minimisation and MD approaches using a classical parameterised force field. Chem. nucleic acids. HPCx/Biology Discussions 55 Royal Institution.MD Calculation of Carboxy Myoglobin (MbCO) with 3830 Water Molecules. molecular dynamics and vibrational analysis package for modelling and simulation of the structure and behaviour of macromolecular systems (proteins. ‡ J. ‡ QM/MM model for study of reacting species ± incorporate the QM energy as part of the system into the force field ± coupling between GAMESS-UK (QM) and CHARMM. Comp. Parallel CHARMM Benchmark Benchmark MD Calculation of Carboxy Myoglobin (MbCO) with 3830 Water Molecules: 14026 atoms. 6th November 2003 . 12- 150 114 104 100 83 CS2 QSN t Alpha Clu t r/667 CS9 4/2000 Myrin t 2k CS12 4/2400 Gbit Eth r CS10 4/2666 Myrin t SGI ri in 3800/ 14k-500 AlphaServer SC ES45/1000 I M SP/p690 89 73 64 59 66 61 54 62 51 64 64 56 48 40 32 Lin r C 1 III/450 F /LAM C 2 Q N t Alph lu t r/667 CS10 4/2666 Myrin t Cray 3E/1200E SGI ri in 3800/ 14k-500 72 50 69 46 24 44 37 16 8 0 16 HPCx/Biology Discussions 8 16 24 32 40 48 56 64 32 64 56 Royal Institution. 1000 steps (1 ps). 12-14 A shift. variants of the Reference System Propagation Algorithm (RESPA) But to date length / time scales only differ by ~ 1 order of magnitude For an example of an effort to link the atomistic and meso-scales see RealityGrid: http://www.realitygrid.g.Multiple Time and Length Scales ‡ QM/MM .g.first step towards multiple length scales ± QM treatment of the active site ‡ reacting centre ‡ problem structures (e.org/information.g. spectroscopy) ± Classical MM treatment of environment ‡ enzyme structure. 6th November 2003 .html HPCx/Biology Discussions 57 Royal Institution. explicit and/or dielectric solvent models ‡ Multiple time scale algorithms for MD ± Recompute different parts of energy expression at different intervals e. zeolite framework. transition metal centres) ‡ excited state processes (e. 6th November 2003 0 8 16 32 Number of CPUs 64 HPCx/Biology Discussions .QM/MM Applications Triosephosphate isomerase TI ) T 128 (IBM SP/Regatta-H) = 143 secs SP/RegattaMeasured Time (seconds) 1600 1487 CS9 P4/2000 + M ri et 2k SG Origi 3800/R14k-500 A haServer SC ES45/1000 1200 714 797 IBM SP/Regatta-H 778 glycolysis. catalytic intercon ersion of DHAP to GAP ‡ Demonstration case within UASI Partners UZH. TURBOMOLE ± CHARMM force-field. DL_POLY Royal Institution.180 atoms + 2 link) 58 ± include residues with possible proton donor/acceptor roles ± GAMESS-UK. MNDO. implemented in CHARMM. and BASF) ‡ Central rea tion in reaction 800 1030 ‡ QM region 35 atoms (DFT BLYP) 419 431 428 274 246 308 196 257 213 170 540 400 ‡ MM region (4. subject to path constraints.clrc.ac.simultaneously E optimise a series of points defining a reaction path or conformational change. P0 P1 P2 P3 P4 P32 P33 P34 P35 P36






‡ Suitable for QM and QM/MM Hamiltonians Reaction Co-ordinate Co‡ Parallelisation per point ‡ Communication is limited to Collaboration with Bernie Brooks (NIH) adjacent points on the path http://www. 6th November 2003 .Monte Carlo exchange of ± ± configurations between an ensemble of replicas at different temperatures Combinatorial approach to ligand binding Replica path method .uk/qcg/chmguk global sum of energy function HPCx/Biology Discussions 59 Royal Institution.Sampling Methods ± Multiple independent simulations ± Replica exchange .cse. CRYSTAL). pathway optimisation.g.g. NAMD) ± heavy CPU or memory demands (e. Monte Carlo etc) HPCx/Biology Discussions 60 Royal Institution.g. ± potential for algorithmic development to exploit 1000s of processors (e.Summary ‡ Many of the codes used by the community have quite poor scaling ‡ Best cases ± large quantum calculations (Crystal. 6th November 2003 . DFT etc) ± very large MD simulations (NAMD) ‡ For credible consortium bid we need to focus on applications which have ± acceptable scaling now (perhaps involving migration to new codes (e.