LAMMPS from scratch · 20 min
Running in parallel
LAMMPS runs on many processors by cutting the box into bricks, one per processor. You predict by hand which grid of bricks LAMMPS chooses, how many atoms and ghost atoms each processor holds, and why five processors get uneven shares, and check every number against real mpirun runs. Then you read the speed-up and the cost of communication from the timing table.
- Domain decomposition: each processor owns a brick and its atoms
- How LAMMPS chooses the processor grid (smallest surface)
- Atoms and ghosts per processor, predicted plane by plane
- Speed-up, efficiency, and why more processors can be slower
1 · Understand: one box, many bricks
Run mpirun -np 4 lmp -in in.melt and four copies of LAMMPS start. Each one gets a brick of the box (src/domain.cpp:371: sublo = boxlo + xprd·xsplit), stores only the atoms inside it plus ghosts around it (page B.6), and computes their forces. Every step they talk through MPI (src/verlet.cpp:264):
| When | Call | What moves between processors |
|---|---|---|
| every step | comm->forward_comm() (src/verlet.cpp:268) | new positions of the atoms other processors hold as ghosts |
| every step | comm->reverse_comm() (src/verlet.cpp:341) | forces computed on ghosts, sent back to their owners (newton on) |
| when the neighbour list is rebuilt | comm->exchange() (src/verlet.cpp:284) | atoms that left a brick, handed to the new owner |
| same | comm->borders() (src/verlet.cpp:286) | the new list of ghosts, as on page B.6 |
The force loop itself is the same code as in serial: each processor just loops over fewer atoms. The browser runs one processor only, so the parallel numbers on this page were measured with real mpirun on a 2-core Linux computer (LAMMPS 22 Jul 2025, MPICH 5.0.2) and are matched by hand below.
2 · Predict the processor grid
For P processors LAMMPS tries every px × py × pz = P and keeps the one where the bricks have the smallest surface, because the surface is where ghosts live and communication happens (src/procmap.cpp:875). For a cube of side L the surface per brick, divided by L², is 1/(px·py) + 1/(px·pz) + 1/(py·pz). A tie keeps the first one found (surf < bestsurf, src/procmap.cpp:878).
3 · Match: atoms and ghosts on each processor
At step 0 the atoms sit on 20 planes per direction, 0.8397981 apart, 200 atoms per plane. Along each split direction, a brick owns the planes between its two walls; its ghosts are the planes within the communication cutoff (2.5 + 0.3 = 2.8) beyond each wall, taken from the neighbouring bricks (or periodic images). Count the FCC sites in the grown brick, subtract the owned atoms: that is Nghost (page B.6).
4 · Observe: is it faster?
32 000 atoms (20 cells per side), 500 steps, on the same 2-core computer:
| P | loop time (s) | speed-up = t₁/t_P | efficiency = speed-up/P | Pair % | Comm % |
|---|---|---|---|---|---|
| 1 | 6.10393 | 1.00 | 100 % | 80.58 | 1.26 |
| 2 | 3.16732 | 1.93 | 96 % | 76.05 | 7.31 |
| 4 | 4.01285 | 1.52 | 38 % | 41.52 | 37.66 |
- 2 processors: 6.104 / 3.167 = 1.93 times faster, 96 % efficient. Comm grows from 1.3 % to 7.3 % of the time: the bricks now have faces that need messages.
- 4 processors on 2 cores: slower than 2. Two processes share each core, and every exchange of positions waits for the process that is not running. Never ask for more MPI processes than physical cores.
- Ghosts grow relative to owned atoms as bricks shrink: 5841 / 4000 = 1.5 ghosts per atom with 1 processor, 1956 / 500 = 3.9 with 8. Each brick does less work but carries more surface. Below a few hundred atoms per processor, communication starts to dominate.
On your own computer
mpirun -np 4 lmp -in in.melt # LAMMPS picks the grid # force a grid in the script, before create_box: processors 4 1 1
The log prints the grid (1 by 2 by 2 MPI processor grid), and Nlocal and Nghost with ave, max and min over the processors. A large gap between max and min means uneven work: the slowest processor sets the speed (the balance command moves the walls to even it out; page F.5 covers performance).
Your turn
1. Which grid does LAMMPS choose for 6 processors, and what is its surface?
2. With 3 processors, why do two get 1400 atoms and one 1200?
3. You have a 32-core node and 4000 atoms. Is mpirun -np 32 a good idea?