The victory gave New Zealand a 1-0 lead in the three-match series after the first match on Sunday was abandoned due to rain.
After a five-hour stoppage for bad weather, the game was reduced to 23 overs a side with the Kiwis set 198 to win under the Duckworth Lewis method. They finished on 203 for six.
With 20 needed off the last over, McCullum hit left-arm spinner Rangana Herath for three sixes and a four off the final four balls.
McCullum scored 32 not out off nine balls and opener Tom Latham also played his part with a fiery 86 off 68 deliveries, for which he was named man-of-the-match.
The target looked a tough task for New Zealand after Sri Lanka had scored 138 for one in 23 overs when Nuwan Kulasekara reduced them to eight for two in the second over.
But a fifth-wicket partnership of 93 off 57 balls between Latham and Luke Ronchi, who made 49 off 26 balls, brought New Zealand back into the match.
Kulasekara returned to dismiss both men in the 21st over but New Zealand held their nerve and McCullum put the finishing touches to their chase.
Earlier, Tillakaratne Dilshan and Kumar Sangakkara shared their second century partnership in a row until bad weather intervened, Dilshan scoring an unbeaten 55 off 72 balls and Sangakkara 71 not out off 59
The third and final one-dayer will be played at Dambulla on Saturday. (Editing Tony Goodson)