Kaushalya blasted five fours and three sixes en route to 56 off 41 balls to pull the underdogs back from the brink after they had lost quick wickets midway through their run-chase in pursuit of 239.
The big-hitting all-rounder fell in the last over and with the scores tied and Sri Lanka nine wickets down, it was left to tail-ender Dilani Manodara to hammer a six off the final ball to seal a memorable victory.
Earlier, Jenny Gunn scored 52 as England recovered from a shaky start to post 238 for eight after losing the toss.
Chamani Seneviratna struck twice in her first four overs as England slipped to 29 for three, before Heather Knight (38) and Arran Brindle (31) added 64 for the fourth wicket.
When they fell in quick succession, Gunn teamed up with 19-year-old debutant Amy Jones (41) in a stand of 83.
As the pair grew in confidence, Gunn hit back-to-back boundaries off Sharina Ravikumar and Jones, impressively taking her chance with Sarah Taylor out injured, plundered three fours in four balls off Siriwardene.
Gunn and Taylor fell in successive overs but it did not stop the acceleration, Katherine Brunt hitting the only six of the innings and Danielle Hazell plundering 19 off eight balls as England added 62 in the last five overs.
Chamani Seneviratna was the pick of the Sri Lankan attack, finishing with two for 35 off eight overs.