The wicketkeeper-batsman sustained the injury while batting during the one-wicket loss to the West Indies on Sunday and will take no further part in the tournament.
Virat Kohli, who skippered the side when Dhoni failed to take the field for the hosts' run chase, will continue as captain for the remainder of the series while Ambati Rayudu will take Dhoni's place in the squad.
The 27-year-old could now be in line for an international debut against either Sri Lanka or the West Indies this week.
"Mahendra Singh Dhoni has been ruled out of the ongoing tri-series in the West Indies, due to a hamstring injury," a statement from the BCCI read.
"The All-India Senior Selection Committee has picked Ambati Rayudu as his replacement in the squad.
"Virat Kohli will lead India in the remainder of the tri-series."