"This is a personal decision on Edwin’s part," Netherlands head coach Tom de la Hunty said.
"It’s a build-up of numerous factors, including his crash in the two-man, the tragic accident in the men’s luge and external family pressures, all of which resulted in Edwin not having confidence in piloting."
The Netherlands are the second crew to withdraw from the four-man after Australia pulled out because two of their crew, Duncan Harvey and Duncan Pugh, were suffering with concussion after rolling over in the two-man.
Liechtenstein are almost certainly out too after failing to start any of the four training runs.
Van Calker said he expected some negative reaction to his decision back home but said it was the right one.
"Some say it a brave decision. Some say scared," he said at a news conference. "I have to look after my boys and can’t close my eyes to that. For me, it’s not about performing, it’s about surviving."
Steve Holcomb, the world’s top-ranked four-man bobsleigh pilot, had some sympathy for Van Calker, who crashed in the two-man competition.
"He had a couple of tough runs in the two-man and the four-man is less forgiving,” the American said after setting the fastest time in the third and fourth training runs.
"Standing at the top, if you’re not 100 percent confident, I can see why he would not want to race."
Whistler’s track came in for criticism before and after the death of Georgia’s Nodar Kumaritashvili in luge training and minor alterations were made to one corner on Monday at the request of four-man bobsleigh crews.
There were no crashes in Tuesday’s four-man training.
"This is a safe track," De la Hunty added. "This is Edwin’s personal decision and the team will support him.
"In one or two years, once this track gets more and more sleds down and the athletes become more familiar, it will be considered the same as any other track."