Rutherford lost his middle stump to Alasdair Evans, who was making his first-class Derbyshire debut, while Fulton was trapped lbw by Mark Footitt after the tourists were put in to bat.
The Black Caps had reached 92 for two at lunch.
New Zealand had no injury concerns going into the three-day game but left out their captain Brendon McCullum and Ross Taylor, who have both just arrived in the country after playing in the Indian Premier League.
A heavy shower delayed the start of play by 15 minutes and there was some life in the pitch for a Derbyshire attack that was without all of their front-line bowlers.
But Evans troubled both openers and deserved the breakthrough which arrived in the 10th over of a breezy morning when Rutherford played back but was beaten by late movement and bowled for 13.
Fulton was starting to get on top of the bowling and had just glanced Footitt to the fine-leg boundary when he played across the line at the paceman's next ball and was lbw for 21 to leave the tourists on 50 for two.
That brought in Martin Guptill, who played a key role in helping Derbyshire win the Division Two title last season, and he was given a rousing reception when he walked out to join Kane Williamson.
Guptill had an anxious moment when he flicked Matt Higginbotham in the air just wide of mid on when he had scored two, but after that he reeled off some fluent strokes on both sides of the wicket and was on 24 at lunch with Williamson unbeaten on 27.