The hosts dominated but Fuerth, beaten 3-0 at home by Bayern Munich on their Bundesliga debut last week, snatched victory thanks to a 67th-minute goal by Felix Klaus.
Klaus, like several of his team mates, had never played in the top division before this season.
Ilir Azemi teed him up and Klaus scored with a dipping left-foot strike that went over the unsighted Christian Wetklo for Fuerth's first goal in the Bundesliga.
Fuerth took part in the last edition of the old German first division in 1962/63 but were not among the 16 clubs invited to take part in the inaugural Bundesliga the following season and had never returned to the top-flight.
Mainz have one point from two games.