Second half tries by inside centre Craig Wing and openside flanker Michael Broadhurst, who both qualified to play for Japan after completing residency requirements, and 13 points from the boot of fullback Ayumu Goromaru helped secure their biggest win to date.
"I'm really proud of the players," Japan coach Eddie Jones was quoted as saying by Japan's Kyodo news agency after the win at the Chichibunomiya Stadium.
"We have created history today. We are the first Japan team to beat a top 10 team in the world."
Wales, shorn of the majority of their first team who have been selected to tour Australia with the British and Irish Lions, dominated possession and territory in the opening half but struggled to turn that into points with Japan defending resolutely.
Goromaru made the most of Japan's rare forays into the opposing half by slotting two penalties, while Wales fly-half Dan Biggar kicked one, as the 'Brave Blossoms' went into the break 6-3 ahead.
Wales were quick out of the blocks in the second period and eventually found a hole in the disciplined Japanese defence after moving the ball right then left, with Biggar floating a lovely long pass wide for winger Tom Prydie to cross for a 45th minute try.
Japan, who have won just one match in seven World Cups but will host the 2019 tournament, did not drop their heads and former Australian Rugby League international Wing brushed through a tackle by Welsh prop Rhys Gill to cross for a try five minutes later.
Goromaru converted from out wide to put Japan 13-8 ahead and it got even better for the Asian Five Nations champions 10 minutes later when they crossed for a second try.
The fleet-footed Japanese continued to find holes in the Wales backline, taking them into the 22 and, after sucking in defenders, created a huge overlap to allow New Zealand-born Broadhurst to touch down in the corner.
Goromaru converted again and added a 77th minute penalty to cap a strong display after missing two kicks in the 22-18 first test defeat in Osaka last week.
"We understand the Welsh didn't have 15 of their best players here. But we played a very good game of test rugby and it's another step forward for the team," Jones added.
Wales coach Robin McBryde, standing in while Warren Gatland leads the Lions in Australia, handed a debut to flanker Josh Navidi after opting to go without a specialist openside in the first test but the move did not prevent Wales coughing up possession.
"We turned the ball over nine times in the Japan 22 and that was early on too," the former Wales hooker said. "If you don't take your opportunities, then you pay the ultimate price."