The lethargic Canaries were made to pay after spurning chances from Leon Barnett, who crashed a header off the post, and half-time substitute Grant Holt, who had a point-blank header saved by Mark Tyler.
The decisive moment came in the 80th minute when JJ O'Donnell crossed from the left to find Rendell, who slid in to find the net and earn Blue Square Premier Luton a place in the fifth round of the FA Cup - emulating Sutton, who beat Coventry in 1989.
Norwich were booed off and will be keen to forget their display, with immediate attentions turning to Wednesday's Barclays Premier League meeting with Tottenham.
For Luton, who last beat top-flight opposition in 1994, the match will live long in the memory.
The 1959 finalists and 1994 semi-finalists were seeking to revive former glories in their first meeting with Norwich in six years - a match which came as the Hatters were tumbling down the divisions and the Canaries were preparing for a revival.
Six changes were made by Norwich, with Declan Rudd, Andrew Surman, David Fox, Elliott Bennett, Simeon Jackson and Harry Kane given a chance to impress.
Luton, 85 places lower in the football pyramid than their hosts, named 10 of the players which beat Wolves in the third round.
Rather than take the initiative, Norwich were drawn into a scrappy battle as Paul Buckle's Luton set about their task well.
Two early corners were forced by the visitors but did not beat the near post.
Centre-backs Ryan Bennett and Barnett were again unflustered as Luton attacked down the left, but further forward the Canaries were toiling, disjointed and harassed as they tried to craft a way through.
A poor David Fox free-kick hit his team-mate Ryan Bennett before a chance fell for Jackson as Elliott Bennett lofted a ball into right channel.
Jackson out-paced Janos Kovacs but his weak shot was saved by Tyler.
The rebound fell for Surman and was blocked by Ronnie Henry.
Surman then attempted to create and pulled a ball back for Kane, whose scuffed effort was blocked by left-back Greg Taylor.
Kane forced a corner from which Norwich came close to the opening goal.
An initial header by Russell Martin was met by Barnett, who bulldozed in to meet the ball above Tyler, but it crashed off the post and was scrambled clear by Alex Lawless.
Having so nearly gone ahead, the hosts were frantically defending a Luton counter-attack and referee Andre Marriner was unmoved when Andre Gray went down in the box under a challenge from Jonny Howson.
Martin hopefully pushed a shot goalwards from the right side of the area in the final move of the half, but it drifted wide.
Kane was replaced at half-time for Grant Holt, who took the captain's armband from Martin and provided a presence up front Norwich had lacked in the opening 45 minutes.
Gray shot tamely as Rudd was called to make a save for the first time before Holt was denied after 55 minutes.
Surman found Jackson who clipped a cross to the back post where Holt met it well, only to find Tyler well positioned to save.
Martin's driven cross was diverted over from under his crossbar by Kovacs and the resulting Fox corner was just too high for Holt.
Tyler saved a curling Surman shot low to his right either side of two efforts from Luton's Jon Shaw, the first blocked by Barnett and the second going narrowly over.
Gray hooked an effort over after the Canaries defence erred under a bouncing ball as the match entered the final 20 minutes.
Wes Hoolahan and Anthony Pilkington replaced Surman and Elliott Bennett with 17 minutes remaining as Norwich sought a cutting edge which never came.
Pilkington headed over Javier Garrido's cross from the left before Norwich were caught napping at the other end and nearly punished.
Lawless latched on to a prod forward from substitute Rendell, but could only steer the ball narrowly wide.
Moments later the Hatters took the lead with a devastating move down the left as three substitutes combined.
Stuart Fleetwood freed O'Donnell down the left, he reached the byline and cut the ball across and Rendell nipped in front of Ryan Bennett to direct the ball into the top corner.
Norwich never looked like finding an equaliser and hopeful shouts for a penalty when the ball bounced off Lathaniel Rowe-Turner's thigh and onto his arm were in vain as Luton claimed a famous win.