Oxford made five changes to the team that went down 2-0 at home to Southend, and the youngsters brought in helped them to turn in an improved performance.
But Pompey grew in stature as the game went on. Sonny Bradley shot straight at Clarke, and John Marquis volleyed over from distance.
Agyemang had only been on the pitch for five minutes when struck a brilliant 65th minute shot with the outside of his right foot from the edge of the area to beat Ryan Clarke.
Clarke pulled off fine stops from Agyemang and Marquis but he was beaten again the 82nd minute, Marquis headed in Yassin Moutaouakil's right-wing cross.
Constable pulled a goal back from close range with a minute left but it was too late as Guy Whittingham's team avenged their 4-1 thumping by Oxford at Fratton Park on the opening day of the season.
Chris Kiwomya's Notts County booked their place in the northern section quarter-finals of the Johnstone's Paint Trophy after seeing off Wolves in a penalty shoot-out at Molineux.
Bjorn Sigurdarson and Kevin Foley both blazed their spot-kicks over the crossbar before James Henry was denied by goalkeeper Fabian Spiess as County triumphed 3-1 to secure a thoroughly deserved success.
The visitors were dominant throughout and really should have settled affairs well before the shootout as the tie finished goalless after 90 minutes.
But they could not find a way past goalkeeper Carl Ikeme, who continued the good form he had shown in Wanderers' 3-0 win at Colchester on Saturday.
Sheffield United were also dumped out of the competition, the Blades losing 1-0 to Hartlepool at Bramall Lane. Pools forward James Poole scored a 37th-minute volley to settle the contest.
Rotherham progressed through to the quarter-finals following a convincing 3-0 win at York.
An unfortunate own goal and an inspired substitution helped Peterborough progress courtesy of a 2-1 victory over Brentford.
Right-back Joe Rafferty headed his first goal in senior football as Rochdale upset the odds with a 1-0 victory at Port Vale.
Leyton Orient beat Coventry 4-2 on penalties at Brisbane Road after the two teams had reached 90 minutes locked in a goalless scrap.
David Mooney, Elliott Omozusi, Gary Sawyer and Lloyd James netted their spot-kicks for the home side and although Leon Clarke and Jordan Clarke were on target in reply for the Sky Blues, Franck Moussa had his effort saved by Jamie Jones while John Fleck missed the target.
Newport County battled back from two goals down to see off Crawley Town 3-2.
Emile Sinclair and Michael Jones opened up a two-goal cushion for Crawley, but Newport hit back through Chris Zebroski and then Adam Chapman, with a penalty, to level things up. Connor Essam then diverted a cross past his own goalkeeper to hand the Welsh side victory.
Matt Bloomfield marked his first start in over a year with a goal and an assist as Wycombe beat Bristol City 2-1 to reach the third round.
Four goals in seven minutes handed Dagenham a 5-2 win at Southend in the second round.
Teenager Jack Barthram scored his first career goal to help Swindon come from behind to beat League Two Plymouth 2-1 at the County Ground.
A penalty by Felipe Morais sent Stevenage through to the third round for the first time as they came from behind to beat MK Dons 2-1.
James Wesolowski scored for the first time in almost two years to help Oldham progress to the northern section quarter-finals with a 2-0 win over Preston at Deepdale.
Carlisle sealed their passage into the quarter-finals after a dramatic penalty shoot-out against Morecambe at the Globe Arena.
A local derby turned into a night of misery for Mansfield as they missed two one-on-ones in their 1-0 to Chesterfield.
Holders Crewe were dumped out after going down 4-0 to League Two Fleetwood.