The 33-year-old Irishman, who captained the Lions on their last tour to South Africa in 2009, was ruled out of the rest of the series after suffering a broken arm in the tourists' bruising 23-21 victory in the first Test in Brisbane last Saturday.
The injury will be a particular blow to the Munster man, who came through a long period on the sidelines with a back injury before returning to action in April to secure his place on a third Lions tour.
"He's a massive loss," Rowntree told reporters on Monday. "He's such a big influence on the group, playing exceptionally well.
"But you've got to get on with these things, that's why we bought so many players ... we'd like him to (stay) because of his influence he has around the group."
O'Connell partnered Wales lock Alun-Wyn Jones in the starting lineup at Lang Park but would be replaced in the second Test by Parling, who came off the bench in the first Test, Rowntree confirmed.
"We're not going to try and lie to you," the forwards coach said. "Geoff is our next lineout caller as such and he's been doing very well coming off the bench and he's taken his chance when he's started for us as well."
The Lions camp earlier signalled their intention by withdrawing Parling from Tuesday's tour match against the Melbourne Rebels.
Parling was to have led a 'dirt-trackers' side against the Super Rugby team at Melbourne Rectangular Stadium, but will be replaced in the starting lineup by Ian Evans for the last non-Test tour match.
Towering Scotland lock Richie Gray will line up alongside the Welshman for the match at Melbourne Rectangular Stadium.
Wales flanker Dan Lydiate replaces Parling as captain, while England specialist blindside Tom Croft comes into the side to take a spot on the bench after starting against the Wallabies.
"Obviously over the moon," Lydiate told reporters of his promotion to captain for the Rebels clash.
"When Gats (coach Warren Gatland) asked me, I didn't hesitate to say yes. Just looking around that team huddle today and to represent these boys as a captain, I'm just chuffed and really excited for tomorrow."
O'Connell's injury capped a quick turnaround in the fortunes for both sides, with Australia captain and lock James Horwill cleared to play in the second Test late on Sunday after a disciplinary hearing dismissed a charge of trampling on Lions lock Wyn Jones.
Wyn Jones required stitches in his eye and Gatland complained about the incident after the match.
Rowntree betrayed his disdain with the judicial decision by sighing heavily. He steadfastly refused to question it, however.
"It's happened, we move on," he said tersely.
"I've got enough to worry about it in terms of looking forward to this week's Test and change of personnel and the game tomorrow night, so we just accept that decision and we get on with it."
The Lions also have an injury query over prop Alex Corbisiero and called up Ireland's Tom Court as cover for the Englishman, who sustained a calf injury in the first Test.
Corbisiero's availability would be confirmed over the next 24-48 hours, Rowntree said.
"I think he stepped up the challenge very well," Rowntree said of the loosehead, who was called up the squad as a replacement after injuries to Cian Healy and Gethin Jenkins, and edged compatriot Mako Vunipola for the starting role in the Brisbane Test.
"It would be a big blow (to lose him) but I think it would be another Test of our strength in depth."
The Lions' injury concerns have paled in comparison to Brisbane match's toll on Australia, with Pat McCabe and Digby Ioane ruled out for the series and a third Wallabies back in Berrick Barnes sidelined for the second Test.
Revised Lions team for Tuesday's tour match against the Melbourne Rebels: 15-Rob Kearney, 14-Sean Maitland, 13-Manu Tuilagi, 12-Brad Barritt, 11-Simon Zebo, 10-Owen Farrell, 9-Conor Murray 8-Tony Faletau, 7-Sean O'Brien, 6-Dan Lydiate (capt), 5-Ian Evans, 4-Richie Gray, 3-Dan Cole, 2-Richard Hibbard, 1-Ryan Grant
Replacements: 16-Rory Best, 17-Tom Court, 18-Matt Stevens, 19-Tom Croft, 20-Justin Tipuric, 21-Ben Youngs, 22-Billy Twelvetrees, 23-Stuart Hogg