Saints headed north of the border needing a bonus-point victory to have any chance of going through, but Peter Horne's last-gasp try instead condemned them to a 27-20 reverse.
"We showed that when we want to we can get performances; the win at Ulster away - the only visiting team to win at Ravenhill this season - was magnificent for us," he said. "But to win these competitions you need to be more consistent. We won three games and lost three games and that isn't good enough."
He added: "We came up here to try and get to the quarter-finals of the Heineken Cup. We knew it was an outside chance, we needed four tries. We were close, but we weren't close enough."
Northampton had threatened to get the points they needed when Ben Foden crossed over late on for their third try to make it 20-20. But Horne ran the length of the field to score the match-winning try late on.
Northampton will now turn their attentions back to the Aviva Premiership, where they stand fifth, and Mallinder believes they have reason to believe they can push up the table.
"We'll take positives in terms of possession and attacking play," he said. "We said at half-time that we were in the game. We only had one try but we had the wind in our favour and the slope in our favour.
"We were confident of getting the win, and the three tries, but it wasn't to be."
Glasgow head coach Gregor Townsend said: "We've shown what we can do at times in every game.
"We went down to Northampton - who were Premiership leaders at the time - and we were 15-0 up. We were leading for 75 minutes over at Castres, so we've shown it, we just haven't had the 80 minutes and the winning performance until now."