Hartley pounced for his team's only try as Saints lacked the firepower to overcome the defending French Top 14 champions.
Stephen Myler kicked the conversion and two penalties to at least ensure Saints left Stade Pierre Antoine with a losing bonus point from the 19-13 setback.
The visitors were well aware of the challenge they faced having been drawn in the same pool as their opponents for a fourth year in a row.
It was bitterly disappointing for Northampton, who came into Europe full of confidence and ambition from the Aviva Premiership.
In contrast, Castres were struggling for form lying in 11th place in the French Top 14.
Saints dominated during the first 25 minutes of the game, getting all the possession they wanted but not being able to break a strong and aggressive French defence line.
And they gifted Castres their first try when Romain Martial ran back an interception almost the full length of the field after 11 minutes.
Despite having all the ammunition they could ask for and Courtney Lawes dominating at the lineout, with two steals in the first 15 minutes, Northampton's flat attacking line could find no solution. Only Myler was able to get on the board with the boot after 31 minutes.
The French side then started to show more, with great efficiency in their kicking game.
They fed on their opponents' mistakes and Rory Kockott was able to add a penalty two minutes before half-time and Ben Foden was sent to the bin in the last period of play.
Kockott increased the French champions' lead four minutes into the second half with a another kick to goal.
Northampton then had to show great resilience to get back in the game in the second half, which started in similar fashion. However, after a series of rolling mauls and lineouts, Hartley was able to take advantage of a Castres fumble and score. Myler converted and Saints were within three points.
Kockott immediately added three points to his personal tally after a penalty given away at the breakdown by Christian Day.
Both teams struggled to inject rhythm and pace into the game after the hour mark, Saints making too many mistakes and poor decisions to convert their territorial domination into points.
Myler and Kockott traded further kicks before Castres' South African scrum-half opened some breathing space with another kick eight minutes from time.
That set up a thrilling climax as Northampton threw everything into salvaging a winning try, yet despite sustained pressure deep inside Castres territory, Saints came up short.
Castres move on to take on Leinster in Dublin next week.