That all changed in 2011 when Dave Rennie succeeded Ian Foster and was given the job of turning their fortunes around.
Success under the former Wellington, Manawatu and New Zealand under-20 coach was immediate.
Rennie became the first rookie coach to win the title in 2012 and on Saturday the Chiefs could become just the fourth side to retain the crown, cementing what many pundits in New Zealand regard as the beginning of a dynasty.
That word is seldom used by the Chiefs, however.
"It's more a media thing and we don't tend to talk about it," Rennie told Reuters via telephone on Thursday. "A dynasty is something you look back on.
"You win X amount of titles and think that was a pretty special time but what we're trying to do is create an environment that people want to be part of.
"From a retention and recruitment point of view we want people to want to move here. It's all about good people really and that's what makes this group strong."
It was not always that way.
In the earliest days of Super Rugby the Chiefs had an uneasily contrived existence, with the North Harbour province feeding into the franchise to ensure the Auckland Blues were not a virtual All Blacks team.
In 1999 they reverted to a more natural geographic membership, though their finishes were mostly mid-table with a semi-final appearance in 2004 and a 61-17 hammering at the hands of the Bulls in the 2009 final their only playoff experience.
They slumped after that loss to the Bulls, finishing 11th in 2010 and 10th in 2011, before Foster decided the team needed a fresh focus.
Rennie took over a team losing a number of established All Blacks, such as Mils Muliaina and Sitiveni Sivivatu, who were heading overseas after the 2011 World Cup.
Losing them was both a blessing and a curse.
While he lost their experience, Rennies could shape the squad the way he wanted, picking young players and nurturing their development.
Rennie's track record in developing younger players was already proven after he had taken the 'Baby Blacks' to three successive under-20 world titles and reinvigorated the fortunes of Manawatu.
Several players from those sides have followed him to Hamilton.
Rennie has also shown an ability to identify talent in unheralded players and turn them into Super Rugby starters.
Winger Asaeli Tikoroituma, for example, was playing provincial rugby for third-tier Wanganui before he moved to Manawatu, while number eight Matt Vant Leven had only played club rugby.
Assistant coach Tom Coventry has also been an important recruiter and enticed the likes of Brodie Retallick and Andrew Horrell to Hamilton from the Hawke's Bay provincial side where he worked as an assistant coach.
Retallick has since established himself as a first-choice All Blacks lock.
Others, like team captain Craig Clarke and Samoa's World Cup captain Mahonri Schwalger, were vastly experienced and the type of players prepared to just roll up their sleeves and go about their business to ensure the team succeeded.
While Sonny Bill Williams sprinkled a little stardust over the 2012 team, it is the likes of Clarke and Schwalger who best reflect Rennie's uncomplicated approach.
A happy working environment is conducive to a productive team culture, one in which the players will go out on to the field and not want to let their team mates down.
It also helps keeps egos from becoming over-inflated.
A hands on coach, Rennie is also careful to share the credit with assistants Wayne Smith, Coventry and Andrew Strawbridge, who, he said, gelled as a unit because none were 'yes men' and tactics and selections were heatedly debated.
Each has an opinion, which he listens to, then as the head coach he calls the play.
"Fair enough too," Rennie told Fairfax Media. "Ultimately we have to take responsibility. If we win a championship then we deflect all the praise to the players."