The 39-year-old Australian was appointed as the successor to Royce Simmons at Langtree Park last April but stayed on at the John Smith's Stadium, intending to see out his fourth season with the Yorkshire club. Huddersfield were top of Super League at the time of Saints' announcement but shortly afterwards went on a downward spiral.
Brown parted company with Giants in the aftermath of a crushing Challenge Cup semi-final defeat by Warrington in July. Addressing his first press conference at St Helens since taking up a three-year contract, he said: "The reality is with European sport when you sign somewhere you generally go the next day, whether you're a coach or a player."
He added: "At the time I made my decision, my intentions were good to stay and the club's intentions were good but it didn't work out unfortunately. It wasn't anyone's fault. I probably should have left the next day but we all wanted it to work. It didn't, so you shake hands and move on."
With St Helens going well at the time under caretaker-coach Mike Rush, Brown took the opportunity to have an extended holiday back in Australia, where he was linked with a number of coaching vacancies in the National Rugby League.
"The ending was not what anyone wanted but we used the time well," he said. "I hadn't had a break for 10 years. We went back to Australia for five weeks and got to visit some people we hadn't been able to see for a long while and had a really nice time.
"When my time at Huddersfield came to an end, I think St Helens had lost two games in 15 or 16 weeks and both were to Wigan, who at that stage were untouchable.
"They were going fine, they didn't need me to come in. I'm not sure I would have brought anything to the table at that stage of the year and the break for me was probably good.
"You always get linked to jobs, it's just the way it is. I was never not coming back to St Helens. We didn't go anywhere or do any interviews anywhere."