Burnley said the transfer of star striker Austin, who hit 28 goals in 40 appearances last season for the Championship club, is off “due to unforeseen complications”.
Hull manager Steve Bruce told talkSPORT that Austin, who was expected to be unveiled by his club this week after a £4.5 million deal was agreed, had failed a medical at the KC Stadium.
"Unfortunately, it looks like it (the deal has collapsed)," he said. "There has been a bit of a problem with his medical, which is the worst scenario for everybody concerned, and more so the boy.
"It's an awful, awful, awful situation to find ourselves in, but unfortunately the medical is not great, so it looks as if it is a non-starter at the moment.
"I think he is surprised because he has not really missed any games over the last couple of years.
"It is a blow for him and it's a blow for us, but there you go. These medicals are so stringent, things have progressed with scans and all the rest of it and obviously he has got a problem."
No details of the medical have been revealed, but Austin has a history of shoulder problems.
Republic of Ireland international McShane was one of the star performers last season as Hull were promoted to the Premier League.
The former Manchester United, West Brom and Sunderland defender has given manager Steve Bruce a pre-season boost by putting pen to paper on a new contract.
"I'm delighted to sign a new deal," said McShane on Hull's official website.
"Last season was such a great season and then I got offered the new contract. I couldn't wait to come in and get it signed and now I can just get on with looking forward to the new season.
"I'm really looking forward to testing myself in the Premier League again. When we got relegated from the top flight in 2010, it was always my aim to get back up there.
"Now I've got the opportunity to play up there again. I'll admit there were times when I didn't think it would happen, but we had such a great season last year and we're back up there now."
McShane has made 98 appearances for Hull, having initially signed on loan in 2008.