Danny Hollands got the other goal as Swindon scored all five in 22 hectic minutes after an evenly-contested first hour.
Until Collins came on for Chris Martin, Swindon were struggling to break down an organised Portsmouth defence.
Within 30 seconds of coming on, Collins transformed the match with a hat-trick in 12 minutes as Portsmouth, without a win in their previous 12 matches, collapsed.
Di Canio said: "We should have done better. I told my players I am very angry and very proud.
"We went out after a poor first-half display and scored five and it should have been seven or eight. But I must admit they were amazing.
"I was lucky with Collins. I sent him on and thanks to him we have won the match. He is a typical number nine but to score four in such a short time was fantastic."