Overnight leader Kim was on course for victory standing on the 17th tee but was told by an official he had been penalised two strokes for grounding a club in an area deemed to be a hazard on the 13th hole.
The South Korean then bogeyed the 17th and could only par the 18th to miss out on the OneAsia title.
Kim then decided to go back to the scene of the crime on the 13th and argued for two hours that he had not grounded his club before finally accepting his penalty and signing his card for a six-over-par 77.
That left compatriot Kang Sung-hoon, who carded a final round 69, to bag the title as Kim finished tied second on three-under with McIlroy and South Korean trio of Lee Sang-hee, Mo Joong-kyung and amateur Lee Chang-woo.
The 26-year-old Kang, who also won the Asian Tour's CJ Invitational last week, felt terrible for Kim.
"I'm a really good friend of his so at the moment it doesn't feel great. Even though I won the tournament, I just feel really sorry for him," he said.
"I was actually out there to celebrate for him, but... I don't know... I don't know what to say. It's horrible."
Former world number one McIlroy only had himself to blame for not ending his title drought on the demanding Woo Jeong Hills Country Club course, near Cheonan, south of Seoul.
Starting the day 10 strokes back, McIlroy put on a masterclass tee-to-green but endured another difficult day on the greens as he signed for a four-under-par 67 and was left wondering what might have been.
"I could have shot anything, absolutely anything. I only missed two greens and had so many chances, but it was like the story of yesterday - I just didn't hole enough putts," the 24-year-old Northern Irishman told reporters.
"I created so many more chances today that it could have been 61, 62. It just wasn't to be.
"I didn't birdie any of the par fives which was disappointing. It was tough. I felt like it could have been so much lower the last couple of days."
The twice Major winner, who won the Order of Merits on both sides of the Atlantic last year, has slipped to sixth in the rankings after a disappointing season on and off the course.
The search for the season's first title continued for McIlroy, who played the Korean event after a four-week break, but his approach play gave him confidence.
"A little frustrating, but I'm happy with how I hit it. I hit the ball really well off the tee and my iron play was very solid as well," said McIlroy, who was the top draw at the one billion Korean won (£580,000) event.
"I feel like my game is in good shape going into the next few weeks, and that's a good thing."
A bit of work on the greens was all he needed before heading to Shanghai for next week's BMW Masters.
"If I keep giving myself all those birdie chances, sooner or later I'm going to start holing a few," he added. "I'll work on my putting over the next couple of days and get ready for Shanghai."
- Sports & Recreation