Coin won four successive games to take the first set from 2-3 down, but the Serbian starlet hit back to take the second set thanks to a vital break at 4-4.
But top seed Ivanovic made a string of unforced errors as Coin ran away with the decider.
"Today I felt nervous at the beginning and then it went away," said the 25-year-old. "I don't know how I did it.
"I am not thinking about anything right now. Just enjoying the moment."
Ivanovic's exit was the earliest by a top seed at the US Open since the start of the professional era in 1968.
"This is very disappointing to me but something I have to accept," she said. "I never saw her play before so I didn't know what to expect.
"She played a lot better than I expected. She served very well."



