Karlovic was fishing on his tinnie off the shore of Indented Head near Geelong early on Monday when a wave knocked him overboard.
Police have said he only survived because he was wearing a life jacket.
“Without a doubt he is alive due to the fact he was wearing a life jacket,” Inspector James Dalton has said.
But the current and the wind was too strong for him to swim to shore.
For almost seven hours the Altona man struggled in the water, bleeding from the mouth.
His tinnie was found doing circles about 9.30am, alerting police to his disappearance.
Almost four hours later police found him in the water, almost 1.5km from his boat.
Karlovic suffered hypothermia, with his lungs full of salt water.
He then spent four days in hospital.
Inspector Dalton said it was very lucky police had been able to find the man.
”To locate him so far away from his tinnie and so far off shore is like finding a needle in a haystack,” he said.