A gunman on a motorcycle opened fire on the car carrying Qassem Aqlani to the embassy in the Yemeni capital Sanaa, Reuters reported.
Yemeni officials suspect Al Qaeda was responsible for the attack.
"This (assassination) operation has the fingerprints of Al Qaeda which carried out similar operations before," an unnamed security source told Reuters.
The attack comes two weeks after senior intelligence official Col. Abdullah al-Ashwal was killed in a drive-by shooting in Sanaa, the AP said.
CNN reported the US Embassy in Yemen was stormed by protesters last month to protest the privately made anti-Islam video “Innocence of Muslims,” which triggered a wave anti-US violence across North Africa and the Middle East.
Extremist activity in Yemen has been on the rise. The government recently launched military offenses in a bid to retake its militant-held south, which was overrun with radical groups in the wake up uprisings against the country's former leader last year.