All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
The rule of Friday (sermons) Khutbas is different from the prayer itself. During the Friday Khutba, the Imam can for instance, move backwards and forewords, he could talk to somebody for some reasons. He is not confined to look towards the Ka’ba.
Muslim scholars have made it lawful for us to translate the Friday Khutba into a language understood by the audience to help the people who are present understand the speech and profit from it. Otherwise the Khutba loses the objective for which it was ordered to us. The Imam should, however read the verses and Hadith in Arabic before translating them into the language of the audience. Since the translation is lawful the Imam could just give the sermon in the target language (language of the audience) and the prayer is valid and correct.
Allah knows best.