All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His Slave and Messenger.
The Sunnah in this regard is for the Muslim to say the Du‘aa’ Al-Istiftaah (the invocation at the beginning of prayer) right after the opening Takbeerah, and then to say the Isti‘aathah (A‘oothu Billaahi min Ash-Shaytaan Ar-Rajeem) and then the Basmalah (Bismillaah Ar-Rahmaan Ar-Raheem) in this order. Scholars said that if the person missed any of these, then the one that comes before the other should be omitted.
Kashshaaf Al-Qinaa‘ reads, "If the worshipper did not say the Du‘aa’ Al-Istiftaah, even deliberately, and said the Isti‘aathah, then saying the Du‘aa’ Al-Istiftaah is omitted. The same applies if he did not say the Isti‘aathah and said Basmalah or missed the Basmalah and recited Al-Faatihah. This is because saying any of these is an act of Sunnah (i.e. it is a recommended act) whose place in prayer was missed, so it should not be said afterwards." [Briefly excerpted]
It is not part of the Sunnah of the Prophet, sallallaahu ‘alayhi wa sallam, to reverse the order; and he, sallallaahu ‘alayhi wa sallam, said: "Whoever introduces in our matter (i.e. Islam) something which does not belong to it, it will be rejected." [Al-Bukhari and Muslim] Another version of the Hadeeth reads: "Whoever introduces a practice which we have not commanded, it will be rejected."
An-Nawawi wrote: "The scholars of Arabic said that 'radd' (lit. 'rejection') here means 'rejected'. It means that it is vain and does not count. This Hadeeth consists of a great principle from among the principles of Islam, and it is an example of his concise words, sallallaahu ‘alayhi wa sallam, as it is explicit about the rejection of every innovation and invention."
Your second question is not clear. If you mean the Du‘aa’ (supplication) after the Tashahhud, then you can say any Du‘aa’ you wish to say, whether it starts with "Rabbana" or "Allaahumma inni thalamtu nafsi..." or any other Du‘aa’.
Imaam An-Nawawi said that it is recommended to make Du‘aa’ after the Tashahhud and before the Tasleem. He wrote: "The worshipper may supplicate Allaah however he wants, whether it is something related to the Hereafter or the world; however, the matters related to the Hereafter are better. He may recite the Du‘aa’ reported specifically for this place in the prayer or reported for another place; and he may also recite an unreported Du‘aa’, for whatever he wishes of the matters of the Hereafter and the world…. The evidence for this are the authentic Ahaadeeth which we shall cite in a special chapter, Allaah Willing, among which is that the Prophet said: '…Then, let him say any Du‘aa’ he wishes to say.'" [Briefly excerpted]
For more benefit, please refer to Fatwa 90952.
Allaah Knows best.