All perfect praise be to Allah, The Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad is His slave and Messenger.
It was confirmed that the Prophet supplicated against the polytheists when they caused much harm, and it was also confirmed that he supplicated in their favor in order to soften their hearts to Islam.
In his book Fat-h Al-Baari, Ibn Hajar said, “His (Al-Bukhari's) saying ‘in order to soften their hearts’ is a wise indication from the author, as he pointed to the difference between the two cases, and that he sometimes supplicated against them and sometimes supplicated in their favor. The first case was when they were strong and caused much harm... and the second case was when their harm was not feared and it was hoped that they would be attracted to Islam, as in the story of (the tribe of) Daws.”
Ibn Battaal said in his commentary on Sharh Al-Bukhari:
“The Prophet liked that people embraced Islam, so he used not to hasten to supplicate against them as long as he hoped that they would accept to embrace Islam. Rather, he used to supplicate for those whom he hoped would turn to Allah, and those of whom he did not expect that they would embrace Islam and feared their harm, he would supplicate against them, as he supplicated that they suffer severe years like the severe years of Yoosuf (Prophet Joseph), and he supplicated against the leaders of Quraysh due to their extensive harm and enmity. His supplication against them was answered and they were killed in the Battle of Badr, and many of those for whom he supplicated for guidance became Muslim.”
The Prophet also supplicated against those who contravened the Sharia, as in the hadeeth about the man who refused to eat with his right hand. An-Nawawi said in his commentary on Saheeh Muslim, “This hadeeth is evidence on the permissibility of supplicating against someone who contravenes a religious ruling without having a sound excuse for doing so.”
In regard to why the Prophet did not supplicate against the people of Taa'if, it was due to him hoping that Allah would bring from their loins offspring who would believe in Allah. This is evidence of his great compassion and patience. Ibn Hajar commented on this saying, “This hadeeth is evidence of his compassion for his people and his constant great patience and forbearance; this corresponds to the saying of Allah (which means): {So by mercy from Allah [O Muhammad] you were lenient with them.} [Quran 3:159]; and His saying (which means): {And We have not sent you [O Muhammad] except as a mercy to the worlds.} [Quran 21:107]”
Also, the Prophet did not supplicate against the Jews because he hoped to soften their hearts towards Islam. Ibn Hajar commented on the hadeeth of responding to their greeting, saying:
“Al-Muhallab said, ‘This hadeeth proves the permissibility of a person in leadership tricking the one who plots against him and confronting him without him [the enemy] knowing if it is hoped that he would turn to Allah.’ I say: restricting it to this condition is something that has to be examined, because the Jews at that time were people who had a covenant with the Muslims, so it appears that this [the Prophet answering them in that manner] was for the benefit of softening their hearts to Islam.”
Allah knows best.