All perfect praise be to Allah, The Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad is His slave and Messenger.
First of all, we ask Allah Almighty to guide your brother to the right path and enable him to regain his senses, as our Lord is Near and answers the supplications. Indeed, Allah commanded us to supplicate Him and promised to answer our supplication; Allah says (what means): {And your Lord says, "Call upon Me; I will respond to you." Indeed, those who disdain My worship will enter Hell [rendered] contemptible.} [Quran 40:60]
So we advise you to supplicate as much as is possible for him, especially his parents, as the parents’ supplication for their child is answered, Allah willing. Abu Hurayrah, may Allah be pleased with him, narrated that the Prophet said, “There are three supplications that will undoubtedly be answered (by Allah): The supplication of a victim of injustice, the supplication of a traveler, and the supplication of a parent for his child.” [Ibn Maajah]
Therefore, you should not despair in trying to rectify him, as Allah is over all things Omnipotent; and people's hearts are in His Hands and He turns them as He wishes. So you should continue to advise him even indirectly, by seeking the help of a wise advisor to admonish him, even if he is not a relative, such as one of his friends, for example, because a stranger may have a greater influence than a relative, in order to clarify the seriousness of his actions to him, especially given that it includes wronging other people; so he should remind him of the evil fate of his actions. Abu Hurayrah, may Allah be pleased with him, narrated that the Prophet said, “Do you know who is the bankrupt?” They (the Companions) said, “The bankrupt among us is the one who has neither money nor property.” He, sallallahu ‘alayhi wa sallam, said, “The real bankrupt among my Ummah (nation) is the one who comes on the Day of Resurrection with prayers, fasts, and zakaah, but he comes having abused some people, slandered some, unlawfully consumed the wealth of some, shed the blood of some, and beaten some, so each of them is given some of his good deeds. If his good deeds fall short to clear the account, then some of their sins are thrown on him and he is thrown into Hellfire.” [Muslim]
As long as he persists on sinning, it is permissible to desert him, especially if it is hoped that deserting him would deter him, and this is better in that case. However, if it is feared that he would become more stubborn (and continue to sin), then it is better to soften his heart hoping that this would benefit him. Please refer to fatwa 84976.
Based on this, there is no harm on you in not inviting him to family gatherings as a way of deterring him, and fearing that the children might be influenced by him is a preponderant benefit.
Allah knows best.