All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
We ask Allaah, the Sustainer of mankind, to heal you, for He is the Healer. No healing is of any avail but His; healing that will leave behind no illness.
We advise you to pray that Allaah blesses you with recovery; indeed, Allaah, The Exalted, answers the supplications of the desperate and distressed; Allaah, The Exalted, says (what means): {Is He [not best] who responds to the desperate one when he calls upon Him and removes evil and makes you inheritors of the earth? Is there a deity with Allaah? Little do you remember!}[Quran 27:62]
You should not despair, for Allaah, The Exalted, is able to do all things and nothing is difficult for Him. You should also keep to patience given its great virtues both in this worldly life and the Hereafter as we clarified in Fatwa 83577.
You have done well by choosing and keeping to the path of righteousness. This is an indication of your sound judgment and rationality and a sign that you are not crazy as your sister described. We ask Allaah to bless you with steadfastness on the path of Truth until death and to admit you into Paradise.
If your sister is actually as you have described, then she is immensely neglectful of the rights of Allaah over her by being indifferent about her religion, abandoning prayer, fasting and Hijab. You have done the right thing by advising her and supplicating Allaah to guide her to the straight path. This is one of the greatest manifestations of your dutifulness and kindness towards her. If your advice and supplications are met with mockery and false accusations of hypocrisy, then this is an indication of her meanness and harshness. The best reaction to such an attitude is to be patient and to pardon and repel the offense with kindness. Such a reaction may very well yield good results. Allaah, The Exalted, says (what means): {And not equal are the good deed and the bad. Repel [evil] by that [deed] which is better; and thereupon the one whom between you and him is enmity [will become] as though he was a devoted friend.}[Quran 41:34]
Ibn Katheer commented on the verse: “The verse means that there is a huge difference between evil deeds (offenses) and good ones (kindness). It means that one should repel people’s evil deeds and offenses by showing kindness towards them. ‘Umar said, ‘You would never punish a person who disobeys Allaah in his relationship with you (i.e. by offending you) with something better than obeying Allaah in your relationship with him (i.e. by being kind to him).’ The part of the verse that says: {… the one whom between you and him is enmity [will become] as though he was a devoted friend} means that if you show kindness to the person who offends you, your kindness will lead him to love you and become an intimate friend and compassionate ally.” [End of quote]
For more benefit, please refer to Fataawa 90528 and 179697.
As for Mubaahalah, it is permissible in Islam. It was defined by Ibn Al-Atheer in An-Nihaayah as follows, “It means mutual imprecation: both disputing parties gather and invoke the curse of Allaah upon the liars and wrongdoers among themselves.”
Ibn Hajar commented on the Prophet’s invitation for the Christians of Najraan - a delegation consisting of 14 Christian priests - to Mubaahalah challenge, saying, “The permissibility of Mubaahalah challenge can be deduced from this incident in case the opponent insists on his (false) opinion after its falsehood had been proved with clear evidence beyond reasonable doubts. It was reported that Ibn ‘Abbaas and then Al-Awzaa‘i and a group of scholars called their opponents to Mubaahalah challenges …” [Fat-h Al-Baari]
However, it would have been better for you not to call for that; it seems that you were driven by extreme anger to do that. If your sister was lying and has accepted the Mubaahalah, then she should blame none but herself if afflictions befall her as a result!
Allaah Knows best.