All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
Patience has its great virtues and benefits, and whenever it is possible for a Muslim to be patient with the harm of the people, he should do so. For more benefit on the excellence of patience, please refer to Fatwa 83577.
Besides, repelling an evil deed with a good deed has good consequences in this world and in the Hereafter. Allaah Says (what means): {Repel (the evil) with one which is better (i.e. Allaah ordered the faithful believers to be patient at the time of anger, and to excuse those who treat them badly), then verily, he, between whom and you there was enmity, (will become) as though he was a close friend.}[Quran 41:34]
Nevertheless, it might be wise sometimes to encounter toughness with toughness. Therefore, it would not be wise to be weak when one must be strong, and whoever is strict where he should be tolerant, he is not wise as well.
Hence, such neighbors should be admonished about the wrong thing that they are doing, and if they continue doing so, then one should do what would deter them and their tikes. So, you may take their case to the authorities if the latter will prevent them from such aggression.
Also, if you can move to another location where you and your children will be safe, then you should do so. It might also be better in your case to look for a pious husband who would marry you and protect you and help you in dealing with the difficulties of life.
Indeed, it is permissible for a woman to look for a pious husband in order to marry him as we clarified in Fatwa 82471.
Allaah Knows best.