All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
The mother is the person who has more right to foster the children as she would be more affectionate to them. The children should stay with their mother until the age of seven. Then the son could choose to stay with his mother or with his father. But when the daughter reaches the age of seven, her father is more appropriate to foster her. This is of course if the parents are free from things that prevent from guardianship like dissoluteness.
If the mother remarries, then she loses the right to foster the children. Therefore, the father has the right to take the child if there is nobody more appropriate to foster him, like the maternal grandmother, or there is someone who can foster him but he/she gives up the right and assigns the father the right of custody of his child.
Anyway, all the matter should be referred to the Islamic courts of justice in the country you are living in. If there is none, then it should be referred to the Muslim scholars and people of knowledge who would deal with this matter. If you can not solve the problem or fear harm from him, you may consult the courts.
For more benefit on the custody of children and right of visiting them, please refer to Fataawa 326341, 86891, 87762, 88331, 89832, 182019, 213352, 82128, 341287, 84575, and 86053.
For more benefit on the sequence of relatives who have more right in the fostering of the children after their mother, please refer to Fataawa 90467 and 84618.
Allah knows best.