All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger. We ask Allaah to exalt his mention as well as that of his family and all his companions.
In principle it is not permissible to spy on a Muslim. Allaah says: (interpretation of the meaning): {And spy not on each other.}[49:12]. But if there is a need for it in order to achieve a benefit or to repel a harm, then there is no harm, Insha Allaah, especially if what your brother is doing results in transgressing other people's rights, and not spying on him makes those rights be lost. Therefore, we hope that there is no harm regarding what you are doing. However, you should not exceed the need as it is in contradiction to the principle. Moreover, you have to conceal his sin and not expose him.
Finally, we draw your attention to the best way of dealing with this issue. It is by reminding him about Allaah and about the bad consequences of his act. You also have to warn him about befriending bad company. And you have to set a good example for him so that he would see that you are on righteousness and upon good conduct. This would increase his desire to repent and turn to Allaah.
Allaah knows best.