How to Help My Younger Brother Overcome Unethical Behavior and Lack of Responsibility in an Islamic Family?

Assalam o Alaikum I recently found out that my younger brother who is 18 yrs old, is involved in some unethical activities that are forbidden in Islam. I can not ask him about them directly because I can't tell him that I have spied on him because that way he will get rebellious and will start hiding everything. I cannot tell anyone in the family due to the same reason because I'm just too afraid that something would go wrong. I have tried to stop him from doing such things by mentioning examples of other people that people who do such things get serious consequences etc. But I don't think that he understands. He is not serious about his academics as well. We are 5 sisters 2 are married and Me and my sister are also older than other siblings and we will also get married and he will be the responsible one after us but we are so much worried what he'll do and how he'll fulfil all the responsibilities of the family. My mother sometimes say that someone must have done some kind of ilm on him that's why he is so much irresponsible, Allah knows better but I don't believe that his non serious behavior is due to some ilm or something. Please guide me what should I do to get him on track.