How do I let him down not harshly?

Say this: " look, I really like being your friend but nothing more. I can't date either. I'm really sorry but I just can't be with you." tell him that in person and not in text or on phone so he knows ur serious. It's not harsh and it makes him understand that you guys aren't meant to be. Good luck! :)