I'm scared of having minority friends?

My family is in the kkk but i have a black friend, how do i tell my friend that he cant come to my house?