What do you think of white people adopting black children?

My fiancee and I found a baby girl we want to adopt...she is from Ghana (dark-skinned). We've tried talking to people about it and I am a little saddened that it's such a controversial subject to many people. I think as long as the child has a loving, safe home that is all that matters...