Is it wrong to want to move this quickly?

"only have when you're with someone you know you're going to be with forever"...this answers part of your question, and you say you have known him for a long time as a friend, so he is obviously someone you trust. Follow your heart. You have nothing to feel guilty about.