If you're in high school, I'd say don't do it.
Standing awkwardly outside of a local laundromat, I watched as this guy pulled into the parking lot to meet us.
He was extremely tall, towering over me at six feet, two inches, and had that teenage bad-boy look to him. I expected him to be as obnoxious and annoying as the friend he brought with him, who was making mean jokes and play-fighting with Leslie. He stood a distance away from us, his eyes shyly focused on his feet.
(Obviously, this is a bit different if you're both out of high school.) Second, how does your friend feel about it? If he's sort of neutral, I probably still wouldn't do it. Overall, it's risky, tricky, and possibly not wise. If you do choose to, then definitely talk with your friend before hand.
Surprising your friend by suddenly dating his sister without warning would not be a good idea.
Basically, dating your best friend’s brother puts you right at the center of pre-existing and self-created family feuds.