Are Princess Peach and Princess Rosalina sisters

No.

Originally, the developers at Nintendo had intended for the two to be related, but eventually decided against it. This can easily be seen by common physical traits between the two and between Peach and Rosalina's mother, as seen within the storybook of Super Mario Galaxy.