"Sweetie" is gender neutral for me; I use it to refer to my romantic partners, who are of various genders. "Buddy" connotes male to me. I think of "babe" as referring to a woman unless it's a form of address between people in a romantic relationship, in which case it can refer to any gender.