Fifi D'Orsay was born Marie-Rose Angelina Yvonne Lussier in Montreal, Canada, to a father who was a postal clerk. The couple had a large family, with Fifi having 11 siblings. She was educated at the Academy of the Sacred Heart in Montreal before graduating and finding work as a secretary. As a young typist she wished to become an actress, and moved to New York City. Once there she found work with the Greenwich Village Follies, after an audition in which she sang "Yes! We Have No Bananas" in French. When asked where she was from, she told the director she was from Paris, France, and that she had worked in the Folies Bergère. The impressed director hired her, billing her as "Mademoiselle Fifi".

While working in the Follies, she became involved with Ed Gallagher, a veteran actor who was half of the successful Broadway comedy team of Gallagher and Shean. Gallagher and D'Orsay put together a vaudeville act, and he coached her in the art of show business. After touring in vaudeville, she headed to Hollywood and adopted the surname "D'Orsay" (after a favorite perfume). Soon after she began working in films, often cast as the "naughty French girl" from "gay Paris".