When it comes down to it, anything culturally offensive doesn’t have a purely objective justification. I mean, what’s so offensive about being naked? Flipping someone off? Exposing yourself to children? Nothing, without the context of cultural norms, of course.

Swear words are offensive simply because people have dubbed them as such. Thus, when people use them, they are consciously choosing to use the agreed offensive word rather than any number of other words that are not as offensive. Using "fudge" reflects a consideration of those who might hear you and be offended. It also reflects a conscious decision to portray a less offensive/extreme personality or emotion. Either of these is enough to make something like "flip" inherently less offensive, even if it is used in a place where many would "drop the f-bomb."

It’s not the location in the sentence that determines the offensiveness, after all, but the meaning it conveys. Swear words are defined specifically as those conveying extreme/offensive meaning, so don’t be surprised when people take them that way. Also, don’t be surprised (or feel judged) if those who wish not to convey extreme/offensive meaning choose to communicate this appropriately by not swearing.