This absolutely is a concern of mine, but I think I can actually give decent experience here.
i've actually constructed a small magnetic mount for mine, so I regularly wear mine on my left hip. I also keep a cellphone holder on this belt in the same area. I actually also have a habit of wearing Scottevest clothing, which is bulky specifically because of its ability to hide devices.
I've worn and used mine in several areas considered to be at least moderately secure, including several airports and the area surrounding One WTC.
I honestly can say I would be prepared to calmly discuss this device with any law enforcement who would ask about it, but I can say that not one has so far.
In terms of appearance, I am an asian male with long hair and a tendency to wear what looks like a cowboy hat, so this may be part of the reason no one thinks i'm a danger; I probably just come off as weird.