Even lacking a password, it is possible, though only for someone who has the technical knowledge to access the raw data and defeat the password. If the data is encrypted, then the raw data will be more difficult to read - one would have to break the cryptogram to unencrypt the data. Basically, it's just really, really hard.
In protecting your data, one should first consider, what is my data worth? If your data has a value, then you should protect it. Chances are you have gone a bit overboard. Yet, USB drives are easily lost.