Some clothing is specifically underwear, while some is also used as swimsuit (if made of suitable material), and both T-shirts and some shorts are suitable as underwear as well as outer clothing. Suitability as outer clothing is, apart from outdoor or indoor climate, largely a social and sometimes even a legal matter. One of the criteria for shorts not to be suitable as outer clothing may be that it has a fly that avoids exposure of the genitals just by an overlap of cloth, without buttons etc. (Source: http://en.wikipedia.org/wiki/Underwear)
Even though my swimsuit bottoms could be worn as outer wear, I chose to wear pants when I went to the bank. But as I sat there asking for a refund of the $70 in fees that they had wrongly charged me, I secretly knew that at any moment, I was ready to go for a nice refreshing dip from the waist down. If the occasion so arose. Which it didn't. But still. It's good to be prepared.