I almost never play in a party. Usually I am a random in one of the competetive playlists. I don't use a mic myself but I can still hear what others say, and I find that most of the people who DO have their mic in are idiots or children anyways, which does not encourage me to use mine. On rare occasions when I do find people who are not morons and genuinely want to talk, then I will put my mic in the next time I die or something and chat with them. But in general I see no point in talking to people who, for the most part, are incapable of saying anything that I would want to hear and respond to.
On a side note, the mics that "come with" xboxes are of notoriously terrible quality and I wouldn't blame anyone for not wanting to use them.