I remember some years back how a friend of mine criticized another friend for wearing leather shoes with denim. He said it was baduy; he even dropped the dreaded j-bomb (jologs). While the comments were probably well deserved then, I believe that the look can be pulled off with flair quite easily, especially with all the options available these days.
I, for one, paired my Theodore-purchased brown Swear Jimmy 1's with my favorite Levi's jeans for my casual Friday outfit last week, and the combination looked pretty good by me. I just tweaked the look by folding the jeans and sporting my air socks to bring the formal feel of the oxfords down.
It's worth mentioning that jeans would look good with penny loafers too. And some classic wingtips. Just make sure to go for slim jeans (not necessarily skinny) and you instantly have another weapon in your ever-expanding style arsenal. (Oh, and you'll avoid my friend's j-bomb too. That's always important.)