This question comes up with regularity on AW, and as we all know, AW is my personal frame of reference for all of reality. Invariably, everybody weighs in on the subject, and virtually invariably, the threads eventually devolve into this script:
(warning: this post is long.)
FantasyWriter: So what you're all saying is that literary fiction is somehow better than that "low-brow" genre fiction.
AllLiteraryWritersInConcert: No, that's not what we're saying at all. Where did you get that from?
ParanormalRomanceWriter: I've read some literary fiction and all it was to me was a bunch of snobbish masturbation, and the writer trying too hard to be smarter than the reader.
ALWIC: Read some better lit fic. There are poor writers in every genre. Also, stop calling us snobs, for the love of god. That's a way dead horse already.
Sci-fiWriter: My extraordinarily heavy sarcasm will fly right over your head as I sneakily imply that it's totally not fair how some literary authors win prestigious awards.
SarcasticForumRegular: *posts smiley eating popcorn*
ALWIC: Uh, you guys have the Hugo and the Nebula. Also, most of us would really enjoy the opportunities that genre fiction present for series and an enormous backlist, which makes you more money in the long run than any advance or literary award.
RandomWriter: Well, it's pretty obvious that literary writers are only in it to show off, because they are snobs. I despise literary fiction and I will never read it. That kind of pointless wankery is so far below me and I shall never abase myself. Did I mention I think you all are snobs?
The net result is that few useful or productive discussions defining literary fiction ever take place. And maybe that's a good thing, because I'm not sure it's really possible to define it.
Many people wiser and with better credentials than mine have said that it's easier to tell what Lit is not than what it is. So what isn't it?
Well, it's not especially concerned with big plots. Taut thrillers are not generally described as "literary." Quest fantasies are not usually literary.
It's not about an expected setup with a gratifying conclusion. Romances, where boy meets girl, obstacle prevents love from forming, obstacle is overcome, and happily-ever-after occurs, are not literary fiction.
It's often said that literary fiction is about character development and internal character arc, and I would tend to agree with this in the general sense, but there are other types of books that also deal with these structures but are not literary fiction. Chick lit/gossip fiction, for example, is usually about a character's internal or emotional struggles, but is not typically considered literary. I'm not sure why this is. I suspect it's for two reasons. First, the voice in chick lit tends to be plucky and assertive (even if the main character doesn't always behave that way with other characters). Literary fiction seldom has a wholly plucky main character. When an unusual amount of pluck is evident, there is usually some juxtaposed tragic element. In my observation, at any rate. I have not read every literary novel out there by a long shot, my friends.
Second, the problems main characters in chick lit face tend to be relatively light, when compared with the heavier conflicts faced by the MCs in literary fiction. In gossip fiction, a happy or at least a pleasant ending is virtually guaranteed, while in literary fiction endings can be significantly darker. A truly awful (but still plausible and satisfying) ending can only be a possibility when a high-stakes conflict is faced. If your book is about reconciling with your drunk bridesmaid after she ruins your wedding, how sharp a stake can your main character really be facing? If your book is about keeping your family from starving to death during the Great Depression, the stakes are a tad higher. So literary fiction is not about generally light-hearted plots.
Literary fiction is almost never simplistic in style. That's not to say that there is not a very large sampling of spare lit out there. Hemingway is just one example of many. His writing is marvelously trimmed but the moods and concepts in his stories and novels are far from simple. There is a complexity underlying his work that is obvious to any reader. Typically the complexity of a literary work manifests itself not only in its plot or theme but also in its style; we have all heard literary prose described variously as "embellished," "intricate," "flowery," "experimental," and even "purple," but that all depends on opinion, of course. It does tend to deal strongly with imagery, with atmosphere, and with mood, and I believe that is largely because its character-focused story arcs tend to occur deep in the realm of sensation and emotion. The style follows the structure.
The thing about literary fiction is that it's got no real home. With fuzzy boundaries like these, it can drift all over the place -- and it does. You will find strong literary elements in science fiction, in fantasy, in mainstream fiction (whatever THAT means), and in romance. Anywhere you can think to look, you will eventually bump into a heavily literary work that still sits neatly within the bounds of some genre or other.
Literary fiction is transient. It is more about feeling than plot. It is more about mood than structure. It is more about character change than character adventure. It's the Flying Dutchman of the genres, forever condemned to sail the turbulent seas beyond the safe ports of backlists and steady writing careers. Don't call snobbery on those poor souls doomed to pace its ill-defined decks. We yearn for the harbors, but we're serving a penance. We loved the sound and texture of language too much, and now we're confined to our drifting curse, always just beyond the reach of steady land.