Dialog is a balance of opposing forces. On the one hand, it’s supposed to be two or more people talking to each other, so it can be considerably less formal than most narrative; on the other hand, it’s not a transcription or recreation of an actual conversation, and thus needs to smooth over some of the verbal tics that occur in reality. It has to reflect the personality of the character speaking it, while still fulfilling whatever plot or background needs the writer has for the scene. And ultimately, it has to balance all these things with readability and the author’s intentions for the story.
Real conversations are full of digressions, half-finished sentences, and pausing words and phrases like “um,” and “er,” and “like, you know.” Conversations in fiction don’t leave these out entirely, but they also don’t usually contain three fragments, two ums, and five you-knows in two lines of dialog. Similarly, even if both characters in a conversation are teenagers from similar backgrounds attending the same high school, it is rare for both of them to pepper their sentences with the same catchphrases, slang, or even “like.” If there is a character who uses a lot of slang or uses “like” and “you know” three or more times per sentence, that character is usually extremely minor, with only a few lines of dialog in the entire novel, or else is meant as an obvious send-up of a particular type or subgroup.
One reason for this is that all these interruptions and digressions make dialog harder to read, and quickly become annoying. What people are accustomed to glossing over in real conversation is much harder to ignore when it’s right there on the page. Thus, the more dialog a particular character has (and the more central he/she is to the plot), the stingier writers usually are with ums, ers, likes, slang, and catchphrases generally.
This doesn’t mean that dialog is written strictly in formal, standard English sentences. It’s still two or more people talking to one another, and it needs to sound like it. This generally means that most of the sentences in dialog will be shorter and less complex than the ones in the narrative. Characters will use contractions (which are seldom present in narrative, unless the story is in first-person) and there will be occasional sentence fragments. Punctuation gets used to indicate a particular character’s delivery of a sentence, whether or not it is technically correct. (“Listen, dammit! Put. That. Down.”)
Two people talking also means that the words within the quotes need to be something that someone would actually say. Tongue-twisters are not always as obvious as “Peter Piper picked a peck of pickled peppers,” so if you have any doubts, read the dialog out loud. Better yet, have someone else read it to you – this reveals not just tongue-twisters and awkward phrasing, but some of the places where the meaning could go in two different directions (if you read it, you don’t catch those, because you know what you meant it to mean).
Keeping dialog believable while also being readable and non-annoying means not overusing any technique just because you can get away with them in dialog. Yes, you can have sentence fragments, but if everyone consistently leaves out the subject of their sentences, it makes conversations hard to read and hard to follow. Yes, you can use dashes and elipses and italics far more often than you can get away with in narrative, but if they’re in every sentence of dialog, long conversations get harder to read. Also, if you overuse any specific technique in every line of your dialog, all your characters start to sound the same.
Many of the tricks for improving dialog involve reading out loud or listening to things being read. Read plays (which are mostly dialog) out loud. Watch different productions of the same play and note how the same dialog means different things in the hands of different actors. (Shakespeare is good for this, because there are zillions of filmed productions of his words.) Write some scenes as playscripts and have several people get together and do a reading, with each person taking a different part. Write characters whose speaking styles are so different from yours – and from each other’s – that you have to think about how you are phrasing things every time they open their mouths.
Or even the same production, if the cast is having fun with the possibilities. I once had the opportunity to see the same actor doing Macbeth on two different nights. When he got to the bit where Lady Macbeth has gone insane and he says “Cure her of that,” the first night he did it in a begging, pleading tone as if his heart would break if she were not made whole again. On the second night, he said it with such deadly cold that the first three rows of audience were trying to back through their seats. Same words, completely different delivery — and thus different characterization and even a somewhat different scenario.
I cannot count the number of times I’ve seen a book lauded as “written the way people really talk!” And my immediate reaction is “No, no it isn’t. Thank gawd.”
Personal bugbear: dialect in dialogue. If’n ah starts a-readin’ a buke an’ ah comes acrawss a load of sennences lahk this, I tend to put it down, or at least skim over those passages. It may be a perfect transcription of the dialect speech into the author’s own speech style, but (a) it’s hard work to read and (b) English writing is not IPA, and if my accent isn’t like the author’s I’ll get a completely misleading idea of what it’s meant to sound like.