Things to avoid Bad AA: Too much AA (over-anti-aliasing)- You only want to use as much AA as is necessary to smooth the edge. If you use too much, the edges can look blurry, and you lose the crispness of the line.
Too little AA– Here the artist has used single pixels to ease the transition, but he has only succeeded in blunting the jagged edge a bit. He could have made a much smoother transition by using longer lines of pixels to show a more gradual transition: AA banding- When segments of AA line up with the lines they’re buffering, AA banding occurs. For a better understanding of AA banding, be sure to read the section on banding. Jaggies: Jaggies occur when a pixel or group of pixels are out of place, interrupting the flow of a line. Jaggies can also occur when a line lacks anti-aliasing. Jaggies get their name from the jagged lines that they create. More broadly, jaggies are the result of any bad pixel technique, but they are most often discussed in reference to line work, so that is the context in which they will be discussed here. How to fix jaggies: Changing the length of the lines often times the problem is just that a segment of the line is too short or too long, and it creates an awkward jump. Using a more uniform length of pixels to smooth the transition is the solution here. Anti-aliasing Unless your line is perfectly horizontal, perfectly vertical, or at 45 degrees, the edges of your line segments are naturally going to be a little jagged. This is because the square nature of the pixel and the grid pattern we’re restricted to makes angled lines and curves difficult to portray. AA is the correct counter-measure in these situations. Bad dithering: There are several common ways dithering is misused. The most common mistake is simply using too much dithering. If dithering is covering half your sprite, it’d probably just be better if you added a new color to the palette. Dithering should ideally be used to taper the ends and edges of an opaque field of pixels. When too much dithering is used, the dithered area turns into a field itself: At this point dithering is no longer serving as a buffer between colors, but creating unwanted texture. Creating texture can be a useful aspect of dithering, but only when used correctly. If you’re trying to buffer and are instead adding texture, then dithering isn’t working out. So how much dithering should you use? Well, it depends on how big your palette is really- or more precisely, the contrast between the two colors you’re trying to dither with. The lower the contrast is between the two (in hue or in value), the less harsh the dithering will be: Banding: Banding, most simply, is when pixels line up. When neighbor pixels end at the same x or y coordinate on the underlying grid, the grid immediately becomes more evident, the pixels are exposed, and the apparent resolution becomes less fine. Here are several instances of banding, all of which occur because the pixels have lined up. These names aren’t common lingo, but will work for the purposes of this tutorial: Hugging: Here an opaque field of color has been outlined by a row of pixels. It’s fine to use outlines, but make sure the outline and the shape it contains don’t line up and reveal the grid. Fat pixels: Fat pixels can occur alone in small squares, together as fat lines, or multiplied as large bands (staircase banding). Skip-one banding: Even if there is a negative space between two bands, the mind will fill in the gap and banding will remain. 45 degree banding: Though the rows of pixels lining up are only 1 pixel thick, banding is still present. Pillow-shading: Shading by surrounding a central area with increasingly darker bands. Pillow-shading is bad because it pays no attention to the light source, and conforms to the shape of the area rather than the form it represents of how light affects it. Pillow shading is often, but not always, combined with banding. The way to fix pillow-shading is simply to pay attention to the direction light is coming from: The reason pillow-shading is wrong is not because the light source is frontal (from the viewer’s direction). You don’t have to place the light source in the corner. The reason pillow-shading is incorrect is because it follows flat shapes rather than focuses on how the three-dimensional forms are lit. So, it is possible to use a frontal light source, so long as you pay attention to the forms: Noise: Much of the time, independent pixels (pixels that do not belong to a pixel cluster) are unable to convey sufficient information by themselves, and their inclusion usually only creates noise. Noise is any sort of information that does not contribute to the piece and serves only to interrupt the area it inhabits and distract the viewer. In pixel art, noise is often composed of independent pixels. For the purposes of this tutorial, single-pixel noise will be what I’m referring to when I use the term “noise”. The reason one must be careful when using a 25% dither (or any dithering, really) is because of the noise all the independent pixels create. Single pixels expose the underlying grid by revealing the resolution of the image. Remember, in the wild, pixels travel in packs. It’s the nature of a pixel to long for a place in a pixel cluster. For this reason, independent pixels should only be used for very specific and purposeful reasons. Justifiable instances of independent pixels include: Use as specular highlights Independent details call a lot of attention to themselves, but sometimes this is precisely what you want. For bright specular highlights, single pixels will often work just fine. For an example, see the white pixel used on the monster’s nose below. Portraying small but essential details Usually this will only matter for details on very small images, like the eyes on a small sprite, or the beak of a tiny bird. Or stars, or little bubbles. Sel-out (broken outlines) Sel-out (short for selective outlining, also known as broken outlines) is anti-aliasing an outline to a background color. This means sel-out is really a type of bad AA, but the term has become popular enough to warrant its own section. The idea is usually to darken the outline at the contours to approach a darker color, so that the sprite will read well on any background, instead of melting into a similarly-colored background. Sel-out is not shading an outline according to a light source. A full outline with light variation won’t create jaggies as badly as a broken outline will: Perhaps this is a simpler example. The half-circle on the left is shaded according to a light source (again, coming from the top left corner). The top of the half-circle on the right has sel-out applied: Sel-out works if it is created for specific scenarios, such as in a game where you know the background will be consistently dark. (via Cure on Pixel Joint Forums)