Refining Edges and Hair: Select and Mask for Clean Cutouts

What “edge refinement” really means

Your first selection is usually a rough guess: it finds the subject, but edges can look jagged, too soft, or contaminated by the old background (especially around hair). Edge refinement is the process of turning that rough selection into a believable cutout by: (1) cleaning the outline (hard edges like shoulders and clothing), (2) preserving fine detail (hair, fur, lace), and (3) reducing background color spill (green trees tinting blonde hair, studio backdrops tinting dark hair).

In Photoshop, Select and Mask is the dedicated workspace for this. The goal is not to “make it perfect in one slider move,” but to follow a repeatable sequence that gets you close, then finish with small manual fixes on the mask.

A repeatable workflow (use this every time)

Step 1: Make an initial selection (fast, not perfect)

Start with any selection method you prefer (Quick Selection, Object Selection, etc.). The only requirement is that you capture the subject broadly. Don’t spend time chasing flyaway hairs yet; just make sure the main silhouette is included.

• Include the entire head and hair mass, even if the edge is messy.
• Avoid cutting into the subject (missing chunks). It’s easier to remove extra background later than to rebuild missing hair.

Step 2: Open Select and Mask

With the selection active, open Select and Mask. You’ll enter a workspace where you can preview the edge and output directly to a mask.

Step 3: Choose a view mode that reveals problems

Different view modes make different issues obvious. Switch views often; don’t commit to just one.

Continue in our app.

• Listen to the audio with the screen off.
• Earn a certificate upon completion.
• Over 5000 courses for you to explore!

Or continue reading below...

Download the app

Practical habit: do most slider work on On White or On Black, then verify on the opposite.

Step 4: Adjust the global edge sliders (in a consistent order)

These sliders affect the entire selection edge. Use small moves and watch the edge at 100% zoom while adjusting.

1) Smooth (fix jagged edges)

Smooth reduces “stair-step” jaggies and bumpy selection edges. Use it for hard shapes like shoulders, arms, hats, and clothing.

• Use a little: too much can round off corners and remove intentional detail.
• If hair starts looking like a helmet, Smooth is likely too high.

2) Feather (control softness)

Feather softens the edge transition. For most cutouts, you want a subtle feather, not a blurry edge.

• Too much feather creates a “cutout sticker” look (soft, hazy outline).
• Use feather sparingly; hair detail is better handled with the Refine Edge Brush than heavy feather.

3) Contrast (tighten a soft edge)

Contrast makes the edge transition more abrupt (it increases the difference between kept and removed pixels). This is useful after a small Feather to prevent the edge from looking foggy.

• If the edge looks too crunchy or aliased, reduce Contrast.
• Think of Contrast as “sharpening the mask edge,” not sharpening the photo.

4) Shift Edge (move the cut line in or out)

Shift Edge moves the boundary of the selection.

• Move slightly inward (negative) to reduce halos and background fringe.
• Move outward (positive) only if you accidentally cut into the subject.

Common beginner win: a small negative Shift Edge often removes the bright outline that appears when placing a subject onto a darker background.

Step 5: Use the Refine Edge Brush for hair and fur (local detail)

The Refine Edge Brush is for areas where the edge is complex: flyaway hair, fur, fuzzy fabric, tree leaves, etc. It analyzes texture and tries to separate subject detail from background.

Where to paint

• Paint along the boundary where hair meets background, not deep inside the face or far out into the background.
• Use short strokes and release often; evaluate each pass.
• If you see it grabbing too much background, undo and use a smaller brush.

Brush size rule of thumb

Set the brush slightly larger than the thickness of the detail you’re trying to recover (for flyaways, small; for fluffy hair mass, medium). Oversized brushes tend to pull in background texture.

Alternate between refine and cleanup

If your refine pass creates holes or transparency in solid areas (like the forehead edge), switch to a normal brush tool inside the workspace (if available) to add/remove from the selection, then refine again only where needed.

Practice scenario: portrait with flyaway hair on a busy background

Goal

Cut out a portrait (head and shoulders) with flyaway hair against a detailed background (trees, city street, patterned wall), then place the subject onto a new, clean background while keeping hair believable and avoiding halos.

Step-by-step exercise

1) Prepare your layers for the test

• Open the portrait image.
• Bring in a new background image (a simple gradient, studio backdrop, or blurred scene) and place it below the portrait layer.
• Temporarily choose a background that contrasts with the hair (e.g., dark background for light hair, light background for dark hair). This makes problems obvious early.

2) Make the initial selection

• Select the subject quickly (don’t worry about flyaways).
• Ensure the selection includes the full hair shape (avoid cutting into hair volume).

3) Enter Select and Mask and set a helpful view

• Open Select and Mask.
• Start with On White (or On Black) and zoom to 100% around the hairline.
• Pan around the entire head: temples, top of head, shoulders, and any fuzzy clothing edges.

4) Dial in global sliders (outline first, then hair)

• Use Smooth to clean jaggies on shoulders/clothing edges.
• Add a tiny Feather if the edge looks too harsh.
• Increase Contrast slightly if feather made the edge hazy.
• Use Shift Edge slightly negative if you see a bright fringe.

Work in small increments. After each slider change, check the hairline and the shoulder edge; you’re balancing both.

5) Refine the hair with the Refine Edge Brush

• Switch to the Refine Edge Brush.
• Paint along flyaway zones: around the crown, sides, and any wispy strands.
• Use short strokes; stop and evaluate often.
• If the background is busy, refine only the areas that truly need it (flyaways), and keep solid edges (jawline, cheeks) clean and controlled.

6) Decontaminate colors (reduce background spill)

Hair often picks up the old background color, creating a tinted halo (green from foliage, blue from sky, etc.). Use Decontaminate Colors when you see that tint along semi-transparent hair edges.

• Turn it on and adjust the amount until the edge looks neutral.
• Watch for side effects: it can shift hair color too much or create slightly “processed” edges.
• If it overcorrects, reduce the amount or plan to correct later with subtle color adjustments on the cutout layer.

Important: Decontaminate typically changes how output is created (often producing a new layer). Choose an output option that keeps your edit flexible.

7) Output to a layer mask (preferred for edits)

Output the result so you can keep refining after leaving the workspace. A mask-based output is ideal because you can paint to fix small issues without redoing the selection.

Spotting and fixing common edge problems

Problem: halos (light or dark outlines)

What it looks like: a bright rim on dark backgrounds or a dark rim on light backgrounds.

Fixes:

• Back in Select and Mask: try a small negative Shift Edge.
• Use Decontaminate Colors if the halo is tinted (color spill).
• After output: paint on the mask with a soft brush to gently hide the halo without biting into the subject.

Problem: hair looks “crunchy” or cut into

What it looks like: missing wisps, unnatural sharp notches, or hairline looks too tight.

Fixes:

• Reduce Contrast if the edge became too harsh.
• Undo and reapply Refine Edge Brush with a smaller brush and shorter strokes.
• On the mask: paint with a soft brush at low opacity to bring back a little edge softness (paint white to reveal, black to hide).

Problem: background texture gets pulled into the hair

What it looks like: bits of trees/wall pattern appear as semi-transparent junk around hair.

Fixes:

• Refine less: only paint where needed, not the entire hair mass.
• Use a smaller Refine Edge Brush and stay closer to the true boundary.
• After output: clean the mask by painting black on obvious unwanted fragments.

Manual correction: the “soft brush on the mask” technique

Select and Mask gets you most of the way, but the final realism often comes from 30–60 seconds of manual cleanup.

Setup

• Click the layer mask thumbnail to target it.
• Choose a soft round brush.
• Use low opacity (build up gradually) and paint along problem areas.

What to paint and why

• To remove halos: paint black gently along the outer edge to hide the fringe without shrinking the subject too much.
• To restore missing hair: paint white lightly where the refine process removed too much.
• To even out semi-transparent patches: use multiple low-opacity strokes rather than one heavy stroke to avoid obvious “mask brush marks.”

Practical tip: if you’re unsure, toggle the mask visibility (Shift-click the mask thumbnail) to compare before/after and ensure you’re improving the edge rather than just changing it.

Quality checks: verify at 100% and on contrasting backgrounds

Check at 100% zoom (non-negotiable)

• Inspect the hairline, ears, jawline, shoulders, and any fuzzy clothing edges.
• Look for: jaggies, halos, missing chunks, and unwanted background fragments.

Check on at least two backgrounds

An edge that looks fine on one background can fail on another. Test quickly by placing solid or simple layers behind the subject:

• One dark background (reveals light halos and leftover bright fringe).
• One light background (reveals dark halos and muddy edges).
• If possible, a saturated color (reveals color contamination around hair).

Make small mask brush corrections after each check until the cutout holds up across backgrounds.