WP Copilot
Search… /
Chunking & Storage

Chunking Strategies for WordPress Content

⏱ 16 min

You can’t embed a whole ten-thousand-word documentation page as one vector and expect useful retrieval, everything in that page gets averaged into a single point, and a question about one paragraph near the bottom scores about as well as a question about the introduction. Chunking, splitting content into smaller pieces before embedding each one, is what makes retrieval precise instead of vague. This lesson covers the real tradeoffs in how you split.

What you'll learn in this lesson
Why chunk size controls retrieval quality
Too large and results get vague, too small and results lose context.
Fixed-size chunking
The simplest approach: split every N characters, with overlap.
Paragraph and heading-aware chunking
Splitting along a post's actual structure instead of an arbitrary character count.
Handling long posts
What changes when a single post produces dozens of chunks instead of one or two.
Prerequisites

Lesson 2’s AiClient::input()->generateEmbedding(). This lesson produces the chunks you’ll feed into that method, and Lesson 5 stores the result of embedding them.

Step 1: Why chunk size is the actual retrieval quality lever

Every chunk becomes one embedding, one point in vector space. That point represents the average meaning of everything in the chunk, so chunk size is a direct tradeoff:

The chunk size tradeoff
Chunks too large
A chunk covering five different subtopics produces a vector that's a blurry average of all five, matching a specific question on any one of them only weakly.
Chunks too small
A chunk that's a single sentence fragment often loses the surrounding context a model needs to actually answer with, "returns are accepted" without saying within how many days is a weak answer.

There’s no universally correct number, but a common practical starting point for prose content is somewhere in the 200 to 500 word range per chunk, small enough to be specific, large enough to hold a complete thought. Treat that as a starting point to tune against your own content and your own test questions, not a fixed rule.

Step 2: Fixed-size chunking

The simplest approach splits content every N characters (or words), regardless of what sits at that boundary:

// File: my-plugin/class-content-chunker.php
function my_plugin_chunk_fixed( string $text, int $chunk_size = 1500, int $overlap = 200 ): array {
	$text   = trim( $text );
	$length = strlen( $text );
	$chunks = array();
	$start  = 0;

	while ( $start < $length ) {
		$chunks[] = substr( $text, $start, $chunk_size );
		$start   += ( $chunk_size - $overlap );
	}

	return array_values( array_filter( $chunks, fn( $c ) => trim( $c ) !== '' ) );
}

The $overlap argument matters more than it looks: without it, a sentence that happens to straddle a chunk boundary gets cut in half, split across two chunks, neither of which contains the whole thought. A couple hundred characters of overlap between consecutive chunks means most sentences end up whole in at least one chunk.

Fixed-size chunking is fast, simple, and works reasonably well as a baseline. Its weakness is that it splits mid-sentence and mid-paragraph by construction, since it doesn’t know where sentences or paragraphs actually end.

Step 3: Paragraph and heading-aware chunking

WordPress content already has natural structure, paragraphs, and for block-based or heading-organized posts, headings. Chunking along that structure instead of a raw character count keeps each chunk a coherent unit of meaning:

// File: my-plugin/class-content-chunker.php
function my_plugin_chunk_by_paragraph( string $html, int $target_words = 350 ): array {
	$text = wp_strip_all_tags( $html );

	// Split on blank-line paragraph breaks.
	$paragraphs = preg_split( '/\n\s*\n/', trim( $text ) );
	$paragraphs = array_values( array_filter( array_map( 'trim', $paragraphs ) ) );

	$chunks       = array();
	$current      = '';
	$current_words = 0;

	foreach ( $paragraphs as $paragraph ) {
		$paragraph_words = str_word_count( $paragraph );

		if ( $current !== '' && ( $current_words + $paragraph_words ) > $target_words ) {
			$chunks[]      = trim( $current );
			$current       = '';
			$current_words = 0;
		}

		$current       .= ( $current === '' ? '' : "\n\n" ) . $paragraph;
		$current_words += $paragraph_words;
	}

	if ( trim( $current ) !== '' ) {
		$chunks[] = trim( $current );
	}

	return $chunks;
}

This groups whole paragraphs together up to a target word count, rather than cutting wherever a fixed character offset happens to land. If your content is organized with headings (<h2>, <h3> blocks), a further refinement is starting a new chunk at every heading boundary regardless of the running word count, since a heading is a strong signal that the topic changed, keeping heading-adjacent content in its own chunk tends to produce more precisely matchable results than word count alone.

Step 4: Handling long posts

A single long documentation page or a comprehensive blog post can reasonably produce dozens of chunks. A few things change once you’re at that scale:

  • Keep a reference back to the source post. Every chunk needs to know which post_id it came from (and ideally which paragraph or section number), both so you can link the user back to the original page and so you can clean up all of a post’s chunks when it’s updated (Lesson 8 covers this).
  • Don’t re-chunk unchanged content. Chunking and embedding an entire long post is wasted work if nothing in it changed since the last index run, check post_modified before reprocessing.
  • Expect a long post to dominate search results if it’s about everything. A ten thousand word “complete guide” post chunked into forty pieces will show up in more searches than a focused three hundred word post, simply because it has forty chances to match instead of one. That’s usually correct behavior, but it’s worth knowing why a particular post tends to appear in retrieval results often.

Test it: compare chunk counts on a real post

wp-env run cli wp eval '
$post = get_post( 1 );
$fixed = my_plugin_chunk_fixed( $post->post_content );
$paragraphs = my_plugin_chunk_by_paragraph( $post->post_content );
echo "Fixed-size chunks: " . count( $fixed ) . "\n";
echo "Paragraph-aware chunks: " . count( $paragraphs ) . "\n";
echo "First paragraph-aware chunk:\n" . $paragraphs[0] . "\n";
'

Read the first chunk from each approach. The paragraph-aware chunk should read as a complete, coherent unit, the fixed-size chunk may cut off mid-sentence at its boundary unless overlap happened to land favorably.

Chunk size too large silently produces vague answers, not errors

Nothing about an oversized chunk throws an error, the embedding generates fine and the similarity search runs fine. The failure mode is purely qualitative: a chunk covering too much ground scores moderately against many different questions instead of scoring high against the one it should actually answer. If retrieval results feel generically relevant but never quite on point, chunk size is usually the first thing to shrink, before you touch anything else in the pipeline.

Recap

Chunk size is the single biggest lever on retrieval quality: too large and every chunk becomes a vague average, too small and chunks lose the context needed for a complete answer. Fixed-size chunking with overlap is a fast, reasonable baseline, paragraph and heading-aware chunking produces more coherent chunks by respecting the content’s actual structure instead of an arbitrary character count. Long posts need a reference back to their source and a check for unchanged content before you reprocess them. The next lesson covers where these chunks and their embeddings actually get stored.

Resources & further reading

← Turning Site Content Into Embeddings Storing Vectors: Your Real Options →