Multimodal Prompts: Combining Text and Image Input

Every previous lesson touching images used one input at a time, a text prompt to generate an image, or an image alone to describe. The genuinely useful case is often both together: a photo and a piece of text the model needs to reason about jointly. Does this product photo actually match this product description? Does this featured image fit the tone of this post’s excerpt? Is this uploaded image appropriate for this page’s subject matter? All of these need the model to hold an image and a piece of text in the same reasoning pass, not describe one and separately judge the other. with_file() and with_text() on the same builder chain do exactly that, and it’s the same combination Lesson 2 used, just with a comparison question instead of a description request.

Step 1: The combined prompt

Both with_file() and with_text() can appear on the same builder chain, order between them doesn’t matter, both get sent to the model together as part of the same message:

// File: my-plugin/class-product-photo-check.php
use WordPress\AiClient\Files\DTO\File;

function my_plugin_check_photo_matches_description( int $attachment_id, string $description ): array {
	$path = get_attached_file( $attachment_id );
	$mime = get_post_mime_type( $attachment_id );

	if ( ! $path || ! file_exists( $path ) ) {
		return array( 'matches' => null, 'reason' => 'Attachment file not found.' );
	}

	$file = new File( $path, $mime );

	$response = wp_ai_client_prompt()
		->with_text(
			'A product listing has this written description: "' . $description . '". ' .
			'Does the attached photo plausibly show the product this description is ' .
			'about? Respond with JSON only: {"matches": true or false, ' .
			'"reason": "one short sentence explaining your answer"}'
		)
		->with_file( $file )
		->generate_text();

	$parsed = json_decode( $response, true );

	if ( ! is_array( $parsed ) || ! isset( $parsed['matches'] ) ) {
		return array( 'matches' => null, 'reason' => 'Model did not return a usable answer.' );
	}

	return $parsed;
}

The instruction text names both inputs explicitly, “this written description” and “the attached photo”, so the model knows it’s being asked to relate the two rather than comment on either in isolation.

Step 2: Using the result in a real workflow

A structured matches boolean is something a workflow can branch on directly, flag a listing for review, block a save, or just log a warning:

// File: my-plugin/class-product-photo-check.php
add_action( 'save_post_product', function ( $post_id, $post ) {
	if ( wp_is_post_revision( $post_id ) ) {
		return;
	}

	$thumbnail_id = get_post_thumbnail_id( $post_id );

	if ( ! $thumbnail_id ) {
		return;
	}

	$result = my_plugin_check_photo_matches_description( $thumbnail_id, $post->post_excerpt );

	if ( false === $result['matches'] ) {
		update_post_meta( $post_id, '_my_plugin_photo_mismatch_warning', sanitize_text_field( $result['reason'] ) );
	} else {
		delete_post_meta( $post_id, '_my_plugin_photo_mismatch_warning' );
	}
}, 10, 2 );

Surfacing _my_plugin_photo_mismatch_warning as an admin notice on the edit screen turns this from a silent background check into something an editor actually sees and can act on before publishing, worth wiring up if you build this out further than the example here.

Step 3: A second practical example, tone matching

The same combined-input shape works for less binary questions too, judging whether an image’s mood fits a piece of writing rather than checking a fact:

// File: my-plugin/class-product-photo-check.php
$response = wp_ai_client_prompt()
	->with_text(
		'A blog post has this excerpt: "' . $excerpt . '". ' .
		'Rate how well the attached featured image matches the tone of that excerpt ' .
		'on a scale of 1 to 5, where 5 is a strong match. Respond with JSON only: ' .
		'{"score": 1 to 5, "reason": "one short sentence"}'
	)
	->with_file( $file )
	->generate_text();

A numeric score is easier to threshold against (“flag anything below 3”) than a free-form judgment would be, the same reasoning behind asking for structured JSON throughout this course rather than parsing prose.

Test it: check a real photo against a real description

wp-env run cli wp eval '
$attachment_id = 123; // a real image attachment ID
$result = my_plugin_check_photo_matches_description(
	$attachment_id,
	"A stainless steel french press coffee maker, 34 ounce capacity."
);
print_r( $result );
'

Try it once with a photo that genuinely matches the description, and once with a photo of something unrelated, you should see matches flip between true and false accordingly, with a reason that actually references what’s in the image.

Recap

with_file() and with_text() combine on a single builder chain to let a model reason about an image and a piece of text together in one pass, real and useful for tasks like checking whether a product photo matches its description or whether a featured image fits a post’s tone. Ask for structured JSON output so the result is something a workflow can branch on, and keep in mind this still rests on the same general-purpose pattern Lesson 2 introduced, a strong signal worth building into a review workflow, not an infallible automatic gate.

Resources & further reading

• PHP AI Client SDK repository, GitHub
• Introducing the AI Client in WordPress 7.0, make.wordpress.org
• get_post_thumbnail_id() function reference, developer.wordpress.org