Rate Limiting AI Traffic at Scale

Course 4’s rate-limiting lesson built one transient-based counter, bucketed by user and a fixed window, and wired it into a single ability’s permission_callback. That’s the right building block, and it’s still correct. What it doesn’t cover is what happens at real production scale: a limit that only looks at one user misses the case where a hundred different users each stay under their own limit while collectively hammering your model provider into a bill you didn’t plan for. And a hard rejection once a limit is hit is a blunt tool, a queued retry or a cached fallback is often the better answer than a flat error. This lesson extends the pattern in both directions: a global ceiling alongside the per-user one, and graceful degradation instead of a wall.

Step 1: One counter isn’t enough at real traffic volume

The Course 4 counter answers “has this user exceeded their own limit.” It says nothing about the site as a whole. A hundred users each making nineteen calls inside their twenty-call limit never trips that check even once, while the site as a whole just made 1,900 model calls in that window, a real cost and a real load on whatever the model provider allows concurrently. Production rate limiting needs both a per-user bucket and a site-wide one, checked together:

// File: my-plugin/class-rate-limits.php
function my_plugin_check_rate_limits( int $user_id, string $bucket_name, int $user_limit, int $global_limit, int $window_seconds ) {
	$window_key = floor( time() / $window_seconds );

	$user_key   = "my_plugin_rl_user_{$bucket_name}_{$user_id}_{$window_key}";
	$global_key = "my_plugin_rl_global_{$bucket_name}_{$window_key}";

	$user_count   = (int) get_transient( $user_key );
	$global_count = (int) get_transient( $global_key );

	if ( $user_count >= $user_limit ) {
		return 'user_limited';
	}

	if ( $global_count >= $global_limit ) {
		return 'global_limited';
	}

	set_transient( $user_key, $user_count + 1, $window_seconds );
	set_transient( $global_key, $global_count + 1, $window_seconds );

	return 'allowed';
}

Returning a distinct reason string, rather than just true/false, matters for Step 3: a user hitting their own limit and the whole site hitting its shared ceiling call for different responses.

Step 2: Wire both checks into the call site

// File: my-plugin.php
add_action( 'wp_abilities_api_init', function () {
	wp_register_ability(
		'my-plugin/generate-product-copy',
		array(
			// ...label, input_schema...
			'permission_callback' => function () {
				if ( ! current_user_can( 'my_plugin_generate_copy' ) ) {
					return false;
				}

				$status = my_plugin_check_rate_limits(
					get_current_user_id(),
					'product_copy',
					20,   // 20 calls per user
					500,  // 500 calls site-wide
					60    // per 60-second window
				);

				if ( 'allowed' !== $status ) {
					return new WP_Error(
						'rate_limited',
						'user_limited' === $status
							? 'You have exceeded your call limit for this action. Try again shortly.'
							: 'This feature is at capacity site-wide right now. Try again shortly.',
						array( 'status' => 429 )
					);
				}

				return true;
			},
			'execute_callback' => function ( $input ) {
				// generation logic
			},
		)
	);
} );

The distinct error messages matter for whoever’s debugging a complaint later: a user hitting their own cap is a different situation from the whole site being saturated, and conflating them into one generic message hides which one actually happened.

Step 3: Degrade gracefully instead of only rejecting

A flat rejection is the right response when there’s genuinely nothing better to do. But for some features, there’s a reasonable fallback that’s better than a hard error. Two practical options, building directly on earlier lessons in this course:

// File: my-plugin/class-rate-limits.php
function my_plugin_generate_with_degradation( string $feature, string $prompt, string $system_instruction, int $user_id ) {
	$status = my_plugin_check_rate_limits( $user_id, $feature, 20, 500, 60 );

	if ( 'allowed' === $status ) {
		return my_plugin_generate_text_cached( $prompt, $system_instruction, DAY_IN_SECONDS );
	}

	// Over limit: try the cache one more time before giving up.
	$key    = my_plugin_ai_cache_key( $prompt, $system_instruction );
	$cached = my_plugin_get_cached_ai_response( $key );

	if ( null !== $cached ) {
		return $cached;
	}

	// No cached fallback available: queue it for later instead of a hard failure.
	as_enqueue_async_action( 'my_plugin_generate_deferred', array( $feature, $prompt, $system_instruction, $user_id ), 'my-plugin-ai' );

	return new WP_Error(
		'rate_limited_deferred',
		'This request has been queued and will complete shortly rather than right now.',
		array( 'status' => 202 )
	);
}

A 202 Accepted with a clear message is a materially better experience than a flat 429, for the class of feature where the caller doesn’t need the answer in this exact request.

Test it

Push a bucket over its per-user limit and confirm the response distinguishes it from a global limit, then push a separate low per-user, low global limit to confirm the global branch fires too:

wp-env run cli wp eval '
for ( $i = 0; $i < 25; $i++ ) {
	$status = my_plugin_check_rate_limits( 1, "test_bucket", 20, 500, 60 );
	echo $i . ": " . $status . "\n";
}
'

You should see allowed for the first 20 calls and user_limited after that. Lower the global limit below 20 in a second run to confirm global_limited fires instead once the shared ceiling, not the per-user one, is what’s actually exceeded.

Recap

A single per-user counter, the Course 4 pattern, misses aggregate load across many users who are each individually within limits. Checking a per-user bucket and a site-wide bucket together closes that gap, and returning a distinct reason for each keeps the response meaningful instead of a single generic denial. Where a reasonable fallback exists, a cached response one lesson back, a queued job two lessons back, degrade gracefully rather than only rejecting, but always tell the caller honestly what actually happened.

Resources & further reading

• Transients API, developer.wordpress.org
• Action Scheduler API documentation, GitHub
• MCP Adapter repository, GitHub