Donatello AI Benchmark / Protocol & evidence

Same prompts. Every attempt. Open evidence.

FLUX.2 [pro] and Ideogram 4.5 medium through Runware: a small, reproducible image API pilot, executed October 6, 2026.

Original protocol · Evidence update 2026-10-06Version 2026-09-11-v1Operated by Donatello

Protocol v1 and its downloads remain available. The separately registered October 6 pilot adds 24 API attempts, technical results and costs. No public votes, independently reviewed quality scores or quality winners are published.

What this test actually found

All 24 scheduled requests returned PNG files that fully decoded at 1024 × 1024. The API reported $1.08 in combined charges. Those are technical outcomes, not proof that every image follows its prompt or is ready for commercial use.

No quality winner. Four chosen prompts are not representative of all creative work. There was no independent blinded review; usability, preference and cost per usable image remain unmeasured. This tests two model configurations via one paid API, not Donatello's customer generator, free mode, or the consumer services of either model maker.

Observed outcomes · 12 attempts per model · USD API charges, not subscription prices
ConfigurationDecoded files / attemptsTotal chargeMedian API response
FLUX.2 [pro]12 / 12$0.368.79 s
Ideogram 4.5 · medium12 / 12$0.725.91 s

Response time runs from dispatch to the complete API JSON response, including queue and network time. Downloads are excluded and were validated later. It is not end-to-end user waiting time. The original full-study protocol uses a different, end-to-end latency definition. With only 12 files per model, we do not report p90 or confidence intervals.

Download SHA-256: 47747dd984443acb0256a026df8e399d48c66dadd173f317183fef5f6c83da6f

Frozen inputs, visible settings

The operator froze the manifest locally at , before dispatch. This is an operator record, not independent public preregistration. Selection: Two available text-to-image API models supporting 1024-square output with bounded published tariffs; not selected on observed outputs.

Four original English prompts selected from protocol v1: product photography, advertising, typography and spatial instructions. Three repetitions for each prompt and model; one 1024-square PNG per request, sequential dispatch, deterministic shuffled round-robin order (schedule seed 20261006). No references, retries, fallback models, retouching or selective replacements.

These settings are distinct strata, not equivalent quality levels. Seeds differ by scheduled attempt and are included in the download; the same seed would not imply identical randomness across models. One EU operator host; the provider's execution region and internal model revision behind its public identifier are undisclosed.

The authorized ceiling was $3 of existing prepaid balance, with $2.94 reserved at conservative published upper bounds. No subscription, top-up or customer-credit charge was needed. Reported costs come from the provider's includeCost response, not an independently audited invoice; taxes and later refunds are not measured.

Public manifest SHA-256 (compact JSON): d9b24ee2010b10c16848016b9f8575a790b0fb63dc813cd24230ffb913e47b5d

Every output, not a highlight reel

Original AI-generated PNGs. Captions expose each attempt's response time and charge. Open the original for full-resolution inspection; the download lists its byte size and SHA-256. No customer content or real-person references were used.

Products · product-01

Studio product photograph of a matte green insulated bottle with a silver cap on a white background, one soft shadow to the right, no branding.

Predeclared checks, not scored in this pilot: Green bottle and silver cap; One rightward shadow; No branding.

Prompt SHA-256: e85ade03e99f1cec798ea3f3424e00417c1f551b6f40573c1ba54015f79bbfd6

FLUX.2 [pro]

Products: FLUX.2 [pro], repetition 1, original AI-generated output
pilot-02 · repetition 1
success · 8.87 s API response · $0.03
Open original PNG
Products: FLUX.2 [pro], repetition 2, original AI-generated output
pilot-03 · repetition 2
success · 8.71 s API response · $0.03
Open original PNG
Products: FLUX.2 [pro], repetition 3, original AI-generated output
pilot-06 · repetition 3
success · 7.75 s API response · $0.03
Open original PNG

Ideogram 4.5 · medium

Products: Ideogram 4.5 · medium, repetition 1, original AI-generated output
pilot-01 · repetition 1
success · 5.22 s API response · $0.06
Open original PNG
Products: Ideogram 4.5 · medium, repetition 2, original AI-generated output
pilot-04 · repetition 2
success · 4.92 s API response · $0.06
Open original PNG
Products: Ideogram 4.5 · medium, repetition 3, original AI-generated output
pilot-05 · repetition 3
success · 5.39 s API response · $0.06
Open original PNG

Advertising · advertising-01

A square advertisement visual for an unbranded lemon sparkling drink: a clear can surrounded by lemon slices and water droplets on a yellow background; leave the upper third empty for copy.

Predeclared checks, not scored in this pilot: Can, lemons and droplets; Yellow background; Empty upper third.

Prompt SHA-256: aff73a99200e746d5d2633d5a47fadf2c3e6a4089058988f1bf2983b9abfb374

FLUX.2 [pro]

Advertising: FLUX.2 [pro], repetition 1, original AI-generated output
pilot-19 · repetition 1
success · 9.65 s API response · $0.03
Open original PNG
Advertising: FLUX.2 [pro], repetition 2, original AI-generated output
pilot-22 · repetition 2
success · 12.17 s API response · $0.03
Open original PNG
Advertising: FLUX.2 [pro], repetition 3, original AI-generated output
pilot-23 · repetition 3
success · 9.39 s API response · $0.03
Open original PNG

Ideogram 4.5 · medium

Advertising: Ideogram 4.5 · medium, repetition 1, original AI-generated output
pilot-20 · repetition 1
success · 7.46 s API response · $0.06
Open original PNG
Advertising: Ideogram 4.5 · medium, repetition 2, original AI-generated output
pilot-21 · repetition 2
success · 8.91 s API response · $0.06
Open original PNG
Advertising: Ideogram 4.5 · medium, repetition 3, original AI-generated output
pilot-24 · repetition 3
success · 6.98 s API response · $0.06
Open original PNG

Typography · type-01

A clean typographic poster with the exact words FRESH IDEAS in large black uppercase letters on a cream background, one green underline, no other writing.

Predeclared checks, not scored in this pilot: Exact text FRESH IDEAS; Black uppercase on cream; One green underline and no extra text.

Prompt SHA-256: 040a5e6be9b3a7358d86c08712bcc01da3d0ef547d9dbe72d8a2968e276624ce

FLUX.2 [pro]

Typography: FLUX.2 [pro], repetition 1, original AI-generated output
pilot-07 · repetition 1
success · 7.44 s API response · $0.03
Open original PNG
Typography: FLUX.2 [pro], repetition 2, original AI-generated output
pilot-10 · repetition 2
success · 8.51 s API response · $0.03
Open original PNG
Typography: FLUX.2 [pro], repetition 3, original AI-generated output
pilot-11 · repetition 3
success · 10.69 s API response · $0.03
Open original PNG

Ideogram 4.5 · medium

Typography: Ideogram 4.5 · medium, repetition 1, original AI-generated output
pilot-08 · repetition 1
success · 5.22 s API response · $0.06
Open original PNG
Typography: Ideogram 4.5 · medium, repetition 2, original AI-generated output
pilot-09 · repetition 2
success · 5.96 s API response · $0.06
Open original PNG
Typography: Ideogram 4.5 · medium, repetition 3, original AI-generated output
pilot-12 · repetition 3
success · 4.90 s API response · $0.06
Open original PNG

Complex instructions · instructions-01

On a white table are exactly three objects: a red cube to the left of a blue sphere and a yellow cone behind the sphere. Eye-level view, neutral background.

Predeclared checks, not scored in this pilot: Exactly three objects; Correct colors and shapes; Specified spatial relationships.

Prompt SHA-256: 2fdf5dc00f5e4a4a4ae9f30cfe406289a56aec09fcd2bcd8b38a0d7adcbe1592

FLUX.2 [pro]

Complex instructions: FLUX.2 [pro], repetition 1, original AI-generated output
pilot-14 · repetition 1
success · 9.88 s API response · $0.03
Open original PNG
Complex instructions: FLUX.2 [pro], repetition 2, original AI-generated output
pilot-15 · repetition 2
success · 8.64 s API response · $0.03
Open original PNG
Complex instructions: FLUX.2 [pro], repetition 3, original AI-generated output
pilot-18 · repetition 3
success · 7.88 s API response · $0.03
Open original PNG

Ideogram 4.5 · medium

Complex instructions: Ideogram 4.5 · medium, repetition 1, original AI-generated output
pilot-13 · repetition 1
success · 6.01 s API response · $0.06
Open original PNG
Complex instructions: Ideogram 4.5 · medium, repetition 2, original AI-generated output
pilot-16 · repetition 2
success · 6.79 s API response · $0.06
Open original PNG
Complex instructions: Ideogram 4.5 · medium, repetition 3, original AI-generated output
pilot-17 · repetition 3
success · 5.86 s API response · $0.06
Open original PNG

Limits, provenance and reuse

Donatello funded and operates this pilot. A successful decode says nothing about exact lettering, object counts, aesthetics, safety for every use or commercial clearance. Repetitions share prompts and are not 24 independent tasks. No user votes or reviewer scores were collected. Results can vary with load, provider changes and settings.

We reviewed output publication terms on October 6: Runware, Black Forest Labs and Ideogram. Public access does not grant a blanket training licence, exclusive copyright or third-party rights. Cite this page, the version 2026-10-06-pilot-v1, configuration and dataset checksum when referring to these technical observations.

Compare documented platform features and terms · See actual Donatello examples · Report a correction