Donatello AI Benchmark / Protocol & evidence
Same prompts. Every attempt. Open evidence.
FLUX.2 [pro] and Ideogram 4.5 medium through Runware: a small, reproducible image API pilot, executed October 6, 2026.
Protocol v1 and its downloads remain available. The separately registered October 6 pilot adds 24 API attempts, technical results and costs. No public votes, independently reviewed quality scores or quality winners are published.
What this test actually found
All 24 scheduled requests returned PNG files that fully decoded at 1024 × 1024. The API reported $1.08 in combined charges. Those are technical outcomes, not proof that every image follows its prompt or is ready for commercial use.
No quality winner. Four chosen prompts are not representative of all creative work. There was no independent blinded review; usability, preference and cost per usable image remain unmeasured. This tests two model configurations via one paid API, not Donatello's customer generator, free mode, or the consumer services of either model maker.
| Configuration | Decoded files / attempts | Total charge | Median API response |
|---|---|---|---|
| FLUX.2 [pro] | 12 / 12 | $0.36 | 8.79 s |
| Ideogram 4.5 · medium | 12 / 12 | $0.72 | 5.91 s |
Response time runs from dispatch to the complete API JSON response, including queue and network time. Downloads are excluded and were validated later. It is not end-to-end user waiting time. The original full-study protocol uses a different, end-to-end latency definition. With only 12 files per model, we do not report p90 or confidence intervals.
Download SHA-256: 47747dd984443acb0256a026df8e399d48c66dadd173f317183fef5f6c83da6f
Frozen inputs, visible settings
The operator froze the manifest locally at , before dispatch. This is an operator record, not independent public preregistration. Selection: Two available text-to-image API models supporting 1024-square output with bounded published tariffs; not selected on observed outputs.
Four original English prompts selected from protocol v1: product photography, advertising, typography and spatial instructions. Three repetitions for each prompt and model; one 1024-square PNG per request, sequential dispatch, deterministic shuffled round-robin order (schedule seed 20261006). No references, retries, fallback models, retouching or selective replacements.
- FLUX.2 [pro] — model identifier
bfl:5@1. Provider-default generation; no prompt-enhancement switch exposed in the registered schema. Provider model and price documentation. - Ideogram 4.5 · medium — model identifier
ideogram:4.5@0. Medium rendering quality; magic prompt off. Provider model and price documentation.
These settings are distinct strata, not equivalent quality levels. Seeds differ by scheduled attempt and are included in the download; the same seed would not imply identical randomness across models. One EU operator host; the provider's execution region and internal model revision behind its public identifier are undisclosed.
The authorized ceiling was $3 of existing prepaid balance, with $2.94 reserved at conservative published upper bounds. No subscription, top-up or customer-credit charge was needed. Reported costs come from the provider's includeCost response, not an independently audited invoice; taxes and later refunds are not measured.
Public manifest SHA-256 (compact JSON): d9b24ee2010b10c16848016b9f8575a790b0fb63dc813cd24230ffb913e47b5d
Every output, not a highlight reel
Original AI-generated PNGs. Captions expose each attempt's response time and charge. Open the original for full-resolution inspection; the download lists its byte size and SHA-256. No customer content or real-person references were used.
Products · product-01
Predeclared checks, not scored in this pilot: Green bottle and silver cap; One rightward shadow; No branding.
Prompt SHA-256: e85ade03e99f1cec798ea3f3424e00417c1f551b6f40573c1ba54015f79bbfd6
FLUX.2 [pro]
![Products: FLUX.2 [pro], repetition 1, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-02.png)
success · 8.87 s API response · $0.03
Open original PNG
![Products: FLUX.2 [pro], repetition 2, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-03.png)
success · 8.71 s API response · $0.03
Open original PNG
![Products: FLUX.2 [pro], repetition 3, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-06.png)
success · 7.75 s API response · $0.03
Open original PNG
Ideogram 4.5 · medium

success · 5.22 s API response · $0.06
Open original PNG

success · 4.92 s API response · $0.06
Open original PNG

success · 5.39 s API response · $0.06
Open original PNG
Advertising · advertising-01
Predeclared checks, not scored in this pilot: Can, lemons and droplets; Yellow background; Empty upper third.
Prompt SHA-256: aff73a99200e746d5d2633d5a47fadf2c3e6a4089058988f1bf2983b9abfb374
FLUX.2 [pro]
![Advertising: FLUX.2 [pro], repetition 1, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-19.png)
success · 9.65 s API response · $0.03
Open original PNG
![Advertising: FLUX.2 [pro], repetition 2, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-22.png)
success · 12.17 s API response · $0.03
Open original PNG
![Advertising: FLUX.2 [pro], repetition 3, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-23.png)
success · 9.39 s API response · $0.03
Open original PNG
Ideogram 4.5 · medium

success · 7.46 s API response · $0.06
Open original PNG

success · 8.91 s API response · $0.06
Open original PNG

success · 6.98 s API response · $0.06
Open original PNG
Typography · type-01
Predeclared checks, not scored in this pilot: Exact text FRESH IDEAS; Black uppercase on cream; One green underline and no extra text.
Prompt SHA-256: 040a5e6be9b3a7358d86c08712bcc01da3d0ef547d9dbe72d8a2968e276624ce
FLUX.2 [pro]
![Typography: FLUX.2 [pro], repetition 1, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-07.png)
success · 7.44 s API response · $0.03
Open original PNG
![Typography: FLUX.2 [pro], repetition 2, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-10.png)
success · 8.51 s API response · $0.03
Open original PNG
![Typography: FLUX.2 [pro], repetition 3, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-11.png)
success · 10.69 s API response · $0.03
Open original PNG
Ideogram 4.5 · medium

success · 5.22 s API response · $0.06
Open original PNG

success · 5.96 s API response · $0.06
Open original PNG

success · 4.90 s API response · $0.06
Open original PNG
Complex instructions · instructions-01
Predeclared checks, not scored in this pilot: Exactly three objects; Correct colors and shapes; Specified spatial relationships.
Prompt SHA-256: 2fdf5dc00f5e4a4a4ae9f30cfe406289a56aec09fcd2bcd8b38a0d7adcbe1592
FLUX.2 [pro]
![Complex instructions: FLUX.2 [pro], repetition 1, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-14.png)
success · 9.88 s API response · $0.03
Open original PNG
![Complex instructions: FLUX.2 [pro], repetition 2, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-15.png)
success · 8.64 s API response · $0.03
Open original PNG
![Complex instructions: FLUX.2 [pro], repetition 3, original AI-generated output](/benchmarks/media/image-pilot-20261006/pilot-18.png)
success · 7.88 s API response · $0.03
Open original PNG
Ideogram 4.5 · medium

success · 6.01 s API response · $0.06
Open original PNG

success · 6.79 s API response · $0.06
Open original PNG

success · 5.86 s API response · $0.06
Open original PNG
Limits, provenance and reuse
Donatello funded and operates this pilot. A successful decode says nothing about exact lettering, object counts, aesthetics, safety for every use or commercial clearance. Repetitions share prompts and are not 24 independent tasks. No user votes or reviewer scores were collected. Results can vary with load, provider changes and settings.
We reviewed output publication terms on October 6: Runware, Black Forest Labs and Ideogram. Public access does not grant a blanket training licence, exclusive copyright or third-party rights. Cite this page, the version 2026-10-06-pilot-v1, configuration and dataset checksum when referring to these technical observations.
Compare documented platform features and terms · See actual Donatello examples · Report a correction