1. Same inputs for every tool
- A fixed test set of 10 empty-room photos (see the test set) covering open-plan, living rooms, bedroom, kitchen/loft, attic with odd windows, small rooms and one partially furnished room. All are CC0 or public domain (credits), so anyone can reproduce the test.
- Photos are uploaded unmodified (same file, same resolution) to every tool.
- Room type is set to the matching option where the tool asks, style is "modern" (or the closest named equivalent, recorded in the results).
- Default settings only. No custom prompts, masks, or manual touch-ups, unless the tool cannot produce any output without them (recorded).
- The first output counts. No re-rolls, no cherry-picking. If a tool returns several variations at once, the first one shown is scored.
- For the ChatGPT/Gemini baselines we use one fixed public prompt: "Virtually stage this empty room in a modern style. Do not change walls, windows, doors, floor or camera angle. Keep the same aspect ratio."
- Every tool is re-tested monthly on the same set. A rotating held-out photo (published after the round) guards against tools tuning to the public set.
2. Automatic checks
Scored with a fixed, documented script (score.py) on original vs. staged image. The steps and thresholds are listed below.
- Aspect ratio kept · pass if within 1% of the original.
- Resolution kept · output width and height ≥ original, resolution ratio reported.
- Framing (pass/fail, reported separately) · some tools return the same room re-framed: slightly zoomed, cropped, shifted or rotated, with the walls intact. Before any structure check we estimate the geometric transform between the original and the output: SIFT feature points are matched on the wall and ceiling region only (top 65% of the frame, where furniture rarely sits), and a similarity model (scale, rotation, shift) is fitted with RANSAC, so furniture and decor that don't match are ignored as outliers. We report zoom, shift (pixels and % of the frame), rotation, the share of the original frame still visible, and the share of the output that lies outside the original frame (newly drawn edges). Pass if ≥ 97% of the original frame is visible, zoom is 0.99-1.02, rotation is at most 0.5° and at most 2% of the output is outside the original frame.
- Architecture drift (the drift score) · the output is then warped into the original's frame and compared only on the overlapping region, so a crop or a small zoom no longer counts as moved walls. Edge maps (Canny) of both images are compared at a fixed working size, masked to non-floor regions (walls, windows, doors, ceiling). We report SSIM on blurred edge maps and the recall of long straight structural lines (Hough) from the original within about 3 px. Drift score 0-100, lower is better. To avoid adding interpolation noise, edges are detected on the output in its own frame and the edge map is warped, not the pixels. Moved or removed windows, doors and columns, or changed wall and ceiling geometry, raise drift. Furniture and decor in front of a wall, and tiled watermarks, also lower the score a little. That is why drift is shown next to human scores and never used alone, and why differences of about 3 points or less are treated as noise.
- Raw drift · the earlier, unaligned score (output simply resized onto the original frame). It mixes framing and architecture and is kept only for comparison with the first runs.
- If alignment fails (too few matching points, or an implausible transform), framing is reported as unknown and architecture drift falls back to the raw score. This is flagged in the per-image score file.
- Side-by-side images show the output as delivered, scaled to the same height as the original and never stretched or cropped. The before/after slider shows the output aligned to the original frame, hatched areas are parts of the original frame the tool did not return.
- Time to result · seconds from submitting the upload to the first output being available, measured by the tester's stopwatch/harness, median over the 10 photos.
3. Blind human scoring (planned, not started)
No blind ratings have been collected yet. Current realism numbers are interim ratings from one editor (see 3b). The plan:
- At least 3 raters per image (real estate agents and photographers), who never see tool names or branding, watermarks are cropped or covered identically for all tools, and output order is randomised.
- Each output is rated 1-10 on Realism (would a buyer believe it is a photo?), Scale (furniture size and placement plausible for the room), and Lighting (shadows and light direction consistent with windows).
- Rater agreement is published, outlier ratings are kept, not deleted.
3b. Interim editor realism ratings
Until blind scoring starts, one editor rates each staged image next to its original, 1-5 on furniture realism, scale/perspective, lighting/shadows match, artifacts (5 = none) and fit to the room type, with a one-line reason for each. These are editor ratings, not automated scores, they are not blind, and they are not used in the overall score.
3c. Prices and features
Prices and features come from each vendor's public pricing and feature pages, read without signing up (dated, with links, on every tool page). Price per photo = plan price ÷ staged photos included, on the cheapest paid option and on the next tier up, monthly billing. EUR prices are converted at the ECB reference rate. If a page doesn't say, we record it as unknown rather than guess.
4. Overall score
Overall = 40% structure kept (100 − drift) + 20% realism + 15% scale + 15% lighting + 10% technical pass (aspect ratio, resolution). Human ratings are rescaled to 0-100. The weights are public and every component is shown separately, so readers can re-weight.
5. Disclosure
- For each tool we state whether results come from a free trial or a paid plan (and which plan), and the date tested.
- We state whether we have an affiliate link with the vendor. Affiliate status never affects inclusion or ranking.
- Paid plans are bought by StagingBench at list price, vendor-provided free accounts are disclosed as such.
6. Vendor right of reply
- Partial automatic-check results for single rooms (currently room 01) may be shown before a round is complete. They are clearly labelled as partial, carry no overall score, and are corrected if a vendor points out a factual error.
- Every vendor receives its raw outputs and scores 7 days before publication.
- Vendors may point out factual errors (wrong plan, wrong setting, bug on our side). Verified errors are corrected and a re-test is run on the same terms.
- Vendors may submit a statement of up to 150 words, published unedited on their tool page.
- Vendors cannot request re-rolls, change the photos, or see other vendors' results early.
7. Scores are never for sale
No vendor can pay for inclusion, ranking, removal, or early access. Sponsorship, if any, is limited to clearly labelled placements outside the leaderboard and is listed on the disclosure page. Sponsors are scored exactly like everyone else.
8. Limitations
- Edge-based drift is a proxy, not a full geometric check. Masks are hand-drawn per photo and published.
- Alignment uses one global similarity transform. A changed camera position or lens (perspective) cannot be modelled and stays in the architecture drift.
- Tools change weekly, a score describes the version tested on the stated date.
- The automatic drift check looks at the top 65% of the frame by default, so it can miss changes to the floor: Collov AI removed two steps in room 05 and still scored low drift. The editor ratings flag such cases.
8. Citing StagingBench
For a short press blurb, preferred URL and how to link, see About · Press / link to us. Contact: contact@stagingbench.org.