Black-Box vs Source-Assisted Benchmarks

Three public write-ups now cover the same two open-source applications. The numbers look like a leaderboard. They aren't one, and the reasons are more useful than the ranking.

Direct answer: Four public counts on Fider/Photoview (49, 31, 52, 55) use different testing modes, counting units and validators. Compare those columns before you compare the integers.

In May 2026 Doyensec published "Comparing AI Application Security Testing Platforms: Aikido vs. XBOW", testing both platforms on Fider and Photoview. In June we published a black-box companion run on the same two applications. On October 8, 2026, Invicti published its own agentic pentest benchmark on the same versions, Fider 0.33.0 and Photoview 2.4.0.

Four numbers are now in circulation:

Source	Platform	Mode	Count	What is counted	Who validated
Doyensec	Aikido	Source-aware (GitHub repo paired)	49	True positives	Doyensec, manually
Doyensec	XBOW	Source-aware (source upload)	31	True positives	Doyensec, manually
Invicti	Invicti Agentic Pentest	Black-box plus source-assisted, combined	52	"Unique, verified findings", endpoint-normalized	Invicti
RedPick	RedPick	Black-box only (URL + credentials)	55	Extended true positives (21 key items + 34 validated extras)	RedPick scoring pipeline

The short version. Read the column headers before the numbers. Each count comes from a different testing mode, a different counting unit and a different validator. Ranking them is the one thing the data can't support. What the data does show: black-box testing with no repository access reached the same range as source-assisted workflows on the same targets.

---

The two targets, briefly

Both applications are open source, which is exactly why they work as shared benchmark targets: anyone can deploy the same version and check a claim.

• Fider is a feedback and feature-voting platform. Users post ideas, vote and comment; administrators moderate, manage tags and configure the site. It is written in Go with a React front end, and it has the kind of role model (visitor, user, collaborator, administrator) where access-control bugs live.
• Photoview is a self-hosted photo gallery with a GraphQL API, user accounts, albums and public share links. Shares, media paths and per-user libraries create a large surface for authorization and path-handling issues.

Neither is a toy built to be broken, like a capture-the-flag lab. They are real projects with real users, which makes the findings more representative and the scoring harder, because there is no official list of "planted" bugs. Every answer key on these two apps is a reviewer's judgment.

Why the four numbers don't sit on one axis

1. Testing mode

Doyensec configured both commercial platforms with source code: Aikido paired the GitHub repository, and XBOW received a source upload. Invicti reports "complementary black-box and source-assisted (white-box) scans" and sums them. RedPick's June run used only a target URL and credentials: no repository, no source upload, no static review.

Source access is a legitimate commercial mode, and it helps. It is also a different test. Many real engagements are black-box by necessity: third-party software, acquired applications, code the client won't share.

Testing guides make the same distinction. The OWASP Web Security Testing Guide describes black-box, grey-box and white-box approaches as complementary, and NIST SP 800-115 separates testing done with and without prior knowledge of the target. Neither treats one as a superset of the other. Source review finds dead code paths and unsafe patterns that an outside attacker may never reach. Black-box testing finds what is reachable, including problems that come from deployment and configuration rather than the code.

Summing the two modes, as a combined count does, answers a third question: "what can we find if we use everything we have?" That's a useful number for a customer who can share source. It just isn't comparable to a single-mode count.

2. Counting unit

• Doyensec counts true positives after manual review, and also reports false positives (2 for Aikido, 1 for XBOW).
• Invicti counts endpoint-normalized findings.
• RedPick's 55 is an extended true positive count: 21 answer-key items found at full credit plus 34 validated findings outside the key. Our June post explains that one finding can satisfy several key items. That's why 62 internally confirmed findings, 20 of which map to the key, produce this figure.

The same vulnerability can count once, several times or not at all, depending on the unit.

A worked example (hypothetical). Imagine an application where a missing ownership check affects three GraphQL mutations: updateAlbum, deleteAlbum and shareAlbum. All three share one resolver helper that forgot to compare the album owner with the current user.

• Counted per root cause, that's one finding.
• Counted per endpoint, it's three.
• Counted against an answer key that lists "IDOR on album management" as one item, it's one key item, and any extra endpoints may or may not earn credit depending on the reviewer.
• If the reviewer decides the delete case is only reachable after another bug is fixed, it might count as zero for that endpoint.

None of these choices is wrong. But a 20% gap between two vendors can come entirely from the counting rule, without any difference in what they actually found.

3. Validator

Only the Aikido and XBOW counts carry an independent third-party review. Invicti's and RedPick's are self-validated. We said so in June, and the caveat applies equally to every vendor-published number, including ours.

Self-validation isn't worthless. It is how every team measures progress internally. But it moves the burden onto the reader. When a vendor validates its own results, the useful questions are: was the answer key fixed before the run, were findings outside the key judged by written criteria, and were rejected findings reported too?

---

Speed

Invicti reports report-ready results in 46 minutes on Fider and 50 minutes on Photoview, across black-box and source-assisted assessments. Doyensec reported its own timing observations for Aikido and XBOW in the original study.

RedPick doesn't publish run times for its benchmark runs, so we can't offer a like-for-like column here. When you compare vendors on speed, ask each for the same measurement: wall-clock time to a report-ready result, per app and per testing mode.

---

What each mode is good for

Your situation	Mode that answers your question
Third-party or vendor-supplied application	Black-box (you usually don't have source)
Internal app, code review already in your SDLC	Black-box or grey-box, to test what review misses
New product before launch, full access available	Source-assisted plus black-box, reported separately
Compliance test with an external-attacker scope	Black-box with credentials for each role in scope
Measuring exposure of an acquired company	Black-box first, source review once access is agreed

If a vendor's benchmark mode doesn't match the row you're in, its score tells you little about how it will perform for you.

What a fair comparison would need

If the industry wants a real leaderboard on these targets, every published result should state:

1. Exact target versions. Fider 0.33.0 and Photoview 2.4.0 are now the de facto standard.
2. Testing mode, reported separately and never summed across modes.
3. Counting unit, with the deduplication rule written down.
4. The validator, and access to the raw findings.
5. Time to a report-ready result, measured the same way.

We'd welcome an independent effort to re-score all published runs against a single answer key. Until then, read every vendor number, ours included, with its mode, unit and validator next to it.

How to use these numbers if you're buying

1. Find the row in the table above that matches your situation.
2. Look only at results produced in that mode.
3. Read how each vendor counts and who validated. Discount self-validated numbers equally, ours included.
4. Ask for a trial on an app you know, in your mode, and reproduce the High and Critical findings yourself. Our buyer's checklist has a two-week evaluation plan.

Takeaways

• Same targets does not mean same test.
• A black-box result and a source-assisted result answer different questions: "what can an outside attacker do?" versus "what can a reviewer with the code find?"
• Before comparing any two AI pentest numbers, check the mode, the unit and the validator.

References: Doyensec, Aikido vs. XBOW (May 2026); Invicti, Agentic Pentest Benchmark on Fider and Photoview (October 8, 2026); RedPick, 55 Findings, No Source Code (June 3, 2026).

---

Related: Doyensec black-box run · AI pentesting buyer's checklist · Benchmarks