AI tool reviews should be useful, reproducible and honest about what was actually tested. This page explains how AI News & Updates plans, runs, scores, checks and corrects its reviews.

Methodology version 1.2. Last updated 21 October 2026.

Download the review rubric

The editable workbook contains the protocol, test cases, raw observation log, scoring weights, formula-driven summary and source/disclosure log. Workbook version 1.1 corrects missing-score handling, weighted scores and closed-test failure rates, and leaves starter reviewer fields blank. The PDF remains the original compact reference edition; use this page and the corrected workbook for current scoring rules.

Excel compatibility: the corrected workbook passed 43 automated formula checks, including recalculation after reopening in the checking tool. Native Microsoft Excel compatibility has not been verified. Check calculations in your spreadsheet application before relying on the results.

Our editorial principles

  1. Test before scoring. A product receives no score without a dated observation and supporting evidence.
  2. Keep tasks comparable. Products in the same comparison receive the same core prompt, eligibility rules, trial count, time limit and scoring weights. We record differences that make perfect parity impossible.
  3. Separate evidence from inference. We label hands-on observations, vendor documentation, independent evidence, editorial inference and unverified claims differently.
  4. Preserve failure. Outages, usage caps, broken citations, lost state and failed exports remain in the record. We do not silently retry until a tool looks good.
  5. Check material claims. Verify the source and check that it supports the claim before publication.
  6. Disclose influence. Free access, affiliate relationships, sponsorship, vendor contact and correction requests are logged. None may alter the protocol or score.
  7. Correct visibly. Material corrections are dated, explained and reflected in the conclusion when necessary.

What we test

Every review begins with a written use case and explicit success criteria. We choose tasks that reflect what the intended reader is likely to do, not a theatrical prompt designed only to make the product fail. Where possible, we run at least three independent trials. Each trial starts in a clean project, collection or conversation, and we record the date, plan, model or mode, material settings, time spent, cost and any usage limit encountered.

For tools that retrieve or summarise external evidence, we sample material claims and trace them to the cited source. A citation must identify a real item, link to the correct item and actually support the nearby claim. A valid-looking DOI attached to the wrong conclusion is a citation failure. Access barriers are recorded separately from mismatched or fabricated references.

We also test recovery. We correct one mistaken constraint, trigger or encounter one failed action, and examine whether the tool preserves understandable state. We then export the work and check whether citations, notes and essential context survive outside the product.

The seven-part scorecard

CriterionWeightWhat it measures
Task completion25%Whether the tool completes the stated task accurately within the constraints.
Evidence traceability20%Whether material claims are readily traceable to matching, accessible sources.
Reliability and recovery15%Consistency, state preservation and clarity when something goes wrong.
Privacy and data handling10%Documented retention terms, controls and handling of uploaded material.
Security and control10%Account, sharing and administrative controls appropriate to the tested audience.
Cost value10%Useful, verifiable outcomes at a transparent and proportionate total cost.
Maintenance10%Export quality, change communication and ongoing correction or upkeep burden.

Each criterion is scored from 0 to 10. Broad anchors are: 0–2 for material failure; 3–4 for partial completion needing substantial rescue; 5–6 for core completion with meaningful limitations; 7–8 for strong, verifiable performance with manageable limits; and 9–10 for excellent, repeatable performance with clear controls and evidence. Blank means untested, not zero.

When all seven criterion scores have supporting evidence, the overall score is the sum of each score multiplied by its published weight. Scores use a 0–10 scale and weights sum to 100%, so the overall score also uses a 0–10 scale.

Incomplete evidence means no overall score. Leave an untested criterion blank, identify what remains untested and report the number of evidenced criteria out of seven. Do not turn missing scores into zero or silently redistribute their weights. A genuine observed failure and an untested task are different states.

Before publishing a comparison, check that each criterion has comparable evidence across the included products, not just a matching number of rows. Report partial and blocked attempts separately, including their reasons. Do not publish a ranked winner while comparable testing is incomplete.

Raw observations must remain separate from calculated summaries. Check the workbook’s missing-data handling and score calculation before relying on its output; a displayed number does not by itself establish that the evidence is complete.

Our five benchmark test cases

  • T01 — Search coverage: collect candidate results and check relevance, traceability, identifiers and duplicates.
  • T02 — Evidence traceability: trace five material claims or summaries to their cited sources.
  • T03 — Screening consistency: apply disclosed eligibility rules to ten borderline records and record reasons.
  • T04 — Synthesis and disagreement: ask the tool to separate supporting, null and conflicting evidence.
  • T05 — Recovery and export: correct a constraint, recover from a failed action and export the work.

Current research-assistant benchmark

Our current protocol compares ResearchRabbit, Elicit, Scite, Consensus and Undermind using this question: For university students, does retrieval practice improve delayed retention compared with rereading?

Eligible records must directly compare retrieval practice, practice testing or active recall with rereading or restudy in higher-education students and measure retention at least 24 hours later. Randomised or quasi-experimental comparisons are eligible. The standard prompt asks each product to identify study design, sample, delay, outcome and direction of effect; provide persistent identifiers or direct paper links; separate cited claims from synthesis; surface disagreement; and state what may have been missed.

The first comparative dataset is now published. On 1 August 2026, we completed three matched attempts for each of the five products—15 run-level observations in total. Failures, blocked runs, account limits and visibly inconsistent result counts were retained rather than replaced with stronger retries.

Download and inspect the evidence

The benchmark found material variation that a single screenshot would have hidden. ResearchRabbit’s visible result count changed substantially between matched runs; Elicit’s tested free allowance prevented every planned task from being completed; Scite produced one off-topic grounded response and one access-blocked run; Consensus was the most consistent of the repeated searches in this limited sample; and Undermind reported 82, 90 and 84 papers across its three runs. These observations are dated evidence from specific account conditions, not permanent product rankings.

The repository includes CSV and XLSX files, a data dictionary, methodology, limitations and suggested attribution. Researchers, librarians and journalists may reuse the dataset with attribution and are invited to report corrections to hello@ainewsandupdates.com.

Evidence labels

LabelMeaning
Hands-on testObserved directly in a dated test; export or screenshot preserved.
Vendor documentationA supplier claim supported by a current URL and access date.
Third-party evidenceAn independent source with author, publication and URL recorded.
InferenceA reasoned editorial conclusion clearly labelled as inference.
UnverifiedA claim that could not be corroborated and is not presented as fact.

How the label appears on an article. A product or pricing guide identifies its sources and review date and shows the inputs for any estimate. A hands-on test gives the reader the setup, inputs, observations and limits. Compare our Vapi pricing guide with our Qdrant local test. Numerical scores require the full evidence record below.

Minimum evidence record for a material conclusion

A reader should be able to trace a conclusion back to a specific observation. For each material conclusion, retain:

  • Scope: the exact task, prompt, eligibility rules and intended audience.
  • Conditions: the date, account tier, visible model or mode, settings and access limits.
  • Observation: the original output or export, its evidence location and whether the attempt completed, partially completed or was blocked.
  • Source check: the claim, source identity, supporting passage or table and any access limitation. A real DOI alone does not establish that the source supports the claim.
  • Judgement: the criterion affected, the reasoning, remaining uncertainty and the reviewer’s check.
  • Disclosure: vendor contact, supplied access, relevant commercial relationships and material corrections.

Keep private or personal material out of public evidence files. Publish enough non-sensitive evidence to explain the conclusion, and state when a result cannot be independently inspected.

Vendor contact, access and corrections

We may contact a vendor before or after publication to check factual details such as the official domain, current pricing, feature names or documentation. We welcome specific, sourced corrections. Vendors do not receive approval rights over our wording, test record, score or verdict.

If a vendor provides free or extended access, that is recorded in the source and disclosure log. We identify the plan actually tested and do not assume that controls listed for an enterprise plan exist on a free plan. If a test hits a usage limit, that limitation remains part of the observation.

Affiliate relationships, sponsorship and paid placement are disclosed near the relevant content. Payment cannot buy a dofollow link, a favourable review, a higher score, removal of a material limitation or access to editorial approval. We prefer useful, earned references and decline low-quality directory or link-scheme opportunities.

Publication and correction policy

Before publication, check the recorded plan and mode, equivalent trial counts, weights, sampled citations, disclosures and wording of material conclusions. A ranking remains unpublished if comparable testing is incomplete. We also state when a conclusion is limited by account access, missing exports, language restrictions, time limits or an incomplete human gold set.

Material corrections include a date, the original affected wording or score, the corrected information, the supporting evidence and any impact on the verdict. Minor spelling or formatting changes may be fixed without a formal correction note when they do not alter meaning.


Questions, evidence or correction requests can be sent to hello@ainewsandupdates.com. For an example of a vendor-supplied factual correction that did not alter our editorial judgement, see our ResearchRabbit review.

Correction note — 21 October 2026: clarified that an overall score requires evidence for all seven criteria, added the minimum evidence record, and replaced the reusable workbook links with version 1.1. Earlier formula handling could include unobserved criteria in an average or produce misleading overall or failure-rate results. These revisions do not change the historical 15-run observations or establish a new product ranking.