Source Credibility Scoring (Document Credibility Score)
AI Skills

Source Credibility Scoring (Document Credibility Score)

Skip to main content
< All Topics
Print

Source Credibility Scoring (Document Credibility Score)

A pre-publication and in-research instrument that scores **sourcing rigor, not political direction**. Patriot University publishes accountability journalism with a point of view; this skill exists to make sure that point of view rests on solid sourcing. It is internal-only — the score never reaches a reader.

Full design: docs/source-credibility-framework.md. Tooling: the /ops/sources registry dashboard, gui/services/document_scorer_service.py (the scorer), /compose/score (the Compose panel), and the opt-in dcs_gate publish gate.

When to use

  • Before publishing any accountability profile, voting guide, or civic analysis —

to check the evidentiary foundation.

  • While researching in Compose — to see each web source’s credibility (FRS),

bias direction (metadata), and class inline.

  • When triaging whether a sourcing weakness is a must-fix before go-live.

Do not use it to balance viewpoints or to soften a critical stance. A document entirely critical of an official scores 100 if it is well-sourced and properly attributed.

The gate that runs first: rigor, not direction

Before any scoring, hold this boundary — it is the analogue of the malice-evaluator’s ideology-vs-malice gate:

Score only whether claims are attributed, supported, and framed in a way the cited sources actually support. Never score the political direction, the strength of the language, or the point of view.

Charged, adversarial language on a properly sourced claim is not a defect. An unsourced or framing-beyond-the-source claim is — regardless of its political direction. Highlight and ask; never rewrite.

The four dimensions (0–25 each → DCS 0–100)

A. Source Mix (deterministic). Resolve every cited outlet against the Source Credibility Registry (per-outlet Factual Reliability Score, 0–100). Reward a high weighted-average FRS, at least one primary/official source (court record, gov database, official filing — detected even in prose), no uncorroborated low-FRS source, and source diversity. **Primary/official sources are exempt from the concentration penalty** — a profile that is 70% court dockets is ideal, not over-reliant.

B. Attribution (Claude + Media Bias Taxonomy rubric). Are factual claims attributed to a named source, linked document, or data reference? Deduct for “critics say” / “some argue” without attribution and for quantitative claims (dollar amounts, statistics, dates) lacking a verifiable source.

C. Claim Support. Ratio of supported to asserted factual claims; deduct for unsupported causal claims (“X caused Y” with no evidence) and controversial claims lacking two independent sources.

D. Red Flags (25 − penalties). Deduct for: citing a known-unreliable source without justification; advocacy-only support for a contested fact; self-sourcing (PU citing PU as external evidence — but slug cross-references are encouraged, not penalized); unresolved [NEEDS VERIFICATION] / [NEEDS CITATION] markers.

Thresholds

  • 85–100 green — publish; no extra review.
  • 70–84 yellow — editor review; the score card highlights weak dimensions.
  • < 70 red — hold; editorial-director sign-off + specific remediation.

The DCS supplements editorial judgment; it never auto-rejects. Even a red score is overridable with documented justification (/compose/score is advisory; the dcs_gate publish gate takes --allow-low-dcs).

The FRS and bias metadata

Each source’s FRS is editorial-primary (PU’s own judgment) with a free Wikipedia WP:RSP confirmatory overlay; editorial always wins. Class floors: a primary-official source floors at 80; known-unreliable ceils at 20. **Bias direction is metadata only — it never gates and never reaches a reader.** WP:RSP covers news/legacy outlets but is silent on advocacy orgs, think tanks, and government sources — PU’s heaviest-usage zone — so editorial FRS carries those.

How it composes


accountability-profile-verification  →  malice-evaluator / spokesperson-misinformation-assessment
        →  source-credibility-scoring (this skill)  →  patriot-sanity-check
  • Runs after verification and severity scoring: those establish that the

evidence exists and what it means; this establishes that the document cites that evidence rigorously.

  • Runs before patriot-sanity-check, whose “source credibility auditor” role

this skill formalizes and feeds with structured per-dimension scores.

What this skill does NOT do

  • It does not balance viewpoints or penalize perspective.
  • It does not auto-reject — a red score is overridable with justification.
  • It does not neutralize or rewrite language — it highlights spans and asks.
  • It does not surface any score to readers — the DCS, FRS, and bias metadata are

internal editorial instruments only.

Was this article helpful?
0 out of 5 stars
5 Stars 0%
4 Stars 0%
3 Stars 0%
2 Stars 0%
1 Stars 0%
5
Please Share Your Feedback
How Can We Improve This Article?
Table of Contents