Source Credibility Scoring (Document Credibility Score)
A pre-publication and in-research instrument that scores **sourcing rigor, not political direction**. Patriot University publishes accountability journalism with a point of view; this skill exists to make sure that point of view rests on solid sourcing. It is internal-only — the score never reaches a reader.
Full design: docs/source-credibility-framework.md. Tooling: the /ops/sources registry dashboard, gui/services/document_scorer_service.py (the scorer), /compose/score (the Compose panel), and the opt-in dcs_gate publish gate.
When to use
- Before publishing any accountability profile, voting guide, or civic analysis —
to check the evidentiary foundation.
- While researching in Compose — to see each web source’s credibility (FRS),
bias direction (metadata), and class inline.
- When triaging whether a sourcing weakness is a must-fix before go-live.
Do not use it to balance viewpoints or to soften a critical stance. A document entirely critical of an official scores 100 if it is well-sourced and properly attributed.
The gate that runs first: rigor, not direction
Before any scoring, hold this boundary — it is the analogue of the malice-evaluator’s ideology-vs-malice gate:
Score only whether claims are attributed, supported, and framed in a way the cited sources actually support. Never score the political direction, the strength of the language, or the point of view.
Charged, adversarial language on a properly sourced claim is not a defect. An unsourced or framing-beyond-the-source claim is — regardless of its political direction. Highlight and ask; never rewrite.
The four dimensions (0–25 each → DCS 0–100)
A. Source Mix (deterministic). Resolve every cited outlet against the Source Credibility Registry (per-outlet Factual Reliability Score, 0–100). Reward a high weighted-average FRS, at least one primary/official source (court record, gov database, official filing — detected even in prose), no uncorroborated low-FRS source, and source diversity. **Primary/official sources are exempt from the concentration penalty** — a profile that is 70% court dockets is ideal, not over-reliant.
B. Attribution (Claude + Media Bias Taxonomy rubric). Are factual claims attributed to a named source, linked document, or data reference? Deduct for “critics say” / “some argue” without attribution and for quantitative claims (dollar amounts, statistics, dates) lacking a verifiable source.
C. Claim Support. Ratio of supported to asserted factual claims; deduct for unsupported causal claims (“X caused Y” with no evidence) and controversial claims lacking two independent sources.
D. Red Flags (25 − penalties). Deduct for: citing a known-unreliable source without justification; advocacy-only support for a contested fact; self-sourcing (PU citing PU as external evidence — but [[slug]] cross-references are encouraged, not penalized); unresolved [NEEDS VERIFICATION] / [NEEDS CITATION] markers.
Thresholds
- 85–100 green — publish; no extra review.
- 70–84 yellow — editor review; the score card highlights weak dimensions.
- < 70 red — hold; editorial-director sign-off + specific remediation.
The DCS supplements editorial judgment; it never auto-rejects. Even a red score is overridable with documented justification (/compose/score is advisory; the dcs_gate publish gate takes --allow-low-dcs).
The FRS and bias metadata
Each source’s FRS is editorial-primary (PU’s own judgment) with a free Wikipedia WP:RSP confirmatory overlay; editorial always wins. Class floors: a primary-official source floors at 80; known-unreliable ceils at 20. **Bias direction is metadata only — it never gates and never reaches a reader.** WP:RSP covers news/legacy outlets but is silent on advocacy orgs, think tanks, and government sources — PU’s heaviest-usage zone — so editorial FRS carries those.
How it composes
accountability-profile-verification → malice-evaluator / spokesperson-misinformation-assessment
→ source-credibility-scoring (this skill) → patriot-sanity-check
- Runs after verification and severity scoring: those establish that the
evidence exists and what it means; this establishes that the document cites that evidence rigorously.
- Runs before
patriot-sanity-check, whose “source credibility auditor” role
this skill formalizes and feeds with structured per-dimension scores.
What this skill does NOT do
- It does not balance viewpoints or penalize perspective.
- It does not auto-reject — a red score is overridable with justification.
- It does not neutralize or rewrite language — it highlights spans and asks.
- It does not surface any score to readers — the DCS, FRS, and bias metadata are
internal editorial instruments only.
