Purpose

Extract and analyze quantifiable linguistic features from speech corpus to:

1. Provide objective baseline metrics (lexical diversity, sentence complexity, function-word patterns)
2. Track changes over time (within-speaker trajectory analysis)
3. Compare across speakers (cohort baselines)
4. Support the cognitive-linguistic trajectory analysis (with mandatory caveats — see cognitive/cl-overview.md)

Validated Linguistic Feature Set

The following features are validated in the corpus-linguistics and computational-linguistics literature for analyzing political speech:

Lexical Diversity

• Type-Token Ratio (TTR): unique-words / total-words. Sensitive to text length.
• Moving-Average TTR (MATTR): TTR computed in sliding 100-word windows; length-controlled. Preferred metric for unequal-length texts.
• Measure of Textual Lexical Diversity (MTLD): TTR-based measure that accumulates until TTR drops below threshold. Length-controlled.
• HD-D (Hypergeometric Distribution Diversity): Probability-based measure. Length-controlled.

Recommended primary metric: MATTR (with MTLD as secondary). Avoid raw TTR for cross-text comparison.

Sentence Complexity

• Mean Sentence Length (words per sentence)
• Subordinate-Clause Density (subordinate clauses / total clauses)
• Mean Tree Depth of dependency parse (using spaCy en_core_web_lg or transformer parsers)
• Mean Words Before Main Verb (proxy for sentence-front complexity)
• Yngve Score (left-branching depth measure)

Function-Word Patterns

• Pronoun frequency by class: 1st person singular (“I”, “me”, “my”), 1st person plural (“we”, “us”, “our”), 2nd person (“you”), 3rd person
• Demonstrative frequency: “this”, “that”, “these”, “those”
• Modal-verb frequency: “can”, “could”, “may”, “might”, “must”, “shall”, “should”, “will”, “would”
• Negation frequency: “no”, “not”, “never”, “nothing”, “nobody”, “none”, contracted forms
• Filler-word frequency (transcribed): “uh”, “um”, “like” — note: heavily affected by transcription quality

Named-Entity Patterns

• Named Entity Recognition (PERSON, ORG, GPE, LOC, NORP, etc. — spaCy default tagset)
• Entity-mention frequency: How often is each named entity mentioned?
• Entity-role classification: Is the entity invoked as ally, enemy, neutral?

LIWC-style Psychological Process Categories

The Linguistic Inquiry and Word Count (LIWC) dictionaries categorize words into psychologically-meaningful categories. LIWC commercial license is required for production use; for non-commercial research, the LIWC-22 academic license is available, and free alternatives exist:

• EmoLex / NRC Emotion Lexicon (Mohammad-Turney): Free, 8 emotions + 2 sentiments
• General Inquirer: Free, comprehensive but older categories
• Empath (Stanford): Free, dynamically-extensible categories

Standard categories of interest:

• Positive emotion words
• Negative emotion words (anger, anxiety, sadness, disgust)
• Cognitive process words (causation, insight, certainty, tentativeness)
• Social process words (family, friends, generic social)
• Time orientation (past, present, future focus)
• Personal pronouns (already covered above)

Repetition Patterns

• Phrase repetition (n-gram repetitions of length 4-8): Detect repeated phrases
• Word repetition (per-paragraph or per-segment): Track repeated keywords
• Repetition entropy: Lower entropy = more repetition

Sentiment and Tone

• VADER sentiment (compound score): Lexicon-based, well-suited for short texts
• Transformer-based sentiment (e.g., cardiffnlp/twitter-roberta-base-sentiment-latest): Higher accuracy for nuanced texts
• Affective tone trends across speech

Coherence and Discourse

• Topic-shift frequency (how often does the topic shift?)
• Discourse-marker frequency (“however”, “moreover”, “therefore” — markers of explicit reasoning)
• Cohesion measures (lexical chains, coreference resolution)

Implementation Stack

For computing the above features:

• spaCy (en_core_web_lg or transformer-based en_core_web_trf) — tokenization, POS, NER, dependency parsing
• lexical_diversity Python package — MATTR, MTLD, HD-D
• NLTK — supplementary tokenization, sentence segmentation
• VADER — sentiment
• textstat — readability and basic metrics
• scikit-learn — n-gram extraction
• transformers / Hugging Face — for transformer-based features
• EmoLex / Empath / General Inquirer — psychological-category dictionaries

Robustness Considerations

Transcription Quality

Linguistic features are highly sensitive to transcription quality:

• Closed-captioning errors inflate type-token ratio (more “unique words” from transcription errors)
• Auto-transcription of filler words is inconsistent
• Sentence-boundary errors distort sentence-complexity metrics

Implementation rule: Always weight or stratify analyses by transcript_quality (human_edited, human_verified, ai_generated, auto_generated_captions). Do not pool across quality levels for fine-grained linguistic analysis.

Genre and Context Effects

• Rallies vs. interviews vs. press conferences vs. teleprompter speeches all differ systematically in linguistic features
• Within-genre comparisons are more reliable than across-genre
• For trajectory analysis (within-speaker over time), stratify by genre

Length Effects

• Many basic metrics (TTR, raw n-gram counts) depend on text length
• Always use length-controlled metrics (MATTR, MTLD, HD-D) for diversity
• For per-text comparisons of long-vs-short texts, be especially cautious

Audience Effects

• Same speaker may use different language with different audiences
• Stratify by audience type (rally crowd, journalist, interviewer, friendly TV host, etc.)

Output Schema

For each speech, produce a feature vector:

{
  "speech_id": "...",
  "speaker": "Donald J. Trump",
  "date": "2025-XX-XX",
  "genre": "rally",
  "audience": "general_election_rally",
  "transcript_quality": "human_edited",
  "word_count": 9842,
  "features": {
    "mattr_w100": 0.69,
    "mtld": 73.4,
    "hdd": 0.85,
    "mean_sentence_length": 17.3,
    "subordinate_clause_density": 0.21,
    "mean_tree_depth": 3.9,
    "first_person_singular_freq_per_1k": 41.2,
    "first_person_plural_freq_per_1k": 18.7,
    "negation_freq_per_1k": 14.1,
    "modal_freq_per_1k": 19.8,
    "vader_compound": -0.32,
    "emolex_negative": 0.082,
    "emolex_anger": 0.041,
    "...": "..."
  }
}

Statistical Treatment

For trajectory and comparative analysis:

• Mann-Kendall trend test (pymannkendall package) — non-parametric trend detection
• Theil-Sen slope estimator — robust slope estimation
• Breakpoint detection (ruptures package, PELT algorithm) — identify regime changes
• Multi-level / hierarchical models — control for genre, audience, transcript quality
• Bootstrap confidence intervals — for any reported metric

Limitations

• Linguistic features describe text properties, not speakers’ minds. Inferences about cognitive state, intent, or personality from these features must be made cautiously and accompanied by caveats.
• Cohort-comparative interpretation is required: A single speaker’s metrics in isolation are hard to interpret without baseline comparison.
• Causal inference is largely impossible from observational speech corpora: many factors covary with the time period (audience changes, genre changes, world events).

See Also

• sc-overview.md
• Analytical Framework — Rhetorical Analysis
• sc-framework-claim-tracking.md
• Analytical Framework — Topics and Frames
• cognitive/cl-overview.md (cognitive-linguistic trajectory track with caveats)
• cognitive/cl-methodology.md (specific methodology for trajectory analysis)