Project 2: Verbosity drift across note types¶
Mentors: To be announced
Problem: Not all clinical note types are equally jargon-dense, but nothing quantifies whether discharge summaries, radiology reports, and visit notes actually differ in how heavily they fragment under tokenization.
Context: Built on mtsamples medical transcription reports across multiple specialties and note types, anchored to ML4LLM Ch.2 · proj2: Book lengths in characters, words, and tokens (helper).
Goals: Do discharge summaries, radiology reports, and visit notes differ systematically in their token-per-word ratio (a proxy for jargon density)?
Deliverables: A notebook that groups mtsamples reports by note type, computes character/word/token counts per report, and compares the token-per-word ratio distribution across note types with box plots. A higher ratio flags note types where clinical vocabulary fragments more heavily under tokenization.
Showcase: TBD
References:
- Making the Most Out of the Limited Context Length: Predictive Power Varies with Clinical Note Type and Note Section: quantifies how token/word statistics differ across clinical note types and sections, the same cross-note-type comparison this project runs on mtsamples.