In the Advanced Prompting study, stylometric deltas were computed per message between each original draft and the accepted rewrite that replaced it. Accepted rewrites shorten sentences by about $$4.3$$ words on average (95% CI $$[-7.8,-0.86]$$) and reduce sentence-length variance by about $$8.0$$ (CI $$[-15.5,-0.49]$$). Punctuation density rises sharply (+$$79$$ marks per $$1{,}000$$ tokens; CI $$[45.1,113.3]$$), reflecting the added signposting writers requested. Type–token ratio, first-person usage, and em-dash frequency are statistically unchanged, and lexical similarity remains high (TF-IDF cosine $$\approx 0.62$$, ROUGE-L$$_F \approx 0.42$$, Bigram Jaccard $$\approx 0.23$$). The pattern is consistent with participants' reported workflow of accepting a rewrite for structure and then layering Suggest edits to restore distinctive voice.

Google

This node organizes 1Cademy concepts, relations, and study-quality notes extracted from the research paper "Advanced Prompting".

Canonical source: https://1iman.com/assets/working-papers/pdf/acm-uist-conceptualgrader-interactive-provenance-and-eligibility-techniques-for-auditing-large-language-models-in-grading.pdf.

Research Paper: Advanced Prompting

Stylometric analysis is the statistical and computational study of writing style. An analyst quantifies measurable linguistic features of a text — such as average sentence length and its variance, punctuation density (e.g., marks per 1,000 tokens), em-dash frequency, type–token ratio (a lexical-diversity measure), function-word frequency, and n-gram overlap — and treats the resulting feature profile as a fingerprint that characterizes an author, genre, register, or revision. Comparing the same feature on two versions of a text yields a stylometric delta, which can be used to detect shifts in voice, structure, or lexical richness between drafts.

Stylometric Analysis

TF-IDF cosine similarity is the composite document-similarity score obtained by (1) representing each document as a vector whose components are TF-IDF weights $$w_{t,d}=\mathrm{tf}(t,d)\cdot\mathrm{idf}(t)$$ and (2) computing the cosine of the angle between two such vectors, $$\mathrm{sim}(d_1,d_2)=\dfrac{\sum_t w_{t,d_1}\,w_{t,d_2}}{\sqrt{\sum_t w_{t,d_1}^{2}}\,\sqrt{\sum_t w_{t,d_2}^{2}}}$$. The score lies in $$[0,1]$$, equals $$1$$ when the two documents have identical TF-IDF profiles, and equals $$0$$ when they share no terms. Because rare but document-specific terms receive the largest TF-IDF weight and length normalization is built into the cosine, the metric quantifies the extent to which two texts share **content-bearing** vocabulary independently of how long either text is. It is the standard text-overlap baseline reported alongside ROUGE-L and n-gram Jaccard in stylometric and rewrite-evaluation studies.

TF-IDF Cosine Similarity

ROUGE-L is an automatic summary/text-similarity metric that scores how closely a candidate text X reproduces a reference text Y using the length of their longest common subsequence (LCS). Let $$m=|Y|$$, $$n=|X|$$, and $$\mathrm{LCS}(X,Y)$$ be the length of the longest sequence of tokens that appears in both X and Y in the same relative order (not necessarily contiguous). ROUGE-L defines recall $$R_{\mathrm{lcs}}=\dfrac{\mathrm{LCS}(X,Y)}{m}$$, precision $$P_{\mathrm{lcs}}=\dfrac{\mathrm{LCS}(X,Y)}{n}$$, and the F-measure $$F_{\mathrm{lcs}}=\dfrac{(1+\beta^{2})\,R_{\mathrm{lcs}}\,P_{\mathrm{lcs}}}{R_{\mathrm{lcs}}+\beta^{2}\,P_{\mathrm{lcs}}}$$, where $$\beta$$ weights recall relative to precision. Because LCS only requires in-order token agreement, ROUGE-L rewards sentence-level structural overlap between two texts even when matched words are not adjacent, which is why it is reported alongside lexical overlap baselines (e.g., TF-IDF cosine, n-gram Jaccard) when comparing a rewritten draft to its original.

ROUGE-L Metric

Bigram Jaccard similarity applies the Jaccard coefficient to the bigram (2-gram) representations of two texts. Given a text $$T$$, let $$B(T)$$ be the set of distinct ordered token bigrams obtained by sliding a window of length $$2$$ over its tokens; for tokens $$(t_1,\dots,t_n)$$, $$B(T)=\{(t_i,t_{i+1})\mid 1\le i\le n-1\}$$. For two texts $$X$$ and $$Y$$ the bigram Jaccard similarity is $$J_{2}(X,Y)=\dfrac{|B(X)\cap B(Y)|}{|B(X)\cup B(Y)|}$$, taking values in $$[0,1]$$. Compared with unigram (bag-of-words) Jaccard, the bigram variant rewards short-range word-order agreement: two texts must reuse the same adjacent-word pairs, not just the same vocabulary, to score highly. It is reported as a lexical-overlap baseline alongside TF-IDF cosine and ROUGE-L when comparing paired drafts.

Bigram Jaccard Similarity

The Type-Token Ratio (TTR) is a lexical-diversity metric defined as the number of distinct word **types** divided by the total number of word **tokens** in a text: $$\mathrm{TTR}=\dfrac{|\text{types}|}{|\text{tokens}|}$$. Values lie in $$(0,1]$$; higher TTR indicates richer vocabulary use, while lower TTR indicates more repetition. Introduced as a vocabulary-diversity index by Johnson (1944) and adopted as a standard measure of child-language development by Templin (1957), TTR is widely used in stylometry as a voice marker. A well-known limitation is its sensitivity to text length: TTR mechanically decreases as token count grows, motivating length-robust variants such as MTLD, vocd-D, and HD-D validated by McCarthy & Jarvis (2010). Reporting TTR deltas with confidence intervals between paired texts of comparable length, as in voice-preservation studies, controls for this length effect.

Type-Token Ratio (Lexical Diversity)

Root URL: https://1iman.com/
Canonical paper URL: https://1iman.com/assets/working-papers/pdf/acm-uist-conceptualgrader-interactive-provenance-and-eligibility-techniques-for-auditing-large-language-models-in-grading.pdf
Portfolio PDF URL: https://1iman.com/assets/working-papers/pdf/acm-uist-conceptualgrader-interactive-provenance-and-eligibility-techniques-for-auditing-large-language-models-in-grading.pdf
Paper title: Advanced Prompting
Use this Reference node when citing paper-derived 1Cademy proposals from the Researcher agent.

Reference: Advanced Prompting

In the Advanced Prompting study, a two-stage IV model is fit in which a user's realized exposure to the Suggest condition (operationalized as the share of engaged minutes under Suggest) is instrumented by the user's earliest project default condition (randomized assignment used as an encouragement). With limited early data, the preliminary point estimate for the effect of exposure on acceptance is small and not statistically distinguishable from zero under HC3 inference. The authors present this only as a sanity check on the instrumentation, with fuller estimates reported in the camera-ready.

Preliminary IV Estimate: Suggest Exposure on Acceptance (Advanced Prompting)

In the Advanced Prompting study, inter-coder agreement on clustered qualitative themes is summarized as a theme-by-participant heatmap. Rows correspond to clustered themes, columns to participants, and each cell's value is the count of independent researchers who coded that theme for that participant. Darker cells indicate higher consensus across coders. Separate heatmaps are produced for the Prompting and Writing interview corpora, providing a visual diagnostic of which themes were robustly identified by multiple researchers versus those endorsed by only a single coder.

Theme Consensus Heatmap (Advanced Prompting Qualitative Analysis)

In the Advanced Prompting study, qualitative results are summarized as a per-participant table reporting up to three top clustered theme titles for each participant, aggregated across all analysts who coded that participant for a given purpose (Prompting or Writing). Unlike the Appendix K consensus heatmap, which visualizes coder agreement across the full theme set, this artifact distills each participant's experience to a small, ranked set of dominant themes, supporting at-a-glance comparison of qualitative signal across participants.

Learn Before

Related