<jupyter_start><jupyter_text>This is my attempt at performing collocation analysis similar to the one performed in Yipu and Sander's paper analysing perspective markers in Mandarin Chinese. This uses the **mclm** package which is completely undocumentedFirst find all the relevant files of your corpus<jupyter_code>advice_files <- list.files(path='../data',pattern='*[0-9]+.txt', recursive=TRUE, full.names = TRUE)

question_files <- list.files(path='../data',pattern='context.txt', recursive=TRUE, full.names = TRUE)<jupyter_output><empty_output><jupyter_text>Now do a super simple surface collocation frequency analysis using the *surf_cooc* function from the *mclm* library. For now, I'll be doing it on the words hypothesized to be indicators of subjectivity versus objectivity - *since* vs *because*, *for that reason* and *as a result*, This is based on the following from the paper *Unifying dimensions in coherence relations* by <NAME>:> ...English since seems to have a preference for epistemic/subjective relations, whereas for that reason and as a result seem to be cue phrases for typical objective causal relationsand <NAME>'s thesis:> This has been argued particularly about backward causal relations, where because can be used in every domain; however, since commonly occurs in subjective epistemic and speech-act relations, while because has been found to prevail in objective contexts(Zufferey & Cartoni, 2012).Although she adds the caveat that the lexicon of English is not as constrained as, say Dutch (perhaps?).Anyway now to do some preliminary surface collocations with the questions  Question analysis of since vs because<jupyter_code>library(mclm)
# Replace token splitter with ////s+ OR <br><br>
q_since_cooc <- surf_cooc(question_files, re_node="since", re_token_splitter="\\s+|<br><br>")
q_because_cooc <- surf_cooc(question_files, re_node="because", re_token_splitter="\\s+|<br><br>")<jupyter_output>Loading required package: ca<jupyter_text>The above can be improved with the other optional arguments to *surf_cooc*:* re_node     can be a regex, so perhaps something more than just the word?* re_boundary    seems important, dunno how to use it* re_drop_token    For now its dropping the pesky . Perhaps more?Now to analyse them a little more<jupyter_code>q_since_cooc
q_since_target_freqlist <- as.data.frame(q_since_cooc['target_freqlist'])
q_since_ref_freqlist <- as.data.frame(q_since_cooc['ref_freqlist'])

q_because_target_freqlist <- as.data.frame(q_because_cooc['target_freqlist'])
q_because_ref_freqlist <- as.data.frame(q_because_cooc['ref_freqlist'])
q_since_ref_freqlist
assoc_scores(q_since_cooc, measures = c("G"))
q_because_assoc <- data.frame(assoc_scores(q_because_cooc, measures = c("G")))
q_because_assoc_att <- q_because_assoc[which(q_because_assoc$dir==1 & q_because_assoc$G>1),]
q_because_assoc_rep <- q_because_assoc[which(q_because_assoc$dir==-1 & q_because_assoc$G>1),]

setDT(q_because_assoc_att, keep.rownames="word")
setDT(q_because_assoc_rep, keep.rownames="word")
q_because_assoc_att[order(q_because_assoc_att$G, decreasing=TRUE),]
q_since_assoc <- data.frame(assoc_scores(q_since_cooc, measures = c("G")))
q_since_assoc_att <- q_since_assoc[which(q_since_assoc$dir==1 & q_since_assoc$G>1),]
q_since_assoc_rep <- q_since_assoc[which(q_since_assoc$dir==-1 & q_since_assoc$G>1),]

setDT(q_since_assoc_att, keep.rownames="word")
setDT(q_since_assoc_rep, keep.rownames="word")
q_since_assoc_att[order(q_since_assoc_att$G, decreasing=TRUE),]<jupyter_output><empty_output><jupyter_text>Is there anything to be seen here in the two tables that show relative attraction of lexical items to contexts which contain *since* vs *because* ?What about words that are repulsed by because and attracted by since (and vice versa) ?  Words attracted by because and repelled by since<jupyter_code>inner_join(q_because_assoc_att, q_since_assoc_rep, by="word")<jupyter_output><empty_output><jupyter_text>Words attracted by since and repelled by because<jupyter_code>inner_join(q_because_assoc_rep, q_since_assoc_att, by="word")<jupyter_output><empty_output><jupyter_text>Words attracted by because and since<jupyter_code>inner_join(q_because_assoc_att, q_since_assoc_att, by="word")<jupyter_output><empty_output><jupyter_text>Words repelled by because and since<jupyter_code>inner_join(q_because_assoc_rep, q_since_assoc_rep, by="word")<jupyter_output><empty_output><jupyter_text>Analyse advice<jupyter_code>a_since_cooc <- surf_cooc(advice_files, re_node="since", re_token_splitter="\\s+|-+", re_drop_token="^Reply\\s\\d")
a_because_cooc <- surf_cooc(advice_files, re_node="because", re_token_splitter="\\s+|-+")
a_since_cooc
a_because_cooc
a_since_target_freqlist <- as.data.frame(a_since_cooc['target_freqlist'])
a_since_ref_freqlist <- as.data.frame(a_since_cooc['ref_freqlist'])

a_because_target_freqlist <- as.data.frame(a_because_cooc['target_freqlist'])
a_because_ref_freqlist <- as.data.frame(a_because_cooc['ref_freqlist'])
a_because_assoc <- data.frame(assoc_scores(a_because_cooc, measures = c("G")))
a_because_assoc_att <- a_because_assoc[which(a_because_assoc$dir==1 & a_because_assoc$G>1),]
a_because_assoc_rep <- a_because_assoc[which(a_because_assoc$dir==-1 & a_because_assoc$G>1),]

setDT(a_because_assoc_att, keep.rownames="word")
setDT(a_because_assoc_rep, keep.rownames="word")
a_since_assoc <- data.frame(assoc_scores(a_since_cooc, measures = c("G")))
a_since_assoc_att <- a_since_assoc[which(a_since_assoc$dir==1 & a_since_assoc$G>1),]
a_since_assoc_rep <- a_since_assoc[which(a_since_assoc$dir==-1 & a_since_assoc$G>1),]

setDT(a_since_assoc_att, keep.rownames="word")
setDT(a_since_assoc_rep, keep.rownames="word")
a_because_assoc_att[order(a_because_assoc_att$G, decreasing=TRUE),]
a_since_assoc_att[order(a_since_assoc_att$G, decreasing=TRUE),]
a_because_assoc_rep[order(a_because_assoc_rep$G, decreasing=TRUE),]
a_since_assoc_rep[order(a_since_assoc_rep$G, decreasing=TRUE),]<jupyter_output><empty_output><jupyter_text>Words attracted by because and repelled by since<jupyter_code>inner_join(a_because_assoc_att, a_since_assoc_rep, by="word")<jupyter_output><empty_output><jupyter_text>Words attracted by since and repelled by because<jupyter_code>inner_join(a_because_assoc_rep, a_since_assoc_att, by="word")<jupyter_output><empty_output><jupyter_text>Words attracted by because and since<jupyter_code>inner_join(a_because_assoc_att, a_since_assoc_att, by="word")<jupyter_output><empty_output><jupyter_text>Words repelled by because and since<jupyter_code>inner_join(a_because_assoc_rep, a_since_assoc_rep, by="word")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10929705215419501, 'fraction_numerical': 0.003325774754346183, 'mean_word_length': 9.273291925465838, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '471570', 'license': 'MIT', 'n_tokens_mistral': 2426, 'n_tokens_neox': 2340, 'n_words': 479, 'path': 'Notebooks/Collocation Analysis.ipynb', 'repo_name': 'venkatasg/Advice-EMNLP2020', 'size': '6629'}