<jupyter_start><jupyter_text>Determining the time of diagnosisthis notebook contains the code necessary to calculate the intervals in which the diagnosis most likely took place (as referenced in Section 1 of the Supplementary Information. First, we load the required modules<jupyter_code>from dateutil.relativedelta import relativedelta
import pandas as pd<jupyter_output><empty_output><jupyter_text>Since several of the conversions are similarly referenced by users, we defined several categories of phrases as part of our mapping. These categories are defined below.<jupyter_code>_YEARS_AGO = {'19 years ago': 19, '36 years ago': 36, '2 years ago': 2, '8 years ago': 8,
              '5 years ago': 5, '3 years ago': 3, 'About 5 years ago': 5, 'Six years ago': 6,
              '20 years ago': 20, '4 years ago': 4, '13 years ago': 13, '7 years ago': 7,
              '14 years ago': 14, 'Two years ago': 2, 'One year ago': 1, 'A year ago': 1,
              '25 years ago': 25, '6 years ago': 6, 'About 4 years ago': 4, 'About 15 years ago': 15,
              '11 years ago': 11, '15 years ago': 15, 'About 3 years ago': 3, 'About 2 years ago': 2,
              'Four years ago': 4, '29 years ago': 29, '23 years ago': 23, '12 years ago': 12,
              'Eight years ago': 8, '10 years ago': 10, '9 years ago': 9, 'Five year ago': 5,
              'In the last year': 1, 'In the past year': 1, 'For 9 years': 9, 'Nearly 5 years ago': 5,
              'For 6 years': 6, 'Last year': 1, 'last year': 1}
_MONTHS_AGO = {'5 months ago': 5, '18 months ago': 18, 'Two years and one month ago': 25,
               '4 months ago': 4, '16 months ago': 16, 'A month ago': 1, 'Half a year ago': 6,
               '6 months ago': 6, 'One year and one month ago': 18, '7 months ago': 7,
               'A year and a half ago': 18, '2 and a half years ago': 30, '8 months since': 8,
               'Two months since': 2, 'Two years this month since': 24}
_DAYS_AGO = {'Three days ago': 3, '5 days ago': 5}
_WEEKS_AGO = {'2 weeks ago': 2, 'Two weeks ago': 2, '3 weeks ago': 3}
_FEW_YEARS = ['A couple of years ago', 'A few years ago', 'Few years ago', '2-3 years ago',
              'Some years ago', 'Several year ago']
_FEW_MONTHS = ['A few months ago', 'Few months back', 'A couple of months ago']
_YEARS_AGO_DATE = {'A year ago today': 1, '5 years ago today': 5, 'Four years ago today': 4,
                   '10 years ago today': 10, 'One year ago today': 1, '6 years ago today': 6,
                   '4 years ago today': 4, '3 years ago today': 3, '2 years ago today': 2,
                   '2 year ago today': 2, 'Today three years ago': 3, '1 year ago today': 1,
                   '7 years since': 7, '1 year since': 1, 'A year since': 1, 'Exactly a year ago': 1,
                   '4 years since': 4, '2 years since': 2, '5 years since': 5}
_ALMOST_YEAR_AGO = {'Nearly four years since': 4, 'Almost 25 years ago': 25, 'Almost a year since': 1,
                    'Almost two years ago': 2, 'Under 2 years ago': 2, 'Just under 2 years ago': 2,
                    'Almost 6 years ago': 6, 'For almost 5 years': 5, 'Been almost 10 year': 10,
                    'Almost 3 years ago': 3, 'Almost 9 years since': 9, 'Almost a year ago': 1}
_OVER_YEAR_AGO = {'For over a year': 1, 'Over a year ago': 1, 'Over 5 years ago': 5,
                  'A little over 9 years ago': 9}
_IN_MONTH = {'In january': 1, 'In February': 2, 'Back in august': 8, 'In december': 12,
             'In october': 10, 'In April': 4, 'In november': 11, 'In august': 8,
             'In October': 10, 'In June': 6, 'In July': 7}
_IN_YEAR = {"In 1979": 1979, "In 1993": 1993, "In 1994": 1994, "Since 1996": 1996,
            "In 1997": 1997, "In 2001": 2001, "In 2003": 2003, "In 2005": 2005,
            "In 2006": 2006, "In 2007": 2007, "In 2008": 2008, "Since 2008": 2008,
            "In 2009": 2009, "In 2010": 2010, "In 2011": 2011, "Since 2011": 2011,
            "In 2012": 2012, "In 2013": 2013, "In 2014": 2014, "In 2015": 2015,
            "In 2016": 2016, "In 2017": 2017}
_IN_MONTH_YEAR = {'May 2015': 'May 2015', 'In september 2015': 'september 2015',
                  'In may 2015': 'may 2015', 'In October 2014': 'October 2014',
                  'May of 2012': 'May 2012', 'In January 2018': 'January 2018',
                  'In november 2016': 'november 2016', 'Around October 2013': 'October 2013',
                  'In march 2017': 'march 2017', 'In december 2014': 'december 2014',
                  'In februari 2017': 'february 2017', 'In februari 2008': 'february 2008'}
_IN_PART_OF_YEAR = {'Late 2015': ("October 2015", "December 2015"),
                    'End of 2015': ("October 2015", "December 2015"),
                    'Beginning of 2016': ("January 2016", "March 2016"),
                    'Early 2015': ("January 2015", "March 2015"),
                    'Early 2017': ("January 2017", "March 2017"),
                    'Late last year': ("October 2017", "December 2017"),
                    'Over the summer': ("June 2017", "August 2017"),
                    'Last summer': ("June 2017", "August 2017")}
_IN_INTERVAL = {
    '2013 - today': pd.Timestamp("2013-1-1", tz="UTC"),
    'Somewhere between 2009 and today': pd.Timestamp("2009-1-1", tz="UTC"),
    'In the last six months': relativedelta(months=6),
    'Within the last two years': relativedelta(years=2),
}
_ON_WEEKDAY = {"On tuesday": 1, "On friday": 4, "Last friday": 4}<jupyter_output><empty_output><jupyter_text>For some tweets, we found a reference to an exact date. If this date did not contain a year, inferred the year manually based on the time at which the tweet was posted.<jupyter_code>_ON_DATE = {
    'Last year march 27': pd.Timestamp("2017 march 27", tz="UTC"),
    'On February 10 2010': pd.Timestamp("February 10 2010", tz="UTC"),
    'On friday 28 October 2016': pd.Timestamp("friday 28 October 2016", tz="UTC"),
    'On Wednesday October 11': pd.Timestamp("Wednesday October 11 2017", tz="UTC"),
    'On 9/3/14': pd.Timestamp("9/3/14", tz="UTC"),
    'On June 27 2017': pd.Timestamp("June 27 2017", tz="UTC"),
    'On feb 15': pd.Timestamp("2016 feb 15", tz="UTC"),
    'On 6 feb 2018': pd.Timestamp("6 feb 2018", tz="UTC"),
    'Recently on august 17': pd.Timestamp("august 17 2017", tz="UTC")
}<jupyter_output><empty_output><jupyter_text>Next, we define the conversion function.<jupyter_code>def convert(ts, freq):
    return ts.to_period(freq).to_timestamp().date()

def convert_timestamp(row, which="start"):
    if which not in ["start", "end"]:
        raise(NotImplementedError)

    if pd.isna(row.time_indication):
        return pd.NaT
    elif row.time_indication in ["today", "Just got", "Just been"]:
        return row.created_at.date()
    elif row.time_indication == "yesterday":
        return row.created_at.date() - pd.Timedelta("1 day")
    elif row.time_indication in ["recently", "Recently"]:
        if which == "start":
            return row.created_at.date() - relativedelta(months=3)
        else:
            return row.created_at.date()
    elif row.time_indication in _YEARS_AGO.keys():
        diff = _YEARS_AGO[row.time_indication]
        to_year = convert(row.created_at, "Y")
        if which == "start":
            return to_year - relativedelta(years=diff)
        else:
            return to_year - relativedelta(years=diff - 1, days=1)
    elif row.time_indication in _FEW_YEARS:
        to_year = convert(row.created_at, "Y")
        if which == "start":
            return to_year - relativedelta(years=3)
        else:
            return to_year - relativedelta(years=1, days=1)
    elif row.time_indication == "This time last year":
        to_month = convert(row.created_at, "M")
        if which == "start":
            return to_month - relativedelta(months=12)
        else:
            return to_month - relativedelta(months=11, days=1)
    elif row.time_indication == "Earlier this year":
        to_year = convert(row.created_at, "Y")
        if which == "start":
            to_month = convert(row.created_at, "M")
            if to_year == to_month:  # If posted in Jan, assume previous year
                return to_year - relativedelta(years=1)
            else:
                return to_year
        else:
            return row.created_at.date()
    elif row.time_indication == "Last week":
        if which == "start":
            return row.created_at.date() - relativedelta(days=row.created_at.weekday() + 7)
        else:
            return row.created_at.date() - relativedelta(days=row.created_at.weekday() + 1)
    elif row.time_indication in _FEW_MONTHS:
        to_month = convert(row.created_at, "M")
        if which == "start":
            return to_month - relativedelta(months=3)
        else:
            return to_month - relativedelta(months=1, days=1)
    elif row.time_indication == "This year":
        if which == "start":
            return convert(row.created_at, "Y")
        else:
            return row.created_at.date()
    elif row.time_indication in _YEARS_AGO_DATE.keys():
        diff = _YEARS_AGO_DATE[row.time_indication]
        return row.created_at.date() - relativedelta(years=diff)
    elif row.time_indication in _MONTHS_AGO.keys():
        diff = _MONTHS_AGO[row.time_indication]
        to_month = convert(row.created_at, "M")
        if which == "start":
            return to_month - relativedelta(months=diff)
        else:
            return to_month - relativedelta(months=diff - 1, days=1)
    elif row.time_indication == "Within the last year":
        if which == "start":
            return row.created_at.date() - relativedelta(years=1)
        else:
            return row.created_at.date()
    elif row.time_indication in _DAYS_AGO.keys():
        return row.created_at.date() - relativedelta(days=_DAYS_AGO[row.time_indication])
    elif row.time_indication in _WEEKS_AGO.keys():
        diff = _WEEKS_AGO[row.time_indication]
        to_week = convert(row.created_at, "W")
        if which == "start":
            return to_week - relativedelta(weeks=diff)
        else:
            return to_week - relativedelta(weeks=diff - 1, days=1)
    elif row.time_indication in ['Few weeks ago', 'A couple of weeks ago']:
        to_week = convert(row.created_at, "W")
        if which == "start":
            return to_week - relativedelta(weeks=3)
        else:
            return to_week - relativedelta(weeks=1, days=1)
    elif row.time_indication in _IN_MONTH:
        month_num = _IN_MONTH[row.time_indication]
        year = row.created_at.year - 1 if row.created_at.month < month_num else row.created_at.year
        dt = pd.Timestamp("{y}-{m}-1".format(y=year, m=month_num), tz="UTC").date()
        if which == "start":
            return dt
        else:
            return dt + relativedelta(months=1, days=-1)
    elif row.time_indication == "In late 90’s":
        if which == "start":
            return pd.Timestamp("1997-1-1", tz="UTC").date()
        else:
            return pd.Timestamp("1999-12-31", tz="UTC").date()
    elif row.time_indication == "About a year ago":
        to_month = convert(row.created_at, "M")
        if which == "start":
            return to_month - relativedelta(months=13)
        else:
            return to_month - relativedelta(months=10, days=1)
    elif row.time_indication in _ON_DATE:
        return _ON_DATE[row.time_indication].date()
    elif row.time_indication == "About 3-4 years ago":
        to_year = convert(row.created_at, "Y")
        if which == "start":
            return to_year - relativedelta(years=4)
        else:
            return to_year - relativedelta(years=2, days=1)
    elif row.time_indication in _IN_MONTH_YEAR:
        dt = pd.Timestamp(_IN_MONTH_YEAR[row.time_indication], tz="UTC")
        if which == "start":
            return dt.date()
        else:
            return dt.date() + relativedelta(months=1, days=-1)
    elif row.time_indication == "Over ten years ago":
        to_year = convert(row.created_at, "Y")
        if which == "start":
            return to_year - relativedelta(years=15)
        else:
            return to_year - relativedelta(years=9, days=1)
    elif row.time_indication in _ON_WEEKDAY:
        reference = _ON_WEEKDAY[row.time_indication]
        dt = row.created_at.date()
        wkd = row.created_at.weekday()
        if wkd == reference:
            return dt
        elif wkd > reference:
            return dt - relativedelta(days=wkd - reference)
        else:
            return dt - relativedelta(days=(7 - (reference - wkd)))
    elif row.time_indication in _IN_PART_OF_YEAR:
        dt_strs = _IN_PART_OF_YEAR[row.time_indication]
        if which == "start":
            return pd.Timestamp(dt_strs[0], tz="UTC").date()
        else:
            return pd.Timestamp(dt_strs[1], tz="UTC").date() + relativedelta(months=1, days=-1)
    elif row.time_indication in _IN_YEAR:
        year = _IN_YEAR[row.time_indication]
        if which == "start":
            return pd.Timestamp("{}-1-1".format(year), tz="UTC").date()
        else:
            #  in case someone references the current year
            return min(row.created_at.date(), pd.Timestamp("{}-12-31".format(year), tz="UTC").date())
    elif row.time_indication == "In 2010-2011":
        if which == "start":
            return pd.Timestamp("2010-1-1", tz="UTC").date()
        else:
            return pd.Timestamp("2011-12-31", tz="UTC").date()
    elif row.time_indication == "In 2012-2013":
        if which == "start":
            return pd.Timestamp("2012-1-1", tz="UTC").date()
        else:
            return pd.Timestamp("2013-12-31", tz="UTC").date()
    elif row.time_indication in _IN_INTERVAL:
        conversion = _IN_INTERVAL[row.time_indication]
        if which == "start":
            if isinstance(conversion, relativedelta):
                return row.created_at.date() - conversion
            else:
                return conversion
        else:
            return row.created_at.date()
    elif row.time_indication == "Within 10 days ago to yesterday":
        dt = row.created_at.date()
        if which == "start":
            return dt - relativedelta(days=10)
        else:
            return dt - relativedelta(days=1)
    elif row.time_indication == "A year ago tomorrow":
        return row.created_at.date() - relativedelta(years=1, days=-1)
    elif row.time_indication in _ALMOST_YEAR_AGO:
        diff = _ALMOST_YEAR_AGO[row.time_indication]
        dt = row.created_at - relativedelta(years=diff)
        if which == "start":
            return dt.date()
        else:
            return convert(dt, "M") + relativedelta(months=3, days=-1)
    elif row.time_indication in _OVER_YEAR_AGO:
        diff = _OVER_YEAR_AGO[row.time_indication]
        dt = row.created_at - relativedelta(years=diff)
        if which == "start":
            return convert(dt, "M") - relativedelta(months=3)
        else:
            return dt.date()
    elif row.time_indication == "A year or more ago":
        if which == "start":
            return convert(row.created_at, "Y") - relativedelta(years=1)
        else:
            return row.created_at.date() - relativedelta(years=1)
    elif row.time_indication == "6 years ago in the weeks after this date":
        dt = row.created_at - relativedelta(years=6)
        if which == "start":
            return convert(dt, "W")
        else:
            return convert(dt, "W") + relativedelta(weeks=4)
    elif row.time_indication == "Over a half year or so":
        dt = row.created_at.date() - relativedelta(months=6)
        if which == "start":
            return dt
        else:
            return dt - relativedelta(months=2)
    elif row.time_indication == "Over 2 hand a half years ago":
        dt = row.created_at.date() - relativedelta(months=30)
        if which == "start":
            return dt
        else:
            return dt - relativedelta(months=6)

    elif row.time_indication == "A year or two ago":
        to_year = convert(row.created_at, "Y")
        if which == "start":
            return to_year - relativedelta(years=2)
        else:
            return to_year
    else:
        return pd.NaT<jupyter_output><empty_output><jupyter_text>Then, we load the file that contains the time indications.<jupyter_code>label_fn = "data/diagnosis_labelling_depression_final.tsv"
labels = pd.read_csv(label_fn, sep="\t", index_col="tweet_id")
labels["created_at"] = pd.to_datetime(labels["created_at"], utc=True)<jupyter_output><empty_output><jupyter_text>Finally, we construct the intervals based on the time indications in the diagnosis tweets. Note that these conversions are already contained in the file named `diagnosis_labelling_depression_final.tsv`.<jupyter_code>conversions = labels[["created_at", "time_indication"]].copy()
conversions["diag_start_interval"] = conversions.apply(convert_timestamp, axis=1)
conversions["diag_end_interval"] = conversions.apply(convert_timestamp, which="end", axis=1)
conversions.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1303455153399425, 'fraction_numerical': 0.048278289435091215, 'mean_word_length': 2.561311886358889, 'pattern_counts': {'":': 75, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '153507', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 6194, 'n_tokens_neox': 5335, 'n_words': 1654, 'path': 'time_conversion.ipynb', 'repo_name': 'mctenthij/circadian_rhythms', 'size': '17047'}