<jupyter_start><jupyter_text>During mylastsemester of DCC, I had to maintain a wordpress blog. After doingmy posts I never looked back at it untilI got an email asking to moderate the comments. On a whim I decided to check back in to find over 2800 comments made by spam bots. Even while I'm writing this dozens are being added. So I decided to analyze them. Why? I'm messed up in the head and bored.<jupyter_code>word_dict = {}
ip_dict = {}
author_dict = {}
website_dict = {}
email_dict = {}
hour_dict = {}
time_dict = {}
num_mistakes_dict = {}
error_dict = {}

with codecs.open('data/spam_data.csv','r',encoding="utf-8") as spam_data:
    reader = csv.DictReader(spam_data)
    for row in reader:
        # Getting word count
        comment = row['Comment']
        
        for word in comment.split():
            word = re.sub("[\?\.,\“!\"\”\(\)]","",word)
            #print(word)
            if word in word_dict:
                word_dict[word] += 1
            else:
                word_dict[word] = 1
    
    
        # Getting IP count
        ip = row['IP Address']
        if ip in ip_dict:
            ip_dict[ip] += 1
        else:
            ip_dict[ip] = 1
        # Getting author count
        author = row['Author']
        if author in author_dict:
            author_dict[author] += 1
        else:
            author_dict[author] = 1
        # Getting website count
        website = row['Website']
        if website in website_dict:
            website_dict[website] += 1
        else:
            website_dict[website] = 1
        # Getting email count
        email = row['Email']
        if email in email_dict:
            email_dict[email] += 1
        else:
            email_dict[email] = 1

        # Getting hour frequency
        hour = re.sub(":[\d]{2}","",row['Submit Time'])

        hour_number = re.match("[\d]+",hour).group(0)
        if "pm" in hour:
            hour = str((int(hour_number)+12)%24)+":00"
        else:
            hour = str((int(hour_number))%24)+":00" 
        if len(hour) < 5:
            hour = "0"+hour
            
        if hour in hour_dict:
            hour_dict[hour] += 1
        else:
            hour_dict[hour] = 1
            
        #Getting time frequency by converting to military time
        time = re.match("([\d]+):([\d]{2})",row['Submit Time'])
        #print(time.groups())
        time_string = ""
        
        if "pm" in row['Submit Time']:
            time_string = str((int(time.groups()[0])+12)%24)+":"+str(time.groups()[1])
        else:
            time_string = str(int(time.groups()[0]))+":"+str(time.groups()[1])
        time_check = re.match("([\d]+):([\d]{2})",time_string)
        if len(time_check.groups()[0]) < 2:
            time_string = "0"+time_string
        if time_string in time_dict:
            time_dict[time_string] += 1
        else:
            time_dict[time_string] = 1
        
        # Grammar checking - number of errors
        parse_tree = xml.etree.ElementTree.fromstring(row['Grammar Check'])
        num_mistakes_dict[row['Comment Number']] = [len(parse_tree),row['Comment']]
        error_dict[row['Comment Number']] = parse_tree<jupyter_output><empty_output><jupyter_text>Now, let's see what are the most common words used by these commentors.<jupyter_code>words_sorted = reversed(sorted(word_dict.items(), key=lambda x: x[1]))
for (word,count) in words_sorted:
    print(word,"-",count)<jupyter_output>I - 4314
to - 4115
the - 3260
you - 2919
your - 2443
a - 2386
this - 2271
and - 2240
for - 2203
of - 1786
is - 1299
it - 1222
in - 1207
on - 1051
that - 1051
site - 1034
my - 1029
have - 1016
with - 898
be - 837
are - 793
as - 754
me - 734
like - 724
web - 685
blog - 678
more - 662
post - 644
website - 608
am - 606
very - 605
will - 601
not - 577
You - 557
up - 556
just - 505
Thanks - 494
really - 494
all - 474
 - 460
been - 439
from - 435
information - 424
some - 405
here - 403
but - 398
out - 393
I’m - 388
so - 388
info - 378
great - 375
what - 368
Thank - 364
was - 359
time - 355
an - 351
do - 333
could - 331
about - 331
It - 324
good - 313
i - 311
posts - 308
The - 307
can - 306
get - 303
look - 300
lot - 300
much - 294
found - 291
know - 288
made - 274
if - 273
people - 270
read - 270
how - 267
by - 266
article - 260
make - 255
actually - 254
well - 250
writing - 247
new - 246
work - 243
I’ve - 240
has - 240
us - 233
again - 233
excellent - 228
This - 228
Your - 226
which - 221
or[...]<jupyter_text>So there are many of the usually boring repeated words: "I", "the", "a", etc. As with almost anytext, these dominate the top of words used. However further down there are moreinteresting words that crop up: "site", "web", "blog", and "website." Most of these comments wanted to heap comments upon my website or blog (post) and/or try toconvince people to visittheir websites and links. WordPress also stores the IP Adresses of every comment author. Below are listed the addresses.<jupyter_code>ips_sorted = reversed(sorted(ip_dict.items(), key=lambda x: x[1]))
for (ip,count) in ips_sorted:
    print(ip,"-",count)<jupyter_output>172.16.17.32 - 2749
172.16.31.10 - 86
172.16.31.10 - 20
172.16.31.10 - 3
172.16.58.3 - 3
192.168.127.12 - 1
172.16.31.10 - 1
172.16.17.32 - 1
172.16.31.10 - 1
172.16.17.32 - 1
172.16.31.10 - 1
192.168.3.11 - 1
172.16.58.3 - 1
172.16.17.32 - 1<jupyter_text>Wow! This one IP address, "172.16.17.32", seems to be the main source of all this spam! What could it be? Oh ... "dcc.umd.edu." Well either WordPress messed up, other IP Adresses are found by commenting via external means, or the commenters are good at hiding their IP addresses.  Now let's see what the author names given.<jupyter_code>authors_sorted = reversed(sorted(author_dict.items(), key=lambda x: x[1]))
for (author,count) in authors_sorted:
    print(author,"-",count)<jupyter_output>Technology - 85
 Home Improvement - 80
 House & Home Improvement - 70
 Business - 68
 Automotive - 45
 Arts - 43
 Sports - 40
 Wedding - 33
 Education - 33
 Pets - 32
 Travel & Leisure - 29
 Pets/animal - 25
 Health - 25
 Women - 25
 technology - 25
 Career & Jobs - 24
 Finance - 24
 Dating - 24
 Food - 23
 Health & Fitness - 23
 Jewelry - 23
 Woman - 22
 Fashion - 20
 Computer & Internet - 18
 Recreation - 18
 Game - 17
 Travel - 17
 business - 16
 Healthy Life - 16
 travel - 16
 Law - 15
 health and fitness - 15
 home improvement - 15
 fashion - 14
 education - 13
 Games - 13
 Web Design - 10
 home improvement stores - 9
 sports - 8
 real estate - 8
 automotive - 8
 drywall repair - 8
 fireplace remodel - 8
 business intelligence - 8
 health articles - 8
 home remodeling - 8
 Law And Legal - 7
 health department - 7
 kitchen remodel cost - 7
 business development - 7
 business ideas - 7
 mental health - 7
 home based business - 7
 Business & Finance - 6
 Pets & Animal - 6
 Computer [...]<jupyter_text>So as can be seen,these authors have some odd names. They seem to be topics or website links. It's almost as if these aren't genuine comments. Now let's see what type of websites these commentors have.<jupyter_code>website_sorted = reversed(sorted(website_dict.items(), key=lambda x: x[1]))
for (website,count) in website_sorted:
    print(website,"-",count)<jupyter_output>http://www.jbxqq.com/ - 24
http://www.asiel-breda.net/ - 16
http://www.znbmh.com/ - 15
http://www.xptnn.com/ - 15
http://www.znbmt.com/ - 14
http://bengalandburmese.com/ - 13
http://audiolibjs.org/ - 13
http://trafficrider.org/ - 13
http://www.lattice-media.com/ - 13
http://www.jfknn.com/ - 13
http://www.mhrestaurants.com/ - 12
http://rf-summit.com/ - 12
http://www.carbonoffsetlist.org/ - 12
http://www.universal-sc.com/ - 12
http://ustechsregister.com/ - 12
http://www.spirit-of-hanf.com/ - 12
http://saveroxbury.org/ - 11
http://thaigeruppercut.com/ - 10
http://www.selimonder.com/ - 10
http://www.refreyell.net/ - 10
http://www.mltwy.com/ - 10
http://www.hlswy.com/ - 10
http://www.ad-techhamburg.com/ - 9
http://raisedgardenbeds.net/ - 9
http://www.allthatantoine.com/ - 9
http://www.wktny.com/ - 9
http://www.proposaldorpresid.com/ - 8
http://www.duluthinfo.com/ - 8
http://downtownkansascityhotels.net/ - 8
http://logo-skateboard.com/ - 8
http://www.otonib.com/ - 8
http://lbe-md.com/ - 8
ht[...]<jupyter_text>There are a wide variety of websites listed here. Admittedly I haven't taken the pleasure of investigating all of them but they seem to be weirdly vacant or malformed blogs. Some have content but they're of the same odd mish-mash of attempted english as most comments. Attempting to be perceived as genuine content but, possibly due to a poor understanding of english or poor automation of content (with no care for quality), are just shallow websites made to be seen.Forwhat purpose? I have noclue. Possible ad revenue from clicks/visits? SEO? I have nary a clue. Now let's see what type of emails these commment authors use.<jupyter_code>emails_sorted = reversed(sorted(email_dict.items(), key=lambda x: x[1]))
for (email, count) in emails_sorted:
    print(email,"-",count)<jupyter_output><EMAIL> - 4
<EMAIL> - 3
<EMAIL> - 3
<EMAIL> - 3
<EMAIL> - 3
<EMAIL> - 3
<EMAIL> - 3
<EMAIL> - 3
<EMAIL> - 3
<EMAIL> - 3
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL> - 2
<EMAIL>[...]<jupyter_text>Nothing seems all that unusual here. That might be because emails aren'thard to fake,they're a simple pattern: randomname + number then add an "@" followed by a well known email provider. It's pretty spread out list and I don't really think there is much to gain because it's too hard to ascertain if these are "real" or "active" emails. Another area of curiosity is the time allthese comments were submitted.<jupyter_code>hours_sorted = sorted(hour_dict.items(), key=lambda x:x[0])
for (hour,count) in hours_sorted:
    print(hour,"-",count)<jupyter_output>00:00 - 74
01:00 - 212
02:00 - 208
03:00 - 189
04:00 - 214
05:00 - 202
06:00 - 142
07:00 - 109
08:00 - 120
09:00 - 103
10:00 - 98
11:00 - 108
12:00 - 197
13:00 - 62
14:00 - 65
15:00 - 55
16:00 - 42
17:00 - 38
18:00 - 51
19:00 - 72
20:00 - 49
21:00 - 133
22:00 - 172
23:00 - 155<jupyter_text>It seems that the most active times for comments were wearlyin the morning. Maybe this has something to do with time zones fromthe spam comments. The highest density is around 1 AM to 6 AM. Then let's try sorting every single minute.<jupyter_code>times_sorted = sorted(time_dict.items(), key=lambda x:x[0])
for (time,count) in times_sorted:
    print(time,"-",count)<jupyter_output>00:00 - 1
00:01 - 1
00:02 - 2
00:03 - 1
00:05 - 2
00:07 - 2
00:09 - 2
00:10 - 2
00:12 - 1
00:13 - 2
00:15 - 3
00:16 - 5
00:17 - 1
00:18 - 4
00:20 - 1
00:22 - 1
00:23 - 3
00:24 - 4
00:25 - 2
00:26 - 1
00:27 - 3
00:29 - 1
00:30 - 1
00:34 - 1
00:35 - 1
00:37 - 2
00:38 - 1
00:39 - 3
00:40 - 1
00:41 - 3
00:42 - 1
00:43 - 2
00:44 - 1
00:45 - 1
00:48 - 1
00:49 - 3
00:52 - 2
00:53 - 2
00:55 - 1
00:59 - 2
01:00 - 4
01:01 - 3
01:02 - 2
01:03 - 4
01:04 - 2
01:05 - 4
01:06 - 5
01:07 - 7
01:08 - 6
01:09 - 3
01:10 - 6
01:11 - 7
01:12 - 1
01:13 - 2
01:14 - 6
01:15 - 3
01:17 - 1
01:18 - 6
01:19 - 3
01:20 - 7
01:21 - 1
01:22 - 2
01:23 - 8
01:24 - 6
01:25 - 6
01:26 - 6
01:27 - 1
01:28 - 1
01:29 - 2
01:30 - 2
01:31 - 5
01:32 - 4
01:33 - 5
01:34 - 5
01:36 - 7
01:37 - 2
01:38 - 7
01:39 - 1
01:40 - 4
01:41 - 6
01:42 - 3
01:43 - 3
01:44 - 2
01:45 - 1
01:46 - 2
01:47 - 3
01:48 - 1
01:49 - 4
01:50 - 2
01:51 - 1
01:52 - 2
01:53 - 3
01:54 - 3
01:55 - 5
01:56 - 5
01:57 - 4
01:58 - 4
01:59 - 1
02:00 - 2
02:01 - 6
[...]<jupyter_text>So again, the same trend can be seen. However I guess it's harderto see withoutthe aid of a wellmade histogram.<jupyter_code>'''time_counts = hour_dict.values()
times = hour_dict.keys()
bar_x_locations = np.arange(len(time_counts))
plt.bar(bar_x_locations, time_counts, align = 'center')
plt.xticks(bar_x_locations, times)
plt.grid()
plt.show()'''<jupyter_output><empty_output><jupyter_text>After all these counts, I decided to take a look at the grammar of all these comments. I did this by sending each comment to be checked by the "After the Deadline" API. Let's see what the most error-filled comments are.<jupyter_code>sorted_mistakes = reversed(sorted(num_mistakes_dict.items(), key=lambda x:x[1][0]))
for (c_id,comment) in sorted_mistakes:
    print(c_id,"-",comment[0])<jupyter_output>177 - 53
279 - 26
2171 - 26
176 - 20
180 - 20
276 - 20
191 - 19
278 - 18
2179 - 18
35 - 17
37 - 17
34 - 16
110 - 14
123 - 14
2180 - 14
140 - 13
91 - 12
277 - 12
1832 - 12
45 - 11
1780 - 11
2730 - 11
2813 - 11
435 - 10
1751 - 10
1886 - 10
2172 - 10
2178 - 10
2197 - 10
2316 - 10
582 - 9
1254 - 9
1554 - 9
1618 - 9
2061 - 9
2187 - 9
2347 - 9
2426 - 9
2542 - 9
308 - 8
322 - 8
423 - 8
603 - 8
805 - 8
893 - 8
998 - 8
1148 - 8
1158 - 8
1248 - 8
1449 - 8
1458 - 8
1654 - 8
1760 - 8
2027 - 8
2055 - 8
2077 - 8
2111 - 8
2280 - 8
2556 - 8
65 - 7
280 - 7
429 - 7
463 - 7
496 - 7
592 - 7
655 - 7
750 - 7
857 - 7
953 - 7
979 - 7
1241 - 7
1318 - 7
1330 - 7
1456 - 7
1633 - 7
1638 - 7
1693 - 7
1979 - 7
2032 - 7
2092 - 7
2122 - 7
2136 - 7
2318 - 7
2412 - 7
2438 - 7
2483 - 7
2550 - 7
2635 - 7
2807 - 7
43 - 6
54 - 6
105 - 6
153 - 6
234 - 6
247 - 6
248 - 6
259 - 6
261 - 6
268 - 6
375 - 6
386 - 6
390 - 6
416 - 6
440 - 6
453 - 6
596 - 6
634 - 6
685 - 6
688 - 6
698 - 6
744 - 6
755 - 6
866 - 6
891 - 6
1013 - 6
1023[...]<jupyter_text>Oh wow! 53 ERRORS? What kind of crazy comment could this be?<jupyter_code>print(num_mistakes_dict['177'][1])<jupyter_output>Personne
 ne vit sans problème et après expérience j’ai compris qu’il avait aucun
 problème sans réelle solution. Je vous conseile de visiter de visiter 
Blackma : Medium Voyant Africain Compétent et sérieux et spécialiste 
africains des travaux occultes. Ils vous apporteront des solutions 
adéquates dans tous les domaines à savoir: Amour et retour d’affection, 
Formule de richesse, Chance au jeux, réussite des examens et emploi, 
désenvoûte ment, protection contre tous dangers, Désintoxication alcool,
 guérrir la folie. Retour de l’être aimé en 72H. Problème familiaux et 
de voisinage.Amour fidélité et prospérité à vie. RÉSULTATS IMMÉDIAT ET 
EFFICACES.<jupyter_text>Ah I see! Obviously the comment author made the cardinal grammatical sin of speaking French. So I guess "After the Deadline" only works for english. Let's see some more of these comments.<jupyter_code>print(num_mistakes_dict['279'][1])
print("-------------------------------------------------------------------------------------------")
print(num_mistakes_dict['2171'][1])
print("-------------------------------------------------------------------------------------------")
print(num_mistakes_dict['176'][1])<jupyter_output>I
 thinkI feelI believe this isthat is one of theamong the so muchsuch a 
lotmost importantsignificantvital informationinfo for me. And i’mi am 
satisfiedgladhappy readingstudying your article. HoweverBut wannawant 
toshould observationremarkstatementcommentary on fewsome 
generalcommonbasicnormal thingsissues, The websitesiteweb site 
tastestyle is perfectidealgreatwonderful, the articles is in point of 
factactuallyreallyin realitytruly excellentnicegreat : D. Just 
rightGoodExcellent taskprocessactivityjob, cheers
-------------------------------------------------------------------------------------------
HiHello,
 Neat post. There isThere’s a probleman issue with yourtogether with 
youralong with your siteweb sitewebsite in internetweb explorer, 
maymightcouldwould checktest this? IE stillnonetheless is the 
marketplacemarket leaderchief and a largea gooda biga huge part 
ofsection ofcomponent toportion ofcomponent ofelement of other 
folksfolksother peoplepeople will [...]<jupyter_text>So, apparently the most error prone comments are either a mish-mash of several words together or commit the vile sin of being in french. So given thee examples, there are obviously some really wonky comments. However are some of the comments with zero errors great literature?<jupyter_code>print(num_mistakes_dict['2854'][1])
print("-------------------------------------------------------------------------------------------")
print(num_mistakes_dict['2849'][1])<jupyter_output>When I got that it remained in a major box plus all the sky was actually pulled out from the bundle the bed remained in.
-------------------------------------------------------------------------------------------
Well I really enjoyed reading it. This article offered by you is very effective for correct planning.<jupyter_text>So these comments aren't grammatically incorrect butthey don't exactly read all that well. That's indicative ofmostof these comments. Whether or not they have errors, most read like they were passed through google translate a few times or , the more probable answer, is that it was written by someone without the best grasp of English. Now let's see the distribution errors we have.<jupyter_code>error_type_dict = {}
for key in error_dict:
    for child in error_dict[key]:
        error_type = child.find('type').text
        if error_type not in error_type_dict:
            error_type_dict[error_type] = 1
        else:
            error_type_dict[error_type] += 1
for key in error_type_dict:
    print(key,"-",error_type_dict[key])<jupyter_output>suggestion - 1770
grammar - 3008
spelling - 1396<jupyter_text>It seems that the most common type of mistakes are grammatical ones. Now let's see what are the most misspelled words.<jupyter_code>misspelled_words = {}
for key in error_dict:
    for child in error_dict[key]:
        error_type = child.find('type').text
        if error_type == "spelling":
            word = child.find('string').text
            if word not in misspelled_words:
                misspelled_words[word] = 1
            else: 
                misspelled_words[word] += 1
                
wrong_words = reversed(sorted(misspelled_words.items(),key=lambda x:x[1]))
for (word,count) in wrong_words:
    print(word,"-",count)<jupyter_output>thats - 49
rss - 45
site - 33
hadnt - 29
bloglist - 28
thisK - 23
informations - 23
Hiya - 23
digg - 22
infos - 22
Heya - 22
cant - 21
youll - 20
wouldve - 20
Googleing - 18
msn - 18
whoah - 18
HelloThis - 17
Thx - 14
ideea - 13
formerly - 13
boringK - 13
MEILLEUR - 12
vous - 12
Adwords - 12
awsome - 12
siteI - 12
alot - 11
favorI - 11
websiteI - 11
theyll - 11
Awsome - 11
hi - 11
cannt - 11
favorIm - 11
Mixview - 10
thnkx - 10
Zunes - 8
iRivers - 8
Sansas - 8
64kbps - 8
access - 8
seek - 8
prÃªt - 7
invstigating - 7
crÃ©dit - 6
blogHave - 6
edginess - 6
know - 6
websiteKeep - 6
possiblity - 6
alsoKI - 6
FRAIS - 5
AUCUN - 5
financement - 5
je - 5
OLED - 5
neatly-favored - 5
examinating - 5
neatly-preferred - 5
smartly-liked - 5
usess - 5
kind-heartedness - 5
siteKeep - 5
blogKeep - 5
lavance - 4
investissement - 4
recherchez - 4
leur - 4
reste - 4
sÃ©rieux - 4
aucun - 4
tous - 4
conseille - 4
employess - 4
wordpress - 4
now - 4
blogBest - 4
infoPerfect - 4
siteHave - 4
discret - 4
labt[...]
{'fraction_non_alphanumeric': 0.13442537701508062, 'fraction_numerical': 0.10696827873114925, 'mean_word_length': 2.53836246550138, 'pattern_counts': {'":': 6, '<': 90, '<?xml version=': 0, '>': 88, 'https://': 0, 'lorem ipsum': 0, 'www.': 21, 'xml': 1}, 'pii_count': 15, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '107903', 'license': 'WTFPL', 'n_tokens_mistral': 8399, 'n_tokens_neox': 6715, 'n_words': 2389, 'path': 'previous_ipython_version/2,854 Comments.ipynb', 'repo_name': 'sidtuli/spam-analysis', 'size': '19777'}