<jupyter_start><jupyter_text>爬取弹幕的流程1. 取Season id2. 取episode id， cid3. 取弹幕 **api = 接口**  1. 取Season ID<jupyter_code>Image(filename = "./seasonID.png")
1689,5451,2967,5559,6018
url = 'https://bangumi.bilibili.com/anime/2967/play#76675'
text = requests.get(url).text
soup = BeautifulSoup(text,'lxml')
print(soup.prettify())<jupyter_output><empty_output><jupyter_text>2. 取episod id, 取cid<jupyter_code>PATH = "./取cid.png"
Image(filename = PATH, width=700, height=700)<jupyter_output><empty_output><jupyter_text>3. 爬弹幕<jupyter_code>url_fanwai = 'https://bangumi.bilibili.com/web_api/get_ep_list?season_id=5451'
import requests
text = requests.get(url_fanwai).text

text
import json
text_js = json.loads(text)
text_js
text_js['result']
text_js['result'][0]
text_js['result'][0]['cid']
Image(filename ='danmu_api.png')
第一个参数是弹幕出现的时间 以秒数为单位。
第二个参数是弹幕的模式1..3 滚动弹幕 4底端弹幕 5顶端弹幕 6.逆向弹幕 7精准定位 8高级弹幕
第三个参数是字号， 12非常小,16特小,18小,25中,36大,45很大,64特别大
第四个参数是字体的颜色 以HTML颜色的十进制为准
第五个参数是Unix格式的时间戳。基准时间为 1970-1-1 08:00:00
第六个参数是弹幕池 0普通池 1字幕池 2特殊池 【目前特殊池为高级弹幕专用】
第七个参数是发送者的ID，用于“屏蔽此弹幕的发送者”功能
第八个参数是弹幕在弹幕数据库中rowID 用于“历史弹幕”功能。
<d p="304.97900,1,25,15138834,1508663765,0,d24ecf2b,3912870146">不要哭，不然眼泪会被冻住的.....</d>
<d p="390.29400,1,25,16777215,1508663908,0,751e6653,3912876862">背后即是祖国，我们无路可退</d>
<d p="5.06000,1,25,16777215,1508664522,0,198e806e,3912902369">第一第一万年第一</d>
<d p="174.24200,1,25,16777215,1508669634,0,887407a6,3913112815">勿谓言之不预也</d>
<d p="18.30700,1,25,16711680,1508669810,0,7df9b0ba,3913120335">中国红</d>
#有问题！最多只能显示1000条，需要爬去所有历史弹幕，怎么办？
#history_url_timestamp_list = "https://comment.bilibili.com/rolldate,cid"
#all_history_url = "https://comment.bilibili.com/dmroll,timestamp,cid"<jupyter_output><empty_output><jupyter_text>取历史弹幕<jupyter_code>cid = '8795676'
history_url_timestamp_list = "https://comment.bilibili.com/rolldate"
all_history_url = "https://comment.bilibili.com/dmroll"

timestamp_list_url=history_url_timestamp_list+","+cid
print(timestamp_list_url)
timestamps = requests.get(timestamp_list_url).text
all_timestamps=json.loads(timestamps)
all_timestamps
all_timestamps[0]['timestamp']
danmu_url="https://comment.bilibili.com/dmroll,{},{}".format(all_timestamps[0]['timestamp'],cid)
from bs4 import BeautifulSoup
danmu = requests.get(danmu_url).text
soup = BeautifulSoup(danmu,'lxml')
print(soup)
cid = '8795676'
history_url_timestamp_list = "https://comment.bilibili.com/rolldate"
all_history_url = "https://comment.bilibili.com/dmroll"

timestamp_list_url=history_url_timestamp_list+","+cid
print(timestamp_list_url)
timestamps = requests.get(timestamp_list_url).text

all_timestamps=json.loads(timestamps)
rows = []

for list in range(len(all_timestamps)):
    danmu_url = "https://comment.bilibili.com/dmroll,{},{}".format(all_timestamps[list]['timestamp'],cid)
    danmu = requests.get(danmu_url).text
    soup = BeautifulSoup(danmu,'lxml')
    if list%100==0:
        print(list)
    for i in soup.select('d'):
        i_text = i.get_text() # 进度条神马情况
        i_p = i['p'].split(',') # 539.89， 1， 25，...  # 8个元素
        i_p.append(i_text) # 9个元素
        rows.append(i_p)
        
records = pd.DataFrame(rows,columns = ['time','format','font','color','data_unix','danmuchi','ID','rowID','danmu'])
url_fanwai = 'https://bangumi.bilibili.com/web_api/get_ep_list?season_id=5451'
text = requests.get(url_fanwai).text
text_json = json.loads(text)
episode_id_list = text_json['result']
episode_id_list[0]['cid']<jupyter_output><empty_output><jupyter_text>采集番外篇所有集的弹幕<jupyter_code>history_url_timestamp_list = "https://comment.bilibili.com/rolldate"
all_history_url = "https://comment.bilibili.com/dmroll"

for j, episode in enumerate(episode_id_list):
    cid = str(episode['cid'])
    timestamp_list_url=history_url_timestamp_list+","+cid
    print(timestamp_list_url)
    timestamps = requests.get(timestamp_list_url).text

    all_timestamps=json.loads(timestamps)
    rows = []

    for list in range(len(all_timestamps)):
        danmu_url = "https://comment.bilibili.com/dmroll,{},{}".format(all_timestamps[list]['timestamp'],cid)
        danmu = requests.get(danmu_url).text
        soup = BeautifulSoup(danmu,'lxml')
        if list%100==0:
            print(list)
        for i in soup.select('d'):
            i_text = i.get_text() # 进度条神马情况
            if not isinstance(i_text,str):
                i_text = ' '
            i_p = i['p'].split(',') # 539.89， 1， 25，...  # 8个元素
            i_p.append(i_text) # 9个元素
            rows.append(i_p)
        
    records = pd.DataFrame(rows,columns = ['time','format','font','color','data_unix','danmuchi','ID','rowID','danmu'])
    filename = '番外篇第{}集.csv'.format((j+1))
    records.to_csv(filename)<jupyter_output><empty_output><jupyter_text>Sentimental Analysis<jupyter_code>url = 'https://comment.bilibili.com/8795676.xml'
text = requests.get(url).text
soup = BeautifulSoup(text,'lxml')
useful_cont = soup.find_all('d')
useful_cont
rows = []

for i in soup.select('d'):
    i_text = i.get_text() # 进度条神马情况
    if not isinstance(i_text,str):
        i_text = ' '
    i_p = i['p'].split(',') # 539.89， 1， 25，...  # 8个元素
    i_p.append(i_text) # 9个元素
    rows.append(i_p)
    
records = pd.DataFrame(rows,columns = ['time','format','font','color','data_unix','danmuchi','ID','rowID','danmu'])
filename = '番外篇第{}集_current.csv'.format(1)
records.to_csv(filename)
records<jupyter_output><empty_output><jupyter_text>1. 弹幕的密度 2. 弹幕分词 去除stopwords 3. 弹幕情感<jupyter_code># 1. 密度
time = records['time']


type(time)
time_float = []
for i in time:
    time_float.append(float(i))
time_float
plt.figure(figsize = [11,9])
n, bins, patches = plt.hist(time_float, 100, facecolor='purple')
plt.show()
# 分词  结巴分词
!pip install jieba
import jieba
fenci = jieba.cut("我在北京上大学，我上的是比清华好的北京大学",cut_all=True)
print("/ ".join(fenci))
fenci = jieba.cut("我在北京读大学，我读的是比清华好的北京大学",cut_all=False)
print("/ ".join(fenci))
fenci = jieba.cut("我爱北京天安门，五环比六环少一环，学好python就不是低端劳动力了，呜呜",cut_all=False)
print("/ ".join(fenci))
jieba.suggest_freq("我读的", tune=True)
fenci = jieba.cut("我在北京读大学，我读的是比清华好的北京大学",cut_all=False)
print("/ ".join(fenci))
fenci = jieba.cut("我爱北京天安门，五环比六环多一环，学好python就不是低端劳动力了，呜呜",cut_all=False)
print("/ ".join(fenci))
jieba.suggest_freq(("比","六环"), tune=True)
fenci = jieba.cut("我爱北京天安门，五环比六环多一环，学好python就不是低端劳动力了，呜呜",cut_all=False)
print("/ ".join(fenci))
jieba.suggest_freq("六环", tune=True)
fenci = jieba.cut("我爱北京天安门，五环比六环多一环，学好python就不是低端劳动力了，呜呜",cut_all=False)
print("/ ".join(fenci))
fenci = jieba.cut("我不去上幼儿园，没意思，吃不饱还要被打针，结巴分词的效果还不错哟，周杰伦都要喜欢了",cut_all=False)
print("/ ".join(fenci))
jieba.suggest_freq("结巴分词", tune=True)

fenci = jieba.cut("我不去上幼儿园，没意思，吃不饱还要被打针，结巴分词的效果还不错哟，周杰伦都要喜欢了",cut_all=False)
print("/ ".join(fenci))
comments = " ".join(records['danmu'])
comments
fenci_all = list(jieba.cut(comments,cut_all=False))

fenci_all
nltk
!pip install snownlp
s1 = SnowNLP("杀人放火")
s1.sentiments
s1 = SnowNLP("")
s1.sentiments
from snownlp import SnowNLP
records['Senti'] = pd.Series()
for i,comment in enumerate(records['danmu']):
    records.loc[i,'Senti'] = SnowNLP(comment).sentiments
records
plt.scatter(time_float,records['Senti'])
#(records['time'],records['Senti'],c='purple',s=0.2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17071468453378, 'fraction_numerical': 0.05513679508654383, 'mean_word_length': 6.215508559919436, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '608635', 'license': 'MIT', 'n_tokens_mistral': 3587, 'n_tokens_neox': 3587, 'n_words': 361, 'path': 'danmu_crawler_BA.ipynb', 'repo_name': 'wyt12345/10__-', 'size': '7164'}