<jupyter_start><jupyter_text>Install dependencies<jupyter_code>!pip install --quiet transformers<jupyter_output>[K     |████████████████████████████████| 3.4 MB 5.0 MB/s 
[K     |████████████████████████████████| 596 kB 45.3 MB/s 
[K     |████████████████████████████████| 3.3 MB 47.6 MB/s 
[K     |████████████████████████████████| 895 kB 64.5 MB/s 
[K     |████████████████████████████████| 67 kB 4.6 MB/s 
[?25h<jupyter_text>Import<jupyter_code>from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from transformers import pipeline
import torch<jupyter_output><empty_output><jupyter_text>Init<jupyter_code>device = 'cuda' if torch.cuda.is_available() else 'cpu'
print ("Device:", device)
torch.set_grad_enabled(False)
tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-cnn-12-6")
model = AutoModelForSeq2SeqLM.from_pretrained("sshleifer/distilbart-cnn-12-6").to(device)
model = model.to(device)
# device = 0 for cuda
summarizer = pipeline("summarization", device=0)<jupyter_output><empty_output><jupyter_text>Logic<jupyter_code>def get_summary(input, tokenizer, model):
  txt = input['text']
  minl = input.get('min_length', 75)
  maxl = input.get('max_length', 150)
  print('minl:', minl)
  print('maxl:', maxl)

  inputs = tokenizer([txt], max_length=1024, truncation=True, return_tensors='pt').to(device)
  print('inputs:', inputs)
  output_ids = model.generate(inputs['input_ids'], num_beams=3, num_return_sequences=1, no_repeat_ngram_size=2, min_length=minl, max_length=maxl, early_stopping=True)
  print('output_ids:', output_ids)
  dec = [tokenizer.decode(ids, skip_special_tokens=True, clean_up_tokenization_spaces=True) for ids in output_ids]
  print('dec:', dec)
  output = dec[0].strip()
  return {'summary': output}<jupyter_output><empty_output><jupyter_text>Run<jupyter_code>text = """
<NAME> has shown again he can influence the digital currency market with just his tweets. After saying that his electric vehicle-making company Tesla will not accept payments in Bitcoin because of environmental concerns, he tweeted that he was working with developers of Dogecoin to improve system transaction efficiency. Following the two distinct statements from him, the world's largest cryptocurrency hit a two-month low, while Dogecoin rallied by about 20 percent. The SpaceX CEO has in recent months often tweeted in support of Dogecoin, but rarely for Bitcoin.  In a recent tweet, Musk put out a statement from Tesla that it was concerned about the rapidly increasing use of fossil fuels for Bitcoin (price in India) mining and transaction, and hence was suspending vehicle purchases using the cryptocurrency.  A day later he again tweeted saying, To be clear, I strongly believe in crypto, but it can't drive a massive increase in fossil fuel use, especially coal. It triggered a downward spiral for Bitcoin value but the cryptocurrency has stabilised since.   A number of Twitter users welcomed Musk's statement. One of them said it's time people started realising that Dogecoin is here to stay and another referred to Musk's previous assertion that crypto could become the world's future currency.
"""
input = {'text': text, 'min_length': 30, 'max_length': 60}
output = get_summary(input, tokenizer, model)
print('output:', output)
outputs = summarizer(text, min_length=30, max_length=60)
result = ''
if len(outputs)>0:
  result = outputs[0].get('summary_text','')

print('summary:', result)<jupyter_output>summary:  Elon Musk has shown again he can influence the digital currency market with just his tweets . The SpaceX CEO has in recent months often tweeted in support of Dogecoin, but rarely for Bitcoin . In a recent tweet, Musk put out a statement from Tesla that it was concerned about the rapidly
{'fraction_non_alphanumeric': 0.13381410256410256, 'fraction_numerical': 0.01655982905982906, 'mean_word_length': 6.066037735849057, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '349435', 'license': 'MIT', 'n_tokens_mistral': 1319, 'n_tokens_neox': 1149, 'n_words': 392, 'path': 'text-summarization/summarizer-details.ipynb', 'repo_name': 'juancolamendy/artificial-intelligence', 'size': '3747'}