Q:

Remove duplicate tweets that are 90% similar

I have extracted the tweets and i want to remove duplicate tweets. If I use padas drop_duplicates(inplace=True) it will remove 100% duplicated tweets. 
I want to know is there a way to remove that are slightly different from each other and 90% similar to each other.
Example
When will this year end? There is only misery and bad stuff! I hate 2020!
When will this year end? There are miseries and bad stuff only! I hate 2020!
These tweets are almost similar, how can I remove them

A:

There's no simple answer to your question, but a couple of naive solutions might look something like the following.
Approach 1

Firstly, you need to define a similarity metric. A common (character based) string comparison metric is Levenshtein distance, but I'd recommend having a look through fuzzywuzzy's README to find one that's appropriate for your use case. For this micro-demo, instead of using †he fuzzywuzzy package, I'm using python-levenshtein.
Secondly, compare all strings to all other tweets and compute the string similarities between them. Note that this is totally impractical if you're dealing with a large number of tweets, but let's roll with it. After comparing the strings, you can filter to get the indexes of other strings which are close matches. 
Using those indexes, we can create a graph of strings, for which I use the networkx package. This is necessary so we can extract the connect components of the graph, for which each connected component represents a network of similar strings. This is not necessarily true, since for deep graphs a string at one end won't necessarily be all that similar to a string at the other end. But in practice, it turns out to work pretty well.

Setup
import networkx as nx
import Levenshtein
import random

df = pd.DataFrame({
    "tweet":["When will this year end? There is only misery and bad stuff! I hate 2020!", 
             "When will this year end? There are miseries and bad stuff only! I hate 2020!", 
             "I am a tweet with no obvious duplicates", 
             "Tweeeeeet!", 
             "Tweeet", 
             "Tweet tweet!"]
})

Logic
def compare(tweet1, threshold=0.7):
    # compare tweets using Levenshtein distance (or whatever string comparison metric) 
    matches = df['tweet'].apply(lambda tweet2: (Levenshtein.ratio(tweet1, tweet2) >= threshold))

    # get positive matches
    matches = matches[matches].index.tolist()

    # convert to list of tuples
    return [*zip(iter(matches[:-1]), iter(matches[1:]))]

# create graph objects
nodes = df.index.tolist()
edges = [*itertools.chain(*df["tweet"].apply(compare))]

# create graphs
G = nx.Graph()
G.add_nodes_from(nodes)
G.add_edges_from(edges)

# get connected component indexes
grouped_indexes = [*nx.connected_components(G)]

# get a random choice index from each group
filtered_indexes = [random.choice([*_]) for _ in grouped_indexes]

df.loc[filtered_indexes]

Output
A filtered subset of the original tweets DataFrame.
    tweet
0   When will this year end? There is only misery ...
2   I am a tweet with no obvious duplicates
5   Tweet tweet!

Approach 2
We can use an unsupervised learning algorithm to cluster strings together, something like k-means This is your bread and butter of unsupervised algorithms, and it has the drawback that you have to know the optimal number of clusters in advance, or more typically work it out via testing. But it has the great advantage that if you're adding more tweets to your dataset, you can quickly apply your clustering model and find similar tweets.
There's a million and one tutorials on how to do this, but the basic approach here would be to (1) clean your text, (2) convert your text into a TFIDF, (3) compute a similarity metric (cosine similarity is common) between each document pair, (4) then train your k-means (or similar) model.
If you're interested in this approach, here's a few random tutorials to follow I found after a quick Google.

https://towardsdatascience.com/k-means-clustering-8e1e64c1561c
http://brandonrose.org/clustering
https://iq.opengenus.org/document-clustering-nlp-kmeans/
...

Hope this helps!

