Blog

Other than that, the code should be simple to read and understand, given all the comments I added throughout the code😅 However, in case you found something unclear and you need some help, don’t hesitate to drop your questions on the forum!

In this method, we’ll be directly comparing both articles as a whole without chunking them by converting both of them into vector embeddings. Then, using cosine similarity, we’ll see if they’re similar to each other.

from scipy.spatial.distance import cosine
import time 
import resources
import openai
def convert_to_vector(text):
"""
Converts a given piece of text into a vector using OpenAI's embeddings API.
"""
text = text.replace("\n", " ")  # Remove newlines for consistent embedding processing
response = openai.embeddings.create(
input=[text],
model="text-embedding-3-small"
)
return response.data[0].embedding  # Return the embedding vector
def calculate_cosine_similarity(vec1, vec2):
"""
Calculates the cosine similarity between two vectors, representing the similarity of their originating texts.
"""
return 1 - cosine(vec1, vec2)  # The cosine function returns the cosine distance, so 1 minus this value gives similarity
def is_similarity_significant(similarity_score):
"""
Determines if a cosine similarity score indicates significant semantic similarity, implying potential plagiarism.
"""
threshold = 0.7  # Define a threshold for significant similarity; adjust based on empirical data
return similarity_score >= threshold  # Return True if the similarity is above the threshold, False otherwise
def search_semantically_similar(text_to_check):
"""
Compares the semantic similarity between the input text and a predefined article text.
It returns a list containing the similarity score and a boolean indicating whether
the similarity is considered significant.
"""
result = []  # Initialize an empty list to store the similarity score and significance flag
input_vector = convert_to_vector(text_to_check)  # Convert the input text to a vector using an embedding model
article_text = resources.article_two  # texts.two contains the text of the article to compare with
article_vector = convert_to_vector(article_text)  # Convert the article text to a vector
similarity = calculate_cosine_similarity(input_vector, article_vector)  # Calculate the cosine similarity between the two vectors
result.append(similarity)  # Append the similarity score to the list
result.append(is_similarity_significant(similarity))  # Append the result of the significance check to the list
return result  # Return the list containing the similarity score and significance flag
def calculate_plagiarism_score(text):
"""
Calculates the plagiarism score of a given text by comparing its semantic similarity
with a predefined article text. The score is expressed as a percentage.
"""
data = search_semantically_similar(text) # Obtain the similarity data for the input text
data[0] = data[0] * 100  # Convert the first item in the data list (similarity score) to a percentage
return data  # Return the plagiarism score and significance
#MAIN SECTION
start_time = time.time()  # Record the start time of the operation
text_to_check = resources.article_one  # Assign the text to check for plagiarism
plagiarism_score = calculate_plagiarism_score(text_to_check)[0]
significance = calculate_plagiarism_score(text_to_check)[1]
end_time = time.time()  # Record the end time of the operation
runtime = end_time - start_time  # Calculate the total runtime
# Output the results
print(f"Plagiarism Score: {plagiarism_score}%")  # Print the calculated plagiarism score
print(f"Is result Significant: {significance}")  # Print the signficance of the score
print(f"Runtime: {runtime} seconds")  # Print the total runtime of the script

As you can see, the code is very similar in structure to method 1. However, the search_semantically_similar function was edited to directly turn both articles into vectors, compare them, and return the result without chunking.

Plus, I added the calculate_plagiarism_score function, which takes the similarity score and generates a percentage of it. Then, it will return the percentage score and True/False statement if the plagiarism score is significant, which will be analyzed by comparing the cosine similarity score with the threshold I initiated to be 0.7

Now it’s time for AI to enter the battlelfield😂

This method is the same as method 1 in concept; however, instead of comparing the chunks by embedding them into vectors and generating the cosine similarity, we’ll compare them using a power prompt and OpenAI’s GPT model.

from SimplerLLM.tools.text_chunker import chunk_by_paragraphs
from SimplerLLM.language.llm import LLM, LLMProvider
import time 
import resources
import json
def compare_chunks(text_chunk):
"""
Compares a text chunk with an article text and generates a response using a OpenAI's Model
"""
article_text = resources.article_two  # The text to compare against
prompt = resources.prompt3  # A template string for creating the comparison prompt
final_prompt = prompt.format(piece=text_chunk, article=article_text)  # Formatting the prompt with the chunk and article texts
llm_instance = LLM.create(provider=LLMProvider.OPENAI)  # Creating an instance of the language model
response = llm_instance.generate_text(final_prompt)  # Generating text/response from the LLM
response_data = json.loads(response)  # Parsing the response into a JSON object
return response_data  # Returning the parsed response data
def calculate_plagiarism_score(text):
"""
Calculates the plagiarism score of a text by comparing its chunks against an article text
and evaluating the responses from OpenAI's Model
"""
text_chunks = chunk_by_paragraphs(text)  # Split the input text into chunks using SimplerLLM built-in method
total_chunks = text_chunks.num_chunks  # The total number of chunks in the input text
similarities_json = {}  # Dictionary to store similarities found
chunk_index = 1  # Index counter for naming the chunks in the JSON
plagiarised_chunks_count = 0  # Counter for the number of chunks considered plagiarised
total_scores = 0  # Sum of scores from the LLM responses
for chunk in text_chunks.chunks:
response_data = compare_chunks(chunk.text)  # Compare each chunk using the LLM
total_scores += response_data["score"]  # Add the score from this chunk to the total scores
if response_data["score"] > 6:  # A score above 6 indicates plagiarism
plagiarised_chunks_count += 1
similarities_json[f"chunk {chunk_index}"] = response_data["article"]  # Record the article text identified as similar
json.dumps(similarities_json)  # Convert the JSON dictionary to a string for easier storage
chunk_index += 1  # Increment the chunk index
plagiarism_result_json = {}  # Dictionary to store the final plagiarism results
plagiarism_score = (plagiarised_chunks_count / total_chunks) * 100 if total_chunks > 0 else 0  # Calculate the plagiarism score as a percentage
plagiarism_result_json["Score"] = plagiarism_score
plagiarism_result_json["Similarities"] = similarities_json # Adding where we found similaritites
plagiarism_result_json["IsPlagiarised"] = (total_scores > total_chunks * 6)  # Recording if the response is really plagiarised
json.dumps(plagiarism_result_json)  # Convert the final results dictionary to a JSON string
return plagiarism_result_json  # Return the plagiarism results as a dictionary
#MAIN SECTION
start_time = time.time()  # Record the start time of the operation
text_to_check = resources.article_one  # Assign the text to check for plagiarism
plagiarism_score = calculate_plagiarism_score(text_to_check)
formatted_plagiarism_score = json.dumps(plagiarism_score, indent=2) # Format the output for better readability
end_time = time.time()  # Record the end time of the operation
runtime = end_time - start_time  # Calculate the total runtime
# Output the results
print(f"Plagiarism Score: {formatted_plagiarism_score}")  # Print the calculated plagiarism score
print(f"Runtime: {runtime} seconds")  # Print the total runtime of the script

In the code,, the main function is the calculate_plagiarism_score, which chunks the articles, sends them to the compare_chunks function to get the similarity score, generates a total plagiarism score, and formats the results as JSON to add some details other than the plagiarism score, keeping them clear and readable.

The compare_chunks function creates a GPT instance using SimplerLLM, then uses a power prompt to analyze both chunks and generate a score out of 10 for how similar they are. Here’s the prompt I’m using:

###TASK
You are an expert in plagiarism checking. Your task is to analyze two pieces of text, an input chunk,
and an article. Then you're gonna check if there are pieces of the article that are similar in meaning to 
the input chunk. After that you're gonna pick the piece of article which is most similar and generate for it
a score out of 10 for how similar it is to the input chunk. Then you're gonna need to generate the output
as a JSON format that contains the input chunk, the article chunk which is the most similar, and the score
out of 10. ### SCORING CRITERIA 
When checking for pieces in the article that are close in meaning to the chunk of text make sure you 
go over the article at least 2 times to make sure you picked the the right chunk in the article which is the most 
similair to the input chunk. Then when picking a score it should be based of how similar are the meanings 
and structure of both these sentences.# INPUTS
input chunk: [{piece}]
article: [{article}]# OUTPUT
The output should be only a valid JSON format nothing else, here's an example structure:
{{
"chunk": "[input chunk]",
"article": "[chunk from article which is similar]",
"score": [score]
}}

As you can see, it is a detailed prompt that is very well crafted to generate a specific result. You can learn how to craft similar prompts yourself by becoming a Prompt Engineer.

This method is a combination of methods 2 and 3, where we’re gonna be comparing both articles as a whole but using AI instead of vector embeddings.

from SimplerLLM.language.llm import LLM, LLMProvider
import time 
import resources
import json
def compare_chunks(text_chunk):
"""
Compares a given text chunk with an article to determine plagiarism using a language model.
Returns dict: The response from the language model, parsed as a JSON dictionary.
"""
article_text = resources.article_two  # The text to compare against
# Formatting the prompt to include both the input text chunk and the article text
comparison_prompt = resources.prompt4.format(piece=text_chunk, article=article_text)
llm_instance = LLM.create(provider=LLMProvider.OPENAI)  # Creating an instance of the language model
response = llm_instance.generate_text(comparison_prompt)  # Generating response
response_data = json.loads(response)  # Parsing the response string into a JSON dictionary
return response_data  # Returning the parsed JSON data
def calculate_plagiarism_score(text_to_analyze):
"""
Calculates the plagiarism score based on the analysis of a given text against a predefined article text.
Returns dict: A JSON dictionary containing the plagiarism score and the raw data from the analysis.
"""
plagiarism_results = {}  # Dictionary to store the final plagiarism score and analysis data
plagiarised_chunk_count = 0  # Counter for chunks considered plagiarised
analysis_data = compare_chunks(text_to_analyze)  # Analyze the input text for plagiarism
total_chunks = len(analysis_data)  # Total number of chunks analyzed
for key, value in analysis_data.items():
# Check if the value is a list with at least one item and contains a 'score' key
if isinstance(value, list) and len(value) > 0 and 'score' in value[0] and value[0]['score'] > 6:
plagiarised_chunk_count += 1
# Check if the value is a dictionary and contains a 'score' key
elif isinstance(value, dict) and 'score' in value and value['score'] > 6:
plagiarised_chunk_count += 1
plagiarism_score = (plagiarised_chunk_count / total_chunks) * 100 if total_chunks > 0 else 0 # Calculate plagiarism score as a percentage
plagiarism_results["Total Score"] = plagiarism_score  # Add the score to the results dictionary
plagiarism_results["Data"] = analysis_data  # Add the raw analysis data to the results dictionary
json.dumps(plagiarism_results)  # Convert the results dictionary to a clear JSON string
return plagiarism_results  # Return the final results dictionary
#MAIN SECTION
start_time = time.time()  # Record the start time of the operation
text_to_check = resources.article_one # Assign the text to check for plagiarism
plagiarism_score = calculate_plagiarism_score(text_to_check)
formatted_plagiarism_score = json.dumps(plagiarism_score, indent=2) # Format the output for better readability
end_time = time.time()  # Record the end time of the operation
runtime = end_time - start_time  # Calculate the total runtime
# Output the results
print(f"Plagiarism Score: {formatted_plagiarism_score}")  # Print the scores
print(f"Runtime: {runtime} seconds")  # Print the total runtime of the script

This code is 80% like the code in method 3. However, instead of comparing each chunk, we send both articles as a whole and let OpenAI’s GPT generate a detailed plagiarism test, comparing all parts of the articles as it wishes. In the end, it returns a detailed output containing a plagiarism score and the top sections are found to be similar in their similarity score.

All this is done using this power prompt:

### TASK
You are an expert in plagiarism checking. Your task is to analyze two pieces of text, an input text,
and an article. Then you're gonna check if there are pieces of the article that are similar in meaning to 
the pieces of the input text. After that you're gonna pick chunk pairs that are most similar to each other
in meaning and structure, a chunk from the input text and a chunk from the article. You will then generate 
a score out of 10 for each pair for how similar they are.
Then you're gonna need to generate the output as a JSON format for each pair that contains 
the input text chunk, the article chunk which are the most similar, and the score out of 10.### SCORING CRITERIA 
When checking for peices in the article that are close in meaning to the chunk of text make sure you 
go over the article at least 2 times to make sure you picked the right pairs of chunks which are most similar.
Then when picking a score it should be based of how similar are the meanings and structure of both these sentences.### INPUTS
input text: [{piece}]
article: [{article}]### OUTPUT
The output should be only a valid JSON format nothing else, here's an example structure:
{{
"pair 1": 
[
"chunk 1": "[chunk from input text]",
"article 1": "[chunk from article which is similar]",
"score": [score]
],
"pair 2": 
[
"chunk 2": "[chunk from input text]",
"article 2": "[chunk from article which is similar]",
"score": [score]
],
"pair 3": 
[
"chunk 3": "[chunk from input text]",
"article 3": "[chunk from article which is similar]",
"score": [score]
],
"pair 4": 
[
"chunk 4": "[chunk from input text]",
"article 4": "[chunk from article which is similar]",
"score": [score]
]
}}

The prompt in methods 3 and 4 is very important to be well-crafted since all the results are based on it. Feel free to tweak and optimize it to your liking and if it generates better results make sure to share it with us in the comments below!

After we tried 2 types of machines to do the work for us, let’s now use human intelligence and see if their results are significant!

Here are the 2 texts I was comparing:

Article 1:What is generative AI?Generative AI refers to deep-learning models that can generate high-quality text, images, and other content based on the data they were trained on.Artificial intelligence has gone through many cycles of hype, but even to skeptics, the release of ChatGPT seems to mark a turning point. OpenAI's chatbot, powered by its latest large language model, can write poems, tell jokes, and churn out essays that look like a human created them. 
Prompt ChatGPT with a few words, and out comes love poems in the form of Yelp reviews, or song lyrics in the style of Nick Cave.

An obstacle most people face when writing academic research papers is finding similar papers easily. I myself faced this problem because it takes too much time to do so.

So, I built a Python script powered by AI to search for related keywords in an input abstract and then get related abstracts on Arxiv.

ArXiv is an open-access archive for nearly 2.4 million academic articles in physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.

Why choose Arxiv?

Simply because it already contains a lot of articles and a free built-in API, which makes it easier to access any article’s abstract directly. This bypasses the need to search the web using paid APIs for articles and then check if they contain an abstract, and if they do use an HTML parser to find the abstract — TOO MUCH WORK 🫠

http://export.arxiv.org/api/query?search_query=all:{abstract_topic}&start=0&max_results={max_results}

Enter your abstract’s topic instead of {abstract_topic}, and how many search results you want instead of {max_results}. Then, paste it into the web browser, and it’ll generate an XML file containing the summary (which is the abstract) and some other details about the article, like its ID, the authors, etc…

The idea is simple; here’s the workflow:

1- Extract from the input abstract the top 5 keywords (topics) that are most representative of its content.

2- Call the API on each of the 5 keywords we extracted

3- Analyze the results and check the similarity of these abstracts to the input abstract.

To use Arxiv’s API, we need a keyword to search for, so we need to extract the top keywords present in the input abstract. You can either use built-in libraries like nltk or spacy, but when I tried using them, the results were not as expected and not so accurate.

So, to get better results, I used OpenAI’s GPT-4 (you can use Gemini if you prefer it), gave it a power prompt, and generated optimal results. Here’s the code:

def extract_keywords(abstract):
# Constructing a prompt for the language model to generate keywords from the abstract
prompt = f"""
### TASK
You are an expert in text analysis and keyword extraction. Your task is to analyse an abstract I'm going to give you
and extract from it the top 5 keywords that are most representative of its content. Then you're going to generate
them in a JSON format in descending order from the most relevant to the least relevant.### INPUTS
Abstract: {abstract}
### OUTPUT
The output should be in JSON format. Here's how it should look like:
[
{{"theme": "[theme 1]"}},
{{"theme": "[theme 2]"}},
{{"theme": "[theme 3]"}},
{{"theme": "[theme 4]"}},
{{"theme": "[theme 5]"}}
]
"""
# Creating an instance of the language model using SimplerLLM
llm_instance = LLM.create(provider=LLMProvider.OPENAI, model_name="gpt-4")
# Generating response from the language model
response = llm_instance.generate_text(user_prompt=prompt)
# Attempting to parse the response as JSON
try:
response_data = json.loads(response)
return json.dumps(response_data, indent=2)
except json.JSONDecodeError:
# Returning an error message if the response is not valid JSON
return json.dumps({"error": "Invalid response from LLM"}, indent=2)

This function uses SimplerLLM, which facilitates the process of calling OpenAI’s API without writing tedious code. In addition, it makes it very easy for you to use Gemini’s API instead of OpenAI by only changing the name of the LLM instance like this:

llm_instance = LLM.create(provider=LLMProvider.GEMINI, model_name="gemini-pro")

Very nice, right?😉

Back to our code.

The power prompt I crafted is the main engine of the above function, so if it weren’t efficiently crafted, the code wouldn’t work at all.

### TASK
You are an expert in text analysis and keyword extraction. Your task is to analyse an abstract I'm going to give you and extract from it the top 5 keywords that are most representative of its content. Then you're going to generate them in a JSON format in descending order from the most relevant to the least relevant.
### INPUTS
Abstract: {abstract}
### OUTPUT
The output should be in JSON format. Here's how it should look like:
[
{{"theme": "[theme 1]"}},
{{"theme": "[theme 2]"}},
{{"theme": "[theme 3]"}},
{{"theme": "[theme 4]"}},
{{"theme": "[theme 5]"}}
]

As you can see, it is a detailed prompt that is very well crafted to generate a specific result. By becoming a Prompt Engineer, you can learn how to craft similar prompts yourself.

After running the above function, we’ll have a JSON-formatted output containing 5 keywords. So, we need to search for abstracts for each of the 5 keywords, and we’ll do that using Arxiv’s API.

However, when you run Arxiv’s API call, you get an XML file like this:

So, to easily extract the ID and summary (abstract), we’ll import xml.etree.ElementTree that helps us easily navigate and extract information from XML-formatted text.

def get_abstracts(json_input):
input_data = json.loads(json_input)
all_summaries_data = []
for theme_info in input_data:
keyword = theme_info['theme']
max_results = 1  # Number of results to fetch for each keyword
# Constructing the query URL for the arXiv API
url = f"http://export.arxiv.org/api/query?search_query=all:{keyword}&start=0&max_results={max_results}&sortBy=submittedDate&sortOrder=descending"
response = requests.get(url)
if response.status_code == 200:
root = ET.fromstring(response.text)
ns = {'atom': 'http://www.w3.org/2005/Atom'}
summaries_data = []
for entry in root.findall('atom:entry', ns):
arxiv_id = entry.find('atom:id', ns).text.split('/')[-1]
summary = entry.find('atom:summary', ns).text.strip()
summaries_data.append({"ID": arxiv_id, "abstract": summary, "theme": keyword})
all_summaries_data.extend(summaries_data[:max_results]) 
else:
print(f"Failed to retrieve data for theme '{keyword}'. Status code: {response.status_code}")
json_output = json.dumps(all_summaries_data, indent=2)
return json_output

In the above function, we’re looping over the 5 keywords we generated, and for each one, we’re calling the API, extracting the ID and abstract from the XML, saving them in a list, and formatting this list into JSON (easier to read).

How can we check for similarity between 2 abstracts? Again, AI 🤖

We’ll be using SimplerLLM again to create an OpenAI instance and a power prompt to perform the analysis and similarity checking.

def score_abstracts(abstracts, reference_abstract):
new_abstracts = json.loads(abstracts)
scored_abstracts = []
for item in new_abstracts:
prompt = f"""
### TASK
You are an expert in abstract evaluation and English Literature. Your task is to analyze two abstracts
and then check how similar abstract 2 is to abstract 1 in meaning. Then you're gonna generate
a score out of 10 for how similar they are. 0 being have nothing in common on different topics, and 10
being exactly the same. Make sure to go over them multiple times to check if your score is correct.
### INPUTS
Abstract 1: {reference_abstract}
Abstract 2: {item["abstract"]}
### OUTPUT
The output should be only the number out of 10, nothing else.
"""
llm_instance = LLM.create(provider=LLMProvider.OPENAI, model_name="gpt-4")
# Generating the similarity score from the language model
response = llm_instance.generate_text(user_prompt=prompt)
# Extracting the score from the response and handling potential errors
try:
score = int(response)
perfect_match = score == 10
except ValueError:
score = 0
perfect_match = False
scored_abstracts.append({
"ID": item["ID"],
"theme": item["theme"],
"score": score,
"perfect_match": perfect_match
})
return scored_abstracts

We’re gonna use the JSON output we got from the function above containing all abstracts and IDs, and we’ll loop over each abstract, run the power prompt on it with the input abstract, and get the similarity score.

As mentioned above, the power prompt is a crucial part of the function; if it is bad, the code won’t work. So, read this article to improve your prompt crafting skills.

After getting the score, if it is 10/10, then the abstract we found is a perfect match for the input abstract.

To run the codes, you’re gonna have to create a .env file that contains your OpenAI API key or Gemini key like this:

And, of course, you’ll need to enter your input abstract to run the code on it:

# MAIN SCRIPT
reference_abstract = """
YOUR_ABSTRACT
"""
json_data = extract_keywords(reference_abstract)   
abstracts = get_abstracts(json_data)
data = json.dumps(score_abstracts(abstracts, reference_abstract),indent=2)
print(data)

Plus, don’t forget to install all the necessary libraries, which you can install by running this in the terminal:

pip install requests simplerllm

Get Code

Now, although the script we created is working properly, why don’t we improve it a little?

The search for abstracts is limited to only Arxiv, and maybe there is a very similar copy to your abstract that is not available on Arxiv but on a different website. So, why don’t we tweak the code a little and make it search on Google directly for similar abstracts, and then turn it into a tool with a nice UI?

To do that, we’ll only need to update the get_abstracts function:

# Search for related abstracts according to keywords and get link and content
def get_google_results(json_input):
keywords = json.loads(json_input)
search_results = []
for theme_info in keywords:
keyword = theme_info['theme']
query = f"{keyword} AND abstract AND site:edu AND -inurl:pdf"
result = search_with_value_serp(query, num_results=1)
for item in result:
try:
url = str(item.URL)
load = load_content(url)  # Assumes load_content is a function that fetches content from the URL
content = load.content
search_results.append({"Link": url, "Content": content, "theme": keyword})
except Exception as e:
print(f"An error occurred with {url}: {e}")
continue
json_output = json.dumps(search_results, indent=2)
return json_output

As you can see, the function now searches on Google using the search_with_value_serp The function is integrated into the SimplerLLM library. Then, I used the load_content function, which is also in the SimplerLLM library. This makes it very easy to access the link’s title and content.

In addition, you have to add your VALUE_SERP_API_KEY in the .env file. This is how it will look like:

Keep in mind that some keywords may not have an abstract similar to it on Google so that the search would return nothing. Therefore, you might get less than 5 links for similar abstracts.

The code above is only a prototype showing a headstart of this function. You can improve it to get better results, design a nice User Interface for it, and make a fully functional tool out of it. Then, you can build a SAAS business based on this tool.

In this way, you’ll have a monthly recurring income from these tools you built! Pretty nice, huh 😉

Remember, if you have any questions, make sure to drop them below in the comments section or on the forum.