Author: Guest Writer

TLDR:

• Traditional data extraction methods often miss deeper insights from unstructured content, particularly in the real estate sector.
• This article explores using Indexify, an open-source framework for real-time, multi-modal data extraction, to better analyze property documents.
• You’ll learn to set up Indexify, including server setup, extraction graph creation, document ingestion, and data querying, plus how to create custom extractors.
• Implementing Indexify can enhance property document analysis, leading to more accurate insights, better decisions, and streamlined management.

· Introduction
· An Overview of Indexify
∘ Extractors
∘ Coordinators
· Prerequisites for the Tutorial
· Setting Up Indexify for Advanced Document Analysis
∘ Installing and Configuring Indexify
∘ Preparing the Document Collection
· Ingesting and Processing Documents with Indexify
∘ Setting Up Extraction Graphs
∘ Custom Extractors
∘ Uploading Documents
· Asking Complex Questions Using Indexify
∘ Semantic Search & Formulating Queries
∘…

Have you ever wondered how to save hours managing YouTube comments automatically and turn them into a goldmine of valuable insights to grow your business?

Let’s see a simple, no-code solution that makes this possible!

Imagine having an AI assistant that:

• – Analyzes every comment on your channel
• – Extracts valuable data about your audience
• – Replies to comments automatically (when appropriate)
• – Organizes everything into an easy-to-understand format

This isn’t just a time-saver; it’s a powerful tool for:

• – Understanding your audience better
• – Growing your business
• – Finding new content ideas
• – Collecting testimonials and social proof

The best part? You can set this up without writing a single line of code!

We’ll be using Make to create this no-code automation.

Make is a powerful platform that lets you connect different apps and services to create custom workflows easily.

Here is a snapshot of the automation system I built (you will get it for free in a few minutes)

Step 1: Set Up Your YouTube Module

Get Make Pro Account For Free

1. In Make, add a YouTube module to fetch new comments from your target channel.
2. Configure it to run on a schedule that works for you (e.g., every hour).

Step 2: Create a Loop for Comments

Add an iterator module to process each comment one by one.

Step 3: Analyze Comments with AI

We’ll use OpenAI’s API to analyze each comment. Here’s what our AI will extract:

{
"can_be_testimonial": true,
"sentiment": "positive",
"requires_reply": true,
"urgency": "moderate",
"question_detection": false,
"can_have_generic_reply": false,
"suggested_reply": "Thank you!",
"comment_language": "en",
"character_count": 15
}

Let’s break down these fields:

• can_be_testimonial: Identifies potential testimonials for your channel or products
• sentiment: Checks if the comment is positive, negative, or neutral
• requires_reply: Determines if the comment needs a response
• urgency: Helps prioritize which comments to address first
• question_detection: Flags comments that ask questions
• can_have_generic_reply: Indicates if a Generic AI-generated response is appropriate
• suggested_reply: Provides a basic response (if applicable)
• comment_language: Identifies the language used
• character_count: Gives you an idea of the comment’s length

And here are the prompts I used:

You are an expert in YouTube comment analysis. 
I will provide you with a [Comment].
Your Role is to analyze the [Comment]. Then Determine the following:
1. Determine if the comment can be used as a positive testimonial or review on my website.
2. Check Sentiment.
3. Determine if it requires an Urgent reply.
4. Determine the level of urgency.
5. Determine if contains a Question.
6. Determine if you can reply with a generic reply like "thank you.."  -  something that doesn't require specific knowledge. 
7. Generate the suggested reply.
8. check the language of the comment.
9. check how many characters.
[YouTube _Comment]: {}

With this system prompt:

RETURN A JSON OBJECT ONLY IN THIS FORMAT:
{
"can_be_testimonial": true,
"sentiment": "positive",
"requires_reply": true,
"urgency": "moderate",
"question_detection": false,
"can_have_generic_reply": false,
"suggested_reply": "Thank you!",
"comment_language": "en",
"character_count": 15
}

If you want to learn more about prompting and automation, take a look at my full course: “Prompt Engineering from Zero to Scripting AI Workflows.”

Step 4: Store Your Data

Add a Google Sheets module to save all this data. You could also use another database of your choice.

Step 5: Automate Replies

If can_have_generic_reply is true, use the AI to generate and post a reply to the comment.

💡If you want see in action how I built this system step by step, you can check out this video.

Want to take your system to the next level? Consider these enhancements:

1. RAG (Retrieval-Augmented Generation): Answer questions using your own Q&A database.
2. Custom Prompts: Create detailed instructions for the AI on how to handle different scenarios.
3. Sentiment-Based Routing: Automatically escalate negative comments to your personal attention.

Offer Your Services to Other YouTubers

With this system in place, you have a valuable service to offer other content creators. Many would gladly pay for a tool that saves them time and provides insights.

Create a Subscription-Based Tool

Use Make’s Custom Apps feature to turn your automation into a scalable, subscription-based service:

1. Lock your automation system behind API keys
2. Clients subscribe to access your tool
3. Enjoy recurring, stable income from your creation

Learn More About Make Custom Apps and How to use them in this video:

Launch Your Own AI Automation Agency

The possibilities are endless! Use Make to build custom automation for various businesses. YouTube comment handling is just the beginning.

Automating YouTube comments isn’t just about saving time — it’s about unlocking the full potential of your audience interactions.

With no-code tools like Make and the power of AI, you can turn a time-consuming task into a strategic advantage for your channel and business.

Download our pre-built Make scenario and start automating today!

Download Make Scenario

Remember, the key to success with automation is continuous improvement.

Start with this basic setup, then iterate and expand based on your unique needs. Happy automating!

There are several reasons why one might choose to use this approach:

Simplified Reproducibility: Fine-tuning a model with input/output pairs for reproducibility allows you to offload much of the complexity and software infrastructure to OpenAI.

This means that the data, along with the seed reference, is embedded within the model itself, streamlining administration and ensuring availability.

Precision in Use Cases: By using seeding, you can precisely target different fine-tuned datasets for various user scenarios.

This enables more tailored and consistent responses depending on the specific context in which the model is being used.

Fine-tuning can also be versioned, and if a newer version of the data is used for fine-tuning, this can be tracked via fine-tuning.

Inference Segmentation: Inference can be segmented into two categories: general information and specific information on which the model is fine-tuned.

This allows the model to distinguish between providing broad, generalised responses and delivering more precise answers based on the specialised training it has received.

Considerations

Fine-tuned models that have already been created from these base models will not be affected by this deprecation.

However, you will no longer be able to create new fine-tuned versions using these models.

Although this approach is appealing due to its simplicity and no-code nature, it does create model dependency, making it difficult to migrate to another model or AI provider.

The advantage of a RAG (Retrieval-Augmented Generation) framework is that it offers a degree of model independence, allowing AI models to be treated more like interchangeable utilities.

This flexibility can simplify transitions between different models or providers, reducing the risk of being locked into a single ecosystem.

Large Language Models primarily rely on unstructured natural language input and generate output through Natural Language Generation (NLG).

A key aspect of fine-tuning, as with RAG, is to provide the Language Model with contextual reference during inference.

This process is known as In-Context Learning (ICL). ICL enables the Language Model to temporarily set aside its base knowledge and instead leverage the specific contextual data supplied, allowing for more accurate and relevant responses based on the given context.

Below is Python code you can copy as-is and paste into a notebook. You can see the question is asked, What is X? and without the necessary context, this question is very ambiguous.

pip install -q openai
#####
import os
#####
os.environ['OPENAI_API_KEY'] = str("")
#####
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o-mini-2024-07-18",
messages=[
{
"role": "system",
"content": "You should help to user to answer on his question."
},
{
"role": "user",
"content": "What is X?"
}
],
temperature=1,
max_tokens=150,
top_p=1,
frequency_penalty=0,
presence_penalty=0,
seed=1
)
print(response)

Therefore, considering the output from the gpt-4o-mini-2024–07–18 model, the model is attempting to disambiguate the input by casting the question into different contexts.

This process involves analysing the question to identify multiple potential interpretations, allowing the model to respond more accurately based on the inferred context.

ChatCompletion(id='chatcmpl-A1UX0BcHnsoy7EPnRUqJHp4F0CToW', 
choices=[Choice(finish_reason='length', 
index=0, logprobs=None, 
message=ChatCompletionMessage(content='The question "What is X?" 
is quite broad and can refer to many things depending on the context. 
Here are a few possibilities:\n\n1. 
**Mathematics**: In an equation, X often represents an unknown variable 
that needs to be solved.\n2. 
**Algebra**: X can be a placeholder for any 
number in algebraic expressions.\n3. 
**Science**: X could refer to a 
variable in experiments or a certain element in chemistry (like X-rays).\n4.
**Popular culture**: X could refer to a character, a title, or a brand name 
(like "X-Men" or "X-Factor").\n5. 
**Programming**: In computer science, X might stand for a parameter or a 
specific data type', refusal=None, role='assistant', 
function_call=None, tool_calls=None))], 
created=1724919054, 
model='gpt-4o-mini-2024-07-18', 
object='chat.completion', 
service_tier=None, 
system_fingerprint='fp_5bd87c427a', 
usage=CompletionUsage(completion_tokens=150, prompt_tokens=26, total_tokens=176))

However, below is a snippet from the training data where the context of X is defined…

The process for generating dialog flows closely mirrors the intrinsic dialog flow generation.

Initially, representative action labels and sample conversations from the domain are included in the prompt for the initial flow generation. Subsequently, feedback, updates, and clean-up steps are applied to refine the dialog flow.

The intrinsic flow method produces broad, general dialog flows but may lack domain-specific patterns observed in real-world conversations.

Conversely, relying solely on domain-specific dialog instances might limit the flow’s scope due to their variability. To address this, a hybrid approach is used, combining the broad coverage of intrinsic flows with the detailed focus from domain-specific data.

This merging process involves prompting GPT-4 to retain distinctive features from the intrinsic flow while eliminating redundant elements, resulting in what is termed the data-guided flow.

Below is an example of an intrinsic flow called city_info_code.txt . Notice how the user intent is defined and encoded. Then each of these intents are mapped to a bot flow which is also coded.

For example U1 goes to B1, and so on.

 B0[Bot: Greeting and Asks for City Information Inquiry] --> U1[User: Wants to Know About Attractions]
B0 --> U2[User: Inquires About City Services]
B0 --> U3[User: Asks About Local Regulations]
B0 --> U4[User: Interested in City Events]
B0 --> U45[User: Asks for Personalized Recommendations]
B0 --> U46[User: Inquires About Local Culture and History]
B0 --> U47[User: Wants Info on Dining and Shopping]
B0 --> U48[User: Seeks Accessibility and Accommodation Info]
U1 --> B1[Bot: Provides Information on Attractions]
U2 --> B2[Bot: Details City Services]
U3 --> B3[Bot: Explains Local Regulations]
U4 --> B4[Bot: Shares Information on Upcoming Events]
U45 --> B5[Bot: Gives Personalized Recommendations]
U46 --> B6[Bot: Discusses Local Culture and History]
U47 --> B7[Bot: Informs About Dining and Shopping]
U48 --> B8[Bot: Advises on Accessibility and Accommodation]
B1 --> U9[User: Requests Further Details or Ready to Explore]
B2 --> U9
B3 --> U9
B4 --> U9
B5 --> U9
B6 --> U9
B7 --> U9
B8 --> U9
U9 --> B9[Bot: Provides Additional Info or Encourages Exploration]

Below is an example of a data flow, called alarm set code, which should be a familiar conversation to many.

Again, notice the intents being defined upfront, and how the bot dialogs are assigned to different user intents. There is also the notion of sub or nested intents.

B0[Bot: Greeting] --> U1[User: Express need for alarm assistance]
U1 --> B1[Bot: Prompt for alarm action]
B1 --> U2[User: Set alarm]
B1 --> U3[User: Edit alarm]
B1 --> U4[User: Delete alarm]
B1 --> U5[User: Copy alarm]
U2 --> B2[Bot: Set Alarm]
U3 --> B3[Bot: Ask for alarm details]
U4 --> B4[Bot: Ask which alarm to delete]
U5 --> B9[Bot: Ask details for copying alarm]
B2 --> U6[User: Confirm alarm set]
B2 --> U7[User: Request to set repeat]
U6 --> B7[Bot: Confirm Completion]
U7 --> B8[Bot: Set repeat and confirm]
B3 --> U8[User: Specify alarm edit details]
B4 --> U9[User: Specify alarm to delete]
B9 --> U14[User: Confirm copy details]
U8 --> B5[Bot: Perform edit and confirm]
U9 --> B6[Bot: Delete alarm and confirm]
U14 --> B10[Bot: Copy alarm and confirm]
B5 --> U10[User: Confirm edit]
B6 --> U11[User: Request additional deletion]
B6 --> U12[User: End conversation]
B10 --> U15[User: Request further changes]
B10 --> U16[User: End conversation]
U10 --> B7
U11 --> B6
U12 --> B11[Bot: Expressing gratitude and farewell]
U15 --> B3
U16 --> B11
B8 --> U13[User: End Conversation]
U13 --> B11

Below is a short snippet of the example conversations used to generate the alarm code reset flow. The complete training file has about 700 conversations. Each line is a conversation defined by the utterances, the session id, domain and task.

{"session_id": "298663b0", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "i need to edit an alarm", "ok which one?", "the \"walk dog\"", "what would you like to do with it?", "change to \"feed cat\"", "changing. do you want to apply this to other alarms with this name?", "yes", "ok", "thanks", "yes"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "9fefc2ac", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I would like to edit a feature on my alarm.", "Alright, what would you like to edit?", "I would like to cancel all my alarms for the next week, but could you first list them all before I do anything?", "Sure. You have 5 alarms for next week. Monday at 8 AM, Tuesday at 9 AM, Wednesday at 10 AM, Thursday at 11 AM, and Friday at 12 PM. Would you like to cancel all of the alarms for next week?", "Alright, thanks. Could you cancel all my alarms?", "Sure. This action cannot be undone once completed. Would you like to proceed with deleting all of your alarms for next week?", "I would like to confirm my cancellation of all my alarms for next week.", "Your alarms for next week have been deleted.", "Very good. Thank you very much!", "You're welcome!"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "63fb88cb", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I needed to change one of my alarms", "I can help you with that, which alarm would you like to change?", "I need to change my 10:30 alarm \"Walk Dog\"", "Ok, what would you like to change it to?", "Let's change it to \"Feed Cat\"", "Ok. You also have \"walk dog\" set for 7PM each day. Would you like to change those alarms as well?", "Erase them. I'm not walking the dog anymore. I hired a kid.", "I can do that, is there anything else I can help you with", "Nope, that's it. Thanks, bot!", "Have a great day"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "8c8066e6", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to set an alarm", "Ok, what day and time?", "Please set an alarm for Monday June 11th at 8am", "Ok, should it be on vibrate or sound?", "Set this alarm to vibrate", "Ok, what should it be called?", "Please Title this alarm 'Pay Bills'", "Ok, anything else?", "No, that's all. Thanks!", "Welcome!"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "dbfc73d9", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello, how may I help you?", "Set an alarm for 6:30pm", "Okay. Do you want it on sound or on vibrate?", "Sound Please", "Okay. What sound should it make? Ringing or do you want a personalized ringtone?", "I want a speaking voice tone i mean", "Okay. What speaking tone would you like?", "A gorgeous lady like", "Okay. When should the alarm go off?", "6:30pm", "What day?", "Everyday", "Okay. Alarm set successfully."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "481aec04", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to move an alarm", "which alarm", "I need my Monday alarm at three changed", "whats the new time and day", "Friday at four please", "do you want to change the name of the alarm", "Please change to ding dong", "done. anything else", "No thanks!", "yw"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "ee342fd6", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need some help with my alarms.", "It would be my pleasure to assist", "I want to set a new alarm.", "I can do that for you. Would you like to title the alarm?", "Wake-up alarm. Set it for 7:00 AM.", "Creating \"Wake-up\" alarm for 7:00am. Would you like any particular sound to wake to?", "Great. Now set it to have an air horn sound.", "Setting to blow horn sound Anything else I can set for you today?", "Excellent. That's all I need. Thank you.", "You're most welcome"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "328d3657", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I want to change the settings for my 7:30pm alarm", "i think you mean at 8. there is no 7:30 alarm", "That's right.", "what do you want to change?", "I want to change the sound. What sound is currently set?", "sound is set at birds chirp right now", "Change that to a chime", "changing", "Thanks", "no problem"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "d22e3a13", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to modify my alarms", "I can help, what would you like to change?", "I want an alarm every Friday at 8 pm", "Every Friday at 8 PM alarm is that correct?", "yes", "Alarm noise or Vibrate?", "both", "What Alarm sound would you like?", "old phone tone", "Ok I have set an alarm for Friday at 8 PM", "thanks"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER"]}
{"session_id": "5cb1b497", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to change an alarm please.", "I can help you with that. What alarm would you like to edit?", "I would like to change my 10:30 am alarm to 11:30 am for Monday.", "Alright, I've made that change for you. Would you like to enable the vibrate feature on this alarm?", "I would like to set this alarm to vibrate.", "Enable vibrate for this alarm. You also have the option of enabling vibrate with sound, would you like to have sound play for this alarm, or only vibrate?", "Only vibrate.", "Ok, I've set this alarm up for you. Would you like to add a title?", "Yes. Add \"Milk Meeting\" as my title.", "I've made this change for you. Anything else I can do?", "That's all."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER"]}
{"session_id": "b479a288", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need some help managing my alarms", "Sure, how can I help with alarms today", "I need an alarm set for ever 2 minutes between 6am and 8am.", "What sound settings do you want for these alarms?", "Shrieking baby sound", "And for what day specifically?", "Monday and Thursday", "Alarms set, anything else I can assist with?", "That is all, thanks bot.", "No problem"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "620925cc", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I want you to set an alarm for me", "What time would you like your alarm set for?", "7am for tomorrow please", "Alright. Your alarm is set for 7 AM.", "And can you set an alarm for Saturday at 3pm?", "Sure. A second alarm is set for Saturday, 3 PM.", "Okay, thank you so much", "You are welcome! May I help you with anything else today?", "No, that was all, goodbyte", "Goodbye. Have a nice day."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "b5b3d0b0", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "Hello. i wanted help with setting some alarms.", "I can help with that. What alarms would you like to set?", "Starting at 6 AM tomorrow, I want to have alarms go off every 30 minutes until 8 AM.", "Done. Your alarms starting at 6 AM tomorrow have been set. Would you like to add more alarms?", "No, but can a choose the sound the alarms will make?", "Yes. Do you want the same sound settings for all alarms ?", "Yes. I want a loud sound, similar to a siren.", "Loud siren has been set to your alarms.", "Thank you. You have been very helpful.", "You are welcome."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "c9fb5fc5", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to rename an alarm please", "which alarm", "Please rename the alarm titled 'walk dog' to 'feed cat'", "done. do you want to apply it to similarly named alarms", "Yes. Please change all alarms titled 'walk dog' to 'feed cat'", "are you sure", "Yes, I'm positive", "changes done", "Great! Thanks so much", "yw"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "84adfea3", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need to delete an alarm", "Okay. Which alarm do you want to delete?", "I am not entirely sure. I know I have 2 alarms at night, one is for 9pm and the other is for 10pm but I can't remember which is which. What are their titles?", "They both have the same name, actually: Nighttime Alarm.", "Oh man, Do I have an alarm named \"pick Jess up\"?", "Yes, you have several alarms under that name too.", "Okay, just delete all of those please", "Alright, I've deleted all of the alarms with that name.", "You rock! Thanks", "You are welcome. Have a good day."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "d6077f73", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "Can you pull up my list of alarms?", "Sure, here is you alarm list.", "What alarm do I have next week on Tuesday?", "You have three meetings for Tuesday.", "Are there alarms for those? I'd like to delete the earliest one", "Three alarms named \"Meetings\" for Tuesday. The earliest alarm is for 8AM, shall I delete that alarm?", "Yes please, delete that", "Alarm \"Meetings\" set for 8AM deleted.", "Great, thanks for that!", "You're welcome"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "c2d70b5c", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I need help editing an alamr", "Ok, which alarm?", "My Monday 3pm alarm move it to Friday at 4pm", "Ok, same sound on that alarm?", "yes please", "and same name?", "Yes keep everything the same", "Anything else I can do for you?", "That's all thank you", "Great"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "2e27a20b", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "Cancel all my alarms", "You have five alarms set for next week, are you sure you want to cancel all of them?", "Yeah but I want you to list them before cancelling", "Ok, your first alarm is for Monday at 3pm. Do you want to cancel that one?", "Yes. but kindly list them", "Your second alarm is for Tuesday at 1pm.", "Yes", "Your third alarm is set for Wednesday at 1pm.", "Yes", "Your fourth alarm is set for Wednesday at 3pm."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "1763c59a", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I'd like to edit an alarm.", "Sure! Which alarm would you like to edit?", "My alarm for tomorrow at 10 AM.", "Got it! What would you like to change it to?", "I'd like to have it play a sound.", "No problem! Which sound would you like this alarm to play?", "\"Waterfalls\".", "All done! This alarm will now play \"Waterfalls\". Is there anything else I can help you with?", "That's all, thank you!", "You are welcome! Have a great day."], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "741540d2", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "I want my Saturday alarm set for 7:00 am. to vibrate and make a bell sound.", "Sure. I can help you to set your alarm for 7:00AM and make it vibrate and make a bell sound.", "Thank you. Could you also set up a Monday to Friday alarm for 5:00 am.?", "Sure. I can set you an alarm from Monday to Friday for 5:00AM. What sound would you like that alarm to make?", "I want it to play thunder sounds.", "All done. Your alarm now makes thunder sounds.", "Could you also set up an alarm for 9:00 pm. everyday?", "Yes. I can also set up an alarm for 9:00PM everyday. What sound would you like it to play?", "Horse sounds, please.", "All done! Your 9:00PM alarm will make horse sounds!"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "0b03aafa", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "Hi! I need help with my device.", "I can help you set and edit alarms. Do you need assistance with that?", "I want to set my alarm for 7:30 PM to vibrate only.", "You do not currently have an alarm set for 7:30 PM. You do have an alarm set for 8 PM.", "Oh okay. please change the setting for that alarm instead.", "To confirm, you want to set your 8 PM alarm to vibrate only. Correct?", "Yes.", "I will make that change. Do you need to make any other changes?", "No. That's all. Thanks!", "I am here to serve"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}
{"session_id": "063c3d96", "domain": "ALARM_SET", "task": "ALARM_SET", "utterances": ["Hello how may I help you?", "i need to set an alarm", "What alarm do you want to set?", "i need to set several alarms", "What are the alarms that you want to set?", "i need to set alarms 15 mins apart between 6am and 8 am", "Okay. Do want the alarms every day of the week?", "yes", "Okay. I have set alarms from Mon-Sun 6 to 8 at 15 minute intervals", "great ty", "You are welcome. Can I help with anything else?"], "roles": ["BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT", "USER", "BOT"]}

• The process of flow creation starts with generating an initial flow by using a large language model (LLM). The level of automation can work and serves well for solving for the blank canvas problem or blank canvas paralysis.
• However, starting out with existing conversations is a much more astute approach. Making use of existing dialog turns between customers and service representatives leverages impactful and meaningful conversations.
• I guess someone can follow the same methodology and only replace the initial LLM dialog generation with real-world conversations.
• The aim of this approach is premised on only involving humans in a weak supervision role, hence elevating the burden on designers and builders.
• The dialogs are also premised on a graph approach with nodes and conditional edges.
• This tool is ideal for making use of existing conversations, creating semantically similar clusters, and creating or generating flows which can be used as a starting point when developing a chatbot.
• The flows can be used as a starting point for designing and building chatbots and creating prototype flows to build from.
• I’m not sure what the best why is to implement the generated flows…having a tool to convert the ASCII into a visual representation or some kind of GUI will be helpful.
• Or some kind of development framework which can interpret and ingest the text based design.

✨✨ Follow me on LinkedIn for updates ✨✨

I’m currently the Chief Evangelist @ Kore AI. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.

Quick Success Data Science

I have a relative who believes rising temperatures are just a “heat island” effect. That is, rapidly growing cities of steel and concrete heat up faster and retain heat longer than green rural areas. He says, “Much of global warming goes away if you focus on temperature measurements taken in the cooler countryside.”

The nice thing about data science is that you can be your own fact-checker. In this Quick Success Data Science project, we’ll test the previous assumption using temperature data from the great state of Texas.

According to the EPA, heat islands are urbanized areas that experience higher temperatures than outlying areas. Artificial structures, such as buildings and roads, absorb and re-emit the sun’s heat more than natural landscapes, such as forests and lakes. Human activity, including driving cars and cooling buildings, generates additional heat. In big cities, where these structures and activities are highly concentrated, “islands” of higher temperatures form relative to the surrounding countryside.

As I have mentioned the architecture and implementation of text based 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀 (𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀) are converging on very much the same principles.

The 𝘯𝘦𝘹𝘵 𝘤𝘩𝘢𝘱𝘵𝘦𝘳 for 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀 is emerging…

And that is… 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀 which are capable of navigating mobile or browser screens, particularly using bounding boxes to define screen elements.

Hence Agents designed to interact with user interfaces (UI) in a way similar to how a human would.

Yet another recent study shows how navigation of a mobile device over multiple apps can be achieved.

The exploration module gathers element information through agent-driven or manual exploration, compiling it into a document.

During the deployment phase, the RAG (Retrieval-Augmented Generation) system retrieves and updates this document in real time, enabling swift task execution.

The bottom of the graphic below shows cross-app task being executed. I still need to dig into this study, but I’m still very much impressed with Ferris-UI from Apple and WebVoyager (LangChain implementation).

Considering the images below, the AppAgent V2 follows very much a well known and defined agent approach of observation, thought, action and summary.