Category: Latest AI News

This category includes the various error messages developers encounter when working with LLM APIs.

For example, developers might face request errors and data value capacity limit errors during API calls for image editing.

Additionally, issues related to the OpenAI server, such as gateway timeout errors, may arise. This subcategory represents 7.5% of the total challenges identified.

1. Automating Task Processing: LLMs can automate tasks like text generation and image recognition, unlike traditional software that requires manual coding.
2. Dealing with Uncertainty: LLMs produce variable and sometimes unpredictable outputs, requiring developers to manage this uncertainty.
3. Handling Large-Scale Datasets: Developing LLMs involves managing large datasets, necessitating expertise in data preprocessing and resource efficiency.
4. Data Privacy and Security: LLMs require extensive data for training, raising concerns about ensuring user data privacy and security.
5. Performance Optimisation: Optimising LLM performance, particularly in output accuracy, differs from traditional software optimisation.
6. Interpreting Model Outputs: Understanding and ensuring the reliability of LLM outputs can be complex and context-dependent.

The results show that 54% of these questions have fewer than three replies, suggesting that they are typically challenging to address.

The introduction of GPT-3.5 and ChatGPT in November 2022, followed by GPT-4 in March 2023, really accelerated the growth of the LLM developer community, markedly increasing the number of posts and users on the OpenAI developer forum.

The challenges faced by LLM developers are multifaceted and diverse, encompassing 6 categories and 27 distinct subcategories.

Challenges such as API call costs, rate limitations, and token constraints are closely associated with the development and use of LLM services.

Developers frequently raise concerns about API call costs, which are affected by the choice of model and the number of tokens used in each request.

Rate limitations, designed to ensure service stability, require developers to understand and manage their API call frequencies. Token limitations present additional hurdles, particularly when handling large datasets or extensive context.

Therefore, LLM providers should develop tools to help developers accurately calculate and manage token usage, including cost optimisation strategies tailored to various models and scenarios. Detailed guidelines on model selection, including cost-performance trade-offs, will aid developers in making informed decisions within their budget constraints.

Additionally, safety and privacy concerns are critical when using AI services. Developers must ensure their applications adhere to safety standards and protect user privacy.

OpenAI should continue to promote its free moderation API, which helps reduce unsafe content in completions by automatically flagging and filtering potentially harmful outputs.

Human review of outputs is essential in high-stakes domains and code generation to account for system limitations and verify content correctness.

Limiting user input text and restricting the number of output tokens can also help mitigate prompt injection risks and misuse, and these measures should be integrated into best practices for safe AI deployment.

By examining relevant posts on the OpenAI developer forum, it is evident that LLM development is rapidly gaining traction, with developers encountering more complex issues compared to traditional software development.

The study aims to analyse the underlying challenges reflected in these posts, investigating trends and problem difficulty levels on the LLM developer forum.

I’m currently the Chief Evangelist @ Kore AI. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.

· Context
· What is EDA
· Evaluation Criteria
· Problem Setup
· ChatGPT-4o
· Claude 3.5 Sonnet
· Gemini Advanced
· Final Results

Welcome back to the second installment of my series, ChatGPT vs. Claude vs. Gemini for Data Analysis! In this series, I aim to compare these AI tools across various data science and analytics tasks to help fellow data enthusiasts and professionals choose the best AI assistant for their needs. If you missed the first article, I compared their performance in writing and optimizing SQL queries — be sure to check it out!

Although the 2024 Olympics have concluded, our AI competition is just heating up. So far, Claude 3.5 Sonnet has taken the lead! But can it maintain its position, or will ChatGPT and Gemini catch up? 🏆

In this second article, we’ll focus on their ability to independently conduct Exploratory Data Analysis (EDA). As a data scientist, imagine the convenience of having an AI tool that can instantly provide data insights and recommendations for a new…

Optimizers are an essential part of everyone working in machine learning.

We all know optimizers determine how the model will converge the loss function during gradient descent. Thus, using the right optimizer can boost the performance and the efficiency of model training.

Besides classic papers, many books explain the principles behind optimizers in simple terms.

However, I recently found that the performance of Keras 3 optimizers doesn’t quite match the mathematical algorithms described in these books, which made me a bit anxious. I worried about misunderstanding something or about updates in the latest version of Keras affecting the optimizers.

So, I reviewed the source code of several common optimizers in Keras 3 and revisited their use cases. Now I want to share this knowledge to save you time and help you master Keras 3 optimizers more quickly.

If you’re not very familiar with the latest changes in Keras 3, here’s a quick rundown: Keras 3 integrates TensorFlow, PyTorch, and JAX, allowing us to use cutting-edge deep learning frameworks easily through Keras APIs.

Working with sensitive data or within a highly regulated environment requires safe and secure cloud infrastructure for data processing. The cloud might seem like an open environment on the internet and raise security concerns. When you start your journey with Azure and don’t have enough experience with the resource configuration it is easy to make design and implementation mistakes that can impact the security and flexibility of your new data platform. In this post, I’ll describe the most important aspects of designing a cloud adaptation framework for a data platform in Azure.

An Azure landing zone is the foundation for deploying resources in the public cloud. It contains essential elements for a robust platform. These elements include networking, identity and access management, security, governance, and compliance. By implementing a landing zone, organizations can streamline the configuration process of their infrastructure, ensuring the utilization of best practices and guidelines.

An Azure landing zone is an environment that follows key design principles to enable application migration, modernization, and development. In Azure, subscriptions are used to isolate and develop application and platform resources. These are categorized as follows:

• Application landing zones: Subscriptions dedicated to hosting application-specific resources.
• Platform landing zone: Subscriptions that contain shared services, such as identity, connectivity, and management resources provided for application landing zones.

These design principles help organizations operate successfully in a cloud environment and scale out a platform.

A data platform implementation in Azure involves a high-level architecture design where resources are selected for data ingestion, transformation, serving, and exploration. The first step may require a landing zone design. If you need a secure platform that follows best practices, starting with a landing zone is crucial. It will help you organize the resources within subscriptions and resource groups, define the network topology, and ensure connectivity with on-premises environments via VPN, while also adhering to naming conventions and standards.

Architecture Design

Tailoring an architecture for a data platform requires a careful selection of resources. Azure provides native resources for data platforms such as Azure Synapse Analytics, Azure Databricks, Azure Data Factory, and Microsoft Fabric. The available services offer diverse ways of achieving similar objectives, allowing flexibility in your architecture selection.

For instance:

• Data Ingestion: Azure Data Factory or Synapse Pipelines.
• Data Processing: Azure Databricks or Apache Spark in Synapse.
• Data Analysis: Power BI or Databricks Dashboards.

We may use Apache Spark and Python or low-code drag-and-drop tools. Various combinations of these tools can help us create the most suitable architecture depending on our skills, use cases, and capabilities.

Azure also allows you to use other components such as Snowflake or create your composition using open-source software, Virtual Machines(VM), or Kubernetes Service(AKS). We can leverage VMs or AKS to configure services for data processing, exploration, orchestration, AI, or ML.

Typical Data Platform Structure

A typical Data Platform in Azure should comprise several key components:

1. Tools for data ingestion from sources into an Azure Storage Account. Azure offers services like Azure Data Factory, Azure Synapse Pipelines, or Microsoft Fabric. We can use these tools to collect data from sources.

2. Data Warehouse, Data Lake, or Data Lakehouse: Depending on your architecture preferences, we can select different services to store data and a business model.

• For Data Lake or Data Lakehouse, we can use Databricks or Fabric.
• For Data Warehouse we can select Azure Synapse, Snowflake, or MS Fabric Warehouse.

3. To orchestrate data processing in Azure we have Azure Data Factory, Azure Synapse Pipelines, Airflow, or Databricks Workflows.

4. Data transformation in Azure can be handled by various services.

• For Apache Spark: Databricks, Azure Synapse Spark Pool, and MS Fabric Notebooks,
• For SQL-based transformation we can use Spark SQL in Databricks, Azure Synapse, or MS Fabric, T-SQL in SQL Server, MS Fabric, or Synapse Dedicated Pool. Alternatively, Snowflake offers all SQL capabilities.

Subscriptions

An important aspect of platform design is planning the segmentation of subscriptions and resource groups based on business units and the software development lifecycle. It’s possible to use separate subscriptions for production and non-production environments. With this distinction, we can achieve a more flexible security model, separate policies for production and test environments, and avoid quota limitations.

Networking

A virtual network is similar to a traditional network that operates in your data center. Azure Virtual Networks(VNet) provides a foundational layer of security for your platform, disabling public endpoints for resources will significantly reduce the risk of data leaks in the event of lost keys or passwords. Without public endpoints, data stored in Azure Storage Accounts is only accessible when connected to your VNet.

The connectivity with an on-premises network supports a direct connection between Azure resources and on-premises data sources. Depending on the type of connection, the communication traffic may go through an encrypted tunnel over the internet or a private connection.

To improve security within a Virtual Network, you can use Network Security Groups(NSGs) and Firewalls to manage inbound and outbound traffic rules. These rules allow you to filter traffic based on IP addresses, ports, and protocols. Moreover, Azure enables routing traffic between subnets, virtual and on-premise networks, and the Internet. Using custom Route Tables makes it possible to control where traffic is routed.

Naming Convention

A naming convention establishes a standardization for the names of platform resources, making them more self-descriptive and easier to manage. This standardization helps in navigating through different resources and filtering them in Azure Portal. A well-defined naming convention allows you to quickly identify a resource’s type, purpose, environment, and Azure region. This consistency can be beneficial in your CI/CD processes, as predictable names are easier to parametrize.

Considering the naming convention, you should account for the information you want to capture. The standard should be easy to follow, consistent, and practical. It’s worth including elements like the organization, business unit or project, resource type, environment, region, and instance number. You should also consider the scope of resources to ensure names are unique within their context. For certain resources, like storage accounts, names must be unique globally.

For example, a Databricks Workspace might be named using the following format:

Example Abbreviations:

A comprehensive naming convention typically includes the following format:

• Resource Type: An abbreviation representing the type of resource.
• Project Name: A unique identifier for your project.
• Environment: The environment the resource supports (e.g., Development, QA, Production).
• Region: The geographic region or cloud provider where the resource is deployed.
• Instance: A number to differentiate between multiple instances of the same resource.

Implementing infrastructure through the Azure Portal may appear straightforward, but it often involves numerous detailed steps for each resource. The highly secured infrastructure will require resource configuration, networking, private endpoints, DNS zones, etc. Resources like Azure Synapse or Databricks require additional internal configuration, such as setting up Unity Catalog, managing secret scopes, and configuring security settings (users, groups, etc.).

Once you finish with the test environment, you‘ll need to replicate the same configuration across QA, and production environments. This is where it’s easy to make mistakes. To minimize potential errors that could impact development quality, it‘s recommended to use an Infrastructure as a Code (IasC) approach for infrastructure development. IasC allows you to create cloud infrastructure as code in Terraform or Biceps, enabling you to deploy multiple environments with consistent configurations.

In my cloud projects, I use accelerators to quickly initiate new infrastructure setups. Microsoft also provides accelerators that can be used. Storing an infrastructure as a code in a repository offers additional benefits, such as version control, tracking changes, conducting code reviews, and integrating with DevOps pipelines to manage and promote changes across environments.

If your data platform doesn’t handle sensitive information and you don’t need a highly secured data platform, you can create a simpler setup with public internet access without Virtual Networks(VNet), VPNs, etc. However, in a highly regulated area, a completely different implementation plan is required. This plan will involve collaboration with various teams within your organization — such as DevOps, Platform, and Networking teams — or even external resources.

You’ll need to establish a secure network infrastructure, resources, and security. Only when the infrastructure is ready you can start activities tied to data processing development.

If you found this article insightful, I invite you to express your appreciation by clicking the ‘clap’ button or liking it on LinkedIn. Your support is greatly valued. For any questions or advice, feel free to contact me on LinkedIn.

The notion of a World State is something I find very interesting, where certain ambient or environmental settings need to be accessed to enable certain actions.

This World State alludes to the research Apple did regarding Ferrit-UI and other research like WebVoyager. Where there is a World the agent needs to interact with. This world currently is constituted by surfaces or screens and needs to navigate browser windows, mobile phone OSs and more.

Milestones are key points which need to be executed in order to achieve or full-fill the user intent. These can also be seen as potential points of failure should it not be possible to execute.

In the example in the image above, the User intent is to send a message, while cellular service is turned off.

The Agent should first understand the User’s intent, and prompt for necessary arguments from the User. After collecting all arguments with the help of the search_contacts tool, the Agent attempted to send the message, figured out it needs to enable cellular service upon failure, and retried.

To evaluate this trajectory, we find the best match for all Milestones against Message Bus and World State in each turn while maintaining topological order.

This is an excellent example of how, for an Agent to be truly autonomous, it needs to be in control of its environment.

Despite the paradigm shift towards a more simplified problem formulation, the stateful, conversational and interactive nature of task oriented dialog remains, and poses a significant challenge for systematic and accurate evaluation of tool-using LLMs.

Stateful

Apple sees state as not only the conversational dialog turns or dialog state, but also the state of the environment in which the agents live.

This includes implicit state dependencies between stateful tools, allowing the agent to track and alter the world state based on its world or common-sense knowledge, which is implicit from the user query.

Something else I find interesting in this study is the notion of a Knowledge Boundary, which inform the user simulator what it should and should not know, providing partial access to expected result, combating hallucination. This is analogous to in and out of domain questions.

Milestones and Minefields, which define key events that must or must not happen in a trajectory, allowing us to evaluate any trajectory with rich intermediate and final execution signals.

For the conversational user interface, there are two scenarios defined…

Single / Multiple Tool Call

The one scenario is where there is a single conversation or dialog/user turn, with multiple tool calling procedures in the background.

Hence the user issues a single request which is not demanding from a NLU dialog state management perspective, but demands heavy lifting in the background.

Single / Multiple User Turn

In other scenarios there might only be a single tool call event or milestone, but multiple dialog turns are required to establish the user intent, disambiguate where necessary, collect relevant and required information from the user, etc.

Most of the code we use to interact with LLMs (Large Language Models) is hidden behind several APIs — and that’s a good thing.

But if you are like me, and want to understand the ins and outs of these magical models, there’s still hope for you. Currently, apart from the researchers working on developing and training new LLMs, there’s mostly two types of people playing with these types of models:

• Users, that interact via applications such as ChatGPT or Gemini.
• Data scientists and developers that work with different libraries, such as llangchain, llama-index or even using Gemini or OpenAI apis, that simplify the process of building on top of these models.

The problem is — and you may have felt it — that there is a fundamental knowledge in text mining and natural language processing that is completely hidden away in consumer products or APIs. And don’t take me wrong — they are great for developing cool use cases around these technologies. But, if you want to a have deeper knowledge to build complex use cases or manipulate LLMs a bit better, you’ll need to check the fundamentals — particularly when the models behave as you…

Previously two options were available JSON Mode & Function Calling…

Enabling OpenAI’s JSON mode doesn’t ensure that the output will adhere to a specific predefined JSON schema. It only guarantees that the JSON will be valid and parse without errors.

The challenge with OpenAI’s JSON Mode lies in the significant variability of the JSON output with each inference, making it impossible to predefine a consistent JSON schema.

To clarify, the chat completion API itself doesn’t call any functions, but the model can generate JSON output that you can use in your code to trigger function calls.

Last year OpenAI introduced JSON mode as a valuable tool for developers aiming to build reliable applications using their models.

Although JSON mode enhances the model’s ability to generate valid JSON outputs, but has I have highlighted in previous articles, it does not ensure that the responses will adhere to a specific schema. Which makes this a more experimental feature than a production ready feature.

Now, OpenAI is introducing Structured Outputs in the API, a new feature designed to guarantee that model-generated outputs will precisely match the JSON Schemas provided by developers.

Structured output is available in two formats, Function Calling & A new option for the response_format parameter.

The following Python code of a Function Calling example can be copied and pasted as-is into a notebook:

# Install the requests library if not already installed
!pip install requests
import requests
import json
# Define your OpenAI API key
api_key = ''
# Define the API endpoint
url = "https://api.openai.com/v1/chat/completions"
# Define the headers with the API key
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
# Define the data for the API request
data = {
"model": "gpt-4o-2024-08-06",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant. The current date is August 6, 2024. You help users query for the data they are looking for by calling the query function."
},
{
"role": "user",
"content": "look up all my orders in may of last year that were fulfilled but not delivered on time"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "query",
"description": "Execute a query.",
"strict": True,
"parameters": {
"type": "object",
"properties": {
"table_name": {
"type": "string",
"enum": ["orders"]
},
"columns": {
"type": "array",
"items": {
"type": "string",
"enum": [
"id",
"status",
"expected_delivery_date",
"delivered_at",
"shipped_at",
"ordered_at",
"canceled_at"
]
}
},
"conditions": {
"type": "array",
"items": {
"type": "object",
"properties": {
"column": {
"type": "string"
},
"operator": {
"type": "string",
"enum": ["=", ">", "=", "                                    },
"value": {
"anyOf": [
{
"type": "string"
},
{
"type": "number"
},
{
"type": "object",
"properties": {
"column_name": {
"type": "string"
}
},
"required": ["column_name"],
"additionalProperties": False
}
]
}
},
"required": ["column", "operator", "value"],
"additionalProperties": False
}
},
"order_by": {
"type": "string",
"enum": ["asc", "desc"]
}
},
"required": ["table_name", "columns", "conditions", "order_by"],
"additionalProperties": False
}
}
}
]
}
# Make the API request
response = requests.post(url, headers=headers, data=json.dumps(data))
# Print the response
print(response.status_code)
print(response.json())

JSON serves as a vital tool for structuring and exchanging data between AI agents and the functions they interact with, ensuring clear, consistent, and reliable communication across various systems and platforms.

✨✨ Follow me on LinkedIn for updates on Large Language Models

I’m currently the Chief Evangelist @ Kore AI. I explore & write about all things at the intersection of AI & language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces & more.

https://openai.com/index/introducing-structured-outputs-in-the-api

https://platform.openai.com/docs/guides/structured-outputs

The basic Retrieval-Augmented Generation (RAG) pipeline uses an encoder model to search for similar documents when given a query.

This is also called semantic search because the encoder transforms text into a high-dimensional vector representation (called an embedding) in which semantically similar texts are close together.

Before we had Large Language Models (LLMs) to create these vector embeddings, the BM25 algorithm was a very popular search algorithm. BM25 focuses on important keywords and looks for exact matches in the available documents. This approach is called keyword search.

If you want to take your RAG pipeline to the next level, you might want to try hybrid search. Hybrid search combines the benefits of keyword search and semantic search to improve search quality.

In this article, we will cover the theory and implement all three search approaches in Python.

Table of Contents

· RAG Retrieval
∘ Keyword Search With BM25
∘ Semantic Search With Dense Embeddings
∘ Semantic Search or Hybrid Search?
∘ Hybrid Search
∘ Putting It All Together
·…