Author: Guest Writer

Recent research and development have highlighted the emergence of Data Design in model training and fine-tuning processes.

This phenomenon is that models are trained not to necessarily imbue the model with knowledge, hence augmenting the Knowledge Intensive nature of the model.

But rather change the behaviour of the model, teaching the model new behaviour.

Can Conversation Designers Excel As Data Designers?

There has been many discussions on the future of conversation designers…and an idea came to mind…many of these datasets require human involvement in terms of annotation and oversight.

And these datasets hold key elements of dialog, reasoning and chains of thought…

So, the question which has been lingering in the back of my mind for the last couple of days is, is this not such a well suited task for conversation designers?

Especially in getting the conversational and thought process topology of the data right?

Allow me to explain, I have been talking much about a data strategy needing to consist of the Eight D’s: data discovery, data design, data development and data delivery.

Data delivery has been discuss much considering RAG and other delivery strategies.

Data Discovery has also been addressed to some degree, for instance XO Platform’s Intent Discovery. However, there is still much to do in finding new development opportunities…

Coming to Data Design…in this article I discuss three recent studies which focusses on teaching language models (both large and small) certain behaviours. While not necessarily imbuing the model with specific world knowledge, but rather improving the behaviour and abilities of the model.

These abilities can include self correction, reasoning abilities, improving contextual understanding, both short and long, and more…

Fine-tuning is one of the most popular techniques for adapting language models to specific tasks.

However, in most cases, this will require large amounts of computing power and resources.

Recent advances, among them PeFT, the parameter-efficient fine-tuning such as the Low-Rank Adaptation method, Representation Fine-Tuning, and ORPO…

Before you can build a machine learning model, you need to load your data into a dataset. Luckily, PyTorch has many commands to help with this entire process (if you are not familiar with PyTorch I recommend refreshing on the basics here).

PyTorch has good documentation to help with this process, but I have not found any comprehensive documentation or tutorials towards custom datasets. I’m first going to start with creating basic premade datasets and then work my way up to creating datasets from scratch for different models!

Before we dive into code for different use cases, let’s understand the difference between the two terms. Generally, you first create your dataset and then create a dataloader. A dataset contains the features and labels from each data point that will be fed into the model. A dataloader is a custom PyTorch iterable that makes it easy to load data with added features.

DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,
batch_sampler=None, num_workers=0, collate_fn=None,
pin_memory=False, drop_last=False, timeout=0,
worker_init_fn=None, *, prefetch_factor=2,
persistent_workers=False)

The most common arguments in the dataloader are batch_size, shuffle (usually only for the training data), num_workers (to multi-process loading the data), and pin_memory (to put the fetched data Tensors in pinned memory and enable faster data transfer to CUDA-enabled GPUs).

It is recommended to set pin_memory = True instead of specifying num_workers due to multiprocessing complications with CUDA.

In the case that your dataset is downloaded from online or locally, it will be extremely simple to create the dataset. I think PyTorch has good documentation on this, so I will be brief.

If you know the dataset is either from PyTorch or PyTorch-compatible, simply call the necessary imports and the dataset of choice:

from torch.utils.data import Dataset
from torchvision import datasets
from torchvision.transforms imports ToTensor
data = torchvision.datasets.CIFAR10('path', train=True, transform=ToTensor())

Each dataset will have unique arguments to pass into it (found here). In general, it will be the path the dataset is stored at, a boolean indicating if it needs to be downloaded or not (conveniently called download), whether it is training or testing, and if transforms need to be applied.

I dropped in that transforms can be applied to a dataset at the end of the last section, but what actually is a transform?

A transform is a method of manipulating data for preprocessing an image. There are many different facets to transforms. The most common transform, ToTensor(), will convert the dataset to tensors (needed to input into any model). Other transforms built into PyTorch (torchvision.transforms) include flipping, rotating, cropping, normalizing, and shifting images. These are typically used so the model can generalize better and doesn’t overfit to the training data. Data augmentations can also be used to artificially increase the size of the dataset if needed.

Beware most torchvision transforms only accept Pillow image or tensor formats (not numpy). To convert, simply use

To convert from numpy, either create a torch tensor or use the following:

From PIL import Image
# assume arr is a numpy array
# you may need to normalize and cast arr to np.uint8 depending on format
img = Image.fromarray(arr)

Transforms can be applied simultaneously using torchvision.transforms.compose. You can combine as many transforms as needed for the dataset. An example is shown below:

import torchvision.transforms.Compose
dataset_transform = transforms.Compose([
transforms.RandomResizedCrop(256),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

Be sure to pass the saved transform as an argument into the dataset for it to be applied in the dataloader.

In most cases of developing your own model, you will need a custom dataset. A common use case would be transfer learning to apply your own dataset on a pretrained model.

There are 3 required parts to a PyTorch dataset class: initialization, length, and retrieving an element.

__init__: To initialize the dataset, pass in the raw and labeled data. The best practice is to pass in the raw image data and labeled data separately.

__len__: Return the length of the dataset. Before creating the dataset, the raw and labeled data should be checked to be the same size.

__getitem__: This is where all the data handling occurs to return a given index (idx) of the raw and labeled data. If any transforms need to be applied, the data must be converted to a tensor and transformed. If the initialization contained a path to the dataset, the path must be opened and data accessed/preprocessed before it can be returned.

Example dataset for a semantic segmentation model:

from torch.utils.data import Dataset
from torchvision import transforms
class ExampleDataset(Dataset):
"""Example dataset"""
def __init__(self, raw_img, data_mask, transform=None):
self.raw_img = raw_img
self.data_mask = data_mask
self.transform = transform
def __len__(self):
return len(self.raw_img)
def __getitem__(self, idx):
if torch.is_tensor(idx):
idx = idx.tolist()
image = self.raw_img[idx]
mask = self.data_mask[idx]
sample = {'image': image, 'mask': mask}
if self.transform:
sample = self.transform(sample)
return sample

Google Analytics 4 is currently estimated to be used by 15.6 million websites in the world, meaning…

What type of company you join is an incredibly important decision. Even if the company is prestigious and pays you well, if the work environment is not a fit, you’ll burn out eventually.

Many people join a startup or a big tech company without a good understanding of what it’s actually like to work there, and often end up disappointed. In this article, I will cover the key differences based on my experience working at companies ranging from a small 10-person startup to big tech giants like Uber and Meta. Hopefully this will help you decide where you want to go.

If you want to skim the article, I am adding a brief summary (“TL;DR” = “Too long, didn’t read”) at the end of each section (something I learned at Uber).

Think of a tech company you know. Chances are, you thought of Google, Meta, Amazon, Apple or a similar large company.

Based on these companies’ reputation, most people assume that anyone who works there meets a very high bar for excellence. While that’s not necessarily true (more on that below), this so-called “halo effect” can help you. Once you have the “stamp of approval” from a big tech company on your resume, it is much easier to find a job afterwards.

Many companies think: “If that person is good enough to be a Data Scientist at Google, they will be good enough for us. I’m sure Google did their due diligence”.

Coming to the US from Germany, most hiring managers and recruiters didn’t know the companies I used to work for. Once I got a job at Uber, I was flooded with offers, including from companies that had rejected me before.

You might find that unfair, but it’s how the system currently works, and you should consider this when choosing a company to work for.

TL;DR: Working for a prestigious company early in your career can open a lot of doors.

As mentioned above, people often assume that FAANG companies only hire the best and brightest.

In reality, that’s not the case. One thing I learned over the years is that any place in the world has a normal distribution of skill and talent once it reaches a certain size. The distribution might be slightly offset on the X axis, but it’s a normal distribution nonetheless.

Many of of the most well-known companies started out being highly selective, but as they grew and ramped up hiring, the level of excellence started reverting to the mean.

Counterintuitively, that means that some small startups have more elite teams than big tech companies because they can afford to hand-pick every single new hire. To be sure, you’ll need to judge the caliber of the people first-hand during the interview process.

TL;DR: You’ll find smart people in both large and small companies; it’s a fallacy that big tech employs higher-caliber people than startups.

How much you’ll earn depends on many factors, including the specific company, the level you’re being offered, how well you negotiate etc.

The main thing to keep in mind: It’s not just about how much you make, but also how volatile and liquid your compensation is. This is affected by the composition of your pay package (salary vs. equity (illiquid private company-stock vs. liquid public company stock)) and the stage of the company.

Here is how you can think about it at a high level:

• Early-stage: Small startups will offer you lower base salaries and try to make up for that by promising high equity upside. But betting on the equity upside of an early-stage startup is like playing roulette. You might hit it big and never have to work again, but you need to be very lucky; the vast majority of startups fail, and very few turn into unicorns.
• Big Tech: Compensation in big tech companies, on the other hand, is more predictable. The base salary is higher (e.g. see the O’Reilly 2016 Data Science Salary Survey) and the equity is typically liquid (i.e. you can sell it as soon as it vests) and less volatile. This is a big advantage since in pre-IPO companies you might have to wait years for your equity to actually be worth something.
• Growth stage: Growth stage companies can be an interesting compromise; they have a much higher chance of exiting successfully, but your equity still has a lot of upside. If you join 2–3 top-tier growth stage companies over the years, there is a good chance you’ll end up with at least one solid financial outcome. Pay in some of these companies can be very competitive; my compensation actually increased when I moved from Meta to Rippling.

TL;DR: Instead of just focusing on salary, choose the pay package that fits your appetite for risk and liquidity needs.

We all want job security.

We might not stay in a job for our entire career, but at least we want to be able to choose ourselves when we leave.

Startups are inherently riskier than big companies. Is the founder up to the job? Will you be able to raise another round of financing? Most of these risks are existential; in other words, the earlier the stage of the company you join, the more likely it is it won’t exist anymore 6–12 months from now.

At companies in later stages, some of these risks have already been eliminated or at least reduced.

In exchange, you’re adding another risk, though: Increased layoff risk. Startups only hire for positions that are business critical since they are strapped for cash. If you get hired, you can be sure they really needed another Data Scientist and there is plenty of work for you to do that is considered central to the startup’s success.

In large companies, though, hiring is often less tightly controlled, so there is a higher risk you’ll be hired into a role that is later deemed “non-essential” and you will be part of sweeping layoffs.

TL;DR: The earlier the company stage, the more risk you take on. But even large companies aren’t “safe” anymore (see: layoffs)

A job at a startup and a large company are very different.

The general rule of thumb is that in earlier-stage companies you’ll have a broader scope. For example, if you join as the first data hire in a startup, you’ll likely act as part Data Engineer, part Data Analyst and part Data Scientist. You’ll need to figure out how to build out the data infrastructure, make data available to business users, define metrics, run experiments, build dashboards, etc.

Your work will also likely range across the entire business, so you might work with Marketing & Sales data one day, and with Customer Support data the next.

In a large company, you’ll have a narrowly defined scope. For example, you might spend most of your time forecasting a certain set of metrics.

The trade-off here is breadth vs. depth & scale: At a startup, your scope is broad, but because you are stretched so thin, you don’t get to go deep on any individual problem. In a large company, you have a narrow scope, but you get to develop deep subject matter expertise in one particular area; if this expertise is in high demand, specializing like this can be a very lucrative path. In addition, anything you do touches millions or even billions of users.

TL;DR: If you want variety, join a startup. If you want to build deep expertise and have impact at scale, join Big Tech. A growth stage company is a good compromise.

When I joined UberEats in 2018, I didn’t get any onboarding. Instead, I was given a set of problems to solve and asked to get going.

If you are used to learning in a structured way, e.g. through lectures in college, this can be off-putting at first. How are you supposed to know how to do this? Where do you even start?

But in my experience, working on a variety of challenging problems is the best way to learn about how a business works and build out your hard and soft skills. For example, coming out of school my SQL was basic at best, but being thrown into the deep end at UberEats forced me to become good at it within weeks.

The major downside of this is that you don’t learn many best practices. What does a best-in-class data infrastructure look like? How do the best companies design their metrics? How do you execute thousands of experiments in a frictionless way while maintaining rigor? Even if you ultimately want to join a startup, seeing what “good” looks like can be helpful so you know what you’re building towards.

In addition, large companies often have formalized training. Where in a startup you have to figure everything out yourself, big tech companies will typically provide sponsored learning and development offerings.

TL;DR: At early-stage companies you learn by figuring things out yourself, at large companies you learn through formal training and absorbing best practices.

We already talked about how working at prestigious companies can help when you’re looking for a new job. But what about your growth within the company?

At an early-stage company, your growth opportunities come as a direct result of the growth of the company. If you join as an early data hire and you and the company are both doing well, it’s likely you’ll get to build out and lead a data team.

Most of the young VPs and C-Level executives you see got there because their careers were accelerated by joining a “rocket ship” company.

There is a big benefit of larger companies, though: You typically have a broader range of career options. You want to work on a different product? No need to leave the company, just switch teams. You want to move to a different city or country? Probably also possible.

TL;DR: Early-stage, high-growth companies offer the biggest growth opportunities (if the company is successful), but large companies provide flexibility.

There are many types of stress. It’s important to figure out which ones you can handle, and which ones are deal-breakers for you.

At fast-growing early-stage companies, the main source of stress comes from:

• Changing priorities: In order to survive, startups need to adapt. The original plan didn’t work out? Let’s try something else. As a result, you can rarely plan longer than a few weeks ahead.
• Fast pace: Early-stage companies need to move fast; after all, they need to show enough progress to raise another financing round before they run out of money.
• Broad scope: As mentioned above, everyone in an early-stage company does a lot of things; it’s easy to feel stretched thin. Most of us in the analytics realm like to do things perfectly, but in a startup you rarely get the chance. If it’s good enough for now, move on to the next thing!

In large companies, stress comes from other factors:

• Complexity: Larger companies come with a lot of complexity. An often convoluted tech stack, lots of established processes, internal tools etc. that you need to understand and learn to leverage. This can feel overwhelming.
• Politics: At large companies, it can sometimes feel like you’re spending more time debating swim lanes with other teams than doing actual work.

TL;DR: Not all stress is created equal. You need to figure out what type of stress you can deal with and choose your company accordingly.

There is no one-size-fits-all answer to this question. However, my personal opinion is that it helps to do at least one stint at a reputable big tech company early in your career, if possible.

This way, you will:

• Get pedigree on your resume that will help you get future jobs
• See what a high-performing data infrastructure and analytics org at scale looks like
• Get structured onboarding, coaching and development

This will provide you with a solid foundation, whether you want to stay in big tech or jump into the crazy world of startups.

Working at a small startup, growth stage company or FAANG tech company is not inherently better or worse. Each company stage has its pros and cons; you need to decide for yourself what you value and what environment is the best fit for you.

For more hands-on advice on how to scale your career in data & analytics, consider following me here on Medium, on LinkedIn or on Substack.

Choosing the right language model depends on an organisation’s specific needs, task complexity, and available resources.

Small language models are ideal for organisations aiming to build applications that run locally on a device (rather than in the cloud).

Some argue that Large language models are better suited for applications requiring the orchestration of complex tasks, advanced reasoning, data analysis, and contextual understanding.

Small language models provide potential solutions for regulated industries and sectors needing high-quality results while keeping data on their own premises.

Latency refers to the delay in communication between Large Language Models (LLMs) and the cloud when retrieving information to generate answers to user prompts. In some use-cases cases, makers can prioritise waiting for high-quality answers, while in others, speed is crucial for user satisfaction. However, for conversational experiences, latency is a non-negotiable.

Cost is also a consideration that makes the use of SLMs very attractive.

Small Language Models (SLMs), which can operate offline, significantly broaden AI’s applicability.

What we’re going to start to see is not a shift from large to small, but a shift from a singular category of models to a portfolio of models where customers get the ability to make a decision on what is the best model for their scenario .~ Source

There are a number of features which are part and parcel of Large Language Models. For instance Natural Language Generation, dialog and conversational context management, reasoning and knowledge.

The knowledge portion is not being used to a large degree as RAG is used to inject contextual reference data at inference. Hence Small Language Models (SLMs) are ideal, even though they lack being knowledge intensive.

Seeing that the knowledge intensive nature of LLMs are not used in any-case.

The model simply does not have the capacity to store too much “factual knowledge”, which can be seen for example with low performance on TriviaQA.

However, we believe such weakness can be resolved by augmentation with a search engine. ~ Source

Another limitation related to the model’s capacity is that the model is primarily restricted to English.

Below is an example of how anyone can interact with the Phi-3 small language model. Within HuggingFace’s HuggingChat, anyone can go to settings and under models select the Phi-3 model.

As we all know, a big part of a data scientist’s job is to clean and preprocess data. A huge part of this involves outlier detection and removal. Large outliers, spikes and bad data can really interfere with training an accurate machine learning model, so it’s important that outliers are handled properly.

The P&F data science team faces a challenge: They must weigh each expert opinion equally, but can’t satisfy everyone. Instead of focusing on expert subjective opinions, they decide to evaluate the chatbot on historical customer questions. Now experts do not need to come up with questions to test the chatbot, bringing the evaluation closer to real-world conditions. The initial reason for involving experts, after all, was their better understanding of real customer questions compared to the P&F data science team.

It turns out that commonly asked questions for P&F are related to paper clip technical instructions. P&F customers want to know detailed technical specifications of the paper clips. P&F has thousands of different paper clip types, and it takes a long time for customer support to answer the questions.

Understanding the test-driven development, the data science team creates a dataset from the conversation history, including the customer question and customer support reply:

Having a dataset of questions and answers, P&F can test and evaluate the chatbot’s performance retrospectively. They create a new column, “Chatbot reply”, and store the chatbot example replies to the questions.

We can have the experts and GPT-4 evaluate the quality of the chatbot’s replies. The ultimate goal is to automate the chatbot accuracy evaluation by utilizing GPT-4. This is possible if experts and GPT-4 evaluate the replies similarly.

Experts create a new Excel sheet with each expert’s evaluation, and the data science team adds the GPT-4 evaluation.

There are conflicts on how different experts evaluate the same chatbot replies. GPT-4 evaluates similarly to expert majority voting, which indicates that we could do automatic evaluations with GPT-4. However, each expert’s opinion is valuable, and it’s important to address the conflicting evaluation preferences among the experts.

P&F organizes a workshop with the experts to create golden standard responses to the historical question dataset

and evaluation best practice guidelines, to which all experts agree.

With the insights from the workshop, the data science team can create a more detailed evaluation prompt for the GPT-4 that covers edge cases (i.e. “chatbot should not ask to raise support tickets”). Now the experts can use time to improve the paper clip documentation and define best practices, instead of laborious chatbot evaluations.

By measuring the percentage of correct chatbot replies, P&F can decide whether they want to deploy the chatbot to the support channel. They approve the accuracy and deploy the chatbot.

Finally, it’s time to save all the chatbot responses and calculate how well the chatbot performs to solve real customer inquiries. As the customer can directly respond to the chatbot, it is also important to record the response from the customer, to understand the customer’s sentiment.

The same evaluation workflow can be used to measure the chatbot’s success factually, without the ground truth replies. But now the customers are getting the initial reply from a chatbot, and we do not know if the customers like it. We should investigate how customers react to the chatbot’s replies. We can detect negative sentiment from the customer’s replies automatically, and assign customer support specialists to handle angry customers.