Blog

Pre-training large models on time series data faces several challenges: the lack of a comprehensive public time series repository, the complexity of diverse time series characteristics, and the infancy of experimental benchmarks for model evaluation, especially under resource-constrained and minimally supervised scenarios. Despite these hurdles, time series analysis remains vital across applications like weather forecasting, heart rate irregularity detection, and anomaly identification in software deployments. Utilizing pre-trained language, vision, and video models offers promise, though adaptation to time series data specifics is necessary for optimal performance.

Applying transformers to time series analysis presents challenges due to the quadratic growth of the self-attention mechanism with input token size. Treating time series sub-sequences as tokens enhances efficiency and effectiveness in forecasting. Utilizing cross-modal transfer learning from language models, ORCA extends pre-trained models to diverse modalities through align-then-refine fine-tuning. Recent studies have utilized this approach to reprogram language pre-trained transformers for time series analysis, albeit resource-intensive models require substantial memory and computational resources for optimal performance.

Researchers from Carnegie Mellon University and the University of Pennsylvania present MOMENT, an open-source family of foundation models for general-purpose time series analysis. It utilizes the Time series Pile, a diverse collection of public time series, to address time series-specific challenges and enable large-scale multi-dataset pretraining. These high-capacity transformer models are pre-trained using a masked time series prediction task on extensive data from various domains, offering versatility and robustness in tackling diverse time series analysis tasks.

MOMENT begins by assembling a diverse collection of public time series data called the Time Series Pile, combining datasets from various repositories to address the scarcity of comprehensive time-series datasets. These datasets encompass long-horizon forecasting, short-horizon forecasting, classification, and anomaly detection tasks. MOMENT’s architecture involves a transformer encoder and a lightweight reconstruction head pre-trained on a masked time series prediction task. The pre-training setup includes variations of MOMENT corresponding to different sizes of encoders, trained with Adam optimizer and gradient checkpointing for memory optimization. MOMENT is designed for fine-tuning downstream tasks such as forecasting, classification, anomaly detection, and imputation, either end-to-end or with linear probing, depending on the task requirements.

The study compares MOMENT with state-of-the-art deep learning and statistical machine learning models across various tasks, contrary to TimesNet, which mainly focuses on transformer-based approaches. These comparisons are essential for evaluating the practical applicability of the proposed methods. Interestingly, statistical and non-transformer-based methods, such as ARIMA for short-horizon forecasting, N-BEATS for long-horizon forecasting, and k-nearest neighbors for anomaly detection, demonstrate superior performance over many deep learning and transformer-based models.

To recapitulate, this research presents MOMENT, the first open-source family of time series foundation models developed through comprehensive stages of data compilation, model pre-training, and systematic addressing of time series-specific challenges. By utilizing the Time Series Pile and innovative strategies, MOMENT demonstrates high performance in pre-training transformer models of various sizes. Also, the study designs an experimental benchmark for evaluating time series foundation models across multiple practical tasks, particularly emphasizing scenarios with limited computational resources and supervision. MOMENT exhibits effectiveness across various tasks, showcasing superior performance, especially in anomaly detection and classification, attributed to its pre-training. The research also underscores the viability of smaller statistical and shallower deep learning methods across many tasks. Ultimately, the study aims to advance open science by releasing the Time Series Pile, along with code, model weights, and training logs, fostering collaboration and further advancements in time series analysis.

Check out the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 42k+ ML SubReddit

Asjad is an intern consultant at Marktechpost. He is persuing B.Tech in mechanical engineering at the Indian Institute of Technology, Kharagpur. Asjad is a Machine learning and deep learning enthusiast who is always researching the applications of machine learning in healthcare.

The need to convert PDF documents into more manageable and editable formats like markdowns is increasingly vital, especially for those dealing with academic and scientific materials. These PDFs often contain complex elements such as multi-language text, tables, code blocks, and mathematical equations. The primary challenge in converting these documents lies in accurately maintaining the original layout, formatting, and content, which standard text converters often need help to handle.

There are already some solutions available aimed at extracting text from PDFs. Optical Character Recognition (OCR) tools are commonly used to interpret and digitize the text contained within these files. However, while these tools can handle straightforward text extraction, they frequently need to improve when preserving the intricate layouts of academic and scientific documents. Issues such as misaligned tables, misplaced text fragments, and loss of critical formatting are commonplace, leading to outputs that require significant manual correction to be helpful.

In response to these challenges, a new tool called “Marker” has been developed that significantly enhances the accuracy and utility of converting PDFs into markdown. Marker is designed to tackle the complexities of high-density information documents like books and research papers. It supports extensive document types and is optimized for content in any language. Crucially, Marker not only extracts text but also carefully maintains the structure and formatting of the original PDF, including accurately converting tables, code blocks, and most mathematical equations into LaTeX format. Additionally, Marker can extract images from the documents and integrate them appropriately into the resultant markdown files.

It has been finely tuned to efficiently handle large volumes of data, utilizing GPU, CPU, or MPS platforms to optimize processing speed and accuracy. This capability ensures that it operates within a reasonable usage of computational resources, typically requiring around 4GB of VRAM, which is on par with other high-performance document conversion tools. Benchmarks comparing Marker to existing solutions highlight its superior ability to maintain the integrity and layout of complex document formats while ensuring the converted text remains true to the original content.

Further setting Marker apart is its tailored approach to handling different types of PDFs. It is particularly effective with digital PDFs, where the need for OCR is minimized, thus allowing for faster and more accurate conversions. The developers have acknowledged some limitations, such as the occasional imperfect conversion of equations to LaTeX and minor issues with table formatting. 

In conclusion, Marker represents a significant step forward in document conversion technology. It addresses the critical challenges faced by users who need to manage complex documents by providing a solution that not only converts text but also respects and reproduces the original formatting and structure. With its robust performance metrics and adaptability to various document types and languages, Marker is poised to become an essential resource for academics, researchers, and anyone involved in extensive document handling. As digital content grows both in volume and complexity, having reliable tools to facilitate easy and accurate conversion will be paramount.

Niharika is a Technical consulting intern at Marktechpost. She is a third year undergraduate, currently pursuing her B.Tech from Indian Institute of Technology(IIT), Kharagpur. She is a highly enthusiastic individual with a keen interest in Machine learning, Data science and AI and an avid reader of the latest developments in these fields.

On May 13, OpenAI held its Spring update event, at which the company announced its newest model, GPT-4o, an AI model with a GPT-4 level of intelligence. The “o” in GPT-4o means omnimodal capabilities due to its ability to process and integrate text, vision, and audio. The event was overall good and properly highlighted everything important and relevant, including the major announcement of the official ChatGPT desktop app for Mac.

Before we elaborate on the new ChatGPT Mac desktop app, here are a few worth mentioning:

• The event started and ended with positivity, and the first big news was that ChatGPT has over 100 million users worldwide. 
• The most exciting announcement was the introduction of its new model, GPT-4o. This new model is accessible to all. It will be available for free to all ChatGPT users.
• The new GPT-4o can better understand and respond to voice commands, allowing you to interrupt between the responses to change the topic or tone or get a better response output. 
• The new and improved vision capability can see what’s going around you and respond to you accordingly, and even help you solve complex questions. 
• The new model can even assist you with coding in real-time just by using the desktop app by highlighting and pressing cmd + C. But for now, this is only available on the Macs.
• Lastly, the new model’s speech translation ability was one of its highlights because of its speed and lack of awkward lags in between.

The most shocking and interesting news was the ChatGPT desktop application for Macs. It surprised a few people as Microsoft had invested over $13 billion in OpenAI. Yet, OpenAI came up with the Mac application first, possibly due to the highly rumored Apple and OpenAI deal, which is said to be close to being a done deal.

As suggested, the Mac application is only available to macOS users and requires macOS 14.0 or later. Although the news is big, not everyone is happy because, as stated before, investors, like Microsoft, expressed disappointment that the first desktop app release was for macOS and not Windows. However, OpenAI has assured that they plan to launch a Windows version later this year, as most of their users are Windows users.

The ChatGPT app can also live and stay pinned in your taskbar for quick and easy access. 

The Mac application can monitor your screen and appear on any opened tab. 

You can easily share any picture or screenshot with it using the drag-and-drop option.

Within seconds, you will receive a response related to the image.

Finally, the app can communicate with you like an assistant and present you with appropriate answers to your queries.

In Conclusion:

The OpenAI Spring update event was quite eventful and exciting. The introduction of the new GPT-4o model with its omnimodal capabilities is particularly intriguing. Additionally, the ChatGPT desktop app announcement for Mac is a significant development. Some are disappointed by the lack of a Windows version, but that will not last long. Overall, it’s clear that OpenAI is making impressive steps in AI technology. Now we wait and hope to get GPT-5 later this year!

Nishant, the Product Growth Manager at Marktechpost, is interested in learning about artificial intelligence (AI), what it can do, and its development. His passion for trying something new and giving it a creative twist helps him intersect marketing with tech. He is assisting the company in leading toward growth and market recognition.

In the dynamic field of AI technology, a pressing challenge for the drug discovery (DD) community, especially in structural biology and computational chemistry, is the creation of innovative models finely tuned for drug design. The core challenge lies in accurately and efficiently predicting molecular properties crucial for understanding protein-ligand interactions and optimizing binding affinities, essential for advancing effective drug development initiatives.

In current structural biology and drug design, researchers commonly depend on existing datasets and methods, which have inherent limitations like structural inaccuracies, crystallographic artifacts, and difficulties in accurately capturing the dynamic nature of protein-ligand interactions. Traditional approaches for predicting molecular properties often lack the necessary detail for complex protein-ligand interactions, neglecting the vital role of dynamics and flexibility in understanding binding mechanisms and affinity.

Researchers from the Institute of Structural Biology, Technical University of Munich, Jülich Supercomputing Centre, Helmholtz AI, Cambridge University, Jagiellonian University, and Institute of Computational Biology propose MISATO, marking a transformative shift in drug discovery and structural biology methodologies. MISATO addresses the limitations of existing methods by integrating quantum-chemically refined ligand data, molecular dynamics (MD) simulations, and advanced AI models. This comprehensive approach facilitates a nuanced understanding of molecular properties, capturing electronic structure details and dynamic behavior crucial for accurate predictions. 

MISATO takes a comprehensive approach, utilizing semi-empirical quantum chemical methods to refine ligand datasets. This method captures electronic properties with high accuracy, while also analyzing both electronic structure details and dynamic behavior, crucial for precise predictions. Additionally, classical MD simulations within MISATO characterize the dynamic behavior and conformational landscape of protein-ligand complexes, offering insights into binding mechanisms and flexibility. AI models integrated into MISATO, such as graph neural networks (GNNs), are trained on this enriched dataset to predict properties like adaptability, binding affinities, and thermodynamic parameters. Extensive experimental validations confirm the efficacy of these models in accurately predicting key molecular properties crucial for drug discovery.

In conclusion, MISATO signifies a key stride in AI-driven drug discovery and structural biology. By integrating quantum chemistry, MD simulations, and advanced AI models, MISATO provides a holistic and robust solution to challenges in structure-based drug design, enhancing accuracy and efficiency and empowering researchers with potent tools.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 42k+ ML SubReddit

Aswin AK is a consulting intern at MarkTechPost. He is pursuing his Dual Degree at the Indian Institute of Technology, Kharagpur. He is passionate about data science and machine learning, bringing a strong academic background and hands-on experience in solving real-life cross-domain challenges.

Large Language Models (LLMs) like GPT-3 and ChatGPT exhibit exceptional capabilities in complex reasoning tasks such as mathematical problem-solving and code generation, far surpassing standard supervised machine learning techniques. The key to unlocking these advanced reasoning abilities lies in the chain of thought (CoT), which refers to the ability of the model to generate intermediate reasoning steps before arriving at the final answer, kind of like how we humans break down a complex problem into smaller steps in our head. This can be achieved through methods like training the model on examples enriched with intermediate reasoning steps or using few-shot prompting to instruct the model to generate a CoT.

Now, you might think that the contents of these intermediate steps is what allows the model to reason better. But interestingly, in this study, the researchers found that even if the intermediate steps are incorrect or completely random, just the act of generating them still helps the model a lot. It’s like the model is being told “Okay, think this through step-by-step” and that alone improves its reasoning ability drastically.

So the researchers wanted to understand why this “chain of thought” approach is so powerful for transformers (the type of model used in GPT-3, etc). They used concepts from circuit complexity theory and adopted the language of computational complexity classes like NC, AC, and TC to analyze this problem.

Essentially, they found that without the chain of thought, transformers are limited to efficiently performing only parallel computations, meaning they can solve problems that can be broken down into independent sub-tasks that can be computed simultaneously.

However, many complex reasoning tasks require inherently serial computations, where one step follows from the previous step. And this is where the chain of thought helps transformers a lot. By generating step-by-step reasoning, the model can perform many more serial computations than it could without CoT.

The researchers proved theoretically that while a basic transformer without CoT can only solve problems up to a certain complexity level, allowing a polynomial number of CoT steps makes transformers powerful enough to solve almost any computationally hard problem, at least from a theoretical perspective.

To back up their theory, they also did some experiments on different arithmetic tasks – ones that can be parallelized and ones that inherently require sequential computations. Sure enough, they found that transformers struggled on the sequential tasks without CoT, but enabling CoT drastically boosted their performance, especially when the transformer model was relatively small/shallow.

In essence, the chain of thought is a simple but powerful trick that vastly increases the reasoning capabilities of transformer models like GPT-3. It allows them to tackle complex tasks requiring sequential logic that parallel models would fail at. 

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 42k+ ML SubReddit

Vineet Kumar is a consulting intern at MarktechPost. He is currently pursuing his BS from the Indian Institute of Technology(IIT), Kanpur. He is a Machine Learning enthusiast. He is passionate about research and the latest advancements in Deep Learning, Computer Vision, and related fields.