Disclaimer:
This post won't dive deep into the nitty-gritty of diffusion models (e.g., noise schedules, re-parameterization of scores, difference between flow matching and score matching, etc.). It will focus on the high-level idea of how to adapt diffusion models to discrete space to build a generative model for discrete data.
Also, I will be primarily using diffusion model/score-matching-flavored rather than flow matching-flavored terminology.
At this point you probably are very familiar with how diffusion models or score/flow matching models work in general.
Just in case, diffusion models generate data following a target distribution through a multi-step Markov denoising process, where the model starts with random noise and gradually removes noise until clean data is reached.
For the purpose of training the model a noising process is also used, which adds noise to clean data, and the two processes are conceptually invertible with regard to each other.
This post discussed the differential equation view of diffusion models: they are learning a time-continuous map from noise to data.
Large language models (LLMs) are the backbone of the latest AI hype and are taking all the attention in the natural language processing (NLP) domain, or even most of the attention in the broad AI and ML community.
As we noticed in end-to-end learning and self-supervised learning for trajectories, trajectories share technical and semantic similarity with sentences in NLP, and that enables us to adapt many NLP techniques to trajectory learning.
The same story applies to adapting techniques introduced in LLMs to build so-called large trajectory models.
In this post, we will discuss the fundamentals of these techniques, what to look for when adapting them to trajectory learning, and the necessity and future directions for large trajectory models.
Trajectory generation, or trajectory synthesis, aims to generate trajectories that are not actually recorded in the available trajectory data, but are still realistic and follow a target distribution.
It is an important task for expanding the scale of trajectory datasets, when the availability of real-world trajectory data is limited.
It is also a unique task that does not fit neatly into the categories of end-to-end learning or self-supervised learning.
In this post, we will focus on the trajectory generation task, and discuss how the problem is defined, how to evaluate the quality and diversity of the generated trajectories, and introduce a general framework for solving the problem.
Self-supervised learning means training a model with unlabeled data, using supervisory signals extracted from the data itself. It usually does not set the model to perform a certain task, but aims to learn task-agnostic information in the data.
Compared to end-to-end learning, self-supervised learning does not rely on labeled data to train the model; in other words, it can be helpful when the availability of labeled data is limited.
This is also very relevant in the context of trajectory learning, since the availability of unlabeled trajectory datasets is usually much higher than that of labeled ones.
Self-supervised learning is also widely used to pre-train deep learning models. Put into the perspective of trajectories, self-supervised learning can help models get a general understanding of trajectory sequences or components in trajectories; the pre-trained models can later be fine-tuned on specific tasks, or be used directly for unsupervised tasks like clustering.
End-to-end learning means training a model to perform a task from input to output, supervising only on how the output aligns with the task's ground truth.
End-to-end is typically the most straightforward option for building a deep learning method for a certain task, and that also applies to most tasks related to spatiotemporal trajectories.
Illustration of end-to-end learning of spatiotemporal trajectories.
In this post we will categorize end-to-end learning tasks of trajectories from a technical standpoint: prediction, classification, and imputation.
For each category of tasks, we will give a general problem formulation and the general framework for solving it, and briefly discuss the motivation and use case of more specific downstream applications that fit into the category.
Schema overview of the three categories of end-to-end trajectory learning tasks.
A spatiotemporal trajectory is a sequence, with each item being a timestamped location. It records the movement of an object or a human through time and space.
Examples of human (left) and vehicle (right) trajectories.
With the development of technologies and devices that can record trajectories, such as GPS-equipped mobile phones, human society has gathered large-scale trajectory data.
Such data can be valuable for traffic planning and city development.
Yet, the scale of the data also creates challenges for traditional data analysis approaches.
To automate the process of trajectory data analysis, including feature extraction, embedding learning, performing tasks such as prediction, classification, and imputation, deep learning techniques have increasingly become prominent in the field.
What is a Trajectory?
A trajectory consists of a sequence of locations, each associated with a timestamp. Together, these points trace the path of a moving object or person through space and time.
A complete trajectory usually records the movement of the target from the beginning to the end of the movement process. For example, a complete taxi trajectory would span from the start of the trip (when the passenger gets into the taxi and the order starts) to the end of the trip (when the passenger gets off the taxi and the order terminates).
Formally, a trajectory can be represented as:
Up to yesterday morning, this blog site was generated from an Obsidian vault with Quartz 4.
Not anymore, since last night I refactored the whole site using Zola, and introduced a very minimalist design.
Comparison of how the blog site looks before (left) and after (right) the refactor.
Aside from artistic changes, the main reason behind this refactor is to use a static site generator (SSG) that has fewer dependencies and straight-forward control of templates.
I will dive deeper into the rationale and the refactor process below.
Why Refactor?
If it ain't broke, don't fix it. Following this philosophy it is unnecessary to refactor my blog site since it is working perfectly.
But this was something that stays in my mind for quite a while and now I finally decided to do it.
Before this refactor my blog site was generated using Quartz 4, an SSG that can turn a set of Markdown notes into static websites, particularly Obsidian-flavored Markdown with WikiLinks and callout blocks.
I will still recommend Quartz 4, if you want to quickly turn your Obsidian vault into a blog site, or if you want to build a wiki site with intertwined inner links jumping between notes. It is a free alternative to Obsidian's official publish service (which, aside from being overpriced in my opinion, doesn't even directly support custom domains and requires you to set up a reverse proxy for that purpose).
