Most software that relies heavily on cloud infrastructure works like this: the full data is stored on the cloud, and only the data needed for now is fetched or updated by the software.
There are lots of problems with this infrastructure.
First, it means the software cannot function without a stable internet connection. In other words, anything related to internet connection can break your software, say, for example: you are on an airplane, your internet provider breaks down, the cloud provider breaks down, your workplace faces a power outage, or your VPN connection is not stable, etc.
Second, there is no guarantee that you can always fetch the data and it is always intact. The cloud provider can block your access to the data for a variety of reasons (geopolitical regulations, policy updates, cancellation of subscription, etc.). Also, if the cloud provider gets hacked, anything can happen to the data.
In conclusion, using software that relies too much on cloud infrastructure can be annoying at best, and can cost you valuable data at worst.
This is a very concise walkthrough of my main home server running NixOS. I assume the reader already has basic knowledge about NixOS.
neofetch screen of my home server.
My home server (or many would rather call it a NAS) serves common home server purposes: bulk storage, basic file sharing, media streaming service, and photo backup.
Physical Setup
Below is a recent photo of my home server, living in the utility closet together with my network equipments.
Real-world photo of the home server.
It is essentially an Intel N305 custom motherboard with SATA back panel and a 3D-printed enclosure. I bought it on Taobao last time I went back to China to visit my family.
Not very important here, as long as you stick to common hardware, it should be relatively straightforward to install NixOS and replicate my setup.
The Transformer network is position-agnostic.
In other words, it doesn't care about the order of the input sequence.
You can easily see the reason from how the Attention mechanism, the primary component in the Transformer, calculates attention weights:
It only depends on the content of the queries and keys, not where they are positioned in a sequence.
But in many cases, we want the network to be able to distinguish the positions of tokens in a sequence.
For example, you certainly don't want the network to interpret the sentence "Jack beats John" exactly the same as "John beats Jack".
Thus the idea of positional encoding is born.
We can explicitly include information about positions of queries and keys in their content.
The network is certainly able to distinguish "1-Jack 2-beats 3-John" from "1-John 2-beats 3-Jack" even when it cannot directly access positional information.
Most modern large language models (LLMs) use the decoder-only Transformer structure coupled with auto-regressive (AR) generation scheme. In other words, they generate text one token at a time, with each new token predicted based on all previously generated tokens. Nowadays, when people talk about and adopt Transformers, they are mostly referring to this framework.
However, you might remember that before the introduction of GPT3, BERT framework was the star of the show, and it provides a completely different formulation of Transformers: BERT is encoder-only with Masked Language Model (MLM) generation scheme. In other words, BERT randomly masks certain tokens in the input and predicts them based on the surrounding context from both directions.
Despite BERT failing to dominate LLMs, they are still highly relevant, both in the NLP domain and in other domains. For the NLP domain, decoder-only structure is fundamentally not suitable for calculating text embeddings (which extends to downstream tasks like RAG), and BERT-like structure is preferred. For other domains, there are also many scenarios where the BERT framework is more suitable. For example, when the data is not inherently sequential, when the task needs bidirectional blank in-filling instead of unidirectional step-by-step generation, or you want to build a multi-task method instead of a pure generative method.
One of the "missing pieces" to build an AI agent for a specific domain, like the spatiotemporal domain I mentioned in this post, is to enable LLMs' understanding of data other than text, so that the feedback loop can be closed. The problem essentially comes down to building a multi-modal LLM which can take (or even produce) data other than text.
There are of course lots of existing successful techniques developed and adopted to solve this general problem, especially for images. In this post I touched on the topic of multi-modal LLMs, but focused on how to feed multi-modal data into LLMs (from an input embedding standpoint). This post will be more focused on a higher-level: how to train an LLM so that it actually understands multi-modal data, with images as the primary example.
Train on Data-Text Pairs
The most straight-forward method to bridge multi-modal data and text is to train an LLM with pairs of data and text. And spoiler: This step is basically inevitable, at least at current state of AI.
Moving beyond large language models (LLMs), agents are the new hot topic in AI research, and needless to say, lots of researchers in spatiotemporal data mining (ST-DM), like us, are trying to propose novel ideas about AI agents for spatiotemporal data.
First of all, what is an agent and how it differentiates from an LLM? As Anthropic stated in this post:
Quotes:
"Agent" can be defined in several ways. Some customers define agents as fully autonomous systems that operate independently over extended periods, using various tools to accomplish complex tasks. Others use the term to describe more prescriptive implementations that follow predefined workflows. At Anthropic, we categorize all these variations as agentic systems, but draw an important architectural distinction between workflows and agents:
Workflows are systems where LLMs and tools are orchestrated through predefined code paths.
Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.
TL;DR: In the context of generative modeling, we examine ODEs, SDEs, and two recent works that share the idea of learning shortcuts that traverse through vector fields defined by ODEs faster. We then discuss the generalization of this idea to both ODE- and SDE-based models.
Holderrieth and Erives. "An Introduction to Flow Matching and Diffusion Models."
Differential Equations
Let's start with a general scenario of generative modeling: suppose you want to generate data $x$ that follows a distribution $p(x)$. In many cases, the exact form of $p(x)$ is unknown. What you can do is follow the idea of normalizing flow: start from a very simple, closed-form distribution $p(x_0)$ (for example, a standard normal distribution), transform this distribution through time $t\in [0, 1]$ with intermediate distributions $p(x_t)$, and finally obtain the estimated distribution $p(x_1)$.
