Daniel Chen

The Only Linear Algebra You Must Know

2026-02-25T00:00:00+00:00

Linear algebra is often taught backwards.

A standard course starts with row reduction, determinants, matrix multiplication, eigenvalues, diagonalization, etc. These are all useful, but taught in that order, the subject can feel like a list of unrelated tricks.

The more useful framing is:

A matrix is not a grid of numbers. A matrix is a coordinate description of a linear transformation.

That sentence is most of linear algebra.

Everything else is bookkeeping.

The few facts worth remembering are stated explicitly below. The proofs are short because the point is not to make the subject feel more formal. The point is to make the conceptual dependencies visible.

Vectors are not columns of numbers

A vector is an element of a vector space.

Definition 1 (vector space). A vector space over a field $\mathbb{F}$ is a set $V$ whose elements can be added together and scaled by elements of $\mathbb{F}$, in a way that behaves the way addition and scaling are supposed to behave: \[ u+v \in V, \quad \lambda v \in V, \] and the usual rules like associativity, commutativity of addition, distributivity, additive identity, and additive inverses all hold.

That definition is intentionally plain. A vector space is just a place where linear combinations make sense.

The important operation is:

\[x_1v_1 + \cdots + x_nv_n\]

Once objects can be added and scaled, linear algebra can begin.

The column of numbers is what appears after choosing a basis.

Definition 2 (coordinate representation). Let $V$ be a finite-dimensional vector space over $\mathbb{F}$, and let $B = \{e_1,\dots,e_n\}$ be a basis of $V$. If \[ v = x_1e_1 + \cdots + x_ne_n, \] then the coordinate representation of $v$ in the basis $B$ is \[ [v]_B = \begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix}. \]

For example, the notation

\[v = \begin{bmatrix} 3 \\ 5 \end{bmatrix}\]

usually means

\[v = 3e_1 + 5e_2\]

where

\[e_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \quad e_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}\]

But the important thing is that the standard basis has been chosen.

The vector is not inherently the pair of numbers (3, 5). The pair of numbers is the vector’s address in a particular coordinate system.

This sounds pedantic until the source of confusion becomes clear: many linear algebra mistakes come from forgetting this distinction.

Bases are coordinate systems

A basis is just a coordinate system for a vector space.

If (B = {e_1, \dots, e_n}) is a basis for a vector space (V), then every vector (v \in V) can be written uniquely as

\[v = x_1 e_1 + \cdots + x_n e_n\]

So choosing (B) gives a coordinate map:

\[\Phi_B : V \to \mathbb{F}^n\]

where

\[\Phi_B(v) = \begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix}\]

This map is doing something conceptually important. It takes an abstract vector and gives coordinates for it.

The vector space did not come with coordinates. The coordinates were added by choosing a basis.

Linear transformations are the actual objects

The real object in linear algebra is a linear transformation.

A function

\[T : V \to W\]

is linear if

\[T(v + w) = T(v) + T(w)\]

and

\[T(\lambda v) = \lambda T(v)\]

That is all linearity means: the function respects the vector space structure.

The reason bases matter is that a linear transformation is completely determined by what it does to basis vectors.

Proposition 1 (a linear map is determined by a basis). Let $T : V \to W$ be linear, and let $B=\{e_1,\dots,e_n\}$ be a basis of $V$. If $T(e_i)$ is known for every basis vector $e_i$, then $T(v)$ is determined for every $v \in V$.

Proof. Every vector $v \in V$ has a unique expansion \[ v = x_1e_1 + \cdots + x_ne_n. \] Since $T$ is linear, \[ T(v) = T(x_1e_1 + \cdots + x_ne_n) = x_1T(e_1) + \cdots + x_nT(e_n). \] So once the values of $T$ on the basis vectors are known, the value of $T$ on every vector is forced. $\square$

This is the structural reason matrices exist.

What is a matrix?

Suppose:

(V) has basis (B = {e_1, \dots, e_n})
(W) has basis (C = {f_1, \dots, f_m})
(T : V \to W) is linear

To describe (T), look at where it sends each basis vector:

\[T(e_1), \dots, T(e_n)\]

Each of these lives in (W), so each one can be expressed using the basis (C).

For each (j):

\[T(e_j) = a_{1j}f_1 + a_{2j}f_2 + \cdots + a_{mj}f_m\]

The coefficients become the (j)-th column of the matrix:

\[A = \begin{bmatrix} a_{11} & \cdots & a_{1n} \\ \vdots & & \vdots \\ a_{m1} & \cdots & a_{mn} \end{bmatrix}\]

So the matrix is not the transformation itself.

It is the coordinate representation of the transformation, after choosing:

coordinates for the input space
coordinates for the output space

Definition 3 (matrix of a linear transformation). Let $T : V \to W$ be linear. Let $B=\{e_1,\dots,e_n\}$ be a basis of $V$, and let $C=\{f_1,\dots,f_m\}$ be a basis of $W$. The matrix of $T$ with respect to input basis $B$ and output basis $C$, written $[T]_{C \leftarrow B}$, is the matrix whose $j$-th column is $[T(e_j)]_C$.

Proposition 2 (the coordinate equation). For every $v \in V$, \[ [T(v)]_C = [T]_{C \leftarrow B}[v]_B. \]

Proof. Write \[ v = x_1e_1 + \cdots + x_ne_n. \] Then \[ T(v) = x_1T(e_1) + \cdots + x_nT(e_n). \] The $j$-th column of $[T]_{C \leftarrow B}$ is $[T(e_j)]_C$, so multiplying $[T]_{C \leftarrow B}$ by the coordinate column $[v]_B$ forms exactly the same linear combination in coordinates. Therefore \[ [T(v)]_C = [T]_{C \leftarrow B}[v]_B. \] $\square$

Equivalently, writing the coordinate maps explicitly, the matrix (A) satisfies:

\[\Phi_C(T(v)) = A \Phi_B(v)\]

or as a commuting diagram:

\[\begin{array}{ccc} V & \xrightarrow{T} & W \\ \downarrow \Phi_B & & \downarrow \Phi_C \\ \mathbb{F}^n & \xrightarrow{A} & \mathbb{F}^m \end{array}\]

This is the whole picture.

The top row is the real transformation.

The bottom row is what that transformation looks like in coordinates.

Matrix multiplication is function composition

Matrix multiplication also becomes less arbitrary from this view.

Suppose:

\[T : U \to V\]

and

\[S : V \to W\]

They can be composed:

\[S \circ T : U \to W\]

If (T) has matrix ([T]), and (S) has matrix ([S]), then the composition has matrix:

\[[S \circ T] = [S][T]\]

Proposition 3 (matrix multiplication is composition). Let $T : U \to V$ and $S : V \to W$ be linear maps. After choosing compatible bases for $U,V,W$, \[ [S \circ T] = [S][T]. \]

Proof. For any $u \in U$, Proposition 2 gives \[ [T(u)] = [T][u]. \] Applying $S$ next gives \[ [S(T(u))] = [S][T(u)] = [S][T][u]. \] But $S(T(u)) = (S \circ T)(u)$, so the matrix that represents $S \circ T$ is $[S][T]$. $\square$

This is not an arbitrary multiplication rule.

It is just the coordinate version of doing one function and then the next.

That is why the order can seem backwards at first:

\[[S][T]\]

means:

do (T)
then do (S)

Functions compose right-to-left, so matrices multiply right-to-left.

Why change of basis is annoying

Change of basis is another topic that becomes simpler once vectors are no longer treated as columns.

The same vector can have different coordinate columns under different bases.

The vector did not change.

The coordinate system changed.

Proposition 4 (change of basis changes descriptions, not vectors). If $B$ and $C$ are two bases for the same vector space $V$, then there is an invertible matrix $P_{C \leftarrow B}$ such that \[ [v]_C = P_{C \leftarrow B}[v]_B \] for every $v \in V$.

Proof. Each basis vector $e_j$ of $B$ can be written in the basis $C$. Put those coordinate columns into a matrix: \[ P_{C \leftarrow B} = \begin{bmatrix} [e_1]_C & \cdots & [e_n]_C \end{bmatrix}. \] If $v = x_1e_1 + \cdots + x_ne_n$, then multiplying by $P_{C \leftarrow B}$ forms \[ x_1[e_1]_C + \cdots + x_n[e_n]_C, \] which is exactly $[v]_C$. The matrix is invertible because changing from $B$-coordinates to $C$-coordinates can be undone by changing back. $\square$

So a change-of-basis matrix is not doing something to the vector itself. It is translating between descriptions of the same vector.

Likewise, two different matrices can represent the same linear transformation if they are written using different bases.

This is why statements like “this matrix has eigenvalues…” are a little imprecise. The main object is the linear operator, and the matrix is one coordinate representation of it.

Why diagonalization matters

Diagonalization is often taught as another algorithm:

find eigenvalues
find eigenvectors
build (PDP^{-1})

But conceptually, diagonalization means:

Find a basis where the transformation acts independently on each coordinate direction.

If an operator is diagonal in some basis, then in that basis it just scales each basis vector:

\[T(e_i) = \lambda_i e_i\]

Proposition 5 (diagonalization is choosing an eigenbasis). Let $T : V \to V$ be linear. If $B=\{e_1,\dots,e_n\}$ is a basis of eigenvectors with \[ T(e_i) = \lambda_i e_i, \] then the matrix $[T]_B$ is diagonal, with diagonal entries $\lambda_1,\dots,\lambda_n$.

Proof. The $i$-th column of $[T]_B$ is $[T(e_i)]_B$. Since $T(e_i)=\lambda_i e_i$, this coordinate column has $\lambda_i$ in the $i$-th slot and zeros everywhere else. So every off-diagonal entry is zero, and the diagonal entries are the eigenvalues. $\square$

That is why diagonal matrices are useful.

They are not useful because zeros are aesthetically pleasing. They are useful because the transformation has been expressed in coordinates where its behavior is decoupled.

The sentence to remember

Linear algebra is the study of linear transformations between vector spaces.

Matrices are coordinate descriptions of those transformations.

With that distinction in place, many standard facts become instances of the same idea:

matrix multiplication is composition
change of basis is changing coordinates
invertible matrices are isomorphisms in coordinates
the identity matrix is the identity transformation in coordinates
diagonalization is finding coordinates where an operator is simple

The useful diagnostic question is:

What transformation does this matrix describe, and which bases are involved?

That question usually identifies the relevant object and the relevant coordinates.

Understanding Transformers III - Cross Attention

2025-11-13T00:00:00+00:00

This post is a follow up to Part II on Encoders, you should start there if you are unfamiliar with encoders.

Recall that the Transformer architecture consists of three parts: the Decoder, the Encoder, and Cross Attention. In parts I and II we studied the decoder only transformer and the encoder only transformer, respectively. Now it is time to combine these things together to form the original encoder-decoder transformer architecture, using the Cross Attention mechanism.

Up to now, all mentions of “Attention” have referred to “Self Attention”, where a token pays attention to other tokens in the same sequence. Cross Attention is a mechanism where a token pays attention to tokens in a different sequence. In particular, Cross Attention is used to connect the Encoder and the Decoder.

Figure 1. The transformer architecture with labels for the encoder, decoder, and cross attention. (Choi, 2024)

How Cross Attention connects the Encoder and the Decoder

Something we should first clarify is how exactly the connection is made. The original diagram above is a bit confusing, but Murel and Noble (2024) provide a more clear explanation:

Figure 2. The cross attention mechanism when expanded fully (Murel & Noble, 2024).

In words, once the encoder has processed the input sentence, the cross attention mechanism feeds the final encoding to each decoder block of the decoding stack.

The Cross Attention Mechanism in Detail

The cross attention mechanism is nearly identical to the self attention mechanism, the main difference now is that in Cross Attention, we port the Key and Value matrices over from the Encoder and the decoder uses its own Query matrix on these encoder matrices. This is illustrated in the diagram below:

Figure 3. The cross attention mechanism in detail (Benveniste, 2024).

The math is entirely the same, the magic occurs just by the fact that we are reusing the Key and Value matrices from the Encoder and the Query matrix from the Decoder and performing this fancy rerouting.

But why?

Doing this allows each decoder block to pay attention to the final encoding of the input sentence. This is useful because it allows the decoder to use the information from the input sentence to help it generate the next token, and also prevents the decoder from “forgetting” the input sentence (vanishing gradient problem).

Conclusion

That’s it, that’s really all there is to the Encoder-Decoder Transformer.

References

Benveniste, D. (2024). What is the difference between self-attention and cross-attention? [LinkedIn post]. LinkedIn. https://www.linkedin.com/posts/damienbenveniste_what-is-the-difference-between-self-attention-activity-7211029906166624257-m0Wn/

Choi, J. (2024, March 2). Where the term “cross-attention” is first used? (couldn’t find the term in Attention is all you need paper) [Question on the Data Science Stack Exchange]. Data Science Stack Exchange. https://datascience.stackexchange.com/questions/128123/where-the-term-cross-attention-is-first-used-couldnt-find-the-term-in-attention-is-all-you-need-paper

Murel, J., & Noble, J. (2024). What is an encoder-decoder model? [Article]. IBM Think. https://www.ibm.com/think/topics/encoder-decoder-model

Why do Magnetic Field Lines go from South to North inside of a bar magnet?

2025-08-23T00:00:00+00:00

When reading about magnetic field lines for bar magnets, you will often hear that field lines go from north to south outside the bar magnet, and they go from south to north inside of the bar magnet, thus forming closed loops. The direction of the field line is interpreted to be the direction of force the bar magnet imposes on a point north pole particle. So it would make sense that field lines go from north to south outside of the magnet, but why does it make sense that they would go from south to north then when inside of a bar magnet? How can you ascertain what would happen to a north pole particle when you cannot physically measure this since you can’t place a particle there physically? In otherwords, if there’s no way to experimentally verify or dispute this, why do we say this?

Figure 1. Magnetic Field lines indicate which direction a north monopole will be pushed due to the magnetic field of the bar magnet (StickMan Physics).

Figure 2. You can empirically verify the existence and behavior of field lines by using iron filaments (E-TCOURSE). However, this only works for the outside of the bar magnet, as you obviously cannot place filaments inside of the bar magnet.

Figure 3. Magnetic Field lines are often shown going from north to south outside of the bar magnet, while south to north inside of the bar magnet. (Nave).

Such a statement has bothered me since my youth when it was first taught to me, and even now most sources don’t really do the topic justice, so I’ll try my best to break this down into intuitive terms.

First principles

To begin to understand this behavior, we need to recall some facts about bar magnets that we can verify.

Fact 1: A bar magnet cannot be split into two monopoles. In fact, when splitting a bar magnet, you will get two bar magnets with both north and south poles, regardless of whether you split it vertically or horizontally.

Note that this is equivalent to the fact that magnetic monopoles (i.e. magnets with only a north or south pole) don’t exist.

Figure 4. Splitting a North-South bar magnet just recursively produces smaller North-South bar magnets, you cannot separate the north from the south pole even if you try (Monash University).

Clearly, if you keep doing this recursion, you end up with a collection of ferromagnetic units, each of which of course has its own north and south poles.

Figure 5. A magnet consists of ferromagnetic sections. (Lehan, 2020)

Now, intuitively, if you place a north pole somewhere in this array of north-south dipoles, the force enacted upon the north pole will tend to move towards the top (north side) of the magnet, hence why the field lines point towards the north pole of the overall bar magnet.

There is really no outside vs inside, at all points in space there are many of these ferromagnetic particles acting on those points inside of the overall bar magnet. Basically, the north pole of the magnet is implied by the particles and not the otherway around - the particles imbue an overall sense of north vs southness of the bar magnet.

References

E-TCOURSE. (2020, November 26). An overview of magnetic field lines and its characteristics. E-TCOURSE News Blog. Retrieved August 23, 2025, from https://www.etcourse.com/news-blog/magnetic-field-lines

Lehan, D. (2020). Permanent and Induced Magnetism (4.7.1 Permanent and induced magnetism). In Physics-SchoolUK.com. Retrieved August 23, 2025, from https://physics-schooluk.com/magnetism_permanent.html

Monash University. (n.d.). Magnetic fields. In Student Academic Success – Physics: Fields and interactions. Retrieved August 23, 2025, from https://www.monash.edu/student-academic-success/physics/fields-and-interactions/magnetic-fields

Nave, C. R. (n.d.). Magnetism & electromagnetism. In HyperPhysics. Georgia State University. Retrieved August 23, 2025, from http://hyperphysics.phy-astr.gsu.edu/hbase/magnetic/elemag.html

StickMan Physics. (n.d.). Magnetic fields. In Magnets and magnetism [Web page]. Retrieved August 23, 2025, from https://stickmanphysics.com/stickman-physics-home/magnets-and-magnetism/magnetic-fields/

Understanding Transformers II - The Encoder

2025-07-13T00:00:00+00:00

This post is a follow up to Part I on Decoders, you should start there if you are unfamiliar with decoders.

Recall that the Transformer architecture consists of three parts: the Decoder, the Encoder, and Cross Attention.

An encoder-only transformer looks like this:

Figure 1. The encoder-only architecture [Adapted from Vaswani et al., 2017].

Encoder-only Transformers - BERT

Studying encoder-only transformers such as BERT (Bidirectional Encoder Representations from Transformers) (Devlin, 2019) allows us to understand encoders as a concept. Unlike decoder-only transformers which consume sentences and predict the next token, encoder-only transformers take sentences and encode them in a higher dimensional space. Encoders don’t do any prediction of next tokens or any meaningful task on their own. If we want them to do something like classification, we need to add task specific layers at the end of the model. For classification problems, this would be the classification head which would be some Multilayer Perceptron that consumes the encodings from the encoder blocks and outputs classification logits.

Figure 2. The BERT architecture with a classifier head (Khalid, 2021).

A key difference here between BERT and GPT-2 (decoder only) is that BERT consumes pairs of sentences. However this is just a choice of input, and is not necessitated by the architecture itself. A model like this can be broken down into simple steps, each corresponding to a part of the diagram above:

Receive an input sentence, consisting of two smaller sentences: “The cat is black. It licks its fur.”
Tokenization and embeddings: The process of tokenization and embedding the tokens is identical to that of decoder models. One additional step is we add a separator token (denoted SEP) between the two sentences.
Positional encodings: Similar to the decoder-only transformer, we have positional vectors that we add to each token embedding to encode information about its position in the sentence. We also have segment encodings to indicate whether the token is in the first or second sentence.

Figure 3. In BERT, we have both positional _and_ segment encodings (Devlin, 2019).
Attention: This corresponds to the “Multi-head attention” box in the transformer diagram. The process is nearly identical to the “Masked Multi-head attention” mechanism in the Decoder block so I will not repeat its details here. Recall that attention is called masked when we disallow tokens from paying attention to tokens ahead of it in the sentence, by zeroing out these values. So, the only difference in the encoder case is we allow for tokens to pay attention to all other tokens in the input sentence, since the attention is not masked here. We have an Add and Normalize step after this is done to avoid vanishing gradient problems.
Identically to the Decoder case, we have a Feed Forward block where the attention value vectors are passed through a Multilayer Perceptron (MLP) to introduce non linearity. Once again we then do an Add and Normalize step.
Steps 4-5 constitute one transformer block. The outputs are then fed to the next block. This process can be repeated as many times as desired. In BERT’s case, this is repeated 12 times.
At the output end of the last transformer block, we receive the encodings and can add any task head to fine tune the model to a specific task.

This completes the architecture.

References

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre‑training of deep bidirectional transformers for language understanding (arXiv:1810.04805). arXiv. https://doi.org/10.48550/arXiv.1810.04805

Khalid, U., Beg, M. O., & Arshad, M. U. (2021, February 22). RUBERT: A bilingual Roman Urdu BERT using cross‑lingual transfer learning (arXiv:2102.11278). arXiv. https://doi.org/10.48550/arXiv.2102.11278

Don’t Use Truthy/Falsy Values

2025-07-07T00:00:00+00:00

I’m not a fan of so called Truthy or Falsy values in programming languages for a variety of reasons. For those of you who are unfamiliar with these, truthy and falsy values are basically default boolean values associated with values. For example, in Python, a number implictly has boolean value True if and only if it is equal to 0. Similarly, an empty list [] has boolean value False but any list with any values, even if they are all 0, has boolean value True. So the lists [0], [1], etc. all have value True.

This means you can write nice little code like this:

def processList(inputList : List[int]) -> str:
    if not inputList:
        return "Nothing to do!"
    
    ...

Instead of:

def processList(inputList : List[int]) -> str:
    if len(inputList) == 0:
        return "Nothing to do!"
    
    ...

The key of course is just that writing !inputList is a lot more consise than writing len(inputList) == 0, when you want to check if the list is empty or not.

I’ll now go through a list of reasons colleagues have presented to me for why truthy falsy values are useful and evaluate their merits.

Various reasons for why Truthy Falsy checks are good

Avoids repetitive code for routine checks

In the example above, its useful to use the shorthand !inputList and anyone who is fluent in their language to take advantage of the option especially when the check comes up often. I agree with this point that if you’re going to do something a lot you should have nice syntax for it, and truthy/falsy values get you that.

If it is a feature of the language you should use it

I never understood this one but have heard it. It sort of assumes that every language is designed perfectly and has no pitfalls.

A common gotcha in python is that default arguments in function args are mutable and shared across all calls to the function.

def myFunc(inputList : List[int] = []) -> None:
    ...
    inputList.append(5) # each time myFunc is called, the inputList default mutates from [], [5], [5,5]

You are often recommended to this instead for safety:

def myFunc(inputList : List[int] = None) -> None:
    if not inputList:
        inputList = [] # always defaults to [] across all function calls
    ...

If you change the type of your variable then you don’t have to update any of your checks again

I’m not sure why you would want this, the example presented to me was that converting the following from a list input to an integer input

def processList(inputList : List[int]) -> str:
    if not inputList:
        return "Nothing to do!"

would allow us to seamlessly change the type without updating the code of the check

def processList(inputNum : int) -> str:
    if not inputNum:
        return "Nothing to do!"

This clearly depends on the fact that your desired check would have remained the same in the new datatype, but I can’t think of immediate cases as to why you would want this. Doesn’t it seem safer to have had the explicit check len(inputList) == 0 so that you immediately get compile/run time errors and you now have to make sure the checks are what you want in the new datatype?

There is then the added case of, if your variable name contains the type like in the example above, you still have to edit the line with the new variable name, so I don’t understand what time was saved.

Finally, I don’t think you change the type of a variable often anyways, so the whole case seems irrelevant.

Various reasons for why truthy/falsy checks are bad:

It violates “Write once, read many”

Using these checks kind of reminds me of a time when programmers around me would write alp everywhere instead of alpha just to save time. This leads to the classic case that anyone reading a line in the code needs to now think about what alp means, similar to how ayone reading !myList needs to think about the False/True cases of list values when len(myList) == 0 would have immediately told you the check condition.

No objective default across languages

The boolean values we associate with objects and primitives are inherently arbitrary. For example, in python the empty list [] has boolean value False, but in Javascript an empty array [] has boolean value true. Therefore, there is no intuitive way to know this knowledge, you just have to use it often and only in one language, and hope you don’t forget that the behavior is different when switching to another language.

No objective preference across languages

In python, it is often recommended to use these truthy/falsy checks just as it is tacitly the preferred style in the community, as part of writing “Pythonic” code.

In C++, the only truthy/falsy value provided by the language is that for numerical values i.e. int, float, double and their variants. This also applies to pointer values which are just size_t values. However we all know a good C++ programmer uses the nullptr keyword when checking if a pointer is null or not as follows:

size_t* ptr;
if (ptr == nullptr){
    std::cout << "Got a null pointer!" << std::endl;
}

even though you could do this:

size_t* ptr;
if (!ptr) {
    std::cout << "Got a null pointer!" << std::endl;
}

So we have an example where whether you should use it or not is subjective based on the community you are a part of.

Contradictory internal style guides

Python programmers love truthy/falsy values because it makes their code so-called “pythonic”.

I now refer you to the following lines from Zen of Python:

...
Explicit is better than implicit
...
Readability counts
...

If you read these sentences, you would probably realize that truthy/falsy values violate both - implicit boolean checks are inherently not explicit, and they are harder to read if you don’t know what the defaults are.

Confusing and inefficient when layered

In python, None evaluates to False in boolean contexts. This means that there are two cases when an Optional[int] evaluates to False: when it is 0, and when it is None.

You may forget this when writing code and do:

def myFunc(myVal : Optional[int] = None) -> None:
    if not myVal:
        print("No val set... performing handling...")
    else:
        if myVal == 0:
            print("Handling no quantity")
        ...

This is a bug. Remember - both the None and 0 values hit the if block, so you need to do this:

def myFunc(myVal : Optional[int] = None) -> None:
    if myVal is None:
        print("No val set... performing handling...")
    else:
        if myVal == 0:
            print("Handling no quantity")
        ...

In this case we weren’t able to use the truthy values at all. We could do !myVal in the else block if we think we’re smart, but that would require you to read all of the code around that block in order to realize that myVal couldn’t be a possible value in the else block. Doing the explicit check is much more readable.

Conclusion

In conclusion, just don’t…

Understanding Transformers I - The Decoder

2025-05-31T00:00:00+00:00

In this blog post I give an introduction to the Transformer architecture as originally presented by Vaswani et al., in their landmark paper, “Attention Is All You Need”. My focus will be on visuals and intuition. Specifically, I intend to explain the Transformer in three parts: Decoders, Encoders, and Cross Attention. This blog post covers part one of this list.

Figure 1. The transformer architecture features encoders, decoders, positional encodings, and token embeddings (Vaswani et al., 2017).

Breaking it down - Decoders, Encoders, and Cross Attention

For reasons, decoder-only architectures such as GPT-2 (released in 2019) and encoder-only architectures such as BERT (released 2018) actually came after the original encoder-decoder transformer paper in 2017. Despite this chronology it is useful to focus on these simpler transformers and build our way up in future sections to eventually combine the encoders and decoders together (via Cross Attention) to form the original transformer.

An easy way to visualize this is to divide the Transformer and relate it to these simpler transformers:

Figure 2. The transformer architecture consists of two parts - an encoder stack, and a decoder stack (Esmailbeigi, 2023).

That is - Decoder only architectures mimic the right side of the Transformer stack, while encoder only architectures mimic the left side.

There is one subtle detail - in decoder only models i.e. the right side, there is of course no cross attention mechanism that connects the encoders to the decoders, so the second multi-head attention unit should be removed:

Figure 3. The encoder decoder transformer features a Cross attention mechanism that connects these together. In a decoder only model this is not present (Tensorflow).

Therefore, a decoder-only transformer would look like this:

Figure 4. The decoder-only architecture [Adapted from Vaswani et al., 2017].

You may notice that this looks very similar to the encoder block, I will discuss the subtle differences in a separate post.

Decoder-only transformers - GPT-2

Let us now focus on the topic of this blog post which is Decoder-only architectures such as GPT-2. A model like this consumes a sentence and predicts the next token. In this sense, it can be understood that the Decoder architecture “decodes” what the next token should be, hence the name.

Some ideas below are inspired by the Transformer Explainer tool (Liang, D. & Klein, D.).

A model like this can be broken down into simple steps, each corresponding to a part of the diagram above:

Receive an input sentence: “Data visualization empowers users to”
Tokenization and embeddings: For each token in the sentence, convert it to an embedding vector of fixed length.
- A token is a string of continuous characters treated as a unit. For example, in GPT-2’s tokenizer, ‘data’, ‘visualization’, ‘users’ and ‘to’ are all tokens. However, ‘em’ and ‘powers’ are separate tokens.
- An embedding is a vector that corresponds to the token, and represents its meaning.
- The embedding vector represents the token’s meaning in the sense that vector operations can be performed to produce similar concepts:
  
  Figure 5. Subtracting the embedding vectors for man and women is equivalent to subtracting the embedding vectors for uncle and aunt (3Blue1Brown, 2024).
Positional encodings Special positional encoding vectors are added to each embedding to add meaning about the position of the token in the sentence. For example, there is a positional encoding vector for position 1, and it is added to the first token only.
Attention: This corresponds to the “Masked Multi-head attention” box in the transformer diagram, it works as follows: For each embedding $\vec{e}$ (corresponding to an input token), apply special transforms (understood to be weights matrices, but this isn’t too important) $W_K$, $W_Q$, and $W_V$ to get the Key, Query, and Value vectors corresponding to the token, respectively. These matrices are learnt during the training process of the transformer.
- Such vectors can be understood as follows:
  - The Key vector $\vec{k} = W_k \vec{e}$ is a vector of dimension $d_k$ such that when multiplied by another token’s Query vector, gives us a magnitude representing how much attention that other token is paying attention to us.
  - The Query vector $\vec{q} = W_q \vec{e}$ is symmetrically a vector of dimension $d_q = d_k$ such that, when multiplied by another token’s Key vector, gives us a magnitude representing how much attention our token is paying attention to that other token.
  - The Value vector $\vec{v} = W_v \vec{e}$ is a vector that represents the original embedding of the token in a new vector space, such that, when multiplied by attention (a scalar number), gives us a vector representing the original token scaled by the attention the current token has for it.
- To calculate the attention token $i$ has for token $j$, we could do something like this: $\vec{q_i}^\intercal \cdot \vec{k_j}$ thus getting a scalar number via the usual dot product. Then we could scale the value vector of token $j$, so that our scaled attention for token $j$ is just $\left(\vec{q_i}^\intercal \cdot \vec{k_j}\right) \vec{v_j}$.
- An astute reader would note that instead of doing each of these calculations as a for loop over all $i$ and $j$, we can get all of this together as one matrix mulitplication $Q^\intercal K V$, where $Q$, $K$, $V$ are matrices such that the ith column is the vector for the ith token.
- The variance of the multiplication of $Q$ and $K$ happens to be $\sqrt{d_k}$ (user3667125, 2020). Then if we divide by this we can reduce the variance of the multiplication to 1. And we can then apply a softmax so that the sum of attention scores a token has for other tokens adds up to 1. This would give us $\text{softmax}\left( \frac{Q^\intercal K}{\sqrt{d_k}} \right) V$.
- Therefore, for each input token’s embedding, we get a list of vectors that represent other vectors in the sentence, scaled by the attention this token has for them.
- This attention is called masked because we disallow tokens from paying attention to tokens ahead of it in the sentence, by zeroing out these values.
- This attention is called multi-head because technically, we run 12 of these attention heads in parallel, each doing the above as described, but for smaller sections of the input embeddings. For example, one attention head would look at the first 12th of each embedding. The second attention head would look at the second 12th of each embedding, etc. And then we just concatenate these results together to produce the same output had we done it all in one single head of attention.
Once we have these scaled attention-value vectors, we run these through a Multilayer Perceptron (MLP). This corresponds to the “Feed Forward” box in the transformer diagram (note: MLPs are only one type of Feed Forward network, but not all Feed Forward networks are MLPs). This adds many layers of non linearity that allows the transformer to learn deep features about these vectors.
Steps 4-5 constitute one transformer block. The outputs are then fed to the next block. This process can be repeated as many times as desired. In GPT-2’s case, this is repeated 12 times.
At the output end of the last transformer block, we apply a softmax layer to convert the outputs to probabilities. This represents the probability of a particular token being the next token in an input sentence.

This completes the architecture. There are a couple of details I left out above because I felt they would disrupt the flow and are of minor importance. They are as follows:

Add and Norm: There are several layers of Add and Normalization, how this works is we take the input before MLP or attention respectively and add it to the output, and then normalize the values to some range ex. between 0 and 1. The point here is to avoid vanishing gradient and exploding gradient problems by feeding the original inputs through the model and then ensuring output numbers don’t get too large.

References

3Blue1Brown. (2024, May 27). How word vectors encode meaning [Video]. YouTube. https://www.youtube.com/watch?v=FJtFZwbvkI4

Esmailbeigi, R. (2023, January 20). BERT, GPT, and BART: A short comparison. Medium. https://medium.com/@reyhaneh.esmailbeigi/bert-gpt-and-bart-a-short-comparison-5d6a57175fca

Liang, D., & Klein, D. (n.d.). Transformer explainer. Polo Club of Data Science, Georgia Institute of Technology. https://poloclub.github.io/transformer-explainer/

TensorFlow. (n.d.). Text generation with a transformer. TensorFlow. Retrieved June 1, 2025, from https://www.tensorflow.org/text/tutorials/transformer#the_cross_attention_layer

user3667125. (2020, August 16). Answer to “Why does this multiplication of Q and K have a variance of d_k, in scaled dot product attention?” Artificial Intelligence Stack Exchange. https://ai.stackexchange.com/a/25057

Vaswani, A., Shazeer, N., Parmar, N., Uszoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (pp. 5998–6008). https://arxiv.org/abs/1706.03762

Creating Meaningful Evaluations for Agentic App Builders

2025-05-17T00:00:00+00:00

For the last two months at Create, my main focus has been building evaluations for our AI agent – the thing that takes a user’s natural language prompt and turns it into a working application. This is a hard problem, and the space is largely unresearched, so I want to share what we’ve learned so far.

What are evaluations?

Evaluations are automated setups where we give the AI agent a set of tasks (an array of prompts that a user would have typed in manually), and then score how well the agent performs on those tasks. Scores range from 0% to 100%, and there can be multiple scores per run. The key property of a meaningful score is that it correlates with quality: a higher score than before implies improvement, a lower score implies regression.

In simpler terms, it is a way to give the AI agent grades on various tasks. If the grade changes day-over-day after our engineers ship code changes, it tells us whether the product experience got better or worse.

Why this is hard

Traditional software tests assert exact outputs. If add(2, 3) returns 5, it passes. Agentic systems don’t work this way. The prompt “build a mentoring marketplace with Stripe payments” has no single correct answer, and the agent could produce many valid implementations that all look completely different.

We tried the common approaches, and each has limitations.

LLM-as-a-judge: pass the output of the agent to another LLM and ask it to score the result. The problem is that LLMs hallucinate success and miss functionality bugs when reading codebases. They have no way to actually run the code or test the web app.

Computer Use Agents (CUA) as judge: a computer use agent like OpenAI’s Operator uses a physical browser to interact with the generated website and judge it. This is better because it actually tests the app, but accuracy is still not where it needs to be. Anthropic’s CUA performs at around 90% accuracy on our text-to-app evaluation set, while OpenAI’s Operator is at around 70%. These error rates get embedded into the provided scores.

String or DOM comparisons break whenever the agent chooses a different but valid layout.

Manual QA doesn’t scale.

What we built

We built a daily evaluation framework that simulates user sessions, scores the results, and gives engineers a signal on whether the agent is getting better or worse. It has three main components.

Prompt sets are sets of prompts drawn from real user sessions, edge cases, and synthetic flows. We have two main categories:

Text-to-app one shot: Given a single prompt, how well can the agent build the app in one go?
Long conversation: Given a long array of prompts in series, how well can the agent implement all of them without introducing regressions?

Each set holds roughly 1 to 50 high quality examples.

Scorers calculate the scores. We use three types:

LLM judges for relevance and coherence.
Computer use agents that launch the generated app in a real browser, click through flows, and observe state.
Humans-in-the-loop for side-by-side comparisons on selected failures.

The human-in-the-loop component is important. Because no automated method is 100% accurate, gold standard evaluation sets must be feasible for human experts to evaluate. There must be a small enough number that humans can review day-over-day, and they need to provide meaningful signal on each run.

Daily tracking: Scores, logs, and qualitative notes go into Braintrust. Engineers can review regressions in minutes. If a change lands, we see the effect the same day.

Scoring vs meaningfulness

Balancing accurate scoring with meaningfulness is difficult.

You can build automatable evaluations by forcing the agent to produce specific, testable outputs – for example, requiring hardcoded data-testid attributes on page elements so you can write deterministic assertions. These are stable and can run without human review.

But such evaluations don’t represent what a real human would type. No real user prompts “build me a landing page and make sure the hero section has data-testid='hero-section'.” Evaluations like these don’t tell you how the product experience is changing.

We went with evaluations that represent real human use, even though it means we need humans in the loop and can’t fully automate scoring. Selecting the right set of evaluation examples requires a lot of experimentation.

What we’ve learned

Evals have been useful for identifying issues with the agent. By their nature, they are a small set of high quality examples that provide a lot of information on each run. The small size allows for humans to build intuition around the behavior of the agent (which can be noisy due to the fundamental nature of LLMs), and the fact that the evals run repeatedly allows engineers to identify persistent issues as well as flaky ones. Some concrete results:

We caught regressions before they hit users.
We flagged prompt templates that degraded output quality.
We stretched one long conversation evaluation from 8 to 165+ turns without errors, by iterating on the eval and using it to drive improvements.

Where this fits in the bigger picture

Create’s platform produces full applications for non-technical users from natural language. That includes auth, payments, databases, deployment, integrations, etc. The agent has to be reliable, and reliability requires metrics and feedback loops.

Evaluations provide information about the current state of the platform independent of production data. This allows us to see how configuration changes to things like models or tools will affect performance.

However, these evaluations are still limited in scope. The ultimate test is how the agent performs across real production sessions.

Industry context

Academic benchmarks like SWE-Bench, WebArena, and AppBench focus on code reasoning or single-turn tasks. They don’t cover the full workflow of generating a working app from a prompt, which is what we need to evaluate. As far as I can tell, there isn’t much existing work on evaluations for text-to-app agents specifically.

Conclusion

Evaluations are a constant work in progress. The question “how well does a generated website represent a prompt?” is still open-ended, and there is no state of the art way to answer it fully automatically. But even imperfect evaluations have already helped us catch real issues, and I expect them to keep being useful as the agent gets more capable.