Row- and column-major ordering (tensor vectorization)

There is no standard for how to order the elements in a tensor during vectorization. There, is, however, a standard nomenclature: you can use a row-major or column-major order.

Let’s first understand these concepts for matrices, then generalize them to higher-order tensors. So consider a matrix M

There are two ways to imagine as a multidimensional array: it could be an array of rows

or of columns

where refers to expansion of the vector along the indicated axis, i.e.

We say that is the row-major representation of , and is the column-major representation. The vectorizations and of these respective matrices is the expansion, in order, of their member vectors, i.e.,

Extension to higher-rank tensors

The key to extending this reasoning to higher-rank tensors is in which index advances more slowly:

• Row-major ordering advances the left (row) index more slowly.
• Column-major ordering advances the right (column) index more slowly.

Generalizing this to higher dimensions, we can say that:

• Row-major ordering advances indices from right-to-left; and
• Column-major ordering advances indices from left-to-right.

For example, consider a rank-3 tensor . We would conventionally write this as

The above representation can be seen as a (row) vector of matrices

But we could also think of as a (column) vector of matrices

which we could expand (somewhat unnaturally) as

It should be noted that the first representation, which is analogous to row-major ordering, feels much more natural. Still, the choice of row- or column-major order is application specific, so we must not make assumptions.

Reading row-wise and column-wise, we see that the row and column orderings and of T would be

The same logic could be extended to arbitrarily high dimension.