Scalars, Vectors, Matrices, and Tensors
- (The University of Chicago - Alvin Wei-Cheng Wong)
- Overview
Scalars, vectors, matrices, and tensors are the fundamental data structures of linear algebra. They scale in dimensionality from zero to multiple dimensions, serving as the building blocks for storing numbers, performing spatial transformations, and driving machine learning (ML) computations.
1. Scalars (0th-Order Tensors):
- Definition: A single numerical value representing magnitude only.
- Role: Acts as a baseline unit or scaling factor to stretch or shrink vectors and matrices.
- Examples: Mass, temperature, learning rate, or a single weight coefficient.
2. Vectors (1st-Order Tensors):
- Definition: An ordered array of numbers organized in a single dimension.
- Role: Represents points or directions in a coordinate space, defining directional movement, features, and embeddings.
- Examples: Spatial coordinates (x, y, z), feature rows, or word embeddings.
3. Matrices (2nd-Order Tensors):
- Definition: A rectangular grid or two-dimensional array of numbers arranged in rows and columns.
- Role: Represents linear transformations (linear maps) that rotate, scale, or shear vectors, as well as storing tabular data or network layer weights.
- Examples: Transformation operators, grayscale images, and neural network weight matrices.
4. Tensors (N-Dimensional Arrays):
- Definition: Tensors are multi-dimensional arrays that extend data past rows and columns into three or more axes. They serve as the core data structure in modern machine learning (ML) and deep learning (DL) frameworks.
- Role: Encapsulates complex, multi-faceted data configurations where information relies on three or more indexing coordinates.
- Examples: Color images (eight x width x color channels) or batches of video frames over time.
Please refer to the following for more details:
- Wikipedia: Outline of linear algebra
- Wikipedia: Scalar.
- Wikipedia: Vector Space.
- Wikipedia: Matrices.
- Wikipedia: Tensor.
- Core Data Structures and Roles in Machine Learning
Linear algebra uses scalars, vectors, matrices, and tensors as data structures to power computer graphics, cryptography, and machine learning (ML). Scalars are zero-dimensional numbers, vectors are one-dimensional lists, matrices are two-dimensional grids, and tensors extend into multi-dimensional arrays for complex data.
Broadly speaking, in linear algebra data is represented in the form of linear equations. These linear equations are in turn represented in the form of matrices and vectors.
Tensors are often used to represent multi-dimensional data, such as color images, volumetric data, or time series data. Scalars, vectors, and matrices are used to represent inputs like text and pictures, which allows ML or DL models to be trained and deployed.
Linear algebra is the backbone of many areas of computing, allowing us to efficiently model and solve a wide variety of problems. From image processing and machine learning (ML) to computer graphics and cryptography, the principles of linear algebra are used in countless areas of computer science.
In machine learning (ML), a vector is an element of a vector space, which is a geometric object collection with an addition rule and a scalar multiplication rule. A vector looks like a directed line segment, though not all vectors are directed line segments.
Scalars, vectors, and matrices are fundamental structures of linear algebra that are important for understanding deep learning (DL). Tensors are a generalization of matrices and can have any number of dimensions. They are often used to represent multi-dimensional data, such as color images, volumetric data, or time series data.
1. Core Data Structures:
- Scalars: Single numbers with zero dimensions.
- Vectors: One-dimensional arrays representing features or directions.
- Matrices: Two-dimensional grids representing tables or transformations.
- Tensors: Multi-dimensional arrays generalizing matrices for rich data.
2. Roles in Machine Learning (ML):
- Inputs: Translate text, images, and audio into numbers.
- Weights: Store parameters inside deep learning neural networks.
- Calculations: Speed up model training via matrix multiplication.
- Computational Linear Algebra: Scalars, Vectors, Matrices and Tensors
In machine learning (ML), scalars, vectors, matrices, and tensors are fundamental data structures built upon linear algebra. Scalars are single numerical values (0-dimensional), vectors are ordered lists of numbers (1-dimensional), matrices are two-dimensional arrays, and tensors are generalizations of matrices to any number of dimensions.
These structures are essential for representing and manipulating various types of data, including text, images, and time series, within ML models.
- Scalars: Represent single numerical values (e.g., temperature, price) and are considered 0-dimensional.
- Vectors: Represent ordered lists of numbers (e.g., coordinates in a space) and are 1-dimensional.
- Matrices: Represent 2D arrays of numbers (e.g., tabular data, images).
- Tensors: Generalize matrices to any number of dimensions, allowing for the representation of complex multi-dimensional data like color images, volumetric data, or time series.
- Linear Equations: In linear algebra, data is often represented as linear equations, which can be expressed using matrices and vectors.
- Deep Learning (DL): Understanding these linear algebra concepts is crucial for grasping the fundamentals of deep learning (DL), as these structures are used to represent inputs and manipulate data within neural networks.
- [Alpacas, Perù]
- Arrays and Linear Algebra
Arrays serve as the building blocks for numerical linear algebra, enabling the representation and manipulation of mathematical structures and the execution of key operations for solving real-world problems.
Arrays are multi-dimensional data structures used to represent mathematical objects like 1D vectors, 2D matrices, and multi-dimensional tensors. They enable numerical implementations of linear algebra, including solving linear equations, calculating eigenvalues and eigenvectors, performing singular-value decomposition, and executing matrix factorizations.
1. Representing Structures:
- Vectors: 1D arrays store vector data where length equals the dimension.
- Matrices: 2D arrays organize elements in rows and columns.
- Tensors: Multi-dimensional arrays handle data in higher dimensions.
2. Numerical Operations:
- Linear equations: Augmented arrays use Gaussian elimination to find solutions.
- Eigenvalues: Arrays form characteristic equations to find values and vectors.
- Decompositions: Factorization techniques break down matrices using array operations.
- Linear Equations
A linear equation is a math statement with an equal sign where the variables have a power of 1. When you graph them on a grid, they make a straight line. Common examples include 2x + 3 = 7 (one variable) and y = 2x + 1 (two variables).
In linear algebra, a linear equation is an algebraic equation where the highest power of the variable is always 1. It's also known as a one-degree equation.
The standard form of a linear equation in one variable is Ax + B = 0, where x is a variable, A is a coefficient, and B is a constant. The standard form for linear equations in two variables is Ax+By=C. For example, 2x+3y=5 is a linear equation in standard form.
When graphed, a linear equation in one or two variables always represents a straight line. For example, x + 2 y = 4 is a linear equation and the graph of this linear equation is a straight line.
[More to come ...]
