You may have heard the term "Vector Search" before. Do not worry, though; it is not "Vector Holmes" investigating a crime from 221B Baker Street.
So, let’s explain, step by step, how to tackle vector data searches.
:::pagebreak:::
What Are Vectors?
In mathematics, a vector is an arrow that indicates a quantity with three elements: its value (magnitude), its direction, and its course. It is used to represent things that cannot be described with a single number, such as velocity, force, or displacement.
Magnitude: It is the length of the arrow, which indicates the numerical value of the magnitude.
Direction: It is the straight line on which the vector acts (it can be horizontal, vertical, or diagonal).
Course: Indicates where the arrow points (it is the endpoint of the direction).
Point of application: It is the starting point of the vector.
Two vectors are equal when they have the same magnitude (modulus), direction, and course. It means that regardless of their position in space, if their components are identical, they are the same vector.
Two vectors are opposite when they have the same magnitude and the same direction, but opposite course, meaning they are facing away from each other.
In a two-dimensional system, a vector is represented by its start (origin) and end (destination) coordinates on the X and Y axes.
These two vectors are represented as follows:
{(1,1),(5,3)}
{(2,4),(7,5)}
In a three-dimensional space, coordinates include the X, Y, and Z axes.
What Do Mathematical Vectors Have to Do with Vector Search?
Vector search is a method of searching known in numerical representations as vectors. It is similar to the vectors we have seen previously, but with the origin point being point 0 in Euclidean space.
Instead of looking for precise keyword matches, this technique examines similarities between vectors, as previously explained, enabling more accurate, semantically and contextually meaningful results, even if the words in the query are not exactly the same.
For example, "Iron Maiden," "Van Halen," and "Scorpions" are all the names of heavy metal bands. If we convert these names into tokens to create vectors, their coordinates will be similar but not identical. There will be slight differences in magnitude and direction. Therefore, they will be “orbiting” in the same “band” zone.
Conversely, "Bob Dylan" or "Genesis" will point in the opposite direction. They are still musical groups, but their styles do not have "the same direction".[
If we perform a traditional search, we must use the exact same word, which is"Music group". It will encompass all groups. However, if we want to look for a specific genre, for instance, "thrash rock", we will not know how to differentiate Metallica from Dire Straits.
In contrast, if we perform a vector search, it would locate these groups based on their proximity in both genre and musical style, showing that Metallica and Slayer are similar.
Are Vectors Really “Arrows”?
In computer science, a vector is a data structure that includes an array of numbers. In our case, these vectors store a digital summary of the dataset to which they have been applied. Consider it a summary or a digital fingerprint.
Images can also be analyzed for similarities. If you had to develop an application to compare two images, how would you do it? If you simply compared every pixel of one image with every pixel of the other, you would only find images that are identical in resolution, color, encoding, and other aspects.
However, if you could examine the images and generate vector embeddings of the content, you would be able to compare them and identify similarities. In the case of images, a vector embedding explains the content of each image and then allows for comparison.
This constitutes a much more robust method for discovering similarities between images.
A vector, speaking in computer science terms, would have a representation similar to the one below
{10,-4,6,34,0,-35,67,203,466,4,356,3,-53,-3,0}
It will be like indicating the vector's final position only, since the vector's origin is point 0.
Vector Search Algorithms
The data, which includes images, texts, audio, etc., is encoded in numerical vectors called "Embeddings". These values have varying dimensions depending on how they are encoded. The more dimensions our vector has, the more specific its definition will be, and the better it can be grouped.
Vector Similarity Searches:
We can use different types of metrics in vector similarity searches, each with its own advantages and disadvantages. The appropriate metric will depend on the data type and the application.
Euclidean Distance:
To determine if two vectors are close, we should measure the “Euclidean distance”, which is the straight-line length between two points in Euclidean space. It equals the square root of the sum of the squared differences between the point coordinates:
In an n-dimensional space, it is measured as the difference between each of the dimensions squared.
Since the value is smaller, we can deduce that the final vector points are closer to each other than the initial ones.
If we compare it with other vectors in the same space, those with the smallest values will be approximate vectors.
The distance between v1 and v3 is less than the distance between v1 and v2.
It is important to know how to calculate the distance between two points because it will be used later for measuring the magnitude of a vector.
Magnitude of a Vector
It is the distance from the vector base to its endpoint. In our case, all vectors originate at point 0, so the magnitude of a vector would be calculated using the following formula:
The data above gives us the following magnitudes of the vectors:
V1 = 5,83095189
V2 = 10
V3 = 5,6568542
Dot Product
The dot product (also called the scalar product) is an operation between two vectors that results in a real number (a scalar). It is one of the most important tools in mathematics and physics because it tells us how "aligned" two vectors are.
It is calculated by multiplying the sum of the coordinates in each dimension of the vectors to be compared.
The information from our example gives us the dot product values as follows:
Dot product v1v2 = (3 * 6) + (5 * 8) = 58
Dot product v2v3 = (6 * 4) + (8 * 4) = 56
Dot product v1v3 = (3 * 4) + (5 * 4) = 32
The larger the dot product, the greater the similarity between these two vectors.
We can say that v1 and v2 are very similar.
Cosine Similarity
This search method allows us to look for vectors not by the distance between their points, but by the angle they form.
The cosine similarity measures the orientation between two vectors in a multidimensional space, ignoring their magnitudes. It is calculated by dividing the dot product of two vectors (A) and (B) by the product of their magnitudes, resulting in a value between -1 and 1.
It is calculated using the following formula:
Following the previous example,
Similarity of the cosine of v1 and v2:
Point product v1 y v2 = (3 * 6) + (5 *8) = 18 + 40 = 58
Magnitude v1 = 5,83095189
Magnitude v2 = 10
Cosine(v1, v2) = 58 / 58,309 = 0,9947
Similarity of the cosine of v2 and v3
Point product v2 y v3 = (6 * 4) + (8 * 4) = 56
Magnitude v2 = 10
Magnitude v3 = 5,6568542
Cosine(v2, v3) = 58 / 58,309 = 0,9899
Similarity of the cosine of v1 y v3
Point product v1 y v3 = (3 * 4) + (5 * 4) = 32
Magnitude v1 = 5,83095189
Magnitude v3 = 5,6568542
Cosine(v1, v3) = 32 / 32,984 = 0,9701
We can confirm that v1 and v2 are more similar than v2 and v3, and v1 and v3.
Note: In the case of multidimensional vectors, we should perform the sum of each of the dimensions.
This approach returns vectors with similar directions, so it can help us find resemblances to make recommendations (this is how Netflix or HBO works).
In the example of music groups, we can start by searching for "thrash rock" bands. Then we can suggest other groups not categorized as thrash rock but within the heavy metal genre. That is, we start by listening to Metallica and, through suggestions, move on to Van Halen. Now you understand why Netflix might suggest watching "The Nightmare Before Christmas" after seeing "Hotel Transylvania”.
As Foghorn Leghorn once said, “It’s math, son!! The numbers don’t lie.”
How Do We Use This Mathematical Knowledge In Vector Search?
First, we need to convert our data (text, images, etc.) into a vector, as we have indicated before.
However, vectors do not create themselves. That means it is not enough just to give them a random value defined by us.
LLM models allow us to create the dot product and use cosine similarity to perform searches within InterSystems IRIS. Euclidean search is not implemented, but could be applied with Python libraries.
Some people have already thought about this because it will allow us to associate correct vectors based on the text context… Yet, I will explain this in the next article.
Note: I am sorry for the math lesson. Yet, I had to explain how it works first to help you better understand the "search magic".
