After an extensive math lesson, we are going to put our new knowledge into practice and create one of the best context-related databases.

Get your magnifying glasses and hats ready, because Vector Holmes is back.

Vector Calculation

To calculate the vector associated with a text, image, or sound, we will use a Python library called sentence-transformers, which allows us to transform content into a vector.
To implement this, we should create the following function:

ClassMethod Embedding(Text) [ Language = python ]
{
    from sentence_transformers import SentenceTransformer

    model_name = 'sentence-transformers/all-MiniLM-L6-v2'

    # Cache in global variables of the embedded Python module (persistent per process).
    global _cached_embedding_model
    if '_cached_embedding_model' not in globals():
        _cached_embedding_model = SentenceTransformer(model_name)

    vector = _cached_embedding_model.encode(
        [Text],
        normalize_embeddings=True,
        convert_to_numpy=True,
        show_progress_bar=False
    )

    return str(vector[0].tolist())
}

This method "vectorizes" the content of our text using a pre-trained library called "all-MiniLM-L6-v2". However, if you wish, you can modify it and utilize your own trained library.

First Steps

Note: To access the terminal of our Docker instance, use the following command:
Docker-compose exec iris iris session iris

Let's use an example of a vectorized search. For this, we will utilize a table called St.vectorsearch.Feeling that has the following fields:

You can create the data by running the following command:

Do ##class(St.vectorsearch.Data).Init()

Next, we are going to load the sentiment data from the /opt/irisbuild/data/training.csv directory, using the Populate command from the St.vectorsearch.Data class.

USER>do ##class(St.vectorsearch.Data).Populate()
Truncating table St_vectorsearch.Feeling
Preparing to load data from file training.csv
Loading data from file training.csv
Total records loaded: 2000
Calculating vectors for records...
Processing 496 of 2000

Be patient, since it will create a vector for each record, and it might take quite a while.

Note: Populate will initialize a data model of 2000 records. There is another file with 5410 records if you want more information in the data model. If you wish to work with that one, use the PopulateFull() command instead.

The values of feelings are as follows:

For example:

“I left with my bouquet of red and yellow tulips under my arm, feeling slightly more optimistic than when I arrived.” It has a value of 1 (Joy).

“I can’t walk into a shop anywhere where I do not feel comfortable.” It has a value of 4 (Fear).

If we use this function to create vectors, the first text we pass will return the next vector:

.034666668623685836791, .012147962115705013276, .020678628236055374146, 
.043785430490970611572, .030321707949042320251, -.0081106657162308692932, 
-.028869708999991416931, -.059950094670057296752, .067945346236228942871, 
-.070874534547328948974, -.060159780085086822509, .016239311546087265014, 
-.034665744751691818237, -.011642633005976676941, .080176420509815216064, 
………
.0099751437082886695861, .0098187308758497238159, .00016082286310847848653, .013545278459787368774, -.0049553057178854942321, .054155148565769195556, .025806473568081855773, -.038503900170326232911, .039657127112150192261, 
-.073851920664310455322, -.070615962147712707519, .066068030893802642822, 
-.082378372550010681152, -.043505564332008361816, -.0054294602014124393463

Note: The vector value has been reduced to keep the text from being too long; the vector actually has 384 values.

If we look for another cheerful text when creating the vector, it will give us the following value.

Example:

“Today, I’m very happy.”

[-0.007015716750174761, 0.05171322077512741, 0.0045058708637952805, -0.04008815810084343, 0.009171668440103531, -0.04867621883749962, 0.08268272876739502, 0.004975424613803625, -0.062008269131183624, 
……
0.036987580358982086, 0.027985544875264168, 0.05382953956723213, 0.031146947294473648, -0.011256292462348938, 0.005441852379590273]

This data does not make sense right now because we are not going to make a value-by-value comparison. That means we are not going to look for a record with exactly the same value as the vector because each vector is different. For that, we will use the formulas explained in the first part.

We are going to use the following SQL commands to see which vectors are the closest to the vector we have calculated.

Dot Product Search

As we indicated in the first part, the dot product search tells us how aligned two different vectors are. That means the closer it is to 1, the closer, or rather more similar, two vectors are.

To perform our search, we will use the condition “VECTOR_DOT_PRODUCT” that compares two vectors to determine their alignment:

SELECT TOP 5 Text, Value
 FROM St_vectorsearch.Feeling
ORDER BY VECTOR_DOT_PRODUCT(Vector, TO_VECTOR('[-0.007015716750174761, 0.05171322077512741, 0.0045058708637952805, -0.04008815810084343, 0.009171668440103531, -0.04867621883749962, 
......
-0.036987580358982086, 0.027985544875264168, 0.05382953956723213, 0.031146947294473648, -0.011256292462348938, 0.005441852379590273]', float)) DESC

This search gives us the following results:

So… “Today, I’m very happy” returns most results with a value of 1, joy.

But… what is the percentage of proximity our text has regarding the rest of the retrieved values?

We can find that out by comparing it with the current vector and displaying the value in a percentage format.

SELECT TOP 5 Text, Value, TO_CHAR(VECTOR_DOT_PRODUCT(Vector, TO_VECTOR('[-0.007015716750174761, 0.05171322077512741, 0.0045058708637952805, -0.04008815810084343, 0.009171668440103531, -0.04867621883749962, 
......
 -0.036987580358982086, 0.027985544875264168, 0.05382953956723213, 0.031146947294473648, -0.011256292462348938, 0.005441852379590273]', float)) * 100 , '990.99%') AS Percentage
 FROM St_vectorsearch.Feeling
ORDER BY VECTOR_DOT_PRODUCT(Vector, TO_VECTOR('[-0.007015716750174761, 0.05171322077512741, 0.0045058708637952805, -0.04008815810084343, 0.009171668440103531, -0.04867621883749962, 
......
-0.036987580358982086, 0.027985544875264168, 0.05382953956723213, 0.031146947294473648, -0.011256292462348938, 0.005441852379590273]', float)) DESC

This search gives us the following results:

The percentage of similarity to the most matching text is 65%. Our mind knows that, among all the texts, the meaning of “Today, I’m very happy” is a feeling of joy, meaning it should be closer to 100% comparing to the text “I am feeling so happy”

Cosine Similarity Search

The cosine similarity search is performed using the cosine of the angle between the vectors being compared.

If we want to perform vector search with cosine similarity, we would have to utilize the condition “VECTOR_COSINE”, just as we have previously done it:

SELECT TOP 5 Text, Value, TO_CHAR(VECTOR_COSINE(Vector, TO_VECTOR('[-0.007015716750174761, 0.05171322077512741, 0.0045058708637952805, -0.04008815810084343, 0.009171668440103531, -0.04867621883749962, 
......
 -0.036987580358982086, 0.027985544875264168, 0.05382953956723213, 0.031146947294473648, -0.011256292462348938, 0.005441852379590273]', float)) * 100 , '990.99%') AS Percentage
 FROM St_vectorsearch.Feeling
ORDER BY VECTOR_COSINE(Vector, TO_VECTOR('[-0.007015716750174761, 0.05171322077512741, 0.0045058708637952805, -0.04008815810084343, 0.009171668440103531, -0.04867621883749962, 
......
-0.036987580358982086, 0.027985544875264168, 0.05382953956723213, 0.031146947294473648, -0.011256292462348938, 0.005441852379590273]', float)) DESC

Oh… surprise!! It gave us back the same results:

As mentioned in the first part, this type of search is more advisable for comparing more than one parameter, such as movie genre, lead actor, etc. In other words, this is how the movie recommendation system of Netflix or HBO works.

Does it Support Multiple Languages?

What will happen if, instead of using the text “Today, I’m very happy,” we do it in Spanish?

“Hoy, estoy muy feliz.”

The created vector (reducing the text for obvious reasons) will resemble the following:

[-0.020053215324878693, 0.08766797184944153, 0.04118209332227707, 0.027171766385436058, -0.026769554242491722, -0.045022152364254,
…..
 0.038541924208402634, 0.018977241590619087, 0.03448185324668884, 0.10012426972389221, 0.08426760882139206, -0.09713190793991089]

And if we use it to perform the search as we did previously, we will get the result below:

We got a combination of love (2), fear (4), and joy (1), when the meaning is identical to the English text.

This rather diverse result appeared because the word "feliz" is closer to "feel" than "happy".

As you have noticed, the creation of vectors is closer to the phonetics of the texts than to their meanings.

I recommend testing the same text in different languages.

Here you have the result of searching for it in French “Aujourd'hui, je suis très heureux”:

Therefore, if we want to use this system to search for text in a document that we have fully "tokenized", it can be solved in a very simple way.

In a nutshell, we are going to change the model we used to create the token to the model paraphrase-multilingual-MiniLM-L12-v2:

model_name = 'sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2'

Therefore, if we modify this line in the Embedding method, we can use multilingual support in data retrieval.

This is the result of using this model with the search in Spanish:

Now, if you create a language-based token, why does it internally "translate" "Feliz" to "Happy"? Yes, I have used double quotation marks around "translate" because it is not a context-based translation like in any other online dictionary. It interprets the word by creating a token, which can cause some confusion when similar words are used.

For example, in Spanish, the word "tiempo" is the same word for the meaning of the word "time" or "weather", so the token for this word could be confusing. Fortunately, the token is not based on a single word, but on a set of words that make up the phrase. It means that the rest of the phrase's context helps us create a token much closer to the final one. Therefore, our vector seems to have translated the text we want to search for.

Case Study

Very important note: I encountered some difficulties developing these practical examples. When trying to create a vector from an IRIS production environment, the process would completely freeze.

The solution to using the St.Vectorsearch.Vector.Embedding method without any issues was to download the model locally instead of querying the online model for each request. When using the model locally, invoking this class from production did not require an Internet connection to download the model, making the vector creation process much faster.

I have created a small demo to show how a text, entered from a form, is "tokenized" and searched in the database to find the five closest results.

http://localhost:52773/csp/user/feeling.html

Note: Make sure production is running. To do this, log in to the Management Portal and start production if it was stopped.

The first time you ask about a feeling, it may take a little while, because it caches the model for future queries.

This website invokes an API running on IRIS that converts the text into a vector. Then it invokes the SQL search as I mentioned previously.

It performs a search for the 5 closest phrases to the calculated vector. Then the website displays those results and indicates which sentiment is repeated more often, showing the arithmetic mean of the most repeated results.

Can We Have More Than One Element to Create a Vector?

In the sentiment example, the phrase included information we used to create a search vector. However, what do we need if we have more than one column to perform the search?

If we want someone to recommend a similar movie to the one we have already seen, the result will depend on many factors. It could be related to the movie release year (because you like older films) or the genre (comedy, drama, science fiction), etc.

In this case, we will create a vector based on the text that includes the fields we want to index. This way, we will have all the fields we wish to use for our query:

Note: The order of the data is important because if we are interested in films from the same year and genre, those columns will have the most weight in the index.

If we are looking for recommendations based on the plot, genre, director, and actors, that would be the order:

“Overview: [Overview]. Genre: [Genre]. Directed by [Director] and starring by [Star1], [Star2], [Star3] and [Star4]. Movie year [Year]. Ranking: [Rating]."

We created the St.Vectorsearch.Movie table with the following fields:
|Field|Type|Description|
|-|-|
|Link|%String|Link to the movie poster|
|Title|%String|Movie title|
|Year|%Integer|Year of the film release|
|Certificate|%String|Age classification (see attached table)|
|Runtime|%Integer|Film duration in minutes|
|Genre|%String|Film genre|
|Rating|%Decimal(3,1)|IMDb rating|
|Overview|%String|Movie description|
|Director|%String|Name of director|
|Star1|%String|Names of actors/actresses|
|Star2|%String|Names of actors/actresses|
|Star3|%String|Names of actors/actresses|
|Star4|%String|Names of actors/actresses|
|Vector|%Vector|Vector value of the film card|

To understand the age classifications better, I will show you the meaning of those values:

In the same way that we initialized the table and loaded the data from the Feeling table, we employ the following commands:

Note: To access the terminal of our Docker instance, use the following command:
Docker-compose exec iris iris session iris

Do ##class(St.vectorsearch.Data).InitMovie()

Next, we will load the movie data from the /opt/irisbuild/data/imdb_top_1000.csv directory, using the PopulateMovie command from the St.vectorsearch.Data class.

USER>do ##class(St.Vectorsearch.Data).PopulateMovie()
Truncating table St_Vectorsearch.Movie
Preparing to load data from file imdb_top_1000.csv
Loading data from file imdb_top_1000.csv
Total records loaded: 999
Calculating vectors for records...
Processing 116 of 999

Note: During the demo initialization process in Docker, the test data loading operations are already performed.

In each movie, we have initialized the vector value using the phrase we have created with the different fields. I encourage everyone to make appropriate changes to the St.Vectorsearch.Data.PopulateMovie class to align with your priorities better.

As I mentioned in the first part, there is another vector approximation method (by cosine similarity), which allows us to search vectors that form the closest angle (those that are closest in the same direction).

That is why it is the best system to use if you wish to find data recommendations.

To use this, we will need the condition “VECTOR_COSINE”, which can tell us the closest angles to the indicated vector.

SELECT TOP 5 ID, Link, Title, Year
FROM St_Vectorsearch.Movie 
WHERE Title <> ? 
ORDER BY VECTOR_COSINE(Vector, TO_VECTOR(?, float)) DESC

In this case, we are going to look for movies that approximate the angle of the vector associated with the film we have, without, of course, having it recommend itself.

You can access the demo via the link below:

http://localhost:52773/csp/user/movies.html

In this case, the recommended films are similar to 1988's "Akira," although I would personally put "Kôkaku Kidôtai," also known as "Ghost in the Shell," first. They are similar because they are both animated films. Mental note: I still need to watch Papurika...

So, that is all for now. You have everything you need to work with vector indices, including practical examples and an application you can use for your experiments.

Please leave any suggestions or ideas you have in the comments, including practical ways to implement this knowledge, etc...

See you at the next “InterSystems for Dummies”!