What is a 𝗩𝗲𝗰𝘁𝗼𝗿 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲?

With the rise of Foundational Models, Vector Databases skyrocketed in popularity. The truth is that a Vector Database is also useful outside of a Large Language Model context.

When it comes to Machine Learning, we often deal with Vector Embeddings. Vector Databases were created to perform specifically well when working with them:

➡️ Storing.
➡️ Updating.
➡️ Retrieving.

When we talk about retrieval, we refer to retrieving set of vectors that are most similar to a query in a form of a vector that is embedded in the same Latent space. This retrieval procedure is called Approximate Nearest Neighbour (ANN) search.

A query here could be in a form of an object like an image for which we would like to find similar images. Or it could be a question for which we want to retrieve relevant context that could later be transformed into an answer via a LLM.

Let’s look into how one would interact with a Vector Database:

𝗪𝗿𝗶𝘁𝗶𝗻𝗴/𝗨𝗽𝗱𝗮𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮.

1. Choose a ML model to be used to generate Vector Embeddings.
2. Embed any type of information: text, images, audio, tabular. Choice of ML model used for embedding will depend on the type of data.
3. Get a Vector representation of your data by running it through the Embedding Model.
4. Store additional metadata together with the Vector Embedding. This data would later be used to pre-filter or post-filter ANN search results.
5. Vector DB indexes Vector Embedding and metadata separately. There are multiple methods that can be used for creating vector indexes, some of them: Random Projection, Product Quantization, Locality-sensitive Hashing.
6. Vector data is stored together with indexes for Vector Embeddings and metadata connected to the Embedded objects.

𝗥𝗲𝗮𝗱𝗶𝗻𝗴 𝗗𝗮𝘁𝗮.

7. A query to be executed against a Vector Database will usually consist of two parts:

➡️ Data that will be used for ANN search. e.g. an image for which you want to find similar ones.
➡️ Metadata query to exclude Vectors that hold specific qualities known beforehand. E.g. given that you are looking for similar images of apartments - exclude apartments in a specific location.

8. You execute Metadata Query against the metadata index. It could be done before or after the ANN search procedure.
9. You embed the data into the Latent space with the same model that was used for writing the data to the Vector DB.
10. ANN search procedure is applied and a set of Vector embeddings are retrieved. Popular similarity measures for ANN search include: Cosine Similarity, Euclidean Distance, Dot Product.

How are you using Vector DBs? Let me know in the comment section!

#RAG #LLM #DataEngineering

https://t.me/CodeProgrammer ✅

𝗪𝗵𝘆 𝗘𝘃𝗲𝗿𝘆 𝗔𝘀𝗽𝗶𝗿𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗦𝗵𝗼𝘂𝗹𝗱 𝗟𝗲𝗮𝗿𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸

If you’re working with large datasets, tools like Pandas can hit limits fast. That’s where 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 comes in—designed to scale effortlessly across big data workloads.

𝗪𝗵𝗮𝘁 𝗶𝘀 𝗣𝘆𝗦𝗽𝗮𝗿𝗸?
PySpark is the Python API for Apache Spark—a powerful engine for distributed data processing. It's widely used to build scalable ETL pipelines and handle millions of records efficiently.

𝗪𝗵𝘆 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗜𝘀 𝗮 𝗠𝘂𝘀𝘁-𝗛𝗮𝘃𝗲 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀:
✔️ Scales to handle massive datasets
✔️ Designed for distributed computing
✔️ Blends SQL with Python for flexible logic
✔️ Perfect for building end-to-end ETL pipelines
✔️ Supports integrations like Hive, Kafka, and Delta Lake

𝗤𝘂𝗶𝗰𝗸 𝗘𝘅𝗮𝗺𝗽𝗹𝗲:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Example").getOrCreate() 
df = spark.read.csv("data.csv", header=True, inferSchema=True) 
df.filter(df["age"] > 30).show()

#PySpark #DataEngineering #BigData #ETL #ApacheSpark #DistributedComputing #PythonForData #DataPipelines #SparkSQL #ScalableAnalytics

✉️ Our Telegram channels: https://t.me/addlist/0f6vfFbEMdAwODBk

📱 Our WhatsApp channel: https://whatsapp.com/channel/0029VaC7Weq29753hpcggW2A

🚀 DataCamp has officially partnered with Polars**—a cutting-edge DataFrame library designed for speed and efficiency!

To mark this exciting collaboration, **DataCamp is offering free access to its brand-new course *“Introduction to Polars”* for the next 90 days. 🎉

This course is a great opportunity for learners and professionals alike to master data cleaning, transformation, and analysis with Polars' high-performance engine, lazy execution, and powerful groupby operations.

Unlock the full potential of data workflows and explore how Polars can supercharge large-scale data processing.

🔗 Start learning now:
https://www.datacamp.com/courses/introduction-to-polars

#DataScience #Polars #Python #BigData #DataEngineering #MachineLearning #DataAnalytics #OpenSource #DataCamp #FreeCourse #LearnDataScience

🌟 Join the communities:

✉️ Our Telegram channels: https://t.me/addlist/0f6vfFbEMdAwODBk

📱 Our WhatsApp channel: https://whatsapp.com/channel/0029VaC7Weq29753hpcggW2A

📖 A comprehensive cheat sheet for working with Polars


🌟 Have you ever worked with pandas and thought that was the fastest way? I thought the same thing until I worked with Polars.

✏️ This cheat sheet explains everything about Polars in a concise and simple way. Not just theory! But also a bunch of real examples, practical experience, and projects that will really help you in the real world.

┌ 🐻‍❄️ Polars Cheat Sheet
├ ♾️ Google Colab
└ 📖 Doc

#Polars #DataEngineering #PythonLibraries #PandasAlternative #PolarsCheatSheet #DataScienceTools #FastDataProcessing #GoogleColab #DataAnalysis #PythonForDataScience



✉️ Our Telegram channels: https://t.me/addlist/0f6vfFbEMdAwODBk

📱 Our WhatsApp channel: https://whatsapp.com/channel/0029VaC7Weq29753hpcggW2A

𝗠𝗮𝘀𝘁𝗲𝗿 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗟𝗶𝗸𝗲 𝗮 𝗣𝗿𝗼 – 𝗔𝗹𝗹-𝗶𝗻-𝗢𝗻𝗲 𝗚𝘂𝗶𝗱𝗲 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀

If you're a data engineer, aspiring Spark developer, or someone preparing for big data interviews — this one is for you.
I’m sharing a powerful, all-in-one PySpark notes sheet that covers both fundamentals and advanced techniques for real-world usage and interviews.

𝗪𝗵𝗮𝘁'𝘀 𝗶𝗻𝘀𝗶𝗱𝗲? • Spark vs MapReduce
• Spark Architecture – Driver, Executors, DAG
• RDDs vs DataFrames vs Datasets
• SparkContext vs SparkSession
• Transformations: map, flatMap, reduceByKey, groupByKey
• Optimizations – caching, persisting, skew handling, salting
• Joins – Broadcast joins, Shuffle joins
• Deployment modes – Cluster vs Client
• Real interview-ready Q&A from top use cases
• CSV, JSON, Parquet, ORC – Format comparisons
• Common commands, schema creation, data filtering, null handling

𝗪𝗵𝗼 𝗶𝘀 𝘁𝗵𝗶𝘀 𝗳𝗼𝗿? Data Engineers, Spark Developers, Data Enthusiasts, and anyone preparing for interviews or working on distributed systems.

#PySpark #DataEngineering #BigData #SparkArchitecture #RDDvsDataFrame #SparkOptimization #DistributedComputing #SparkInterviewPrep #DataPipelines #ApacheSpark #MapReduce #ETL #BroadcastJoin #ClusterComputing #SparkForEngineers

✉️ Our Telegram channels: https://t.me/addlist/0f6vfFbEMdAwODBk

📱 Our WhatsApp channel: https://whatsapp.com/channel/0029VaC7Weq29753hpcggW2A