.join()

Certainly! Let's break down the Python code step by step:

1. Read the Data: First, you read the data from an Excel file into a pandas DataFrame named dk and another DataFrame named filter_1.

import pandas as pd

# Read data from Excel files
dk = pd.read_excel("./data/ne3.xlsx")
filter_1 = pd.read_excel("./data/ne2.xlsx", sheet_name="data")

2. Data Preprocessing:
- Convert all values to strings in filter_1.
- Create a new column home_sets containing sets of values from columns 2 onwards in filter_1.
- Remove 'nan' from each set.

# Convert all values to strings
filter_1 = filter_1.applymap(str)

# Create a new column containing sets of values from other columns
filter_1['home_sets'] = filter_1.iloc[:, 2:].apply(lambda x: set(x), axis=1)

# Remove 'nan' from each set
for i in filter_1.index:
    filter_1['home_sets'][i].discard('nan')

3. Matching Descriptions with Models:
- Create a list of tuples (word_to_model) containing unique combinations of model names and their corresponding normalized forms.
- Process each description in the dk_need DataFrame and find matches with the models from word_to_model using string matching.

import re

# Function to normalize text
def re_text(x):
    text = re.sub(r'[^A-Za-z0-9А-Яа-я ]+', '', x).replace("  ", "").lower()
    return text

# Apply normalization function to 'combine2' column in dk_need DataFrame
dk_need['combine2'] = dk_need['combine2'].apply(re_text)

# Create a list of tuples containing unique combinations of model names and their normalized forms
word_to_model = []
for _, row in filter_1.iterrows():
    for home in row['home_sets']:
        arent_ = re_text(home).replace(' ', '')
        word_to_model.append((row['name'], arent_))

# Function to process descriptions and find matches with models
def process_descriptions(descriptions, word_to_model):
    matched_models = []
    for _, row_ in descriptions.iterrows():
        desc = row_['combine2']
        desc_ = desc.replace(' ', '')
        match = next(({'combine2': desc, 'arentir': name}
                      for name, arent_ in word_to_model
                      if (arent_ in desc) or (name in desc) or (arent_ in desc_) or (name in desc_)),
                     {'combine2': desc,  'arentir': 'Not matched'})
        matched_models.append(match)
    return matched_models

# Process each description in parallel using ThreadPoolExecutor
import concurrent.futures

with concurrent.futures.ThreadPoolExecutor() as executor:
    # Split descriptions into chunks
    chunks = [dk_need.iloc[i:i + 100] for i in range(0, len(dk_need), 100)]
    # Process each chunk in parallel
    results = executor.map(lambda chunk: process_descriptions(chunk, word_to_model), chunks)

# Combine results
all_matched_models = [item for sublist in results for item in sublist]

# Convert the results into a DataFrame
df_models = pd.DataFrame(all_matched_models)

4. Explanation:
- The code first normalizes the text in the combine2 column of the dk_need DataFrame to remove special characters, convert to lowercase, and remove extra spaces.
- It then creates a list of tuples (word_to_model) containing unique combinations of model names and their corresponding normalized forms.
- Using multithreading with ThreadPoolExecutor, it processes each chunk of descriptions in parallel and finds matches with models based on the normalized forms.
- Finally, it combines all the results into a DataFrame (df_models) containing the matched descriptions and model names.

This Python code performs similar functionality to the SQL code provided earlier, but it's implemented using pandas and Python's concurrent programming features for parallel processing.

Iterrows
Re.sub
Append