Python-Pandas Operations For Working with Data

▎Common Python Terms

1. Variable: A symbolic name that is a reference or pointer to an object. When you create a variable, you allocate some memory for its value.

2. Data Type: A classification that specifies which type of value a variable can hold and what operations can be performed on it (e.g., int, float, str, bool, list, dict).

3. Function: A block of organized, reusable code that is used to perform a single, related action. Functions provide better modularity for your application and a high degree of code reusing.

4. Method: A function that is associated with an object or a class. It operates on the data (attributes) of that object.

5. Class: A blueprint for creating objects. It defines a set of attributes (data) and methods (functions) that characterize any object created from it.

6. Object: An instance of a class. It is a collection of data (attributes) and functions (methods) that operate on that data.

7. Module: A file containing Python definitions and statements. The file name is the module name with the suffix .py appended. Modules allow you to logically organize your Python code.

8. Package: A Python module can be grouped into a package. A package is a directory of Python modules containing an __init__.py file (which can be empty). Packages allow for a hierarchical structuring of the module namespace.

9. Interpreter: The program that reads and executes Python code. Python code is not compiled to machine code directly; it is compiled to bytecode, which is then interpreted.

10. Syntax: The set of rules that defines the combinations of symbols that are considered to be correctly structured programs in a particular language.

11. Indentation: Python uses whitespace (spaces or tabs) to define code blocks (e.g., inside loops, functions, or conditional statements) rather than braces like in many other languages.

12. List: An ordered, mutable (changeable) collection of items. Items can be of different data types.

13. Tuple: An ordered, immutable (unchangeable) collection of items. Like lists, tuples can contain items of different data types.

14. Dictionary: An unordered collection of key-value pairs. Keys must be unique and immutable (e.g., strings, numbers, tuples), while values can be of any data type.

15. String: An immutable sequence of characters, used to represent text data.

16. Loop: A control flow statement that allows code to be executed repeatedly. Common loops are for and while.

17. Conditional Statement: Statements that perform different computations or actions depending on whether a programmer-specified boolean condition evaluates to true or false. Common are if, elif, else.

18. Exception Handling: A mechanism to handle runtime errors or unexpected events gracefully, preventing the program from crashing. Uses try, except, finally blocks.

19. Library/Framework: A collection of pre-written code (functions, classes) that developers can use to perform common tasks without writing the code from scratch. Examples: NumPy, Pandas, Django, Flask.

20. IDE (Integrated Development Environment): A software application that provides comprehensive facilities to computer programmers for software development. It typically includes a code editor, debugger, and build automation tools. Examples: VS Code, PyCharm.

🐍 5 Python Mistakes Beginners Make (And Don’t Realize)

If you’re learning Python…
you’ve probably done these already 👇

1️⃣ Using = Instead of == ❌

if x = 5:   # ❌ Error

👉 = is assignment
👉 == is comparison


2️⃣ Not Understanding Mutable Objects 🔗

a = [1, 2]
b = a

b.append(3)
print(a)

Output: [1, 2, 3]

👉 Both changed 😳
👉 Because they point to SAME object


3️⃣ Forgetting Indentation ⚠️

if True:
print("Hello")   # ❌

👉 Python depends on indentation
👉 One mistake → code breaks


4️⃣ Using is Instead of == 🤯

a = 1000
b = 1000

print(a is b)

👉 Might return False
👉 Because is checks memory, not value


5️⃣ Modifying List While Looping 🔥

nums = [1, 2, 3]

for x in nums:
    nums.remove(x)

👉 Leads to unexpected bugs
👉 Loop skips elements


💡 Key Idea

Most bugs in Python
come from misunderstanding how it works internally 🚀

▎Top 5 Resources for Every Aspiring Python Developer


▎1. Automate the Boring Stuff with Python

• Type: Book/Online Course
• Author: Al Sweigart
• This resource is perfect for beginners who want to learn Python through practical projects. The book covers automation tasks like web scraping, working with spreadsheets, and more. The online course is available for free on platforms like Udemy.

• Link: Automate the Boring Stuff


▎2. Python for Everybody

• Type: Online Course
• Instructor: Dr. Charles Severance
• This course is designed for beginners and covers the basics of programming using Python. It's structured to help learners understand data handling, web scraping, and database management.

• Link: Python for Everybody


▎3. Real Python

• Type: Online Tutorials/Articles
• Real Python offers a wealth of tutorials, articles, and video courses on various Python topics. It caters to all skill levels and includes practical examples and projects to reinforce learning.

• Link: Real Python


▎4. Python Crash Course

• Type: Book
• Author: Eric Matthes
• This hands-on guide is ideal for beginners and intermediate learners. It covers the fundamentals of Python programming and includes projects such as building web applications and data visualizations.

• Link: Python Crash Course


▎5. The Hitchhiker's Guide to Python

• Type: Book/Online Resource
• Authors: Kenneth Reitz and Tanya Schlusser
• This comprehensive guide offers best practices for writing Python code. It covers everything from installation to advanced topics, making it a valuable resource for both new and experienced developers.

• Link: The Hitchhiker's Guide to Python

🏎 Python Performance: Vectorization vs. Loops in Pandas 🐼

In standard Python, we are taught to use for loops to process data. However, in Data Science, loops are the enemy of speed. If you use a loop to process a million rows in a Pandas DataFrame, your code will be 100x to 1000x slower than it needs to be.

👉 To be a pro Data Analyst, you must stop "looping" and start "vectorizing."


🐢 The Slow Way: Iterating with Loops
Python is an interpreted language, meaning every time a loop runs a calculation on a row, there is massive "overhead." The computer has to check the data type, find the memory address, and perform the math over and over again.

🚀 The Fast Way: Vectorization
Pandas (and NumPy) use Vectorization, which performs operations on entire arrays (columns) at once. This pushes the heavy lifting down to highly optimized C and Fortran code under the hood.


💻 The "Speed Race" Code

Let's say we have 1 million rows of prices and we want to apply a 10% tax.

import pandas as pd
import numpy as np
import time

# Create a DataFrame with 1 million rows
df = pd.DataFrame({'price': np.random.randint(1, 100, size=1_000_000)})

# ❌ THE SLOW WAY: Manual Loop (Don't do this!)
start = time.time()
taxes = []
for p in df['price']:
    taxes.append(p * 0.1)
df['tax_loop'] = taxes
print(f"Loop time: {time.time() - start:.4f} seconds")

# ✅ THE FAST WAY: Vectorization (The Pandas Way)
start = time.time()
df['tax_vec'] = df['price'] * 0.1
print(f"Vectorized time: {time.time() - start:.4f} seconds")

The result? The loop might take ~0.1 seconds, while the vectorized version takes ~0.001 seconds. On massive datasets, this is the difference between a task taking 10 minutes or 2 seconds.


🛠 When can't you vectorize?
If you have extremely complex logic (like an if/else that depends on three different external APIs), you might use .apply(). While .apply() is slightly better than a manual for loop, it is still significantly slower than true vectorization. Always try math-based column operations first.

👉 Write your code for the column, not for the row!

Hey there! 😊

Have you ever heard of "dunder" methods in Python?

It’s a fun little secret that can make your classes behave in some really cool ways! Let me break it down for you..

So, you know how when you create a class, it’s just a blueprint for an object, right? Well, dunder methods (which is short for "double underscore") let you customize how those objects act.

Here are a few key ones to know:

• __str__: This one lets you define what happens when you try to print your object. So instead of just seeing something like <MyClass object at 0x...>, you could make it say something meaningful, like the creature's name or description.

• __add__: This method lets you define what happens when you use the + operator with your objects. For example, if you have two magical creatures, you could combine them into a new one!

• __len__: With this, you can control what happens when someone uses len() on your object. Maybe your creature has a certain number of abilities, and you want len() to return that number.

Using these dunder methods makes your code much more intuitive and fun to work with.

So next time you're coding, think about how you can use these dunder methods to make your classes even cooler!

🏰 Python Context Managers: The Elegant Way to Handle Resources ✨

In Python, when you work with external resources like files, network connections, or database sessions, you often need to perform setup actions before using the resource and crucial cleanup actions afterward. Failing to clean up properly can lead to resource leaks, data corruption, or system instability.

👉 Context Managers provide a standardized and elegant way to manage these setup and teardown operations, ensuring resources are always handled correctly.


🐢 The Clumsy Way: Manual Resource Management (try...finally)
Without context managers, you'd typically rely on try...finally blocks to guarantee cleanup. While functional, this approach can be prone to errors if not implemented carefully.

Let's look at opening and reading a file:

file = None # Initialize to None
try:
    file = open('my_data.txt', 'r')
    content = file.read()
    # ... process content ...
    print("File read using try...finally")
except FileNotFoundError:
    print("Error: my_data.txt not found.")
finally:
    if file: # Need to check if file was successfully opened
        file.close()
        print("File closed using finally block")

This pattern requires explicit checks and can quickly become messy when managing multiple resources simultaneously.

🚀 The Elegant Way: The with Statement
Python's with statement, in conjunction with context managers, automates resource management. It ensures that the __exit__ method of the context manager is always called, even if errors occur.

Consider the same file operation using the with statement:

try:
    with open('my_data.txt', 'r') as file:
        content = file.read()
        # ... process content ...
        print("File read using 'with' statement")
    # The file is *automatically* closed here, even if an error happened inside the block.
    print("File automatically closed after 'with' block.")
except FileNotFoundError:
    print("Error: my_data.txt not found.")

This is significantly cleaner, more readable, and much less error-prone. The with statement handles the resource's lifecycle for you.


💡 How Context Managers Work: The __enter__ and __exit__ Protocol
A context manager is an object that defines two special methods:

• __enter__(self): This method is executed when the with statement is entered. It performs the setup actions and can return a value (like the file object) that will be assigned to the variable after the as keyword.
• __exit__(self, exc_type, exc_val, exc_tb): This method is executed when the with block is exited, either normally or due to an exception. It's responsible for performing cleanup actions. If an exception occurred, its details are passed as arguments. Returning True from __exit__ can suppress the exception.

You can create your own context managers by defining a class with these methods or by using the contextlib.contextmanager decorator with a generator function.


🎯 Today's Goal (What you should do)

✔️ Understand the necessity of proper resource management in Python.
✔️ Learn why manual try...finally blocks can be cumbersome and error-prone.
✔️ Master the syntax and benefits of the with statement for automatic resource handling.
✔️ Grasp the core __enter__ and __exit__ protocol that powers context managers.
✔️ Recognize how context managers simplify complex setup and teardown logic.

🌟 Python *args & *kwargs: Flexible Functions, Unlimited Arguments ✨

Have you ever wanted to write a function that can accept any number of arguments without pre-defining them all? That's where *args and *kwargs come in!

👉 These special syntaxes let your functions be incredibly flexible, handling varying inputs like a pro.

*args (Asterisk Args):
* Collects any number of positional arguments into a tuple.
* Think of it for inputs where order matters, but quantity doesn't.

*kwargs (Double Asterisk Kwargs):
* Collects any number of keyword arguments into a dictionary.
* Think of it for named inputs (like settings or attributes).

def process_data(fixed_arg, *args, **kwargs):
    print(f"Fixed argument: {fixed_arg}")
    if args:
        print(f"Additional positional arguments (tuple): {args}")
    if kwargs:
        print(f"Keyword arguments (dict): {kwargs}")

# Usage Examples:
process_data("Required")
# Output: Fixed argument: Required

process_data("Required", 1, 2, 3)
# Output:
# Fixed argument: Required
# Additional positional arguments (tuple): (1, 2, 3)

process_data("Required", name="Alice", age=30, city="NY")
# Output:
# Fixed argument: Required
# Keyword arguments (dict): {'name': 'Alice', 'age': 30, 'city': 'NY'}

process_data("Required", 10, "hello", option="fast", debug=True)
# Output:
# Fixed argument: Required
# Additional positional arguments (tuple): (10, 'hello')
# Keyword arguments (dict): {'option': 'fast', 'debug': True}

🎯 Use *args and *kwargs to build versatile, future-proof functions that adapt to diverse needs!

📘A Byte of Python

✍️ Author: Swaroop C H

Read Online

#Python
────────────────────
👉 @free_programming_books_bds 👈

▎Common Pandas Terms

1. Series: A one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.).

2. DataFrame: A two-dimensional, size-mutable, and potentially heterogeneous tabular data structure with labeled axes (rows and columns).

3. Index: The labels for the rows of a Series or DataFrame, used for fast identification and alignment of data.

4. read_csv: A widely used function to load data from a Comma-Separated Values file into a Pandas DataFrame.

5. head() / tail(): Methods used to quickly inspect the first or last few rows (default is 5) of a DataFrame or Series.

6. loc: A label-based data selection method used to access a group of rows and columns by their labels or a boolean array.

7. iloc: An integer-location based selection method used to access data by its numerical position (0-based indexing).

8. Shape: An attribute that returns a tuple representing the dimensionality of the DataFrame (number of rows, number of columns).

9. Describe: A method that generates descriptive statistics (mean, count, std, min, max, etc.) for numerical columns in a DataFrame.

10. GroupBy: A process involving splitting the data into groups based on some criteria, applying a function, and combining the results.

11. Aggregation (agg): The process of computing a summary statistic (like sum, mean, or count) for each group in a dataset.

12. Merge: A function used to combine two DataFrames based on a common key or index, similar to a SQL JOIN operation.

13. Concatenation (concat): The process of "gluing" together multiple DataFrames or Series along a particular axis (either rows or columns).

14. dropna: A method used to remove missing values (NaN) from a Series or DataFrame.

15. fillna: A method used to replace missing values (NaN) with a specified value or a calculated value (like the mean or median).

16. Apply: A powerful method that allows you to apply a function along an axis of the DataFrame or on a Series.

17. Pivot Table: A method used to summarize and reshape data into a spreadsheet-style table, often used for multi-dimensional analysis.

18. Melt: A function used to transform a "wide" DataFrame into a "long" format, unpivoting columns into rows.

19. Vectorization: The process of performing operations on entire arrays (columns) at once without the need for explicit Python loops, ensuring high performance.

20. DatetimeIndex: A specialized type of index in Pandas that handles date and time information, enabling powerful time-series analysis and resampling.

🔢 Python enumerate(): Loop with Index, The Pythonic Way! ✨

If you ever need to loop through a list and get both the item and its index? Stop using range(len())!

👉 Python's enumerate() function gives you a clean, efficient, and Pythonic way to do exactly that.

my_fruits = ["apple", "banana", "cherry"]

# ❌ The Clumsy Way (Avoid!)
# for i in range(len(my_fruits)):
#     print(f"Fruit {i}: {my_fruits[i]}")

# ✅ The Pythonic Way: enumerate()
for index, fruit in enumerate(my_fruits):
    print(f"Fruit {index}: {fruit}")

Output:

Fruit 0: apple
Fruit 1: banana
Fruit 2: cherry

🎯 Always use enumerate() when you need an index alongside your iterable items. It's cleaner, safer, and makes your loops shine!

⚡️ Python Sets: Stop Nesting Loops for Comparisons

Nested loops to find common items between lists are a performance killer. As your data grows, checking if item in list inside another loop slows down your code exponentially.

Python Sets use hash tables to turn these comparisons into lightning-fast math operations.

# Two lists with 1 million items
list_a = list(range(1_000_000))
list_b = list(range(500_000, 1_500_000))

# ❌ THE SLOW WAY: Nested lookup (O(n^2))
# This could take minutes on large lists
# common = [x for x in list_a if x in list_b]

# ✅ THE PRO WAY: Set Math (O(n))
# This happens almost instantly
common = set(list_a) & set(list_b) # Intersection
diff = set(list_a) - set(list_b)   # Items in A but not B

🎯 Stop "searching" through lists to find overlaps or differences. Convert your data to set() and use math symbols (&, -, ^) to handle large-scale comparisons in milliseconds.

📘 Annotated Algorithms in Python

✍️ Author: Massimo Di Pierro

🗓 Year: 2021

📄 Pages: 376

🧠 This open book is assembled from lectures given by the author over a period of 10 years at the School of Computing of DePaul University. The lectures cover multiple classes, including Analysis and Design of Algorithms, Scientific Computing, Monte Carlo Simulations, and Parallel Algorithms. These lectures teach the core knowledge required by any scientist interested in numerical algorithms and by students interested in computational finance.

#Algorithms

▎Asynchronous Programming in Python

Asynchronous programming allows applications to handle multiple tasks simultaneously without blocking. This is especially useful for I/O-bound operations, such as web requests, where waiting can lead to inefficiencies.

▎Key Concepts

• Event Loop: Manages and dispatches events or tasks.
• Coroutines: Functions defined with async def that can pause execution.
• Tasks: Wrappers for coroutines that run concurrently.

▎Benefits

1. Improved Performance: Handles more requests in less time.
2. Better Resource Utilization: Non-blocking I/O optimizes system resource use.
3. Responsive Applications: Keeps user interfaces responsive during background processing.

▎Getting Started with asyncio

The asyncio library provides the tools for asynchronous programming. Here’s a simple example simulating data fetching from multiple URLs:

import asyncio
import random

async def fetch_data(url):
    print(f"Fetching data from {url}...")
    await asyncio.sleep(random.uniform(1, 3))  # Simulate network delay
    print(f"Data fetched from {url}")
    return f"Data from {url}"

async def main():
    urls = ["http://example.com", "http://example.org", "http://example.net"]
    tasks = [fetch_data(url) for url in urls]
    results = await asyncio.gather(*tasks)
    print("All data fetched:", results)

# Run the main function
asyncio.run(main())

▎Explanation

• fetch_data(url): An asynchronous function simulating data fetching.
• asyncio.sleep(): A non-blocking sleep that allows other tasks to run.
• asyncio.gather(): Runs multiple coroutines concurrently.

▎Real-World Application: Web Scraping

Using aiohttp, you can perform asynchronous HTTP requests efficiently. Here’s an example:

import aiohttp
import asyncio

async def fetch(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()

async def scrape(urls):
    tasks = [fetch(url) for url in urls]
    return await asyncio.gather(*tasks)

urls = ["http://example.com", "http://example.org", "http://example.net"]

# Run the scraping function
results = asyncio.run(scrape(urls))
print("Scraped data:", results)