Here are top 40 commonly asked pyspark questions that you can prepare for interviews.

𝗥𝗗𝗗𝘀 -
1. What is an RDD in Apache Spark? Explain its characteristics.
2. How are RDDs fault-tolerant in Apache Spark?
3. What are the different ways to create RDDs in Spark?
4. Explain the difference between transformations and actions in RDDs.
5. How does Spark handle data partitioning in RDDs?
6. Can you explain the lineage graph in RDDs and its significance?
7. What is lazy evaluation in Apache Spark RDDs?
8. How can you persist RDDs in memory for faster access?
9. Explain the concept of narrow and wide transformations in RDDs.
10. What are the limitations of RDDs compared to DataFrames and Datasets?

𝗗𝗮𝘁𝗮𝗳𝗿𝗮𝗺𝗲 𝗮𝗻𝗱 𝗗𝗮𝘁𝗮𝘀𝗲𝘁𝘀 -
1. What are DataFrames and Datasets in Apache Spark?
2. What are the differences between DataFrame and RDD?
3. Explain the concept of a schema in a DataFrame.
4. How are DataFrames and Datasets fault-tolerant in Spark?
5. What are the advantages of using DataFrames over RDDs?
6. Explain the Catalyst optimizer in Apache Spark.
7. How can you create DataFrames in Apache Spark?
8. What is the significance of Encoders in Datasets?
9. How does Spark SQL optimize the execution plan for DataFrames?
10. Can you explain the benefits of using Datasets over DataFrames?

𝗦𝗽𝗮𝗿𝗸 𝗦𝗤𝗟 -
1. What is Spark SQL, and how does it relate to Apache Spark?
2. How does Spark SQL leverage DataFrame and Dataset APIs?
3. Explain the role of the Catalyst optimizer in Spark SQL.
4. How can you run SQL queries on DataFrames in Spark SQL?
5. What are the benefits of using Spark SQL over traditional SQL queries?

𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 -
1. What are some common performance bottlenecks in Apache Spark applications?
2. How can you optimize the shuffle operations in Spark?
3. Explain the significance of data skew and techniques to handle it in Spark.
4. What are some techniques to optimize Spark job execution time?
5. How can you tune memory configurations for better performance in Spark?
6. What is dynamic allocation, and how does it optimize resource usage in Spark?
7. How can you optimize joins in Spark?
8. What are the benefits of partitioning data in Spark?
9. How does Spark leverage data locality for optimization?

All the best 👍👍

Date: 09-04-2025
Company name: Capgemini
Role: Data Scientist

01. What are decorators in Python?

Answer-  Decorators are used to add some design patterns to a function without changing its structure. Decorators generally are defined before the function they are enhancing. To apply a decorator we first define the decorator function. Then we write the function it is applied to and simply add the decorator function above the function it has to be applied to. For this, we use the @ symbol before the decorator.

02. What is the ACID property in a database?

Answer- The full form of ACID is atomicity, consistency, isolation, and durability.
•Atomicity refers to the fact that if any aspect of a transaction fails, the whole transaction fails and the database state remains unchanged.
•Consistency means that the data meets all validity guidelines.
•Concurrency management is the primary objective of isolation.
•Durability ensures that once a transaction is committed, it will occur regardless of what happens in between such as a power outage, fire, or some other kind of disturbance.

03. Explain One-hot encoding and Label Encoding. How do they affect the dimensionality of the given dataset?

Answer-  One-hot encoding is the representation of categorical variables as binary vectors. Label Encoding is converting labels/words into numeric form. Using one-hot encoding increases the dimensionality of the data set. Label encoding doesn’t affect the dimensionality of the data set. One-hot encoding creates a new variable for each level in the variable whereas, in Label encoding, the levels of a variable get encoded as 1 and 0.

04.  Given a table Employee having columns empName and empId, what will be the result of the SQL query below?

Answer-  Select empName from Employee order by 2 asc;

“Order by 2” is valid when there are at least 2 columns used in SELECT statement. Here this query will throw error because only one column is used in the SELECT statement.

————————————————————

Stay Safe & Happy Learning 💙

Planning for Data Engineering Interview.

Focus on SQL & Python first. Here are some important questions which you should know.

𝐈𝐦𝐩𝐨𝐫𝐭𝐚𝐧𝐭 𝐒𝐐𝐋 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬

1- Find out nth Order/Salary from the tables.
2- Find the no of output records in each join from given Table 1 & Table 2
3- YOY,MOM Growth related questions.
4- Find out Employee ,Manager Hierarchy (Self join related question) or
Employees who are earning more than managers.
5- RANK,DENSERANK related questions
6- Some row level scanning medium to complex questions using CTE or recursive CTE, like (Missing no /Missing Item from the list etc.)
7- No of matches played by every team or Source to Destination flight combination using CROSS JOIN.
8-Use window functions to perform advanced analytical tasks, such as calculating moving averages or detecting outliers.
9- Implement logic to handle hierarchical data, such as finding all descendants of a given node in a tree structure.
10-Identify and remove duplicate records from a table.


𝐈𝐦𝐩𝐨𝐫𝐭𝐚𝐧𝐭 𝐏𝐲𝐭𝐡𝐨𝐧 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬

1- Reversing a String using an Extended Slicing techniques.
2- Count Vowels from Given words .
3- Find the highest occurrences of each word from string and sort them in order.
4- Remove Duplicates from List.
5-Sort a List without using Sort keyword.
6-Find the pair of numbers in this list whose sum is n no.
7-Find the max and min no in the list without using inbuilt functions.
8-Calculate the Intersection of Two Lists without using Built-in Functions
9-Write Python code to make API requests to a public API (e.g., weather API) and process the JSON response.
10-Implement a function to fetch data from a database table, perform data manipulation, and update the database.

All the best 👍👍

Date: 12-04-2025
Company name: Flipkart
Role: Data Scientist

01. What are Support Vectors in SVM?

Answer-  Support Vector Machine (SVM) is an algorithm that tries to fit a line (or plane or hyperplane) between the different classes that maximizes the distance from the line to the points of the classes.

In this way, it tries to find a robust separation between the classes. The Support Vectors are the points of the edge of the dividing hyperplane.

02. Explain Correlation and Covariance?

Answer- Covariance signifies the direction of the linear relationship between two variables, whereas correlation indicates both the direction and strength of the linear relationship between variables.

03. What is P-value?

Answer-  P-values are used to make a decision about a hypothesis test. P-value is the minimum significant level at which you can reject the null hypothesis. The lower the p-value, the more likely you reject the null hypothesis.

04.  Does Random Forest need Pruning? Why or why not?

Answer-

(a) Pruning is a data compression technique in machine learning and search algorithms that reduces the size of decision trees by removing sections of the tree that are non-critical and redundant to classify instances.

(b) Random Forest usually does not require pruning because it will not over-fit like a single decision tree. This happens due to the fact that the trees are bootstrapped and that multiple random trees use random features so the individual trees are strong without being correlated with each other.

————————————————————


Stay Safe & Happy Learning 💙

DevOps Tech Stack

Date: 09-05-2025
Company name: Tredence
Role: Data Scientist

01. What is the exploding gradient problem while using the back propagation technique?

Answer:  When large error gradients accumulate and result in large changes in the neural network weights during training, it is called the exploding gradient problem. The values of weights can become so large as to overflow and result in NaN values. This makes the model unstable and the learning of the model to stall just like the vanishing gradient problem. This is one of the most commonly asked interview questions on machine learning.

02. What do you mean by Associative Rule Mining (ARM)?

Answer- Associative Rule Mining is one of the techniques to discover patterns in data like features (dimensions) which occur together and features (dimensions) which are correlated. It is mostly used in Market-based Analysis to find how frequently an itemset occurs in a transaction. Association rules have to satisfy minimum support and minimum confidence at the very same time.

03. What is the primary difference between R square and adjusted R square?

Answer- In linear regression, you use both these values for model validation. However, there is a clear distinction between the two. R square accounts for the variation of all independent variables on the dependent variable. In other words, it considers each independent variable for explaining the variation. In the case of Adjusted R square, it accounts for the significant variables alone for indicating the percentage of variation in the model. By significant, we refer to the P values less than 0.05.

04. What are hard margin and soft Margin SVMs?

Answer-  Hard margin SVMs work only if the data is linearly separable and these types of SVMs are quite sensitive to the outliers. But our main objective is to find a good balance between keeping the margins as large as possible and limiting the margin violation i.e. instances that end up in the middle of margin or even on the wrong side, and this method is called soft margin SVM.

————————————————————



Stay Safe & Happy Learning 💙

Data Lake vs Data Warehouse

SQL Cheatsheet 📝

This SQL cheatsheet is designed to be your quick reference guide for SQL programming. Whether you’re a beginner learning how to query databases or an experienced developer looking for a handy resource, this cheatsheet covers essential SQL topics.

1. Database Basics
   - CREATE DATABASE db_name;
   - USE db_name;

2. Tables
   - Create Table: CREATE TABLE table_name (col1 datatype, col2 datatype);
   - Drop Table: DROP TABLE table_name;
   - Alter Table: ALTER TABLE table_name ADD column_name datatype;

3. Insert Data
   - INSERT INTO table_name (col1, col2) VALUES (val1, val2);

4. Select Queries
   - Basic Select: SELECT * FROM table_name;
   - Select Specific Columns: SELECT col1, col2 FROM table_name;
   - Select with Condition: SELECT * FROM table_name WHERE condition;

5. Update Data
   - UPDATE table_name SET col1 = value1 WHERE condition;

6. Delete Data
   - DELETE FROM table_name WHERE condition;

7. Joins
   - Inner Join: SELECT * FROM table1 INNER JOIN table2 ON table1.col = table2.col;
   - Left Join: SELECT * FROM table1 LEFT JOIN table2 ON table1.col = table2.col;
   - Right Join: SELECT * FROM table1 RIGHT JOIN table2 ON table1.col = table2.col;

8. Aggregations
   - Count: SELECT COUNT(*) FROM table_name;
   - Sum: SELECT SUM(col) FROM table_name;
   - Group By: SELECT col, COUNT(*) FROM table_name GROUP BY col;

9. Sorting & Limiting
   - Order By: SELECT * FROM table_name ORDER BY col ASC|DESC;
   - Limit Results: SELECT * FROM table_name LIMIT n;

10. Indexes
   - Create Index: CREATE INDEX idx_name ON table_name (col);
   - Drop Index: DROP INDEX idx_name;

11. Subqueries
   - SELECT * FROM table_name WHERE col IN (SELECT col FROM other_table);

12. Views
   - Create View: CREATE VIEW view_name AS SELECT * FROM table_name;
   - Drop View: DROP VIEW view_name;


Hope it helps :)

 CGI Interview Questions 

1) Introduce Yourself

2) What is Full Stack Development?

3) Difference between GET and POST method

4) Explain OOPs Concepts with Examples

5) What is the difference between SQL and NoSQL?

6) Explain SDLC and Agile Model

7) Describe your Final Year Project

8) Write an SQL query to count employees in each department

9) SQL query to find duplicate records in a table

10) Difference between INNER JOIN and LEFT JOIN

11) SQL query to fetch top 3 highest salaries

12) Write a program to check if a number is a palindrome

13) Write a program to check if two strings are anagrams

14) Find the missing number in a sorted array

15) Implement Stack using Array

16) What is Time and Space Complexity?

17) What are your strengths and weaknesses?

18) Why do you want to join CGI?

19) Are you open to working night shifts or relocating?

20) How do you manage deadlines or pressure?

Accenture Interview Questions
1. Tell me about yourself. 
2. Tell me about your project and your roles in your project
3. Are u designing the framework, so tell me the folder structure of your framework.
4. Tell me what type of framework you are using.
5. Tell me the maven command.
6. Write a program to reverse a string.
7. Open the url Amazon, search for mobiles, scroll the page two times, and find the xpath of 7th listed Mobile phone....write a generic xpath. It should work when I use the same xpath in the new tab also.
8. Suppose there are multiple mobile numbers given in a website ..how u find which are the valid mobile number and which are invalid..give count also.
9. Write a selenium program to read the data from the Excel sheet.
10. Difference between method overloading and method overriding
11. What is the primary key, and how is it different from the foreign key.
12. Http status codes
13. Http methods call
14. What is inheritance where u apply in your framework. 
15. Explain the testng.xml file.