TOP PYSPARK INTERVIEW QUESTION 2023

-
What is Apache Spark and how does it differ from Hadoop? What are the benefits of using Spark over MapReduce? What is a Spark RDD and what operations can be performed on it? How does Spark handle fault-tolerance and data consistency? Explain the difference between Spark transformations and actions. What is a Spark DataFrame and how is it different from an RDD? What is Spark SQL and how does it work? How can you optimize a Spark job to improve its performance? How does Spark handle memory management and garbage collection? Explain the role of Spark Driver and Executors. What is PySpark and how does it differ from Apache Spark? How do you create a SparkContext in PySpark? What is the purpose of SparkContext? What is RDD (Resilient Distributed Dataset)? How is it different from DataFrame and Dataset? What are the different ways to create RDD in PySpark? What is the use of persist() method in PySpark? How does it differ from cache() method? What is the use of broadcast variables in PySpark...
Post a Comment

How spark read a large file (petabyte) when file can not be fit in spark's main memory

-
What will happen for large files in these cases?
1) Spark gets a location from NameNode for data . Will Spark stop in this same time because data size is too long as per information from NameNode?
2) Spark do partition of data as per datanode block size but all data can not be stored into main memory. Here we are not using StorageLevel. So what will happen here?
3) Spark do partition the data, some data will store on main memory once this main memory store's data will process again spark will load other data from disc.
Ans. First of all, Spark only starts reading in the data when an action (like count, collect or write) is called. Once an action is called, Spark loads in data in partitions - the number of concurrently loaded partitions depend on the number of cores you have available. So in Spark you can think of 1 partition = 1 core = 1 task. Note that all concurrently loaded partitions have to fit into memory, or you will get an OOM.
Assuming that you have several stages, Spark then runs the transformations from the first stage on the loaded partitions only. Once it has applied the transformations on the data in the loaded partitions, it stores the output as shuffle-data and then reads in more partitions. It then applies the transformations on these partitions, stores the output as shuffle-data, reads in more partitions and so forth until all data has been read.
If you apply no transformation but only do for instance a count, Spark will still read in the data in partitions, but it will not store any data in your cluster and if you do the count again it will read in all the data once again. To avoid reading in data several times, you might call cache or persist in which case Spark will try to store the data in you cluster. On cache (which is the same as persist(StorageLevel.MEMORY_ONLY) it will store all partitions in memory - if it doesn't fit in memory you will get an OOM. If you call persist(StorageLevel.MEMORY_AND_DISK) it will store as much as it can in memory and the rest will be put on disk. If data doesn't fit on disk either the OS will usually kill your workers.
Note that Spark has its own little memory management system. Some of the memory that you assign to your Spark job is used to hold the data being worked on and some of the memory is used for storage if you call cache or persist.
I hope this explanation helps :)


Comments

  1. Pankaj28 February 2020 at 14:30

    Hello

    ReplyDelete
  2. sherazabbasi28 December 2021 at 13:08

    hi

    ReplyDelete
  3. sherazabbasi3 January 2022 at 11:07

    Most of the time I don’t make comments on websites, but I'd like to say that this article really forced me to do so. Really nice post!
    large file transfer

    ReplyDelete

Post a Comment

Popular posts from this blog

Spark SQL “case when” and “when otherwise”

-
Image
  Like SQL “case when”  statement  and Swith statement from popular programming languages, Spark SQL Dataframe also supports similar syntax using “when otherwise” or we can also use “case when” statement. So let’s see an example on how to check for multiple conditions and replicate SQL CASE statement. Using “when otherwise” on DataFrame. Using “case when” on DataFrame. Using && and || operator First Let’s do the imports that are needed and create spark context and DataFrame. import org . apache . spark . sql . functions . { when , _ } val spark : SparkSession = SparkSession . builder ( ) . master ( "local[1]" ) . appName ( "SparkByExamples.com" ) . getOrCreate ( ) import spark . sqlContext . implicits . _ val data = List ( ( "James" , "" , "Smith" , "36636" , "M" , 60000 ) , ( "Michael" , "Rose" , "" , "40288" , ...
Read more

Hive failed renaming table with error "New location for this table already exist" ?

-
  How to rename a hive table without changing location? If you got this type error :  Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. New location for this table dbname.tablename already exists : hdfs://hanameservice/user/hive/warehouse/dbname.db/table name: Ans : Rename Table ALTER TABLE table_name RENAME TO new_table_name;  we can do that. You just need to follow below three commands in sequence. Lets say you have a external table test_1 in hive. And you want to rename it test_2 which should point test_2 location not test_1. Then you need to convert this table into Managed table using below command. test_1 -> pointing to test_1 location ALTER TABLE db_name.test_1 SET TBLPROPERTIES('EXTERNAL'='FALSE'); Rename the table name. ALTER TABLE db_name.test_1 RENAME TO db_name.test_2; Again convert the managed table after renaming to external table. ALTER TABLE db_name.test_2 SET TBLPROPERTIES('EXTERNAL'...
Read more

Top Hive Commands with Examples

-
Image
      In this blog post, let’s discuss top Hive commands with examples: If you are familiar with SQL, it’s a cakewalk. Many users can simultaneously query the data using Hive-QL. What is HQL? Hive defines a simple SQL-like query language to querying and managing large datasets called Hive-QL ( HQL ). It’s easy to use if you’re familiar with SQL Language. Hive allows programmers who are familiar with the language to write the custom MapReduce framework to perform more sophisticated analysis. Components of Hive: Metastore : Hive stores the schema of the Hive tables in a Hive Metastore. Metastore is used to hold all the information about the tables and partitions that are in the warehouse. By default, the metastore is run in the same process as the Hive service and the default Metastore is DerBy Database. SerDe : Serializer, Deserializer gives instructions to hive on how to process a record. Hive Commands : Data Defini...
Read more