100% Real & Accurate Associate-Developer-Apache-Spark-3.5 Questions and Answers with Free and Fast Updates [Q16-Q37]

Q16. What is the difference betweendf.cache()anddf.persist()in Spark DataFrame?

Bothcache()andpersist()can be used to set the default storage level (MEMORY_AND_DISK_SER)

Both functions perform the same operation. Thepersist()function provides improved performance asits default storage level isDISK_ONLY.

persist()- Persists the DataFrame with the default storage level (MEMORY_AND_DISK_SER) andcache()- Can be used to set different storage levels to persist the contents of the DataFrame.

cache()- Persists the DataFrame with the default storage level (MEMORY_AND_DISK) andpersist()- Can be used to set different storage levels to persist the contents of the DataFrame

Q17. A data engineer noticed improved performance after upgrading from Spark 3.0 to Spark 3.5. The engineer found that Adaptive Query Execution (AQE) was enabled.
Which operation is AQE implementing to improve performance?

Dynamically switching join strategies

Collecting persistent table statistics and storing them in the metastore for future use

Improving the performance of single-stage Spark jobs

Optimizing the layout of Delta files on disk

Q18. A Spark application developer wants to identify which operations cause shuffling, leading to a new stage in the Spark execution plan.
Which operation results in a shuffle and a new stage?

DataFrame.groupBy().agg()

DataFrame.filter()

DataFrame.withColumn()

DataFrame.select()

Q19. Given this view definition:
df.createOrReplaceTempView(“users_vw”)
Which approach can be used to query the users_vw view after the session is terminated?
Options:

Query the users_vw using Spark

Persist the users_vw data as a table

Recreate the users_vw and query the data using Spark

Save the users_vw definition and query using Spark

Q20. Which UDF implementation calculates the length of strings in a Spark DataFrame?

df.withColumn(“length”, spark.udf(“len”, StringType()))

df.select(length(col(“stringColumn”)).alias(“length”))

spark.udf.register(“stringLength”, lambda s: len(s))

df.withColumn(“length”, udf(lambda s: len(s), StringType()))

Q21. A data engineer is working on a real-time analytics pipeline using Apache Spark Structured Streaming. The engineer wants to process incoming data and ensure that triggers control when the query is executed. The system needs to process data in micro-batches with a fixed interval of 5 seconds.
Which code snippet the data engineer could use to fulfil this requirement?
A)

B)

C)

D)

Options:

Uses trigger(continuous=’5 seconds’) – continuous processing mode.

Uses trigger() – default micro-batch trigger without interval.

Uses trigger(processingTime=’5 seconds’) – correct micro-batch trigger with interval.

Uses trigger(processingTime=5000) – invalid, as processingTime expects a string.

Q22. A data scientist has identified that some records in the user profile table contain null values in any of the fields, and such records should be removed from the dataset before processing. The schema includes fields like user_id, username, date_of_birth, created_ts, etc.
The schema of the user profile table looks like this:

Which block of Spark code can be used to achieve this requirement?
Options:

filtered_df = users_raw_df.na.drop(thresh=0)

filtered_df = users_raw_df.na.drop(how=’all’)

filtered_df = users_raw_df.na.drop(how=’any’)

filtered_df = users_raw_df.na.drop(how=’all’, thresh=None)

Q23. Given the code fragment:

import pyspark.pandas as ps
psdf = ps.DataFrame({‘col1’: [1, 2], ‘col2’: [3, 4]})
Which method is used to convert a Pandas API on Spark DataFrame (pyspark.pandas.DataFrame) into a standard PySpark DataFrame (pyspark.sql.DataFrame)?

psdf.to_spark()

psdf.to_pyspark()

psdf.to_pandas()

psdf.to_dataframe()

Q24. A data engineer is running a batch processing job on a Spark cluster with the following configuration:
10 worker nodes
16 CPU cores per worker node
64 GB RAM per node
The data engineer wants to allocate four executors per node, each executor using four cores.
What is the total number of CPU cores used by the application?

160

64

80

40

Q25. A data engineer is working on the DataFrame:

(Referring to the table image: it has columnsId,Name,count, andtimestamp.) Which code fragment should the engineer use to extract the unique values in theNamecolumn into an alphabetically ordered list?

df.select(“Name”).orderBy(df[“Name”].asc())

df.select(“Name”).distinct().orderBy(df[“Name”])

df.select(“Name”).distinct()

df.select(“Name”).distinct().orderBy(df[“Name”].desc())

Q26. A data scientist is working with a Spark DataFrame called customerDF that contains customer information.
The DataFrame has a column named email with customer email addresses. The data scientist needs to split this column into username and domain parts.
Which code snippet splits the email column into username and domain columns?

customerDF.select(
col(“email”).substr(0, 5).alias(“username”),
col(“email”).substr(-5).alias(“domain”)
)

customerDF.withColumn(“username”, split(col(“email”), “@”).getItem(0))
.withColumn(“domain”, split(col(“email”), “@”).getItem(1))

customerDF.withColumn(“username”, substring_index(col(“email”), “@”, 1))
.withColumn(“domain”, substring_index(col(“email”), “@”, -1))

customerDF.select(
regexp_replace(col(“email”), “@”, “”).alias(“username”),
regexp_replace(col(“email”), “@”, “”).alias(“domain”)
)

Q27. A data engineer writes the following code to join two DataFramesdf1anddf2:
df1 = spark.read.csv(“sales_data.csv”) # ~10 GB
df2 = spark.read.csv(“product_data.csv”) # ~8 MB
result = df1.join(df2, df1.product_id == df2.product_id)

Which join strategy will Spark use?

Shuffle join, because AQE is not enabled, and Spark uses a static query plan

Broadcast join, as df2 is smaller than the default broadcast threshold

Shuffle join, as the size difference between df1 and df2 is too large for a broadcast join to work efficiently

Shuffle join because no broadcast hints were provided

Q28. A Spark DataFramedfis cached using theMEMORY_AND_DISKstorage level, but the DataFrame is too large to fit entirely in memory.
What is the likely behavior when Spark runs out of memory to store the DataFrame?

Spark duplicates the DataFrame in both memory and disk. If it doesn’t fit in memory, the DataFrame is stored and retrieved from the disk entirely.

Spark splits the DataFrame evenly between memory and disk, ensuring balanced storage utilization.

Spark will store as much data as possible in memory and spill the rest to disk when memory is full, continuing processing with performance overhead.

Spark stores the frequently accessed rows in memory and less frequently accessed rows on disk, utilizing both resources to offer balanced performance.

Q29. A data scientist at a financial services company is working with a Spark DataFrame containing transaction records. The DataFrame has millions of rows and includes columns fortransaction_id,account_number, transaction_amount, andtimestamp. Due to an issue with the source system, some transactions were accidentally recorded multiple times with identical information across all fields. The data scientist needs to remove rows with duplicates across all fields to ensure accurate financial reporting.
Which approach should the data scientist use to deduplicate the orders using PySpark?

df = df.dropDuplicates()

df = df.groupBy(“transaction_id”).agg(F.first(“account_number”), F.first(“transaction_amount”), F.first (“timestamp”))

df = df.filter(F.col(“transaction_id”).isNotNull())

df = df.dropDuplicates([“transaction_amount”])

Q30. A developer is working with a pandas DataFrame containing user behavior data from a web application.
Which approach should be used for executing agroupByoperation in parallel across all workers in Apache Spark 3.5?
A)
Use the applylnPandas API
B)

C)

D)

Use theapplyInPandasAPI:
df.groupby(“user_id”).applyInPandas(mean_func, schema=”user_id long, value double”).show()

Use themapInPandasAPI:
df.mapInPandas(mean_func, schema=”user_id long, value double”).show()

Use a regular Spark UDF:
from pyspark.sql.functions import mean
df.groupBy(“user_id”).agg(mean(“value”)).show()

Use a Pandas UDF:
@pandas_udf(“double”)
def mean_func(value: pd.Series) -> float:
return value.mean()
df.groupby(“user_id”).agg(mean_func(df[“value”])).show()

Q31. Given a CSV file with the content:

And the following code:
from pyspark.sql.types import *
schema = StructType([
StructField(“name”, StringType()),
StructField(“age”, IntegerType())
])
spark.read.schema(schema).csv(path).collect()
What is the resulting output?

[Row(name=’bambi’), Row(name=’alladin’, age=20)]

[Row(name=’alladin’, age=20)]

[Row(name=’bambi’, age=None), Row(name=’alladin’, age=20)]

The code throws an error due to a schema mismatch.

Q32. A data scientist is analyzing a large dataset and has written a PySpark script that includes several transformations and actions on a DataFrame. The script ends with acollect()action to retrieve the results.
How does Apache Spark™’s execution hierarchy process the operations when the data scientist runs this script?

The script is first divided into multiple applications, then each application is split into jobs, stages, and finally tasks.

The entire script is treated as a single job, which is then divided into multiple stages, and each stage is further divided into tasks based on data partitions.

Thecollect()action triggers a job, which is divided into stages at shuffle boundaries, and each stage is split into tasks that operate on individual data partitions.

Spark creates a single task for each transformation and action in the script, and these tasks are grouped into stages and jobs based on their dependencies.

Q33. An engineer has a large ORC file located at/file/test_data.orcand wants to read only specific columns to reduce memory usage.
Which code fragment will select the columns, i.e.,col1,col2, during the reading process?

spark.read.orc(“/file/test_data.orc”).filter(“col1 = ‘value’ “).select(“col2”)

spark.read.format(“orc”).select(“col1”, “col2”).load(“/file/test_data.orc”)

spark.read.orc(“/file/test_data.orc”).selected(“col1”, “col2”)

spark.read.format(“orc”).load(“/file/test_data.orc”).select(“col1”, “col2”)

Q34. The following code fragment results in an error:
@F.udf(T.IntegerType())
def simple_udf(t: str) -> str:
return answer * 3.14159
Which code fragment should be used instead?

@F.udf(T.IntegerType())
def simple_udf(t: int) -> int:
return t * 3.14159

@F.udf(T.DoubleType())
def simple_udf(t: float) -> float:
return t * 3.14159

@F.udf(T.DoubleType())
def simple_udf(t: int) -> int:
return t * 3.14159

@F.udf(T.IntegerType())
def simple_udf(t: float) -> float:
return t * 3.14159

Q35. A Spark engineer must select an appropriate deployment mode for the Spark jobs.
What is the benefit of using cluster mode in Apache Spark™?

In cluster mode, resources are allocated from a resource manager on the cluster, enabling better performance and scalability for large jobs

In cluster mode, the driver is responsible for executing all tasks locally without distributing them across the worker nodes.

In cluster mode, the driver runs on the client machine, which can limit the application’s ability to handle large datasets efficiently.

In cluster mode, the driver program runs on one of the worker nodes, allowing the application to fully utilize the distributed resources of the cluster.

Q36. An MLOps engineer is building a Pandas UDF that applies a language model that translates English strings into Spanish. The initial code is loading the model on every call to the UDF, which is hurting the performance of the data pipeline.
The initial code is:

def in_spanish_inner(df: pd.Series) -> pd.Series:
model = get_translation_model(target_lang=’es’)
return df.apply(model)
in_spanish = sf.pandas_udf(in_spanish_inner, StringType())
How can the MLOps engineer change this code to reduce how many times the language model is loaded?

Convert the Pandas UDF to a PySpark UDF

Convert the Pandas UDF from a Series # Series UDF to a Series # Scalar UDF

Run thein_spanish_inner()function in amapInPandas()function call

Convert the Pandas UDF from a Series # Series UDF to an Iterator[Series] # Iterator[Series] UDF

Q37. Given a DataFramedfthat has 10 partitions, after running the code:
result = df.coalesce(20)
How many partitions will the result DataFrame have?

10

Same number as the cluster executors

1

20

Free Exam Dumps Torrent

100% Real & Accurate Associate-Developer-Apache-Spark-3.5 Questions and Answers with Free and Fast Updates [Q16-Q37]

Leave a Reply Cancel reply

Related Posts

Best Way To Study For Databricks Databricks-Certified-Professional-Data-Engineer Exam Brilliant Databricks-Certified-Professional-Data-Engineer Exam Questions PDF [Q22-Q38]

Leave a Reply Cancel reply