QUESTION 156 You have a Snowflake table, ‘raw_data’, which contains a column ‘data url’ storing URLs pointing to CSV files with varying schemas. Each CSV file represents sales data, but the column names and data types can differ. You need to create a process to automatically discover the schema of each CSV file, load the data into Snowflake, and standardize the column names to ‘order id’, ‘product id’, ‘quantity’, and ‘price’. Which of the following approaches best addresses this requirement, considering scalability and minimal manual intervention?
Create a stored procedure that iterates through each URL in ‘raw_data’ , downloads the CSV file using ‘SYSTEM$URL_GET , parses the CSV header to determine the column names, manually maps the discovered column names to the standardized names, creates a temporary table with the discovered schema, loads the data into the temporary table, transforms the data to use the standardized column names, and then inserts the transformed data into a final target table. Drop the temporary table after successful insertion.
Use Snowpipe with auto-ingest to continuously load the CSV files into a VARIANT column in a staging table. Create a series of views on top of the staging table, each view attempting to extract data based on different potential schema variations. Union all the views together to create a single consolidated view.
Create a Python-based external function that downloads the CSV file from the URL using a library like ‘pandas’, infers the schema using ‘pandas.read_csv’ , maps the discovered column names to the standardized names, and returns the data as a JSON string. Then, create a Snowflake table with a VARIANT column, call the external function for each URL, and load the returned JSON data into the table. Create a view on top of it.
Create a Snowflake external table that points to the external stage. Define a single file format to be used by external table. Define a pipe that uses ‘COPY INTO’ to ingest data into external table from the files found at the file URLs.
Leverage a combination of Snowflake Scripting and External functions: create external function that infer the schema of the CSV, create temporary table based on identified schema, fetch the CSV data using SYSTEM$URL GET using snowflake scripting, copy the data into the temporary table, tranform the data into required structure, ingest into target table and finally drop the temporary table
Option C is the most suitable approach. It leverages the power of Python and the ‘pandas’ library within an external function to handle the complexities of schema discovery and standardization. The external function isolates the data transformation logic, making the Snowflake SQL code cleaner. Option E is also valid as it encapsulates the schema discovery and dynamic table creation in Snowflake Scripting. Options A is error prone and not scalable. Option B uses ‘VARIANT column, but requires creation of a lot of views. Option D is incorrect since External Tables do not support data coming from URLs but rather from external stages.