What happens when you ask Spark to Load a 1GB CSV file

 Ever wondered what happens when you ask Spark to Load a 1GB CSV file? Let's break it down step by step, in simple terms, from the moment you hit go until you see the results.

Image Source

Step 1: Getting Started

You command Spark to begin its tasks, either by typing commands or running a program.

Step 2: Checking the File

Spark makes sure the file you want to use exists and can be accessed.

Step 3: Slicing and Dicing

Spark determines how to divide the large 1GB CSV file into smaller partitions so it can work on them in parallel.

Step 4: Reading the Data

Spark reads the CSV file from wherever it's stored, like a hard drive or the cloud.

Step 5: Understanding the Columns

Spark quickly checks the CSV file to see what data it contains. It looks at the first few rows to guess things like column names and what type of data is in each column.

Step 6: Structuring the Data

Spark organizes the data into rows and columns, making it look like a table, which helps in easier processing.

Step 7: Storing the Data

Here's the crucial part: after organizing the data, Spark stores it in either RAM (memory) or disk storage, depending on the size of the data and available resources. If the data fits into memory, Spark prefers to keep it there for faster access. However, if the data is too large to fit into memory, Spark spills it onto disk storage to avoid out-of-memory errors.

Step 8: Partitioning the Data

Spark divides the organized data into partitions. Each partition contains a portion of the data and is processed independently by different executors in parallel.

Step 9: Showing the Goods

Finally, when you ask to see the data (maybe by using `show()`), Spark gets to work. It crunches the numbers, organizes everything, and then shows you the results on your screen.

Conclusion:

That's the journey your data takes when you unleash Spark on a 1GB CSV file. Spark's partitioning magic ensures efficient parallel processing, and its smart storage management ensures optimal performance, whether it's storing data in RAM or on disk.

References:

Comments

Post a Comment

Popular posts from this blog

Flutter for Single-Page Scrollable Websites with Navigator 2.0

Image
  Single page scrollable websites are everywhere. In this website design pattern, all the content is presented in one long-scrolling layout that contains multiple sections. Visitors can scroll or jump to a section by clicking buttons of a sticky menu. This pattern is a good fit for small content such as brochure websites, software library documentation, portfolios, and landing pages that are used to convert users [ 1 ]. Designers also love this pattern because it is simple, clean, and enables cool scroll animations. Jetbrains Mono single-page scrollable website Hive docs multi-page scrollable website In this series of articles, we will explore how to build a  single-page scrollable website  using Flutter. We will benefit from the Navigator 2.0 API to provide a good navigation experience to the users. You can find the source code of this series’ sample apps on my  Github page . The implementation details of the sample apps will be explained in the following articles: ...
Read more

A Data Science Portfolio is More Valuable than a Resume

Image
A Data Science Portfolio is More Valuable than a Resume Benjamin Obi Tayo Ph.D. Following Nov 20  · 5 min read D ata Science is a practical field. Hands-on skills are very important especially when you are interested in working outside academia as a practicing data scientist. In academia, you need more theoretical and research skills. While in-depth knowledge in the theoretical foundations of data science is important, as a practicing data scientist, hands-on experience is very crucial and one way to showcase you hands-on skills is via building a very great portfolio. Companies interested in hiring you would definitely be asking you for a portfolio, as it gives evidence of your strengths in fundamental data science concepts. While a data science resume is important, a data science portfolio is more valuable than a resume. This article will discuss 3 important platforms that can be used for portfolio buildi...
Read more

Better File Storage in Oracle Cloud

Image
Use Oracle Application Express with Oracle Cloud Infrastructure REST APIs to upload, download, and manage big files in Oracle Cloud. By Adrian Png November 27, 2019 Oracle Cloud Infrastructure  provides a comprehensive suite of REST APIs for accessing and managing cloud resources. For example, administrators can use REST APIs to create and manage autonomous databases and initiate backups. Oracle Application Express (Oracle APEX)  makes working with REST easy. The  APEX_WEB_SERVICE  package, available since Oracle APEX 4.0, provides developers with a function for consuming REST APIs. Oracle APEX 5.0 introduced the  APEX_JSON  package, allowing easy parsing of JSON responses. Oracle APEX 18.1 introduced the Web Source Module (WSM), which enables developers to access REST services and use the data in APEX components such as reports, interactive reports, and interactive grids. These features, when used together with Oracle Cloud Infrastructure REST ...
Read more