⚠️Attention! Spark = Pandas + Big Data support

Be careful when applying your Pandas knowledge to Spark!!!

Of course, Pandas and Spark operate on the same data type — tables. However, the way they interact with them is significantly different.
For example, the main difference is that Pandas runs in a single process on a single machine and loads all the data into memory, while Spark is designed to work with large distributed data sets and can process terabytes and petabytes of data without loading it entirely into the memory of a single node

However, unfortunately, many programmers often transfer their knowledge from Pandas to Spark, assuming similar architectures, which leads to performance bottlenecks.

You can learn more about solving this problem from this article