Minimizing the soul-sucking part of working with Spark, one bug at a time

PySpark debugging pt. 2

Photo by 傅甬 华 on Unsplash

1. Invalid log directory

Error: invalid log directory /usr/local/spark/work/app-
20201203195256-0001/2/

This can be a bit tricky, check your firewall rules and make sure all of your nodes have access to the storage being used. So…