Spark is great, and the more I work with it on my PhD thesis the more changes I make to my local installation on my rMBP. One of the modifications I came across the other day is how to dial down logging messages in one of the Spark shells. Specifically, how to dial dow the messages in PySpark when programming in Python.
Just follow the instructions in this post over at Stack Overflow if you're using OS X.
TLDR; version is that Spark uses Log4j for its logging services, so all you have to do is edit the conf/log4j.properties to match your needs.
