The updates to graphing with PySpark onDatabricks have made it much nicer to work with. Options exist to aggregate data direct in the graph, handle time series data and have independent axes for two different data series in the same graph.
Whenever you use the display function to display a pyspark dataframe, Databricks gives you the option to perform a visualisation. You can add one or more visualisations to a cell by clicking the + button (highlighted green below) just below it that appears when you display a dataframe.
Once you add a visualisation you get to set various options. You can choose between different types of graph, select which columns to plot against which axes and set other properties. There is the ability for you to plot multiple columns on the Y axis if desired. You can also group data if you wish and apply different logic to what you are plotting such as sums and averages. This allows for rapid visualisation of data prior to grouping operations in PySpark.
Other useful abilities are the ability to easily apply logarithmic axes, and the ability to plot two sets of data with scales on the left and right of the the table. Also the plotting has specific modes for time series and categorical data
On top of that you can customise colours, specify data labelling conventions and easily export the graphs if required. It makes rapid visualisation of your data in Databricks much easier than it used to be. The legacy visualisation is still present if you desire and there is also a handy visual proofiler tool to give you basic information about your dataframe.
I have placed an example workbook with some time series data on Databricks Community Edition. Note that it uses a couple of mismatched datasets for sunspots and births found here. This is just to allow for easy plotting of two variables. Unsurprisingly there is not a correlation! You can also get it from Github, note that while it is a notebook the graphing is Databricks specific. hopefully it might help you with graphing with PySpark on Databricks
