https://blog.octo.com/en/how-to-hack-spark-to-do-some-data-lineage/
blog-post-hack-spark-for-data-lineage#data-lineage1Data lineage, or data tracking, is generally defined as a type of data lifecycle that includes data origins and data movement over time. It can also describe transformations applied to the data as it passes through various processes. Data lineage can help analyse how information is used and track key information that serves a particular purpose. blog-post-hack-spark-for-data-lineage#data-lineage1