<story><title>English as the new programming language for Apache spark</title><url>https://www.databricks.com/blog/introducing-english-new-programming-language-apache-spark</url></story><parent_chain></parent_chain><comment><author>pixelmonkey</author><text>One thing I find somewhat amusing about this is that all of the generated code is against the PySpark API. And the PySpark API is itself an interop layer to the native Scala APIs for Spark.&lt;p&gt;So you have LLM-based English prompts as an interop layer to Python + PySpark, which is itself an interop layer onto the Spark core. Also, the generated Spark SQL strings inside the DataFrame API have their own little compiler into Spark operations.&lt;p&gt;When Databricks wrote PySpark, it was because many programmers knew Python but weren&amp;#x27;t willing to learn Scala just to use Spark. Now, they are offering a way for programmers to not bother learning the PySpark APIs, and leverage the interop layers all the way down, starting from English prompts.&lt;p&gt;This makes perfect sense when you zoom out and think about what their goal is -- to get your data workflows running on their cluster runtime. But it does make a programmer like me -- who developed a lot of systems while Spark was growing up -- wonder just how many layers future programmers will be forced to debug through when things go wrong. Debugging PySpark code is hard enough, even when you know Python, the PySpark APIs, and the underlying Spark core architecture well. But if all the PySpark code I had ever written had started from English prompts, it might make debugging those inevitable job crashes even more bewildering.&lt;p&gt;I haven&amp;#x27;t, in this description, mentioned the &amp;quot;usual&amp;quot; programming layers we have to contend with, like Python&amp;#x27;s interpreter, the JVM, underlying operating system, cloud APIs, and so on.&lt;p&gt;If I were to take a guess, programmers of the future are going to need more help debugging across programming language abstractions, system abstraction layers, and various code-data boundaries than they currently make do with.</text></comment>
<story><title>English as the new programming language for Apache spark</title><url>https://www.databricks.com/blog/introducing-english-new-programming-language-apache-spark</url></story><parent_chain></parent_chain><comment><author>greggyb</author><text>&amp;quot;Moving four week average using only calendar weeks with complete data&amp;quot;&lt;p&gt;&amp;quot;Moving four week average to today inclusive&amp;quot;&lt;p&gt;&amp;quot;Moving four week average to today exclusive (or to yesterday)&amp;quot;&lt;p&gt;&amp;quot;Moving four week daily average&amp;quot;&lt;p&gt;&amp;quot;Moving four week weekly average&amp;quot;&lt;p&gt;&amp;quot;Moving four week daily average, but the denominator should only count days with data&amp;quot;&lt;p&gt;&amp;quot;Moving four week average, but I actually mean week-to-date total as of today&amp;quot;&lt;p&gt;And these are just a few of the variations I have had to implement recently. I have no reason to expect things to end up one way or the other, but it would sure suck if the English SDK only correctly implements a subset of these.</text></comment>