<story><title>Test for lists in Cython</title><url>https://github.com/00sapo/cython_list_test</url></story><parent_chain><item><author>cfcf14</author><text>Have you been successful in implementing non-trivial computational code in numba&amp;#x2F;numpy? I&amp;#x27;ve always found it starts to really break for anything which isn&amp;#x27;t really trivial, and the errors are mostly non-prescriptive and highly verbose.</text></item><item><author>PartiallyTyped</author><text>Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It&amp;#x27;s not caveat free, but it also removes the hustle of labeling arrays as donated in Jax.&lt;p&gt;You may find this interesting &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;scikit-hep&amp;#x2F;iminuit&amp;#x2F;blob&amp;#x2F;develop&amp;#x2F;tutorial&amp;#x2F;automatic_differentiation.ipynb&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;scikit-hep&amp;#x2F;iminuit&amp;#x2F;blob&amp;#x2F;develop&amp;#x2F;tutorial&amp;#x2F;...&lt;/a&gt;</text></item><item><author>jeremiecoullon</author><text>I&amp;#x27;ve been using JAX (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;jax.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;jax.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;&lt;/a&gt;) for scientific computing in general (in particular MCMC algorithms), as it&amp;#x27;s really fast. Even on a CPU you get massive speedups compared to numpy (can be up to 2 or 3 orders of magnitude faster in some cases).&lt;p&gt;The main selling point of the library is automatic differential and compilation to XLA, but I&amp;#x27;ve been using it even when I don&amp;#x27;t need gradients, as it&amp;#x27;s really fast (due to compilation). I also really like the random number generator as it&amp;#x27;s very good for reproducibility.&lt;p&gt;I&amp;#x27;ve played around with Julia in the past and really liked it, but in terms speed Jax has pretty much solved that problem for me</text></item><item><author>ZuLuuuuuu</author><text>I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don&amp;#x27;t have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, especially now that we can easily get an 8 core CPU for a reasonable price.&lt;p&gt;There is multiprocessing module but you cannot share memory between processes. I guess the best options are either writing a C extension or using Numba. Writing C extensions require either distributing binary packages or C compiler to be present on the target computer which is not always ideal. So is using Numba the best solution currently? I tried it a bit in the past but the errors I got was a bit hard to interpret compared to regular Python errors.&lt;p&gt;I wish the multithreading module had support for native threads. Is there any PEPs trying to bring easy multi-core support for Python?</text></item></parent_chain><comment><author>joshuaellinger</author><text>I just implemented both a CSV parser and an address standardizer in numba (both CPU and GPU) running in parallel feed through a message queue with a bunch of workers subprocs.&lt;p&gt;It takes a bit of getting used to but the performance gains on impressive.  Basically, my bottlenecks shift from compute to i&amp;#x2F;o.&lt;p&gt;I think you have to balance it against writing in C&amp;#x2F;C++.  Mentally, it is basically the same work as writing in C (you manage memory&amp;#x2F;you write complicated for-loops) but you have good array support with numpy.  The primary advantage for me that everything stays in the python runtime environment.  You just run the code without any extra steps.&lt;p&gt;...&lt;p&gt;What is missing from the timing type &amp;#x27;toy&amp;#x27; benchmarks is an understanding that there is typically more than one bottleneck in a real problem and it is easy to choose the wrong one to optimize and get little gains.&lt;p&gt;After starting C (30 years ago now), spending a long time in C#, then switching to Python a few years ago, I think the unappreciated advantage of python is that I have to abandon all pretense of caring about speed and just get stuff working.  It basically solves the pre-mature optimization problem for me by being a fast interpreted language rather than a slow compiled language.</text></comment>
<story><title>Test for lists in Cython</title><url>https://github.com/00sapo/cython_list_test</url></story><parent_chain><item><author>cfcf14</author><text>Have you been successful in implementing non-trivial computational code in numba&amp;#x2F;numpy? I&amp;#x27;ve always found it starts to really break for anything which isn&amp;#x27;t really trivial, and the errors are mostly non-prescriptive and highly verbose.</text></item><item><author>PartiallyTyped</author><text>Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It&amp;#x27;s not caveat free, but it also removes the hustle of labeling arrays as donated in Jax.&lt;p&gt;You may find this interesting &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;scikit-hep&amp;#x2F;iminuit&amp;#x2F;blob&amp;#x2F;develop&amp;#x2F;tutorial&amp;#x2F;automatic_differentiation.ipynb&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;scikit-hep&amp;#x2F;iminuit&amp;#x2F;blob&amp;#x2F;develop&amp;#x2F;tutorial&amp;#x2F;...&lt;/a&gt;</text></item><item><author>jeremiecoullon</author><text>I&amp;#x27;ve been using JAX (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;jax.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;jax.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;&lt;/a&gt;) for scientific computing in general (in particular MCMC algorithms), as it&amp;#x27;s really fast. Even on a CPU you get massive speedups compared to numpy (can be up to 2 or 3 orders of magnitude faster in some cases).&lt;p&gt;The main selling point of the library is automatic differential and compilation to XLA, but I&amp;#x27;ve been using it even when I don&amp;#x27;t need gradients, as it&amp;#x27;s really fast (due to compilation). I also really like the random number generator as it&amp;#x27;s very good for reproducibility.&lt;p&gt;I&amp;#x27;ve played around with Julia in the past and really liked it, but in terms speed Jax has pretty much solved that problem for me</text></item><item><author>ZuLuuuuuu</author><text>I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don&amp;#x27;t have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, especially now that we can easily get an 8 core CPU for a reasonable price.&lt;p&gt;There is multiprocessing module but you cannot share memory between processes. I guess the best options are either writing a C extension or using Numba. Writing C extensions require either distributing binary packages or C compiler to be present on the target computer which is not always ideal. So is using Numba the best solution currently? I tried it a bit in the past but the errors I got was a bit hard to interpret compared to regular Python errors.&lt;p&gt;I wish the multithreading module had support for native threads. Is there any PEPs trying to bring easy multi-core support for Python?</text></item></parent_chain><comment><author>grej</author><text>We have a very large portion of production code written in numba we’ve been running for about 3 years and I made a small contribution to the library. There are a lot of gotchas to numba when the codebase gets large but the benefits far outweigh the downsides. I highly recommend numba.&lt;p&gt;Edit: also note that a big part of the umap library is written in numba.</text></comment>