Helping Pandas, sans Pandas, with one step more

A long carried question of mine has been this - how many Pandas does this push trouble? Lacking context, this does sound as bizarre as most of my questions. A good parallel for the context would be Rasmus Lerdorf's pitch for PHP7.

That the carbon footprints of poor code runs go amiss does trouble one. A certain amount of frugal lifestyle has only made the concern all the more pertinent. I didn't start programming in the era of the 64Ks. But, most production systems I've worked with, were a restricted case to say the least.

Natural indeed then it is to ask, "where do we shave the fat off from?". One ensuing question thereof is going to be one of the impact it has on the larger footprint. From slimming the burn rate to burning less to cool the servers, the impacts are solid indeed.

In the same vein, the rewrite of one of my data pipelines opened a lot. The first changes retained Python. The increments pushed to C++. And finally, C made a sea of a difference, to say the least. From sharing to memory, appreciation for tiniest of concerns only doubled with C.

While at this, I'd suggest the original Unix explainer to every developer. The sub 14.4k binary now takes a radical diversion from the road thus far taken. In splitting concerns to the lowest of levels, the binary takes away every blob of fat off, deals with its core. One of the interesting outcomes I must has been how consistent this one has been, in performance. Not to mention, how it showed the mirror to the line of needs and wants.

On the first, the runtime reduction is significant; sub 1s from a sub minute system. I do smirk once in a while, looking back at the past. When my first improvements to the Python stack happened in 2016/17, was wondering about CRON. That CRON is granular to the minute meant, the new pipeline would wait twice its runtime before it can run again. How the turn tables, the new pipeline says, waiting 59x, over the previous winner.

Yet another take away has been one of needs and wants. Now, this is something that I've had different thoughts about for long enough. Contrary to most talks, I've sealed enough in concrete a certain view of mine. That over-beefed-up specs encourage inefficiency is stronger than ever, even without empirical proofs.

The original rewrite happened on a 4th Gen Intel (current being 11th, for perspective) with an SSD. By all bounds of fairness, an old enough one compared to the flashy ones advocated by most workplaces. And that's where another itch was also persistent. The consistent unbelievable numbers, are they placebo after all?

Thanks to Murphy, did get an answer to it last evening. Over the last week, both the one mentioned above and my backup notebook conked, forcing me to be off code for a bit. In some odd mood, fired up Dad's notebook lying around, for one reason; interviews in case. As always, ended up doing everything but that; revisiting the pipeline in particular.

The biggest surprise was when I ran it on a Celeron N2840/4G/HDD. The numbers above are an average show from this old as Methuselah box, running Xubuntu live.

One major take away there, that a million rounds of studying didn't do but this work did. Accommodating problems to small enough a size for computing units to load, does win a lot. There might still be a lot of scope to narrow this down (have a few penned). But, may be I've hit the point of inflection for now, so on to other problems at hand to solve.