kronosapiens.github.io - Creating superfunctions in Python

Creating superfunctions in Python

Kronosapiens.github.io Spined HTML

Creating superfunctions in Python AbacusWell-nighCreating superfunctions in Python Jul 16, 2014 Last week I ran into an interesting problem while towers out the ParagonMeasure backend. Much of the work of the past two months has involved the diamond and implementation of a data wringer library to parse, organize, and query subject typing data. The library is nearly mature, in the sense that we can now go from a raw CSV file to a full-length matrix comparing multiple subjects over time. The problem that I’m grappling with now is the problem of integrating the higher-level functionality into our increasingly exploratory tools. While towers the library, I wrote a number of functions to indulge us to hands compare various data series, experimen with various parameters and filters, and get a sense of what our data looked like. Those functions, though, were written virtually whatever modules I had managed to well-constructed at the time, with little thought to forwards-compatibility. Inevitably, as new and increasingly ramified functions were written and our ways for representing our data evolved, the older exploratory functions stopped working with the new data representations. These functions were deprecated, new functions were written to work with the new modules, and the library kept growing. Eventually, we arrived at a structure in which we encapsulated all of the knowledge needed to summate a specific full-length out of a larger dataset into something tabbed a “Feature Function”, a function capable of taking in some intermediate data structure, filtering it, analyzing it, aggregating it, and packaging it for turnout into a full-length matrix.Subtractingfeatures to our wringer became a matter of writing a newFull-lengthFunction equal to the spec, and subtracting that function to feature_matrix(). I was pleased with this convention. There are a number of steps that need to occur, in order, to go from raw data to a full feature_matrix. One of the toughest things well-nigh writing this library was keeping track of where in the process each function was expected to live, in order to know how that function needed to receive input and package output to play nicely with the other functions. TheFull-lengthFunction institute meant that every new full-length could start from the same place, perform some unique analysis, and package the results into a standard format. It worked, for a while. But a rencontre emerged when I was asked to go when to some older exploratory functions and use them to generate some new graphs, using the new data. TheFull-lengthFunctions were written to exist in a particular place in the tampering flow, and now I was stuff asked to generate plots of wrong-headed full-length functions with wrong-headed parameters; in essence, to use an API which I hadn’t built. The easiest thing to do would have been to gut the full-length functions and indistinguishable a lot of their policies elsewhere, in a new environment where I could tenancy the parameters that I needed to to generate the exploratory plots. But I desperately wanted to stave this. I find duplicating lawmaking to be upsetting; it’s a form of technical debt that I can scrutinizingly never justify. The second-easiest thing to do would have been to write new exploratory functions which wonted an egregious number of substantially redundant arguments, in order for them to re-create the environment that theFull-lengthFunctions expect. That would have been suboptimal, as the exploratory functions (designed to facilitate speedy development) error-prone and unpleasant to use. The third option was to make theFull-lengthFunctions very unstipulated – let them winnow substantially wrong-headed data, and requite them the worthiness to refine the data themselves. This solution would have been terrible from an efficiency and optimization perspective, resulting in an unacceptable value of indistinguishable work. Whatever my solution was, it needed to make theFull-lengthFunctions increasingly hands plugged into exploratory functions, without sacrificing the efficiency of their primary use. I wanted a DRY solution that was easy to use – I wanted to be worldly-wise to pass myFull-lengthFunctions into wrong-headed new functions, and have them just know what to do. I realized that I wanted theseFull-lengthFunctions to have knowledge well-nigh their own requirements (beyond the policies they define), so that functions to which they are passed can recreate the environment that theFull-lengthFunctions expect to be executed in. Enter function attributes. Python allows you to assign wrong-headed nature to a function, in the same way that you can assign nature to any object (which is unsurprising – functions are first-class citizens, objects, in Python). My preferred syntax looks something like this: def PAOFFSET(unigrams, df=True, agg=True, metrics=['O0']): unigrams = clean(unigrams, metrics) wringer = unigrams['ND_Offset'] ... return wringer PAOFFSET.q = 1 PAOFFSET.trait = 'ND_Offset' In this case, the PAOFFSET function has been given two attributes. The first, q, represents the size of the QGram that this function expects to recieves as an treatise (in this case, one, creating unigrams). The second, ND_Offset, refers to the post in the data that this function is designed to analyze. Regarding syntax, you’ll note that the function nature are specified without the tropical of the function definition. This is considering the nature are not executed in any way with the function, but are rather properties of the function as an object. So, they are prescribed to the function object at the soonest possible point – immediately without the function is defined. Normally, this function would be tabbed only without the subject’s data had been processed to pericope the desired rows. With these attributes, however, it is now possible to waif PAOFFSET into some wrong-headed function and provide that function with the knowledge to recreate the environment that PAOFFSET expects. This is a good solution, to my mind, considering it keeps PAOFFSET optimized for it’s main use inside the feature_matrix() function, while permitting for unexpected and exploratory uses of that same feature. An spare feature, much increasingly mainstream, was the wing of a number of new keyword arguments (or kwargs**!!!!!!!, as the tomfool kids undeniability them). These kwargs substantially vivify or deactivate parts of theFull-lengthFunction to indulge for increasingly tenancy over the output (allowing for variegated levels of indexing and team for variegated circumstances). So. Function nature + keyword arguments = superfunction. You might think for a moment that this sounds a lot like the concept of a Class. Nope. Superfunction. Comments Please enable JavaScript to view the comments powered by Disqus. Abacus Abacus kronovet@gmail.com kronosapiens kronosapiens I'm Daniel Kronovet, a data scientist living in Tel Aviv.