Slot Wrapper Python

Register here: http://gg.gg/w6x32
There are a large number of structures which are used in the definition ofobject types for Python. This section describes these structures and how theyare used.
All Python objects ultimately share a small number of fields at the beginningof the object’s representation in memory. These are represented by thePyObject and PyVarObject types, which are defined, in turn,by the expansions of some macros also used, whether directly or indirectly, inthe definition of all other Python objects.PyObject¶
All object types are extensions of this type. This is a type whichcontains the information Python needs to treat a pointer to an object as anobject. In a normal “release” build, it contains only the object’sreference count and a pointer to the corresponding type object.Nothing is actually declared to be a PyObject, but every pointerto a Python object can be cast to a PyObject*. Access to themembers must be done by using the macros Py_REFCNT andPy_TYPE.PyVarObject¶
So if there is a built-in base class that is not object it fails, even if it’s still wrapping the standard implementation of that slot. I would consider this a bug in Python-it shouldn’t be doing identity comparison on the slot wrapper object itself.
*Since slot wrappers are loaded before the method table, the existence of a sqcontains slot, for example, would generate a wrapped method named contains and preclude the loading of a corresponding PyCFunction with the same name. With the flag defined, the PyCFunction will be loaded in place of the wrapper object and will co-exist with the.
*This difference is an implementation detail that shows up because of internal C-level slots that PyPy does not have. On CPython,.add is a method-wrapper, and list.add is a slot wrapper. On PyPy these are normal bound or unbound method objects. This can occasionally confuse some tools that inspect built-in types.
This is an extension of PyObject that adds the ob_sizefield. This is only used for objects that have some notion of length.This type does not often appear in the Python/C API.Access to the members must be done by using the macrosPy_REFCNT, Py_TYPE, and Py_SIZE.PyObject_HEAD¶
This is a macro used when declaring new types which represent objectswithout a varying length. The PyObject_HEAD macro expands to:
See documentation of PyObject above.PyObject_VAR_HEAD¶
This is a macro used when declaring new types which represent objectswith a length that varies from instance to instance.The PyObject_VAR_HEAD macro expands to:
See documentation of PyVarObject above.Py_TYPE(o)¶
This macro is used to access the ob_type member of a Python object.It expands to:Py_REFCNT(o)¶
This macro is used to access the ob_refcnt member of a Pythonobject.It expands to:Py_SIZE(o)¶
This macro is used to access the ob_size member of a Python object.It expands to:PyObject_HEAD_INIT(type)¶
This is a macro which expands to initialization values for a newPyObject type. This macro expands to:PyVarObject_HEAD_INIT(type, size)¶
This is a macro which expands to initialization values for a newPyVarObject type, including the ob_size field.This macro expands to:PyCFunction¶
Type of the functions used to implement most Python callables in C.Functions of this type take two PyObject* parameters and returnone such value. If the return value is NULL, an exception shall havebeen set. If not NULL, the return value is interpreted as the returnvalue of the function as exposed in Python. The function must return a newreference.Python Wrapper ExamplePyCFunctionWithKeywords¶
Type of the functions used to implement Python callables in C that takekeyword arguments: they take three PyObject* parameters and returnone such value. See PyCFunction above for the meaning of the returnvalue.PyMethodDef¶
Structure used to describe a method of an extension type. This structure hasfour fields:FieldC TypeMeaningml_namechar *name of the methodml_methPyCFunctionpointer to the Cimplementationml_flagsintflag bits indicating how thecall should be constructedml_docchar *points to the contents of thedocstring
The ml_meth is a C function pointer. The functions may be of differenttypes, but they always return PyObject*. If the function is not ofthe PyCFunction, the compiler will require a cast in the method table.Even though PyCFunction defines the first parameter asPyObject*, it is common that the method implementation uses thespecific C type of the self object.
The ml_flags field is a bitfield which can include the following flags.The individual flags indicate either a calling convention or a bindingconvention. Of the calling convention flags, only METH_VARARGS andMETH_KEYWORDS can be combined. Any of the calling convention flagscan be combined with a binding flag.METH_VARARGS¶
This is the typical calling convention, where the methods have the typePyCFunction. The function expects two PyObject* values.The first one is the self object for methods; for module functions, it isthe module object. The second parameter (often called args) is a tupleobject representing all arguments. This parameter is typically processedusing PyArg_ParseTuple() or PyArg_UnpackTuple().METH_KEYWORDS¶
Methods with these flags must be of type PyCFunctionWithKeywords.The function expects three parameters: self, args, and a dictionary ofall the keyword arguments. The flag must be combined withMETH_VARARGS, and the parameters are typically processed usingPyArg_ParseTupleAndKeywords().METH_NOARGS¶
Methods without parameters don’t need to check whether arguments are given ifthey are listed with the METH_NOARGS flag. They need to be of typePyCFunction. The first parameter is typically named self and willhold a reference to the module or object instance. In all cases the secondparameter will be NULL.METH_O¶
Methods with a single object argument can be listed with the METH_Oflag, instead of invoking PyArg_ParseTuple() with a ’O’ argument.They have the type PyCFunction, with the self parameter, and aPyObject* parameter representing the single argument.
These two constants are not used to indicate the calling convention but thebinding when use with methods of classes. These may not be used for functionsdefined for modules. At most one of these flags may be set for any givenmethod.METH_CLASS¶
The method will be passed the type object as the first parameter ratherthan an instance of the type. This is used to create class methods,similar to what is created when using the classmethod() built-infunction.METH_STATIC¶
The method will be passed NULL as the first parameter rather than aninstance of the type. This is used to create static methods, similar towhat is created when using the staticmethod() built-in function.
One other constant controls whether a method is loaded in place of anotherdefinition with the same method name.METH_COEXIST¶
The method will be loaded in place of existing definitions. WithoutMETH_COEXIST, the default is to skip repeated definitions. Since slotwrappers are loaded before the method table, the existence of asq_contains slot, for example, would generate a wrapped method named__contains__() and preclude the loading of a correspondingPyCFunction with the same name. With the flag defined, the PyCFunctionwill be loaded in place of the wrapper object and will co-exist with theslot. This is helpful because calls to PyCFunctions are optimized morethan wrapper object calls.PyMemberDef¶
Structure which describes an attribute of a type which corresponds to a Cstruct member. Its fields are:FieldC TypeMeaningnamechar *name of the membertypeintthe type of the member in theC structoffsetPy_ssize_tthe offset in bytes that themember is located on thetype’s object structflagsintflag bits indicating if thefield should be read-only orwritabledocchar *points to the contents of thedocstring
type can be one of many T_ macros corresponding to various Ctypes. When the member is accessed in Python, it will be converted to theequivalent Python type.Macro nameC typeT_SHORTshortT_INTintT_LONGlongT_FLOATfloatT_DOUBLEdoubleT_STRINGchar *T_OBJECTPyObject *T_OBJECT_EXPyObject *T_CHARcharT_BYTEcharT_UBYTEunsigned charT_UINTunsigned intT_USHORTunsigned shortT_ULONGunsigned longT_BOOLcharT_LONGLONGlong longT_ULONGLONGunsigned long longT_PYSSIZETPy_ssize_t
T_OBJECT and T_OBJECT_EX differ in thatT_OBJECT returns None if the member is NULL andT_OBJECT_EX raises an AttributeError. Try to useT_OBJECT_EX over T_OBJECT because T_OBJECT_EXhandles use of the del statement on that attribute more correctlythan T_OBJECT.
flags can be 0 for write and read access or READONLY forread-only access. Using T_STRING for type impliesREADONLY. Only T_OBJECT and T_OBJECT_EXmembers can be deleted. (They are set to NULL).PyGetSetDef¶
Structure to define property-like access for a type. See also description ofthe PyTypeObject.tp_getset slot.FieldC TypeMeaningnamechar *attribute namegetgetterC Function to get the attributesetsetteroptional C function to set ordelete the attribute, if omittedthe attribute is readonlydocchar *optional docstringclosurevoid *optional function pointer,providing additional data forgetter and setter
The get function takes one PyObject* parameter (theinstance) and a function pointer (the associated closure):
It should return a new reference on success or NULL with a set exceptionon failure.
set functions take two PyObject* parameters (the instance andthe value to be set) and a function pointer (the associated closure):
In case the attribute should be deleted the second parameter is NULL.Should return 0 on success or -1 with a set exception on failure.Overview¶
This document describes the guts of jpype. It is intendedlay out the architecture of the jpype code to aid intrepid lurkersto develop and debug the jpype code once I am run over by a bus.For most of this document I will use the royal we, except whereI am giving personal opinions expressed only by yours truly, theauthor Thrameos.History¶
When I started work on this project it had already existed for over 10 years.The original developer had intended a much larger design with modules tosupport multiple languages such as Ruby. As such it was constructed withthree layers of abstraction. It has a wrapper layer over Java in C++, awrapper layer for the Python api in C++, and an abstraction layer intendedto bridge Python and other interpreted languages. This multilayer abstractionment that every debugging call had to drop through all of those layers.Memory management was split into multiple pieces with Java controlling aportion of it, C++ holding a bunch of resources, Python holding additionalresources, and HostRef controlling the lifetime of objects shared between thelayers. It also had its own reference counting system for handing Javareferences on a local scale.
This level of complexity was just about enough to scare off all but the mosthardened programmer. Thus I set out to eliminate as much of this as I could.Java already has its own local referencing system to deal in the form ofLocalFrames. It was simply a matter of setting up a C++ object tohold the scope of the frames to eliminate that layer. The Java abstractionwas laid out in a fashion somewhat orthagonally to the Java inheritancediagram. Thus that was reworked to something more in line which could besafely completed without disturbing other layers. The multilanguageabstraction layer was already pierced in multiple ways for speed. However,as the abastraction interwove throughout all the library it was a terriblelift to remove and thus required gutting the Python layer as well to supportthe operations that were being performed by the HostRef.
The remaining codebase is fairly slim and reasonably streamlined. Thisrework cut out about 30% of the existing code and sped up the internaloperations. The Java C++ interface matches the Java class hierachy.Architecture¶
JPype is split into several distinct pieces.jpype Python moduleThe majority of the front end logic for the toolkit is in Python jpype module.This module deals with the construction of class wrappers and control functions.The classes in the layer are all prefixed by J._jpype CPython module
The native module is supported by a CPython module called _jpype. The _jpypemodule is located in native/python and has C style classes with a prefix PyJP.
This CPython layer acts as a front end for passing to the C++ layer.It performs some error checking. In addition to the module functions in_JModule, the module has multiple Python classes to support the native jpypecode such as _JClass, _JArray, _JValue, _JValue, etc.CPython API wrapperIn addition to the exposed Python module layer, there is also a C++ wrapperfor the Python API. This is located in native/python and has the prefixJPPy for all classes. jp_pythontypes wraps the required parts ofthe CPython API in C++ for use in the C++ layer.C++ JNI layerThe guts that drive Java are in the C++ layer located in native/common. This layerhas the namespace JP. The code is divided into wrappers for each Java type,a typemanager for mapping from Java names to class instances, support classesfor proxies, and a thin JNI layer used to help ensure rigerous use of the samedesign patterns in the code. The primary responsibility of this layer istype conversion and matching of method overloads.Java layerIn addition to the C++ layer, jpype has a native Java layer. This codeis compiled as a “thunk” which is loaded into the JVM in the form of aa binary stored as a string. Code for Java is found in native/java.The Java layer is divided into two parts,a bootstrap loader and a jar containing the support classes. The Javalayer is responsible managing the lifetime of shared Python, Java, and C++ objects.jpype module¶
The jpype module itself is made of a series of support classes whichact as factories for the individual wrappers that are created to mirroreach Java class. Because it is not possible to wrap all Java classeswith staticly created wrappers, instead jpype dynamically createsPython wrappers as requested by the user.
The wrapping process is triggered in two ways. The user can manuallyrequest creating a class by importing a class wrapper with jpype.importsor JPackage or by manually invoking it with JClass. Or the class wrappercan be created automatically as a result of a return type or exceptionthrown to the user.
Because the classes are created dynamically, the class structureuses a lot of Python meta programming.Each class wrapper derives from the class wrappers of each of thewrappers corresponding to the Java classes that each class extendsand implements. The key to this is to hacked mro. The mroorders each of the classes in the tree such that the most drivedclass methods are exposed, followed by each parent class. Thismust be ordered to break ties resulting from multiple inheritanceof interfaces. The factory classes are grafted into the type systemusing __instancecheck__ and __subtypecheck__.resource types¶
JPype largely maps to the same concepts as Python with a few special elements.The key concept is that of a Factory which serves to create Java resourcesdynamically as requested. For example there is no Python notation tocreate a int[][] as the concept of dimensions are fluid in Python.Thus a factory type creates the actual object instance type withJArray(JInt,2) Like Python objects, Java objects derives from atype object which is called JClass that serves as a meta type forall Java derived resources. Additional type like object JArrayand JInterface serve to probe the relationships between types.Java object instances are created by calling the Java class wrapper justlike a normal Python class. A number of pseudo classes serve as placeholdersfor Java types so that it is not necessary to create the type instancewhen using. These aliased classes are JObject, JString, andJException. Underlying all Java instances is the concept of ajvalue.jvalue¶
In the earlier design, wrappers, primitives and objects were all seperateconcepts. At the JNI layer these are unified by a common element calledjvalue. A jvalue is a union of all primitives with the jobject. The jobjectcan represent anything derived from Java object including the pseudo classjstring.
This has been replaced with a Java slot concept which holds an instance ofJPValue which holds a pointer to the C++ Java type wrapper and a Javajvalue union. We will discuss this object further in the CPython section.Bootstrapping¶
The most challenging part in working with the jpype module other than theneed to support both major Python versions with the same codebase is thebootstrapping of resources. In order to get the system working, we must passthe Python resources so the _jpype CPython module can acquire resources and thenconstruct the wrappers for java.lang.Object and java.lang.Class. The keydifficulty is that we need reflection to get methods from Java and thoseare part of java.lang.Class, but class inherits from java.lang.Object.Thus Object and the interfaces that Class inherits must all be createdblindly. The order of bootstrapping is controlled by specific sequenceof boot actions after the JVM is started in startJVM. The class instanceclass_ may not be accessed until after all of the basic class, object,and exception types have been loaded.Factories¶
The key objects exposed to the user (JClass, JObject, and JArray) are eachfactory meta classes. These classes serve as the gate keepers to creating themeta classes or object instances. These factories inherit from the Java class metaand have a class_ instance inserted after the the JVM is started. They do nothave exposed methods as they are shadows for action for actual Java types.
The user calls with the specified arguments to create a resource. The factorycalls the __new__ method when creating an instance of the derived object. Andthe C++ wrapper calls the method with internally construct resource such as_JClass or _JValue. Most of the internal calls currently create theresource directly without calling the factories. The gateway for this isPyJPValue_create which delegates the process to the corresponding specializedtype.Style¶
One of the aspects of the jpype design is elegance of the factory patterns.Rather than expose the user a large number of distinct concepts with differentnames, the factories provide powerfull functionality with the same syntax forrelated things. Boxing a primitive, casting to a specific type, and creatinga new object are all tied together in one factory, JObject. By also making thatfactory an effective base class, we allow it to be used for issubtype andisinstance.
This philosophy is further enhanced by silent customizers which integratePython functionality into the wrappers such that Java classes can be usedeffectively with Python syntax. Consistent use and misuse of Python conceptssuch as with for defining blocks such as try with resources and synchronizedhide the underlying complexity and give the feeling to the user that themodule is integrated completely as a solution such as jython.
When adding a new feature to the Python laye

https://diarynote.indered.space