diff --git a/.gitattributes b/.gitattributes index afd4cde..d015960 100644 --- a/.gitattributes +++ b/.gitattributes @@ -2,4 +2,4 @@ *.less linguist-language=Python *.js linguist-language=Python *.html linguist-language=Python -*.md linguist-language=Python +*.md linguist-language=Markdown diff --git a/BasicObject/bytearray/bytearray.md b/BasicObject/bytearray/bytearray.md index fcdbe56..4dae9b2 100644 --- a/BasicObject/bytearray/bytearray.md +++ b/BasicObject/bytearray/bytearray.md @@ -46,7 +46,7 @@ The **ob_alloc** field represents the real allocated size in bytes ## append -after append a charracter 'a', **ob_alloc** becomes 2, **ob_bytes** and **ob_start** all points to same address +After appending a character 'a', **ob_alloc** becomes 2, and **ob_bytes** and **ob_start** both point to the same address ```python3 a.append(ord('a')) @@ -57,7 +57,7 @@ a.append(ord('a')) ## resize -The size grow pattern is shown in the code +The size growth pattern is shown in the code ```python3 /* Need growing, decide on a strategy */ @@ -72,7 +72,7 @@ The size grow pattern is shown in the code ``` -In appending, ob_alloc is 2, and request size is 2, 2 <= 2 * 1.125, so the new allocated size is 2 + (2 >> 3) + 3 ==> 5 +When appending, ob_alloc is 2 and the requested size is 2. Since 2 <= 2 * 1.125, the new allocated size is 2 + (2 >> 3) + 3 ==> 5 ```python3 a.append(ord('b')) @@ -99,7 +99,7 @@ b[0:5] = [1,2] ![after_slice](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytearray/after_slice.png) -as long as the slice operation is going to shrink the bytearray, and the **new_size < allocate / 2** is False, the resize operation won't shrink the real malloced size +As long as the slice operation is going to shrink the bytearray and **new_size < allocate / 2** is False, the resize operation won't shrink the actual malloced size ```python3 b[2:6] = [3, 4] @@ -108,7 +108,7 @@ b[2:6] = [3, 4] ![after2_slice](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytearray/after2_slice.png) -now, in the shrink operation, the **new_size < allocate / 2** is True, the resize operation will be triggered +Now, in the shrink operation, **new_size < allocate / 2** is True, so the resize operation will be triggered ```python3 b[0:3] = [7,8] @@ -119,7 +119,7 @@ b[0:3] = [7,8] The growing pattern in slice operation is the same as the append operation -request size is 6, 6 < 6 * 1.125, so new allocated size is 6 + (6 >> 3) + 3 ==> 9 +The requested size is 6. Since 6 < 6 * 1.125, the new allocated size is 6 + (6 >> 3) + 3 ==> 9 ```python3 b[0:3] = [1,2,3,4] @@ -130,7 +130,7 @@ b[0:3] = [1,2,3,4] ## ob_exports -what's field **ob_exports** mean ? If you need detail, you can refer to [less-copies-in-python-with-the-buffer-protocol-and-memoryviews](https://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews) and [PEP 3118](https://www.python.org/dev/peps/pep-3118/) +What does the field **ob_exports** mean? If you need details, you can refer to [less-copies-in-python-with-the-buffer-protocol-and-memoryviews](https://eli.thegreenplace.net/2011/11/28/less-copies-in-python-with-the-buffer-protocol-and-memoryviews) and [PEP 3118](https://www.python.org/dev/peps/pep-3118/) ```python3 buf = bytearray(b"abcdefg") @@ -139,9 +139,9 @@ buf = bytearray(b"abcdefg") ![exports](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytearray/exports.png) -the **bytearray** implements the **buffer protocol**, and **memoryview** is able to access the internal data block via the **buffer protocol**, **mybuf** and **buf** are all sharing the same internal block +The **bytearray** implements the **buffer protocol**, and **memoryview** is able to access the internal data block via the **buffer protocol**. **mybuf** and **buf** both share the same internal block. -field **ob_exports** becomes 1, which indicate how many objects currently sharing the internal block via **buffer protocol** +The field **ob_exports** becomes 1, which indicates how many objects are currently sharing the internal block via the **buffer protocol** ```python3 mybuf = memoryview(buf) @@ -151,7 +151,7 @@ mybuf[1] = 3 ![exports_1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytearray/exports_1.png) -so does **mybuf2** object(**ob_exports** doesn't change because you need to call the c function defined by **buf** object via the **buffer protocol**, **mybuf2** barely calls the slice function of **mybuf**) +The same applies to the **mybuf2** object (**ob_exports** doesn't change because you need to call the C function defined by the **buf** object via the **buffer protocol**; **mybuf2** only calls the slice function of **mybuf**) ```python3 mybuf2 = mybuf[:4] diff --git a/BasicObject/bytes/bytes.md b/BasicObject/bytes/bytes.md index 2252792..5170087 100644 --- a/BasicObject/bytes/bytes.md +++ b/BasicObject/bytes/bytes.md @@ -39,7 +39,7 @@ s = b"" ## ascii characters -let's initialize a byte object with ascii characters +Let's initialize a bytes object with ASCII characters ```python3 s = b"abcdefg123" @@ -65,17 +65,17 @@ s = "我是帅哥".encode("utf8") The field **ob_shash** should store the hash value of the byte object, value **-1** means not computed yet. -The first time the hash value computed, it will be cached in the **ob_shash** field +The first time the hash value is computed, it will be cached in the **ob_shash** field. -the cached hash value can save recalculation and speeds up dictionary lookups +The cached hash value can save recalculation and speed up dictionary lookups ## ob_size -field **ob_size** is inside every **PyVarObject**, the **PyBytesObject** uses this **field** to store size information to keep O(1) time complexity for **len()** operation and tracks the size of non-ascii string(may be null characters inside) +The field **ob_size** is inside every **PyVarObject**. **PyBytesObject** uses this field to store size information to keep O(1) time complexity for the **len()** operation and to track the size of non-ASCII strings (which may contain null characters inside) ## summary -The **PyBytesObject** is a python wrapper of c style null terminate string, with **ob_shash** for caching hash value and **ob_size** for storing the size information of **PyBytesObject** +The **PyBytesObject** is a Python wrapper of C-style null-terminated strings, with **ob_shash** for caching the hash value and **ob_size** for storing size information The implementation of **PyBytesObject** looks like the **embstr** encoding in redis diff --git a/BasicObject/class/class.md b/BasicObject/class/class.md index 59cc275..26cc4d0 100644 --- a/BasicObject/class/class.md +++ b/BasicObject/class/class.md @@ -18,7 +18,7 @@ # memory layout -the **PyMethodObject** represents the type **method** in c-level +**PyMethodObject** represents the type **method** at the C level ```python3 class C(object): @@ -35,7 +35,7 @@ class C(object): # fields -the layout of **c.f1** +The layout of **c.f1** ![example0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/example0.png) @@ -51,7 +51,7 @@ as you can see from the layout, field **im_func** stores the [function](https:// ## im_self -field **im_self** stores the instance object this method bound to +The field **im_self** stores the instance object this method is bound to ```python3 >>> c @@ -67,7 +67,7 @@ when you call ``` -the **PyMethodObject** delegate the real call to **im_func** with **im_self** as the first argument +**PyMethodObject** delegates the real call to **im_func** with **im_self** as the first argument ```c static PyObject * @@ -99,11 +99,11 @@ static int numfree = 0; ``` -free_list is a single linked list, it's used for **PyMethodObject** to safe malloc/free overhead +free_list is a singly linked list. It's used for **PyMethodObject** to save malloc/free overhead. -**im_self** field is used to chain the element +The **im_self** field is used to chain the elements. -the **PyMethodObject** will be created when you trying to access the bound-method, not when the instance is created +The **PyMethodObject** will be created when you try to access the bound-method, not when the instance is created ```python3 >>> c1 = C() @@ -121,7 +121,7 @@ the **PyMethodObject** will be created when you trying to access the bound-metho ``` -now, let's see an example of free_list +Now let's see an example of free_list ```python3 >>> c1_f1_1 = c1.f1 @@ -156,7 +156,7 @@ assume the free_list is empty now # classmethod and staticmethod -let's define an object with **classmethod** and **staticmethod** +Let's define an object with **classmethod** and **staticmethod** ```python3 class C(object): @@ -193,7 +193,7 @@ the **@classmethod** keeps type of **c1.fc** as **method** ![classmethod](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/classmethod.png) -how **classmethod** work internally? +How does **classmethod** work internally? **classmethod** is a **type** in python3 @@ -208,7 +208,7 @@ typedef struct { ![classmethod1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/classmethod1.png) -let's see what's under the hood +Let's see what's under the hood ```python3 fc = classmethod(lambda self : self) @@ -228,9 +228,9 @@ class C(object): ``` -get a different result when access the same object in a different way, why? +We get a different result when accessing the same object in a different way. Why? -when you trying to access the **fc1** in instance cc, the **descriptor protocol** will try several different paths to get the attribute in the following step +When you try to access **fc1** in instance cc, the **descriptor protocol** will try several different paths to get the attribute in the following steps * call `__getattribute__` of the object C * `C.__dict__["fc1"]` is a data descriptor? * yes, return `C.__dict__['fc1'].__get__(instance, Class)` @@ -246,7 +246,7 @@ for more detail, please refer to this blog [object-attribute-lookup](https://blo because **classmethod** implements `__get__` and `__set__`, it's a data descriptor, when you try to access attribute **cc.fc1**, you will actually call `fc1.__get__`, and caller will get whatever it returns -we can see the `__get__` function of classmethod object(defined as `cm_descr_get` in C) +We can see the `__get__` function of the classmethod object (defined as `cm_descr_get` in C) ```c static PyObject * @@ -268,7 +268,7 @@ cm_descr_get(PyObject *self, PyObject *obj, PyObject *type) ![classmethod_get](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/classmethod_get.png) -when you access fc1 by **cc.fc1**, the **descriptor protocol** will call the function above, which returns whatever in the **cm_callable**, wrapped by **PyMethod_New()** function, which makes the return object a new bounded-PyMethodObject +When you access fc1 via **cc.fc1**, the **descriptor protocol** will call the function above, which returns whatever is in **cm_callable**, wrapped by the **PyMethod_New()** function, which makes the returned object a new bound PyMethodObject ## staticmethod @@ -291,7 +291,7 @@ typedef struct { ``` -this is the layout of **staticmethod** object +This is the layout of the **staticmethod** object ![staticmethod1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/staticmethod1.png) @@ -315,7 +315,7 @@ class C(object): ![staticmethod2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/staticmethod2.png) -we can see the `__get__` function of staticmethod object +We can see the `__get__` function of the staticmethod object ```c static PyObject * diff --git a/BasicObject/complex/complex.md b/BasicObject/complex/complex.md index 79f1192..7f1ca0d 100644 --- a/BasicObject/complex/complex.md +++ b/BasicObject/complex/complex.md @@ -13,9 +13,9 @@ # memory layout -the **PyComplexObject** stores two double precision floating point number inside +**PyComplexObject** stores two double-precision floating-point numbers inside. -the handling process and representation are mostly the same as [float](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/float.md) object +The handling process and representation are mostly the same as the [float](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/float.md) object ![layout](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/complex/layout.png) @@ -35,7 +35,7 @@ d = complex(0.1, -0.2) ![example1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/complex/example1.png) -let's read the add function +Let's read the add function ```c Py_complex @@ -62,9 +62,9 @@ complex_add(PyObject *v, PyObject *w) ``` -the add operation is quite simple, sum the **real**, sum the **image** part and return the new value +The add operation is quite simple: sum the **real** parts, sum the **imag** parts, and return the new value. -the sub/divide/pow/neg operations are similar +The sub/divide/pow/neg operations are similar ```python3 >>> e = c + d diff --git a/BasicObject/dict/5_5.png b/BasicObject/dict/5_5.png deleted file mode 100644 index 4fb00c9..0000000 Binary files a/BasicObject/dict/5_5.png and /dev/null differ diff --git a/BasicObject/dict/PyDictObject.png b/BasicObject/dict/PyDictObject.png new file mode 100644 index 0000000..5e76487 Binary files /dev/null and b/BasicObject/dict/PyDictObject.png differ diff --git a/BasicObject/dict/after_py36.png b/BasicObject/dict/after_py36.png new file mode 100644 index 0000000..62a9591 Binary files /dev/null and b/BasicObject/dict/after_py36.png differ diff --git a/BasicObject/dict/after_py36_search.png b/BasicObject/dict/after_py36_search.png new file mode 100644 index 0000000..4d797a7 Binary files /dev/null and b/BasicObject/dict/after_py36_search.png differ diff --git a/BasicObject/dict/after_py36_space.png b/BasicObject/dict/after_py36_space.png new file mode 100644 index 0000000..eb42c57 Binary files /dev/null and b/BasicObject/dict/after_py36_space.png differ diff --git a/BasicObject/dict/before_py36.png b/BasicObject/dict/before_py36.png new file mode 100644 index 0000000..bfd4252 Binary files /dev/null and b/BasicObject/dict/before_py36.png differ diff --git a/BasicObject/dict/dict.md b/BasicObject/dict/dict.md index b33b829..2ba2273 100644 --- a/BasicObject/dict/dict.md +++ b/BasicObject/dict/dict.md @@ -1,12 +1,13 @@ # dict -# contents +# Contents -because the **PyDictObject** is a little bit more complicated than other basic objects, I will not show `__setitem__`/`__getitem__` step by step, instead, I will illustrate in the middle of some concept +[Related File](#related-file) + +[Evolvement and Implementation](#evolvement-and-implementation) -* [related file](#related-file) * [memory layout](#memory-layout) - * [combined table && split table](#combined-table-&&-split-table) + * [combined table and split table](#combined-table-and-split-table) * [indices and entries](#indices-and-entries) * [hash collisions and delete](#hash-collisions-and-delete) * [resize](#resize) @@ -16,39 +17,54 @@ because the **PyDictObject** is a little bit more complicated than other basic o * [why mark as DKIX_DUMMY](#why-mark-as-DKIX_DUMMY) * [delete in entries](#delete-in-entries) * [end](#end) +* [read more](#read-more) -# related file -* cpython/Objects/dictobject.c -* cpython/Objects/clinic/dictobject.c.h -* cpython/Include/dictobject.h -* cpython/Include/cpython/dictobject.h +# Related File +* cpython/Objects/dictobject.c ([File URL](https://github.com/python/cpython/blob/master/Objects/dictobject.c)) +* cpython/Objects/clinic/dictobject.c.h ([File URL](https://github.com/python/cpython/blob/master/Objects/clinic/dictobject.c.h)) +* cpython/Include/dictobject.h ([File URL](https://github.com/python/cpython/blob/master/Include/dictobject.h)) -# memory layout +# Evolvement and Implementation + +Before we dive into the memory layout of a Python dictionary, let's imagine what a normal dictionary object looks like. +Usually, we implement a dictionary as a hash table. It takes **O(1)** time to look up an element, and that's exactly how CPython does it. +This is an entry, which points to a hash table inside the Python dictionary object before Python 3.6 + +![before_py36](./before_py36.png) + +If you have many large sparse hash tables, it will waste lots of memory. To represent the hash table more compactly, you can split indices and real key-value objects inside the hash table (after Python 3.6) + +![after_py36](./after_py36.png) -before we dive into the memory layout of python dictionary, let's imagine what normal dictionary object looks like +`Indices` points to an array of indices, `entries` points to where the original content is stored -usually, we implement a dictionary as a hash table, it takes O(1) time to look up an element, that's how exactly CPython does. +You can think of `indices` as a simpler version of a hash table, and `entries` as an array that stores each original hash value, key, and value as an element -this is an entry, which points to a hash table inside the python dictionary object before python3.6 +Whenever you search for an element or insert a new element, you take the hash result mod the size of `indices` to get an index in the `indices` array, then get the result you want from `entries` according to that index. For example, the result of `hash("key2") % 8` is `3`, and the value in `indices[3]` is `1`, so we can go to `entries` and find what we need in `entries[1]` -![entry_before](https://img-blog.csdnimg.cn/20190311111041784.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNzIwMzI5,size_16,color_FFFFFF,t_70) -If you have many large sparse hash tables, it will waste lots of memory. In order to represent the hash table in a more compact way, you can split indices and real key-value object inside the hashtable(After python3.6) +![after_py36_search](./after_py36_search.png) -![entry_after](https://img-blog.csdnimg.cn/20190311114021201.png) +We can benefit from more compact space with the above approach. For example, in a 64-bit operating system, every pointer takes 8 bytes. We need `8 * 3 * 8` which is `192` originally, while we only need `8 * 3 * 3 + 1 * 8` which is `80`. This saves about 58% of memory usage. -It takes about half of the original memory to store the same hash table, and we can traverse the hash table in the same order as we insert/delete items. Before python3.6 it's not possible to retain the order of key/value in the hash table due to the resize/rehash operation. For those who needs more detail, please refer to [python-dev](https://mail.python.org/pipermail/python-dev/2012-December/123028.html) and [pypy-blog](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html) +Because `entries` stores elements in the insertion order, we can traverse the hash table in the same order as we insert items. In the old implementation, the hash table stored elements in the order of the hash key, so it appeared **unordered** when you traversed the hash table. That's why dict before Python 3.6 is **unordered** and after Python 3.6 is **ordered**. -Now, let's see the memory layout +![after_py36_space](./after_py36_space.png) -![memory layout](https://img-blog.csdnimg.cn/20190308144931301.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNzIwMzI5,size_16,color_FFFFFF,t_70) +[PyPy](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html) implemented the compact and ordered dict first, and CPython implemented it in [PEP 468](https://www.python.org/dev/peps/pep-0468/) and released it in Python 3.6. -## combined table && split table +# memory layout + +Now, let's see the memory layout. + +![PyDictObject](./PyDictObject.png) -I'm confused when I first look at the definition of the **PyDictObject**, what's **ma_values** ? why **PyDictKeysObject** looks different from the indices/entries structure above? +## combined table and split table -the source code says +I was confused when I first looked at the definition of **PyDictObject**. What's **ma_values**? Why does **PyDictKeysObject** look different from the indices/entries structure above? + +The source code says ```python3 /* @@ -68,7 +84,7 @@ Or: ``` -In what situation will different dict object shares same keys but different values, only contain unicode keys and no dummy keys(no deleted object), and preserve same insertion order? Let's try. +In what situation will different dict objects share the same keys but different values, only contain Unicode keys and no dummy keys (no deleted objects), and preserve the same insertion order? Let's try. ```c # I've altered the source code to print some state information @@ -85,7 +101,7 @@ b2 = B() >>> b2.b 1 -# __dict__ obect appears, b1.__dict__ and b2.__dict__ are all split table, they shares the same PyDictKeysObject +# __dict__ object appears, b1.__dict__ and b2.__dict__ are both split tables, they share the same PyDictKeysObject >>> b1.b = 3 in lookdict_split, address of PyDictObject: 0x10bc0eb40, address of PyDictKeysObject: 0x10bd8cca8, key_str: b >>> b2.b = 4 @@ -107,14 +123,14 @@ in lookdict_split, address of PyDictObject: 0x10bdbbc00, address of PyDictKeysOb ``` -The split table implementation can save lots of memory if you have many instances of same class. For more detail, please refer to [PEP 412 -- Key-Sharing Dictionary](https://www.python.org/dev/peps/pep-0412/) +The splittable implementation can save lots of memory if you have many instances of the same class. For more detail, please refer to [PEP 412 -- Key-Sharing Dictionary](https://www.python.org/dev/peps/pep-0412/) -![dict_shares](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict_shares.png) +![key_shares](./key_shares.png) ## indices and entries -Let's analyze some source code to understand how indices/entries implement in **PyDictKeysObject**, what **char dk_indices[]** means in **PyDictKeysObject**? -(It took me sometimes to figure out) +Let's analyze some source code to understand how indices/entries are implemented in **PyDictKeysObject**. What does **char dk_indices[]** mean in **PyDictKeysObject**? +(It took me some time to figure out) ```c /* @@ -153,7 +169,7 @@ dk_size == 256. ``` -Let's rewrite the marco +Let's rewrite the macro ```c #define DK_ENTRIES(dk) \ @@ -174,11 +190,11 @@ PyDictKeyEntry *entries = (PyDictKeyEntry *) pointer_to_entries; now, the overview is clear -![dictkeys_basic](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dictkeys_basic.png) +![dictkeys_basic](./dictkeys_basic.png) # hash collisions and delete -how CPython handle hash collisions in dict object? Instead of depending on a good hash function, python uses "perturb" strategy, let's read some source code and have a try +How does CPython handle hash collisions in dict objects? Instead of depending on a good hash function, Python uses the "perturb" strategy. Let's read some source code and try it out. ```python3 @@ -190,7 +206,7 @@ use j % 2**i as the next table index; ``` -I've altered the source code to print some information +I've altered the source code to print some information. ```python3 @@ -212,44 +228,47 @@ index: 6 ix: -1 DKIX_EMPTY index: 7 ix: -1 DKIX_EMPTY '{1: 1}' - ``` -![hh_1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_1.png) +`indices` is the index, value **-1(DKIX_EMPTY)** means the current location is empty, now I set `d[1] = "value1"` in the code, `hash(1) & mask == 1` will find the location 1 in the `indices`, it's `-1(DKIX_EMPTY)` originally, means it's free to use, we take the position and change `-1` to the next free index in `entries`, which is `0`, so we store `0` to `indices[1]`. -```python3 -d[4] = 4 +![hh_1](./hh_1.png) +```python3 +d[4] = "value4" ``` -![hh_2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_2.png) +![hh_2](./hh_2.png) ```python3 -d[7] = 111 - +d[7] = "value7" ``` -![hh_3](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_3.png) +![hh_3](./hh_3.png) ```python3 # delete, mark as DKIX_DUMMY # notice, dk_usable and dk_nentries don't change del d[4] - ``` -![hh_4](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_4.png) +![hh_4](./hh_4.png) ```python3 # notice, dk_usable and dk_nentries now change -d[0] = 0 - +d[0] = "value3" ``` -![hh_5](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_5.png) +![hh_5](./hh_5.png) + +If we insert an element with hash key result 16, `hash (16) & mask == 0`, but `0` is already taken, we encounter hash collision + +There are various ways to handle hash collisions. `Redis` uses open hashing (linked list) to solve the problem. CPython `dict` object implements closed hashing with a more sporadic algorithm (read the source code if you're interested). + +![dict_prob](./dict_prob.png) ```python3 -d[16] = 16 +d[16] = "value16" # hash (16) & mask == 0 # but position 0 already taken by key: 0, value: 0 # currently perturb = 16, PERTURB_SHIFT = 5, i = 0 @@ -263,7 +282,7 @@ d[16] = 16 ``` -![hh_6](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_6.png) +# ![hh_6](./hh_6.png) # resize @@ -271,39 +290,72 @@ d[16] = 16 # now, dk_usable is 0, dk_nentries is 5 # let's insert one more item d[5] = 5 -# step1: resize, when resizing, the deleted object which mark as DKIX_DUMMY in entries won't be copied +# step1: resize, when resizing, the deleted objects marked as DKIX_DUMMY in entries won't be copied ``` -![resize](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/resize.png) +![resize](./resize.png) ```python3 # step2: insert key: 5, value: 5 - - ``` # variable size indices -Notice, the indices array is variable size. when the size of your hash table is <= 128, type of each item is int_8, int16 and int64 for bigger table. The variable size indices array can save memory usage. + +Notice that the indices array has variable size. When the size of your hash table is <= 128, the type of each item is int8. For larger tables, it uses int16, int32, or int64. The variable size of the indices array can save memory. + +![indices](./indices.png) # free list ```python3 +#ifndef PyDict_MAXFREELIST +#define PyDict_MAXFREELIST 80 +#endif static PyDictObject *free_list[PyDict_MAXFREELIST]; +``` + +CPython also uses free_list to reuse deleted hash tables, to avoid memory fragmentation and improve performance. + +There exists a per-process global variable named **free_list**. + +![free_list0](./free_list0.png) + +If we create a new `dict` object, the memory request is delegated to CPython's [memory management system](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management.md) +```python3 +d = dict() +``` + +![free_list1](./free_list1.png) + +```python3 +del a ``` -CPython also use free_list to reuse the deleted hash table, to avoid memory fragment and improve performance, I've illustrated free_list in [list object](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/list.md#why-free-list) +The destructor of the `dict` type will store the current `dict` in **free_list** (if **free_list** is not full) + +![free_list2](./free_list2.png) + +Next time you create a new `dict` object, **free_list** will be checked for available objects you can use directly. If available, allocate from **free_list**; if not, allocate from CPython's [memory management system](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management.md) + +```python3 +d = dict() +``` + + + +![free_list3](./free_list3.png) # delete -the [lazy deletion](https://en.wikipedia.org/wiki/Lazy_deletion) strategy is used for deletion in dict object +The [lazy deletion](https://en.wikipedia.org/wiki/Lazy_deletion) strategy is used for deletion in dict objects ## why mark as DKIX_DUMMY -why mark as **DKIX_DUMMY** in the indices? +Why mark as **DKIX_DUMMY** in the indices? -there're totally three kinds of value for each slot in indices, **DKIX_EMPTY**(-1), **DKIX_DUMMY**(-2) and **Active/Pending**(>=0), if you mark as **DKIX_EMPTY** instead of **DKIX_DUMMY**, the "perturb" strategy for inserting and finding key/values will fail. Imagine you have a hash table in size 8, this is the indices in hash table +There are three kinds of values for each slot in indices: **DKIX_EMPTY**(-1), **DKIX_DUMMY**(-2), and **Active/Pending**(>=0). If you mark a slot as **DKIX_EMPTY** instead of **DKIX_DUMMY**, the "perturb" strategy for inserting and finding key/values will fail. Imagine you have a hash table of size 8; these are the indices in the hash table. ```python3 indices: [0] [1] [DKIX_EMPTY] [2] [DKIX_EMPTY] [DKIX_EMPTY] [3] [4] @@ -311,9 +363,7 @@ index: 0 1 2 3 4 5 6 7 ``` -when you search for an item with hash value 0, and real position is in entries[4] - -this is the searching process according to the "perturb" strategy +When you search for an item with hash value 0, and the real position is in entries[4], this is the searching process according to the "perturb" strategy. ```python3 0 -> 1 -> 6 -> 7(found, no need to go on) @@ -335,7 +385,7 @@ the searching process now becomes ``` -the item being searched is in indices[7], but "perturb" strategy stopped before indices[7] due to the wrong **DKIX_EMPTY** in indices[6], if we mark deleted as **DKIX_DUMMY** +The item being searched is in indices[7], but the "perturb" strategy stopped before indices[7] due to the wrong **DKIX_EMPTY** in indices[6]. If we mark deleted as **DKIX_DUMMY** ```python3 indices: [0] [1] [DKIX_EMPTY] [2] [DKIX_EMPTY] [DKIX_EMPTY] [DKIX_DUMMY] [4] @@ -350,16 +400,22 @@ the searching process becomes ``` -also, the indices with **DKIX_DUMMY** can be inserted for new item +Also, indices with **DKIX_DUMMY** can be used for new items ## delete in entries -dict object need to guarantee the [inserted order](https://mail.python.org/pipermail/python-dev/2017-December/151283.html), the delete operation can't shuffle objects in **entries** +Dict objects need to guarantee [insertion order](https://mail.python.org/pipermail/python-dev/2017-December/151283.html), so the delete operation can't shuffle objects in **entries** -leave the entries[i] to empty, and packing these empty entries later in the resize operation can keep the time complexity of delete operation in amortized O(1) +Leaving entries[i] empty and packing these empty entries later in the resize operation keeps the time complexity of delete operations at amortized O(1) -deleting entries from dictionaries is not very common, in some case a little bit slower is acceptable([PyPy Status Blog](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html)) +Deleting entries from dictionaries is not very common, so being a little bit slower in some cases is acceptable ([PyPy Status Blog](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html)) # end -now, you understand how python dictionary object works internally. \ No newline at end of file +Now you understand how Python dictionary objects work internally. + +# read more + +* [pypy-blog](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html) + +* [python-dev](https://mail.python.org/pipermail/python-dev/2012-December/123028.html) diff --git a/BasicObject/dict/dict_cn.md b/BasicObject/dict/dict_cn.md index 72125b4..bc8f8ed 100644 --- a/BasicObject/dict/dict_cn.md +++ b/BasicObject/dict/dict_cn.md @@ -1,57 +1,76 @@ -# dict - -# 目录 - -因为 **PyDictObject** 比其他的基本对象稍微复杂一点点,我不会在这里一步一步的展示 `__setitem__`/`__getitem__` 的过程,这些基本步骤会穿插在一些概念的解释里面一起说明 +# 字典 * [相关位置文件](#相关位置文件) -* [内存构造](#内存构造) - * [combined table && split table](#combined-table-&&-split-table) - * [indices and entries](#indices-and-entries) +* [演变和实现](#演变和实现) +* [内存构造](#演变和实现) + * [combined table 和 split table](#combined-table-和-split-table) + * [indices 和 entries](#indices-和-entries) * [哈希碰撞与删除](#哈希碰撞与删除) * [表扩展](#表扩展) -* [类型可变的indices数组](#类型可变的indices数组) -* [free list](#free-list) +* [indices数组](#indices数组) +* [缓冲池](#缓冲池) * [删除操作](#删除操作) - * [为什么标记成 DKIX_DUMMY](#为什么标记成-DKIX_DUMMY) - * [entries 中的删除](#entries-中的删除) -* [结束](#结束) + * [为什么标记成 DKIX_DUMMY](#为什么标记成-DKIX_DUMMY) + * [entries 中的删除](#entries-中的删除) +* [更多资料](#更多资料) # 相关位置文件 + * cpython/Objects/dictobject.c * cpython/Objects/clinic/dictobject.c.h * cpython/Include/dictobject.h * cpython/Include/cpython/dictobject.h -# 内存构造 -在深入看python 字典对象的内存构造之前,我们先来想象一下,如果让我们自己造一个字典对象会长成什么样呢? +# 演变和实现 -通常情况下,我们会用哈希表来实现一个字典对象,你只用花O(1)的时间即可以完成对一个元素的查找,这也是 CPython 的实现方式 +在深入看 CPython 字典对象的内部构造和实现之前,我们先来想象一下,如果让我们自己造一个字典对象会长成什么样呢? -下图是在 python3.6 版本之前,一个指向 python 内部字典对象的入口指针 +通常情况下,我们会用哈希表来实现一个字典对象,平均情况下你只需要花费 O(1) 的时间复杂度即可以完成对一个元素的查找,这也是 CPython 的实现方式 -![entry_before](https://img-blog.csdnimg.cn/20190311111041784.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNzIwMzI5,size_16,color_FFFFFF,t_70) +下图是在 [python3.6](https://github.com/python/cpython/commit/742da040db28e1284615e88874d5c952da80344e) 版本之前,一个指向 python 内部字典对象的入口指针 -如果你有很多稀疏的哈希表,会浪费很多内存空间,为了用更紧凑的方式来实现哈希表,可以把 哈希索引 和 真正的键值对分开存放,下图是 python3.6 版本以后的实现方式 +![before_py36](./before_py36.png) -![entry_after](https://img-blog.csdnimg.cn/20190311114021201.png) +如果你的应用场景下有比较多的字典对象, 当它们以稀疏的哈希表存储时,会浪费较多的内存空间,为了用更紧凑的方式来实现哈希表,可以把 哈希索引 和 真正的键值对分开存放,下图是 python3.6 版本以后的实现方式 -只花费了之前差不多一半的内存,就存下了同样的一张哈希表,而且我们可以在 resize 时保持键值对的顺序,需要更详细的内容,请参考 [python-dev](https://mail.python.org/pipermail/python-dev/2012-December/123028.html) 和 [pypy-blog](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html) +下图是 python3.6 版本以后的实现方式 -现在,我们来看看 **PyDictObject** 的内存构造 +![after_py36](./after_py36.png) -![memory layout](https://img-blog.csdnimg.cn/20190308144931301.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNzIwMzI5,size_16,color_FFFFFF,t_70) +`indices` 指向了一列索引, `entries` 指向了原本的存储哈希表内容的结构 -## combined table && split table +你可以把 `indices` 理解成新的简化版的哈希表, `entries` 理解成一个数组, 数组中的每个元素是原本应该存储的哈希结果,键和值 -第一次看到 **PyDictObject** 的定义是很懵逼的,**ma_values** 是什么? 怎么 **PyDictKeysObject** 和上面看到的 indices/entries 结构长得不太一样? +查找或者插入一个元素的时候, 根据键的哈希值结果取模 `indices` 的长度, 就能得到对应的数组下标, 再根据对应的数组下标到 `entries` 中获取到对应的结果, 比如 `hash("key2") % 8` 的结果是 `3`, 那么 `indices[3]` 的值是 `1`, 这时候到 `entries` 中找到对应的 `entries[1]` 既为所求的结果 -源代码是这么注释的,我给翻译一下 +![after_py36_search](./after_py36_search.png) -```python3 +这么做的好处是空间利用率得到了较大的提升, 我们以 64 位操作系统为例, 每个指针的长度为 8 字节, 则原本需要 `8 * 3 * 8` 为 `192` + +现在变成了 `8 * 3 * 3 + 1 * 8` 为 `80`, 节省了 58% 左右的内存空间 + +并且由于 `entries` 是按照插入顺序进行插入的数组, 对字典进行遍历时能按照插入顺序进行遍历, 而在旧的哈希表方案中, 由于不同键的哈希值不一样, 哈希表中的顺序是按照哈希值大小排序的, 遍历时从前往后遍历表现起来就是 "无序" 的, 这也是为什么 python3.6 以前的版本字典对象是 **无序** 的但是 python3.6 以后的版本字典对象是 **有序** 的原因 + +![after_py36_space](./after_py36_space.png) + +这个方案最初是由 [PyPy](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html) 提出, 并在 [pep 468](https://www.python.org/dev/peps/pep-0468/) 中引入到了 CPython, 在 python3.6 版本进行了实现 + +# 内存构造 + +我们来看看 **PyDictObject** 的内存构造 + +![PyDictObject](./PyDictObject.png) + +## combined table 和 split table + +第一次看到 **PyDictObject** 的定义是很疑惑的,**ma_values** 是什么? 怎么 **PyDictKeysObject** 和上面看到的 indices/entries 结构长得不太一样 ? + +下面是经过翻译的源代码注释 + +```c /* 字典对象可以按照以下两种方式来表示 The DictObject can be in one of two forms. @@ -75,10 +94,9 @@ Or: 所有共享的 哈希键值 必须是按照同样顺序插入的 All dicts sharing same key must have same insertion order. */ - ``` -什么意思呢? 在什么情况下字段对象会共享同样的键,但是不共享值呢? 而且键还只能是 unicode 对象,字典对象中的键还不能被删除? 我们来试试看 +什么意思呢? 在什么情况下字典对象会共享同样的键,但是不共享值呢? 而且键还只能是 unicode 对象,字典对象中的键还不能被删除? 我们来试试看 ```python3 # 我更改了部分源代码,解释器执行 split table 查询时会打印出一些信息 @@ -114,25 +132,21 @@ in lookdict_split, address of PyDictObject: 0x10bdbbc00, address of PyDictKeysOb # 现在,看不到 lookdict_split 的输出了,说明已经变成了 combined table >>> b2.b 4 - ``` -split table 可以在你对同个class有非常多实例的时候节省很多内存,这些实例在满足上述条件时,都会共享同一个 PyDictKeysObject, 更多关于实现方面的细节请参考 [PEP 412 -- Key-Sharing Dictionary](https://www.python.org/dev/peps/pep-0412/) +`split table` 可以在你对同个`class` 有非常多实例的时候节省很多内存,这些实例在满足上述条件时,都会共享同一个 `PyDictKeysObject`, 更多关于实现方面的细节请参考 [PEP 412 -- Key-Sharing Dictionary](https://www.python.org/dev/peps/pep-0412/) -![dict_shares](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict_shares.png) +![key_shares](./key_shares.png) -## indices and entries +## indices 和 entries -我们进到源代码里看一下 cpython 如何在 **PyDictKeysObject** 中实现 indices/entries, **char dk_indices[]** 又是什么意思? +我们进到源代码里看一下 CPython 如何在 **PyDictKeysObject** 中实现 indices/entries, **char dk_indices[]** 又是什么意思? ```c /* -这是代码本身的注释,我来翻译一下 +这是代码本身的注释,我进行了翻译 dk_indices 是真正的哈希表,他在 entries 存储了哈希的索引,或者 DKIX_EMPTY(-1) 和 DKIX_DUMMY(-2) 的一种 -dk_indices is actual hashtable. It holds index in entries, or DKIX_EMPTY(-1) -or DKIX_DUMMY(-2). dk_size 字段表示 indices 的大小,但是这个字段的类型是可以根据表的大小变化的 -Size of indices is dk_size. Type of each index in indices is vary on dk_size: * int8 for dk_size <= 128 * int16 for 256 <= dk_size <= 2**15 @@ -140,13 +154,8 @@ Size of indices is dk_size. Type of each index in indices is vary on dk_size: * int64 for 2**32 <= dk_size dk_entries 是一个数组,里面的对象类型是 PyDictKeyEntry, 他的大小可以用 USABLE_FRACTION 这个宏来表示, DK_ENTRIES(dk) 可以获得真正哈希键对值的第一个入口 -dk_entries is array of PyDictKeyEntry. It's size is USABLE_FRACTION(dk_size). -DK_ENTRIES(dk) can be used to get pointer to entries. # 注意, DKIX_EMPTY 和 DKIX_DUMMY 是用负数表示的,所有 dk_indices 里面的索引类型也需要用有符号整数表示,int16 用来表示 dk_size 为 256 的表 -NOTE: Since negative value is used for DKIX_EMPTY and DKIX_DUMMY, type of -dk_indices entry is signed integer and int16 is used for table which -dk_size == 256. */ #define DK_SIZE(dk) ((dk)->dk_size) @@ -186,86 +195,84 @@ PyDictKeyEntry *entries = (PyDictKeyEntry *) pointer_to_entries; ``` -现在就很清晰了 +现在就比较清晰 + +![dictkeys_basic](./dictkeys_basic.png) -![dictkeys_basic](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dictkeys_basic.png) +`indices` 和 `entries` 是一段连续的连起来的空间, `indices` 指向前面介绍的哈希索引, `entries` 指向真正存储数据的数组头部 # 哈希碰撞与删除 -cpython 是怎么处理字典对象里的哈希碰撞的呢? 除了依靠一个好的哈希函数,cpython 还依赖 perturb 策略, -我们来读一读源代码看看 +CPython 是怎么处理字典对象里的哈希碰撞的呢? 除了依靠一个好的哈希函数,cpython 还实现了一个 perturb 策略 +我们可以截取部分源代码注释看下 ```python3 - j = ((5*j) + 1) mod 2**i -0 -> 1 -> 6 -> 7 -> 4 -> 5 -> 2 -> 3 -> 0 [and here it's repeating] +0 -> 1 -> 6 -> 7 -> 4 -> 5 -> 2 -> 3 -> 0 [开始下次循环] perturb >>= PERTURB_SHIFT; j = (5*j) + 1 + perturb; use j % 2**i as the next table index; - ``` -我更改了部分源代使得每次打印出更多信息 +我更改了部分源代使得每次打印出更多信息, 更改后的代码可在参考资料中找到 ```python3 - >>> d = dict() ->>> d[1] = 1 -: 1, ix: -1, address of ep0: 0x10870d798, dk->dk_indices: 0x10870d790 -ma_used: 0, ma_version_tag: 11313, PyDictKeyObject.dk_refcnt: 1, PyDictKeyObject.dk_size: 8, PyDictKeyObject.dk_usable: 5, PyDictKeyObject.dk_nentries: 0 -DK_SIZE(dk): 8, DK_IXSIZE(dk): 1, DK_SIZE(dk) * DK_IXSIZE(dk): 8, &((int8_t *)((dk)->dk_indices))[DK_SIZE(dk) * DK_IXSIZE(dk)]): 0x10870d798 - +>>> d[1] = "value1" >>> repr(d) -ma_used: 1, ma_version_tag: 11322, PyDictKeyObject.dk_refcnt: 1, PyDictKeyObject.dk_size: 8, PyDictKeyObject.dk_usable: 4, PyDictKeyObject.dk_nentries: 1 +ma_used: 1, ma_version_tag: 16667, PyDictKeyObject.dk_refcnt: 1, PyDictKeyObject.dk_size: 8, PyDictKeyObject.dk_usable: 4, PyDictKeyObject.dk_nentries: 1 index: 0 ix: -1 DKIX_EMPTY -index: 1 ix: 0 me_hash: 1, me_key: 1, me_value: 1 +index: 1 ix: 0 me_hash: 1, me_key: 1, me_value: 'value1' index: 2 ix: -1 DKIX_EMPTY index: 3 ix: -1 DKIX_EMPTY index: 4 ix: -1 DKIX_EMPTY index: 5 ix: -1 DKIX_EMPTY index: 6 ix: -1 DKIX_EMPTY index: 7 ix: -1 DKIX_EMPTY -'{1: 1}' - +"{1: 'value1'}" ``` -这里 indices 是索引, 索引里面 **-1(DKIX_EMPTY)** 表示这个坑位是空的, 现在代码里设置了 d[1] = 1, hash(1) & mask == 1, 会对应到 indices 的下标为 1 的坑位上, 这个坑位原本是 -1(DKIX_EMPTY) 表示没有被占用过, 马上占用他, 把这里的 -1 改成 entries 里面第一个空余的真正的存储 key 和 value 的位置, 这个位置是 0, 所以就把 0 存储到了 indices[1] 里 +这里 `indices` 是索引, 索引里面 **-1(DKIX_EMPTY)** 表示这个位置是空的, 现在代码里设置了 `d[1] = "value1"`, 而`hash(1) & mask == 1`, 会对应到 indices 的下标为 1 的位置上, 这个位置原本是 `-1(DKIX_EMPTY)`, 表示没有被占用过, 马上占用他, 把这里的 `-1` 改成 entries 里面第一个空余的真正的存储 key 和 value 的位置, 这个位置是 0, 所以就把 0 存储到了 `indices[1]` 里 -![hh_1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_1.png) +![hh_1](./hh_1.png) ```python3 -d[4] = 4 - +d[4] = "value4" ``` -![hh_2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_2.png) +![hh_2](./hh_2.png) -```python3 -d[7] = 111 +每新增一个元素, `dk_usable` 就减小1, `dk_nentries` 就增加1 +```python3 +d[7] = "value7" ``` -![hh_3](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_3.png) +![hh_3](./hh_3.png) ```python3 -# 删除的时候并不会把索引清除,而是把对应那格的索引标记成 DKIX_DUMMY, DKIX_DUMMY 数字表示为 -2 +# 在删除的时候并不会把索引清除,而是把对应那格的索引标记成 DKIX_DUMMY, DKIX_DUMMY 数字表示为 -2 # 并且 dk_usable 和 dk_nentries 并没有改变 del d[4] - ``` -![hh_4](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_4.png) +![hh_4](./hh_4.png) ```python3 # 注意, dk_usable 和 dk_nentries 现在变了 -d[0] = 0 - +d[0] = "value3" ``` -![hh_5](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_5.png) +![hh_5](./hh_5.png) + +我们此时插入一个哈希结果为 16 的键的话, `hash (16) & mask == 0`, 而 `0` 已经被占用, 就产生了哈希碰撞, 在一些常规的哈希表的实现中, 比如 `Redis` 的 `hashtable` 使用的是开链法, CPython 的 `set` 对象使用的是线性探测法, 而CPython 的 `dict` 对象中为了避免上述两种方法的缺点, 实现了一套更加分散的探测方法(篇幅限制就不展开细说了, 有兴趣的同学可以参考源码注释) + +![dict_prob](./dict_prob.png) + +我们现在插入 `d[16] = "value16"` 看看结果 ```python3 -d[16] = 16 +d[16] = "value16" # hash (16) & mask == 0 # 但是索引位置 0 已经被 key: 0, value: 0 这个对象占用了 # 当前 perturb = 16, PERTURB_SHIFT = 5, i = 0 @@ -276,44 +283,76 @@ d[16] = 16 # 所以, perturb >>= PERTURB_SHIFT ===> perturb == 0 # i = (i*5 + perturb + 1) & mask ===> i = 6 # 此时索引位置 6 是空的,插入 - ``` -![hh_6](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/hh_6.png) +![hh_6](./hh_6.png) # 表扩展 +现在, `dk_usable` 为 0, `dk_nentries` 为 5, 说明当前表中存储的元素已经达到了阈值, 再插入一组对象试试 + ```python3 -# 现在, dk_usable 为 0, dk_nentries 为 5 -# 再插入一个对象试试 -d[5] = 5 -# 第一步,表达到了阈值,扩展表,并复制 -# 索引里被标记成 DKIX_DUMMY 不会被复制,所以索引对应的位置后面的元素都会往前移 +d[5] = "value5" +``` + +第一步,表达到了阈值,扩展表,并复制 + +索引里被标记成 `DKIX_DUMMY` 不会被复制,所以索引对应的位置后面的元素都会往前移 +第二步, 插入 + +```python3 +key: 5, value: "value5" ``` -![resize](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/resize.png) +![resize](./resize.png) + +# indices数组 + +indices 数组的大小是可变的,当你的哈希表大小 <= 128 时,索引数组的元素类型为 int_8, 表变大时会相应地用 int16, int32 或者 int64 来表示,这样做可以节省内存使用,这个策略在上面代码注释部分解释过了 + +![indices](./indices.png) + +# 缓冲池 ```python3 -# 第二步, 插入 key: 5, value: 5 +#ifndef PyDict_MAXFREELIST +#define PyDict_MAXFREELIST 80 +#endif +static PyDictObject *free_list[PyDict_MAXFREELIST]; +``` + +CPython 同时使用了 free_list 来重新循环使用那些被删除掉的字典对象,这样做的好处是可以避免内存碎片并且提高性能 + +每个进程都拥有一个全局变量 `free_list` +![free_list0](./free_list0.png) + +如果我们创建一个新的 `dict` 对象, 创建新对象的内存分配过程会用到 CPython 的 [内存管理机制](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management_cn.md) + +```python3 +d = dict() ``` -# 类型可变的indices数组 -indices 数组的大小是可变的,当你的哈希表大小 <= 128 时,索引数组的元素类型为 int_8, 表变大时会用 int16 或者 int64 来表示,这样做可以节省内存使用,这个策略在上面代码注释部分说明过了 +![free_list1](./free_list1.png) -# free list +之后我们再删除这个字典对象 ```python3 -static PyDictObject *free_list[PyDict_MAXFREELIST]; +del a +``` +![free_list2](./free_list2.png)下一次你创建一个新的 `dict` 对象时, 会优先检查 `free_list` 中是否有可用的对象, 如果有的话则从 `free_list` 取, 如果没有的话, 再从 CPython 的 [内存管理机制](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management_cn.md) 申请 + +```python3 +d = dict() ``` -cpython 也会用 free_list 来重新循环使用那些被删除掉的字典对象,可以避免内存碎片和提高性能,需要图解的同学可以参考 [list](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/list_cn.md#%E4%B8%BA%E4%BB%80%E4%B9%88%E7%94%A8-free-list), cpython set 使用了同样的策略,里面有图片解释 +![free_list3](./free_list3.png) # 删除操作 -字典中元素的删除使用的是 [lazy deletion](https://en.wikipedia.org/wiki/Lazy_deletion) 策略(上面已展示过) +字典中元素的删除使用的是 [惰性删除](https://en.wikipedia.org/wiki/Lazy_deletion) 的策略(前面已展示过) ## 为什么标记成 DKIX_DUMMY @@ -368,12 +407,16 @@ index: 0 1 2 3 4 5 6 7 ## entries 中的删除 -dict 对象需要保证字典中元素按照 [插入顺序](https://mail.python.org/pipermail/python-dev/2017-December/151283.html) 来进行保存, 删除操作不能对 **entries** 中的元素进行排序 +删除操作不对 **entries** 中的元素进行重新排序, 这么做可以保证字典中的元素存储顺序是按照 [插入顺序](https://mail.python.org/pipermail/python-dev/2017-December/151283.html) 来进行保存的 + +并且把 `entries[i]` 当成空的值, 在后续表扩展/压缩时再重新压缩这部分空余的空间, 可以保持删除操作的平均时间复杂度为 O(1), 这在算法中是一种平摊(amortized)的思想 + +并且通常情况下字典对象的删除操作并不普遍, 只有部分情况下会导致性能变慢 + -把 entries[i] 当成空的值, 在后续表扩展/压缩时再重新压缩这部分空余的空间, 这样做可以保持删除操作的平均时间复杂度为 O(1) -并且通常情况下字典对象的删除操作并不普遍, 只有部分情况下会导致性能变慢([PyPy Status Blog](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html)) +# 更多资料 -#### 结束 +* [pypy-blog](https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html) -现在,我们大致弄明白了 cpython 字典对象的内部实现了! \ No newline at end of file +* [python-dev](https://mail.python.org/pipermail/python-dev/2012-December/123028.html) diff --git a/BasicObject/dict/dict_prob.png b/BasicObject/dict/dict_prob.png new file mode 100644 index 0000000..07d0c4a Binary files /dev/null and b/BasicObject/dict/dict_prob.png differ diff --git a/BasicObject/dict/dict_shares.png b/BasicObject/dict/dict_shares.png deleted file mode 100644 index b1b08f3..0000000 Binary files a/BasicObject/dict/dict_shares.png and /dev/null differ diff --git a/BasicObject/dict/free_list0.png b/BasicObject/dict/free_list0.png new file mode 100644 index 0000000..4bd1695 Binary files /dev/null and b/BasicObject/dict/free_list0.png differ diff --git a/BasicObject/dict/free_list1.png b/BasicObject/dict/free_list1.png new file mode 100644 index 0000000..5544aed Binary files /dev/null and b/BasicObject/dict/free_list1.png differ diff --git a/BasicObject/dict/free_list2.png b/BasicObject/dict/free_list2.png new file mode 100644 index 0000000..d63aea2 Binary files /dev/null and b/BasicObject/dict/free_list2.png differ diff --git a/BasicObject/dict/free_list3.png b/BasicObject/dict/free_list3.png new file mode 100644 index 0000000..031be67 Binary files /dev/null and b/BasicObject/dict/free_list3.png differ diff --git a/BasicObject/dict/hh_1.png b/BasicObject/dict/hh_1.png index e75cecc..76515ea 100644 Binary files a/BasicObject/dict/hh_1.png and b/BasicObject/dict/hh_1.png differ diff --git a/BasicObject/dict/hh_2.png b/BasicObject/dict/hh_2.png index 9f2a843..c9ded3c 100644 Binary files a/BasicObject/dict/hh_2.png and b/BasicObject/dict/hh_2.png differ diff --git a/BasicObject/dict/hh_3.png b/BasicObject/dict/hh_3.png index af108dc..827d4b0 100644 Binary files a/BasicObject/dict/hh_3.png and b/BasicObject/dict/hh_3.png differ diff --git a/BasicObject/dict/hh_4.png b/BasicObject/dict/hh_4.png index 77de9f8..8da28b4 100644 Binary files a/BasicObject/dict/hh_4.png and b/BasicObject/dict/hh_4.png differ diff --git a/BasicObject/dict/hh_5.png b/BasicObject/dict/hh_5.png index 8466115..cf538f0 100644 Binary files a/BasicObject/dict/hh_5.png and b/BasicObject/dict/hh_5.png differ diff --git a/BasicObject/dict/hh_6.png b/BasicObject/dict/hh_6.png index 18f07b3..b221ad7 100644 Binary files a/BasicObject/dict/hh_6.png and b/BasicObject/dict/hh_6.png differ diff --git a/BasicObject/dict/indices.png b/BasicObject/dict/indices.png new file mode 100644 index 0000000..4b166a5 Binary files /dev/null and b/BasicObject/dict/indices.png differ diff --git a/BasicObject/dict/key_shares.png b/BasicObject/dict/key_shares.png new file mode 100644 index 0000000..42d74ac Binary files /dev/null and b/BasicObject/dict/key_shares.png differ diff --git a/BasicObject/dict/resize.png b/BasicObject/dict/resize.png index b089e97..671b92f 100644 Binary files a/BasicObject/dict/resize.png and b/BasicObject/dict/resize.png differ diff --git a/BasicObject/enum/enum.md b/BasicObject/enum/enum.md index c7ebf74..f7213e2 100644 --- a/BasicObject/enum/enum.md +++ b/BasicObject/enum/enum.md @@ -15,7 +15,7 @@ # memory layout -**enumerate** is a **type**, the instance of **enumerate** object is an iterable object, you can iter through the real delegated object and counting index at the same time +**enumerate** is a **type**. The instance of an **enumerate** object is an iterable object; you can iterate through the real delegated object while counting the index at the same time ![layout](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/enum/layout.png) @@ -36,9 +36,9 @@ e = enumerate(gen()) ``` -before iter through the object **e**, the **en_index** field is 0, **en_sit** stores the actual generator object being iterated, **en_result** points a tuple object with two empty value +Before iterating through the object **e**, the **en_index** field is 0, **en_sit** stores the actual generator object being iterated, and **en_result** points to a tuple object with two empty values. -we will see the meaning of **en_longindex** later +We will see the meaning of **en_longindex** later ![example0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/enum/example0.png) @@ -51,9 +51,9 @@ we will see the meaning of **en_longindex** later ``` -now, the **en_index** becomes 1, the tuple in **en_result** is the last tuple object returned, the elements in the tuple are changed, but the address in **en_result** doesn't change, not because of the [free-list mechanism in tuple](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md#free-list) +Now the **en_index** becomes 1. The tuple in **en_result** is the last tuple object returned. The elements in the tuple are changed, but the address in **en_result** doesn't change, and this is not because of the [free-list mechanism in tuple](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md#free-list). -it's a trick in the **enumerate** iterating function +It's a trick in the **enumerate** iterating function ```c static PyObject * @@ -94,15 +94,15 @@ enum_next(enumobject *en) ``` -it's clear, because only the current enumerate object keep a reference to the old `tuple object -> (None, None)` object +It's clear: because only the current enumerate object keeps a reference to the old `tuple object -> (None, None)` object. -**enum_next** reset the 0th element in the tuple to 0, and 1th element in the tuple to 'I', so the address **en_result** points to is the same +**enum_next** resets the 0th element in the tuple to 0 and the 1st element in the tuple to 'I', so the address **en_result** points to is the same ![example1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/enum/example1.png) -because the reference count of the tuple object `(0, 'I') # id(4469348888)` now becomes 2, one from the enumerate object and one from variable t1, the **enum_next** function will create and return a new tuple instead of resetting the one stored in **en_result** +Because the reference count of the tuple object `(0, 'I') # id(4469348888)` now becomes 2 (one from the enumerate object and one from variable t1), the **enum_next** function will create and return a new tuple instead of resetting the one stored in **en_result**. -the **en_index** incremented, and **en_result** points to the old tuple object`(0, 'I') # id(4469348888)` +The **en_index** is incremented, and **en_result** still points to the old tuple object `(0, 'I') # id(4469348888)` ```python3 >>> next(e) @@ -123,7 +123,7 @@ after `del t1`, the reference count of tuple object `(0, 'I') # id(4469348888)` ![example3](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/enum/example3.png) -the termination state is indicated by the object inside the **en_sit** field, nothing changed in the **enumerate** object +The termination state is indicated by the object inside the **en_sit** field; nothing changed in the **enumerate** object ```python3 >>> next(e) @@ -137,7 +137,7 @@ StopIteration ## en_longindex -usually, the **index** of enumerate object is stored in the **en_index** field, **en_index** is of type **Py_ssize_t** in c, and **Py_ssize_t** is defined as +Usually, the **index** of an enumerate object is stored in the **en_index** field. **en_index** is of type **Py_ssize_t** in C, and **Py_ssize_t** is defined as ```c #ifdef HAVE_SSIZE_T @@ -150,26 +150,26 @@ typedef Py_intptr_t Py_ssize_t; ``` -most time it's a **ssize_t**, which is type **int** in 32-bit os and **long int** in 64-bit os +Most of the time it's a **ssize_t**, which is type **int** in 32-bit OS and **long int** in 64-bit OS. -in my machine, it's type **long int** +On my machine, it's type **long int**. -what if the index is so big that a signed 64-bit can't hold? +What if the index is so big that a signed 64-bit integer can't hold it? ```python3 e = enumerate(gen(), 1 << 62) ``` -the **en_index** can hold the value +The **en_index** can hold the value ![longindex0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/enum/longindex0.png) -the max value **en_index** can represent is ((1 << 63) - 1) (PY_SSIZE_T_MAX) +The max value **en_index** can represent is ((1 << 63) - 1) (PY_SSIZE_T_MAX). -now the actual index is larger than PY_SSIZE_T_MAX, so the **en_longindex** is used to represent the actual index +Now the actual index is larger than PY_SSIZE_T_MAX, so **en_longindex** is used to represent the actual index. -what **en_longindex** points to is a [PyLongObject(python type int)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/long.md), which can represent variable size integer +What **en_longindex** points to is a [PyLongObject (Python type int)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/long.md), which can represent variable-size integers ```python3 >>> e = enumerate(gen(), (1 << 63) + 100) diff --git a/BasicObject/float/float.md b/BasicObject/float/float.md index f736df3..56b9ba6 100644 --- a/BasicObject/float/float.md +++ b/BasicObject/float/float.md @@ -19,7 +19,7 @@ # memory layout -**PyFloatObject** is no more than a wrapper of c type **double**, which takes 8 bytes to represent a floating point number +**PyFloatObject** is simply a wrapper around the C type **double**, which takes 8 bytes to represent a floating-point number you can refer to [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754-1985)/[IEEE-754标准与浮点数运算](https://blog.csdn.net/m0_37972557/article/details/84594879) for more detail @@ -29,7 +29,7 @@ you can refer to [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754-1985)/[IEEE-7 ## 0 -the binary representation of 0.0 in **IEEE 754** format is 64 zero bits +The binary representation of 0.0 in **IEEE 754** format is 64 zero bits ```python3 f = 0.0 @@ -58,13 +58,13 @@ f = 0.1 ## 1.1 -the difference between 1.1 and 0.1 is the last few exponent bits +The difference between 1.1 and 0.1 is the last few exponent bits ![1.1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/1.1.png) ## -0.1 -the difference between -0.1 and 0.1 is the first sign bit +The difference between -0.1 and 0.1 is the first sign bit ![-0.1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/-0.1.png) @@ -79,11 +79,11 @@ static PyFloatObject *free_list = NULL; ``` -free_list is a single linked list, store at most PyFloat_MAXFREELIST **PyFloatObject** +free_list is a singly linked list that stores at most PyFloat_MAXFREELIST **PyFloatObject**s ![free_list](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/free_list.png) -The linked list is linked via the field **ob_type** +The linked list is linked via the **ob_type** field ```python3 >>> f = 0.0 diff --git a/BasicObject/func/func.md b/BasicObject/func/func.md index 6556db6..6afde09 100644 --- a/BasicObject/func/func.md +++ b/BasicObject/func/func.md @@ -24,7 +24,7 @@ # memory layout -everything is an object in python, including function, a function is defined as **PyFunctionObject** in the c level +Everything is an object in Python, including functions. A function is defined as **PyFunctionObject** at the C level ```python3 def f(a, b=2): @@ -41,16 +41,15 @@ the type **function** indicates the user-defined method/classes, for **builtin_f # field -let's figure out the meaning of each field in the **PyFunctionObject** +Let's figure out the meaning of each field in **PyFunctionObject** ## func_code -**func_code** field stores an instance of **PyCodeObject**, which contains information of a code block +The **func_code** field stores an instance of **PyCodeObject**, which contains information about a code block. -A code block must contain python virtual machine's instruction, argument number, argument -body and etc +A code block contains the Python virtual machine's instructions, argument count, argument body, etc. -I will explain **PyCodeObject** in other article +I will explain **PyCodeObject** in another article ```python3 >>> f.__code__ @@ -60,7 +59,7 @@ I will explain **PyCodeObject** in other article ## func_globals -the global namespace attached to the function object +The global namespace attached to the function object ```python3 >>> type(f.__globals__) @@ -72,7 +71,7 @@ the global namespace attached to the function object ## func_defaults -a tuple stores all the default argument of the function object +A tuple that stores all the default arguments of the function object ```python3 >>> f.__defaults__ @@ -82,7 +81,7 @@ a tuple stores all the default argument of the function object ## func_kwdefaults -field **func_kwdefaults** is a python dictionary, which stores the [keyword-only argument](https://www.python.org/dev/peps/pep-3102/) with default value +The field **func_kwdefaults** is a Python dictionary that stores [keyword-only arguments](https://www.python.org/dev/peps/pep-3102/) with default values ```python3 def f2(a, b=4, *x, key=111): @@ -95,7 +94,7 @@ def f2(a, b=4, *x, key=111): ## func_closure -the **func_closure** field is a tuple, indicate all the enclosing level of the current function object +The **func_closure** field is a tuple that indicates all the enclosing levels of the current function object ```python3 def wrapper(func): @@ -164,7 +163,7 @@ call done ## func_doc -usually, it's an **unicode** object for explanation +Usually, it's a **unicode** object for documentation ```python3 def f(): @@ -179,7 +178,7 @@ print(f.__doc__) ## func_name -the name of the **PyFunctionObject** object +The name of the **PyFunctionObject** object ```python3 def i_have_a_very_long_long_long_name(): @@ -194,7 +193,7 @@ print(i_have_a_very_long_long_long_name.__name__) ## func_dict -**func_dict** field stores the attribute of the function object +The **func_dict** field stores the attributes of the function object ```python3 >>> f.__dict__ @@ -207,7 +206,7 @@ print(i_have_a_very_long_long_long_name.__name__) ## func_module -**func_module** field indicate the module which the **PyFunctionObject** attached to +The **func_module** field indicates the module to which the **PyFunctionObject** is attached ```python3 >>> f.__module__ @@ -233,7 +232,7 @@ def a(x: "I am a int" = 3, y: "I am a float" = 4) -> "return a list": ## func_qualname -it's used for nested class/function representation, it contains a dotted path leading to the object from the module top-level, refer [PEP 3155 -- Qualified name for classes and functions](https://www.python.org/dev/peps/pep-3155/) for more detail +It's used for nested class/function representation. It contains a dotted path leading to the object from the module top-level. Refer to [PEP 3155 -- Qualified name for classes and functions](https://www.python.org/dev/peps/pep-3155/) for more detail ```python3 def f(): diff --git a/BasicObject/gen/gen.md b/BasicObject/gen/gen.md index b57339c..d730603 100644 --- a/BasicObject/gen/gen.md +++ b/BasicObject/gen/gen.md @@ -22,7 +22,7 @@ ## memory layout generator -there's a common defination among **generator**, **coroutine** and **async generator** +There's a common definition among **generator**, **coroutine**, and **async generator** ```c #define _PyGenObject_HEAD(prefix) \ @@ -43,7 +43,7 @@ there's a common defination among **generator**, **coroutine** and **async gener ``` -the definition of **generator** object is less than 4 lines +The definition of the **generator** object is less than 4 lines ```c typedef struct { @@ -68,13 +68,13 @@ typedef struct { ``` -we can draw the layout according to the code now +We can draw the layout according to the code now ![layout_gen](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/layout_gen.png) ## example generator -let's define and iter through a generator +Let's define and iterate through a generator ```python3 def fib(n): @@ -118,7 +118,7 @@ def fib(n): ``` -we initialize a new generator, the **f_lasti** in **gi_frame** act as the program counter in the python virtual machine, it indicates the next instruction offset from the code block inside the **gi_code** +We initialize a new generator. The **f_lasti** in **gi_frame** acts as the program counter in the Python virtual machine; it indicates the next instruction offset from the code block inside **gi_code** ```python3 >>> fib.__code__ @@ -128,11 +128,11 @@ we initialize a new generator, the **f_lasti** in **gi_frame** act as the progra ``` -the **gi_code** inside the f object is the **code** object that represents the function fib +The **gi_code** inside the f object is the **code** object that represents the function fib. -the **gi_running** is 0, indicating the generator is not executing right now +The **gi_running** is 0, indicating the generator is not executing right now. -**gi_name** and **gi_qualname** all points to same **unicode** object, all fields in **gi_exc_state** have value 0x00 +**gi_name** and **gi_qualname** both point to the same **unicode** object; all fields in **gi_exc_state** have value 0x00 ![example_gen_0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/example_gen_0.png) @@ -146,13 +146,13 @@ result None ``` -looking into the object f, nothing changed +Looking into the object f, nothing changed. -but the **f_lasti** in **gi_frame** now in the position 52(the first place keyword **yield** appears) +But the **f_lasti** in **gi_frame** is now at position 52 (the first place the keyword **yield** appears) ![example_gen_1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/example_gen_1.png) -step one more time, due to the while loop, the **f_lasti** still points to the same position +Step one more time. Due to the while loop, the **f_lasti** still points to the same position ```python3 >>> r = f.send("handsome") @@ -164,7 +164,7 @@ result 'handsome' ``` -send again, the **f_lasti** indicate the code offset in the position of second **yield** +Send again. The **f_lasti** indicates the code offset at the position of the second **yield** ```python3 >>> r = f.send("handsome2") @@ -200,9 +200,9 @@ result 'handsome5' ``` -now, the while loop terminated by the break statement +Now the while loop is terminated. -the **f_lasti** is in the position of the first **except** statement, the **exc_type** points to the type of the exception, **exc_value** points to the instance of the exception, and **exc_traceback** points to the traceback object +The **f_lasti** is at the position of the first **except** statement. **exc_type** points to the type of the exception, **exc_value** points to the instance of the exception, and **exc_traceback** points to the traceback object ```python3 >>> r = f.send("handsome6") @@ -215,7 +215,7 @@ the **f_lasti** is in the position of the first **except** statement, the **exc_ ![example_gen_2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/example_gen_2.png) -the **f_lasti** is in the position of the second **except** statement, **exc_type**, **exc_value**, and **exc_traceback** now relate to ModuleNotFoundError +The **f_lasti** is at the position of the second **except** statement. **exc_type**, **exc_value**, and **exc_traceback** now relate to ModuleNotFoundError ```python3 >>> r = f.send("handsome7") @@ -229,9 +229,9 @@ the **f_lasti** is in the position of the second **except** statement, **exc_typ ![example_gen_3](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/example_gen_3.png) -the **f_lasti** is in the position of the first **finally** statement, the ModuleNotFoundError is handled properly, at the top of the exception stack is the **ZeroDivisionError** +The **f_lasti** is at the position of the first **finally** statement. The ModuleNotFoundError is handled properly; at the top of the exception stack is the **ZeroDivisionError**. -actually the information about [exception handling](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/exception.md) is stored in [frame object](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame.md), **gi_exec_state** is used for representing whether current generator ojbect is handling exception and the detail of the most nested exception +Actually, the information about [exception handling](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/exception.md) is stored in [frame object](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame.md). **gi_exc_state** is used for representing whether the current generator object is handling an exception and the details of the most nested exception ```python3 >>> r = f.send("handsome8") @@ -245,13 +245,13 @@ result 'handsome8' ![example_gen_4](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/example_gen_4.png) -now, the **StopIteration** is raised +Now the **StopIteration** is raised. -the frameObject in **gi_frame** field is freed +The frameObject in the **gi_frame** field is freed. -field **gi_frame** points to a null pointer, indicating that the generator is terminated +The field **gi_frame** points to a null pointer, indicating that the generator is terminated. -and states in **gi_exc_state** is restored +The state in **gi_exc_state** is restored ```python3 >>> r = f.send("handsome9") @@ -278,17 +278,17 @@ AttributeError: 'NoneType' object has no attribute 'f_lasti' ## memory layout coroutine -most parts of the definition of the **coroutine** type and **generator** are the same +Most parts of the definition of the **coroutine** type and **generator** are the same. -the coroutine-only field named **cr_origin**, tracking the trackback of the **coroutine** object, is disabled by default, can be enabled by **sys.set_coroutine_origin_tracking_depth**, for more detail please refer to [docs.python.org(set_coroutine_origin_tracking_depth)](https://docs.python.org/3/library/sys.html#sys.set_coroutine_origin_tracking_depth) +The coroutine-only field named **cr_origin**, which tracks the traceback of the **coroutine** object, is disabled by default. It can be enabled by **sys.set_coroutine_origin_tracking_depth**. For more detail, please refer to [docs.python.org(set_coroutine_origin_tracking_depth)](https://docs.python.org/3/library/sys.html#sys.set_coroutine_origin_tracking_depth) ![layout_coro](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/layout_coro.png) ## example coroutine -let's try to run an example with **coroutine** type defined to understand each field's meaning +Let's try to run an example with the **coroutine** type defined to understand each field's meaning. -as usual, I've altered the source code so that my **repr** function is able to print all the low-level detail of the object +As usual, I've altered the source code so that my **repr** function is able to print all the low-level details of the object ```python3 import sys @@ -336,7 +336,7 @@ if __name__ == "__main__": ``` -if you call a function defined with the **async** keyword, the calling result is an object of type **coroutine** +If you call a function defined with the **async** keyword, the result is an object of type **coroutine** ```python3 >>> c = cor() @@ -354,7 +354,7 @@ in the **test** function, before the first **await** statement at the moment ``` -the content in field **cr_origin** in my computer is the calling stack from bottom to top +The content in field **cr_origin** on my computer is the calling stack from bottom to top ![example_coro_0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/example_coro_0.png) @@ -388,15 +388,15 @@ in the 6.01 seconds, both **cor_list[0]** and **cor_list[1]** returned, and thei ## memory layout async generator -the layout of **async generator** is the same as **generator** type, except for the **ag_finalizer**, **ag_hooks_inited** and **ag_closed** +The layout of **async generator** is the same as the **generator** type, except for **ag_finalizer**, **ag_hooks_inited**, and **ag_closed** ![layout_async_gen](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/layout_async_gen.png) ## example async generator -the **set_asyncgen_hooks** function is used for setting up a **firstiter** and a **finalizer**, **firstiter** will be called before when an asynchronous generator is iterated for the first time, finalizer will be called when asynchronous generator is about to be gc +The **set_asyncgen_hooks** function is used for setting up a **firstiter** and a **finalizer**. **firstiter** will be called when an asynchronous generator is iterated for the first time; **finalizer** will be called when an asynchronous generator is about to be garbage collected. -the **run_forever** function in asyncio base event loop has defined +The **run_forever** function in the asyncio base event loop has defined ```python3 def run_forever(self): @@ -434,7 +434,7 @@ in firstiter: ``` -let's define and iter through an async iterator +Let's define and iterate through an async iterator ```python3 import asyncio diff --git a/BasicObject/iter/iter.md b/BasicObject/iter/iter.md index f2c6207..2af6290 100644 --- a/BasicObject/iter/iter.md +++ b/BasicObject/iter/iter.md @@ -29,9 +29,9 @@ ## memory layout iter -The python sequence iterator is a wrapper of a python object with _\_getitem_\_ defined +The Python sequence iterator is a wrapper of a Python object with _\_getitem_\_ defined. -so the sequence iterator can iter through the target object by calling the object[index], and set index to index + 1 +So the sequence iterator can iterate through the target object by calling object[index], and set index to index + 1 ```c static PyObject * @@ -75,7 +75,7 @@ iter_iternext(PyObject *iterator) ### iter0 -let's iter through an iterator object +Let's iterate through an iterator object ```python3 class A(object): @@ -98,7 +98,7 @@ type(a) # iterator ### iter1 -the **next(a)** calls object[0], and return the result +**next(a)** calls object[0] and returns the result ```python3 >>> next(a) @@ -120,7 +120,7 @@ the **next(a)** calls object[0], and return the result ### iter3 -the current **it_index** is 2, so the next(a) calls object[2] which returns 4 +The current **it_index** is 2, so next(a) calls object[2] which returns 4 ```python3 >>> next(a) @@ -152,9 +152,9 @@ the current **it_index** is 2, so the next(a) calls object[2] which returns 4 ### iter end -now, the **it_idnex** is 5, next(a) will raise an IndexError +Now the **it_index** is 5, and next(a) will raise an IndexError. -the content in **it_index** is still 5, but the content in **it_seq** becomes 0x00, which indicate end of iteration +The content in **it_index** is still 5, but the content in **it_seq** becomes 0x00, which indicates end of iteration ```python3 >>> next(a) @@ -167,7 +167,7 @@ StopIteration ![iterend](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/iter/iterend.png) -notice, if you call next(a) again, the **"raise by myself"** won't be printed, since the **it_seq** field no longer points to the instance of **class A**, it loses the way to access the instance of **class A**, and can no longer call the _\_getitem_\_ function +Notice: if you call next(a) again, **"raise by myself"** won't be printed, since the **it_seq** field no longer points to the instance of **class A**. It has lost the way to access the instance of **class A** and can no longer call the _\_getitem_\_ function ```python3 >>> next(a) @@ -219,7 +219,7 @@ calliter_iternext(calliterobject *it) ``` -a callable_iterator calls whatever stores in **it_callable** each time you iter through it, until the result match the **it_sentinel** or a StopIteration raised +A callable_iterator calls whatever is stored in **it_callable** each time you iterate through it, until the result matches **it_sentinel** or a StopIteration is raised ## example citer @@ -246,9 +246,9 @@ type(r) # callable_iterator ### citer1 -the callable_iterator calls the _\_call_\_ method of the instance of A, and compare the result with the sentinel **2** +The callable_iterator calls the _\_call_\_ method of the instance of A and compares the result with the sentinel **2**. -the next operation won't change any state of the callable_iterator object, the state inside **it_callable** is changed anyway +The next operation won't change any state of the callable_iterator object; the state inside **it_callable** is changed instead ```python3 >>> next(r) @@ -260,9 +260,9 @@ the next operation won't change any state of the callable_iterator object, the s ### citer end -this time the instance of A returns PyLongObject 2, which is the same as the object stored inside **it_sentinel** field, +This time the instance of A returns PyLongObject 2, which is the same as the object stored in the **it_sentinel** field. -so the callable_iterator clears its state and return NULL to the outer scope, which raise a StopIteration +So the callable_iterator clears its state and returns NULL to the outer scope, which raises a StopIteration ```python3 >>> next(r) diff --git a/BasicObject/list/list.md b/BasicObject/list/list.md index 2610b05..0e91c13 100644 --- a/BasicObject/list/list.md +++ b/BasicObject/list/list.md @@ -36,7 +36,7 @@ l = list() ``` -field `ob_size` stores the actual size, it's type is `Py_ssize_t` which is usually 64 bit, `1 << 64` can represent a very large size, usually you will run out of RAM before overflow the `ob_size` field +The field `ob_size` stores the actual size. Its type is `Py_ssize_t` which is usually 64 bits. `1 << 64` can represent a very large size; usually you will run out of RAM before overflowing the `ob_size` field ![list_empty](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/list_empty.png) @@ -85,9 +85,9 @@ you can see that it's actually more like a `C++ vector` # pop -the append operation will only trigger the `list`'s resize when the `list` is full +The append operation will only trigger the `list`'s resize when the `list` is full -what about pop ? +What about pop? ```python3 >>> l.pop() @@ -103,9 +103,9 @@ what about pop ? ``` -the realloc is called and the malloced size is shrinked, actually the resize function will be called each time you call `pop` +The realloc is called and the malloced size is shrunk. Actually, the resize function will be called each time you call `pop` -but the actual realloc will be called only if the newsize falls lower than half the allocated size +But the actual realloc will be called only if the newsize falls lower than half the allocated size ```python3 /* cpython/Objects/listobject.c */ @@ -131,7 +131,7 @@ new_allocated = (size_t)newsize + (newsize >> 3) + (newsize < 9 ? 3 : 6); ## timsort -the algorithm CPython used in sorting `list` is **timsort**, it's quiet complicated +The algorithm CPython uses for sorting `list` is **timsort**. It's quite complicated ```python3 >>> l = [5, 9, 17, 11, 10, 14, 2, 8, 12, 19, 4, 13, 3, 0, 16, 1, 6, 15, 18, 7] @@ -145,11 +145,11 @@ a structure named `MergeState` is created for helping the **timsort** algorithm ![MergeState](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/MergeState.png) -this is the state after preparing +This is the state after preparing ![sort_begin1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/sort_begin1.png) -assume `minrun` is 5, we will see what `minrun` is and how `minrun` calculated later, for now, we run the sort algorithm and ignore these details for illustration +Assume `minrun` is 5. We will see what `minrun` is and how `minrun` is calculated later. For now, we run the sort algorithm and ignore these details for illustration ![sort_begin2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/sort_begin2.png) @@ -165,7 +165,7 @@ after `binary_sort` the second group the second group is sorted by [binary_sort](#binary_sort), and the next index of `pending` stores the information of the second group -we can learn from the above graph that `pending` act as a stack, every time a group is sorted, the group info will be pushed onto this stack, a function named `merge_collapse` will be called after the psuh operation +We can learn from the above graph that `pending` acts as a stack. Every time a group is sorted, the group info will be pushed onto this stack, and a function named `merge_collapse` will be called after the push operation ```python3 /* cpython/Objects/listobject.c */ @@ -226,7 +226,7 @@ merge_collapse(MergeState *ms) ``` -the current state is case 3, `merge_at` will merge the two runs at stack indices `i` and `i+1` +The current state is case 3. `merge_at` will merge the two runs at stack indices `i` and `i+1` ## merge_at @@ -234,7 +234,7 @@ the current state is case 3, `merge_at` will merge the two runs at stack indices ![ms](https://github.com/zpoint/Algorithms/blob/master/screenshots/ms.gif) -after `merge_collapse`, the first two runs are merged and valid length of `pending` becomes 1 +After `merge_collapse`, the first two runs are merged and the valid length of `pending` becomes 1 ![sort_begin5](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/sort_begin5.png) @@ -242,13 +242,13 @@ after [binary_sort](#binary_sort) the next run ![sort_begin6](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/sort_begin6.png) -the `merge_collapse` won't merge any of the `run` because the stack invariants are good +The `merge_collapse` won't merge any of the `run`s because the stack invariants are satisfied after [binary_sort](#binary_sort) the final `run` ![sort_begin7](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/sort_begin7.png) -it meets `case 1`, and the last two `run` will be merged first +It meets `case 1`, and the last two `run`s will be merged first ![sort_begin8](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/sort_begin8.png) @@ -258,15 +258,15 @@ in the while loop of `merge_collapse`, merge will happen again in `case 3`, afte ## galloping mode -if we are merging these two arrays +If we are merging these two arrays ![galloping_mode0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/galloping_mode0.png) -we can use binary search to find the the max element that is smaller than the first element in the right, and copy these elements together instead of merging one by one +We can use binary search to find the max element that is smaller than the first element in the right, and copy these elements together instead of merging one by one ![galloping_mode1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/galloping_mode1.png) -we can also do that from right to left +We can also do that from right to left ![galloping_mode2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/galloping_mode2.png) @@ -274,17 +274,17 @@ read more detail in [listsort.txt](https://github.com/python/cpython/blob/master ## binary_sort -before delegating the call to `binary_sort`, a function named `count_run` will be called first to calculate the longest increasing or descending subarray begin at index 0 +Before delegating the call to `binary_sort`, a function named `count_run` will be called first to calculate the longest increasing or descending subarray beginning at index 0 ![binary_sort0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort0.png) so that `binary_sort` can sort the range begin at index 2, and end at index 4 -the subarray before `start` is sorted, and we need to sort all the rest elements from `start` to the end +The subarray before `start` is sorted, and we need to sort all the remaining elements from `start` to the end ![binary_sort1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort1.png) -first we set a variable `pivot`'s value to `start`'s value, perform binary search in the left subarray to find the first element that is greater than `pivot` +First we set a variable `pivot`'s value to `start`'s value, then perform binary search in the left subarray to find the first element that is greater than `pivot` ```python3 do { @@ -297,33 +297,33 @@ first we set a variable `pivot`'s value to `start`'s value, perform binary searc ``` -then we move every element from `l` to `start` forawrd, and set element in `start` to `pivot` +Then we move every element from `l` to `start` forward, and set the element at `start` to `pivot` ![binary_sort2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort2.png) -we perform binary search again, this time element in index 2 is selected, we move every element from the selected index to `start` forward again +We perform binary search again. This time the element at index 2 is selected, and we move every element from the selected index to `start` forward again ![binary_sort3](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort3.png) -and set the value in selected element's index to `pivot` +And set the value at the selected element's index to `pivot` ![binary_sort4](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort4.png) -we need to perform the final binary search +We need to perform the final binary search ![binary_sort5](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort5.png) -the selected element is in index 2, after move every element from `l` to `start` forawrd +The selected element is at index 2. After moving every element from `l` to `start` forward ![binary_sort6](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort6.png) -and set element in `start` to `pivot`, we've finished the binary_sort algorithm +And set the element at `start` to `pivot`, we've finished the binary_sort algorithm ![binary_sort7](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort7.png) ## run -actually `minrun` will be computed in the following function, if the current `run` number is lower than 64, it will be [binary_sort](#binary_sort) directly, else half of it's size will be shrink until there's a result size lower than 64 +Actually `minrun` will be computed in the following function. If the current `run` number is lower than 64, it will be sorted with [binary_sort](#binary_sort) directly. Otherwise, half of its size will be shrunk until there's a result size lower than 64 I've changed this constant to a smaller value so that the example above can fit into my graph @@ -361,11 +361,11 @@ static int numfree = 0; ``` -there exists per process global variable named **free_list** +There exists a per-process global variable named **free_list** ![free_list0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/free_list0.png) -if we create a new `list` object, the memory request is delegate to CPython's [memory management system](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management.md) +If we create a new `list` object, the memory request is delegated to CPython's [memory management system](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management.md) ```python3 a = list() @@ -379,11 +379,11 @@ del a ``` -the destructor of the `list` type will stores the current `list` to **free_list** (if **free_list** is not full) +The destructor of the `list` type will store the current `list` in **free_list** (if **free_list** is not full) ![free_list2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/free_list2.png) -next time you create a new `list` object, **free_list** will be checked if there is available objects you can use directly, if so, allocate from **free_list**, if not, allocate from CPython's [memory management system](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management.md) +Next time you create a new `list` object, **free_list** will be checked for available objects you can use directly. If available, allocate from **free_list**; if not, allocate from CPython's [memory management system](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management.md) ```python3 b = list() @@ -392,7 +392,7 @@ b = list() ![free_list3](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/free_list3.png) -by caching `list` objects in **free_list** can +Caching `list` objects in **free_list** can * improve performance * reduce memory fragmentation diff --git a/BasicObject/list/list_cn.md b/BasicObject/list/list_cn.md index 6cab3a0..1cae3b4 100644 --- a/BasicObject/list/list_cn.md +++ b/BasicObject/list/list_cn.md @@ -129,7 +129,7 @@ new_allocated = (size_t)newsize + (newsize >> 3) + (newsize < 9 ? 3 : 6); ## timsort -CPyton 用来对 `list` 对象排序的算法名称叫做 **timsort**, 它有一点复杂 +CPython 用来对 `list` 对象排序的算法名称叫做 **timsort**, 它有一点复杂 ```python3 >>> l = [5, 9, 17, 11, 10, 14, 2, 8, 12, 19, 4, 13, 3, 0, 16, 1, 6, 15, 18, 7] @@ -147,7 +147,7 @@ CPyton 用来对 `list` 对象排序的算法名称叫做 **timsort**, 它有一 ![sort_begin1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/sort_begin1.png) -假设 `minrun` 为 5, 我们后面会解释 `minrun` 是什么并且是怎么计算的, 现在我们先忽略一部分细节, 跑一边算法看看 +假设 `minrun` 为 5, 我们后面会解释 `minrun` 是什么并且是怎么计算的, 现在我们先忽略一部分细节, 跑一遍算法看看 ![sort_begin2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/sort_begin2.png) @@ -163,7 +163,7 @@ CPyton 用来对 `list` 对象排序的算法名称叫做 **timsort**, 它有一 第二组内的元素被 [binary_sort](#binary_sort) 排好序了, 并且 `pending` 的下一个空闲的位置存储了第二组相关的信息 -我们从上图可以发现, `pending` 在这里的作用和调用栈的 stack 类似, 每次给下一组拍完序之后, 这组相关的信息就会被压入这个栈中, 每次压入后, 一个名为 `merge_collapse` 的函数都会被调用 +我们从上图可以发现, `pending` 在这里的作用和调用栈的 stack 类似, 每次给下一组排完序之后, 这组相关的信息就会被压入这个栈中, 每次压入后, 一个名为 `merge_collapse` 的函数都会被调用 ```python3 /* cpython/Objects/listobject.c */ @@ -294,7 +294,7 @@ merge_collapse(MergeState *ms) ``` -之后我们把 `l` 到 `start` 的每一个元素都往前移一格, 移完后把 `start` 这个位置的元素值设置为 `pivot` 的值 +之后我们把 `l` 到 `start` 的每一个元素都往后移一格, 移完后把 `start` 这个位置的元素值设置为 `pivot` 的值 ![binary_sort2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort2.png) @@ -310,7 +310,7 @@ merge_collapse(MergeState *ms) ![binary_sort5](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort5.png) -搜索结果为下标为 2, 在把 `l` 到 `start` 的每个值前移一格后 +搜索结果为下标为 2, 在把 `l` 到 `start` 的每个值后移一格后 ![binary_sort6](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/binary_sort6.png) @@ -395,4 +395,4 @@ b = list() * 减小内存碎片 # read more -* [Timsort——自适应、稳定、高效排序算法](https://blog.csdn.net/sinat_35678407/article/details/82974174) \ No newline at end of file +* [Timsort——自适应、稳定、高效排序算法](https://blog.csdn.net/sinat_35678407/article/details/82974174) diff --git a/BasicObject/long/long.md b/BasicObject/long/long.md index 0eaeffd..91815a7 100644 --- a/BasicObject/long/long.md +++ b/BasicObject/long/long.md @@ -26,17 +26,17 @@ Thanks @MambaWong for pointing out the errors [#22](https://github.com/zpoint/C ![memory layout](https://img-blog.csdnimg.cn/20190314164305131.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNzIwMzI5,size_16,color_FFFFFF,t_70) -after python3, there's only type named **int**, the **long** type in python2.x is **int** type in python3.x +After Python 3, there's only a type named **int**. The **long** type in Python 2.x is the **int** type in Python 3.x. -the structure of **long object** looks like the structure of [tuple object](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md#memory-layout), obviously, there's only one field to store the real **int** value, that's **ob_digit** +The structure of **long object** looks like the structure of [tuple object](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md#memory-layout). Obviously, there's only one field to store the real **int** value: **ob_digit**. -But how does CPython represent the variable size **int** in byte level? Let's see +But how does CPython represent the variable-size **int** at the byte level? Let's see # how is element stored inside ## ingeter 0 -notice, when the value is 0, the **ob_digit** field doesn't store anything, the value 0 in **ob_size** indicate that **long object** represent integer 0 +Notice: when the value is 0, the **ob_digit** field doesn't store anything. The value 0 in **ob_size** indicates that the **long object** represents integer 0 ```python3 i = 0 @@ -47,7 +47,7 @@ i = 0 ## ingeter 1 -there are two different types of **ob_digit** depends on your system. +There are two different types of **ob_digit** depending on your system. ```c #if PYLONG_BITS_IN_DIGIT == 30 @@ -71,7 +71,7 @@ typedef long stwodigits; /* signed variant of twodigits */ I've modified the source code to change the value of **PYLONG_BITS_IN_DIGIT** to 15 -but when it's going to represent integer 1, **ob_size** becomes 1 and field in **ob_digit** represent the value 1 with type **unsigned short** +But when it represents integer 1, **ob_size** becomes 1 and the field in **ob_digit** represents the value 1 with type **unsigned short** ```python3 i = 1 @@ -82,7 +82,7 @@ i = 1 ## ingeter -1 -when i becomes -1, the only difference from the integer 1 is the value in **ob_size** field, CPython interpret **ob_size** as a signed type to differ the positive and negative sign +When i becomes -1, the only difference from integer 1 is the value in the **ob_size** field. CPython interprets **ob_size** as a signed type to differentiate between positive and negative signs ```python3 i = -1 @@ -93,43 +93,43 @@ i = -1 ## ingeter 1023 -the basic unit is type **digit**, which provide 2 bytes(16bits) for storage. And 1023 takes the rightmost 10 bits, -so the value **ob_size** field is still 1. +The basic unit is type **digit**, which provides 2 bytes (16 bits) for storage. Since 1023 takes the rightmost 10 bits, +the value of the **ob_size** field is still 1. ![1023](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/1023.png) ## ingeter 32767 -the integer 32767 represent in the same way as usual +The integer 32767 is represented in the same way as usual ![32767](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/32767.png) ## ingeter 32768 -CPython don't use all the 16 bits in **digit** field, the first bit of every **digit** is reserved, we will see why later +CPython doesn't use all the 16 bits in the **digit** field. The first bit of every **digit** is reserved; we will see why later ![32768](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/32768.png) ## little endian and big endian -notice, because the **digit** is the smallest unit in the CPython abstract level, The order between bytes inside a single ob_digit is the same as your machine order(mine is little endian) +Notice: because **digit** is the smallest unit at the CPython abstract level, the order between bytes inside a single ob_digit is the same as your machine order (mine is little endian). -Order between **digit** in the **ob_digit** array are represent as most-significant-digit-in-the-right-most order(little endian order) +The order between **digit**s in the **ob_digit** array is represented in most-significant-digit-in-the-rightmost order (little endian order). -we can have a better understanding with the integer value -262143 +We can have a better understanding with the integer value -262143. -the minus sign is stored in the **ob_size** field +The minus sign is stored in the **ob_size** field. -the interger 262143(2^18 = 262144) in binary representation is 00000011 11111111 11111111 +The integer 262143 (2^18 = 262144) in binary representation is 00000011 11111111 11111111 ![262143](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/262143.png) ## reserved bit -why the left-most bit in **digit** is reserved? Why order between **digit** in the **ob_digit** array are represented as little-endian? +Why is the leftmost bit in **digit** reserved? Why is the order between **digit**s in the **ob_digit** array represented as little-endian? -let's try to add two integer value +Let's try to add two integer values ```python3 i = 1073741824 - 1 # 1 << 30 == 1073741824 @@ -144,48 +144,48 @@ k = i + j ``` -first, initialize a temporary **PyLongObject** with size = max(size(i), size(j)) + 1 +First, initialize a temporary **PyLongObject** with size = max(size(i), size(j)) + 1 ![temp_add](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/temp_add.png) -step1, sum the firt **digit** in each **ob_digit** array to a variable named **carray** +Step 1: sum the first **digit** in each **ob_digit** array into a variable named **carry** ![step_1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_1.png) -step2, set the value in temp[0] to (carry & PyLong_MASK) +Step 2: set the value in temp[0] to (carry & PyLong_MASK) ![step_2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_2.png) -step3, right shift the carray up to the leftmost bit +Step 3: right shift the carry up to the leftmost bit ![step_3_rshift](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_3_rshift.png) -step4, add the second **digit** in each **ob_digit** array to the result of **carray** +Step 4: add the second **digit** in each **ob_digit** array to the result of **carry** ![step_4](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_4.png) -step5, set the value in temp[1] to (carry & PyLong_MASK) +Step 5: set the value in temp[1] to (carry & PyLong_MASK) ![step_4](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_5.png) -step6, right shift the carray again +Step 6: right shift the carry again ![step_3_rshift](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_3_rshift.png) -go to step4 and repeat until no more **digit** left, set the final carray to the last index of temp +Go to step 4 and repeat until no more **digit** is left. Set the final carry to the last index of temp ![step_final](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_final.png) -the variable temp contains the sum, now, you see the reserved bit is used for the **carry** or **borrow** when you add/sub an integer, the **digit** in **ob_digit** array are stored in little-endian order so that the add/sub operation can process each **digit** from left to right +The variable temp contains the sum. Now you see the reserved bit is used for the **carry** or **borrow** when you add/subtract an integer. The **digit**s in the **ob_digit** array are stored in little-endian order so that the add/subtract operation can process each **digit** from left to right. -the sub operation is similar to the add operation, so you can read the source code directly +The subtraction operation is similar to the addition operation, so you can read the source code directly ![k](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/k.png) ## small ints -CPython also use a buffer pool to store the frequently used integer +CPython also uses a buffer pool to store frequently used integers ```c diff --git a/BasicObject/long/long_cn.md b/BasicObject/long/long_cn.md index b61006e..a4d86e8 100644 --- a/BasicObject/long/long_cn.md +++ b/BasicObject/long/long_cn.md @@ -149,7 +149,7 @@ k = i + j ![temp_add](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/temp_add.png) -第一步, 把两个数 **ob_digit** 里的第一个坑位里的 **digit** 加起来, 并加到 **carray** 里 +第一步, 把两个数 **ob_digit** 里的第一个坑位里的 **digit** 加起来, 并加到 **carry** 里 ![step_1](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_1.png) @@ -157,11 +157,11 @@ k = i + j ![step_2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_2.png) -第三步, 右移 carray, 值保留下最左边的一位(这一位其实就是之前两个数相加的进位) +第三步, 右移 carry, 值保留下最左边的一位(这一位其实就是之前两个数相加的进位) ![step_3_rshift](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_3_rshift.png) -第四步, 把两边下一个 **ob_digit** 的对应位置的值加起来, 并把结果与 **carray** 相加 +第四步, 把两边下一个 **ob_digit** 的对应位置的值加起来, 并把结果与 **carry** 相加 ![step_4](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_4.png) @@ -173,7 +173,7 @@ k = i + j ![step_3_rshift](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_3_rshift.png) -回到步骤四, 直到两边都没有剩余的 **digit** 可以相加为止, 把最后的 carray 存储到 temp 最后一格 +回到步骤四, 直到两边都没有剩余的 **digit** 可以相加为止, 把最后的 carry 存储到 temp 最后一格 ![step_final](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/step_final.png) @@ -212,4 +212,4 @@ print(id(f), id(g)) # 4484604176 4484604016 ``` -![small_ints](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/small_ints.png) \ No newline at end of file +![small_ints](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/small_ints.png) diff --git a/BasicObject/method/method.md b/BasicObject/method/method.md index bad1bd6..d5deab2 100644 --- a/BasicObject/method/method.md +++ b/BasicObject/method/method.md @@ -19,7 +19,7 @@ # memory layout -There's a type named **builtin_function_or_method** in python, as the type name described, all the builtin function or method defined in the c level belong to **builtin_function_or_method** +There's a type named **builtin_function_or_method** in Python. As the type name describes, all built-in functions or methods defined at the C level belong to **builtin_function_or_method** ```python3 >>> print @@ -35,7 +35,7 @@ There's a type named **builtin_function_or_method** in python, as the type name ## print -let's read code snippet first +Let's read a code snippet first ```c #define PyCFunction_Check(op) (Py_TYPE(op) == &PyCFunction_Type) @@ -51,7 +51,7 @@ typedef PyObject *(*PyNoArgsFunction)(PyObject *); ``` -The **PyCFunction** is a type in c, any c function with signature(accept two PyObject* as parameters and return a PyObject *) can be cast to type **PyCFunction** +**PyCFunction** is a type in C. Any C function with the signature (accepting two PyObject* as parameters and returning a PyObject*) can be cast to type **PyCFunction** ```c // a c function named builtin_print @@ -69,15 +69,15 @@ static PyMethodDef builtin_methods[] = { ![print](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/method/print.png) -a **PyMethodDef** should be attached to a **module** object, and a **self** arugment, and then a **PyCFunctionObject** will be generated +A **PyMethodDef** should be attached to a **module** object with a **self** argument, and then a **PyCFunctionObject** will be generated. -what user really interactive with is **PyCFunctionObject** +What the user actually interacts with is **PyCFunctionObject** ![print2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/method/print2.png) -let's check for more detail of each field +Let's check each field for more detail. -the type in **m_self** field is **module**, and type in **m_module** field is **str** +The type in the **m_self** field is **module**, and the type in the **m_module** field is **str** ![print3](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/method/print3.png) @@ -85,17 +85,17 @@ the type in **m_self** field is **module**, and type in **m_module** field is ** ## ml_name -as you can see in the above picture, field **ml_name** is the name of the built-in method, it's a c style null terminated string "print" +As you can see in the above picture, the field **ml_name** is the name of the built-in method; it's a C-style null-terminated string "print" ## ml_meth -the real c function pointer that does the job +The actual C function pointer that does the job ## ml_flags -bit flag indicates how the c function's behave in the python level +A bit flag that indicates how the C function behaves at the Python level. -the function in **call.c** begin with **_PyMethodDef_** will delegate the work to the **PyCFunction**, but with different calling behaviour according to different **ml_flags** +The functions in **call.c** beginning with **_PyMethodDef_** will delegate work to the **PyCFunction**, but with different calling behavior according to different **ml_flags** for more detail please refer to [c-api Common Object Structures](https://docs.python.org/3/c-api/structures.html) @@ -120,12 +120,12 @@ static int numfree = 0; ``` -cpython use a buffer pool with size 256 to reuse the deallocated **PyCFunctionObject** object, free_list is a single linked list, all elements are chained by the **m_self** field +CPython uses a buffer pool with size 256 to reuse deallocated **PyCFunctionObject** objects. free_list is a singly linked list; all elements are chained by the **m_self** field. -the similar technique appears in float object, the float object chained through the **ob_type** field, I will not draw again, user who need the graph representation please click the link [float(free_list)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/float.md#free_list) +A similar technique appears in float objects. Float objects are chained through the **ob_type** field. I will not draw again; users who need the graphical representation please click the link [float(free_list)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/float.md#free_list) # classmethod # staticmethod -they're more related in **classobject**, I will talk about them later in [class object](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/class.md) \ No newline at end of file +They're more related to **classobject**. I will talk about them later in [class object](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/class.md) \ No newline at end of file diff --git a/BasicObject/set/set.md b/BasicObject/set/set.md index 37f4d5d..67844ac 100644 --- a/BasicObject/set/set.md +++ b/BasicObject/set/set.md @@ -27,7 +27,7 @@ * static PyObject * set_new(PyTypeObject *type, PyObject *args, PyObject *kwds) * static PyObject * make_new_set(PyTypeObject *type, PyObject *iterable) -the following picture shows value in each field in an empty set +The following picture shows the value of each field in an empty set ![make new set](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/make_new_set.png) @@ -38,8 +38,8 @@ the following picture shows value in each field in an empty set * static int set_add_entry(PySetObject *so, PyObject *key, Py_hash_t hash) -initialize a new empty set, whenever an empty set created, the **setentry** field points to **smalltable** field inside the same PySetObject, default value of **PySet_MINSIZE** is 8, it means if there are less than 8 objects in PySetObject, they are stored in the **smalltable**, if there are more than 8 objects, the resize operation will make **setentry** points to a new malloced block -(Actually, the first time resize operation takes place, the smalltable won't be filled, there is a factor to trigger the resize operation, please keep reading) +Initialize a new empty set. Whenever an empty set is created, the **setentry** field points to the **smalltable** field inside the same PySetObject. The default value of **PySet_MINSIZE** is 8, which means if there are fewer than 8 objects in PySetObject, they are stored in the **smalltable**. If there are more than 8 objects, the resize operation will make **setentry** point to a new malloced block. +(Actually, the first time a resize operation takes place, the smalltable won't be filled. There is a factor that triggers the resize operation; please keep reading) ```python3 s = set() @@ -48,7 +48,7 @@ initialize a new empty set, whenever an empty set created, the **setentry** fiel ![set_empty](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/set_empty.png) -the value in **mask** field is the size of malloced entries - 1, currently it's 7 +The value in the **mask** field is the size of malloced entries minus 1; currently it's 7 ```python3 s.add(0) # hash(0) & mask == 0 @@ -57,7 +57,7 @@ s.add(0) # hash(0) & mask == 0 ![set_add_0](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/set_add_0.png) -add a value 5 +Add a value 5 ```python3 s.add(5) # hash(5) & mask == 0 @@ -68,8 +68,8 @@ s.add(5) # hash(5) & mask == 0 * ### hash collision -add a value 16, because the mask is 7, hash(16) & 7 ===> 0 -cpython use **LINEAR_PROBES** to solve hash collision instead of **CHAIN**(linked list) +Add a value 16. Because the mask is 7, hash(16) & 7 ===> 0 +CPython uses **LINEAR_PROBES** to solve hash collisions instead of **CHAIN** (linked list) ```c // pseudocode @@ -95,7 +95,7 @@ while (1) ``` -the 0th position has been taken, so the **LINEAR_PROBES** take the next empty position, which is 1th position +The 0th position has been taken, so **LINEAR_PROBES** takes the next empty position, which is the 1st position ```python3 s.add(16) # hash(16) & mask == 0 @@ -104,7 +104,7 @@ s.add(16) # hash(16) & mask == 0 ![set_add_16](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/set_add_16.png) -the 0th position has been taken, the first time **LINEAR_PROBES** find the 1th position, which also has been taken, the second time **LINEAR_PROBES** find the 2th position which is empty. +The 0th position has been taken. The first time, **LINEAR_PROBES** finds the 1st position, which has also been taken. The second time, **LINEAR_PROBES** finds the 2nd position, which is empty. ```python3 s.add(32) # hash(32) & mask == 0 @@ -115,7 +115,7 @@ s.add(32) # hash(32) & mask == 0 * ## **resize** -let's insert one more item with value 64, still repeat the **LINEAR_PROBES** progress, insert to the position at index 3 +Let's insert one more item with value 64, still repeating the **LINEAR_PROBES** process, inserting at index 3 ```python3 s.add(64) # hash(64) & mask == 0 @@ -124,8 +124,8 @@ s.add(64) # hash(64) & mask == 0 ![set_add_64](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/set_add_64.png) -now, value in field **fill** is 5, **mask** is 7, it will trigger the resize operation -the trigger condition is **fill*5 !< mask * 3** +Now the value in field **fill** is 5 and **mask** is 7, which will trigger the resize operation. +The trigger condition is **fill*5 !< mask * 3** ```python3 // from cpython/Objects/setobject.c @@ -135,8 +135,8 @@ return set_table_resize(so, so->used>50000 ? so->used*2 : so->used*4); ``` -after resize,value 32 and 64 still repeat the **LINEAR_PROBES** progress, inserted at index 1 and index 2, because the value in **mask** field becomes 31, hash(16) is less than the mask, so 16 can be inserted to index 15 -And field **table** no longer points to **smalltable**, instead, it points to a new malloced block +After resize, values 32 and 64 still repeat the **LINEAR_PROBES** process, inserted at index 1 and index 2. Because the value in the **mask** field becomes 31 and hash(16) is less than the mask, 16 can be inserted at index 16. +The field **table** no longer points to **smalltable**; instead, it points to a new malloced block ![set_add_64_resize](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/set_add_64_resize.png) @@ -150,7 +150,7 @@ And field **table** no longer points to **smalltable**, instead, it points to a * static int set_clear_internal(PySetObject *so) * static void set_empty_to_minsize(PySetObject *so) -call clear to restore the initial state +Call clear to restore the initial state ```python3 s.clear() diff --git a/BasicObject/str/str.md b/BasicObject/str/str.md index d9816c6..db5ee68 100644 --- a/BasicObject/str/str.md +++ b/BasicObject/str/str.md @@ -17,7 +17,7 @@ # memory layout -you can read [How Python saves memory when storing strings](https://rushter.com/blog/python-strings-and-memory/) first to have an overview of how **unicode** stored internally +You can read [How Python saves memory when storing strings](https://rushter.com/blog/python-strings-and-memory/) first to have an overview of how **unicode** is stored internally ![layout](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/layout.png) @@ -25,7 +25,7 @@ for those who are interested in bit-fields in C please refer to [When to use bit # conversion -before we look into how unicode object create, resize, let's look into the c function **PyUnicode_AsUTF8** +Before we look into how unicode objects are created and resized, let's look at the C function **PyUnicode_AsUTF8** ```c /* Whenever I try to covert PyUnicodeObject to const char* pointer, @@ -122,7 +122,7 @@ calling PyUnicode_AsUTF8(unicode): s ``` -have you notice the field **utf8_length** in **PyCompactUnicodeObject** is 115, and chr(115) is 's', yes, this is the begin address of the c style null terminate string +Have you noticed the field **utf8_length** in **PyCompactUnicodeObject** is 115, and chr(115) is 's'? Yes, this is the beginning address of the C-style null-terminated string ![s_s](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/s_s.png) @@ -136,15 +136,15 @@ s = "aaa" # interned -all unicode object with "interned" set to 1 will be held in this dictionary, -any other new created unicode object with the same value will points to the same object, it act as singleton +All unicode objects with "interned" set to 1 will be held in this dictionary. +Any other newly created unicode object with the same value will point to the same object; it acts as a singleton ```c static PyObject *interned = NULL; ``` -if you delete the string and initialize a new same string, the id of them are the same, the first time you created it, it's stored in the **interned** dictionary +If you delete the string and initialize a new string with the same value, the IDs are the same. The first time you created it, it was stored in the **interned** dictionary ```python3 >>> id(s) @@ -162,7 +162,7 @@ if you delete the string and initialize a new same string, the id of them are th # kind -there are totally four values of **kind** field in **PyASCIIObject**, it means how characters are stored in the unicode object internally. +There are four possible values for the **kind** field in **PyASCIIObject**, indicating how characters are stored in the unicode object internally. * PyUnicode_WCHAR_KIND @@ -174,7 +174,7 @@ I haven't found a way to define an unicode object represent with **PyUnicode_WCH * ascii flag set true: U+0000-U+007F * ascii flag set false: at least one character in U+0080-U+00FF -the **utf8_length** field still stores the null terminated c style string, except the interned field is 0, only the characters in specific range will CPython store the unicode object in the interned dictionary. +The **utf8_length** field still stores the null-terminated C-style string, except the interned field is 0. Only characters in a specific range will cause CPython to store the unicode object in the interned dictionary. ```python3 >>> s1 = "\u007F\u0000" @@ -188,18 +188,18 @@ the **utf8_length** field still stores the null terminated c style string, excep ![1_byte_kind](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/1_byte_kind.png) -let's define an unicode object with a character **\u0088** inside +Let's define a unicode object with a character **\u0088** inside ```python3 s = "\u0088\u0011\u00f1" ``` -now, because the first character is **U+0088**, the ascii flag becomes 0, and **PyUnicode_UTF8(unicode)** no longer return the address of **utf8_length** field, instead, it returns the value in the `char *utf8` field, and that's 0 +Now, because the first character is **U+0088**, the ascii flag becomes 0, and **PyUnicode_UTF8(unicode)** no longer returns the address of the **utf8_length** field. Instead, it returns the value in the `char *utf8` field, which is 0 -if **PyUnicode_UTF8(unicode)** is zero, where are the three bytes located? -we haven't used the data field in **PyUnicodeObject**, let's print whatever inside **data** field. -it took me some time to figure out how to print those bytes in the latin1 field. +If **PyUnicode_UTF8(unicode)** is zero, where are the three bytes located? +We haven't used the data field in **PyUnicodeObject**. Let's print whatever is inside the **data** field. +It took me some time to figure out how to print those bytes in the latin1 field. ```c static PyObject * @@ -247,7 +247,7 @@ unicode_repr(PyObject *unicode) ``` -after recompile the above code, we are able to track latin1 fields in repr() function +After recompiling the above code, we are able to track latin1 fields in the repr() function ```python3 >>> repr(s) @@ -263,7 +263,7 @@ PyUnicodeObject->latin1: 0x88 0x11 0xf1 * all characters are in range U+0000-U+FFFF * at least one character is in the range U+0100-U+FFFF -we can use the same code to trace bytes stored in **data** field, now the field name is **ucs2**(**ucs2** or **latin1** have a different name, but same address, they are inside same c union structure) +We can use the same code to trace bytes stored in the **data** field. Now the field name is **ucs2** (**ucs2** and **latin1** have different names but the same address; they are inside the same C union structure) ```python3 >>> s = "\u0011\u0111\u1111" @@ -272,7 +272,7 @@ kind: PyUnicode_2BYTE_KIND, PyUnicodeObject->ucs2: 0x11 0x111 0x1111 ``` -now, the **kind** field is **PyUnicode_2BYTE_KIND** and it takes 2 bytes to store each character +Now the **kind** field is **PyUnicode_2BYTE_KIND** and it takes 2 bytes to store each character ![2_byte_kind](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/2_byte_kind.png) @@ -281,7 +281,7 @@ now, the **kind** field is **PyUnicode_2BYTE_KIND** and it takes 2 bytes to stor * all characters are in the range U+0000-U+10FFFF * at least one character is in the range U+10000-U+10FFFF -now, kind field become **PyUnicode_4BYTE_KIND** +Now the kind field becomes **PyUnicode_4BYTE_KIND** ```python3 >>> s = "\u00ff\U0010FFFF\U00100111\U0010FFF1" @@ -295,7 +295,7 @@ kind: PyUnicode_4BYTE_KIND, PyUnicodeObject->ucs4: 00xff 0x10ffff 0x100111 0x10f ## unicode memory usage summary -we now know that there are three kinds of storage mechanism, how many bytes CPython will consume to store an unicode object depends on the maximum range of your character. All characters inside the unicode object must be in the same size, if CPython use variable size representation such as utf-8, it would be impossible to do index operation in O(1) time +We now know that there are three kinds of storage mechanisms. How many bytes CPython consumes to store a unicode object depends on the maximum range of your characters. All characters inside the unicode object must be the same size; if CPython used variable-size representation such as UTF-8, it would be impossible to do index operations in O(1) time ```python3 # if represented as utf8 encoding, and stores 1 million variable size characters @@ -309,7 +309,7 @@ s[999999] # compact -if flag in **compact** field in 1, it means all characters are stored within **PyUnicodeObject**, no matter what **kind** field is. The example above all have **compact** field set to 1. Otherwise, the data block will not be stored inside the **PyUnicodeObject** object directly, the data block will be a newly malloced location. -the difference between **compact** 0 and **compact** 1 is the same as the difference between Redis string encoding **raw** and **embstr** +If the flag in the **compact** field is 1, it means all characters are stored within **PyUnicodeObject**, no matter what the **kind** field is. The examples above all have the **compact** field set to 1. Otherwise, the data block will not be stored inside the **PyUnicodeObject** object directly; the data block will be in a newly malloced location. +The difference between **compact** 0 and **compact** 1 is the same as the difference between Redis string encoding **raw** and **embstr** -now, we understand most of the unicode implementations in CPython, for more detail, you can directly refer to the source code \ No newline at end of file +Now we understand most of the unicode implementation in CPython. For more detail, you can refer directly to the source code \ No newline at end of file diff --git a/BasicObject/tuple/tuple.md b/BasicObject/tuple/tuple.md index 1951fcd..a63f227 100644 --- a/BasicObject/tuple/tuple.md +++ b/BasicObject/tuple/tuple.md @@ -16,8 +16,8 @@ ![memory layout](https://img-blog.csdnimg.cn/20190313121821367.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNzIwMzI5,size_16,color_FFFFFF,t_70) -The structure of **tuple** object is more simple than other python object. -Obviously, **ob_item** is an array of PyObject* pointer, all elements will be stored inside the **ob_item** array, But how exactly each element stored in **PyTupleObject**? Is the first element begin at the 0 index? What is the resize strategy? +The structure of the **tuple** object is simpler than other Python objects. +Obviously, **ob_item** is an array of PyObject* pointers, and all elements will be stored inside the **ob_item** array. But how exactly is each element stored in **PyTupleObject**? Does the first element begin at index 0? What is the resize strategy? Let's see @@ -42,7 +42,7 @@ t = ("aa", "bb", "cc", "dd") ``` -**ob_size** represents the size of **PyTupleObject** object, because tuple object is immutable, the **ob_item** is the start address of an array of pointer to **PyObject**, and the size of this array is **ob_size**, there's no need for resize operation. +**ob_size** represents the size of the **PyTupleObject** object. Because a tuple object is immutable, **ob_item** is the start address of an array of pointers to **PyObject**, and the size of this array is **ob_size**. There's no need for a resize operation. ![tuple_4](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple_4.png) @@ -59,8 +59,8 @@ static int numfree[PyTuple_MAXSAVESIZE]; ``` -let's exam what **free_list** and **numfree** is -we assume that python interpreter don't create/deallocate any tuple object internally during the following code +Let's examine what **free_list** and **numfree** are. +We assume that the Python interpreter doesn't create/deallocate any tuple objects internally during the following code ```python3 >>> t = tuple() @@ -84,7 +84,7 @@ we assume that python interpreter don't create/deallocate any tuple object inter 4459413360 >>> del t >>> t = ("bb", ) ->>> id(t) # it's not repeatable, because I assume that python intepreter don't create/deallocate any tuple object during execution +>>> id(t) # it's not repeatable, because I assume that the Python interpreter doesn't create/deallocate any tuple objects during execution 4459413360 >>> t2 = ("cc", ) >>> del t @@ -94,7 +94,7 @@ we assume that python interpreter don't create/deallocate any tuple object inter ![delete_2](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/delete_2.png) -num_free[i] means how many objects left in free_list[i], when you create a new tuple with size i, cpython will use the top object at free_list[i] +num_free[i] indicates how many objects are left in free_list[i]. When you create a new tuple with size i, CPython will use the top object at free_list[i] ```python3 >>> t = ("aa", ) diff --git a/BasicObject/type/type.md b/BasicObject/type/type.md index c208def..2ab7c40 100644 --- a/BasicObject/type/type.md +++ b/BasicObject/type/type.md @@ -21,13 +21,13 @@ # mro -the [C3 superclass linearization](https://en.wikipedia.org/wiki/C3_linearization) is implemented in python for Method Resolution Order (MRO) after python2.3, prior to python2.3, the Method Resolution Order is a Depth-First, Left-to-Right algorithm +The [C3 superclass linearization](https://en.wikipedia.org/wiki/C3_linearization) is implemented in Python for Method Resolution Order (MRO) after Python 2.3. Prior to Python 2.3, the Method Resolution Order was a Depth-First, Left-to-Right algorithm for those who are interested in detail, please refer to [Amit Kumar: Demystifying Python Method Resolution Order - PyCon APAC 2016](https://www.youtube.com/watch?v=cuonAMJjHow&t=400s) ## before python2.3 -let's define an example +Let's define an example ```python3 from __future__ import print_function @@ -56,7 +56,7 @@ class F(D, E): ![mro_ hierarchy](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/type/mro_hierarchy.png) -the **mro** order implementation before python2.3 is a Depth-First, Left-Most algorithm +The **mro** order implementation before Python 2.3 was a Depth-First, Left-Most algorithm mro of **D** is computed as `DAB` @@ -74,12 +74,12 @@ I am B ## after python2.3 -the **merge** part of the **C3** algorithm can be simply summarized as the following steps +The **merge** part of the **C3** algorithm can be simply summarized as the following steps: -* take the head of the first list -* if this head is not in the tail of any of the other lists, then add it to the linearization of C and remove it from the lists in the merge -* otherwise look at the head of the next list and take it, if it is a good head -* then repeat the operation until all the class are removed or it is impossible to find good heads(the inherted order need to be monotonous, or the algorithm won't terminate, for more detail please refer to the first video in [read more](#read-more)) +* Take the head of the first list +* If this head is not in the tail of any of the other lists, then add it to the linearization of C and remove it from the lists in the merge +* Otherwise, look at the head of the next list and take it if it is a good head +* Then repeat the operation until all the classes are removed or it is impossible to find good heads (the inherited order needs to be monotonous, or the algorithm won't terminate; for more detail please refer to the first video in [read more](#read-more)) mro of **D** is computed as @@ -106,7 +106,7 @@ E + B + C mro of **F** is computed as ```python3 -F + merge(L(D), L(E), BC) +F + merge(L(D), L(E), DE) F + merge(DAB, EBC, DE) F + D + merge(AB, EBC, E) F + D + A + merge(B, EBC, E) @@ -151,7 +151,7 @@ class F(D, E): ``` -in python2, class `A`, `B`,`C`, `D`, and `E` are not inherted from the `object`, even after python2.3, objects not inherted from `object` will not use the **C3** algorithm for **MRO**, the old style Depth-First, Left-Most algorithm will be used +In Python 2, classes `A`, `B`, `C`, `D`, and `E` are not inherited from `object`. Even after Python 2.3, objects not inherited from `object` will not use the **C3** algorithm for **MRO**; the old-style Depth-First, Left-Most algorithm will be used ```python3 # ./python2.7 @@ -162,7 +162,7 @@ I am B ``` -while in python3, they both implicit inherted from `object`, objects inherted from `object` use the **C3** algorithm for **MRO** +While in Python 3, they both implicitly inherit from `object`. Objects inherited from `object` use the **C3** algorithm for **MRO** ```python3 # ./python3 @@ -174,7 +174,7 @@ I am E # creation of class -if we `dis` the above code +If we `dis` the above code ```python3 26 96 LOAD_BUILD_CLASS @@ -195,7 +195,7 @@ and the following opcodes push all the variables `__build_class__` needed to sta `110 CALL_FUNCTION 4` calls the `__build_class__` to generate the class -the `__build_class__` will find the `metaclass`, `name`, `bases`, and `ns`(namespace) and delegates the call to `metaclass.__call__` attribute +The `__build_class__` will find the `metaclass`, `name`, `bases`, and `ns` (namespace) and delegate the call to the `metaclass.__call__` attribute for example, the `class F` @@ -207,26 +207,26 @@ ns: {'__module__': '__main__', '__qualname__': 'F'}, cls: ``` -in the example `class F`, what does `.__call__` do ? +In the example `class F`, what does `.__call__` do? -the function is defined in `cpython/Objects/typeobject.c` +The function is defined in `cpython/Objects/typeobject.c`. -prototype is `static PyObject *type_call(PyTypeObject *type, PyObject *args, PyObject *kwds)` +The prototype is `static PyObject *type_call(PyTypeObject *type, PyObject *args, PyObject *kwds)`. -we can draw the following procedure according to the source code +We can draw the following procedure according to the source code ![creation_of_class](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/type/creation_of_class.png) # creation of instance -if we add the following line to the tail of the above codes +If we add the following line to the end of the above code ```python3 f = F() ``` -we can find other snippet of the `dis` result +We can find another snippet of the `dis` result ```python3 @@ -236,7 +236,7 @@ we can find other snippet of the `dis` result ``` -it simply calls the `__call__` attribute of `F` +It simply calls the `__call__` attribute of `F` ```python3 @@ -249,13 +249,16 @@ it simply calls the `__call__` attribute of `F` # metaclass -in the above procedures, we can learn that **metaclass** controls the creation of a **class**, the creation of **instance** doesn't invoke the **metaclass**, it just calls the `__call__` attribute of the class to create the instance +In the above procedures, we can learn that the **metaclass** controls the creation of a **class**. The creation of an **instance** doesn't invoke the **metaclass**; it just calls the `__call__` attribute of the class to create the instance ![difference_between_class_instance](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/type/difference_between_class_instance.png) -we can change the behaviour of a class on the fly by manually define a **metaclass** +We can change the behavior of a class on the fly by manually defining a **metaclass** ```python3 +class F(object): + pass + class MyMeta(type): def __new__(mcs, name, bases, attrs, **kwargs): @@ -282,7 +285,7 @@ class AnotherF(F, metaclass=MyMeta): pass -print(Animal1.leg) # 4 +print(Animal1.leg) # 4 print(Normal.leg) # 0 print(AnotherF) # @@ -293,4 +296,4 @@ pretty fun! # read more * [Amit Kumar: Demystifying Python Method Resolution Order - PyCon APAC 2016](https://www.youtube.com/watch?v=cuonAMJjHow&t=400s) * [The Python 2.3 Method Resolution Order(MRO)](https://www.python.org/download/releases/2.3/mro/) -* [understanding-python-metaclasses](https://blog.ionelmc.ro/2015/02/09/understanding-python-metaclasses) \ No newline at end of file +* [understanding-python-metaclasses](https://blog.ionelmc.ro/2015/02/09/understanding-python-metaclasses) diff --git a/BasicObject/type/type_cn.md b/BasicObject/type/type_cn.md index 85fbda1..45b2799 100644 --- a/BasicObject/type/type_cn.md +++ b/BasicObject/type/type_cn.md @@ -100,7 +100,7 @@ D + merge(L(A), L(B), AB) D + merge(A, B, AB) # A 是下一个 list 的头部, 并且不在任何其他 list 的尾部中, 取出 A, 并把 A 从其他的所有的 list 中移除 D + A + merge(B, B) -# list 中的第一个元素是头部元素e而不是尾部元素, 所以 B 符合要求 +# list 中的第一个元素应该当成头部元素而不是尾部元素, 所以 B 符合要求 D + A + B ``` @@ -118,7 +118,7 @@ E + B + C **F** 的 MRO 计算结果如下 ```python3 -F + merge(L(D), L(E), BC) +F + merge(L(D), L(E), DE) F + merge(DAB, EBC, DE) F + D + merge(AB, EBC, E) F + D + A + merge(B, EBC, E) @@ -267,6 +267,9 @@ f = F() 根据以上发现, 我们可以自己定义一个 **metaclass** 在代码运行过程中个性化定制最终的 class ```python3 +class F(object): + pass + class MyMeta(type): def __new__(mcs, name, bases, attrs, **kwargs): @@ -293,7 +296,7 @@ class AnotherF(F, metaclass=MyMeta): pass -print(Animal1.leg) # 4 +print(Animal1.leg) # 4 print(Normal.leg) # 0 print(AnotherF) # @@ -304,4 +307,4 @@ print(AnotherF) # # 相关阅读 * [Amit Kumar: Demystifying Python Method Resolution Order - PyCon APAC 2016](https://www.youtube.com/watch?v=cuonAMJjHow&t=400s) * [The Python 2.3 Method Resolution Order(MRO)](https://www.python.org/download/releases/2.3/mro/) -* [understanding-python-metaclasses](https://blog.ionelmc.ro/2015/02/09/understanding-python-metaclasses) \ No newline at end of file +* [understanding-python-metaclasses](https://blog.ionelmc.ro/2015/02/09/understanding-python-metaclasses) diff --git a/Extension/C/c.md b/Extension/C/c.md index 8eaca8b..96727d5 100644 --- a/Extension/C/c.md +++ b/Extension/C/c.md @@ -12,13 +12,13 @@ # overview -recently I get a task to improve someone else's API performance written in Django server, the API will start an async job and in some cases last several minutes, I manage to improve a 40 seconds CPU bound task to lower than 1 second +Recently I got a task to improve someone else's API performance written in Django server. The API will start an async job and in some cases last several minutes. I managed to improve a 40-second CPU-bound task to lower than 1 second # example ## profile -[line_profiler](https://github.com/rkern/line_profiler) is a handful tool to begin with +[line_profiler](https://github.com/rkern/line_profiler) is a handy tool to begin with I am running with `python2.7` @@ -57,11 +57,11 @@ if __name__ == "__main__": ``` -notice, if you're running django server with `python manage.py runserver`, the default behaviour of `runserver` will spawn a thread to handle your request, there will be at least two thread register the `profile.print_stats`, their output may interleave together and become confused +Notice, if you're running Django server with `python manage.py runserver`, the default behavior of `runserver` will spawn a thread to handle your request. There will be at least two threads registering the `profile.print_stats`, and their output may interleave together and become confusing you may need to call `profile.print_stats()` manually in your code without register the `profile.print_stats` -this is the output +This is the output ```shell script Timer unit: 1e-06 s @@ -101,7 +101,7 @@ Line # Hits Time Per Hit % Time Line Contents ``` -we can see that `if i in meaningless_dict.keys():` takes about 128 seconds to run, this is mainly because `dict.keys()` will generate a new list like object and insert all the `keys` in the `dict` into the newly generated list, and `in dict.keys()` will become a `O(n)` list search +We can see that `if i in meaningless_dict.keys():` takes about 128 seconds to run. This is mainly because `dict.keys()` will generate a new list-like object and insert all the `keys` in the `dict` into the newly generated list, and `in dict.keys()` will become an `O(n)` list search for python3.x, `dict.keys()` will return a `dict_keys` object, which is a view of the `dict`, the performance issue will be gone @@ -130,23 +130,23 @@ Line # Hits Time Per Hit % Time Line Contents ``` -the runtime of the same line becomes about 3 seconds, it consumes about 18.6% runtime of the function, there still exist other time consuming line, can we do better ? +The runtime of the same line becomes about 3 seconds, it consumes about 18.6% runtime of the function. There still exist other time-consuming lines. Can we do better? ## C module -we can rewrite the **my_cpu_bound_task** in C +We can rewrite the **my_cpu_bound_task** in C -you can find all the C API you need from [python2.7-capi](https://docs.python.org/2.7/c-api/index.html) or [python3.7-capi](https://docs.python.org/3.7/c-api/index.html) +You can find all the C API you need from [python2.7-capi](https://docs.python.org/2.7/c-api/index.html) or [python3.7-capi](https://docs.python.org/3.7/c-api/index.html) -you can also refer to guide of [Writing a C Extension Module](http://madrury.github.io/jekyll/update/programming/2016/06/20/python-extension-modules.html) +You can also refer to the guide [Writing a C Extension Module](http://madrury.github.io/jekyll/update/programming/2016/06/20/python-extension-modules.html) ## python2 a [setup.py](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/profile_py2/my_mod/setup.py) is needed, and a source code file stores all the C function, I name it [my_module.c](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/profile_py2/my_mod/my_module.c) -I am not going to copy pasted all the C source code in the readme +I am not going to copy and paste all the C source code in the readme -it's avaliable in [CPython-Internals/Extension/C/profile_py2/my_mod/](https://github.com/zpoint/CPython-Internals/tree/master/Extension/C/profile_py2/my_mod) +It's available in [CPython-Internals/Extension/C/profile_py2/my_mod/](https://github.com/zpoint/CPython-Internals/tree/master/Extension/C/profile_py2/my_mod) ```shell script # only available for python2.x @@ -190,9 +190,9 @@ Line # Hits Time Per Hit % Time Line Contents a [setup.py](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/profile_py3/my_mod/setup.py) is needed, and a source code file stores all the C function, I name it [my_module.c](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/profile_py3/my_mod/my_module.c) -I am not going to copy pasted all the C source code in the readme +I am not going to copy and paste all the C source code in the readme -it's avaliable in [CPython-Internals/Extension/C/profile_py3/my_mod/](https://github.com/zpoint/CPython-Internals/tree/master/Extension/C/profile_py3/my_mod) +It's available in [CPython-Internals/Extension/C/profile_py3/my_mod/](https://github.com/zpoint/CPython-Internals/tree/master/Extension/C/profile_py3/my_mod) ```shell script # only available for python3.x diff --git a/Extension/C/profile_py2/my_mod/setup.py b/Extension/C/profile_py2/my_mod/setup.py index ce18b23..39944c8 100644 --- a/Extension/C/profile_py2/my_mod/setup.py +++ b/Extension/C/profile_py2/my_mod/setup.py @@ -1,6 +1,6 @@ from distutils.core import setup, Extension -my_module = Extension('my_module', sources=['my_module.c']) +my_module = Extension('my_module', sources=['my_module.c'], extra_compile_args=["-std=c99"]) setup(name='my_module', version='1.0', diff --git a/Extension/CPP/cpp.md b/Extension/CPP/cpp.md new file mode 100644 index 0000000..aa753b6 --- /dev/null +++ b/Extension/CPP/cpp.md @@ -0,0 +1,145 @@ +# C++ extension + +# contents + +* [overview](#overview) +* [example](#example) +* [integrate with NumPy](#integrate-with-NumPy) +* [bypass the GIL](#bypass-the-GIL) +* [read more](#read-more) + +# overview + +We've written a [C extension](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/c.md) before to improve performance + +Now we want to take a further step to develop a complex module in C++ and expose it as a Python API + +The C++ compiler is compatible with C and I want to use the power of `STL` in `C++11`(or newer) standard + +# example + +```python3 +# setup.py +from distutils.core import setup, Extension + +my_module = Extension('m_example', sources=['./example.cpp'], extra_compile_args=['-std=c++11']) + +setup(name='m_example', + version='1.0', + description='my module to use C++ STL', + ext_modules=[my_module]) +``` + +and the [example.cpp](https://github.com/zpoint/CPython-Internals/blob/master/Extension/CPP/example/example.cpp) get elements from the Python array and stores all integer to a C++ vectors and use `std::sort` from `` to sort the vector, finally returns the first element as Python object to the caller + +Run the example + +```bash +cd Cpython-Internals/Extension/CPP/example +python3 setup.py build +mv build/lib.macosx-10.15-x86_64-3.8/m_example.cpython-38-darwin.so ./ +zpoint@zpoints-MacBook-Pro example % python3 +Python 3.8.4 (default, Jul 14 2020, 02:58:48) +[Clang 11.0.3 (clang-1103.0.32.62)] on darwin +Type "help", "copyright", "credits" or "license" for more information. +>>> import m_example +>>> m_example.example([6,4,3]) +3 +>>> +``` + +# integrate with NumPy + +Sometimes we need to work with NumPy, let's begin with an example + +We take a two-dimensional NumPy array, a one-dimensional NumPy array as input, and a double value, do some computation, and write the result back to the one-dimensional array + +The numpy array should have dtype float64 + +two_dim is `3*N` array + +one_dim is `N` array + +```python3 +import numpy as np +def compute(two_dim: np.array, one_dim: np.array, val: np.float) -> None: + # do some computation and store result to one_dim + for index in range(len(one_dim)): + one_dim[index] = (two_dim[0][index] + two_dim[1][index] + two_dim[2][index]) * val + # ... +``` + +We are going to write the above function as extension module in C++ + +[example.cpp](https://github.com/zpoint/CPython-Internals/blob/master/Extension/CPP/m_numpy/example.cpp) + +```bash +cd Cpython-Internals/Extension/CPP/m_numpy +python3 setup.py build +mv build/lib.macosx-10.15-x86_64-3.8/m_example.cpython-38-darwin.so ./ +zpoint@zpoints-MacBook-Pro m_numpy % python3 +Python 3.8.4 (default, Jul 14 2020, 02:58:48) +[Clang 11.0.3 (clang-1103.0.32.62)] on darwin +Type "help", "copyright", "credits" or "license" for more information. +>>> import numpy as np +>>> import m_example +>>> one_dim = np.zeros([2], dtype=np.float) +>>> two_dim = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], dtype=np.float) +>>> m_example.example(two_dim, one_dim, 0.5) +>>> one_dim +array([4.5, 6. ]) +``` + +# bypass the GIL + +What if we want to schedule a parallel task to utilize the runtime of our task? + +two_dim becomes `X * 3 * N` array + +one_dim is `X * N` array + +```python3 +import numpy as np +def compute(two_dim: np.array, one_dim: np.array, val: np.float) -> None: + # do some computation and store result to one_dim + for task_index in range(len(one_dim)): + curr_one_dim = one_dim[task_index] + curr_two_dim = two_dim[task_index] + for index in range(len(one_dim)): + curr_one_dim[index] = (curr_two_dim[0][index] + curr_two_dim[1][index] + curr_two_dim[2][index]) * val + # ... +``` + +I want to separate these tasks into several different threads, and let the OS schedule them to work together + +We've learned about the [GIL](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil.md) before. We know that the interpreter in different threads will acquire the mutex before executing every Python bytecode, so as long as our code in different threads is not executed by the Python interpreter, everything will be fine + +We use the `std::future` from `C++ STL` to schedule our thread according to the environment variable `CONCURRENCY_NUM` + +```bash +cd Cpython-Internals/Extension/CPP/m_parallel +python3 setup.py build +mv build/lib.macosx-10.15-x86_64-3.8/m_example.cpython-38-darwin.so ./ +zpoint@zpoints-MacBook-Pro m_parallel % python3 +Python 3.8.4 (default, Jul 14 2020, 02:58:48) +[Clang 11.0.3 (clang-1103.0.32.62)] on darwin +Type "help", "copyright", "credits" or "license" for more information. +>>> import os +>>> import numpy as np +>>> import m_example +>>> os.putenv("CONCURRENCY_NUM", "2") +>>> one_dim = np.zeros([4, 2], dtype=np.float) +>>> two_dim = np.array([[[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]], dtype=np.float) +>>> m_example.example(two_dim, one_dim, 0.5) +>>> print(one_dim) +[[4.5 6. ] + [4.5 6. ] + [4.5 6. ] + [4.5 6. ]] +``` + + + +# read more + +* [numpy-api](https://numpy.org/doc/stable/reference/c-api/array.html?highlight=array%20api) diff --git a/Extension/CPP/cpp_cn.md b/Extension/CPP/cpp_cn.md new file mode 100644 index 0000000..98504d0 --- /dev/null +++ b/Extension/CPP/cpp_cn.md @@ -0,0 +1,142 @@ +# C++ extension + +# 目录 + +* [介绍](#介绍) +* [示例](#示例) +* [和 NumPy 结合](#和-NumPy-结合) +* [绕过 GIL](#绕过-GIL) +* [更多资料](#更多资料) + +# 介绍 + +我们在之前已经尝试过使用 [C 扩展](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/c_cn.md) 来提高性能了, 现在我们尝试更进一步, 在 C++ 下编写一个扩展, 并同样的利用 python API 暴露给 python 调用 + +C++ 的编译器是兼容 C 语言的, 我们将会借助 `C++11`(或以上) 的标准的 `STL` 来实现一些功能 + +# 示例 + +```python3 +# setup.py +from distutils.core import setup, Extension + +my_module = Extension('m_example', sources=['./example.cpp'], extra_compile_args=['-std=c++11']) + +setup(name='m_example', + version='1.0', + description='my module to use C++ STL', + ext_modules=[my_module]) +``` + +[example.cpp](https://github.com/zpoint/CPython-Internals/blob/master/Extension/CPP/example/example.cpp) 从 Python 数组中获取元素, 之后把它作为整数类型存储在了 `C++` 的 `vector` 中, 然后用 `` 中的 `std::sort` 方法对其进行排序, 最后把排好序的第一个值作为 Python 对象返回给调用者 + +运行这个示例 + +```bash +cd Cpython-Internals/Extension/CPP/example +python3 setup.py build +mv build/lib.macosx-10.15-x86_64-3.8/m_example.cpython-38-darwin.so ./ +zpoint@zpoints-MacBook-Pro example % python3 +Python 3.8.4 (default, Jul 14 2020, 02:58:48) +[Clang 11.0.3 (clang-1103.0.32.62)] on darwin +Type "help", "copyright", "credits" or "license" for more information. +>>> import m_example +>>> m_example.example([6,4,3]) +3 +>>> +``` + +# 和 NumPy 结合 + +有些时候我们需要和 NumPy 结合使用, 让我们从一个新的示例开始 + +我们以一个二维数组, 一个一维数组, 和一个双精度浮点数作为输入, 对输入进行一些运算, 之后把结果写回一维数组中 + +numpy 数组中的元素类型我们先指定为 float64 + +two_dim 维度是 `3*N` + +one_dim 维度是 `N` + +```python3 +import numpy as np +def compute(two_dim: np.array, one_dim: np.array, val: np.float) -> None: + # do some computation and store result to one_dim + for index in range(len(one_dim)): + one_dim[index] = (two_dim[0][index] + two_dim[1][index] + two_dim[2][index]) * val + # ... +``` + +我们会把上面的示例函数用 C++ 扩展的方式进行重写 + +[example.cpp](https://github.com/zpoint/CPython-Internals/blob/master/Extension/CPP/m_numpy/example.cpp) + +```bash +cd Cpython-Internals/Extension/CPP/m_numpy +python3 setup.py build +mv build/lib.macosx-10.15-x86_64-3.8/m_example.cpython-38-darwin.so ./ +zpoint@zpoints-MacBook-Pro m_numpy % python3 +Python 3.8.4 (default, Jul 14 2020, 02:58:48) +[Clang 11.0.3 (clang-1103.0.32.62)] on darwin +Type "help", "copyright", "credits" or "license" for more information. +>>> import numpy as np +>>> import m_example +>>> one_dim = np.zeros([2], dtype=np.float) +>>> two_dim = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], dtype=np.float) +>>> m_example.example(two_dim, one_dim, 0.5) +>>> one_dim +array([4.5, 6. ]) +``` + +# 绕过 GIL + +如果我们想要通过并行任务的方式, 来缩小我们任务的执行时间呢 ? + +two_dim 的维度变成了 `X * 3 * N` + +one_dim 的维度变成了 `X * N` + +```python3 +import numpy as np +def compute(two_dim: np.array, one_dim: np.array, val: np.float) -> None: + # do some computation and store result to one_dim + for task_index in range(len(one_dim)): + curr_one_dim = one_dim[task_index] + curr_two_dim = two_dim[task_index] + for index in range(len(one_dim)): + curr_one_dim[index] = (curr_two_dim[0][index] + curr_two_dim[1][index] + curr_two_dim[2][index]) * val + # ... +``` + +我想要把这些任务均匀地分布到不同的线程中, 并让操作系统对这些任务进行调度 + +我们先前已经学过了 [GIL](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil_cn.md) 相关的知识了, 我们知道不同线程中的解释器会在执行 python 字节码之前获取同一个互斥锁, 所以只要我们在不同线程中的代码不通过解释器去执行即可 + +我们这里会使用到 `C++ STL` 中的 `std::future` 来让我们不同的线程执行任务(具体起多少个 `future` 根据环境变量 `CONCURRENCY_NUM` 来判断) + +```bash +cd Cpython-Internals/Extension/CPP/m_parallel +python3 setup.py build +mv build/lib.macosx-10.15-x86_64-3.8/m_example.cpython-38-darwin.so ./ +zpoint@zpoints-MacBook-Pro m_parallel % python3 +Python 3.8.4 (default, Jul 14 2020, 02:58:48) +[Clang 11.0.3 (clang-1103.0.32.62)] on darwin +Type "help", "copyright", "credits" or "license" for more information. +>>> import os +>>> import numpy as np +>>> import m_example +>>> os.putenv("CONCURRENCY_NUM", "2") +>>> one_dim = np.zeros([4, 2], dtype=np.float) +>>> two_dim = np.array([[[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]], [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]], dtype=np.float) +>>> m_example.example(two_dim, one_dim, 0.5) +>>> print(one_dim) +[[4.5 6. ] + [4.5 6. ] + [4.5 6. ] + [4.5 6. ]] +``` + + +# 更多资料 + +* [numpy-api](https://numpy.org/doc/stable/reference/c-api/array.html?highlight=array%20api) diff --git a/Extension/CPP/example/example.cpp b/Extension/CPP/example/example.cpp new file mode 100644 index 0000000..a82906f --- /dev/null +++ b/Extension/CPP/example/example.cpp @@ -0,0 +1,50 @@ +/* check your system path, sometimes it's */ +/* only for illustration purpose, not good code for production */ +#include +#include +#include + +static PyObject* example(PyObject *self, PyObject *args) { + /* def example(arr: typing.List[int]) -> int: */ + PyObject *py_arr; + std::vector i_vec; + long res; + if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &py_arr)) { + PyErr_SetString(PyExc_ValueError, "expected one list parameter"); + return nullptr; + } + Py_ssize_t list_len = PyList_GET_SIZE(py_arr); + i_vec.reserve(list_len); + for (Py_ssize_t i = 0; i < list_len; ++i) { + PyObject *py_int = PyList_GET_ITEM(py_arr, i); + res = PyLong_AsLong(py_int); + if (res == -1 && PyErr_Occurred()) { + return nullptr; + } + i_vec.push_back(res); + } + std::sort(i_vec.begin(), i_vec.end()); + return PyLong_FromLong(i_vec[0]); +} + +static PyMethodDef MyMethods[] = { + {"example", example, METH_VARARGS, "example function"}, + {NULL, NULL, 0, NULL} /* Sentinel */ +}; + +static struct PyModuleDef m_example = { + PyModuleDef_HEAD_INIT, + "example", + "example module to provide python api", + -1, + MyMethods +}; + +PyMODINIT_FUNC PyInit_m_example(void) { + PyObject *m; + + m = PyModule_Create(&m_example); + if (m == NULL) + return NULL; + return m; +} diff --git a/Extension/CPP/example/setup.py b/Extension/CPP/example/setup.py new file mode 100644 index 0000000..de6b058 --- /dev/null +++ b/Extension/CPP/example/setup.py @@ -0,0 +1,8 @@ +from distutils.core import setup, Extension + +my_module = Extension('m_example', sources=['./example.cpp'], extra_compile_args=['-std=c++11']) + +setup(name='m_example', + version='1.0', + description='my module to use C++ STL', + ext_modules=[my_module]) diff --git a/Extension/CPP/m_numpy/example.cpp b/Extension/CPP/m_numpy/example.cpp new file mode 100644 index 0000000..3a6e013 --- /dev/null +++ b/Extension/CPP/m_numpy/example.cpp @@ -0,0 +1,62 @@ +#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION +#include +#include + +PyObject* failure(PyObject *type, const char *message) { + PyErr_SetString(type, message); + return NULL; +} + +static PyObject* example(PyObject *self, PyObject *args) { + /* def example(two_dim: np.array, one_dim: np.array, val: np.float) -> None */ + PyArrayObject *two_dim, *one_dim; + double val; + if (!PyArg_ParseTuple(args, "O!O!d", &PyArray_Type, &two_dim, &PyArray_Type, &one_dim, &val)) + return failure(PyExc_KeyError, "expected one two dimension array, one one dimension array and one double value"); + + if (PyArray_DESCR(two_dim)->type_num != NPY_FLOAT64) + return failure(PyExc_TypeError, "Type np.float expected for two_dim array."); + + if (PyArray_DESCR(one_dim)->type_num != NPY_FLOAT64) + return failure(PyExc_TypeError, "Type np.float expected for one_dim array."); + + if (PyArray_NDIM(two_dim) != 2 || PyArray_DIM(two_dim, 0) != 3) + return failure(PyExc_TypeError, "two_dim dimension error."); + + if (PyArray_NDIM(one_dim) != 1) + return failure(PyExc_TypeError, "one_dim dimension error."); + + npy_intp size_one_dim = PyArray_SIZE(one_dim); + double* d_one_dim = (double*)PyArray_GETPTR1(one_dim, 0); + double* d_two_dim_0 = (double*)PyArray_GETPTR1(two_dim, 0); + double* d_two_dim_1 = (double*)PyArray_GETPTR1(two_dim, 1); + double* d_two_dim_2 = (double*)PyArray_GETPTR1(two_dim, 2); + for (npy_intp i = 0; i < size_one_dim; ++i) { + d_one_dim[i] = (d_two_dim_0[i] + d_two_dim_1[i] + d_two_dim_2[i]) * val; + } + return Py_None; + +} + +static PyMethodDef MyMethods[] = { + {"example", example, METH_VARARGS, "example function"}, + {NULL, NULL, 0, NULL} /* Sentinel */ +}; + +static struct PyModuleDef m_example = { + PyModuleDef_HEAD_INIT, + "example", + "example module to provide numpy python api", + -1, + MyMethods +}; + +PyMODINIT_FUNC PyInit_m_example(void) { + PyObject *m; + + m = PyModule_Create(&m_example); + if (m == NULL) + return NULL; + import_array(); + return m; +} diff --git a/Extension/CPP/m_numpy/setup.py b/Extension/CPP/m_numpy/setup.py new file mode 100644 index 0000000..a8fca98 --- /dev/null +++ b/Extension/CPP/m_numpy/setup.py @@ -0,0 +1,9 @@ +import numpy +from distutils.core import setup, Extension + +my_module = Extension('m_example', sources=['./example.cpp'], extra_compile_args=['-std=c++11'], include_dirs=[numpy.get_include()]) + +setup(name='m_example', + version='1.0', + description='my module to use C++ for numpy', + ext_modules=[my_module]) diff --git a/Extension/CPP/m_parallel/example.cpp b/Extension/CPP/m_parallel/example.cpp new file mode 100644 index 0000000..5616b8f --- /dev/null +++ b/Extension/CPP/m_parallel/example.cpp @@ -0,0 +1,101 @@ +#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION +#include +#include +#include +#include +#include + +PyObject* failure(PyObject *type, const char *message) { + PyErr_SetString(type, message); + return NULL; +} + +int load_thread_num() { + int ret = -1; + char* num; + num = getenv("CONCURRENCY_NUM"); + if (num != nullptr) { + ret = std::stoi(num); + } + // printf("load_thread_num return: %d\n", ret); + return ret > 1 ? ret : 1; +} + +void parallel_run(PyArrayObject *origin_one_dim, PyArrayObject *origin_two_dim, int start, int end, double val) { + npy_intp size_one_dim = PyArray_DIM(origin_one_dim, 1); + for (int curr_start = 0; curr_start < end; ++ curr_start) { + double* d_one_dim = (double*)PyArray_GETPTR2(origin_one_dim, curr_start, 0); + double* d_two_dim_0 = (double*)PyArray_GETPTR2(origin_two_dim, curr_start, 0); + double* d_two_dim_1 = (double*)PyArray_GETPTR2(origin_two_dim, curr_start, 1); + double* d_two_dim_2 = (double*)PyArray_GETPTR2(origin_two_dim, curr_start, 2); + for (npy_intp i = 0; i < size_one_dim; ++i) { + d_one_dim[i] = (d_two_dim_0[i] + d_two_dim_1[i] + d_two_dim_2[i]) * val; + } + } +} + +static PyObject* example(PyObject *self, PyObject *args) { + /* def example(two_dim: np.array, one_dim: np.array, val: np.float) -> None */ + PyArrayObject *two_dim, *one_dim; + double val; + if (!PyArg_ParseTuple(args, "O!O!d", &PyArray_Type, &two_dim, &PyArray_Type, &one_dim, &val)) + return failure(PyExc_KeyError, "expected one three dimension array, one two dimension array and one double value"); + + if (PyArray_DESCR(two_dim)->type_num != NPY_FLOAT64) + return failure(PyExc_TypeError, "Type np.float expected for two_dim array."); + + if (PyArray_DESCR(one_dim)->type_num != NPY_FLOAT64) + return failure(PyExc_TypeError, "Type np.float expected for one_dim array."); + + if (PyArray_NDIM(two_dim) != 3 || PyArray_DIM(two_dim, 1) != 3) + return failure(PyExc_TypeError, "two_dim dimension error."); + + if (PyArray_NDIM(one_dim) != 2) + return failure(PyExc_TypeError, "one_dim dimension error."); + + npy_intp dim1_one = PyArray_DIM(one_dim, 0); + + int right, steps; + int thread_num = load_thread_num(); + if (thread_num > dim1_one) + thread_num = dim1_one; + std::vector> future_vector; + future_vector.reserve(thread_num); + steps = dim1_one / thread_num; + for (int i = 0; i < dim1_one; ) { + // [i, i+steps) + right = i + steps; + if (right > dim1_one) { + right = dim1_one; + } + future_vector.push_back(std::async(std::launch::async, parallel_run, one_dim, two_dim, i, right, val)); + i = right; + } + for (auto &future : future_vector) { + future.get(); + } + return Py_None; +} + +static PyMethodDef MyMethods[] = { + {"example", example, METH_VARARGS, "example function"}, + {NULL, NULL, 0, NULL} /* Sentinel */ +}; + +static struct PyModuleDef m_example = { + PyModuleDef_HEAD_INIT, + "example", + "example module to provide numpy python api", + -1, + MyMethods +}; + +PyMODINIT_FUNC PyInit_m_example(void) { + PyObject *m; + + m = PyModule_Create(&m_example); + if (m == NULL) + return NULL; + import_array(); + return m; +} diff --git a/Extension/CPP/m_parallel/setup.py b/Extension/CPP/m_parallel/setup.py new file mode 100644 index 0000000..1ab824a --- /dev/null +++ b/Extension/CPP/m_parallel/setup.py @@ -0,0 +1,9 @@ +import numpy +from distutils.core import setup, Extension + +my_module = Extension('m_example', sources=['./example.cpp'], extra_compile_args=['-std=c++11', '-pthread'], include_dirs=[numpy.get_include()]) + +setup(name='m_example', + version='1.0', + description='my module to use C++ for numpy, in parallel', + ext_modules=[my_module]) diff --git a/Interpreter/code/code.md b/Interpreter/code/code.md index abacd0e..ea7b34c 100644 --- a/Interpreter/code/code.md +++ b/Interpreter/code/code.md @@ -24,7 +24,7 @@ > a code object is CPython's internal representation of a piece of runnable Python code, such as a function, a module, a class body, or a generator expression. When you run a piece of code, it is parsed and compiled into a code object, which is then run by the CPython virtual machine (VM) -for more detail, please refer to [What is a code object in Python?](https://www.quora.com/What-is-a-code-object-in-Python) +For more detail, please refer to [What is a code object in Python?](https://www.quora.com/What-is-a-code-object-in-Python) ![layout](https://img-blog.csdnimg.cn/20190208112516130.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNzIwMzI5,size_16,color_FFFFFF,t_70) @@ -32,7 +32,7 @@ for more detail, please refer to [What is a code object in Python?](https://www. ## example -let's run an example written in python +Let's run an example written in Python ```python3 def a(x, y, *args, z=3, **kwargs): @@ -67,7 +67,7 @@ print("co_cellvars", a.__code__.co_consts[1].co_cellvars) ``` -output +Output ```python3 co_argcount 2 @@ -159,7 +159,7 @@ def _unpack_opargs(code): ``` -so, **co_code** is the opcode and argument stores in binary format +So, **co_code** is the opcode and argument stored in binary format ```python3 >>> c = b'd\x01}\x00t\x00\x88\x01\x88\x00|\x00\x83\x03\x01\x00d\x00S\x00' @@ -169,7 +169,7 @@ so, **co_code** is the opcode and argument stores in binary format ``` -the binary format can be translated to +The binary format can be translated to ```python3 0 100 1 (LOAD_CONST) @@ -195,11 +195,11 @@ the binary format can be translated to > This means line number table, and stores a compressed mapping of bytecode instructions to line numbers. -let's see an example +Let's see an example. -the first pair (0, 1) in **co_lnotab** means byteoffset 0, line offset: 1 + co_firstlineno(7) == 8 +The first pair (0, 1) in **co_lnotab** means byte offset 0, line offset: 1 + co_firstlineno(7) == 8 -the second pair (4, 1) in **co_lnotab** means byteoffset 4, line offset 1 + 8(previous offset) == 9 +The second pair (4, 1) in **co_lnotab** means byte offset 4, line offset 1 + 8 (previous offset) == 9 ```python3 import dis @@ -227,15 +227,15 @@ print(dis.dis(f2)) ## co_zombieframe -you can read the following comment from `Objects/frameobject.c` +You can read the following comment from `Objects/frameobject.c` -for more detail, please refer to [frame object(zombie frame)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame.md#zombie-frame) +For more detail, please refer to [frame object(zombie frame)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame.md#zombie-frame) > each code object will hold a single "zombie" frame. This retains the allocated and initialized frame object from an invocation of the code object. The zombie is reanimated the next time we need a frame object for that code object. Doing this saves the malloc/ realloc required when using a free_list frame that isn't the correct size. It also saves some field initialization. ## co_extra -this field stores a pointer to a **_PyCodeObjectExtra** object +This field stores a pointer to a **_PyCodeObjectExtra** object ```c typedef struct { @@ -245,8 +245,8 @@ typedef struct { ``` -since it has a size field and an array of (void *) pointer, it can store almost everything +Since it has a size field and an array of (void *) pointers, it can store almost everything. -usually, it's a function pointer related to the interpreter for debug or JIT usage +Usually, it's a function pointer related to the interpreter for debug or JIT usage. -for more detail please refer to [PEP 523 -- Adding a frame evaluation API to CPython](https://www.python.org/dev/peps/pep-0523/) \ No newline at end of file +For more detail please refer to [PEP 523 -- Adding a frame evaluation API to CPython](https://www.python.org/dev/peps/pep-0523/) \ No newline at end of file diff --git a/Interpreter/compile/compile.md b/Interpreter/compile/compile.md new file mode 100644 index 0000000..01af1d8 --- /dev/null +++ b/Interpreter/compile/compile.md @@ -0,0 +1,226 @@ +# compile + +# contents + +* [related file](#related-file) +* [pgen](#pgen) + * [the dfa of `ExampleGrammar`](#the-dfa-of-ExampleGrammar) + * [parse](#parse) + * [make dfa](#make-dfa) + +* [read more](#read-more) + +# related file + +* Python/pythonrun.c +* Parser/tokenizer.c +* Parser/tokenizer.h +* Parser/parsetok.c +* Include/grammar.h +* Parser/metagrammar.c +* Include/metagrammar.h +* Parser/pgen.c + +# pgen + +This is the layout of "grammar" structure + +![grammar](./grammar.png) + + + +```bash +make regen-grammar +gcc -g -O0 -Wall -L/usr/local/opt/llvm/lib Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/token.o Parser/pgen.o Objects/obmalloc.o Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain.o -ldl -framework CoreFoundation -o Parser/pgen +# Regenerate Include/graminit.h and Python/graminit.c +# from Grammar/Grammar using pgen +Parser/pgen ./Grammar/Grammar \ + ./Include/graminit.h.new \ + ./Python/graminit.c.new +Translating labels ... +python3 ./Tools/scripts/update_file.py ./Include/graminit.h ./Include/graminit.h.new +python3 ./Tools/scripts/update_file.py ./Python/graminit.c ./Python/graminit.c.new +``` + +There's a program inside the `Parser/` directory. The above command will compile and run the `Parser/pgenmain.o` and output a program named `Parser/pgen`, which takes a **grammar file** as input and generates `grammar` structure, `dfa` tables, etc. (above diagram) as output (two C files, `graminit.c` and `graminit.h`) + +![pgen](./pgen.png) + +Let's focus on `pgen` + +The built-in `Parser/metagrammar.c` and `Include/metagrammar.h` will be used as the input grammar file's grammar + +`metagrammar` is used for parsing the `EBNF` format's grammar file + +`graminit` is used for parsing `Python` 's source code + +They are both generated in the same way, with different `Grammar` files + +![pgen2](./pgen2.png) + +The `Parser/metagrammar.c` and `Include/metagrammar.h` 's grammar + +>```bash +>MSTART: (NEWLINE | RULE)* ENDMARKER +>RULE: NAME ':' RHS NEWLINE +>RHS: ALT ('|' ALT)* +>ALT: ITEM+ +>ITEM: '[' RHS ']' | ATOM ['*' | '+'] +>ATOM: NAME | STRING | '(' RHS ')' +>``` + +It's like a compiler which can compile itself. For more detail please refer to [Python compiler from grammar to dfa](https://aoik.me/blog/posts/python-compiler-from-grammar-to-dfa) + +Let's focus on how `pgen` works with a simple example + +```bash +% cat Grammar/ExampleGrammar +START: (NEWLINE | RULE)* ENDMARKER +RULE: NUMBER (ADD NUMBER)* +ADD: '+' | '-' +``` + +We can use the default `pgen` to generate code for our `ExampleGrammar` + +```bash +make regen-grammar +Parser/pgen ./Grammar/ExampleGrammar \ + ./Include/examplegrammar.h \ + ./Parser/examplegrammar.c +``` + +With a little manual editing to make the following compile command work, the final [examplegrammar.c](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/examplegrammar.c) + +And compile the edited generated grammar code and compile a new `pgen2` program [example_grammar.sh](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/example_grammar.sh) + +```bash +sh example_grammar.sh +``` + +After getting `Parser/pgen2`, we can verify that our grammar works correctly + +```bash +% cat my_file.txt +1 + 3 + 4 +2 - 5 +# run the following command will fail, our purpose is to see the dfa state +# if you comment `Parser/parser.c`'s macro and set the `Parser/parser.c`'s macro to '#define D(x) x' and rerun the above compile command +# you can see the output of DFA state is in the ACCEPT state +# it means our grammar works correctly +Parser/pgen2 ./my_file.txt \ + ./Include/my_file.h \ + ./Parser/my_file.c +``` + +## the dfa of `ExampleGrammar` + + + +![dfa_example_grammar](./dfa_example_grammar.png) + +This is the dfa due to [examplegrammar.c](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/examplegrammar.c) or [ExampleGrammar](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/ExampleGrammar) + +## parse + +The first step is to parse the Grammar file according to the [BuiltInGrammar](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/BuiltInGrammar) and build a CST tree + +The call stack of `Parser/pgenmain.c` + +![pgmain](./pgmain.png) + + + +`parsetok` will call `PyTokenizer_Get->tok_get` to get tokens, for each token, `PyParser_AddToken` will be called + +`tokens` are predefined in `Parser/token.c` and `Include/token.h`, which are also auto-generated from `Grammar/Tokens` by the command `make regen-token` + +This is the helper structure when parsing + +![parser](./parser.png) + +There are too many details. I am not going to show every element's value in every structure. Let's go through our `ExampleGrammar` with `my_file.txt` token by token + +After `push` + +![parse1](./parse1.png) + +After `shift` + +![parse2](./parse2.png) + +After `push` + +![parse3](./parse3.png) + +After `shift` + +![parse4](./parse4.png) + +Because `ADD_1` is the accept state, `POP` will be called, after `POP` + +![parse5](./parse5.png) + +After `push` and `shift` + + + +![parse6](./parse6.png) + +And so on ... + +![parse7](./parse7.png) + + + +## make dfa + +The function defined in `Parser/pgen.c` will take the above tree as input, and a `grammar` structure as output + +```c +grammar * +pgen(node *n) +{ + nfagrammar *gr; + grammar *g; + + gr = metacompile(n); + g = maketables(gr); + translatelabels(g); + addfirstsets(g); + freenfagrammar(gr); + return g; +} +``` + +The comment in `Parser/pgen.c` + +> ```c +> /* +> Input is a grammar in extended BNF (using * for repetition, + for +> at-least-once repetition, [] for optional parts, | for alternatives and +> () for grouping). This has already been parsed and turned into a parse +> tree. +> +> Each rule is considered as a regular expression in its own right. +> It is turned into a Non-deterministic Finite Automaton (NFA), which +> is then turned into a Deterministic Finite Automaton (DFA), which is then +> optimized to reduce the number of states. See [Aho&Ullman 77] chapter 3, +> or similar compiler books (this technique is more often used for lexical +> analyzers). +> +> The DFA's are used by the parser as parsing tables in a special way +> that's probably unique. Before they are usable, the FIRST sets of all +> non-terminals are computed. +> */ +> ``` + + + +# read more + +* [what-is-the-difference-between-an-abstract-syntax-tree-and-a-concrete-syntax-tree](https://stackoverflow.com/questions/1888854/what-is-the-difference-between-an-abstract-syntax-tree-and-a-concrete-syntax-tre) +* [What's the Difference Between BNF, EBNF, ABNF?](http://xahlee.info/parser/bnf_ebnf_abnf.html) +* [A Meta-Grammar for PEG Parsers](https://medium.com/@gvanrossum_83706/a-meta-grammar-for-peg-parsers-3d3d502ea332) +* [Python's compiler - from grammar to dfa](https://aoik.me/blog/posts/python-compiler-from-grammar-to-dfa) +* [the-origins-of-pgen](http://python-history.blogspot.com/2018/05/the-origins-of-pgen.html) + diff --git a/Interpreter/compile/compile_3.10.md b/Interpreter/compile/compile_3.10.md new file mode 100644 index 0000000..e1e7b76 --- /dev/null +++ b/Interpreter/compile/compile_3.10.md @@ -0,0 +1,43 @@ +# compile + +# contents + +* [related file](#related-file) +* [tokenizer](#tokenizer) + +The following contents are based on version 3.10.0 alpha 0 + +```bash +git reset --hard 38d3864efe914fda64553e2ec75c9ec15574483f +``` + +# related file + +* Python/pythonrun.c +* Parser/tokenizer.c +* Parser/tokenizer.h +* Parser/parsetok.c + +```bash +% make regen-pegen +PYTHONPATH=./Tools/peg_generator python3 -m pegen -q c \ + ./Grammar/python.gram \ + ./Grammar/Tokens \ + -o ./Parser/parser.new.c +python3 ./Tools/scripts/update_file.py ./Parser/parser.c ./Parser/parser.new.c +% make regen-keyword +\# Regenerate Lib/keyword.py from Grammar/python.gram and Grammar/Tokens +# using Tools/peg_generator/pegen +PYTHONPATH=./Tools/peg_generator python3 -m pegen.keywordgen \ + ./Grammar/python.gram \ + ./Grammar/Tokens \ + ./Lib/keyword.py.new +python3 ./Tools/scripts/update_file.py ./Lib/keyword.py ./Lib/keyword.py.new +``` + + + +# read more + +* [parsing expression grammar](https://en.wikipedia.org/wiki/Parsing_expression_grammar) + diff --git a/Interpreter/compile/compile_cn.md b/Interpreter/compile/compile_cn.md new file mode 100644 index 0000000..1026461 --- /dev/null +++ b/Interpreter/compile/compile_cn.md @@ -0,0 +1,220 @@ +# compile + +# 目录 + +* [相关位置文件](#相关位置文件) +* [pgen](#pgen) + * [the dfa of `ExampleGrammar`](#the-dfa-of-ExampleGrammar) + * [parse](#parse) + * [make dfa](#make-dfa) + +* [更多资料](#更多资料) + +# 相关位置文件 + +* Python/pythonrun.c +* Parser/tokenizer.c +* Parser/tokenizer.h +* Parser/parsetok.c +* Include/grammar.h +* Parser/metagrammar.c +* Include/metagrammar.h +* Parser/pgen.c + +# pgen + +这是 `grammar` 结构体的定义 + +![grammar](./grammar.png) + + + +```bash +make regen-grammar +gcc -g -O0 -Wall -L/usr/local/opt/llvm/lib Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/token.o Parser/pgen.o Objects/obmalloc.o Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain.o -ldl -framework CoreFoundation -o Parser/pgen +# Regenerate Include/graminit.h and Python/graminit.c +# from Grammar/Grammar using pgen +Parser/pgen ./Grammar/Grammar \ + ./Include/graminit.h.new \ + ./Python/graminit.c.new +Translating labels ... +python3 ./Tools/scripts/update_file.py ./Include/graminit.h ./Include/graminit.h.new +python3 ./Tools/scripts/update_file.py ./Python/graminit.c ./Python/graminit.c.new +``` + +在 `Parser/` 目录下, 上面的命令会编译 `Parser/pgenmain.o` 并生成一个叫 `Parser/pgen` 的程序, 这个程序的输入是语法文件, 并且把 `grammar` 结构体, `dfa` 表结构等上图所示的结构体作为输出(总共是两个c文件, `graminit.c` 和 `graminit.h`) + +![pgen](./pgen.png) + +我们先来关注 `pgen` 这个程序 + + `Parser/metagrammar.c` 和 `Include/metagrammar.h` 这两个内建的语法结构体会被用作 Python 语法文件的语法, 生成 `graminit.c` 和 `graminit.h` 之后, 就可以再次编译然后用 Python 的 Grammar 去进行解析 Python 的语法 + +`metagrammar` 的作用是解析 `EBNF` 格式的语法文件 + +`graminit` 的作用是解析 `Python` 源代码 + +他们都是通过同样的方式, 利用不用的 `Grammar` 自动生成的 + +![pgen2](./pgen2.png) + +其中 `Parser/metagrammar.c` 和 `Include/metagrammar.h` 语法如下 + +>```bash +>MSTART: (NEWLINE | RULE)* ENDMARKER +>RULE: NAME ':' RHS NEWLINE +>RHS: ALT ('|' ALT)* +>ALT: ITEM+ +>ITEM: '[' RHS ']' | ATOM ['*' | '+'] +>ATOM: NAME | STRING | '(' RHS ')' +>``` + +(上面的语法结构在源码中无法找到, 是 [python compiler from grammar to dfa](https://aoik.me/blog/posts/python-compiler-from-grammar-to-dfa) 中作者倒推出来的) + +如果提供给内建的`pgen`相同的上述语法文件, 理论上会生成一模一样的 `Parser/metagrammar.c` 和 `Include/metagrammar.h` 文件, 这有点类似编译器自举的概念 + +我们通过写一行非常简单的语法规则来看 `pgen` 是如何工作的 + +```bash +% cat Grammar/ExampleGrammar +START: (NEWLINE | RULE)* ENDMARKER +RULE: NUMBER (ADD NUMBER)* +ADD: '+' | '-' +``` + +我们可以利用默认的 `pgen` 程序来生成我们自己的 `ExampleGrammar` + +```bash +make regen-grammar +Parser/pgen ./Grammar/ExampleGrammar \ + ./Include/examplegrammar.h \ + ./Parser/examplegrammar.c +``` + +为了让上述命令能正常工作, 需要进行一点手动的小改动, 最终的 [examplegrammar.c](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/examplegrammar.c) + +并且编译改动好的程序, 生成一个新的 `pgen2` 程序 [example_grammar.sh](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/example_grammar.sh) + +```bash +sh example_grammar.sh +``` + +在得到 `Parser/pgen2` 之后, 我们能验证下我们的语法文件是否正常工作 + +```bash +% cat my_file.txt +1 + 3 + 4 +2 - 5 +# 运行如下命令是会失败的, 我们的目的是看构建语法树和dfa的过程, 不是真的构建出一个能用的 dfa +# 如果你把 `Parser/parser.c` 中的宏注释掉, 并且留下或新建一行 '#define D(x) x' 然后重新运行上述的命令 +# 你能看到我们的 DFA 状态是在 ACCEPT 的状态下的 +# 表示我们的语法文件是正常工作的 +Parser/pgen2 ./my_file.txt \ + ./Include/my_file.h \ + ./Parser/my_file.c +``` + +## the dfa of `ExampleGrammar` + + + +![dfa_example_grammar](./dfa_example_grammar.png) + +这是根据 [examplegrammar.c](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/examplegrammar.c) 或者 [ExampleGrammar](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/ExampleGrammar) 用上述命令自动生成的 dfa + +## parse + +第一步是根据 [BuiltInGrammar](https://github.com/zpoint/CPython-Internals/tree/master/Interpreter/compile/gen/BuiltInGrammar) 来解析语法文件, 并且生成一颗语法树 + + `Parser/pgenmain.c` 的调用栈如下 + +![pgmain](./pgmain.png) + +`parsetok` 会调用 `PyTokenizer_Get->tok_get` 来获得 tokens, 对于每个 token 都会调用`PyParser_AddToken` 方法 + +`tokens` 是在 `Parser/token.c` 和 `Include/token.h` 中预定义好的, 这两个文件也是通过 `Grammar/Tokens` 文件自动生成的, 自动生成的命令为 `make regen-token` + +这是解析时使用的结构体 + +![parser](./parser.png) + +如果每一步都展开的话, 就会花费太多时间在细节上了, 我们就来用我们上面的 `ExampleGrammar` 来简单的解析下上面的 `my_file.txt` 文件看看 + +在 `push`之后 + +![parse1](./parse1.png) + +在 `shift`之后 + +![parse2](./parse2.png) + +在 `push`之后 + +![parse3](./parse3.png) + +在 `shift`之后 + +![parse4](./parse4.png) + +因为 `ADD_1` 是在 accept 状态下, 所以我们会调用 `POP`, 在 `POP `之后 + +![parse5](./parse5.png) + +在 `push` 和 `shift`之后 + + + +![parse6](./parse6.png) + +重复上述一系列操作之后 + +![parse7](./parse7.png) + + + +## make dfa + +`Parser/pgen.c` 中的下面的函数会把上述的语法树作为输入, 然后把 `grammar` 结构作为输出 + +```c +grammar * +pgen(node *n) +{ + nfagrammar *gr; + grammar *g; + + gr = metacompile(n); + g = maketables(gr); + translatelabels(g); + addfirstsets(g); + freenfagrammar(gr); + return g; +} +``` + + `Parser/pgen.c` 中的注释如下 + +> ```c +> /* +> 输入是一个语法文件/结构, 格式是扩展的 BNF (* 表示重复, + 表示大于等于一次重复, [] 表示可选, +> | 表示或, () 表示分组) 输入的部分已经是一个语法树结构了 +> +> 每一个规则的右边部分都会被当成一个正则表达式, 并被转换成 NFA, 之后再转换成 DFA, +> 之后再通过优化算法减小状态机的状态数量, 可以翻阅 [Aho&Ullman 77] chapter 3, +> 或者相关的编译技术书籍查看相关资料 (这个技术通常是在 lexical analyzers 上用的多) +> +> DFA 被 parser 以一种特殊的方式作为 parsing tables, 这样的处理方式可能在别的地方是看不到的 +> 并且在 DFA 能使用之前, 所有的 non-terminals 的 FIRST 集合都会被提前计算出来 +> */ +> ``` + + + +# 更多资料 + +* [what-is-the-difference-between-an-abstract-syntax-tree-and-a-concrete-syntax-tree](https://stackoverflow.com/questions/1888854/what-is-the-difference-between-an-abstract-syntax-tree-and-a-concrete-syntax-tre) +* [What's the Difference Between BNF, EBNF, ABNF?](http://xahlee.info/parser/bnf_ebnf_abnf.html) +* [A Meta-Grammar for PEG Parsers](https://medium.com/@gvanrossum_83706/a-meta-grammar-for-peg-parsers-3d3d502ea332) +* [Python's compiler - from grammar to dfa](https://aoik.me/blog/posts/python-compiler-from-grammar-to-dfa) +* [the-origins-of-pgen](http://python-history.blogspot.com/2018/05/the-origins-of-pgen.html) + diff --git a/Interpreter/compile/dfa_example_grammar.png b/Interpreter/compile/dfa_example_grammar.png new file mode 100644 index 0000000..2c0d959 Binary files /dev/null and b/Interpreter/compile/dfa_example_grammar.png differ diff --git a/Interpreter/compile/gen/BuiltInGrammar b/Interpreter/compile/gen/BuiltInGrammar new file mode 100644 index 0000000..d174033 --- /dev/null +++ b/Interpreter/compile/gen/BuiltInGrammar @@ -0,0 +1,6 @@ +MSTART: (NEWLINE | RULE)* ENDMARKER +RULE: NAME ':' RHS NEWLINE +RHS: ALT ('|' ALT)* +ALT: ITEM+ +ITEM: '[' RHS ']' | ATOM ['*' | '+'] +ATOM: NAME | STRING | '(' RHS ')' \ No newline at end of file diff --git a/Interpreter/compile/gen/ExampleGrammar b/Interpreter/compile/gen/ExampleGrammar new file mode 100644 index 0000000..1263e06 --- /dev/null +++ b/Interpreter/compile/gen/ExampleGrammar @@ -0,0 +1,3 @@ +START: NUMBER (ADD NUMBER)* ENDMARKER +ADD: '+' | '-' + diff --git a/Interpreter/compile/gen/example_grammar.sh b/Interpreter/compile/gen/example_grammar.sh new file mode 100644 index 0000000..a12ccfd --- /dev/null +++ b/Interpreter/compile/gen/example_grammar.sh @@ -0,0 +1,10 @@ +gcc -c -Wno-unused-result -Wsign-compare -g -O0 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include/internal -I. -I./Include -I/usr/local/opt/llvm/include -DPy_BUILD_CORE -o Parser/examplegrammar.o Parser/examplegrammar.c + +gcc -c -Wno-unused-result -Wsign-compare -g -O0 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include/internal -I. -I./Include -I/usr/local/opt/llvm/include -DPy_BUILD_CORE -o Parser/parsetok_pgen.o Parser/parsetok_pgen.c + +gcc -c -Wno-unused-result -Wsign-compare -g -O0 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include/internal -I. -I./Include -I/usr/local/opt/llvm/include -DPy_BUILD_CORE -o Parser/parser.o Parser/parser.c + +gcc -c -Wno-unused-result -Wsign-compare -g -O0 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include/internal -I. -I./Include -I/usr/local/opt/llvm/include -DPy_BUILD_CORE -o Parser/token.o Parser/token.c + +gcc -c -Wno-unused-result -Wsign-compare -g -O0 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include/internal -I. -I./Include -I/usr/local/opt/llvm/include -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c +gcc -g -O0 -Wall -L/usr/local/opt/llvm/lib Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/examplegrammar.o Parser/firstsets.o Parser/grammar.o Parser/token.o Parser/pgen.o Objects/obmalloc.o Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain.o -ldl -framework CoreFoundation -o Parser/pgen2 diff --git a/Interpreter/compile/gen/examplegrammar.c b/Interpreter/compile/gen/examplegrammar.c new file mode 100644 index 0000000..70a31a5 --- /dev/null +++ b/Interpreter/compile/gen/examplegrammar.c @@ -0,0 +1,65 @@ +/* Generated by Parser/pgen */ + +#include "pgenheaders.h" +#include "grammar.h" + +static arc arcs_0_0[1] = { + {2, 1}, +}; +static arc arcs_0_1[2] = { + {3, 0}, + {4, 2}, +}; +static arc arcs_0_2[1] = { + {0, 2}, +}; +static state states_0[3] = { + {1, arcs_0_0}, + {2, arcs_0_1}, + {1, arcs_0_2}, +}; +static arc arcs_1_0[2] = { + {5, 1}, + {6, 1}, +}; +static arc arcs_1_1[1] = { + {0, 1}, +}; +static state states_1[2] = { + {2, arcs_1_0}, + {1, arcs_1_1}, +}; +static dfa dfas[2] = { + {256, "START", 0, 3, states_0, + "\004"}, + {257, "ADD", 0, 2, states_1, + "\140"}, +}; +static label labels[7] = { + {0, "EMPTY"}, + {256, 0}, + {2, 0}, + {257, 0}, + {0, 0}, + {14, 0}, + {15, 0}, +}; + +grammar _PyParser_Grammar = { + 2, + dfas, + {7, labels}, + 256 +}; + +grammar * +meta_grammar(void) +{ + return &_PyParser_Grammar; +} + +grammar * +Py_meta_grammar(void) +{ + return meta_grammar(); +} diff --git a/Interpreter/compile/grammar.png b/Interpreter/compile/grammar.png new file mode 100644 index 0000000..cd5641e Binary files /dev/null and b/Interpreter/compile/grammar.png differ diff --git a/Interpreter/compile/parse1.png b/Interpreter/compile/parse1.png new file mode 100644 index 0000000..ecf6ff6 Binary files /dev/null and b/Interpreter/compile/parse1.png differ diff --git a/Interpreter/compile/parse2.png b/Interpreter/compile/parse2.png new file mode 100644 index 0000000..5f82459 Binary files /dev/null and b/Interpreter/compile/parse2.png differ diff --git a/Interpreter/compile/parse3.png b/Interpreter/compile/parse3.png new file mode 100644 index 0000000..d99bd3f Binary files /dev/null and b/Interpreter/compile/parse3.png differ diff --git a/Interpreter/compile/parse4.png b/Interpreter/compile/parse4.png new file mode 100644 index 0000000..aca7e21 Binary files /dev/null and b/Interpreter/compile/parse4.png differ diff --git a/Interpreter/compile/parse5.png b/Interpreter/compile/parse5.png new file mode 100644 index 0000000..c5d73e7 Binary files /dev/null and b/Interpreter/compile/parse5.png differ diff --git a/Interpreter/compile/parse6.png b/Interpreter/compile/parse6.png new file mode 100644 index 0000000..6c21e67 Binary files /dev/null and b/Interpreter/compile/parse6.png differ diff --git a/Interpreter/compile/parse7.png b/Interpreter/compile/parse7.png new file mode 100644 index 0000000..33fc187 Binary files /dev/null and b/Interpreter/compile/parse7.png differ diff --git a/Interpreter/compile/parser.png b/Interpreter/compile/parser.png new file mode 100644 index 0000000..6623695 Binary files /dev/null and b/Interpreter/compile/parser.png differ diff --git a/Interpreter/compile/pgen.png b/Interpreter/compile/pgen.png new file mode 100644 index 0000000..100d0a0 Binary files /dev/null and b/Interpreter/compile/pgen.png differ diff --git a/Interpreter/compile/pgen2.png b/Interpreter/compile/pgen2.png new file mode 100644 index 0000000..670078f Binary files /dev/null and b/Interpreter/compile/pgen2.png differ diff --git a/Interpreter/compile/pgmain.png b/Interpreter/compile/pgmain.png new file mode 100644 index 0000000..e6a07e7 Binary files /dev/null and b/Interpreter/compile/pgmain.png differ diff --git a/Interpreter/compile2/ast.png b/Interpreter/compile2/ast.png new file mode 100644 index 0000000..7d6d2f1 Binary files /dev/null and b/Interpreter/compile2/ast.png differ diff --git a/Interpreter/compile2/compile.md b/Interpreter/compile2/compile.md new file mode 100644 index 0000000..1358c5f --- /dev/null +++ b/Interpreter/compile2/compile.md @@ -0,0 +1,239 @@ +# CST TO AST + +# contents + +* [related file](#related-file) +* [pythonrun](#pythonrun) +* [CST TO AST](#CST-TO-AST) +* [read more](#read-more) + +# related file + +* Python/ast.c + +* Python/pythonrun.c + +* Include/Python-ast.h + +* Python/Python-ast.c + +* Python/asdl.c + +* Include/asdl.h + + + +The following command will generate `Include/Python-ast.h` and `Python/Python-ast.c` from `Parser/Python.asdl`, which will be used to construct `AST` from the previos [parse tree](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/compile/compile.md#parse) + +```shell +% make regen-ast +# Regenerate Include/Python-ast.h using Parser/asdl_c.py -h +./install-sh -c -d ./Include +python3 ./Parser/asdl_c.py \ + -h ./Include/Python-ast.h.new \ + ./Parser/Python.asdl +python3 ./Tools/scripts/update_file.py ./Include/Python-ast.h ./Include/Python-ast.h.new +# Regenerate Python/Python-ast.c using Parser/asdl_c.py -c +./install-sh -c -d ./Python +python3 ./Parser/asdl_c.py \ + -c ./Python/Python-ast.c.new \ + ./Parser/Python.asdl +python3 ./Tools/scripts/update_file.py ./Python/Python-ast.c ./Python/Python-ast.c.new + +``` + +> * Lowercase names are non-terminals. +> * Uppercase names are terminals. +> * Literal tokens are in double quotes. +> * `[]` means zero or one. +> * `{}` means one or more. +> * `?` means it is optional, `*` means 0 or more + +# pythonrun + +We take the interactive loop as example, the call stack of `pythonrun` + +![pythonrun](./pythonrun.png) + +# CST TO AST + +Let's focus on `PyAST_FromNodeObject` + +If we execute + +```python3 +a = 2 +``` + +This is the CST of above expression + +```shell +n_type: 256, n_type_str: single_input, n_str: (null), n_children: 1 + n_type: 270, n_type_str: simple_stmt, n_str: (null), n_children: 2 + n_type: 271, n_type_str: small_stmt, n_str: (null), n_children: 1 + n_type: 272, n_type_str: expr_stmt, n_str: (null), n_children: 3 + n_type: 274, n_type_str: testlist_star_expr, n_str: (null), n_children: 1 + n_type: 305, n_type_str: test, n_str: (null), n_children: 1 + n_type: 309, n_type_str: or_test, n_str: (null), n_children: 1 + n_type: 310, n_type_str: and_test, n_str: (null), n_children: 1 + n_type: 311, n_type_str: not_test, n_str: (null), n_children: 1 + n_type: 312, n_type_str: comparison, n_str: (null), n_children: 1 + n_type: 315, n_type_str: expr, n_str: (null), n_children: 1 + n_type: 316, n_type_str: xor_expr, n_str: (null), n_children: 1 + n_type: 317, n_type_str: and_expr, n_str: (null), n_children: 1 + n_type: 318, n_type_str: shift_expr, n_str: (null), n_children: 1 + n_type: 319, n_type_str: arith_expr, n_str: (null), n_children: 1 + n_type: 320, n_type_str: term, n_str: (null), n_children: 1 + n_type: 321, n_type_str: factor, n_str: (null), n_children: 1 + n_type: 322, n_type_str: power, n_str: (null), n_children: 1 + n_type: 323, n_type_str: atom_expr, n_str: (null), n_children: 1 + n_type: 324, n_type_str: atom, n_str: (null), n_children: 1 + n_type: 1, n_type_str: NAME, n_str: a, n_children: 0 + n_type: 22, n_type_str: EQUAL, n_str: =, n_children: 0 + n_type: 274, n_type_str: testlist_star_expr, n_str: (null), n_children: 1 + n_type: 305, n_type_str: test, n_str: (null), n_children: 1 + n_type: 309, n_type_str: or_test, n_str: (null), n_children: 1 + n_type: 310, n_type_str: and_test, n_str: (null), n_children: 1 + n_type: 311, n_type_str: not_test, n_str: (null), n_children: 1 + n_type: 312, n_type_str: comparison, n_str: (null), n_children: 1 + n_type: 315, n_type_str: expr, n_str: (null), n_children: 1 + n_type: 316, n_type_str: xor_expr, n_str: (null), n_children: 1 + n_type: 317, n_type_str: and_expr, n_str: (null), n_children: 1 + n_type: 318, n_type_str: shift_expr, n_str: (null), n_children: 1 + n_type: 319, n_type_str: arith_expr, n_str: (null), n_children: 1 + n_type: 320, n_type_str: term, n_str: (null), n_children: 1 + n_type: 321, n_type_str: factor, n_str: (null), n_children: 1 + n_type: 322, n_type_str: power, n_str: (null), n_children: 1 + n_type: 323, n_type_str: atom_expr, n_str: (null), n_children: 1 + n_type: 324, n_type_str: atom, n_str: (null), n_children: 1 + n_type: 2, n_type_str: NUMBER, n_str: 2, n_children: 0 + n_type: 4, n_type_str: NEWLINE, n_str: , n_children: 0 +``` + +In diagram representation, you can check the `Grammar/Grammar` file + +![cst](./cst.png) + +The CST to AST part is written in `Python/ast.c`, The function inside the file is a hand writting parsing according to gramma file, while the return type `expr_ty` is the above auto generated c structure + +```c +// in Python/ast.c +static expr_ty +ast_for_atom(struct compiling *c, const node *n) +{ + /* atom: '(' [yield_expr|testlist_comp] ')' | '[' [testlist_comp] ']' + | '{' [dictmaker|testlist_comp] '}' | NAME | NUMBER | STRING+ + | '...' | 'None' | 'True' | 'False' + */ + node *ch = CHILD(n, 0); + + switch (TYPE(ch)) { + case NAME: { + PyObject *name; + const char *s = STR(ch); + size_t len = strlen(s); + if (len >= 4 && len <= 5) { + if (!strcmp(s, "None")) + return Constant(Py_None, LINENO(n), n->n_col_offset, c->c_arena); + if (!strcmp(s, "True")) + return Constant(Py_True, LINENO(n), n->n_col_offset, c->c_arena); + if (!strcmp(s, "False")) + return Constant(Py_False, LINENO(n), n->n_col_offset, c->c_arena); + } + name = new_identifier(s, c); + if (!name) + return NULL; + /* All names start in Load context, but may later be changed. */ + return Name(name, Load, LINENO(n), n->n_col_offset, c->c_arena); + } + // ... + case NUMBER: { + PyObject *pynum = parsenumber(c, STR(ch)); + if (!pynum) + return NULL; + + if (PyArena_AddPyObject(c->c_arena, pynum) < 0) { + Py_DECREF(pynum); + return NULL; + } + return Constant(pynum, LINENO(n), n->n_col_offset, c->c_arena); + } + // ... + default: + PyErr_Format(PyExc_SystemError, "unhandled atom %d", TYPE(ch)); + return NULL; + } +} +``` + +`Name` and `Load` are defined in `Python/Python-ast.c` + +```c +// Python/Python-ast.c +expr_ty +Name(identifier id, expr_context_ty ctx, int lineno, int col_offset, PyArena + *arena) +{ + expr_ty p; + /* ... */ + p->kind = Name_kind; + p->v.Name.id = id; + p->v.Name.ctx = ctx; + p->lineno = lineno; + p->col_offset = col_offset; + return p; +} + +// Include/Python-ast.h +typedef enum _expr_context { Load=1, Store=2, Del=3, AugLoad=4, AugStore=5, + Param=6 } expr_context_ty; +// ... +typedef struct _expr *expr_ty; +``` + +`_Py_asdl_seq_new` is defined in `Python/asdl.c` and `asdl_seq_SET` is defined in `Include/asdl.h` + +```c +// Python/ast.c +ast_for_global_stmt(struct compiling *c, const node *n) +{ + /* global_stmt: 'global' NAME (',' NAME)* */ + identifier name; + asdl_seq *s; + int i; + + REQ(n, global_stmt); + s = _Py_asdl_seq_new(NCH(n) / 2, c->c_arena); + if (!s) + return NULL; + for (i = 1; i < NCH(n); i += 2) { + name = NEW_IDENTIFIER(CHILD(n, i)); + if (!name) + return NULL; + asdl_seq_SET(s, i / 2, name); + } + return Global(s, LINENO(n), n->n_col_offset, c->c_arena); +} + +// Include/asdl.h +typedef struct { + Py_ssize_t size; + void *elements[1]; +} asdl_seq; +// ... +#define asdl_seq_SET(S, I, V) (S)->elements[I] = (V) +``` + +This is the AST after conversion + +![ast](./ast.png) + +The following [function](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/compile2/inspect_ast.c) can be used to inspect the above AST sctucture + +# read more + +* [using-asdl-to-describe-asts-in-compilers](https://eli.thegreenplace.net/2014/06/04/using-asdl-to-describe-asts-in-compilers) +* [What is Zephyr ASDL](https://www.oilshell.org/blog/2016/12/11.html) +* [Python's compiler - from CST to AST](https://aoik.me/blog/posts/python-compiler-from-cst-to-ast) +* [Design of CPython’s Compiler](https://devguide.python.org/compiler/) + diff --git a/Interpreter/compile2/compile_cn.md b/Interpreter/compile2/compile_cn.md new file mode 100644 index 0000000..e14dfe2 --- /dev/null +++ b/Interpreter/compile2/compile_cn.md @@ -0,0 +1,239 @@ +# CST TO AST + +# 目录 + +* [相关位置文件](#相关位置文件) +* [pythonrun](#pythonrun) +* [CST 到 AST](#CST-到-AST) +* [更多资料](#更多资料) + +# 相关位置文件 + +* Python/ast.c + +* Python/pythonrun.c + +* Include/Python-ast.h + +* Python/Python-ast.c + +* Python/asdl.c + +* Include/asdl.h + + + +下面的命令会从 `Parser/Python.asdl` 中生成 `Include/Python-ast.h` 和 `Python/Python-ast.c`, 这两个自动生成的文件的结构体会被用来从之前的 [语法树](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/compile/compile_cn.md#parse) 中生成 `AST` + +```shell +% make regen-ast +# Regenerate Include/Python-ast.h using Parser/asdl_c.py -h +./install-sh -c -d ./Include +python3 ./Parser/asdl_c.py \ + -h ./Include/Python-ast.h.new \ + ./Parser/Python.asdl +python3 ./Tools/scripts/update_file.py ./Include/Python-ast.h ./Include/Python-ast.h.new +# Regenerate Python/Python-ast.c using Parser/asdl_c.py -c +./install-sh -c -d ./Python +python3 ./Parser/asdl_c.py \ + -c ./Python/Python-ast.c.new \ + ./Parser/Python.asdl +python3 ./Tools/scripts/update_file.py ./Python/Python-ast.c ./Python/Python-ast.c.new + +``` + +> * 小写名称是 non-terminals. +> * 大写名称是 terminals +> * 文本 tokens 用双引号引起来 +> * `[]` 表示 >= 0 个 +> * `{}` 表示 >= 1 个 +> * `?` 表示可能有可能没有, `*` 表示 >= 0 个 + +# pythonrun + +我们以交互式循环为例, 展开 `pythonrun` 的调用栈看看 + +![pythonrun](./pythonrun.png) + +# CST 到 AST + +我们把关注点放到 `PyAST_FromNodeObject` 上 + +如果我们执行 + +```python3 +a = 2 +``` + +这是上述语句生成的 CST 结构 + +```shell +n_type: 256, n_type_str: single_input, n_str: (null), n_children: 1 + n_type: 270, n_type_str: simple_stmt, n_str: (null), n_children: 2 + n_type: 271, n_type_str: small_stmt, n_str: (null), n_children: 1 + n_type: 272, n_type_str: expr_stmt, n_str: (null), n_children: 3 + n_type: 274, n_type_str: testlist_star_expr, n_str: (null), n_children: 1 + n_type: 305, n_type_str: test, n_str: (null), n_children: 1 + n_type: 309, n_type_str: or_test, n_str: (null), n_children: 1 + n_type: 310, n_type_str: and_test, n_str: (null), n_children: 1 + n_type: 311, n_type_str: not_test, n_str: (null), n_children: 1 + n_type: 312, n_type_str: comparison, n_str: (null), n_children: 1 + n_type: 315, n_type_str: expr, n_str: (null), n_children: 1 + n_type: 316, n_type_str: xor_expr, n_str: (null), n_children: 1 + n_type: 317, n_type_str: and_expr, n_str: (null), n_children: 1 + n_type: 318, n_type_str: shift_expr, n_str: (null), n_children: 1 + n_type: 319, n_type_str: arith_expr, n_str: (null), n_children: 1 + n_type: 320, n_type_str: term, n_str: (null), n_children: 1 + n_type: 321, n_type_str: factor, n_str: (null), n_children: 1 + n_type: 322, n_type_str: power, n_str: (null), n_children: 1 + n_type: 323, n_type_str: atom_expr, n_str: (null), n_children: 1 + n_type: 324, n_type_str: atom, n_str: (null), n_children: 1 + n_type: 1, n_type_str: NAME, n_str: a, n_children: 0 + n_type: 22, n_type_str: EQUAL, n_str: =, n_children: 0 + n_type: 274, n_type_str: testlist_star_expr, n_str: (null), n_children: 1 + n_type: 305, n_type_str: test, n_str: (null), n_children: 1 + n_type: 309, n_type_str: or_test, n_str: (null), n_children: 1 + n_type: 310, n_type_str: and_test, n_str: (null), n_children: 1 + n_type: 311, n_type_str: not_test, n_str: (null), n_children: 1 + n_type: 312, n_type_str: comparison, n_str: (null), n_children: 1 + n_type: 315, n_type_str: expr, n_str: (null), n_children: 1 + n_type: 316, n_type_str: xor_expr, n_str: (null), n_children: 1 + n_type: 317, n_type_str: and_expr, n_str: (null), n_children: 1 + n_type: 318, n_type_str: shift_expr, n_str: (null), n_children: 1 + n_type: 319, n_type_str: arith_expr, n_str: (null), n_children: 1 + n_type: 320, n_type_str: term, n_str: (null), n_children: 1 + n_type: 321, n_type_str: factor, n_str: (null), n_children: 1 + n_type: 322, n_type_str: power, n_str: (null), n_children: 1 + n_type: 323, n_type_str: atom_expr, n_str: (null), n_children: 1 + n_type: 324, n_type_str: atom, n_str: (null), n_children: 1 + n_type: 2, n_type_str: NUMBER, n_str: 2, n_children: 0 + n_type: 4, n_type_str: NEWLINE, n_str: , n_children: 0 +``` + +图像表示的话如下, 你可以参数 `Grammar/Grammar` 语法文件 + +![cst](./cst.png) + +CST 到 AST 的转换部分是在 `Python/ast.c` 中写好的, 这个文件中对应的转换函数都是根据语法规则去进行对应的处理, 返回结构比如 `expr_ty` 则是上述自动生成的 C 文件生成的 + +```c +// in Python/ast.c +static expr_ty +ast_for_atom(struct compiling *c, const node *n) +{ + /* atom: '(' [yield_expr|testlist_comp] ')' | '[' [testlist_comp] ']' + | '{' [dictmaker|testlist_comp] '}' | NAME | NUMBER | STRING+ + | '...' | 'None' | 'True' | 'False' + */ + node *ch = CHILD(n, 0); + + switch (TYPE(ch)) { + case NAME: { + PyObject *name; + const char *s = STR(ch); + size_t len = strlen(s); + if (len >= 4 && len <= 5) { + if (!strcmp(s, "None")) + return Constant(Py_None, LINENO(n), n->n_col_offset, c->c_arena); + if (!strcmp(s, "True")) + return Constant(Py_True, LINENO(n), n->n_col_offset, c->c_arena); + if (!strcmp(s, "False")) + return Constant(Py_False, LINENO(n), n->n_col_offset, c->c_arena); + } + name = new_identifier(s, c); + if (!name) + return NULL; + /* All names start in Load context, but may later be changed. */ + return Name(name, Load, LINENO(n), n->n_col_offset, c->c_arena); + } + // ... + case NUMBER: { + PyObject *pynum = parsenumber(c, STR(ch)); + if (!pynum) + return NULL; + + if (PyArena_AddPyObject(c->c_arena, pynum) < 0) { + Py_DECREF(pynum); + return NULL; + } + return Constant(pynum, LINENO(n), n->n_col_offset, c->c_arena); + } + // ... + default: + PyErr_Format(PyExc_SystemError, "unhandled atom %d", TYPE(ch)); + return NULL; + } +} +``` + +`Name` 和 `Load` 在 `Python/Python-ast.c` 中定义好了 + +```c +// Python/Python-ast.c +expr_ty +Name(identifier id, expr_context_ty ctx, int lineno, int col_offset, PyArena + *arena) +{ + expr_ty p; + /* ... */ + p->kind = Name_kind; + p->v.Name.id = id; + p->v.Name.ctx = ctx; + p->lineno = lineno; + p->col_offset = col_offset; + return p; +} + +// Include/Python-ast.h +typedef enum _expr_context { Load=1, Store=2, Del=3, AugLoad=4, AugStore=5, + Param=6 } expr_context_ty; +// ... +typedef struct _expr *expr_ty; +``` + +`_Py_asdl_seq_new` 在 `Python/asdl.c` 中定义好了, 而 `asdl_seq_SET` 在 `Include/asdl.h` 中定义好了 + +```c +// Python/ast.c +ast_for_global_stmt(struct compiling *c, const node *n) +{ + /* global_stmt: 'global' NAME (',' NAME)* */ + identifier name; + asdl_seq *s; + int i; + + REQ(n, global_stmt); + s = _Py_asdl_seq_new(NCH(n) / 2, c->c_arena); + if (!s) + return NULL; + for (i = 1; i < NCH(n); i += 2) { + name = NEW_IDENTIFIER(CHILD(n, i)); + if (!name) + return NULL; + asdl_seq_SET(s, i / 2, name); + } + return Global(s, LINENO(n), n->n_col_offset, c->c_arena); +} + +// Include/asdl.h +typedef struct { + Py_ssize_t size; + void *elements[1]; +} asdl_seq; +// ... +#define asdl_seq_SET(S, I, V) (S)->elements[I] = (V) +``` + +这是转换后的 AST 结构 + +![ast](./ast.png) + +这里的 [函数](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/compile2/inspect_ast.c) 可以用来打印上述 AST 的结构内容 + +# 更多资料 + +* [using-asdl-to-describe-asts-in-compilers](https://eli.thegreenplace.net/2014/06/04/using-asdl-to-describe-asts-in-compilers) +* [What is Zephyr ASDL](https://www.oilshell.org/blog/2016/12/11.html) +* [Python's compiler - from CST to AST](https://aoik.me/blog/posts/python-compiler-from-cst-to-ast) +* [Design of CPython’s Compiler](https://devguide.python.org/compiler/) + diff --git a/Interpreter/compile2/cst.png b/Interpreter/compile2/cst.png new file mode 100644 index 0000000..b3a8944 Binary files /dev/null and b/Interpreter/compile2/cst.png differ diff --git a/Interpreter/compile2/inspect_ast.c b/Interpreter/compile2/inspect_ast.c new file mode 100644 index 0000000..988a747 --- /dev/null +++ b/Interpreter/compile2/inspect_ast.c @@ -0,0 +1,50 @@ +void pr_expr_ty(expr_ty value) +{ + printf("value->kind: %d\n", value->kind); + PyObject *op = NULL; + switch (value->kind) + { + case Constant_kind: + op = value->v.Constant.value; + printf("value->v.Constant.value: %s\n", PyUnicode_AsUTF8(op->ob_type->tp_repr(op))); + break; + case Name_kind: + op = value->v.Name.id; + printf("identifier: %s, _expr_context: %d\n", PyUnicode_AsUTF8(op->ob_type->tp_repr(op)), value->v.Name.ctx); + break; + case BinOp_kind: + printf("left:\n"); + pr_expr_ty(value->v.BinOp.left); + printf("operator: %d\n", value->v.BinOp.op); + printf("right:\n"); + pr_expr_ty(value->v.BinOp.right); + break; + default: + printf("unknown kind\n"); + } + +} + +void pr_asdl_seq(asdl_seq* target) +{ + printf("target->size: %zd\n", target->size); + for (Py_ssize_t i = 0; i < target->size; ++i) + { + pr_expr_ty((expr_ty)target->elements[i]); + } + +} + +void pr_ast(stmt_ty s) +{ + printf("in pr_ast, kind: %d\n", s->kind); + switch (s->kind) + { + case Assign_kind: + pr_asdl_seq(s->v.Assign.targets); + pr_expr_ty(s->v.Assign.value); + break; + default: + printf("unknown kind\n"); + } +} \ No newline at end of file diff --git a/Interpreter/compile2/pythonrun.png b/Interpreter/compile2/pythonrun.png new file mode 100644 index 0000000..5cf15fd Binary files /dev/null and b/Interpreter/compile2/pythonrun.png differ diff --git a/Interpreter/compile3/ast_after.png b/Interpreter/compile3/ast_after.png new file mode 100644 index 0000000..f79a858 Binary files /dev/null and b/Interpreter/compile3/ast_after.png differ diff --git a/Interpreter/compile3/ast_before.png b/Interpreter/compile3/ast_before.png new file mode 100644 index 0000000..53a1f68 Binary files /dev/null and b/Interpreter/compile3/ast_before.png differ diff --git a/Interpreter/compile3/compile.md b/Interpreter/compile3/compile.md new file mode 100644 index 0000000..8e5159a --- /dev/null +++ b/Interpreter/compile3/compile.md @@ -0,0 +1,260 @@ +# AST TO byte code + +# contents + +* [related file](#related-file) +* [optimize](#optimize) +* [symtable](#symtable) +* [CodeObject](#CodeObject) +* [read more](#read-more) + +# related file + +* Python/compile.c + +* Python/pythonrun.c + +* Python/symtable.c + +* Include/symtable.h + +* Python/ast_opt.c + + + +Let's generate the python code object from previous [AST tree](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/compile2/compile.md) + +The aforementioned `pythonrun` call stack + +![pythonrun](./pythonrun.png) + +We will foucus on `PyAST_CompileObject` this time + +```c +PyCodeObject * +PyAST_CompileObject(mod_ty mod, PyObject *filename, PyCompilerFlags *flags, + int optimize, PyArena *arena) +{ + struct compiler c; + PyCodeObject *co = NULL; + PyCompilerFlags local_flags; + int merged; + // ... omit ... + + if (!_PyAST_Optimize(mod, arena, c.c_optimize)) { + goto finally; + } + + c.c_st = PySymtable_BuildObject(mod, filename, c.c_future); + if (c.c_st == NULL) { + if (!PyErr_Occurred()) + PyErr_SetString(PyExc_SystemError, "no symtable"); + goto finally; + } + + co = compiler_mod(&c, mod); + + finally: + compiler_free(&c); + assert(co || PyErr_Occurred()); + return co; +} +``` + +# optimize + +`_PyAST_Optimize` will do some optimize such as constant folding + +```c +static int +fold_binop(expr_ty node, PyArena *arena, int optimize) +{ + expr_ty lhs, rhs; + lhs = node->v.BinOp.left; + rhs = node->v.BinOp.right; + if (lhs->kind != Constant_kind || rhs->kind != Constant_kind) { + return 1; + } + + PyObject *lv = lhs->v.Constant.value; + PyObject *rv = rhs->v.Constant.value; + PyObject *newval; + + switch (node->v.BinOp.op) { + case Add: + newval = PyNumber_Add(lv, rv); + break; + case Sub: + newval = PyNumber_Subtract(lv, rv); + break; + case Mult: + newval = safe_multiply(lv, rv); + break; + case Div: + newval = PyNumber_TrueDivide(lv, rv); + break; + case FloorDiv: + newval = PyNumber_FloorDivide(lv, rv); + break; + case Mod: + newval = safe_mod(lv, rv); + break; + case Pow: + newval = safe_power(lv, rv); + break; + case LShift: + newval = safe_lshift(lv, rv); + break; + case RShift: + newval = PyNumber_Rshift(lv, rv); + break; + case BitOr: + newval = PyNumber_Or(lv, rv); + break; + case BitXor: + newval = PyNumber_Xor(lv, rv); + break; + case BitAnd: + newval = PyNumber_And(lv, rv); + break; + default: // Unknown operator + return 1; + } + return make_const(node, newval, arena); +} +``` + +For example + +```python3 +a = 3 + 4 +``` + +The AST is + +![ast_before](./ast_before.png) + +After `fold_binop`, it becomes + +![ast_after](./ast_after.png) + +The constant folding will also + +* expand binary op(as above) + +* expand your unary op(`not/is/isnot`) +* change you list literal to tuple +* fetch the constant sub index +* etc ... + +# symtable + +After `_PyAST_Optimize`, the `PySymtable_BuildObject` will traverse the AST and add stores the the function/class/name definition to a Python dict object + +![symtable](./symtable.png) + +# CodeObject + +`compiler_mod` will traverse the AST, generate the appropriate byte code instruction and helper data, finally generate and return a`PyCodeObject` + +```c +static int +compiler_visit_expr1(struct compiler *c, expr_ty e) +{ + // ... + case Constant_kind: + ADDOP_LOAD_CONST(c, e->v.Constant.value); + break; + // ... +} + +static int +compiler_addop_load_const(struct compiler *c, PyObject *o) +{ + // add the constant value to a dictionary object: c->u->u_consts, returns the index after converted to list(offset of arg list) + Py_ssize_t arg = compiler_add_const(c, o); + if (arg < 0) + return 0; + // add the `LOAD_CONST arg` to the next instruction + return compiler_addop_i(c, LOAD_CONST, arg); +} + +// Below functions can be found in Python/compile.c +static int +compiler_addop_i(struct compiler *c, int opcode, Py_ssize_t oparg) +{ + struct instr *i; + int off; + + // ... + + off = compiler_next_instr(c, c->u->u_curblock); + if (off < 0) + return 0; + i = &c->u->u_curblock->b_instr[off]; + i->i_opcode = opcode; + i->i_oparg = Py_SAFE_DOWNCAST(oparg, Py_ssize_t, int); + compiler_set_lineno(c, off); + return 1; +} + +static Py_ssize_t +compiler_add_const(struct compiler *c, PyObject *o) +{ + PyObject *key = merge_consts_recursive(c, o); + if (key == NULL) { + return -1; + } + + Py_ssize_t arg = compiler_add_o(c, c->u->u_consts, key); + Py_DECREF(key); + return arg; +} + +static Py_ssize_t +compiler_add_o(struct compiler *c, PyObject *dict, PyObject *o) +{ + PyObject *v; + Py_ssize_t arg; + + v = PyDict_GetItemWithError(dict, o); + if (!v) { + if (PyErr_Occurred()) { + return -1; + } + arg = PyDict_GET_SIZE(dict); + v = PyLong_FromSsize_t(arg); + if (!v) { + return -1; + } + if (PyDict_SetItem(dict, o, v) < 0) { + Py_DECREF(v); + return -1; + } + Py_DECREF(v); + } + else + arg = PyLong_AsLong(v); + return arg; +} +``` + +The constants are stored in dict format, key represent the constant `PyObject`, value is the offset `{'a': 0}` + +It will be converted to a list object in the order of value in `Python/compile.c->consts_dict_keys_inorder` + +Finally, the `Python/compile.c->makecode` converts the compiled instrcutions and helper data to a newly create `PyCodeObject` + +```shell + 1 0 LOAD_CONST 0 (7) + 2 STORE_NAME 0 (a) + 4 LOAD_CONST 1 (None) + 6 RETURN_VALUE +``` + + + +# read more + +* [Python's compiler - from AST to code object](https://aoik.me/blog/posts/python-compiler-from-ast-to-code-object) + diff --git a/Interpreter/compile3/compile_cn.md b/Interpreter/compile3/compile_cn.md new file mode 100644 index 0000000..2a7d9ed --- /dev/null +++ b/Interpreter/compile3/compile_cn.md @@ -0,0 +1,259 @@ +# AST 到 字节码 + +# 目录 + +* [related file](#related-file) +* [optimize](#optimize) +* [symtable](#symtable) +* [CodeObject](#CodeObject) +* [read more](#read-more) + +# 相关位置文件 + +* Python/compile.c + +* Python/pythonrun.c + +* Python/symtable.c + +* Include/symtable.h + +* Python/ast_opt.c + + + +我们来尝试从之前的 [语法树](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/compile2/compile_cn.md) 中生成对应的字节码 + +前面提到过的 `pythonrun` 的调用栈如下 + +![pythonrun](./pythonrun.png) + +这次关注的是 `PyAST_CompileObject` 这个函数 + +```c +PyCodeObject * +PyAST_CompileObject(mod_ty mod, PyObject *filename, PyCompilerFlags *flags, + int optimize, PyArena *arena) +{ + struct compiler c; + PyCodeObject *co = NULL; + PyCompilerFlags local_flags; + int merged; + // ... 忽略 ... + + if (!_PyAST_Optimize(mod, arena, c.c_optimize)) { + goto finally; + } + + c.c_st = PySymtable_BuildObject(mod, filename, c.c_future); + if (c.c_st == NULL) { + if (!PyErr_Occurred()) + PyErr_SetString(PyExc_SystemError, "no symtable"); + goto finally; + } + + co = compiler_mod(&c, mod); + + finally: + compiler_free(&c); + assert(co || PyErr_Occurred()); + return co; +} +``` + +# optimize + +`_PyAST_Optimize` 会针对AST做一些优化, 比如常量折叠 + +```c +static int +fold_binop(expr_ty node, PyArena *arena, int optimize) +{ + expr_ty lhs, rhs; + lhs = node->v.BinOp.left; + rhs = node->v.BinOp.right; + if (lhs->kind != Constant_kind || rhs->kind != Constant_kind) { + return 1; + } + + PyObject *lv = lhs->v.Constant.value; + PyObject *rv = rhs->v.Constant.value; + PyObject *newval; + + switch (node->v.BinOp.op) { + case Add: + newval = PyNumber_Add(lv, rv); + break; + case Sub: + newval = PyNumber_Subtract(lv, rv); + break; + case Mult: + newval = safe_multiply(lv, rv); + break; + case Div: + newval = PyNumber_TrueDivide(lv, rv); + break; + case FloorDiv: + newval = PyNumber_FloorDivide(lv, rv); + break; + case Mod: + newval = safe_mod(lv, rv); + break; + case Pow: + newval = safe_power(lv, rv); + break; + case LShift: + newval = safe_lshift(lv, rv); + break; + case RShift: + newval = PyNumber_Rshift(lv, rv); + break; + case BitOr: + newval = PyNumber_Or(lv, rv); + break; + case BitXor: + newval = PyNumber_Xor(lv, rv); + break; + case BitAnd: + newval = PyNumber_And(lv, rv); + break; + default: // Unknown operator + return 1; + } + return make_const(node, newval, arena); +} +``` + +比如 + +```python3 +a = 3 + 4 +``` + +AST 表示如下 + +![ast_before](./ast_before.png) + +在 `fold_binop` 之后, 变成了 + +![ast_after](./ast_after.png) + +这个常量折叠的方法同样会做如下操作 + +* 展开二元操作(如上图) + +* 展开一元操作(`not/is/isnot`) + +* 把你的手写 list 转换成 tuple +* 展开常量可迭代对象的常量下标 `[]` +* 等待 ... + +# symtable + +在 `_PyAST_Optimize` 之后, `PySymtable_BuildObject` 会遍历 AST 并把 function/class/name 等定义/声明的变量加到一个 Python 字典对象中 + +![symtable](./symtable.png) + +# CodeObject + +`compiler_mod` 会遍历这个 AST, 并且生成对应的指令集和所需要的数据, 之后生成对应的 `PyCodeObject` 对象 + +```c +static int +compiler_visit_expr1(struct compiler *c, expr_ty e) +{ + // ... + case Constant_kind: + ADDOP_LOAD_CONST(c, e->v.Constant.value); + break; + // ... +} + +static int +compiler_addop_load_const(struct compiler *c, PyObject *o) +{ + // 把对应的常量加到字典对象 c->u->u_consts 中, 并把对应常量的最后的位移位置返回 + Py_ssize_t arg = compiler_add_const(c, o); + if (arg < 0) + return 0; + // 把 `LOAD_CONST arg` 加到下一个指令中 + return compiler_addop_i(c, LOAD_CONST, arg); +} + +// 下列的函数在 Python/compile.c 中可以找到 +static int +compiler_addop_i(struct compiler *c, int opcode, Py_ssize_t oparg) +{ + struct instr *i; + int off; + + // ... + + off = compiler_next_instr(c, c->u->u_curblock); + if (off < 0) + return 0; + i = &c->u->u_curblock->b_instr[off]; + i->i_opcode = opcode; + i->i_oparg = Py_SAFE_DOWNCAST(oparg, Py_ssize_t, int); + compiler_set_lineno(c, off); + return 1; +} + +static Py_ssize_t +compiler_add_const(struct compiler *c, PyObject *o) +{ + PyObject *key = merge_consts_recursive(c, o); + if (key == NULL) { + return -1; + } + + Py_ssize_t arg = compiler_add_o(c, c->u->u_consts, key); + Py_DECREF(key); + return arg; +} + +static Py_ssize_t +compiler_add_o(struct compiler *c, PyObject *dict, PyObject *o) +{ + PyObject *v; + Py_ssize_t arg; + + v = PyDict_GetItemWithError(dict, o); + if (!v) { + if (PyErr_Occurred()) { + return -1; + } + arg = PyDict_GET_SIZE(dict); + v = PyLong_FromSsize_t(arg); + if (!v) { + return -1; + } + if (PyDict_SetItem(dict, o, v) < 0) { + Py_DECREF(v); + return -1; + } + Py_DECREF(v); + } + else + arg = PyLong_AsLong(v); + return arg; +} +``` + +常量在上述操作时是以字典形式存储的, `{'a': 0}` 键表示对应常量的 `PyObject`, 值表示参数位移 + +它最终会被 `Python/compile.c->consts_dict_keys_inorder` 这个函数按照值里的位移顺序被转换成对应的列表对象, + +最终, `Python/compile.c->makecode` 会把上述的指令和对应的数据结构存储到新建的 `PyCodeObject ` 上 + +```shell + 1 0 LOAD_CONST 0 (7) + 2 STORE_NAME 0 (a) + 4 LOAD_CONST 1 (None) + 6 RETURN_VALUE +``` + +# read more + +* [Python's compiler - from AST to code object](https://aoik.me/blog/posts/python-compiler-from-ast-to-code-object) + diff --git a/Interpreter/compile3/pythonrun.png b/Interpreter/compile3/pythonrun.png new file mode 100644 index 0000000..5cf15fd Binary files /dev/null and b/Interpreter/compile3/pythonrun.png differ diff --git a/Interpreter/compile3/symtable.png b/Interpreter/compile3/symtable.png new file mode 100644 index 0000000..44a4208 Binary files /dev/null and b/Interpreter/compile3/symtable.png differ diff --git a/Interpreter/descr/descr.md b/Interpreter/descr/descr.md index 36a8ab4..8f54c39 100644 --- a/Interpreter/descr/descr.md +++ b/Interpreter/descr/descr.md @@ -30,7 +30,7 @@ Thanks @Hanaasagi for pointing out the errors [#19](https://github.com/zpoint/CP ## built in types -let's see an example first before we look into how descriptor object implements +Let's see an example first before we look into how the descriptor object is implemented ```python3 print(type(str.center)) # @@ -38,11 +38,11 @@ print(type("str".center)) # ``` -what is type **method_descriptor** ? why will `str.center` returns a **method_descriptor** object, but `"str".center` returns a **builtin_function_or_method** ? how does attribute access work in python ? +What is type **method_descriptor**? Why does `str.center` return a **method_descriptor** object, but `"str".center` returns a **builtin_function_or_method**? How does attribute access work in Python? ### instance attribute access -this is the defination of `inspect.ismethoddescriptor` and `inspect.isdatadescriptor` +This is the definition of `inspect.ismethoddescriptor` and `inspect.isdatadescriptor` ```python3 def ismethoddescriptor(object): @@ -68,7 +68,7 @@ def isdatadescriptor(object): ``` -according to the comments, we know that **data descriptor** has both `__set__` and `__get__` attribute defined while **method descriptor** has only `__get__` defined +According to the comments, we know that a **data descriptor** has both `__set__` and `__get__` attributes defined while a **method descriptor** has only `__get__` defined if we **dis** the `print(type(str.center))` @@ -86,7 +86,7 @@ if we **dis** the `print(type(str.center))` ``` -we can see that the core **opcode** is `LOAD_ATTR`, follow the `LOAD_ATTR` to the `Python/ceval.c`, we can find the definition +We can see that the core **opcode** is `LOAD_ATTR`. Following `LOAD_ATTR` to `Python/ceval.c`, we can find the definition ```c case TARGET(LOAD_ATTR): { @@ -147,7 +147,7 @@ The only difference between them is the second parameter, which is of type `PyOb From the above `PyObject_GetAttr` function, we can learn that `tp_getattro` have a higher priority than `tp_getattr` -we can see how type **str** is defined in `Objects/unicodeobject.c` +We can see how type **str** is defined in `Objects/unicodeobject.c` ```c PyTypeObject PyUnicode_Type = { @@ -174,7 +174,7 @@ PyTypeObject PyUnicode_Type = { ``` -it's using the widely used c function `PyObject_GenericGetAttr` in cpython as it's `tp_getattro`, which is defined in `Objects/object.c` +It's using the widely used C function `PyObject_GenericGetAttr` in CPython as its `tp_getattro`, which is defined in `Objects/object.c` ```c PyObject * @@ -301,11 +301,11 @@ _PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name, ``` -we can draw the process according to the code above +We can draw the process according to the code above ![_str__attribute_access](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/descr/_str__attribute_access.png) -until now, I made a mistake, the `tp_getattro` in `PyUnicode_Type` will not be called in `str.center`, otherwise, it will be called in `"str".center`, you can tell the differences in the following codes +Up until now, I made a mistake. The `tp_getattro` in `PyUnicode_Type` will not be called in `str.center`; rather, it will be called in `"str".center`. You can tell the differences in the following code ```python3 >>> type("str") @@ -324,13 +324,13 @@ until now, I made a mistake, the `tp_getattro` in `PyUnicode_Type` will not be c ``` -so, the procedure above describes the attribute access of `"str".center` +So, the procedure above describes the attribute access of `"str".center` ### class attribute access -let's find the definition of `` and how exactly `str.center` works (mostly same as `"str".center`) +Let's find the definition of `` and how exactly `str.center` works (mostly the same as `"str".center`) -for the type ``, the `LOAD_ATTR` calls the `type_getattro` +For the type ``, `LOAD_ATTR` calls `type_getattro` ```c PyTypeObject PyType_Type = { @@ -387,9 +387,9 @@ method_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) ``` -now, we have the answers of -* why will `str.center` returns a **method_descriptor** object, but `"str".center` returns a **builtin_function_or_method** ? -* how does attribute access work in python ? +Now, we have the answers to: +* Why does `str.center` return a **method_descriptor** object, but `"str".center` returns a **builtin_function_or_method**? +* How does attribute access work in Python? ## self defined types @@ -604,16 +604,16 @@ Because `type(newly_created_class)` will always return ``, and the # method_descriptor -let's find out the answer of -* what is type **method_descriptor** ? +Let's find out the answer to: +* What is type **method_descriptor**? -it's defined in `Include/descrobject.h` +It's defined in `Include/descrobject.h` ## memory layout ![PyMethodDescrObject](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/descr/PyMethodDescrObject.png) -we can see that when you try to access `"str".center` and `str.center`, they both shares the same **PyMethodDescrObject**, which is a wrapper of the **PyMethodDef** object +We can see that when you try to access `"str".center` and `str.center`, they both share the same **PyMethodDescrObject**, which is a wrapper of the **PyMethodDef** object for those who are interested in **PyMethodDef**, please refer to [method](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/method/method.md) @@ -625,7 +625,7 @@ descr->d_common->d_type: 0x10fdefc10, descr->d_common->d_type.tp_name: str, repr ``` -there exists various descriptor type +There exist various descriptor types **PyMemberDescrObject**: wrapper of **PyMemberDef** @@ -633,7 +633,7 @@ there exists various descriptor type # how to change the behavior of attribute access -we know that when you try to access the attribute of an object, the python virtual machine will +We know that when you try to access the attribute of an object, the Python virtual machine will 1. execute the opcode `LOAD_ATTR` 2. `LOAD_ATTR` will try to call `tp_getattro` function of the object, if success go to 4 @@ -642,13 +642,13 @@ we know that when you try to access the attribute of an object, the python virtu the default `tp_getattro` will be installed in the creation process of the newly created type, different behaviour depends on what methods are override by user, generally, the default `tp_getattro` is `PyObject_GenericGetAttr` which implements the **descriptor protocol**(we learned above from the source code) -when we define a python object, if we need to change the behavior of attribute access +When we define a Python object, if we need to change the behavior of attribute access: -we are not able to change the behavior of opcode `LOAD_ATTR`, it's written in C +We are not able to change the behavior of opcode `LOAD_ATTR`; it's written in C. -instead, we can provide our own `__getattribute__` and `__getattr__` to change the function installed in the `tp_getattro` slot of the newly created type +Instead, we can provide our own `__getattribute__` and `__getattr__` to change the function installed in the `tp_getattro` slot of the newly created type. -notice, provide your own `__getattribute__` may violate the **descriptor protocol**, I will not recommend you to do that(usually we only need to define our own `__getattr__`) +Note: Providing your own `__getattribute__` may violate the **descriptor protocol**. I would not recommend doing that (usually we only need to define our own `__getattr__`) ```python3 class A(object): diff --git a/Interpreter/exception/exception.md b/Interpreter/exception/exception.md index 59468dc..0a31523 100644 --- a/Interpreter/exception/exception.md +++ b/Interpreter/exception/exception.md @@ -15,17 +15,17 @@ # memory layout -there are various exception types defined in `Include/cpython/pyerrors.h`, the most widely used **PyBaseExceptionObject**(also the base part of any other exception type) +There are various exception types defined in `Include/cpython/pyerrors.h`. The most widely used is **PyBaseExceptionObject** (also the base part of any other exception type) ![base_exception](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/base_exception.png) -all other basic exception types defined in same C file are shown +All other basic exception types defined in the same C file are shown ![error_layout1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/error_layout1.png) ![error_layout2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/error_layout2.png) -what's following the definition of the basic exception types above are the definition of derived exception types according to exception hierarchy, because there're dozens of derived exceptions need to be defined, some **marco** are used for shortening the codes +Following the definition of the basic exception types above are the definitions of derived exception types according to the exception hierarchy. Because there are dozens of derived exceptions that need to be defined, some **macros** are used for shortening the code ```c #define SimpleExtendsException(EXCBASE, EXCNAME, EXCDOC) \ @@ -141,7 +141,7 @@ BaseException # exception handling -let's define an example +Let's define an example ```python3 import sys @@ -182,7 +182,7 @@ def t2(): # position 1 ``` -and try to compile it +And try to compile it ```python3 python.exe -m dis .\test.py @@ -406,11 +406,11 @@ Disassembly of : ``` -we can see that the opcode `0 SETUP_FINALLY 178 (to 202)` maps to the outermost `finally` statement, 0 is the byte offset of opcode, 178 is the parameter of the `SETUP_FINALLY`, the real handler offset is calculated as `INSTR_OFFSET() + oparg`(`INSTR_OFFSET` is the byte offset of the first opcode to the next opcode(here is 24), `oparg` is the parameter 178), which adds up to 202 in result +We can see that the opcode `0 SETUP_FINALLY 178 (to 202)` maps to the outermost `finally` statement. 0 is the byte offset of the opcode, 178 is the parameter of `SETUP_FINALLY`. The real handler offset is calculated as `INSTR_OFFSET() + oparg` (`INSTR_OFFSET` is the byte offset of the first opcode to the next opcode (here it's 24), `oparg` is the parameter 178), which adds up to 202 `2 SETUP_FINALLY 26 (to 52)` maps to the first `except` statement, the byte offset of opcode is 26(`LOAD_GLOBAL`) and the parameter of the `SETUP_FINALLY` is also 26, 26(parameter) + 26(opcode offset) is 52 -what does `SETUP_FINALLY` do ? +What does `SETUP_FINALLY` do? ```c /* cpython/Python/ceval.c */ @@ -439,21 +439,21 @@ void PyFrame_BlockSetup(PyFrameObject *f, int type, int handler, int level) ``` -for those who need detail of frame object, please refer to [frame object](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame.md) +For those who need details about the frame object, please refer to [frame object](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame.md). -the `PyFrame_BlockSetup` will be called in the following opcode `SETUP_FINALLY`, `SETUP_ASYNC_WITH`, and `SETUP_WITH`(parameter `int type` equals `SETUP_FINALLY` in these opcode) +`PyFrame_BlockSetup` will be called in the following opcodes: `SETUP_FINALLY`, `SETUP_ASYNC_WITH`, and `SETUP_WITH` (parameter `int type` equals `SETUP_FINALLY` in these opcodes) `RAISE_VARARGS`, `END_FINALLY` and `END_ASYNC_FOR` will call `PyFrame_BlockSetup` in some cases -what will happen if we call the function `t2()` ? +What will happen if we call the function `t2()`? -this is the definition of `PyTryBlock` +This is the definition of `PyTryBlock` ![try_block](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/try_block.png) in `position 1` -the value `CO_MAXBLOCKS` is 20, you can't set up more than 20 blocks inside a frame(`try/finally/with/async with`) +The value `CO_MAXBLOCKS` is 20. You can't set up more than 20 blocks inside a frame (`try/finally/with/async with`) ![pos1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/pos1.png) @@ -520,11 +520,11 @@ exception_unwind: bacause `66 SETUP_FINALLY 116 (to 184)` exists before the `print("position 5\n\n")` statement, the third try block will exist -the block is created by the compiler, the opcode `184 LOAD_CONST 0 (None)` is between `position 8` and `position 9`, it does nothing except for load variable `e1` and deallocates `e1` +The block is created by the compiler. The opcode `184 LOAD_CONST 0 (None)` is between `position 8` and `position 9`. It does nothing except load variable `e1` and deallocate `e1` ![pos5](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/pos5.png) -in `position 6`, the following two opcodes create two more try block +in `position 6`, The following two opcodes create two more try blocks ```python3 12 82 SETUP_FINALLY 80 (to 164) diff --git a/Interpreter/frame/example3.png b/Interpreter/frame/example3.png index 52aa182..98a4920 100644 Binary files a/Interpreter/frame/example3.png and b/Interpreter/frame/example3.png differ diff --git a/Interpreter/frame/frame.md b/Interpreter/frame/frame.md index 5097cf8..0483629 100644 --- a/Interpreter/frame/frame.md +++ b/Interpreter/frame/frame.md @@ -18,21 +18,21 @@ # memory layout -the **PyFrameObject** is the stack frame in python virtual machine, it contains space for the currently executing code object, parameters, variables in different scope, try block info and etc +The **PyFrameObject** is the stack frame in the Python virtual machine. It contains space for the currently executing code object, parameters, variables in different scopes, try block info, and more -for more information please refer to [stack frame strategy](http://en.citizendium.org/wiki/Memory_management) +for more information please refer to [stack frame strategy](https://en.citizendium.org/wiki/Stack_frame) -![layout](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/layout.png) +![layout](./layout.png) # example -every time you make a function call, a new **PyFrameObject** will be created and attached to the current function call +Every time you make a function call, a new **PyFrameObject** will be created and attached to the current function call. -it's not intuitive to trace a frame object in the middle of a function, I will use a [generator object](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/gen.md) to do the explanation +It's not intuitive to trace a frame object in the middle of a function. I will use a [generator object](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/gen.md) to do the explanation. -you can always get the frame of the current environment by executing `sys._current_frames()` +You can always get the frame of the current environment by executing `sys._current_frames()`. -if you need the meaning of each field, please refer to [Junnplus' blog](https://github.com/Junnplus/blog/issues/22) or read source code directly +If you need the meaning of each field, please refer to [Junnplus' blog](https://github.com/Junnplus/blog/issues/22) or read the source code directly ## f_valuestack/f_stacktop/f_localsplus @@ -65,7 +65,7 @@ the **ob_size** is the sum of code->co_stacksize, code->co_nlocals, code->co_cel **code->co_cellvars**: a tuple containing the names of all variables in the function that are also used in a nested function -**code->nfrees**: the names of all variables used in the function that is defined in an enclosing function scope +**code->co_freevars**: the names of all variables used in the function that is defined in an enclosing function scope for more information about **PyCodeObject** please refer to [What is a code object in Python?](https://www.quora.com/What-is-a-code-object-in-Python) and [code object](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/code/code.md) @@ -81,7 +81,7 @@ def g2(a, b=1, c=2): ``` -the **dis** result +The **dis** result ```python3 # ./python.exe -m dis frame_dis.py @@ -124,7 +124,7 @@ Disassembly of : ``` -let's iter through the generator +Let's iterate through the generator ```python3 >>> gg = g2("param a") @@ -133,13 +133,13 @@ let's iter through the generator ![example0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example0.png) -after the first **next** returns, the first **opcode** `0 LOAD_FAST 0 (a)` will be executed and the current execution flow is in the middle of the second **opcode** `2 YIELD_VALUE` +After the first **next** returns, the first **opcode** `0 LOAD_FAST 0 (a)` will be executed and the current execution flow is in the middle of the second **opcode** `2 YIELD_VALUE`. -the field **f_lasti** is 2, indicate that the virtual program counter is in `2 YIELD_VALUE` +The field **f_lasti** is 2, indicating that the virtual program counter is at `2 YIELD_VALUE`. -the **opcode** `LOAD_FAST` will push the paramter to the **f_valuestack**, and **opcode** `YIELD_VALUE` will **pop** the top element in the **f_valuestack**, the defination of **pop** is `#define BASIC_POP() (*--stack_pointer)` +The **opcode** `LOAD_FAST` will push the parameter to **f_valuestack**, and **opcode** `YIELD_VALUE` will **pop** the top element from **f_valuestack**. The definition of **pop** is `#define BASIC_POP() (*--stack_pointer)`. -the value(address 0x100a5b538) in **f_valuestack** is the same as the previous step(previous picture), but the first element the address(0x100a5b538) pointed to is different, currently it's a pointer to a PyUnicodeObject('param a') or an invalid address(if the PyUnicodeObject is deallocated)) +The value (address 0x100a5b538) in **f_valuestack** is the same as the previous step (previous picture), but the first element the address (0x100a5b538) points to is different. Currently, it's a pointer to a PyUnicodeObject('param a') or an invalid address (if the PyUnicodeObject is deallocated) ```python3 >>> next(gg) @@ -147,7 +147,7 @@ the value(address 0x100a5b538) in **f_valuestack** is the same as the previous s ``` -![example1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1.png) +![example1](./example1.png) ```python3 >>> next(gg) @@ -156,43 +156,43 @@ the value(address 0x100a5b538) in **f_valuestack** is the same as the previous s ``` -the opcode `6 LOAD_GLOBAL 0 (str)` `8 LOAD_FAST 1 (b)` and `10 LOAD_FAST 2 (c)` in line 3 pushes **str**(parameter str is stored in the frame-f_code->co_names field), **b**(int 1) and **c**(int 2) to **f_valuestack**, opcode `12 BINARY_ADD` pops off the top 2 elements in **f_valuestack**(**b** and **c**), sum these two values, store to the top of the **f_valuestack**, this is what the **f_valuestack** looks like after `12 BINARY_ADD` +The opcodes `6 LOAD_GLOBAL 0 (str)`, `8 LOAD_FAST 1 (b)`, and `10 LOAD_FAST 2 (c)` in line 3 push **str** (parameter str is stored in the frame->f_code->co_names field), **b** (int 1), and **c** (int 2) to **f_valuestack**. Opcode `12 BINARY_ADD` pops off the top 2 elements in **f_valuestack** (**b** and **c**), sums these two values, and stores the result at the top of **f_valuestack**. This is what **f_valuestack** looks like after `12 BINARY_ADD` -![example1_2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_2.png) +![example1_2](./example1_2.png) -the opcode `14 CALL_FUNCTION 1` will pop the function and argument off the stack and delegate the actual function call +The opcode `14 CALL_FUNCTION 1` will pop the function and argument off the stack and delegate the actual function call. -after the function call, result `'3'` is pushed onto the stack +After the function call, the result `'3'` is pushed onto the stack -![example1_2_1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_2_1.png) +![example1_2_1](./example1_2_1.png) -opcode `16 STORE_FAST 2 (c)` pops off the top element in the **f_valuestack** and stores it into the 2th position of the **f_localsplus** +Opcode `16 STORE_FAST 2 (c)` pops off the top element in **f_valuestack** and stores it into the 2nd position of **f_localsplus** -![example1_2_2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_2_2.png) +![example1_2_2](./example1_2_2.png) -opcode `18 LOAD_FAST 2 (c)` push the 2th element in the **f_localsplus** onto the **f_valuestack**, and `20 YIELD_VALUE ` pops it and send it to the caller +Opcode `18 LOAD_FAST 2 (c)` pushes the 2nd element in **f_localsplus** onto **f_valuestack**, and `20 YIELD_VALUE` pops it and sends it to the caller. -field **f_lasti** is 20, indicate that it's currently executing the opcode `20 YIELD_VALUE` +Field **f_lasti** is 20, indicating that it's currently executing the opcode `20 YIELD_VALUE` -![example2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example2.png) +![example2](./example2.png) after `24 LOAD_GLOBAL 1 (range)` and `26 LOAD_CONST 1 (3)` -![example1_3_1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_3_1.png) +![example1_3_1](./example1_3_1.png) after `28 CALL_FUNCTION 1` -![example1_3_2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_3_2.png) +![example1_3_2](./example1_3_2.png) after `30 STORE_FAST 3 (new_g)` -![example1_3_3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_3_3.png) +![example1_3_3](./example1_3_3.png) after `32 LOAD_FAST 3 (new_g)` -![example1_3_4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_3_4.png) +![example1_3_4](./example1_3_4.png) -the opcode `34 GET_YIELD_FROM_ITER` makes sure the stack's top is an iterable object +The opcode `34 GET_YIELD_FROM_ITER` makes sure the stack's top is an iterable object `36 LOAD_CONST 0 (None)` pushes `None` onto the stack @@ -202,11 +202,13 @@ the opcode `34 GET_YIELD_FROM_ITER` makes sure the stack's top is an iterable ob ``` -field **f_lasti** is 36, indicate that it's before the `38 YIELD_FROM` +Field **f_lasti** is 36, indicating that it's after `38 YIELD_FROM`. -![example3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example3.png) +At the end of `YIELD_FROM`, the following code `f->f_lasti -= sizeof(_Py_CODEUNIT);` resets `f_lasti` to the beginning of `YIELD_FROM` [Thanks to @RyanHe123](https://github.com/zpoint/CPython-Internals/issues/46) -the frame object deallocated after the **StopIteration** raised (the opcode `44 RETURN_VALUE` also executed) +![example3](./example3.png) + +The frame object is deallocated after **StopIteration** is raised (the opcode `44 RETURN_VALUE` is also executed) ```python3 >>> next(gg) @@ -224,9 +226,9 @@ StopIteration ## f_blockstack -f_blockstack is an array, element type is **PyTryBlock**, size is **CO_MAXBLOCKS**(20) +f_blockstack is an array. The element type is **PyTryBlock** and the size is **CO_MAXBLOCKS** (20). -the definition of **PyTryBlock** +The definition of **PyTryBlock** ```c typedef struct { @@ -237,7 +239,7 @@ typedef struct { ``` -let's define a generator with some blocks +Let's define a generator with some blocks ```python3 def g3(): @@ -261,11 +263,11 @@ def g3(): ``` -![blockstack0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack0.png) +![blockstack0](./blockstack0.png) -in the first **yield** statement, the first **try block** is set up +In the first **yield** statement, the first **try block** is set up. -**f_iblock** is 1, indicate that there's currently one block +**f_iblock** is 1, indicating that there's currently one block **b_type** 122 is the opcode `SETUP_FINALLY`, **b_handler** 20 is the opcode location of the `except ZeroDivisionError`, **b_level** 0 is the stack pointer's position to use @@ -275,7 +277,7 @@ in the first **yield** statement, the first **try block** is set up ``` -![blockstack1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack1.png) +![blockstack1](./blockstack1.png) **b_type** 257 is the opcode `EXCEPT_HANDLER`, `EXCEPT_HANDLER` has a special meaning @@ -298,7 +300,7 @@ in the first **yield** statement, the first **try block** is set up ``` -![blockstack2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack2.png) +![blockstack2](./blockstack2.png) **f_iblock** is 3, the second try block comes from `finally:`(opcode position 116), and the third try block comes from `except ModuleNotFoundError:`(opcode position 62) @@ -308,7 +310,7 @@ in the first **yield** statement, the first **try block** is set up ``` -![blockstack3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack3.png) +![blockstack3](./blockstack3.png) ```python3 >>> next(gg) @@ -318,9 +320,9 @@ in the first **yield** statement, the first **try block** is set up **b_type** of the third try block becomes 257 and **b_handler** becomes -1, means this block is currently being handling -![blockstack4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack4.png) +![blockstack4](./blockstack4.png) -the other two try block is handled properly +The other two try blocks are handled properly ```python3 >>> next(gg) @@ -334,9 +336,9 @@ the other two try block is handled properly ``` -![blockstack5](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack5.png) +![blockstack5](./blockstack5.png) -frame object deallocated +Frame object is deallocated ```python3 >>> next(gg) @@ -348,7 +350,7 @@ StopIteration ## f_back -**f_back** is a pointer which points to the previous frame, it makes the related frames a single linked list +**f_back** is a pointer that points to the previous frame. It makes the related frames a singly linked list ```python3 import inspect @@ -364,7 +366,7 @@ g4(3) ``` -output +Output ```python3 depth 3 @@ -378,15 +380,15 @@ depth 0 ``` -![f_back](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/f_back.png) +![f_back](./f_back.png) # free_list mechanism ## zombie frame -the first time a code object attached to a frame object, after the execution of the code block, the frame object will not be freed, it becomes a "zombie" frame, next time the code block executes again, it will reuse the same frame object +The first time a code object is attached to a frame object, after the execution of the code block, the frame object will not be freed. It becomes a "zombie" frame. The next time the code block executes again, it will reuse the same frame object. -the strategy saves malloc/realloc overhead and some field initialization +This strategy saves malloc/realloc overhead and some field initialization ```python3 def g5(): @@ -410,7 +412,7 @@ StopIteration ## free_list sub -there's a single linked list store the deallocated frame object, it saves malloc/free overhead +There's a singly linked list that stores the deallocated frame objects. It saves malloc/free overhead ```c static PyFrameObject *free_list = NULL; @@ -420,7 +422,7 @@ static int numfree = 0; /* number of frames currently in free_list */ ``` -When a **PyFrameObject** is on the free list, only the following members have a meaning +When a **PyFrameObject** is on the free list, only the following members have meaning ```python3 ob_type == &Frametype @@ -430,7 +432,7 @@ ob_size size of localsplus ``` -the creating process will check if the stack size is enough +The creation process will check if the stack size is enough ```c if (Py_SIZE(f) < extras) { @@ -438,7 +440,7 @@ if (Py_SIZE(f) < extras) { ``` -let's see an example +Let's see an example ```python3 import inspect @@ -452,7 +454,7 @@ def g6(): ``` -![free_list0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/free_list0.png) +![free_list0](./free_list0.png) the frame attached to variable **gg** is deallocated, because it's the first frame execute the code block, it becomes the "zombie" frame of the **code** object @@ -468,7 +470,7 @@ StopIteration ``` -![free_list1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/free_list1.png) +![free_list1](./free_list1.png) ```python3 >>> next(gg1) @@ -480,7 +482,7 @@ StopIteration ``` -![free_list2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/free_list2.png) +![free_list2](./free_list2.png) ```python3 >>> next(gg2) @@ -492,4 +494,4 @@ StopIteration ``` -![free_list3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/free_list3.png) +![free_list3](./free_list3.png) diff --git a/Interpreter/frame/frame_cn.md b/Interpreter/frame/frame_cn.md index ecd1660..b5980d3 100644 --- a/Interpreter/frame/frame_cn.md +++ b/Interpreter/frame/frame_cn.md @@ -22,7 +22,7 @@ 需要更多有关栈帧的信息请参考 [stack frame strategy](http://en.citizendium.org/wiki/Memory_management) -![layout](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/layout.png) +![layout](./layout.png) # 示例 @@ -65,7 +65,7 @@ for (i=0; ico_cellvars**: 一个元组, 包含了自己用到并且内嵌函数也用到的所有的变量名称 -**code->nfrees**: 自己是内嵌函数的情况下, 包含了外部函数和自己同时用到的变量名称 +**code->co_freevars**: 自己是内嵌函数的情况下, 包含了外部函数和自己同时用到的变量名称 更多关于 **PyCodeObject** 的信息请参考 [What is a code object in Python?](https://www.quora.com/What-is-a-code-object-in-Python) 和 [code 对象](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/code/code_cn.md) @@ -131,7 +131,7 @@ Disassembly of : ``` -![example0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example0.png) +![example0](./example0.png) 第一次 **next** 返回时, **opcode** `0 LOAD_FAST 0 (a)` 已经被执行了, 并且当前的执行位置是在 `2 YIELD_VALUE` 中 @@ -147,7 +147,7 @@ Disassembly of : ``` -![example1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1.png) +![example1](./example1.png) ```python3 >>> next(gg) @@ -158,39 +158,39 @@ Disassembly of : 在第三行代码的 opcode `6 LOAD_GLOBAL 0 (str)` `8 LOAD_FAST 1 (b)` 和 `10 LOAD_FAST 2 (c)` 分别把 **str**(str 存储在了 frame-f_code->co_names 这个字段中), **b**(int 1) 和 **c**(int 2) 推入 **f_valuestack**, opcode `12 BINARY_ADD` 弹出 **f_valuestack**(**b** and **c**) 顶部的两个元素, 相加之后存储回 **f_valuestack** 的顶部, 下图的 **f_valuestack** 为 `12 BINARY_ADD` 执行之后的样子 -![example1_2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_2.png) +![example1_2](./example1_2.png) opcode `14 CALL_FUNCTION 1` 会弹出可执行对象和可执行对象对应的参数, 并用这些参数传递给可执行对象, 之后执行 执行完成之后, 执行结果 `'3'` 会被压回堆中 -![example1_2_1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_2_1.png) +![example1_2_1](./example1_2_1.png) opcode `16 STORE_FAST 2 (c)` 弹出 **f_valuestack** 顶部的元素, 并把它存储到了 **f_localsplus** 下标为 2 的位置中 -![example1_2_2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_2_2.png) +![example1_2_2](./example1_2_2.png) opcode `18 LOAD_FAST 2 (c)` 把 **f_localsplus** 位置下标为 2 的元素推入 **f_valuestack**, 之后 `20 YIELD_VALUE ` 弹出这个元素并把它传递给调用者 字段 **f_lasti** 位置的值为 20, 表明当前正在 opcode `20 YIELD_VALUE` 的位置 -![example2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example2.png) +![example2](./example2.png) 在 `24 LOAD_GLOBAL 1 (range)` 和 `26 LOAD_CONST 1 (3)` 之后 -![example1_3_1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_3_1.png) +![example1_3_1](./example1_3_1.png) 在 `28 CALL_FUNCTION 1` 之后 -![example1_3_2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_3_2.png) +![example1_3_2](./example1_3_2.png) 在 `30 STORE_FAST 3 (new_g)` 之后 -![example1_3_3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_3_3.png) +![example1_3_3](./example1_3_3.png) 在 `32 LOAD_FAST 3 (new_g)` 之后 -![example1_3_4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example1_3_4.png) +![example1_3_4](./example1_3_4.png) opcode `34 GET_YIELD_FROM_ITER` 作用是保证堆顶的元素是一个可迭代对象 @@ -202,9 +202,11 @@ opcode `18 LOAD_FAST 2 (c)` 把 **f_localsplus** 位置下标为 ``` -字段 **f_lasti** 现在值是 36, 表明他在 `38 YIELD_FROM` 之前 +字段 **f_lasti** 现在值是 36, 表明他在 `38 YIELD_FROM` 之后 -![example3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/example3.png) +在 `YIELD_FROM` 的末尾这一行代码 `f->f_lasti -= sizeof(_Py_CODEUNIT);` 会把 `f_lasti` 重置为刚进入 `YIELD_FROM` 的位置 [感谢 @RyanHe123 指正](https://github.com/zpoint/CPython-Internals/issues/46) + +![example3](./example3.png) 栈帧对象在 **StopIteration** 抛出后就进入了释放阶段(opcode `44 RETURN_VALUE` 执行之后) @@ -261,7 +263,7 @@ def g3(): ``` -![blockstack0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack0.png) +![blockstack0](./blockstack0.png) 在第一个 **yield** 声明时, 第一个 **try block** 已经设置好了 @@ -275,7 +277,7 @@ def g3(): ``` -![blockstack1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack1.png) +![blockstack1](./blockstack1.png) **b_type** 257 是 opcode `EXCEPT_HANDLER` 的值, opcode `EXCEPT_HANDLER` 有特殊的含义 @@ -301,7 +303,7 @@ def g3(): ``` -![blockstack2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack2.png) +![blockstack2](./blockstack2.png) **f_iblock** 值为 3, 第二个 try block 来自 `finally:`(opcode 位置 116), 第三个来自 `except ModuleNotFoundError:`(opcode 位置 62) @@ -311,7 +313,7 @@ def g3(): ``` -![blockstack3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack3.png) +![blockstack3](./blockstack3.png) ```python3 >>> next(gg) @@ -321,7 +323,7 @@ def g3(): 第三个 try block 的 **b_type** 变为了 257 并且 **b_handler** 变为 -1, 表明当前的 block 正在处理中 -![blockstack4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack4.png) +![blockstack4](./blockstack4.png) 另外两个 try block 也正确的处理完了 @@ -337,7 +339,7 @@ def g3(): ``` -![blockstack5](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/blockstack5.png) +![blockstack5](./blockstack5.png) frame 对象进入释放阶段 @@ -381,7 +383,7 @@ depth 0 ``` -![f_back](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/f_back.png) +![f_back](./f_back.png) # free_list 机制 @@ -455,7 +457,7 @@ def g6(): ``` -![free_list0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/free_list0.png) +![free_list0](./free_list0.png) 和 **gg** 对象关联的 frame 进入了回收阶段, 因为当前的 **code** 对象 "zombie" frame 字段为空, 所以这个 frame 成为了 **code** 对象的 "zombie" frame @@ -471,7 +473,7 @@ StopIteration ``` -![free_list1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/free_list1.png) +![free_list1](./free_list1.png) ```python3 >>> next(gg1) @@ -483,7 +485,7 @@ StopIteration ``` -![free_list2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/free_list2.png) +![free_list2](./free_list2.png) ```python3 >>> next(gg2) @@ -495,4 +497,4 @@ StopIteration ``` -![free_list3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/free_list3.png) +![free_list3](./free_list3.png) diff --git a/Interpreter/gc/gc.md b/Interpreter/gc/gc.md index 7c4bcfb..8da40dd 100644 --- a/Interpreter/gc/gc.md +++ b/Interpreter/gc/gc.md @@ -31,18 +31,18 @@ # introduction -the garbage collection in CPython consists of two components +The garbage collection in CPython consists of two components * **reference counting** (mostly defined in `Include/object.h`) * **generational garbage collection** (mostly defined in `Modules/gcmodule.c`) ## reference counting -as [wikipedia](https://en.wikipedia.org/wiki/Reference_counting) says +As [Wikipedia](https://en.wikipedia.org/wiki/Reference_counting) says > In computer science, reference counting is a technique of storing the number of references, pointers, or handles to a resource such as an object, block of memory, disk space or other resource. It may also refer, more specifically, to a garbage collection algorithm that uses these reference counts to deallocate objects which are no longer referenced -in python +In Python ```python3 s = "hello world" @@ -68,7 +68,7 @@ s = "hello world" ![refcount2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/refcount2.png) -let's compile a script +Let's compile a script ```python3 s = [] @@ -77,7 +77,7 @@ del s ``` -the output is +The output is ```python3 1 0 BUILD_LIST 0 @@ -92,9 +92,9 @@ the output is ``` -in the first line `s = []` +In the first line `s = []` -`0 BUILD_LIST 0` simply create a new list, initialize it's reference count to 1, and push it onto the stack +`0 BUILD_LIST 0` simply creates a new list, initializes its reference count to 1, and pushes it onto the stack `2 STORE_NAME 0` @@ -132,13 +132,13 @@ case TARGET(STORE_NAME): { ``` -before execute line 2 `s2 = s`, reference count of the newly created list object is 1 +Before executing line 2 `s2 = s`, the reference count of the newly created list object is 1 -`4 LOAD_NAME 0` gets the value stores in local namespace with key 's' ---- the newly created list object, increment it's reference count, and push it onto the stack +`4 LOAD_NAME 0` gets the value stored in local namespace with key 's' ---- the newly created list object, increments its reference count, and pushes it onto the stack -until now, the reference count of the newly created list object is 2(one from 's', one from the stack) +Until now, the reference count of the newly created list object is 2 (one from 's', one from the stack) -`6 STORE_NAME 1` executes the above code again, the difference is the name now becomes 's2', after this opcode, reference count of the list becomes 2 +`6 STORE_NAME 1` executes the above code again. The difference is the name now becomes 's2'. After this opcode, the reference count of the list becomes 2 `8 DELETE_NAME 0 (s)` @@ -174,7 +174,7 @@ case TARGET(DELETE_NAME): { ### when will it be triggered -if the reference count becomes 0, the deallocate procedure will be triggered imeediately +If the reference count becomes 0, the deallocation procedure will be triggered immediately ```c /* cpython/Include/object.h */ @@ -202,7 +202,7 @@ static inline void _Py_DECREF(const char *filename, int lineno, ### reference cycle problem -there're some situations that the reference counting can't handle +There are some situations that reference counting can't handle ### example1 @@ -227,7 +227,7 @@ reference count of **a1** and **a2** are both 2 ``` -now, the reference from local namespace is deleted, and they both have a reference to each other, there's no way for the reference count of **a1**/**a2** to become 0 +Now, the reference from the local namespace is deleted, and they both have a reference to each other. There's no way for the reference count of **a1**/**a2** to become 0 ![ref_cycle2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/ref_each2.png) @@ -248,13 +248,13 @@ now, the reference from local namespace is deleted, and they both have a referen ``` -the reference from local namespace is deleted, the reference count of **a** will stays 1 and never become 0 +The reference from the local namespace is deleted. The reference count of **a** will stay 1 and never become 0 ![ref_cycle1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/ref_cycle2.png) ## generational garbage collection -If there's only one mechanism **reference counting** working, the interpreter will risk the memory leak issue due to the reference cycle in the above examples +If only one mechanism (**reference counting**) were working, the interpreter would risk memory leak issues due to the reference cycles in the above examples **generational garbage collection** is the other component used for detecting and garbage collecting the unreachable object @@ -274,7 +274,7 @@ class A: ### track -there must be a way to keep track of all the heap allocated **PyObject** +There must be a way to keep track of all the heap-allocated **PyObject**s **generation0** is a pointer, to the head of the linked list, each element in the linked list consists of two parts, the first part is **PyGC_Head**, the second part is **PyObject** @@ -284,29 +284,29 @@ there must be a way to keep track of all the heap allocated **PyObject** ![track](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/track.png) -when you create a python object such as `a = list()`, object **a** of type **list** will be appended to the end of the linked list in **generation0**, so **generation0** is able to track all the newly created python object +When you create a Python object such as `a = list()`, object **a** of type **list** will be appended to the end of the linked list in **generation0**, so **generation0** is able to track all the newly created Python objects -actually, the linked list is linked via **_gc_next**, **_gc_prev** is not a valid pointer, it's used for storing bit flag and other information +Actually, the linked list is linked via **_gc_next**. **_gc_prev** is not a valid pointer; it's used for storing bit flags and other information ### generational -if all newly created objects are appended to the tail of one linked list, somehow it will be a very huge linked list +If all newly created objects are appended to the tail of one linked list, it will eventually become a very huge linked list. -for some program designed for running for a long time, there must exist some long lived objects, repeating garbage collecting these objects will waste lots of CPU time +For programs designed for running for a long time, there must exist some long-lived objects. Repeatedly garbage collecting these objects will waste lots of CPU time. -generations is used in the purpose of doing less collections +Generations are used for the purpose of doing fewer collections. -CPython used three generations totally, the newly created objects is stored in the first generation, when an object survive a round of gc, it will be moved to next generation +CPython uses three generations in total. Newly created objects are stored in the first generation. When an object survives a round of GC, it will be moved to the next generation. -lower generation will be collected more frequently, higher generation will be collected less frequently +Lower generations will be collected more frequently; higher generations will be collected less frequently. -when a generation is being collected, all the generations lower than the current will be merged together before collection +When a generation is being collected, all the generations lower than the current one will be merged together before collection ![generation](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/generation.png) ### update_refs -let's run a procedure of garbage collection +Let's run a procedure of garbage collection ```python3 # assume a1, a2, b survived in generation0, and move to generation1 @@ -326,13 +326,13 @@ let's run a procedure of garbage collection ![update_ref1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/update_ref1.png) -first step is to merge every generation lower than the current generation, mark this merged generation as **young**, mark the generation next to the current generation as **old** +The first step is to merge every generation lower than the current generation, mark this merged generation as **young**, and mark the generation next to the current generation as **old** ![young_old](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/young_old.png) -function **update_refs** will copy object's reference count, stores in the **_gc_prev** field of the object, the right most 2 bit of **_gc_prev** is reserved of some marking +Function **update_refs** will copy the object's reference count and store it in the **_gc_prev** field of the object. The rightmost 2 bits of **_gc_prev** are reserved for some marking. -so, the copy of the reference count will left shift 2 before stores in the **_gc_prev** +So, the copy of the reference count will be left-shifted by 2 before being stored in **_gc_prev** `1 / 0x06` means the copy of the reference count is `1`, but the value actually stores in **_gc_prev** is `0x06` @@ -340,31 +340,31 @@ so, the copy of the reference count will left shift 2 before stores in the **_gc ### subtract_refs -function **subtract_refs** will traverse the **young** generation, iterate through every object in the linked list +Function **subtract_refs** will traverse the **young** generation, iterating through every object in the linked list. -for every object in the generation, check if every other object accessible from the current object is collectable and in the same generation, if so, decrement the copied reference count in **_gc_prev** (calls **tp_traverse** with the function **visit_decref**) +For every object in the generation, check if every other object accessible from the current object is collectable and in the same generation. If so, decrement the copied reference count in **_gc_prev** (calls **tp_traverse** with the function **visit_decref**). -this step aims to decrement reference count referenced by those objects in same generation +This step aims to decrement reference counts referenced by those objects in the same generation ![subtract_refs](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/subtract_refs.png) ### move_unreachable -**move_unreachable** will create a linked list named **unreachable**, traverse the current **generation** named **young**, move those objects with the value of the copy of the reference count <= 0 to **unreachable** +**move_unreachable** will create a linked list named **unreachable**, traverse the current **generation** named **young**, and move those objects with the value of the copy of the reference count <= 0 to **unreachable**. -if the current object has the value of the copy of the reference count > 0, for every other object in same generation reachable from the current object, if the accessible object has a copy of the reference count <= 0, reset the reference count to 1, and move that object to the tail of **generation** if it's currentlt in **unreachable** +If the current object has the value of the copy of the reference count > 0, for every other object in the same generation reachable from the current object, if the accessible object has a copy of the reference count <= 0, reset the reference count to 1 and move that object to the tail of **generation** if it's currently in **unreachable**. -let's see an example +Let's see an example. -first object in **generation** is the local **namespace**, because **namespace** object is reachable, all objects reachable from the local **namespace** is also reachable, so the **_gc_prev** of `c` and `d2` are all set to `0x06` +The first object in **generation** is the local **namespace**. Because the **namespace** object is reachable, all objects reachable from the local **namespace** are also reachable, so the **_gc_prev** of `c` and `d2` are all set to `0x06` ![move_unreachable1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable1.png) -the second object is `a1`, since `a1`'s copy of the reference count is <= 0, it's moved to **unreachable** +The second object is `a1`. Since `a1`'s copy of the reference count is <= 0, it's moved to **unreachable** ![move_unreachable2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable2.png) -so does `a2` +So does `a2` ![move_unreachable3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable3.png) @@ -372,17 +372,17 @@ so does `a2` ![move_unreachable4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable4.png) -now, it comes to `c` +Now, it comes to `c`. -because the copy of the reference count of `c`'s value is > 0, it will stay in the **generation** +Because the copy of the reference count of `c`'s value is > 0, it will stay in the **generation** ![move_unreachable5](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable5.png) -because `d1`'s copy of the reference count is <= 0, it's moved to **unreachable** +Because `d1`'s copy of the reference count is <= 0, it's moved to **unreachable** ![move_unreachable6](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable6.png) -because `d2`'s copy of the reference count is > 0, and `d1` is reachable from `d2` +Because `d2`'s copy of the reference count is > 0, and `d1` is reachable from `d2` ![move_unreachable7](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable7.png) @@ -390,21 +390,21 @@ because `d2`'s copy of the reference count is > 0, and `d1` is reachable from `d ![move_unreachable8](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable8.png) -when it comes to `d1`, copy of the reference count is > 0 +When it comes to `d1`, the copy of the reference count is > 0. -and we reach the end of the **generation**, so every object stays in **unreachable** can be garbage collected +We have reached the end of the **generation**, so every object that stays in **unreachable** can be garbage collected. -all objects survive this round of garbage collections will be moved to the elder **generation** +All objects that survive this round of garbage collection will be moved to the older **generation** ![move_unreachable9](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/move_unreachable9.png) ### finalizer -what if the object needed to be garbage collected has defined it's own finalizer ? +What if an unusual object (which contains a cyclic reference) that needs to be garbage collected has defined its own finalizer? -before python3.4, those objects won't be collected even if they are moved to **unreachable** +Before Python 3.4, those objects wouldn't be collected. They would be moved to `gc.garbage` and you would need to call `__del__` and collect them manually. -after python3.4, [pep-0442](https://legacy.python.org/dev/peps/pep-0442/) solves the problem +After Python 3.4, [PEP-0442](https://legacy.python.org/dev/peps/pep-0442/) solves the problem ```python3 class A: @@ -433,47 +433,47 @@ gc.collect() ``` -this is the layout after **move_unreachable** +This is the layout after **move_unreachable** ![finalize1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/finalize1.png) -step1, all objects defined it's own finalizer, the `__del__` methods will be called +Step 1: For all objects that have defined their own finalizer, the `__del__` methods will be called after `__del__` ![finalize2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/finalize2.png) -step2, do **update_refs** in **unreachable** +Step 2: Do **update_refs** in **unreachable** -notice, the first bit flag of `b` in **_gc_prev** is set, indicate that it's finalizer is called +Notice that the first bit flag of `b` in **_gc_prev** is set, indicating that its finalizer has been called ![finalize3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/finalize3.png) -step3, do **subtract_refs** in **unreachable** +Step 3: Do **subtract_refs** in **unreachable** -this step aims to elimate the reference cycle among objects in **unreachable** +This step aims to eliminate the reference cycle among objects in **unreachable**. -then, traverse every object in **unreachable**, check if any object with the value of copy of the reference count > 0, if so, it means at least one object in **unreachable** makes outer object create new reference to this object, then go to step4 +Then, traverse every object in **unreachable** and check if any object has the value of copy of the reference count > 0. If so, it means at least one object in **unreachable** makes an outer object create a new reference to this object, then go to step 4. -if not, after traverse, collecte all objects in **unreachable** and return +If not, after traversing, collect all objects in **unreachable** and return ![finalize4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/finalize4.png) -stp4, move all objects in **unreachable** to **old** generation +Step 4: Move all objects in **unreachable** to **old** generation -if there's any object in **unreachable** defined it's own `__del__`, the `__del__` methods will be called in step1, and all objects in **unreachable** will survive this round of garbage collection +If any object in **unreachable** has defined its own `__del__`, the `__del__` methods will be called in step 1, and all objects in **unreachable** will survive this round of garbage collection ![finalize5](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/finalize5.png) -in the next round of gc, object `a1` and `a2` will be collected, because object `b`'s reference count is greater than 0 after the **subtract_refs**, it won't be moved to **unreachable** +In the next round of GC, objects `a1` and `a2` will be collected. Because object `b`'s reference count is greater than 0 after **subtract_refs**, it won't be moved to **unreachable**. -if the `__del__` methods is called, the bit flag in **_gc_prev** will be set, so the `__del__` will be called only once +If the `__del__` method is called, the bit flag in **_gc_prev** will be set, so `__del__` will be called only once ### threshold -there are three generations totally, and three **threshold**, each generation has a **threshold** +There are three generations in total, and three **thresholds**. Each generation has a **threshold**. -before performing collection, CPython will exam from high generation to low generation, if currently the number of objects in the current generation is greater than **threshold**, **gc** will begin in the current generation +Before performing collection, CPython will examine from high generation to low generation. If the number of objects in the current generation is greater than the **threshold**, **GC** will begin in the current generation ```python3 >>> gc.get_threshold() @@ -481,7 +481,7 @@ before performing collection, CPython will exam from high generation to low gene ``` -it can be set manually +It can be set manually ```python3 gc.set_threshold(500) @@ -491,42 +491,44 @@ gc.set_threshold(100, 20) ### when will generational gc be triggered -one way is to call `gc.collect()` manually, it will collect the highest collection directly +One way is to call `gc.collect()` manually. It will collect the highest generation directly ```python3 ->>> imporr gc +>>> import gc >>> gc.collect() ``` -the other way is when the intepreter malloc a new space from the heap +The other way is when the interpreter mallocs a new space from the heap ![generation_trigger1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/generation_trigger1.png) -the `collect` procedure will check from the highest to lowest generation +The `collect` procedure will check from the highest to lowest generation. -first check the **generation2**, **generation2**'s count is less than **threashold** +First, check **generation2**. **generation2**'s count is less than the **threshold** ![generation_trigger2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/generation_trigger2.png) -then check **generation1**, **generation1**'s count is greater than **threashold**, collect procedure begins in **generation1** +Then check **generation1**. **generation1**'s count is greater than the **threshold**, so the collect procedure begins in **generation1** ![generation_trigger3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/generation_trigger3.png) -the collect procedure will merge all the generations lower than the current, and do what's described above +The collect procedure will merge all the generations lower than the current one and do what's described above ![generation_trigger4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/generation_trigger4.png) -if the gc is collecting the highest generation, all the free_list will be freed, if you read other article such as [list](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/list.md) or [tuple](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md), you can find what's free_list +If the GC is collecting the highest generation, all the free_list will be freed. If you read other articles such as [list](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/list.md) or [tuple](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md), you can find what free_list is # summary -because the gc algorithm CPython used is not a parallel algorithm, a global lock such as [gil](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil.md) is necessary to protect the critical region when set up the track of an object to gc, or when garbage collecting any of the generations +Because the GC algorithm CPython uses is not a parallel algorithm, a global lock such as the [GIL](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil.md) is necessary to protect the critical region when setting up the track of an object to GC, or when garbage collecting any of the generations. -while other garbage collector in other programming language such as [Java-g1](http://idiotsky.top/2018/07/28/java-g1/) use **Young GC** or **Mix GC**(combined with Tri-Color algorithm for global concurrent marking) to do the garbage collection +Other garbage collectors in other programming languages such as [Java-g1](http://idiotsky.top/2018/07/28/java-g1/) use **Young GC** or **Mix GC** (combined with the Tri-Color algorithm for global concurrent marking) to do the garbage collection # read more * [Garbage collection in Python: things you need to know](https://rushter.com/blog/python-garbage-collector/) -* [the garbage collector](https://pythoninternal.wordpress.com/2014/08/04/the-garbage-collector/) \ No newline at end of file +* [the garbage collector](https://pythoninternal.wordpress.com/2014/08/04/the-garbage-collector/) +* [关于 `__del__` 和 `finalizers` 和 `unreachable`](https://github.com/zpoint/CPython-Internals/issues/28) + diff --git a/Interpreter/gc/gc_cn.md b/Interpreter/gc/gc_cn.md index ff3dba8..05721b7 100644 --- a/Interpreter/gc/gc_cn.md +++ b/Interpreter/gc/gc_cn.md @@ -105,6 +105,7 @@ $ ./python.exe -m dis test.py `2 STORE_NAME 0` ```c +/* cpython/Python/ceval.c */ case TARGET(STORE_NAME): { /* name 是一个 str 对象, 值为 's' ns 是 local namespace @@ -149,6 +150,7 @@ case TARGET(STORE_NAME): { `8 DELETE_NAME 0 (s)` ```c +/* cpython/Python/ceval.c */ case TARGET(DELETE_NAME): { /* name 这里为 's' ns 为 the local namespace @@ -408,9 +410,9 @@ CPython 总共使用了 3 代, 新创建的对象都会被存储到第一代中( ### finalizer -如果一个需要被垃圾回收的对象定义了自己的 `__del__` 怎么办呢? 毕竟有可能在 `__del__` 中增加了新的引用 +如果一个存在引用循环等非常规的对象, 需要被垃圾回收, 并且定义了自己的 `__del__` 怎么办呢? 毕竟有可能在 `__del__` 中增加了新的引用 -在 python3.4 之前, 这部分对象即使被移动到了 **unreachable**, 也是不会被释放回 heap 空间中的 +在 python3.4 之前, 这部分对象会被移动到 `gc.garbage` 中, 你需要手动调用他们的 `__del__` 方法并且对他们进行回收 在 python3.4 后, [pep-0442](https://legacy.python.org/dev/peps/pep-0442/) 解决了这个问题 @@ -504,7 +506,7 @@ CPython 中一共有 3 代, 对应了 3 个 **threshold**, 每一代对应一个 一个方式是直接调用 `gc.collect()`, 不传参数的情况下直接从最老年代开始回收 ```python3 ->>> imporr gc +>>> import gc >>> gc.collect() ``` @@ -538,4 +540,7 @@ CPython 中一共有 3 代, 对应了 3 个 **threshold**, 每一代对应一个 # 更多资料 * [Garbage collection in Python: things you need to know](https://rushter.com/blog/python-garbage-collector/) -* [the garbage collector](https://pythoninternal.wordpress.com/2014/08/04/the-garbage-collector/) \ No newline at end of file +* [the garbage collector](https://pythoninternal.wordpress.com/2014/08/04/the-garbage-collector/) + +* [关于 `__del__` 和 `finalizers` 和 `unreachable`](https://github.com/zpoint/CPython-Internals/issues/28) + diff --git a/Interpreter/gil/gil.md b/Interpreter/gil/gil.md index 52e2b37..de7d594 100644 --- a/Interpreter/gil/gil.md +++ b/Interpreter/gil/gil.md @@ -1,6 +1,6 @@ -# gil +# GIL -# contents +# Contents * [related file](#related-file) * [introduction](#introduction) @@ -17,69 +17,69 @@ * [switch_cond and switch_mutex](#switch_cond-and-switch_mutex) * [when will the gil be released](#when-will-the-gil-be-released) -# related file +# Related files * cpython/Python/ceval.c * cpython/Python/ceval_gil.h * cpython/Include/internal/pycore_gil.h -# introduction +# Introduction -this is the defination of the [**Global Interpreter Lock**](https://wiki.python.org/moin/GlobalInterpreterLock) +This is the definition of the [**Global Interpreter Lock**](https://wiki.python.org/moin/GlobalInterpreterLock). > In CPython, the global interpreter lock, or GIL, is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once. This lock is necessary mainly because CPython's memory management is not thread-safe. (However, since the GIL exists, other features have grown to depend on the guarantees that it enforces.) -## thread scheduling before python32 +## Thread scheduling before Python 3.2 -basically, the **tick** is a counter for how many opcodes current thread executed continuously without releasing the **gil** +Basically, the **tick** is a counter for how many opcodes the current thread has executed continuously without releasing the **GIL**. -if the current thread is running a CPU-bound task, it will release the **gil** and offer an opportunity for other thread to run for every 100 **ticks** +If the current thread is running a CPU-bound task, it will release the **gil** and offer an opportunity for another thread to run for every 100 **ticks**. -if the current thread is running an IO-bound task, the **gil** will be relaesed manually if you call `sleep/recv/send(...etc)` even without count to 100 **ticks** +If the current thread is running an IO-bound task, the **GIL** will be released manually if you call `sleep/recv/send(...etc)` even without count to 100 **ticks**. -you can call `sys.setcheckinterval()` to set other **tick** count value instead of 100 +You can call `sys.setcheckinterval()` to set other **tick** count value instead of 100. ![old_gil](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/old_gil.png) (picture from [Understanding the Python GIL(youtube)](https://www.youtube.com/watch?v=Obt-vMVdM8s)) -because the **tick** is not time-based, some thread might run far longer than other threads +Because the **tick** is not time-based, some threads might run far longer than others. -in multi-core machine, if two threads both running CPU-bound tasks, the os might schedule the two threads running on different cores, there might be a situation that one thread holding the **gil** executing it's task in it's **100 ticks cycle** in a core, while the thread in the other core wakes up preiodly try to acquire the **gil** but fail, spinning the CPU +On a multi-core machine, if two threads are both running CPU-bound tasks, the OS might schedule them to run on different cores. There might be a situation where one thread holding the **GIL** is executing its task in its **100 ticks cycle** on one core, while the thread on the other core wakes up periodically trying to acquire the **GIL** but fails, spinning the CPU. -the job(thread) schedule mechanism is fully controlled by the operating system, the thread handling IO-bound task have to wait for other thread to release the **gil**, and other thread might re-acquire the **gil** after it release the **gil**, which makes the current IO-bound task thread wait even longer (actually, the thread that cause os's context-switch by it self will have higher priority than those thread forced by os, programmer can utilize this feature by putting the IO-bound thread to sleep as soon as possible) +The job (thread) scheduling mechanism is fully controlled by the operating system. The thread handling an IO-bound task has to wait for another thread to release the **GIL**, and that other thread might re-acquire the **GIL** after releasing it, which makes the current IO-bound task thread wait even longer (actually, the thread that causes the OS's context-switch by itself will have higher priority than those threads forced by the OS; programmers can utilize this feature by putting the IO-bound thread to sleep as soon as possible). ![gil_battle](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil_battle.png) (picture from [Understanding the Python GIL(youtube)](https://www.youtube.com/watch?v=Obt-vMVdM8s)) -## thread scheduling after python32 +## Thread scheduling after Python 3.2 -due to some performance issue in multi-core machine, the implementation of the **gil** has changed a lot after python3.2 +Due to some performance issues on multi-core machines, the implementation of the **GIL** has changed a lot since Python 3.2. -if there's only one thread, it can run forever without check and release **gil** +If there's only one thread, it can run forever without checking and releasing the **GIL**. -if there're more than one threads, the thread currently blocking by the **gil** will wait for a period of timeout and set the **gil_drop_request** to 1, and continue waiting, the thread currently holding the **gil** will release the **gil** and wait for same period of timeout if the **gil_drop_request** is set to 1, thread currently blocking will be signaled and is able to acqure the **gil** +If there is more than one thread, the thread currently blocked by the **GIL** will wait for a timeout period and set the **gil_drop_request** to 1, then continue waiting. The thread currently holding the **GIL** will release the **GIL** and wait for the same timeout period if the **gil_drop_request** is set to 1. The thread currently blocking will be signaled and is able to acquire the **GIL**. ![new_gil](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/new_gil.png) (picture from [Understanding the Python GIL(youtube)](https://www.youtube.com/watch?v=Obt-vMVdM8s)) -the thread set the **gil_drop_request** to 1 might not be the thread acquire the **gil** +The thread that sets **gil_drop_request** to 1 might not be the thread that acquires the **GIL**. -if the current thread is waiting for the interval, and owner of the **gil** changed during the waiting **interval**, after wake up, the current thread need to wait, set **gil_drop_request** to 1 and wait again +If the current thread is waiting for the interval and the owner of the **GIL** changes during the waiting **interval**, after waking up, the current thread needs to wait, set **gil_drop_request** to 1, and wait again. ![new_gil2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/new_gil2.png) (picture from [Understanding the Python GIL(youtube)](https://www.youtube.com/watch?v=Obt-vMVdM8s)) -for those who are interested in detail, please refer to [Understanding the Python GIL(article)](http://www.dabeaz.com/GIL/) +For those who are interested in further details, please refer to [Understanding the Python GIL(article)](http://www.dabeaz.com/GIL/). -## memory layout +## Memory layout ![git_layout](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil_layout.png) -# fields +# Fields -the python intepreter is a program written in C, every executable program written in C have a `main` function +The Python interpreter is a program written in C. Every executable program written in C has a `main` function. -those `main` related functions are defined in `cpython/Modules/main.c`, you will find that the `main` related function does some inilialization for the intepreter status before execute the `main loop`, the `_gil_runtime_state` will be created and initialized in the inilialization +Those `main`-related functions are defined in `cpython/Modules/main.c`. You will find that the `main`-related function does some initialization for the interpreter status before executing the `main loop`. The `_gil_runtime_state` will be created and initialized during this initialization. ```python3 ./python.exe @@ -97,17 +97,17 @@ those `main` related functions are defined in `cpython/Modules/main.c`, you will ``` -**interval** is the suspend timeout before set the `gil_drop_request` in microseconds, 5000 microseconds is 0.005 seconds +**interval** is the suspend timeout before setting the `gil_drop_request` in microseconds, 5000 microseconds is 0.005 seconds. -it's stored as microseconds in C and represent as seconds in python +It's stored as microseconds in C and represented as seconds in Python. ## last_holder -**last_holder** stores the C address of the last PyThreadState hloding the **gil**, this helps us know whether anyone else was scheduled after we dropped the **gil** +**last_holder** stores the C address of the last `PyThreadState` holding the **GIL**, this helps us know whether anyone else was scheduled after we dropped the **GIL**. ## locked -**locked** is a field of type **_Py_atomic_int**, -1 indicate uninitialized, 0 means no one is currently holding the **gil**, 1 means someone is holding it. This is atomic because it can be read without any lock taken in ceval.c +**locked** is a field of type **_Py_atomic_int**, -1 indicates uninitialized, 0 means no one is currently holding the **GIL**, 1 means someone is holding it. This is atomic because it can be read without any lock taken in `ceval.c`. ```c /* cpython/Python/ceval_gil.h */ @@ -144,9 +144,9 @@ static void drop_gil(PyThreadState *tstate) ## switch_number -**switch_number** is a counter for number of **gil** switches since the beginning +**switch_number** is a counter for the number of **GIL** switches since the beginning. -it's used in function `take_gil` +It's used in function `take_gil`. ```c static void take_gil(PyThreadState *tstate) @@ -181,17 +181,17 @@ static void take_gil(PyThreadState *tstate) ## mutex -**mutex** is a mutex used for protecting `locked`, `last_holder`, `switch_number` and other variables in `_gil_runtime_state` +**mutex** is a mutex used for protecting `locked`, `last_holder`, `switch_number`, and other variables in `_gil_runtime_state`. ## cond -**cond** is a condition variable, combined with **mutex**, used for signaling release of **gil** +**cond** is a condition variable, combined with **mutex**, used for signaling the release of the **GIL**. ## switch_cond and switch_mutex -**switch_cond** is another condition variable, combined with **switch_mutex** can be used for making sure that the thread acquire the **gil** is not the thread release the **gil**, avoiding waste of time slice +**switch_cond** is another condition variable, combined with **switch_mutex** can be used for making sure that the thread acquiring the **GIL** is not the thread that released the **GIL**, avoiding a waste of the time slice. -it can be turned off without the defination of `FORCE_SWITCHING` +It can be turned off without the definition of `FORCE_SWITCHING`. ```c static void drop_gil(PyThreadState *tstate) @@ -225,15 +225,15 @@ static void drop_gil(PyThreadState *tstate) ``` -# when will the gil be released +# When will the GIL be released -the `main_loop` in `cpython/Python/ceval.c` is a big `for loop`, and a big `switch statement` +The `main_loop` in `cpython/Python/ceval.c` is a big `for loop`, and a big `switch statement`. -the big `for loop` loads opcode one by one, and the big `switch statement` execute different c code according to the opcode +The big `for loop` loads opcode one by one, and the big `switch statement` executes different C code according to the opcode. -the `for loop` will check the variable `gil_drop_request` and release the `gil` if necessary +The `for loop` will check the variable `gil_drop_request` and release the `gil` if necessary. -not every opcode will check the `gil_drop_request`, some opcode ends with `FAST_DISPATCH()` will go to the next statement directly, while some opcode ends with `DISPATCH()` act as `continue statement` and will go to the beginning of the for loop +Not every opcode will check the `gil_drop_request`. Some opcodes that end with `FAST_DISPATCH()` will go to the next statement directly, while some opcodes that end with `DISPATCH()` act as a `continue statement` and will go to the beginning of the for loop. ```c /* cpython/Python/ceval.c */ diff --git a/Interpreter/gil/gil_cn.md b/Interpreter/gil/gil_cn.md index 151770c..6f93704 100644 --- a/Interpreter/gil/gil_cn.md +++ b/Interpreter/gil/gil_cn.md @@ -45,7 +45,7 @@ 因为 **tick** 并不是以时间为基准计数, 而是以 opcode 个数为基准的计数, 有一些 opcode 代码复杂耗时长, 一些耗时短, 进而导致同样的 100 个 **tick**, 一些线程的执行时间总是执行的比另一些线程长 -在多核机器上, 如果两个线程都在执行 CPU 密集型的任务, 操作系统有可能让这两个线程在不同的核心上运行, 也许会出现以下的情况, 当一个拥有了 **gil** 的线程在一个核心上执行 100 次 **tick** 的过程中, 在另一个核心上运行的线程频繁的进行抢占 **gil**, 抢占失败的循环, 导致 CPU 瞎忙影响性能 +在多核机器上, 如果两个线程都在执行 CPU 密集型的任务, 操作系统有可能让这两个线程在不同的核心上运行, 也许会出现以下的情况, 当一个拥有了 **gil** 的线程在一个核心上执行 100 次 **tick** 的过程中, 在另一个核心上运行的线程频繁的进行抢占 **gil**, 抢占失败的循环, 导致 CPU 空转影响性能 当前的实现完全的把任务(线程)调度交给了操作系统, 哪个线程会抢到锁被唤醒, 哪个线程抢不到锁被阻塞, 程序员是无法控制的, 这也就导致了以下情况发生的概率, 想象一下一个处理 IO 密集型的线程已经收到了 IO 的信号, 但是需要等待另一个线程释放 **gil** 才能去处理, 在等待的过程中, 另一个线程执行满了 **tick** 次数, 释放了 **gil**, 但是之后自己又抢到了自己释放的 **gil**, 导致前面的 IO 密集型的线程又需要至少多等待 100 个 **tick** 才能处理已经接收到的信号, 有可能在信号丢失前都无法及时处理(实际上, 自己主动触发操作系统进行任务调度的线程会比被操作系统强制触发任务调度的线程在执行队列里有更高的优先级, 程序员可以让 IO 密集型任务尽快的进入等待状态, 主动触发任务调度, 提高这个线程的优先级)(详情参考操作系统相关资料) @@ -159,15 +159,18 @@ static void take_gil(PyThreadState *tstate) unsigned long saved_switchnum; saved_switchnum = _PyRuntime.ceval.gil.switch_number; - /* 释放 gil.mutex, 并待 INTERVAL 微秒(默认 5000) 或者等待过程中收到 gil.cond 的信号 */ + /* 释放 gil.mutex, 并在以下两种条件下唤醒 + 1. 等待 INTERVAL 微秒(默认 5000) + 2. 还没有等待到 5000 微秒但是收到了 gil.cond 的信号 + */ COND_TIMED_WAIT(_PyRuntime.ceval.gil.cond, _PyRuntime.ceval.gil.mutex, INTERVAL, timed_out); /* 当前持有 gil.mutex 这把互斥锁 */ if (timed_out && _Py_atomic_load_relaxed(&_PyRuntime.ceval.gil.locked) && _PyRuntime.ceval.gil.switch_number == saved_switchnum) { - /* 如果超过了等待时间, 并且这段等待时间里没有进行 gil 的换手, 则让当前持有 gil 的线程进行释放 - 把 gil_drop_request 值设为 1 */ + /* 如果超过了等待时间, 并且这段等待时间里 gil 的持有者没有变更过, 则尝试让当前持有 gil 的线程进行释放gil + 把 gil_drop_request 值设为 1, 持有锁的线程看到这个值的时候, 会尝试放弃 gil */ SET_GIL_DROP_REQUEST(); } /* 继续回到 while 循环, 检查 gil 是否为锁住状态 */ @@ -187,7 +190,7 @@ static void take_gil(PyThreadState *tstate) ## switch_cond and switch_mutex -**switch_cond** 是另一个 condition variable, 和 **switch_mutex** 结合起来可以用来保证释放后重新获得 **gil** 的线程不是同一个前面释放 **gil** 的线程, 避免 **gil** 换手但是线程未切换浪费 cpu 时间 +**switch_cond** 是另一个 condition variable, 和 **switch_mutex** 结合起来可以用来保证释放后重新获得 **gil** 的线程不是同一个前面释放 **gil** 的线程, 避免 **gil** 切换时线程未切换浪费 cpu 时间 这个功能如果编译时未定义 `FORCE_SWITCHING` 则不开启 @@ -276,7 +279,7 @@ main_loop: } fast_next_opcode: - /* 忽略 */ + /* 忽略 */ switch (opcode) { case TARGET(NOP): { FAST_DISPATCH(); diff --git a/Interpreter/memory_management/memory_management.md b/Interpreter/memory_management/memory_management.md index 95eebb4..a5db7d5 100644 --- a/Interpreter/memory_management/memory_management.md +++ b/Interpreter/memory_management/memory_management.md @@ -42,27 +42,27 @@ # introduction -CPython has implemented it's own memory management mechanism, when you create a new object in python program, it's not directly malloced from the heap +CPython has implemented its own memory management mechanism. When you create a new object in a Python program, it's not directly malloced from the heap ![level](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/level.png) -we will figure out how the **python interpreter** part works internally in the following example +We will figure out how the **Python interpreter** part works internally in the following example ## object allocator -this is the created procedure of a `tuple` object with size n +This is the creation procedure of a `tuple` object with size n -step1, check if there's a chance to reuse `tuple` object in [free_list](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md#free-list), if so, goes to step4 +Step 1: Check if there's a chance to reuse a `tuple` object in [free_list](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md#free-list). If so, go to step 4. -step2, call **PyObject_GC_NewVar** to get a `PyTupleObject` with size `n` from memory management system(if the size of the object is fixed, **_PyObject_GC_New** will called instead) +Step 2: Call **PyObject_GC_NewVar** to get a `PyTupleObject` with size `n` from the memory management system (if the size of the object is fixed, **_PyObject_GC_New** will be called instead). -step3, call **_PyObject_GC_TRACK** to link the newly created object to the double linked list in [gc](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/gc.md#track).**generation0** +Step 3: Call **_PyObject_GC_TRACK** to link the newly created object to the doubly linked list in [gc](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/gc.md#track).**generation0** -step4, return +Step 4: Return ![tuple_new](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/tuple_new.png) -the maximum size of memory you are able to allocated is limited in **_PyObject_GC_Alloc** +The maximum size of memory you are able to allocate is limited in **_PyObject_GC_Alloc** ```c typedef ssize_t Py_ssize_t; @@ -77,46 +77,46 @@ if (basicsize > PY_SSIZE_T_MAX - sizeof(PyGC_Head)) ## raw memory allocator -follow the call stack, we can find that the **raw memory allocator** is mostly defined in `cpython/Objects/obmalloc.c` +Following the call stack, we can find that the **raw memory allocator** is mostly defined in `cpython/Objects/obmalloc.c` ```c #define SMALL_REQUEST_THRESHOLD 512 ``` -the procedure is described below +The procedure is described below: -* if the request memory block is greater than **SMALL_REQUEST_THRESHOLD**(512 bytes), the request will be delegated to the operating system's allocator -* else, the request will be delegated to python's **raw memory allocator** +* If the requested memory block is greater than **SMALL_REQUEST_THRESHOLD** (512 bytes), the request will be delegated to the operating system's allocator. +* Otherwise, the request will be delegated to Python's **raw memory allocator**. ## block -we need to know some concept before we look into how python's memory allocator work +We need to know some concepts before we look into how Python's memory allocator works. -**block** is the smallest unit in python's memory management system, the size of a block is the same size as a single **byte** +**block** is the smallest unit in Python's memory management system. The size of a block is the same as a single **byte** ```c typedef uint8_t block ``` -there're lots of memory **block** in differenct size, word **block** in memory **block** has a different meaning from type **block**, we will see later +There are lots of memory **blocks** of different sizes. The word **block** in memory **block** has a different meaning from the type **block**; we will see this later ## pool -a **pool** stores a collection of memory **block** of the same size +A **pool** stores a collection of memory **blocks** of the same size. -usually, the total size of memory blocks in a pool is 4kb, which is the same as most of the system page size +Usually, the total size of memory blocks in a pool is 4KB, which is the same as most system page sizes. -initially, the addresses for different memory block in the same pool are continously +Initially, the addresses for different memory blocks in the same pool are contiguous ![pool](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool.png) ## arena -an **arena** stores 256kb malloced from heap, it divides the memory block to 4kb per unit +An **arena** stores 256KB malloced from the heap. It divides the memory block into 4KB per unit. -whenever there needs a new pool, the allocator will ask an **arena** for a new memory block with size 4kb, and initialize the memory block as a new **pool** +Whenever a new pool is needed, the allocator will ask an **arena** for a new memory block with size 4KB and initialize the memory block as a new **pool** ![arena](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena.png) @@ -126,17 +126,17 @@ whenever there needs a new pool, the allocator will ask an **arena** for a new m ## usedpools -there's a C array named **usedpools** plays an important role in the memory management mechanism +There's a C array named **usedpools** that plays an important role in the memory management mechanism. -the defination of **usedpools** in C is convoluted, the following picture shows how these mentioned objects organized +The definition of **usedpools** in C is convoluted. The following picture shows how these mentioned objects are organized ![usedpools](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/usedpools.png) -element in **usedpools** is of type `pool_header *`, size of **usedpools** is 128, but only half of the elements are in used +Elements in **usedpools** are of type `pool_header *`. The size of **usedpools** is 128, but only half of the elements are in use. -**usedpools** stores **pool** object with free blocks, every **pool** object in **usedpools** has at least one free memory block +**usedpools** stores **pool** objects with free blocks. Every **pool** object in **usedpools** has at least one free memory block. -if you get **pool 1** from **idx0**, you can get one memory block(8 bytes) from **pool 1** each time, if you get **pool 4** from **idx2**, you can get one memory block(24 bytes) from **pool 4** each time, and so on +If you get **pool 1** from **idx0**, you can get one memory block (8 bytes) from **pool 1** each time. If you get **pool 4** from **idx2**, you can get one memory block (24 bytes) from **pool 4** each time, and so on ```python3 cpython/Objects/obmalloc.c @@ -171,7 +171,7 @@ and so on ### why only half of the usedpools elements are used -every memory request will be routed to the **idxn** in **usedpools**, there must be a very fast way to access **idxn** for the underlying memory request with size **nbytes** +Every memory request will be routed to the **idxn** in **usedpools**. There must be a very fast way to access **idxn** for the underlying memory request with size **nbytes** ```c #define ALIGNMENT_SHIFT 3 @@ -185,13 +185,13 @@ if the request size **nbytes** is 7, (7 - 1) >> 3 is 0, idxn = 0 + 0, usedpools[ if the request size **nbytes** is 24, (24 - 1) >> 3 is 2, idxn = 2 + 2, usedpools[2 + 2] will be the target list, so the head of **idx2** is the target pool -the `usedpools`‘s size is two times larger than the size it actually used, so that you are able to calcuate the **idx** and access the first free pool in a very short time +The `usedpools`'s size is two times larger than what it actually uses, so that you are able to calculate the **idx** and access the first free pool in a very short time # example ## overview -assume we are going to reqest 5 bytes from python's memory allocator, because the request size is less than **SMALL_REQUEST_THRESHOLD**(512 bytes), it's routed to python's raw memory allocator instead of the system's allocator(**malloc** system call) +Assume we are going to request 5 bytes from Python's memory allocator. Because the request size is less than **SMALL_REQUEST_THRESHOLD** (512 bytes), it's routed to Python's raw memory allocator instead of the system's allocator (**malloc** system call) ```c size = (uint)(nbytes - 1) >> ALIGNMENT_SHIFT = (5 - 1) >> 3 = 0 @@ -199,13 +199,13 @@ pool = usedpools[0 + 0] ``` -so **pool** header will be the first element in **idx0**, follow the linked list, we can find that the first **pool** which can offer memory blocks is **pool1** +So the **pool** header will be the first element in **idx0**. Following the linked list, we can find that the first **pool** which can offer memory blocks is **pool1** ![example0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/example0.png) ## how does memory block organize in pool -assume this is the current state of **pool1** +Assume this is the current state of **pool1** **ref.count** is a counter for how many **blocks** are currently in use @@ -221,21 +221,19 @@ assume this is the current state of **pool1** **maxnextoffset** act as the high water mark for **nextoffset**, if **nextoffset** exceeds **maxnextoffset**, it means all blocks in the pool are in used, and the pool is full -the total size of a **pool** is 4kb, same as system page size +The total size of a **pool** is 4KB, same as the system page size ![pool_organize0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize0.png) ### allocate in pool with no freed block -"no freed block" doesn't mean the pool is full and no any other room left, instead, it means no block allocated in the pool ever called **free** +"No freed block" doesn't mean the pool is full and has no other room left. Instead, it means no block allocated in the pool has ever called **free**. -when we allocate a memory block <= 8 bytes - -the block **freeblock** pointed to will be chosen +When we allocate a memory block <= 8 bytes, the block that **freeblock** points to will be chosen ![pool_organize1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize1.png) -because the value stored in block **freeblock** pointed to is a NULL pointer, there's no other **freed** block in the middle of the **pool**, we must get a block from the tail of the pool, these blocks from the tail are never used before in the pool +Because the value stored in the block that **freeblock** points to is a NULL pointer, there's no other **freed** block in the middle of the **pool**. We must get a block from the tail of the pool; these blocks from the tail have never been used before in the pool **nextoffset** and **maxnextoffset** will be used to check whether there's room in the tail of the **pool** @@ -247,23 +245,23 @@ because the value stored in block **freeblock** pointed to is a NULL pointer, th ![pool_organize2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize2.png) -request one more memory block +Request one more memory block **nextoffset** now exceeds **maxnextoffset** ![pool_organize3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize3.png) -request one more memory block +Request one more memory block ![pool_organize4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize4.png) -because the **nextoffset** is greater than **maxnextoffset**, the **pool** is full, there're no more freeblocks to use, so **freeblock** becomes a null pointer +Because **nextoffset** is greater than **maxnextoffset**, the **pool** is full. There are no more freeblocks to use, so **freeblock** becomes a null pointer. -we need to unlink the pool from the **usedpools** +We need to unlink the pool from the **usedpools**. ![pool_organize_full](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_full.png) -in higher level view +In a higher-level view ![arena_pool_full](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_pool_full.png) @@ -271,163 +269,163 @@ in higher level view #### free in full pool -it's idealize that the used blocks and free blocks are seperated by a **freeblock** pointer in a **pool** and every memory request will be routed to the beginning of the seperated pointer +It's ideal when the used blocks and free blocks are separated by a **freeblock** pointer in a **pool** and every memory request is routed to the beginning of the separated pointer. -what if there're some used blocks in the random location of the pool being freed ? +What if there are some used blocks in random locations of the pool being freed? -assume we are freeing the block in address `0x10ae3dfe8` in **pool1** +Assume we are freeing the block at address `0x10ae3dfe8` in **pool1** ![pool_organize_free0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free0.png) -step1, set the value in the **block** to the same value as **freeblock** +Step 1: Set the value in the **block** to the same value as **freeblock** ![pool_organize_free1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free1.png) -step2, make the **freeblock** points to the current freeing **block** +Step 2: Make **freeblock** point to the current freeing **block** ![pool_organize_free2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free2.png) -decrement the value in **ref.count** +Decrement the value in **ref.count**. -step3, check whether the **pool** is full(check whether the value in the freeing **block** is NULL pointer), if so, relink the **pool** to **usedpools** and return, else go to step4 +Step 3: Check whether the **pool** is full (check whether the value in the freeing **block** is a NULL pointer). If so, relink the **pool** to **usedpools** and return; otherwise go to step 4 ![pool_organize_free3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free3.png) -step4 -* check if the all the **pools** in the current **arena** are free, if so free the entire **arena** -* if this is the only free pool in the arena, add the arena back to the `usable_arenas` list -* sort **usable_arenas** to make sure the **arena** with smaller count of free pools in the front +Step 4: +* Check if all the **pools** in the current **arena** are free. If so, free the entire **arena**. +* If this is the only free pool in the arena, add the arena back to the `usable_arenas` list. +* Sort **usable_arenas** to make sure the **arena** with the smaller count of free pools is in front #### free in not full pool -let's free the final block in address `0x10ae3dff8` in **pool1** +Let's free the final block at address `0x10ae3dff8` in **pool1**. -the steps are same as above steps +The steps are the same as above. -step1 +Step 1 ![pool_organize_free4](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free4.png) -step2 +Step 2 ![pool_organize_free5](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free5.png) -decrement the value in **ref.count** +Decrement the value in **ref.count**. -step3, the **pool** is not full, so goes to step4 +Step 3: The **pool** is not full, so go to step 4. -step4, return +Step 4: Return ![pool_organize_free6](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free6.png) -let's free the second to the last block in address `0x10ae3dff0` in **pool1** +Let's free the second-to-last block at address `0x10ae3dff0` in **pool1**. -step1 +Step 1 ![pool_organize_free7](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free7.png) -step2 +Step 2 ![pool_organize_free8](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free8.png) -decrement the value in **ref.count** +Decrement the value in **ref.count**. -step3, the **pool1** is not full, so goes to step4 +Step 3: **pool1** is not full, so go to step 4. -step4, return +Step 4: Return ![pool_organize_free9](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_free9.png) ### allocate in pool with freed block -now, there're some blocks called **free** previously in the **pool1** +Now there are some blocks that were previously freed in **pool1**. -let's request one more memory block +Let's request one more memory block. -because the value stored in block **freeblock** pointed to is a not a NULL pointer, it's a pointer to other block in the same **pool**, it means this block is used before and freed +Because the value stored in the block that **freeblock** points to is not a NULL pointer, it's a pointer to another block in the same **pool**. This means this block was used before and then freed ![pool_organize_allocate_after_free0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_allocate_after_free0.png) -we increment the **ref.count** +We increment the **ref.count**. -make the **freeblock** points to what's stored in the current block +Make **freeblock** point to what's stored in the current block. -the careful reader will realize that **freeblock** is actually a pointer to a single linked list, the linked list will chain all the freed **blocks** in the same **pool** together +The careful reader will realize that **freeblock** is actually a pointer to a singly linked list. The linked list will chain all the freed **blocks** in the same **pool** together. -if the single linked list reaches the end, it means there're no more blocks used previously and freed, we should try to get a block from those never used room before in the **pool**, we already learn this strategy from [allocate in pool with no freed block](#allocate-in-pool-with-no-freed-block) +If the singly linked list reaches the end, it means there are no more blocks that were used previously and freed. We should try to get a block from those never-used rooms in the **pool**. We already learned this strategy from [allocate in pool with no freed block](#allocate-in-pool-with-no-freed-block) ![pool_organize_allocate_after_free1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_allocate_after_free1.png) -request one more memory block +Request one more memory block **freeblock** follows to the next position in the single linked list ![pool_organize_allocate_after_free2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/pool_organize_allocate_after_free2.png) -if we request one more memory block, the **pool** will be full and unlink from the **usedpools**, we've learned before [free in pool](#free-in-pool) +If we request one more memory block, the **pool** will be full and unlinked from the **usedpools**. We've learned this before in [free in pool](#free-in-pool) ### how does pool organize in arena #### arena overview part1 -the **arena** stores a collection of **pool** +The **arena** stores a collection of **pools**. -this is the initial state +This is the initial state. -the **usedpools** is empty and there're no alive **arena** object +The **usedpools** is empty and there are no alive **arena** objects ![arena_orgnaize_overview0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize_overview0.png) -when we request a **block** with 10 bytes, 16 **arena** will be created, each **arena** will contain 64 **pool**, each **pool** will occupy 4kb +When we request a **block** with 10 bytes, 16 **arenas** will be created. Each **arena** will contain 64 **pools**, and each **pool** will occupy 4KB ![arena_orgnaize_overview1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize_overview1.png) -the first free **pool** in the first avaliable **arena** will be inserted to the **idx1** +The first free **pool** in the first available **arena** will be inserted into **idx1** ![arena_orgnaize_overview2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize_overview2.png) -and the **idx1** of **usedpools** is not empty any more, follow the procedure [allocate in pool with no freed block](#allocate-in-pool-with-no-freed-block), we can get the **block** from **poo1** +And the **idx1** of **usedpools** is not empty anymore. Following the procedure [allocate in pool with no freed block](#allocate-in-pool-with-no-freed-block), we can get the **block** from **pool1** #### allocate in arena with no freed pool -"no freed pool" doesn't mean the arena is full and no any other room left, instead, it means no pool allocated in the arena ever called **free** +"No freed pool" doesn't mean the arena is full and has no other room left. Instead, it means no pool allocated in the arena has ever called **free**. -assume this is the current state of the first arena +Assume this is the current state of the first arena ![arena_orgnaize0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize0.png) -request one more **pool** +Request one more **pool**. -because the **freepools** is a null pointer, and **nfreepools** is greater than 0, the address **pool_address** pointed to will be used as the new **pool** +Because **freepools** is a null pointer and **nfreepools** is greater than 0, the address that **pool_address** points to will be used as the new **pool** ![arena_orgnaize1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize1.png) -**nfreepools** is decremented and **pool_address** will point to next free **pool** +**nfreepools** is decremented and **pool_address** will point to the next free **pool**. ![arena_orgnaize2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize2.png) -request one more **pool**, the **arena** is full +Request one more **pool**. The **arena** is full ![arena_orgnaize3](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize3.png) #### free in arena -when all the **blocks** in a pool are freed, the **pool** will also be unlink in the **usedpools** and insert to the front of the **arena** +When all the **blocks** in a pool are freed, the **pool** will also be unlinked from **usedpools** and inserted at the front of the **arena**. -assume we are freeing **pool3** +Assume we are freeing **pool3** **nextpool** in **pool3** will be set to the same value as **freepools** in **arena** ![arena_orgnaize_free0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize_free0.png) -value of **freepools** in **arena** will be set to the address of **pool3** +The value of **freepools** in **arena** will be set to the address of **pool3** **nfreepools** will be incremented ![arena_orgnaize_free1](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize_free1.png) -let's free **pool63** +Let's free **pool63** **nextpool** of the current freeing **pool** will be set to the value in **freepools** and **freepools** in **arena** will point to the current freeing **pool** @@ -435,13 +433,13 @@ let's free **pool63** ![arena_orgnaize_free2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize_free2.png) -the free strategy of **arena** is similar to **pool**, **freepools** is the pointer to the header of the single linked list +The free strategy of **arena** is similar to **pool**. **freepools** is the pointer to the head of the singly linked list #### allocate in arena with freed pool -request one more **pool** +Request one more **pool**. -**nfreepools** is greater than 0, and **freepools** is not a NULL pointer, the first **pool** in the single linked list will be taken +**nfreepools** is greater than 0, and **freepools** is not a NULL pointer, so the first **pool** in the singly linked list will be taken ![arena_orgnaize_allocate0](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnaize_allocate0.png) @@ -451,35 +449,35 @@ request one more **pool** #### arena overview part2 -initially, there're 16 **arenas** +Initially, there are 16 **arenas** ![arena_orgnize_overview_part20](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnize_overview_part20.png) -if all the **pools** in the **arena** are freed, the memory(256kb) these **pools** used will be returned to operating system, and the **arena** structure be moved to a single linked list named **unused_arena_objects**, what's in **unused_arena_objects** is only a shell, it does not contains any **pool** free to use +If all the **pools** in the **arena** are freed, the memory (256KB) these **pools** used will be returned to the operating system, and the **arena** structure will be moved to a singly linked list named **unused_arena_objects**. What's in **unused_arena_objects** is only a shell; it does not contain any **pool** free to use. -prior to Python 2.5, **pools** in arenas were never free()'ed. this strategy is used since Python 2.5 +Prior to Python 2.5, **pools** in arenas were never free()'ed. This strategy has been used since Python 2.5 ![arena_orgnize_overview_part21](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnize_overview_part21.png) -if all **arenas** are used up, and we request a new **arena** +If all **arenas** are used up and we request a new **arena** ![arena_orgnize_overview_part22](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnize_overview_part22.png) -python's allocator will malloc 256kb from operating system, and the newly malloced 256kb memory block will be linked to the **arena** structure in the first **unused_arena_objects** +Python's allocator will malloc 256KB from the operating system, and the newly malloced 256KB memory block will be linked to the **arena** structure in the first **unused_arena_objects**. -the first **arena** in the **unused_arena_objects** will be taken, and **unused_arena_objects** will be an empty pointer +The first **arena** in **unused_arena_objects** will be taken, and **unused_arena_objects** will become an empty pointer ![arena_orgnize_overview_part23](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnize_overview_part23.png) -if all **arenas** are used up, and we request a new **arena** again when the **unused_arena_objects** is empty +If all **arenas** are used up and we request a new **arena** again when **unused_arena_objects** is empty ![arena_orgnize_overview_part24](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnize_overview_part24.png) -size of the **arenas** will be doubled, when you need to request a new **pool** from **arena**, the next **arena**(position 17) will be chosen +The size of the **arenas** will be doubled. When you need to request a new **pool** from **arena**, the next **arena** (position 17) will be chosen ![arena_orgnize_overview_part25](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/arena_orgnize_overview_part25.png) -you can call `sys._debugmallocstats()` to get some static information about the memory management +You can call `sys._debugmallocstats()` to get some statistics about the memory management # read more * [Memory management in Python](https://rushter.com/blog/python-memory-managment/) \ No newline at end of file diff --git a/Interpreter/memory_management/memory_management_cn.md b/Interpreter/memory_management/memory_management_cn.md index c8dedfa..8dfec8d 100644 --- a/Interpreter/memory_management/memory_management_cn.md +++ b/Interpreter/memory_management/memory_management_cn.md @@ -184,7 +184,7 @@ pool = usedpools[size + size] 如果申请的大小 **nbytes** 为 7, (7 - 1) >> 3 为 0, idxn = 0 + 0, usedpools[0 + 0] 就是被选中的链表, **idx0** 中的第一个 **pool** 既为获取到的 **pool** -如果申请的大小 **nbytes** 为 24, (24 - 1) >> 3 为 2, idxn = 2 + 2, usedpools[0 + 0] 就是被选中的链表, **idx2** 中的第一个 **pool** 既为获取到的 **pool** +如果申请的大小 **nbytes** 为 24, (24 - 1) >> 3 为 2, idxn = 2 + 2, usedpools[2 + 2] 就是被选中的链表, **idx2** 中的第一个 **pool** 既为获取到的 **pool** `usedpools` 的大小为实际需要大小的两倍, 你可以通过以上的位移和加减运算就得到对应的 **idx** 位置, 这样设计极大地缩短了获得对应 **pool** 的入口的时间 @@ -485,4 +485,4 @@ python 的内存管理机制会重新像操作系统申请 256kb 的空闲空间 # 相关阅读 -* [Memory management in Python](https://rushter.com/blog/python-memory-managment/) \ No newline at end of file +* [Memory management in Python](https://rushter.com/blog/python-memory-managment/) diff --git a/Interpreter/module/module.md b/Interpreter/module/module.md index ebe4b27..188cc86 100644 --- a/Interpreter/module/module.md +++ b/Interpreter/module/module.md @@ -23,25 +23,25 @@ # memory layout -there's a struct named **PyModuleDef** defined in `Include/moduleobject.h` +There's a struct named **PyModuleDef** defined in `Include/moduleobject.h` ![layout_PyModuleDef](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/module/layout_PyModuleDef.png) -the **PyModuleObject** is defined in `Objects/moduleobject.c`, which contains a field with type **PyModuleDef** +The **PyModuleObject** is defined in `Objects/moduleobject.c`, which contains a field with type **PyModuleDef** ![layout_PyModuleObject](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/module/layout_PyModuleObject.png) # example -the field **md_dict** is the `__dict__` attribute of the module object +The field **md_dict** is the `__dict__` attribute of the module object. -**PyModuleDef** is optional, the index located in the m_base field is used for find the module by index in sys.modules, not by name +**PyModuleDef** is optional. The index located in the m_base field is used to find the module by index in sys.modules, not by name. -**m_size** stores the size of per-module data +**m_size** stores the size of per-module data. -**m_clear** and **m_free** are used for deallolcation +**m_clear** and **m_free** are used for deallocation. -for more detail please refer to [PEP 3121 -- Extension Module Initialization and Finalization](https://www.python.org/dev/peps/pep-3121/) +For more detail please refer to [PEP 3121 -- Extension Module Initialization and Finalization](https://www.python.org/dev/peps/pep-3121/) ```python3 import _locale @@ -59,13 +59,11 @@ import re # how to debug import -when you try to look into the source code to find out how `import` works - -follow the call stack +When you try to look into the source code to find out how `import` works, follow the call stack ![import_call_stack](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/module/import_call_stack.png) -the core callable object is `interp->importlib` which is initialized in +The core callable object is `interp->importlib`, which is initialized in ```c /* cpython/Python/pylifecycle.c */ @@ -84,7 +82,7 @@ initimport(PyInterpreterState *interp, PyObject *sysmod) ``` -search the `_frozen_importlib`, you can find the half-binary file `Python/importlib.h` with content +Searching for `_frozen_importlib`, you can find the half-binary file `Python/importlib.h` with content ```c /* Auto-generated by Programs/_freeze_importlib.c */ @@ -96,13 +94,13 @@ const unsigned char _Py_M__importlib_bootstrap[] = { ``` -it turns out that `_freeze_importlib.c` will freeze the python code in `Lib/importlib/_bootstrap.py` to the half-binary format`Python/importlib.h` +It turns out that `_freeze_importlib.c` will freeze the Python code in `Lib/importlib/_bootstrap.py` to the half-binary format `Python/importlib.h` ![importlib](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/module/importlib.png) -it's interesting that the `import` procedure is actually written in pure python, the attribute `PyId__find_and_load` of `interp->importlib` maps to the python function name `_find_and_load` defined in `Lib/importlib/_bootstrap.py` +It's interesting that the `import` procedure is actually written in pure Python. The attribute `PyId__find_and_load` of `interp->importlib` maps to the Python function name `_find_and_load` defined in `Lib/importlib/_bootstrap.py`. -whenever you modify the source code in `Lib/importlib/_bootstrap.py`, you need to regenerate the `Python/importlib.h` file, and recompile the source code +Whenever you modify the source code in `Lib/importlib/_bootstrap.py`, you need to regenerate the `Python/importlib.h` file and recompile the source code ```python3 # compile ./Programs/_freeze_importlib.c, and use the program to freeze @@ -117,7 +115,7 @@ make # how does import work -let's compile a script with only one line `import _locale` +Let's compile a script with only one line `import _locale` ```python3 ./python.exe -m dis test.py @@ -130,15 +128,15 @@ let's compile a script with only one line `import _locale` ``` -the core opcde here is `IMPORT_NAME` +The core opcode here is `IMPORT_NAME`. -it's tedious to copy and paste all the source code related to `IMPORT_NAME`, even with annotation +It's tedious to copy and paste all the source code related to `IMPORT_NAME`, even with annotation. -imagine that there're two or more threads currently importing the same `_locale` module, how does CPython handle this situation ? +Imagine that there are two or more threads currently importing the same `_locale` module. How does CPython handle this situation? -if you read the source code and following images, you will notice that the lock mechanism is used for preventing race condition +If you read the source code and the following images, you will notice that the lock mechanism is used for preventing race conditions. -the procedure is listed +The procedure is listed 1. opcode `IMPORT_NAME` will check if the name being imported is in sys.module, if so return what's in the sys.module 2. try to acquire the lock `_imp` @@ -163,7 +161,7 @@ in `position 5`, it will acquire the lock `_imp` before the call of every `finde ## how does finder work -there're currently three different finder in my `sys.meta_path` +There are currently three different finders in my `sys.meta_path` ```python3 >>> sys.meta_path @@ -173,7 +171,7 @@ there're currently three different finder in my `sys.meta_path` ### BuiltinImporter -`BuiltinImporter` will handle all the built-in module, when we call +`BuiltinImporter` will handle all the built-in modules. When we call ```python3 `import _locale` diff --git a/Interpreter/pyobject/pyobject.md b/Interpreter/pyobject/pyobject.md index 5e3f26d..55c03f4 100644 --- a/Interpreter/pyobject/pyobject.md +++ b/Interpreter/pyobject/pyobject.md @@ -6,7 +6,7 @@ * [memory layout](#memory-layout) * [overview](#overview) * [ceval](#ceval) -* [intepreter and thread](#intepreter-and-thread) +* [interpreter and thread](#interpreter-and-thread) * [read more](#read-more) # related file @@ -19,33 +19,33 @@ # memory layout -this is the layout of **PyObject**, it's the basic part of every other python object +This is the layout of **PyObject**. It's the basic part of every other Python object -every object in python can be cast to **PyObject**, i.e, list, tuple and etc +Every object in Python can be cast to **PyObject**, e.g., list, tuple, etc. `PyObject_HEAD_EXTRA` is the head of the double linked list, it's used for keeping track in [Garbage Collection](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/gc.md) `ob_refcnt` stores reference count of the current object, it's also used in [Garbage Collection](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/gc.md) -`ob_type` is a pointer to it's type object, i.e, `type("abc") ===> str`, `"abc"` is the PyObject, and `ob_type` in the PyObject will points to `str` +`ob_type` is a pointer to its type object. For example, `type("abc") ===> str`, `"abc"` is the PyObject, and `ob_type` in the PyObject will point to `str` ![PyObject](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/PyObject.png) # overview -whenever you execute a python program for a `.py` file, the compile phase will translate the human readable source code to a form called python bytecode +Whenever you execute a Python program from a `.py` file, the compile phase will translate the human-readable source code to a form called Python bytecode -the `pyc` file will be generated if necessary, and the interpreter phase will start the main loop, execute the bytecode in the `pyc` file line by line +The `pyc` file will be generated if necessary, and the interpreter phase will start the main loop, executing the bytecode in the `pyc` file line by line -the compiled `pyc` file will not boost the run time of the program, instead, the load time of the program will be faster because it doesn't need to generate `pyc` file again +The compiled `pyc` file will not boost the runtime of the program. Instead, the load time of the program will be faster because it doesn't need to generate the `pyc` file again -according to [pep-3147](https://www.python.org/dev/peps/pep-3147/), after python 3.3, `pyc` file will only be generated in the `import` mechanism, and `pyc` file will exists under the `__pycache__` directory +According to [PEP 3147](https://www.python.org/dev/peps/pep-3147/), after Python 3.3, `pyc` files will only be generated through the `import` mechanism, and `pyc` files will exist under the `__pycache__` directory ![executePy](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/executePy.png) # ceval -the main loop of the interpreter is defined in `cpython/Python/ceval.c` +The main loop of the interpreter is defined in `cpython/Python/ceval.c` ```c main_loop: @@ -61,21 +61,21 @@ main_loop: ``` -we can draw the procedure +We can draw the procedure ![ceval](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/ceval.png) -# intepreter and thread +# interpreter and thread -this is the defination of **PyInterpreterState** +This is the definition of **PyInterpreterState** ![PyInterpreterState](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/PyInterpreterState.png) -this is the defination of **PyThreadState** +This is the definition of **PyThreadState** ![PyThreadState](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/PyThreadState.png) -if we have two thread currently running +If we have two threads currently running ![organize](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/organize.png) diff --git a/Interpreter/pyobject/pyobject_cn.md b/Interpreter/pyobject/pyobject_cn.md index c2c162e..9ccaedb 100644 --- a/Interpreter/pyobject/pyobject_cn.md +++ b/Interpreter/pyobject/pyobject_cn.md @@ -35,7 +35,7 @@ python 解释器不仅仅是一个解释器, 它包括很多个部分 -当你通过 `.py` 文件来执行一个 python 程序时, 编译的部分会把你写的源代码转换为一种叫做 python 字节码的形式, 这个时候会生成 `pyc` 文件, 生成的字节码还有参数等信息会存储在这个 `pyc` 中, 解释器会读取这个文件, 开始 main loop 循环, 一个字节码一个字节码的读取, 并执行 +当你通过 `.py` 文件来执行一个 python 程序时, 编译的部分会把你写的源代码转换为一种叫做 python 字节码的形式, import 目录下的文件中的字节码会被缓存在 `pyc` 文件中, 解释器会开始 main loop 循环, 一个字节码一个字节码的读取, 并执行 这个 `pyc` 文件并不会提高你的程序运行速度, 它只会提高你程序的加载速度, 你只要没有修改过源代码文件, 则下次启动可以跳过生成 `pyc` 的部分, 仅此而已 diff --git a/Interpreter/slot/slot.md b/Interpreter/slot/slot.md index 9335ee9..1a91619 100644 --- a/Interpreter/slot/slot.md +++ b/Interpreter/slot/slot.md @@ -56,9 +56,9 @@ class A(object): ``` -what's the difference of accessing attribute `wing` and `x` of instance `a` ? +What's the difference between accessing attribute `wing` and `x` of instance `a`? -what's the difference of accessing attribute `wing` and `x` of type `A` ? +What's the difference between accessing attribute `wing` and `x` of type `A`? # instance access @@ -75,13 +75,11 @@ AttributeError: wing ``` -according to the **attribute accessing procedure** described in [descr](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/descr/descr.md) - -we can draw the procedure of accessing `a.wing` in a brief view +According to the **attribute accessing procedure** described in [descr](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/descr/descr.md), we can draw the procedure of accessing `a.wing` in a brief view ![instance_desc](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/instance_desc.png) -the **descriptor protocol** help us find an object named `descr`, type is `member_descriptor`, if you call +The **descriptor protocol** helps us find an object named `descr` whose type is `member_descriptor`. If you call ```python3 repr(descr) @@ -89,11 +87,11 @@ descr: ``` -the `descr` has a member named `d_member`, which stores information about the attribute name, type and offset +The `descr` has a member named `d_member`, which stores information about the attribute name, type, and offset. -with the informations stored in `d_member`, you are able to access the address of attribute in a very fast way +With the information stored in `d_member`, you are able to access the address of the attribute in a very fast way. -in the current example, if the begin address of the instance `a` is `0x00`, address after adding the offset is `0x18`, you can cast the address in `0x18` to a proper object according to `type` stored in `d_member` +In the current example, if the beginning address of the instance `a` is `0x00`, the address after adding the offset is `0x18`. You can cast the address at `0x18` to a proper object according to `type` stored in `d_member` ```c /* Include/structmember.h */ @@ -130,7 +128,7 @@ switch (l->type) { ``` -because wing attribute of instance `a` is set before, `AttributeError` won't be raised +Because the wing attribute of instance `a` is set before, `AttributeError` won't be raised ## instance access x @@ -140,11 +138,11 @@ because wing attribute of instance `a` is set before, `AttributeError` won't be ``` -type of `descr` is `int`, it's not a data descriptor(does not have `__get__` or `__set__` attribute), so the `descr` will be returned directly +The type of `descr` is `int`. It's not a data descriptor (does not have `__get__` or `__set__` attribute), so the `descr` will be returned directly ![instance_normal](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/instance_normal.png) -you are not able to create any other attribute in instance `a` if `A` defined `__slots__`, we will figure out why later +You are not able to create any other attribute in instance `a` if `A` defines `__slots__`. We will figure out why later ```python3 >>> a.not_exist = 100 @@ -166,7 +164,7 @@ AttributeError: 'A' object has no attribute 'not_exist' ``` -the procedure of accessing `A.wing` is nearly the same as `a.wing` +The procedure of accessing `A.wing` is nearly the same as `a.wing` ![type_desc](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/type_desc.png) @@ -178,7 +176,7 @@ the procedure of accessing `A.wing` is nearly the same as `a.wing` ``` -the procedure of accessing `A.x` is nearly the same as `a.x` +The procedure of accessing `A.x` is nearly the same as `a.x` ![type_normal](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/type_normal.png) @@ -188,29 +186,29 @@ the procedure of accessing `A.x` is nearly the same as `a.x` ### how does attributes initialize in the creation of `class A` ? -we can learn about the creation procedure in [type->creation of class](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/type/type.md#creation-of-class) +We can learn about the creation procedure in [type->creation of class](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/type/type.md#creation-of-class). -`type` object has many fields in it's C structure, the following picture only shows what's necessary for this topic +The `type` object has many fields in its C structure. The following picture only shows what's necessary for this topic. -what's in the `__slots__` is sorted and stored as a tuple in `ht_slots` +What's in `__slots__` is sorted and stored as a tuple in `ht_slots`. -the two attributes in `__slots__` is preallocated in the tail of the newly created type object `A` and two `PyMemberDef` pointers are stored in the order of `ht_slots` +The two attributes in `__slots__` are preallocated in the tail of the newly created type object `A`, and two `PyMemberDef` pointers are stored in the order of `ht_slots`. -while attribute `x` stays in the `tp_dict` field +Meanwhile, attribute `x` stays in the `tp_dict` field. -the `tp_dict` field does not have a key named `__dict__` +The `tp_dict` field does not have a key named `__dict__` ![type_create](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/type_create.png) ### how does attributes initialize in the creation of `instance a` ? -the memory location of the attributes in `__slots__` are preallocated +The memory locations of the attributes in `__slots__` are preallocated ![instance_create](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/instance_create.png) ### lookup procedure in MRO ? -it just iter through every type object in MRO, and if the name in `tp_dict` field, retuen `tp_dict[name]` +It just iterates through every type object in MRO, and if the name is in the `tp_dict` field, returns `tp_dict[name]` ```c /* cpython/Objects/typeobject.c */ diff --git a/Interpreter/thread/thread.md b/Interpreter/thread/thread.md index 476a71c..6189882 100644 --- a/Interpreter/thread/thread.md +++ b/Interpreter/thread/thread.md @@ -23,15 +23,15 @@ # prerequisites -the following contents will show you how CPython implements thread related function, you are able to get the answer of "Does posix semaphore or mutex used as a python thread lock ?" +The following contents will show you how CPython implements thread-related functions. You will be able to get the answer to "Is POSIX semaphore or mutex used as a Python thread lock?" -if you are confused about what **posix thread** is or what **posix semaphore** is, you need to refer to Chapter 11 and Chapter 12 of [APUE](https://www.amazon.com/Advanced-Programming-UNIX-Environment-3rd/dp/0321637739) and [UNP vol 2](https://www.amazon.com/UNIX-Network-Programming-Interprocess-Communications/dp/0130810819) +If you are confused about what **POSIX thread** is or what **POSIX semaphore** is, you need to refer to Chapter 11 and Chapter 12 of [APUE](https://www.amazon.com/Advanced-Programming-UNIX-Environment-3rd/dp/0321637739) and [UNP vol 2](https://www.amazon.com/UNIX-Network-Programming-Interprocess-Communications/dp/0130810819) -if you are interested in how thread/intepreter organized, please refer to [overwview](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/pyobject.md#intepreter-and-thread) +If you are interested in how thread/interpreter is organized, please refer to [overview](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/pyobject.md#intepreter-and-thread) # memory layout -a bootstate structure stores every informations a new python thread needed +A bootstate structure stores all the information a new Python thread needs ![bootstate](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/thread/bootstate.png) @@ -41,9 +41,9 @@ a bootstate structure stores every informations a new python thread needed ![thread](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/thread/thread.png) -the following content will only shows the posix part defined in **thread_pthread.h**, for other platform, even if the API of the system call is different, the idea is the same +The following content will only show the POSIX part defined in **thread_pthread.h**. For other platforms, even if the API of the system call is different, the idea is the same -notice, you're not encouraged to use module `_thread` directly, use `threading` instead, the following code use `_thread` for illustration +Notice, you're not encouraged to use module `_thread` directly. Use `threading` instead. The following code uses `_thread` for illustration # example @@ -61,7 +61,7 @@ module **threading** is a wrapper for the built-in **_thread** module, **threadi # start_new_thread -**PyEval_InitThreads** will create [gil](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil.md) if it's not created before, thread scheduling strategy has changed after python3.2, there's no need to give another thread a chance to run periodly when there's only one python intepreter thread +**PyEval_InitThreads** will create [GIL](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil.md) if it's not created before. Thread scheduling strategy has changed after Python 3.2. There's no need to give another thread a chance to run periodically when there's only one Python interpreter thread ```c /* cpython/Modules/_threadmodule.c */ @@ -108,7 +108,7 @@ thread_PyThread_start_new_thread(PyObject *self, PyObject *fargs) # allocate_lock -this is the normal lock object +This is the normal lock object ```python3 >>> r = _thread.allocate_lock() @@ -125,7 +125,7 @@ True ![lock_object](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/thread/lock_object.png) -in the posix system, the defination and allocation of **lock_lock** +In the POSIX system, the definition and allocation of **lock_lock** ```c /* cpython/Include/pythread.h */ @@ -169,14 +169,14 @@ PyThread_allocate_lock(void) ``` -from the above code we can know that CPython perfer implement the field in **lock_lock** as [posix semaphores](http://www.csc.villanova.edu/~mdamian/threads/posixsem.html) to [posix mutex](http://www.skrenta.com/rt/man/pthread_mutex_init.3.html), below examples use semaphores for illustration +From the above code we can see that CPython prefers to implement the field in **lock_lock** as [POSIX semaphores](http://www.csc.villanova.edu/~mdamian/threads/posixsem.html) over [POSIX mutex](http://www.skrenta.com/rt/man/pthread_mutex_init.3.html). The examples below use semaphores for illustration ```python3 >>> r.acquire_lock() ``` -if you try to acquire the lock, several posix system call will be invoked according to the timeout value +If you try to acquire the lock, several POSIX system calls will be invoked according to the timeout value ```python3 /* cpython/Python/thread_pthread.h */ @@ -204,13 +204,13 @@ while (1) { ``` -if you acquire the lock successfully, field locked will become 1 +If you acquire the lock successfully, the locked field will become 1 ![lock_object_locked](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/thread/lock_object_locked.png) # allocate_rlock -**rlock** is the abbreviation of recursive lock, as [wikipedia](https://en.wikipedia.org/wiki/Reentrant_mutex) says +**RLock** is the abbreviation of recursive lock. As [Wikipedia](https://en.wikipedia.org/wiki/Reentrant_mutex) says > In computer science, the reentrant mutex (recursive mutex, recursive lock) is a particular type of mutual exclusion (mutex) device that may be locked multiple times by the same process/thread, without causing a deadlock. @@ -222,9 +222,9 @@ if you acquire the lock successfully, field locked will become 1 ``` -the procedure of allocation of **lock_lock** in rlock object is the same as the above example +The procedure of allocation of **lock_lock** in the RLock object is the same as the above example -before acquire the lock, **rlock_owner** and **rlock_count** are all set to 0 +Before acquiring the lock, **rlock_owner** and **rlock_count** are both set to 0 ```python3 >>> r.acquire() @@ -240,9 +240,9 @@ before acquire the lock, **rlock_owner** and **rlock_count** are all set to 0 ![rlock_object_acquire2](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/thread/rlock_object_acquire2.png) -we found that **rlock_owner** is the current thread's ident, and **rlock_count** is a counter of how many times the lock is currently acquired +We found that **rlock_owner** is the current thread's ident, and **rlock_count** is a counter of how many times the lock is currently acquired -the acquire procedure is very clear +The acquire procedure is very clear ```c /* cpython/Modules/_threadmodule.c */ @@ -286,7 +286,7 @@ rlock_acquire(rlockobject *self, PyObject *args, PyObject *kwds) ``` -the release procedure +The release procedure ```c /* cpython/Modules/_threadmodule.c */ @@ -327,7 +327,7 @@ thread_PyThread_exit_thread(PyObject *self, PyObject *Py_UNUSED(ignored)) # stack_size -you can set the stack size to 100kb by calling +You can set the stack size to 100KB by calling ```python3 >>> _thread.stack_size(102400) # threading.stack_size(102400) @@ -379,4 +379,4 @@ _pythread_pthread_set_stacksize(size_t size) # thread local -the implementation of thread local storage is a per thread [dict](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict.md) object stored inside the **PyThreadState** structure \ No newline at end of file +The implementation of thread local storage is a per-thread [dict](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict.md) object stored inside the **PyThreadState** structure diff --git a/Media/Python-logo-notext.svg b/Media/Python-logo-notext.svg new file mode 100644 index 0000000..0326792 --- /dev/null +++ b/Media/Python-logo-notext.svg @@ -0,0 +1,48 @@ + + diff --git a/Modules/asyncio/fileio.md b/Modules/asyncio/fileio.md new file mode 100644 index 0000000..ebbbf47 --- /dev/null +++ b/Modules/asyncio/fileio.md @@ -0,0 +1,14 @@ +# asyncio + +# contents + +* [related file](#related-file) +* [memory layout](#memory-layout) +* [read more](#read-more) + +# related file +# memory layout + +# read more + +[Kavya Joshi - A tale of concurrency through creativity in Python: a deep dive into how gevent works.](https://www.youtube.com/watch?v=GunMToxbE0E) \ No newline at end of file diff --git a/Modules/io/fileio/fileio.md b/Modules/io/fileio/fileio.md index 7f74791..be680f3 100644 --- a/Modules/io/fileio/fileio.md +++ b/Modules/io/fileio/fileio.md @@ -43,7 +43,7 @@ For those who need the detail of python dict object, please refer to my previous ## close -After call the close method, the **fd** becomes -1, and one more key **__IOBase_closed** inserted to **dict** field +After calling the close method, the **fd** becomes -1, and one more key **__IOBase_closed** is inserted into the **dict** field ```python3 >>> f.close() @@ -67,7 +67,7 @@ the **fd** and **dict** object are all reused, and **writable**, **appending**, ## fd -let's pass an integer to parameter name +Let's pass an integer to the parameter name ```python3 >>> f = open("../../Desktop/2.txt", "rb") diff --git a/Modules/pickle/bytes.png b/Modules/pickle/bytes.png new file mode 100644 index 0000000..1dc0f28 Binary files /dev/null and b/Modules/pickle/bytes.png differ diff --git a/Modules/pickle/float.png b/Modules/pickle/float.png new file mode 100644 index 0000000..2ff482f Binary files /dev/null and b/Modules/pickle/float.png differ diff --git a/Modules/pickle/int.png b/Modules/pickle/int.png new file mode 100644 index 0000000..e922852 Binary files /dev/null and b/Modules/pickle/int.png differ diff --git a/Modules/pickle/int2.png b/Modules/pickle/int2.png new file mode 100644 index 0000000..8737e8f Binary files /dev/null and b/Modules/pickle/int2.png differ diff --git a/Modules/pickle/list1.png b/Modules/pickle/list1.png new file mode 100644 index 0000000..2f451b5 Binary files /dev/null and b/Modules/pickle/list1.png differ diff --git a/Modules/pickle/object1.png b/Modules/pickle/object1.png new file mode 100644 index 0000000..16959d4 Binary files /dev/null and b/Modules/pickle/object1.png differ diff --git a/Modules/pickle/pickle.md b/Modules/pickle/pickle.md new file mode 100644 index 0000000..ceb8f92 --- /dev/null +++ b/Modules/pickle/pickle.md @@ -0,0 +1,220 @@ +# pickle + +# contents + +* [related file](#related-file) +* [introduction](#introduction) +* [implementation](#implementation) + * [None](#None) + * [bool](#bool) + * [int](#int) + * [float](#float) + * [bytes](#bytes) + * [str](#str) + * [tuple](#tuple) + * [list](#list) + * [type](#type) + * [object](#object) + +# related file + +* Lib/pickle.py +* Modules/_pickle.c +* Modules/clinic/_pickle.c.h + +# introduction + +We use the `pickle` module to serialize and deserialize our Python objects. There are several protocol versions for `pickle`. The current version is `4` + + The `pickle` module will use the faster `_pickle` implemented in `C`(`Modules/_pickle.c`) if possible, if the `_pickle` module not found, the `python` implemented `pickle`(`Lib/pickle.py`) will be used + +| Type | Implementation | +| :----------: | :------------: | +| None | save_none | +| bool | save_bool | +| int | save_long | +| float | save_float | +| bytes | save_bytes | +| str | save_str | +| tuple | save_tuple | +| list | save_list | +| dict | save_dict | +| set | save_set | +| frozenset | save_frozenset | +| FunctionType | save_global | +| | save_reduce | + +# implementation + +Whenever you call dump, some extra information will be added to the result + +The first byte is an identifier indicate that the following binary content is encoded in "pickle protocol" + +The second byte is the protocol version + +The final byte is a stop symbol indicate that it's the end of the binary content + +![pickle_head](pickle_head.png) + + + +## None + +```python3 +NONE = b'N' # push None + +def save_none(self, obj): + self.write(NONE) +``` + +The `data` is `N` here, with the aforementioned information added to it + +```python3 +>>> import pickle +>>> pickle.dumps(None) +b'\x80\x04N.' +``` + +## bool + +`bool` is similar to `None` + +```python3 +NEWTRUE = b'\x88' # push True +NEWFALSE = b'\x89' # push False + +def save_bool(self, obj): + if self.proto >= 2: + self.write(NEWTRUE if obj else NEWFALSE) +``` + +The `data` here is `b'\x88'(True)` and `b'\x89'(False)` + +```python3 +>>> import pickle +>>> pickle.dumps(True) +b'\x80\x04\x88.' +>>> pickle.dumps(False) +b'\x80\x04\x89.' +``` + +## int + +The integer will be saved in various formats according to its value + +![int](int.png) + +![int2](int2.png) + + + +## float + +The float is saved in [IEEE_754](https://en.wikipedia.org/wiki/IEEE_754-1985) standard + +![float](float.png) + +## bytes + +`bytes` object is saved directly as the `data` part below + +The `head` part varies according to the data size + +![bytes](bytes.png) + +## str + +`str` is similar to [bytes](#bytes), except that `str` is encoded in `utf-8` format before dump + +![str](str.png) + +## tuple + +`tuple` is more complicated than other basic type + +If the `tuple` is empty + +![tuple0](tuple0.png) + +Let's see an example + +```python3 +dumps(("a", "b", (2, ))) +b'\x80\x04\x95\x0f\x00\x00\x00\x00\x00\x00\x00\x8c\x01a\x94\x8c\x01b\x94K\x02\x85\x94\x87\x94.' +``` + +`\x80\x04` is pickle protocol and pickle version + +`\x95\x0f\x00\x00\x00\x00\x00\x00\x00` is frame symbol(`\x95`) and frame size(8 bytes) in little endian + +`.` in last byte is the `STOP` symbol + +Besides are the data + + ![tuple1](tuple1.png) + +I found that dumps does not support self-referencing tuples(how to [Build Self-Referencing Tuples](https://stackoverflow.com/questions/11873448/building-self-referencing-tuples)) + +## list + +Let's see an example again + +```python3 +dumps(["a", "b", (2, )]) +b'\x80\x04\x95\x11\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x01a\x94\x8c\x01b\x94K\x02\x85\x94e.' +``` + +The first several bytes are pickle protocol, pickle version and frame size + +The last byte is `STOP` symbol + +The data can be described as (`]\x94(\x8c\x01a\x94\x8c\x01b\x94K\x02\x85\x94e`) + +`list` will be dumped batch by batch(default batch size `1000`) + +![list1](./list1.png) + +`dict` and `set` are similar to `list` and `tuple`, beginning and ending with a `type` symbol to indicate the type, and iterating through each object and recursively calling `dump` for each object + +## type + +If what's to be saved is a type + +```python3 +class A(object): + a = "a" + b = "b" + + def run(self): + print(self.a, self.b) + +pickle.dumps(A) +b'\x80\x04\x95\x12\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94.' +``` + + The data part is `\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94` + +![type1](./type1.png) + +`dumps(A)` saves the `module_name` (`__main__`) and the `object` name (`A`) in [str](#str) format + +## object + +If what's to be saved is an instance + +```python3 +a = A() +pickle.dumps(a) +b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94.' +``` + +The data part is `\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94` + +The only difference is that there is some extra information appended after the previous dumped result + +A `TUPLE` indicate the `args` needed for instance call, in the current case `a = A()` the args is empty, so it's an `EMPTY_TUPLE` + +A `NEWOBJ` symbol indicate that it needs to call `cls.__new__(cls, *args)` after load the dumped result + +![object1](./object1.png) + diff --git a/Modules/pickle/pickle_cn.md b/Modules/pickle/pickle_cn.md new file mode 100644 index 0000000..12b31b0 --- /dev/null +++ b/Modules/pickle/pickle_cn.md @@ -0,0 +1,220 @@ +# pickle + +# 目录 + +* [相关位置文件](#相关位置文件) +* [简介](#简介) +* [实现](#实现) + * [None](#None) + * [bool](#bool) + * [int](#int) + * [float](#float) + * [bytes](#bytes) + * [str](#str) + * [tuple](#tuple) + * [list](#list) + * [type](#type) + * [object](#object) + +# 相关位置文件 + +* Lib/pickle.py +* Modules/_pickle.c +* Modules/clinic/_pickle.c.h + +# 简介 + +我们用 `pickle` 来对 `python ` 中的对象进行序列化和反序列化, `pickle` 总共有好几个版本, 当前的版本号是 4 + + `pickle` 模块会在可能的情况运行 `C(Modules/_pickle.c)` 实现的代码, 这样效率更高, 但是如果无法加载, 还有一个纯 `python` 实现的 `pickle`(`Lib/pickle.py`) 作为备选 + +| 类型 | 实现 | +| :----------: | :------------: | +| None | save_none | +| bool | save_bool | +| int | save_long | +| float | save_float | +| bytes | save_bytes | +| str | save_str | +| tuple | save_tuple | +| list | save_list | +| dict | save_dict | +| set | save_set | +| frozenset | save_frozenset | +| FunctionType | save_global | +| | save_reduce | + +# 实现 + +每当你调用 `dump` 时, 结果数据除了对象本身, 还会包含一些额外的信息 + +第一个字节是一个标识符, 表示接下来的内容都是用 `pickle` 序列化的内容 + +第二个字节表示 `pickle` 版本号 + +最后一个字节是一个结束标识符, 表示看到这个字符时, 当前这一段 `pickle` 序s列化的结构信息已经结束了 + +![pickle_head](pickle_head.png) + + + +## None + +```python3 +NONE = b'N' # push None + +def save_none(self, obj): + self.write(NONE) +``` + +这里的 `data` 是 `N` , 除了 `N` 以外, 前面提到的额外的信息分别加到了头部和尾部 + +```python3 +>>> import pickle +>>> pickle.dumps(None) +b'\x80\x04N.' +``` + +## bool + +`bool` 和 `None` 表示方式类似 + +```python3 +NEWTRUE = b'\x88' # push True +NEWFALSE = b'\x89' # push False + +def save_bool(self, obj): + if self.proto >= 2: + self.write(NEWTRUE if obj else NEWFALSE) +``` + +这里的 `data` 是 `b'\x88'(True)` 和 `b'\x89'(False)` + +```python3 +>>> import pickle +>>> pickle.dumps(True) +b'\x80\x04\x88.' +>>> pickle.dumps(False) +b'\x80\x04\x89.' +``` + +## int + +整数类型会根据值的范围不同, 以好几种格式保存 + +![int](int.png) + +![int2](int2.png) + + + +## float + +浮点数以 [IEEE_754](https://en.wikipedia.org/wiki/IEEE_754-1985) 标准的格式存储 + +![float](float.png) + +## bytes + +`bytes` 类型表示的就是二进制内容了, 就直接按下图所示存储在 `data` 部分里 + +根据这个 `bytes` 的大小范围不同, 头部信息会有好几种存储格式 + +![bytes](bytes.png) + +## str + +`str` 和 [bytes](#bytes) 类似, 不同点是 `str` 会在序列化之前, 以 `utf-8` 方式进行编码, 之后再存储 + +![str](str.png) + +## tuple + +`tuple` 比其他几种基础类型稍微复杂一点 + +如果 `tuple` 是空的 + +![tuple0](tuple0.png) + +我们再来看一个例子 + +```python3 +dumps(("a", "b", (2, ))) +b'\x80\x04\x95\x0f\x00\x00\x00\x00\x00\x00\x00\x8c\x01a\x94\x8c\x01b\x94K\x02\x85\x94\x87\x94.' +``` + +`\x80\x04` 是 `pickle` 协议标识符和版本号 + +`\x95\x0f\x00\x00\x00\x00\x00\x00\x00` 是 frame 标志(`\x95`) 以为 frame 的大小(8 字节) (以小端存储), 这里以 frame 作为基本单位进行分块写入 + +最后一个字节的 `.` 表示 `STOP` 标志 + +其余的部分则为对应的元素数据 + + ![tuple1](tuple1.png) + +这里我还发现了 dumps 方法不支持包含了自己/有引用循环的元组(如何构造这种元组参考 [Build Self-Referencing Tuples](https://stackoverflow.com/questions/11873448/building-self-referencing-tuples)) + +## list + +我们再来看一个例子 + +```python3 +dumps(["a", "b", (2, )]) +b'\x80\x04\x95\x11\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x01a\x94\x8c\x01b\x94K\x02\x85\x94e.' +``` + +前几个字节是`pickle` 协议标识符和版本号, 还有 frame 大小 + +最后一个字节是 `STOP` 标志 + +中间的数据部分则是 `]\x94(\x8c\x01a\x94\x8c\x01b\x94K\x02\x85\x94e` + +`list` 会分批进行序列化(默认批大小为1000) + +![list1](./list1.png) + +`dict` 和 `set` 和 `list` 还有 `tuple ` 相似, 头尾分别用对应的标志表示类型, 之后遍历对象对于每个对象再递归调用 `save` 方法进行序列化存储 + +## type + +如果序列化的是一个自定义类型 + +```python3 +class A(object): + a = "a" + b = "b" + + def run(self): + print(self.a, self.b) + +pickle.dumps(A) +b'\x80\x04\x95\x12\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94.' +``` + +data 部分是 `\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94` + +![type1](./type1.png) + +`dumps(A)` 仅序列化了 `module_name` (`__main__`) 和具体的对象的名称(`A`) (以 [str](#str) 格式序列化) + +## object + +如果序列化的是自定义类型的实例呢 + +```python3 +a = A() +pickle.dumps(a) +b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94.' +``` + +data 部分是 `\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94` + +和上面唯一的不同点是在上面的基础上, 尾部又增加了一些额外的信息 + +一个 `tuple` 对象, 表示实例化的时候需要的参数(`args`), 在这个示例里, `a = A()` 实例化需要的参数为空, 所以就是 `EMPTY_TUPLE` + +一个 `NEWOBJ` 标识符, 表示在加载到这里的时候, 需要调用 `cls.__new__(cls, *args)` 进行反序列化操作 + +![object1](./object1.png) + diff --git a/Modules/pickle/pickle_head.png b/Modules/pickle/pickle_head.png new file mode 100644 index 0000000..8ac37c8 Binary files /dev/null and b/Modules/pickle/pickle_head.png differ diff --git a/Modules/pickle/str.png b/Modules/pickle/str.png new file mode 100644 index 0000000..ff8cc2e Binary files /dev/null and b/Modules/pickle/str.png differ diff --git a/Modules/pickle/tuple0.png b/Modules/pickle/tuple0.png new file mode 100644 index 0000000..04aaa82 Binary files /dev/null and b/Modules/pickle/tuple0.png differ diff --git a/Modules/pickle/tuple1.png b/Modules/pickle/tuple1.png new file mode 100644 index 0000000..7df34fd Binary files /dev/null and b/Modules/pickle/tuple1.png differ diff --git a/Modules/pickle/type1.png b/Modules/pickle/type1.png new file mode 100644 index 0000000..57ac340 Binary files /dev/null and b/Modules/pickle/type1.png differ diff --git a/Modules/re/re.md b/Modules/re/re.md index 22d7076..ce2be23 100644 --- a/Modules/re/re.md +++ b/Modules/re/re.md @@ -25,7 +25,7 @@ # how regex work -what happened when you execute the following code ? +What happens when you execute the following code? ```python3 import re @@ -33,21 +33,21 @@ re.search("abc\d+abc", "xxabc123abcd") # re.DEBUG) ``` -the call stack +The call stack ![call_stack](https://github.com/zpoint/CPython-Internals/blob/master/Modules/re/call_stack.png) -the overview of different phases of **re.search** +The overview of different phases of **re.search** ![overview](https://github.com/zpoint/CPython-Internals/blob/master/Modules/re/overview.png) -let's see what's going on in each step +Let's see what's going on in each step ## parse -the core code in **parse** phase is in **cpython/Lib/sre_parse.py** +The core code in the **parse** phase is in **cpython/Lib/sre_parse.py** -there is a class named **Tokenizer** to split your regex text into tokens, and **_parse** function is in charge of binding the token with the **sre_constants** code +There is a class named **Tokenizer** to split your regex text into tokens, and the **_parse** function is in charge of binding the token with the **sre_constants** code ```python3 # pseudo code @@ -60,7 +60,7 @@ while True: ``` -let's run the parse phase with regex pattern `abc\d+abc` +Let's run the parse phase with regex pattern `abc\d+abc` ```python3 next_token: a @@ -108,14 +108,14 @@ parse result: [(LITERAL, 97), (LITERAL, 98), (LITERAL, 99), (MAX_REPEAT, (1, MAX ## compile -this is the input of the **compile** phase +This is the input of the **compile** phase ```python3 [(LITERAL, 97), (LITERAL, 98), (LITERAL, 99), (MAX_REPEAT, (1, MAXREPEAT, [(IN, [(CATEGORY, CATEGORY_DIGIT)])])), (LITERAL, 97), (LITERAL, 98), (LITERAL, 99)] ``` -the compile function in **sre_compile.py** +The compile function in **sre_compile.py** ```python3 def _code(p, flags): @@ -135,9 +135,9 @@ def _code(p, flags): ``` -the **info block** has the following structure `[INFO, length_of_prefix_code, mask_indicate_whether_has_prefix, min_length_for_the_pattern, max_length_for_the_pattern, prefix1, prefix2..., prefixn, overlap_table_for_prefix]` +The **info block** has the following structure `[INFO, length_of_prefix_code, mask_indicate_whether_has_prefix, min_length_for_the_pattern, max_length_for_the_pattern, prefix1, prefix2..., prefixn, overlap_table_for_prefix]` -after the **_compile_info** function, code becomes +After the **_compile_info** function, code becomes ```python3 code == [INFO, 12, 1, 7, MAXREPEAT, 3, 3, 97, 98, 99, 0, 0, 0] @@ -160,7 +160,7 @@ the following 97, 98, 99 is the prefix `"abc"` and the following 0, 0, 0 here is generated by the **_generate_overlap_table** -after compile the **info block**, the **pattern** will be compiled in `_compile(code, p.data, flags)` +After compiling the **info block**, the **pattern** will be compiled in `_compile(code, p.data, flags)` ```python3 for op, av in pattern: @@ -181,7 +181,7 @@ op: MAX_REPEAT av: (1, MAXREPEAT, [(IN, [(CATEGORY, CATEGORY_DIGIT)])]) ``` -the handling process of MAX_REPEAT is a little bit different from others, the code snippet handling this situation is shown +The handling process of MAX_REPEAT is a little bit different from others. The code snippet handling this situation is shown ```python3 if op is MAX_REPEAT: @@ -199,14 +199,14 @@ code[skip] = _len(code) - skip # change the length ``` -the **code** object in position 1 +The **code** object in position 1 ```python3 code: [LITERAL, 97, LITERAL, 98, LITERAL, 99, REPEAT_ONE, 0, 1, MAXREPEAT] ``` -after the recursive call, **code** object in position 2, the length now becomes 9 +After the recursive call, **code** object in position 2, the length now becomes 9 ```python3 op: IN av: [(CATEGORY, CATEGORY_DIGIT)] @@ -223,7 +223,7 @@ code: [LITERAL, 97, LITERAL, 98, LITERAL, 99, REPEAT_ONE, 9, 1, MAXREPEAT, IN, 4 ``` -combine the **_compile_info** and **_compile** together, the code to the next phase is +Combining the **_compile_info** and **_compile** together, the code to the next phase is ```python3 [INFO, 12, 1, 7, MAXREPEAT, 3, 3, 97, 98, 99, 0, 0, 0, LITERAL, 97, LITERAL, 98, LITERAL, 99, REPEAT_ONE, 9, 1, MAXREPEAT, IN, 4, CATEGORY, CATEGORY_UNI_DIGIT, FAILURE, SUCCESS, LITERAL, 97, LITERAL, 98, LITERAL, 99, SUCCESS] @@ -232,11 +232,11 @@ combine the **_compile_info** and **_compile** together, the code to the next ph ## match -the match phase is written in c +The match phase is written in C -it's a big for loop, act as a state machine, every complied opcode pass down here will be passed to this loop, the loop will exam the pattern according to the opcode, if mismatch in any state, go to fail +It's a big for loop, acting as a state machine. Every compiled opcode passed down here will be passed to this loop. The loop will examine the pattern according to the opcode. If there's a mismatch in any state, go to fail -if success, go on, there will be another article talking about the match phase(if I have time) +If success, go on. There will be another article talking about the match phase (if I have time) ```c for (;;) { @@ -254,16 +254,16 @@ for (;;) { ## sre_ucs1_search -follow the call stack to here +Follow the call stack to here ```python3 status = sre_search(&state, PatternObject_GetCode(self)); ``` -the defination of **sre_search** is +The definition of **sre_search** is -the search will delegate to different function according to the real size of the **unicode** object, for detail of memory usage of **unicode** object, please refer to [how unicode implemented](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/str.md) +The search will delegate to different functions according to the real size of the **unicode** object. For details of memory usage of the **unicode** object, please refer to [how unicode implemented](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/str.md) ```python3 LOCAL(Py_ssize_t) @@ -279,7 +279,7 @@ sre_search(SRE_STATE* state, SRE_CODE* pattern) ``` -if we search for **sre_ucs1_search** in the folder, we can't find anything +If we search for **sre_ucs1_search** in the folder, we can't find anything ```python3 find . -name '*' -exec grep -nHr "sre_ucs1_search" {} \; @@ -288,7 +288,7 @@ Binary file ./libpython3.8m.a matches ``` -the trick here is for reusing the common part in **sre_lib.h** with different definitions +The trick here is for reusing the common part in **sre_lib.h** with different definitions ```c /* generate 8-bit version */ @@ -314,7 +314,7 @@ the trick here is for reusing the common part in **sre_lib.h** with different de ``` -we can find the defination of **search** in **sre_lib.h** +We can find the definition of **search** in **sre_lib.h** ```python3 LOCAL(Py_ssize_t) @@ -322,7 +322,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern) ``` -which will be expanded to three different form +which will be expanded to three different forms ```python3 inline Py_ssize_t sre_ucs1_search(SRE_STATE* state, SRE_CODE* pattern) @@ -331,13 +331,13 @@ inline Py_ssize_t sre_ucs4_search(SRE_STATE* state, SRE_CODE* pattern) ``` -when I go inside the expanded **sre_ucs1_search**, I found the [match](#match) phase +When I go inside the expanded **sre_ucs1_search**, I found the [match](#match) phase # fifo cache -when you called the **re.compile**, there will be a FIFO cache in the python level +When you call **re.compile**, there will be a FIFO cache at the Python level -**_cache** is of type dictionary, and because type dict is now ordered([dict implementation](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict.md#delete-in-entries)), it act as a fifo queue +**_cache** is of type dictionary, and because dict type is now ordered ([dict implementation](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict.md#delete-in-entries)), it acts as a FIFO queue ```python3 >>> r1 = re.compile("a\d+b") @@ -355,7 +355,7 @@ when you called the **re.compile**, there will be a FIFO cache in the python lev ``` -the cache mechanism is written in python +The cache mechanism is written in Python ```python3 _cache = {} # ordered! diff --git a/README.md b/README.md index 3af2152..9d3e847 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,22 @@ # Cpython Internals -* [简体中文](https://github.com/zpoint/CPython-Internals/blob/master/README_CN.md) -* [한국어](https://github.com/zpoint/CPython-Internals/blob/master/README_KR.md) -* **Watch** this repo if you need to be notified when there's update +![cpython logo](Media/Python-logo-notext.svg) -This repository is my notes/blog for [cpython](https://github.com/python/cpython) source code +* [简体中文](README_CN.md) +* [한국어](README_KR.md) +* **Watch** this repo if you need to be notified when there's an update -Trying to illustrate every detail of cpython implementation +This repository contains my notes/blog for [cpython](https://github.com/python/cpython) source code + +It attempts to illustrate every detail of CPython implementation ```shell script # based on version 3.8.0a0 cd cpython git reset --hard ab54b9a130c88f708077c2ef6c4963b632c132b3 - ``` -The following contents are suitable for those who have python programming experience and interested in internal of python interpreter, for those who needs beginner or advanced material please refer to [awesome-python-books](https://github.com/Junnplus/awesome-python-books) +The following content is suitable for those who have Python programming experience and are interested in the internals of the Python interpreter. For those who need beginner or advanced material, please refer to [awesome-python-books](https://github.com/Junnplus/awesome-python-books) # Table of Contents @@ -25,68 +26,74 @@ The following contents are suitable for those who have python programming experi * [Interpreter](#Interpreter) * [Extension](#Extension) * [Learning material](#Learning-material) +* [Contribution](#Contribution) +* [License](#License) # Objects - - [x] [dict](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict.md) - - [x] [long/int](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/long.md) - - [x] [unicode/str](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/str.md) - - [x] [set](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/set.md) - - [x] [list(timsort)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/list.md) - - [x] [tuple](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md) - - [x] [bytes](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytes/bytes.md) - - [x] [bytearray(buffer protocol)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytearray/bytearray.md) - - [x] [float](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/float.md) - - [x] [func(user-defined method)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/func/func.md) - - [x] [method(builtin method)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/method/method.md) - - [x] [iter](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/iter/iter.md) - - [x] [gen(generator/coroutine/async generator)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/gen.md) - - [x] [class(bound method/classmethod/staticmethod)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/class.md) - - [x] [complex](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/complex/complex.md) - - [x] [enum](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/enum/enum.md) - - [x] [type(mro/metaclass/creation of class/instance)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/type/type.md) + - [x] [dict](BasicObject/dict/dict.md) + - [x] [long/int](BasicObject/long/long.md) + - [x] [unicode/str](BasicObject/str/str.md) + - [x] [set](BasicObject/set/set.md) + - [x] [list(timsort)](BasicObject/list/list.md) + - [x] [tuple](BasicObject/tuple/tuple.md) + - [x] [bytes](BasicObject/bytes/bytes.md) + - [x] [bytearray(buffer protocol)](BasicObject/bytearray/bytearray.md) + - [x] [float](BasicObject/float/float.md) + - [x] [func(user-defined method)](BasicObject/func/func.md) + - [x] [method(builtin method)](BasicObject/method/method.md) + - [x] [iter](BasicObject/iter/iter.md) + - [x] [gen(generator/coroutine/async generator)](BasicObject/gen/gen.md) + - [x] [class(bound method/classmethod/staticmethod)](BasicObject/class/class.md) + - [x] [complex](BasicObject/complex/complex.md) + - [x] [enum](BasicObject/enum/enum.md) + - [x] [type(mro/metaclass/creation of class/instance)](BasicObject/type/type.md) # Modules - [ ] io - - [x] [fileio](https://github.com/zpoint/CPython-Internals/blob/master/Modules/io/fileio/fileio.md) - - [ ] [pickle](https://github.com/zpoint/CPython-Internals/blob/master/Modules/pickle/pickle.md) + - [x] [fileio](Modules/io/fileio/fileio.md) + - [x] [pickle](Modules/pickle/pickle.md) # Lib - - [x] [re(regex)](https://github.com/zpoint/CPython-Internals/blob/master/Modules/re/re.md) + - [x] [re(regex)](Modules/re/re.md) - [ ] asyncio # Interpreter - - [x] [gil(Global Interpreter Lock)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil.md) - - [x] [gc(Garbage Collection)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/gc.md) - - [x] [memory management](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management.md) - - [x] [descr(how does attribute access work/`__get__`/`__getattribute__`/`__getattr__`)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/descr/descr.md) - - [x] [exception(exception handling)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/exception.md) - - [x] [module(how does import work)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/module/module.md) - - [x] [frame](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame.md) - - [x] [code](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/code/code.md) - - [x] [slots/`__slots__`(how does attribute initialized in the creation of class/instance)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/slot.md) - - [x] [thread](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/thread/thread.md) - - [x] [PyObject(overview)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/pyobject.md) + - [x] [gil(Global Interpreter Lock)](Interpreter/gil/gil.md) + - [x] [gc(Garbage Collection)](Interpreter/gc/gc.md) + - [x] [memory management](Interpreter/memory_management/memory_management.md) + - [x] [descr(how does attribute access work/`__get__`/`__getattribute__`/`__getattr__`)](Interpreter/descr/descr.md) + - [x] [exception(exception handling)](Interpreter/exception/exception.md) + - [x] [module(how does import work)](Interpreter/module/module.md) + - [x] [frame](Interpreter/frame/frame.md) + - [x] [code](Interpreter/code/code.md) + - [x] [slots/`__slots__`(how does attribute initialized in the creation of class/instance)](Interpreter/slot/slot.md) + - [x] [thread](Interpreter/thread/thread.md) + - [x] [PyObject(overview)](Interpreter/pyobject/pyobject.md) # Extension - - [x] [C API(profile python code and write pure C extension)](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/c.md) + - [x] [C API(profile python code and write pure C extension)](Extension/C/c.md) - [ ] Cython(C extension) - [x] [Boost C++ libaries (C\+\+ extension)](https://github.com/zpoint/Boost-Python-Examples) + - [ ] [C++ extension](Extension/CPP/cpp.md) + - [x] integrate with NumPy + - [x] bypass the GIL # Grammar -I will come back to this part when I finish reading [< < Compilers > >](https://www.amazon.com/Compilers-Principles-Techniques-Tools-2nd/dp/0321486811) and [< < SICP > >](https://www.amazon.com/Structure-Interpretation-Computer-Programs-Engineering/dp/0262510871) and have a better understanding of this kind of stuffs - -In the meantime, my routine work will have a higher priority, so you may need months to see updates in this part + - [x] [Compile Phase](Interpreter/compile/compile.md) + - [x] [Grammar/MetaGrammar to DFA](Interpreter/compile/compile.md) + - [x] [CST to AST](Interpreter/compile2/compile.md) + - [x] [AST to python byte code](Interpreter/compile3/compile.md) # Learning material -I will only recommend what I've read +I only recommend materials I've read * [CPython internals - Interpreter and source code overview(youtube video)](https://www.youtube.com/watch?v=LhadeL7_EIU&list=PLzV58Zm8FuBL6OAv1Yu6AwXZrnsFbbR0S) * [< < Inside The Python Virtual Machine > >](https://leanpub.com/insidethepythonvirtualmachine) @@ -94,4 +101,24 @@ I will only recommend what I've read * [rushter(blog/eng)](https://rushter.com/) * [YET ANOTHER PYTHON INTERNALS BLOG(blog/eng)](https://pythoninternal.wordpress.com/) * [Junnplus(blog/cn)](https://github.com/Junnplus/blog/issues) -* [manjusaka(blog/cn)](https://manjusaka.itscoder.com/) \ No newline at end of file +* [manjusaka(blog/cn)](https://manjusaka.itscoder.com/) +* [aoik-Python's compiler series(blog/eng)](https://aoik.me/blog/posts/python-compiler-from-grammar-to-dfa) + +# Contribution + +All kinds of contributions are welcome + +* submit a pull request + * if you want to share any knowledge you know + * post a new article + * correct any technical mistakes + * correct English grammar + * translation + * anything else +* open an issue + * any suggestions + * any questions + * correct mistakes + * anything else + +# [License](https://creativecommons.org/licenses/by-nc-sa/4.0/) diff --git a/README_CN.md b/README_CN.md index e537814..bc2cc1e 100644 --- a/README_CN.md +++ b/README_CN.md @@ -1,7 +1,7 @@ # Cpython Internals -* [English](https://github.com/zpoint/CPython-Internals/blob/master/README.md) -* [한국어](https://github.com/zpoint/CPython-Internals/blob/master/README_KR.md) +* [English](README.md) +* [한국어](README_KR.md) * 如果你需要接收更新通知, 点击右上角的 **Watch**, 当有文章更新时会在 issue 发布相关标题和链接 * 如果有任何链接无法打开请自行搭梯子O(∩_∩)O @@ -27,61 +27,71 @@ git reset --hard ab54b9a130c88f708077c2ef6c4963b632c132b3 * [解释器相关](#解释器相关) * [扩展](#扩展) * [学习资料](#学习资料) +* [参与贡献](#参与贡献) + +* [授权](#授权) # 基本对象 -- [x] [dict](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict_cn.md) -- [x] [long/int](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/long_cn.md) -- [x] [unicode/str](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/str_cn.md) -- [x] [set](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/set_cn.md) -- [x] [list(timsort)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/list_cn.md) -- [x] [tuple](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple_cn.md) -- [x] [bytes](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytes/bytes_cn.md) -- [x] [bytearray(buffer protocol)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytearray/bytearray_cn.md) -- [x] [float](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/float_cn.md) -- [x] [func(user-defined method)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/func/func_cn.md) -- [x] [method(builtin method)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/method/method_cn.md) -- [x] [iter](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/iter/iter_cn.md) -- [x] [gen(generator/coroutine/async generator)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/gen_cn.md) -- [x] [class(bound method/classmethod/staticmethod)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/class_cn.md) -- [x] [complex](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/complex/complex_cn.md) -- [x] [enum](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/enum/enum_cn.md) -- [x] [type(mro/metaclass/类/实例的创建过程)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/type/type_cn.md) +- [x] [dict](BasicObject/dict/dict_cn.md) +- [x] [long/int](BasicObject/long/long_cn.md) +- [x] [unicode/str](BasicObject/str/str_cn.md) +- [x] [set](BasicObject/set/set_cn.md) +- [x] [list(timsort)](BasicObject/list/list_cn.md) +- [x] [tuple](BasicObject/tuple/tuple_cn.md) +- [x] [bytes](BasicObject/bytes/bytes_cn.md) +- [x] [bytearray(buffer protocol)](BasicObject/bytearray/bytearray_cn.md) +- [x] [float](BasicObject/float/float_cn.md) +- [x] [func(user-defined method)](BasicObject/func/func_cn.md) +- [x] [method(builtin method)](BasicObject/method/method_cn.md) +- [x] [iter](BasicObject/iter/iter_cn.md) +- [x] [gen(generator/coroutine/async generator)](BasicObject/gen/gen_cn.md) +- [x] [class(bound method/classmethod/staticmethod)](BasicObject/class/class_cn.md) +- [x] [complex](BasicObject/complex/complex_cn.md) +- [x] [enum](BasicObject/enum/enum_cn.md) +- [x] [type(mro/metaclass/类/实例的创建过程)](BasicObject/type/type_cn.md) # 模块 - [ ] io - - [x] [fileio](https://github.com/zpoint/CPython-Internals/blob/master/Modules/io/fileio/fileio_cn.md) - - [ ] [pickle](https://github.com/zpoint/CPython-Internals/blob/master/Modules/pickle/pickle_cn.md) + - [x] [fileio](Modules/io/fileio/fileio_cn.md) + - [x] [pickle](Modules/pickle/pickle_cn.md) # 库 - - [x] [re(正则)](https://github.com/zpoint/CPython-Internals/blob/master/Modules/re/re_cn.md) + - [x] [re(正则)](Modules/re/re_cn.md) - [ ] asyncio # 解释器相关 - - [x] [gil(全局解释器锁)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil_cn.md) - - [x] [gc(垃圾回收机制)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/gc_cn.md) - - [x] [memory management(内存管理机制)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management_cn.md) - - [x] [descr(访问(类/实例)属性时发生了什么/`__get__`/`__getattribute__`/`__getattr__`)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/descr/descr_cn.md) - - [x] [exception(异常处理机制)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/exception_cn.md) - - [x] [module(import实现机制)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/module/module_cn.md) - - [x] [frame](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame_cn.md) - - [x] [code](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/code/code_cn.md) - - [x] [slots/`__slots__`(属性在类/实例创建时是如何初始化的)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/slot_cn.md) - - [x] [thread(线程)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/thread/thread_cn.md) - - [x] [PyObject(基础篇/概述)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/pyobject_cn.md) + - [x] [gil(全局解释器锁)](Interpreter/gil/gil_cn.md) + - [x] [gc(垃圾回收机制)](Interpreter/gc/gc_cn.md) + - [x] [memory management(内存管理机制)](Interpreter/memory_management/memory_management_cn.md) + - [x] [descr(访问(类/实例)属性时发生了什么/`__get__`/`__getattribute__`/`__getattr__`)](Interpreter/descr/descr_cn.md) + - [x] [exception(异常处理机制)](Interpreter/exception/exception_cn.md) + - [x] [module(import实现机制)](Interpreter/module/module_cn.md) + - [x] [frame](Interpreter/frame/frame_cn.md) + - [x] [code](Interpreter/code/code_cn.md) + - [x] [slots/`__slots__`(属性在类/实例创建时是如何初始化的)](Interpreter/slot/slot_cn.md) + - [x] [thread(线程)](Interpreter/thread/thread_cn.md) + - [x] [PyObject(基础篇/概述)](Interpreter/pyobject/pyobject_cn.md) # 扩展 - - [x] [C API(python 性能分析和 C 扩展)](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/c_cn.md) + - [x] [C API(python 性能分析和 C 扩展)](Extension/C/c_cn.md) - [ ] Cython(C extension) - [x] [Boost C++ libaries (C\+\+ extension)](https://github.com/zpoint/Boost-Python-Examples) + - [x] [C++ 扩展](Extension/CPP/cpp_cn.md) + - [x] 写 NumPy 扩展 + - [x] 绕过 GIL # 语法 -这一部分我计划在读完 [< < Compilers > >](https://www.amazon.com/Compilers-Principles-Techniques-Tools-2nd/dp/0321486811) 和 [< < SICP > >](https://www.amazon.com/Structure-Interpretation-Computer-Programs-Engineering/dp/0262510871) 之后对这类知识有更好的理解之后再来更新, 与此同时我还需要处理优先级更高的日常工作, 所以可能数月之后这部分才会有更新 +[编译阶段](Interpreter/compile/compile.md) + +* [x] [从语法/元语法到DFA](Interpreter/compile/compile_cn.md) +* [x] [从 CST 到 AST](Interpreter/compile2/compile_cn.md) +* [x] [从 AST 到 字节码](Interpreter/compile3/compile_cn.md) # 学习资料 @@ -95,3 +105,24 @@ git reset --hard ab54b9a130c88f708077c2ef6c4963b632c132b3 * [YET ANOTHER PYTHON INTERNALS BLOG(blog/eng)](https://pythoninternal.wordpress.com/) * [Junnplus(blog/中文)](https://github.com/Junnplus/blog/issues) * [manjusaka(blog/中文)](https://manjusaka.itscoder.com/) +* [aoik-Python's compiler series(blog/eng)](https://aoik.me/blog/posts/python-compiler-from-grammar-to-dfa) + +## 参与贡献 + +欢迎所有形式的贡献 + +* 提交一个 pull request + * 如果你想分享任何你知道的相关知识 + * 发布一篇文章 + * 更正技术性错误 + * 更正语法性错误 + * 翻译 + * 其他情况, 不限于上述 +* 提交一个 issue + * 任何建议 + * 任何问题 + * 更正错误 + * 其他情况, 不限于上述 + +# [授权](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.zh) + diff --git a/README_KR.md b/README_KR.md index 57d5569..b24e2cf 100644 --- a/README_KR.md +++ b/README_KR.md @@ -1,7 +1,9 @@ -# Cpython 내부 +# Cpython Internals! -* [English](https://github.com/zpoint/CPython-Internals/blob/master/README.md) -* [简体中文](https://github.com/zpoint/CPython-Internals/blob/master/README_CN.md) +![cpython logo](Media/Python-logo-notext.svg) + +* [English](README.md) +* [简体中文](README_CN.md) * 업데이트가 있을 때 알림을 받으려면, **Watch** 하세요. 이 저장소는 [cpython](https://github.com/python/cpython) 소스코드에 대한 나의 노트/블로그 입니다. @@ -24,66 +26,74 @@ git reset --hard ab54b9a130c88f708077c2ef6c4963b632c132b3 * [라이브러리(Lib)](#라이브러리(Lib)) * [인터프리터(Interpreter)](#인터프리터(Interpreter)) * [확장(Extension)](#확장(Extension)) -* [공부할 자료](#공부할-자료) +* [공부할 자료(Learning material)](#공부할-자료) +* [기여(Contribution)](#Contribution) +* [라이센스(License)](#License) + # 객체들(Objects) - - [x] [dict](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/dict/dict.md) - - [x] [long/int](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/long/long.md) - - [x] [unicode/str](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/str/str.md) - - [x] [set](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/set/set.md) - - [x] [list(timsort)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/list/list.md) - - [x] [tuple](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/tuple/tuple.md) - - [x] [bytes](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytes/bytes.md) - - [x] [bytearray(buffer protocol)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/bytearray/bytearray.md) - - [x] [float](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/float/float.md) - - [x] [func(user-defined method)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/func/func.md) - - [x] [method(builtin method)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/method/method.md) - - [x] [iter](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/iter/iter.md) - - [x] [gen(generator/coroutine/async generator)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/gen/gen.md) - - [x] [class(bound method/classmethod/staticmethod)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/class/class.md) - - [x] [complex](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/complex/complex.md) - - [x] [enum](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/enum/enum.md) - - [x] [type(mro/metaclass/creation of class/instance)](https://github.com/zpoint/CPython-Internals/blob/master/BasicObject/type/type.md) + - [x] [dict](BasicObject/dict/dict.md) + - [x] [long/int](BasicObject/long/long.md) + - [x] [unicode/str](BasicObject/str/str.md) + - [x] [set](BasicObject/set/set.md) + - [x] [list(timsort)](BasicObject/list/list.md) + - [x] [tuple](BasicObject/tuple/tuple.md) + - [x] [bytes](BasicObject/bytes/bytes.md) + - [x] [bytearray(buffer protocol)](BasicObject/bytearray/bytearray.md) + - [x] [float](BasicObject/float/float.md) + - [x] [func(user-defined method)](BasicObject/func/func.md) + - [x] [method(builtin method)](BasicObject/method/method.md) + - [x] [iter](BasicObject/iter/iter.md) + - [x] [gen(generator/coroutine/async generator)](BasicObject/gen/gen.md) + - [x] [class(bound method/classmethod/staticmethod)](BasicObject/class/class.md) + - [x] [complex](BasicObject/complex/complex.md) + - [x] [enum](BasicObject/enum/enum.md) + - [x] [type(mro/metaclass/creation of class/instance)](BasicObject/type/type.md) # 모듈들(Modules) - [ ] io - - [x] [fileio](https://github.com/zpoint/CPython-Internals/blob/master/Modules/io/fileio/fileio.md) - - [ ] [pickle](https://github.com/zpoint/CPython-Internals/blob/master/Modules/pickle/pickle.md) + - [x] [fileio](Modules/io/fileio/fileio.md) + - [ ] [pickle](Modules/pickle/pickle.md) # 라이브러리(Lib) - - [x] [re(regex)](https://github.com/zpoint/CPython-Internals/blob/master/Modules/re/re.md) + - [x] [re(regex)](Modules/re/re.md) - [ ] asyncio # 인터프리터(Interpreter) - - [x] [gil(Global Interpreter Lock)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gil/gil.md) - - [x] [gc(Garbage Collection)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/gc/gc.md) - - [x] [memory management](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/memory_management/memory_management.md) - - [x] [descr(how does attribute access work/`__get__`/`__getattribute__`/`__getattr__`)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/descr/descr.md) - - [x] [exception(exception handling)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/exception/exception.md) - - [x] [module(how does import work)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/module/module.md) - - [x] [frame](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/frame/frame.md) - - [x] [code](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/code/code.md) - - [x] [slots/`__slots__`(how does attribute initialized in the creation of class/instance)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/slot/slot.md) - - [x] [thread](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/thread/thread.md) - - [x] [PyObject(overview)](https://github.com/zpoint/CPython-Internals/blob/master/Interpreter/pyobject/pyobject.md) + - [x] [gil(Global Interpreter Lock)](Interpreter/gil/gil.md) + - [x] [gc(Garbage Collection)](Interpreter/gc/gc.md) + - [x] [memory management](Interpreter/memory_management/memory_management.md) + - [x] [descr(how does attribute access work/`__get__`/`__getattribute__`/`__getattr__`)](Interpreter/descr/descr.md) + - [x] [exception(exception handling)](Interpreter/exception/exception.md) + - [x] [module(how does import work)](Interpreter/module/module.md) + - [x] [frame](Interpreter/frame/frame.md) + - [x] [code](Interpreter/code/code.md) + - [x] [slots/`__slots__`(how does attribute initialized in the creation of class/instance)](Interpreter/slot/slot.md) + - [x] [thread](Interpreter/thread/thread.md) + - [x] [PyObject(overview)](Interpreter/pyobject/pyobject.md) # 확장(Extension) - - [x] [C API(profile python code and write pure C extension)](https://github.com/zpoint/CPython-Internals/blob/master/Extension/C/c.md) + - [x] [C API(profile python code and write pure C extension)](Extension/C/c.md) - [ ] Cython(C extension) - [x] [Boost C++ libaries (C\+\+ extension)](https://github.com/zpoint/Boost-Python-Examples) + - [ ] [C++ extension](Extension/CPP/cpp.md) + - [x] integrate with NumPy + - [x] bypass the GIL -# 문법(Grammar) -[< < Compilers > >](https://www.amazon.com/Compilers-Principles-Techniques-Tools-2nd/dp/0321486811) 와 [< < SICP > >](https://www.amazon.com/Structure-Interpretation-Computer-Programs-Engineering/dp/0262510871) 를 읽어 본 후 이런 종류의 것들에 대한 이해가 높아지면, 이 부분을 업데이트 할 것 입니다. +# 문법(Grammar) -그 동안, 내 일상적인 작업의 우선 순위가 더 높으므로, 이 부분의 업데이트를 보려면 몇 달이 필요할 수 있습니다. +- [x] [Compile Phase](Interpreter/compile/compile.md) + - [x] [Grammar/MetaGrammar to DFA](Interpreter/compile/compile.md) + - [x] [CST to AST](Interpreter/compile2/compile.md) + - [x] [AST to python byte code](Interpreter/compile3/compile.md) -# 공부할 자료 +# 공부할 자료(Learning material) 읽어본 자료들만 추천할 것 입니다. @@ -94,3 +104,23 @@ git reset --hard ab54b9a130c88f708077c2ef6c4963b632c132b3 * [YET ANOTHER PYTHON INTERNALS BLOG(blog/eng)](https://pythoninternal.wordpress.com/) * [Junnplus(blog/cn)](https://github.com/Junnplus/blog/issues) * [manjusaka(blog/cn)](https://manjusaka.itscoder.com/) +* [aoik-Python's compiler series(blog/eng)](https://aoik.me/blog/posts/python-compiler-from-grammar-to-dfa) + +# 기여(Contribution) + +모든 기여는 환영입니다! + +* pull request 제출하기 + * 당신이 공유하기를 원하는 지식 + * 새로운 내용 추가 + * 기술적 결함 교정 + * 영어 문법 교정 + * 번역 + * 그밖에 다른 무엇이든지 +* issue 열기 + * 제안 + * 질문 + * 실수 교정 + * 그밖에 다른 무엇이든지 + +# [라이센스(License)](https://creativecommons.org/licenses/by-nc-sa/4.0/) diff --git a/_config.yml b/_config.yml new file mode 100644 index 0000000..e879b3e --- /dev/null +++ b/_config.yml @@ -0,0 +1,5 @@ +title: CPython Internals +description: Dive into CPython internals, trying to illustrate every detail of CPython implementation +remote_theme: pages-themes/cayman@v0.2.0 +plugins: + - jekyll-remote-theme