1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
|
Metadata-Version: 2.1
Name: fast-query-parsers
Version: 1.0.3
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python
Classifier: Programming Language :: Rust
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development
Classifier: Typing :: Typed
License-File: LICENSE
Summary: Ultra-fast query string and url-encoded form-data parsers
Keywords: query,query string,qs,query parameters,rust,asgi,starlite,litestar,parser
Author: Na'aman Hirschfeld <nhirschfeld@gmail.com>
Author-email: Na'aman Hirschfeld <nhirschfeld@gmail.com>
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Source Code, https://github.com/starlite-api/fast-query-parsers
<!-- markdownlint-disable -->
<p align="center">
<img src="https://github.com/litestar-org/branding/blob/473f54621e55cde9acbb6fcab7fc03036173eb3d/assets/Branding%20-%20SVG%20-%20Transparent/Logo%20-%20Banner%20-%20Inline%20-%20Light.svg#gh-light-mode-only" alt="Litestar Logo - Light" width="100%" height="auto" />
<img src="https://github.com/litestar-org/branding/blob/473f54621e55cde9acbb6fcab7fc03036173eb3d/assets/Branding%20-%20SVG%20-%20Transparent/Logo%20-%20Banner%20-%20Inline%20-%20Dark.svg#gh-dark-mode-only" alt="Litestar Logo - Dark" width="100%" height="auto" />
</p>
<!-- markdownlint-restore -->
<div align="center">
<!-- prettier-ignore-start -->
| Project | | Status |
|-----------|:----|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| CI/CD | | [](https://github.com/litestar-org/fast-query-parsers/actions/workflows/publish.yaml) [](https://github.com/litestar-org/fast-query-parsers/actions/workflows/ci.yaml) |
| Package | | [](https://badge.fury.io/py/litestar)   |
| Community | | [](https://reddit.com/r/litestarapi) [](https://discord.gg/X3FJqy8d2j) [](https://matrix.to/#/#litestar:matrix.org) [](https://blog.litestar.dev) [](https://twitter.com/LitestarAPI) [](https://blog.litestar.dev) |
| Meta | | [](https://github.com/litestar-org/fast-query-parsers) [](https://spdx.org/licenses/) [](https://github.com/sponsors/litestar-org) [](https://github.com/astral-sh/ruff) [](https://github.com/psf/black) |
<!-- prettier-ignore-end -->
</div>
# Fast Query Parsers
This library includes ultra-fast Rust based query string and urlencoded parsers. These parsers are used
by [`Litestar`](https://github.com/litestar-org/litestar), but are developed separately - and can of course be used separately.
> [!IMPORTANT]\
> [**_Starlite has been renamed to Litestar_**](https://litestar.dev/about/organization.html#litestar-and-starlite)
</div>
## Installation
```shell
pip install fast-query-parsers
```
## Usage
The library exposes two function `parse_query_string` and `parse_url_encoded_dict`.
### `parse_query_string`
This function is used to parse a query string into a list of key/value tuples.
```python
from fast_query_parsers import parse_query_string
result = parse_query_string(b"value=1&value=2&type=dollar&country=US", "&")
# [("value", "1"), ("value", "2"), ("type", "dollar"), ("country", "US")]
```
The first argument to this function is a byte string that includes the query string to be parsed, the second argument is
the separator used.
#### Benchmarks
Query string parsing is more than x5 times faster than the standard library:
```text
stdlib parse_qsl parsing query string: Mean +- std dev: 2.86 us +- 0.03 us
.....................
parse_query_string parsing query string: Mean +- std dev: 916 ns +- 13 ns
.....................
stdlib parse_qsl parsing urlencoded query string: Mean +- std dev: 8.30 us +- 0.10 us
.....................
parse_query_string urlencoded query string: Mean +- std dev: 1.50 us +- 0.03 us
```
### `parse_url_encoded_dict`
This function is used to parse a url-encoded form data dictionary and parse it into the python equivalent of JSON types.
```python
from urllib.parse import urlencode
from fast_query_parsers import parse_url_encoded_dict
encoded = urlencode(
[
("value", "10"),
("value", "12"),
("veggies", '["tomato", "potato", "aubergine"]'),
("nested", '{"some_key": "some_value"}'),
("calories", "122.53"),
("healthy", "true"),
("polluting", "false"),
("json", "null"),
]
).encode()
result = parse_url_encoded_dict(encoded, parse_numbers=True)
# result == {
# "value": [10, 12],
# "veggies": ["tomato", "potato", "aubergine"],
# "nested": {"some_key": "some_value"},
# "calories": 122.53,
# "healthy": True,
# "polluting": False,
# "json": None,
# }
```
This function handles type conversions correctly - unlike the standard library function `parse_qs`. Additionally, it
does not nest all values inside lists.
Note: the second argument passed to `parse_url_encoded_dict` dictates whether numbers should be parsed. If `True`,
the value will be parsed into an int or float as appropriate, otherwise it will be kept as a string.
By default the value of this arg is `True`.
#### Benchmarks
Url Encoded parsing is more than x2 times faster than the standard library, without accounting for parsing of values:
```text
stdlib parse_qs parsing url-encoded values into dict: Mean +- std dev: 8.99 us +- 0.09 us
.....................
parse_url_encoded_dict parse url-encoded values into dict: Mean +- std dev: 3.77 us +- 0.08 us
```
To actually mimic the parsing done by `parse_url_encoded_dict` we will need a utility along these lines:
```python
from collections import defaultdict
from contextlib import suppress
from json import loads, JSONDecodeError
from typing import Any, DefaultDict, Dict, List
from urllib.parse import parse_qsl
def parse_url_encoded_form_data(encoded_data: bytes) -> Dict[str, Any]:
"""Parse an url encoded form data into dict of parsed values"""
decoded_dict: DefaultDict[str, List[Any]] = defaultdict(list)
for k, v in parse_qsl(encoded_data.decode(), keep_blank_values=True):
with suppress(JSONDecodeError):
v = loads(v) if isinstance(v, str) else v
decoded_dict[k].append(v)
return {k: v if len(v) > 1 else v[0] for k, v in decoded_dict.items()}
```
With the above, the benchmarks looks like so:
```text
python parse_url_encoded_form_data parsing url-encoded values into dict: Mean +- std dev: 19.7 us +- 0.1 us
.....................
parse_url_encoded_dict parsing url-encoded values into dict: Mean +- std dev: 3.69 us +- 0.03 us
```
## Contributing
All contributions are of course welcome!
### Repository Setup
1. Run `cargo install` to setup the rust dependencies and `poetry install` to setup the python dependencies.
2. Install the pre-commit hooks with `pre-commit install` (requires [pre-commit](https://pre-commit.com/)).
### Building
Run `poetry run maturin develop --release --strip` to install a release wheel (without debugging info). This wheel can be
used in tests and benchmarks.
### Benchmarking
There are basic benchmarks using pyperf in place. To run these execute `poetry run python benchrmarks.py`.
|
