Dotted
Sometimes you want to fetch data from a deeply nested data structure. Dotted notation helps you do that.
Let's say you have a dictionary containing a dictionary containing a list and you wish to fetch the ith value from that nested list.
>>> import dotted
>>> d = {'hi': {'there': [1, 2, 3]}}
>>> dotted.get(d, 'hi.there[1]')
2
API
Probably the easiest thing to do is pydoc the api layer.
$ pydoc dotted.api
Get
See grammar discussion below about things you can do get data via dotted.
>>> import dotted
>>> dotted.get({'a': {'b': {'c': {'d': 'nested'}}}}, 'a.b.c.d')
'nested'
Update
Update will mutate the object if it can. It always returns the changed object though. If it's not mutable, then get via the return.
>>> import dotted
>>> l = []
>>> t = ()
>>> dotted.update(l, '[0]', 'hello')
['hello']
>>> l
['hello']
>>> dotted.update(t, '[0]', 'hello')
('hello',)
>>> t
()
```
Update via pattern
You can update all fields that match pattern given by either a wildcard OR regex.
>>> import dotted
>>> d = {'a': 'hello', 'b': {'bye'}}
>>> dotted.update(d, '*', 'me')
{'a': 'me', 'b': 'me'}
Remove
You can remove a field or do so only if it matches value. For example,
>>> import dotted
>>> d = {'a': 'hello', 'b': 'bye'}
>>> dotted.remove(d, 'b')
{'a': 'hello'}
>>> dotted.remove(d, 'a', 'bye')
{'a': 'hello'}
Remove via pattern
Similar to update, all patterns that match will be removed. If you provide a value as well, only the matched patterns that also match the value will be removed.
Match
Use to match a dotted-style pattern to a field. Partial matching is on by default. You can match via wildcard OR via regex. Here's a regex example:
>>> import dotted
>>> dotted.match('/a.+/', 'abced.b')
'abced.b'
>>> dotted.match('/a.+/', 'abced.b', partial=False)
With the groups=True parameter, you'll see how it was matched:
>>> import dotted
>>> match('hello.*', 'hello.there.bye', groups=True)
('hello.there.bye', ('hello', 'there.bye'))
In the above example, hello matched to hello and * matched to there.bye (partial
matching is enabled by default).
Expand
You may wish to expand all fields that match a pattern in an object.
>>> import dotted
>>> d = {'hello': {'there': [1, 2, 3]}, 'bye': 7}
>>> dotted.expand(d, '*')
('hello', 'bye')
>>> dotted.expand(d, '*.*')
('hello.there',)
>>> dptted.expand(d, '*.*[*]')
('hello.there[0]', 'hello.there[1]', 'hello.there[2]')
>>> dotted.expand(d, '*.*[1:]')
('hello.there[1:]',)
Grammar
Dotted notation looks similar to python. Both dot fields and bracketed fields
use keys() and __getitem__() internally. A dot field expects to see a
dictionary-like object. A slot field is biased towards sequences (like lists, tuples,
and strs) but can act on dicts as well. Dotted also supports slicing notation as well as
transforms discussed below.
Key fields
A key field is expressed as a or part of a dotted expression, such as a.b. The
grammar parser is permissive for what can be in a key field. Pretty much any non-reserved
char will match. Note that key fields will only work on objects that have a keys()
method. Basically, they work with dictionary or dictionary-like objects.
>>> import dotted
>>> dotted.get({'a': {'b': 'hello'}}, 'a.b')
'hello'
If the key field starts with a space or -, you should either quote it OR you may use
a \ as the first char.
Bracketed fields
You may also use bracket notation, such as a[0] which does a __getitem__ at key 0.
The parser prefers numeric types over string types (if you wish to look up a non-numeric
field using brackets be sure to quote it). Bracketed fields will work with pretty much
any object that can be looked up via __getitem__.
>>> import dotted
>>> dotted.get({'a': ['first', 'second', 'third']}, 'a[0]')
'first'
>>> dotted.get({'a': {'b': 'hello'}}, 'a["b"]')
'first'
Numeric types
The parser will attempt to interpret a field numerically if it can, such as field.1
will interpret the 1 part numerically.
>>> import dotted
>>> dotted.get({'7': 'me', 7: 'you'}, '7')
'you'
Quoting
Sometimes you need to quote a field which you can do by just putting the field in quotes.
>>> import dotted
>>> dotted.get({'has . in it': 7}, '"has . in it"')
7
The numericize # operator
Non-integer numeric fields may be interpreted incorrectly if they have decimal point. To
solve, use the numerize operator # at the front of a quoted field, such as #'123.45'.
This will coerce to a numeric type (e.g. float).
>>> import dotted
>>> d = {'a': {1.2: 'hello', 1: {2: 'fooled you'}}}
>>> dotted.get(d, 'a.1.2')
'fooled you'
>>> dotted.get(d, 'a.#"1.2"')
'hello'
Slicing
Dotted slicing works like python slicing and all that entails.
>>> import dotted
>>> d = {'hi': {'there': [1, 2, 3]}, 'bye': {'there': [4, 5, 6]}}
>>> dotted.get(d, 'hi.there[::2]')
[1, 3]
>>> dotted.get(d, '*.there[1:]')
([2, 3], [5, 6])
The append + operator
Both bracketed fileds and slices support the '+' operator which refers to the end of sequence. You may append an item or slice to the end a sequence.
>>> import dotted
>>> d = {'hi': {'there': [1, 2, 3]}, 'bye': {'there': [4, 5, 6]}}
>>> dotted.update(d, '*.there[+]', 8)
{'hi': {'there': [1, 2, 3, 8]}, 'bye': {'there': [4, 5, 6, 8]}}
>>> dotted.update(d, '*.there[+:]', [999])
{'hi': {'there': [1, 2, 3, 8, 999]}, 'bye': {'there': [4, 5, 6, 8, 999]}}
The append-unique +? operator
If you want to update only unique items to a list, you can use the ?
postfix. This will ensure that it's only added once (see match-first below).
>>> import dotted
>>> items = [1, 2]
>>> dotted.update(items, '[+?]', 3)
[1, 2, 3]
>>> dotted.update(items, '[+?]', 3)
[1, 2, 3]
The invert - operator
You can invert the meaning of the notation by prefixing a -. For example,
to remove an item using update:
>>> import dotted
>>> d = {'a': 'hello', 'b': 'bye'}
>>> dotted.update(d, '-b', dotted.ANY)
{'a': 'hello'}
>>> dotted.remove(d, '-b', 'bye again')
{'a': 'hello', 'b': 'bye again'}
Patterns
You may use dotted for pattern matching. You can match to wildcards or regular expressions. You'll note that patterns always return a tuple of matches.
>>> import dotted
>>> d = {'hi': {'there': [1, 2, 3]}, 'bye': {'there': [4, 5, 6]}}
>>> dotted.get(d, '*.there[2]')
(3, 6)
>>> dotted.get(d, '/h.*/.*')
([1, 2, 3],)
Dotted will return all values that match the pattern(s).
Wildcards
The wildcard pattern is *. It will match anything.
Regular expressions
The regex pattern is enclosed in slashes: /regex/. Note that if the field is a non-str,
the regex pattern will internally match to its str representation.
The match-first operatoer
You can also postfix any pattern with a ?. This will return only
the first match.
>>> import dotted
>>> d = {'hi': {'there': [1, 2, 3]}, 'bye': {'there': [4, 5, 6]}}
>>> dotted.get(d, '*?.there[2]')
(3,)
Transforms
You can optionally add transforms to the end of dotted notation. These will
be applied on get and update. Transforms are separated by the | operator
and multiple may be chained together. Transforms may be parameterized using
the : operator.
>>> import dotted
>>> d = [1, '2', 3]
>>> dotted.get(d, '[1]')
'2'
>>> dotted.get(d, '[1]|int')
2
>>> dotted.get(d, '[0]|str:number=%d')
'number=1'
You may register new transforms via either register or the @transform
decorator. Look at transforms.py for preregistered.
Filters
The key-value filter
You may filter by key-value to narrow your result set. You may use with key or
bracketed fields. Key-value fields may be disjunctively (OR) specified via the ,
delimiter.
A key-value field on key field looks like: keyfield.key1=value1,key2=value2....
This will return all key-value matches on a subordinate dict-like object. For example,
>>> d = {
... 'a': {
... 'id': 1,
... 'hello': 'there',
... },
... 'b': {
... 'id': 2,
... 'hello': 'there',
... },
... }
>>> dotted.get(d, '*.id=1')
({'id': 1, 'hello': 'there'},)
>>> dotted.get(d, '*.id=*')
({'id': 1, 'hello': 'there'}, {'id': 2, 'hello': 'there'})
A key-value field on a bracketed field looks like: [key1=value1,key2=value2...].
This will return all items in a list that match key-value filter. For example,
>>> d = {
... 'a': [{'id': 1, 'hello': 'there'}, {'id': 2, 'hello': 'there'}],
... 'b': [{'id': 3, 'hello': 'there'}, {'id': 4, 'hello': 'bye'}],
... }
>>> dotted.get(d, 'a[hello="there"][*].id')
(1, 2)
>>> dotted.get(d, '*[hello="there"][*].id')
r == (1, 2, 3)
The key-value first filter
You can have it match first by appending a ? to the end of the filter.
>>> d = {
... 'a': [{'id': 1, 'hello': 'there'}, {'id': 2, 'hello': 'there'}],
... 'b': [{'id': 3, 'hello': 'there'}, {'id': 4, 'hello': 'bye'}],
... }
>>> dotted.get(d, 'a[hello="there"?]')
return [{'id': 1, 'hello': 'there'}]
Conjunction vs disjunction
To conjunctively connect filters use the . operator. Filters offer the ability to act
disjunctively as well by using the , operator.
For example, given
*.key1=value1,key2=value2.key3=value3. This will filter
(key1=value1 OR key2=value2) AND key3=value3.
Note that this gives you the abilty to have a key filter multiple values, such as:
*.key1=value1,key2=value2.
