Manipulate JSON-like data with NumPy-like idioms.


Keywords
apache-arrow, cern-root, columnar-format, data-analysis, jagged-array, json, numba, numpy, pandas, python, ragged-array, rdataframe, scikit-hep
License
BSD-3-Clause
Install
pip install awkward==2.6.5

Documentation

PyPI version Conda-Forge Python 3.8β€’3.12 BSD-3 Clause License Build Test

Scikit-HEP NSF-1836650 DOI Documentation Gitter

Awkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms.

Arrays are dynamically typed, but operations on them are compiled and fast. Their behavior coincides with NumPy when array dimensions are regular and generalizes when they're not.

Motivating example

Given an array of lists of objects with x, y fields (with nested lists in the y field),

import awkward as ak

array = ak.Array([
    [{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}],
    [],
    [{"x": 4.4, "y": [1, 2, 3, 4]}, {"x": 5.5, "y": [1, 2, 3, 4, 5]}]
])

the following slices out the y values, drops the first element from each inner list, and runs NumPy's np.square function on everything that is left:

output = np.square(array["y", ..., 1:])

The result is

[
    [[], [4], [4, 9]],
    [],
    [[4, 9, 16], [4, 9, 16, 25]]
]

The equivalent using only Python is

output = []
for sublist in array:
    tmp1 = []
    for record in sublist:
        tmp2 = []
        for number in record["y"][1:]:
            tmp2.append(np.square(number))
        tmp1.append(tmp2)
    output.append(tmp1)

The expression using Awkward Arrays is more concise, using idioms familiar from NumPy, and it also has NumPy-like performance. For a similar problem 10 million times larger than the one above (single-threaded on a 2.2 GHz processor),

  • the Awkward Array one-liner takes 1.5 seconds to run and uses 2.1 GB of memory,
  • the equivalent using Python lists and dicts takes 140 seconds to run and uses 22 GB of memory.

Awkward Array is even faster when used in Numba's JIT-compiled functions.

See the Getting started documentation on awkward-array.org for an introduction, including a no-install demo you can try in your web browser.

Getting help

Installation

Awkward Array can be installed from PyPI using pip:

pip install awkward

The awkward package is pure Python, and it will download the awkward-cpp compiled components as a dependency. If there is no awkward-cpp binary package (wheel) for your platform and Python version, pip will attempt to compile it from source (which has additional dependencies, such as a C++ compiler).

Awkward Array is also available on conda-forge:

conda install -c conda-forge awkward

Because of the two packages (awkward-cpp may be updated in GitHub but not on PyPI), pip install through git (pip install git+https://...) will not work. Instead, use the Installation for developers section below.

Installation for developers

Clone this repository recursively to get the header-only C++ dependencies, then generate sources with nox, compile and install awkward-cpp, and finally install awkward as an editable installation:

git clone --recursive https://github.com/scikit-hep/awkward.git
cd awkward

nox -s prepare
python -m pip install -v ./awkward-cpp
python -m pip install -e .

Tests can be run in parallel with pytest:

python -m pytest -n auto tests

For more details, see CONTRIBUTING.md, or one of the links below.

Documentation, Release notes, Roadmap, Citations

The documentation is on awkward-array.org, including

The Release notes for each version are in the GitHub Releases tab.

The Roadmap, Plans, and Deprecation Schedule are in the GitHub Wiki.

To cite Awkward Array in a paper, see the "Cite this repository" drop-down menu on the top-right of the GitHub front page. The BibTeX is

@software{Pivarski_Awkward_Array_2018,
author = {Pivarski, Jim and Osborne, Ianna and Ifrim, Ioana and Schreiner, Henry and Hollands, Angus and Biswas, Anish and Das, Pratyush and Roy Choudhury, Santam and Smith, Nicholas and Goyal, Manasvi},
doi = {10.5281/zenodo.4341376},
month = {10},
title = {{Awkward Array}},
year = {2018}
}

Acknowledgements

Support for this work was provided by NSF cooperative agreements OAC-1836650 and PHY-2323298 (IRIS-HEP), grant OAC-1450377 (DIANA/HEP), PHY-2121686 (US-CMS LHC Ops), and OAC-2103945 (Awkward Array).

We also thank Erez Shinan and the developers of the Lark standalone parser, which is used to parse type strings as type objects.

Thanks especially to the gracious help of Awkward Array contributors (including the original repository).

Jim Pivarski
Jim Pivarski

πŸ’» πŸ“– πŸš‡ 🚧
Ianna Osborne
Ianna Osborne

πŸ’»
Pratyush Das
Pratyush Das

πŸ’»
Anish Biswas
Anish Biswas

πŸ’»
glass-ships
glass-ships

πŸ’» ⚠️
Henry Schreiner
Henry Schreiner

πŸ’» πŸš‡
Nicholas Smith
Nicholas Smith

πŸ’» ⚠️
Lindsey Gray
Lindsey Gray

πŸ’» ⚠️
Ellipse0934
Ellipse0934

⚠️
Dmitry Kalinkin
Dmitry Kalinkin

πŸš‡
Charles Escott
Charles Escott

πŸ’»
Mason Proffitt
Mason Proffitt

πŸ’»
Michael Hedges
Michael Hedges

πŸ’»
Jonas Rembser
Jonas Rembser

πŸ’»
Jaydeep Nandi
Jaydeep Nandi

πŸ’»
benkrikler
benkrikler

πŸ’»
bfis
bfis

πŸ’»
Doug Davis
Doug Davis

πŸ’»
Joosep Pata
Joosep Pata

πŸ€”
Martin Durant
Martin Durant

πŸ€”
Gordon Watts
Gordon Watts

πŸ€”
Nikolai Hartmann
Nikolai Hartmann

πŸ’»
Simon Perkins
Simon Perkins

πŸ’»
.hard
.hard

πŸ’» ⚠️
HenryDayHall
HenryDayHall

πŸ’»
Angus Hollands
Angus Hollands

⚠️ πŸ’»
ioanaif
ioanaif

πŸ’» ⚠️
Bernhard M. Wiedemann
Bernhard M. Wiedemann

🚧
Matthew Feickert
Matthew Feickert

🚧
Santam Roy Choudhury
Santam Roy Choudhury

⚠️
Jeroen Van Goey
Jeroen Van Goey

πŸ“–
Ahmad-AlSubaie
Ahmad-AlSubaie

πŸ’»
Manasvi Goyal
Manasvi Goyal

πŸ’»
Aryan Roy
Aryan Roy

πŸ’»
Saransh
Saransh

πŸ’»
Laurits Tani
Laurits Tani

πŸ“–
Daniel Savoiu
Daniel Savoiu

πŸ’»
Ray Bell
Ray Bell

πŸ“–
Andrea Zonca
Andrea Zonca

πŸ’»
Chris Burr
Chris Burr

πŸš‡
ZoΓ« Bilodeau
ZoΓ« Bilodeau

πŸ’»
Raymond Ehlers
Raymond Ehlers

🚧
Markus LΓΆning
Markus LΓΆning

πŸ“–
Kush Kothari
Kush Kothari

πŸ’» ⚠️
Jonas RΓΌbenach
Jonas RΓΌbenach

πŸ’»
Jerry Ling
Jerry Ling

πŸ“–
Luis Antonio Obis Aparicio
Luis Antonio Obis Aparicio

πŸ’»
Topher Cawlfield
Topher Cawlfield

πŸ’»

πŸ’»: code, πŸ“–: documentation, πŸš‡: infrastructure, 🚧: maintenance, ⚠: tests and feedback, πŸ€”: foundational ideas.