NECSTDB

Database for NECST.

TL;DR

NECST, an abbreviation of NEw Control System for Telescope, is a flexible control system for radio telescopes. Its efficient data storage is provided here.

The database contains tables, which keep individual topic of time series data with some metadata attached to them, e.g. spectral data from one spectrometer board (array of data + timestamp), weather data (temperature + humidity + wind speed + ... + timestamp), etc.

Features

This package provides:

• database writer with quite flexible data format support
• database reader which supports various output format

Installation

pip install necstdb

Usage

Database Writer

1. Create new database instance

   >>> db_dir = "path/to/new_database_directory"
   >>> db = necstdb.opendb(db_dir, mode="w")  # "w" stands for "write"

2. Create table with data information

   >>> data1_info = {
   ...     "data": [
   ...         {"key": "recorded_time", "format": "d", "size": 8},
   ...         {"key": "n_scan_lines", "format": "i", "size": 4},
   ...         {"key": "obsmode", "format": "3s", "size": 3},
   ...         {"key": "timestamp", "format": "d", "size": 8},
   ...     ],
   ...     "memo": "generated by db_logger_operation",
   ...     "necstdb_version": necstdb.__version__,
   ... }
   >>> db.create_table("data1", data1_info)

   The keys "key" and "format" in "data" list are required. The format characters and the sizes are listed below. For more information, see the struct module documentation and/or the ROS message wiki.

   • Changed in v0.2.5: "size" in "data" list is no longer required, but is optional.

   The "memo" and "necstdb_version" keys are not necessary. You can also add any other keys to the data information dict.

   ROS format	Format character	Size [byte]
   bool	?	1
   int8	b	1
   int16	h	2
   int32	i	4
   int64	q	8
   uint8	B	1
   uint16	H	2
   uint32	I	4
   uint64	Q	8
   float32	f	4
   float64	d	8
   string	[length]s	1 * length

   NOTE: Array data are also supported. To write them, use repeat count syntax (e.g. 3d) or format character sequence syntax (e.g. ddd) for the "format", and the sum of the elements' sizes for the "size".

   {"key": "data_array", "format": "3d", "size": 3 * 8}  # repeat count syntax
   {"key": "data_array", "format": "ddd", "size": 8 + 8 + 8}  # format character sequence syntax

3. Write data into table

   >>> table1 = db.open_table("data1", mode="ab")  # "ab" stands for "append binary"
   >>> data1 = [1.6294488758e9, 3, b"SKY", 1.6294488757e9]  # string data are not allowed, use bytes instead
   >>> table1.append(*data1)

   Call the append method every time you get new data.

   NOTE: Data to pass to append method should be flattened. The nested structure will be reconstructed on reading.

   data = [1, 2, [3, 4, 5], 6, 7]
   data = necstdb.utils.flatten_data(data)
   table.append(*data)

   NOTE: Multi-dimensional data are not supported.

   NOTE: Shape of array or length of string in every data must be the same as the one in data information. (Mismatch of string length won't raise an error, but stored data can be broken or incomplete.)

4. Save files into the database

   >>> db.save_file("example.txt", "Content of the file.")

Database Reader

1. Open the database instance

   >>> db = necstdb.opendb("path/to/database_directory")

2. Read a desired topic

   >>> data1 = db.open_table("data1").read(astype="array")
   >>> data1
   array([[1.6294488758e9, 3, b'SKY', 1.6294488757e9], ...], dtype=[('recorded_time', '<f8'), ('n_scan_lines', '<i4'), ('obsmode', '|S3'), ('timestamp', '<f8')])

   The supported astype keywords for read method are:

   Output type	Keywords	Notes
   tuple	"tuple"	*1
   dict	"dict"
   pandas.DataFrame	"pandas", "dataframe", "data_frame", "df"	*2
   numpy.ndarray	"array", "structuredarray", "structured_array", "sa"
   bytes	"buffer", "raw"

   *1: Array data are not supported, but will be flattened.
   *2: Changed in v0.2.5: Keyword df is now supported.

3. Read files saved in the database

   >>> db.read_file("example.txt")
   "Content of the file.", ""

   This method assumes the file can be read as str. This attempt will fail for binary files, so use asbytes option explicitly.

   >>> db.read_file("example_binary.data", asbytes=True)
   b"Content of the file.", b""

   This method returns 2 str or bytes values. The second value is metadata for the file saved using db.save_file(filename, data, metadata) method.

Misc

List all the tables contained in the database

>>> db = necstdb.opendb("path/to/database_directory")
>>> db.list_tables()
['data1', 'spectral_data', 'weather_data', ...]

Archive the database

>>> db = necstdb.opendb("path/to/database_directory")
>>> db.checkout(saveto="path/to/archive.tar.gz", compression="gz")

Get informations of all tables in the database

>>> db = necstdb.opendb("path/to/database_directory")
>>> db.get_info()
"""
                    file size [byte]    #records    record size [byte]  format
table name
data1               2448                102         24                  di3sd
spectral_data       41948160            320         131088              d32768fd
weather_data        6328                113         56                  ddddddd
"""

Read particular columns and/or rows of the database

>>> db = necstdb.opendb("path/to/database_directory")
>>> data = db.open_table("data1").read(num=5, start=3, cols=["timestamp", "obsmode"], astype="tuple")  # order of cols won't be preserved
((b'SKY', 1.6294488788e9)  # 4th element (caution 0-based indexing)
 (b'SKY', 1.6294488798e9)
 (b'SKY', 1.6294488808e9)
 (b'SKY', 1.6294488818e9)
 (b'HOT', 1.6294488828e9))

Flatten nested array

>>> data = [1, 2, 3, [4, 5], 6]
>>> flattened = necstdb.utils.flatten_data(data)
[1, 2, 3, 4, 5, 6]