Pikka Bird Collector (Python)

Pikka Bird ops monitoring tool Collector component.

Pikka Bird Collector gathers metrics reports, sending them to Pikka Bird Server. Pikka Bird Collector is a Python application.

Supported collectors include: system, MongoDB, MySQL, PostgreSQL, RabbitMQ, and Redis. For more details, see below.

Pikka Bird is currently in a draft phase, which means that payloads and schemas might be changed in a backwards-incompatible fashion. Although it is unlikely, in extreme cases this could require you to reinstall with an empty database. If this upsets you too much, please say hello and come back later. :) Currently, it is not recommended that you use Pikka Bird as a replacement for any of your usual monitoring tools.

More sleep lost by tiredpixel.

Installation

Install the following externals:

• Python

  The default version supported is defined in .python-version. Any other versions supported as defined in .travis.yml.

• Pikka Bird Server

  Pikka Bird Server collects the metrics Pikka Bird Collector gathers.

Install using Pip:

pip install pikka-bird-collector

There are currently no released server packages (stay tuned).

Usage

Run Pikka Bird Server.

To run the collector once:

pikka-bird-collector

To run the collector eternally, staggering to average once per minute:

pikka-bird-collector -e 60

To load service confs:

pikka-bird-collector -c test/fixtures/config/config.json

where the config is something like:

# config.json

{
  "redis": {
    "6379": null,
    "6380": {
      "password": "PASSWORD"
    },
    "6381": {}
  }
}

or maybe something like:

# config.yaml

redis:
  6379:
  6380:
    password: "PASSWORD"
  6381: {}

or even a directory of files, e.g. -c test/fixtures/config/conf.d

Help is at hand:

pikka-bird-collector -h

Collectors

Here follows an overview of each collector supported. Usually, there is nothing extra to install in order to start monitoring one of these services; merely add a conf declaration as above, which might need to be no more than just the port.

For more details about each collector, please see the documentation in pikka_bird_collector/collectors/.

System

Collector for system.

{'memory': {'swap': {'b': 2147483648, 'sout': {'b': 1213657088}, 'free': {'b': 950534144, '/': 0.44}, 'used': {'b': 1196949504, '/': 0.56}, 'sin': {'b': 66084589568}}, 'virtual': {'inactive': {'b': 968835072, '/': 0.11}, 'avail': {'b': 4248592384, '/': 0.49}, 'unavail': {'b': 4341342208, '/': 0.51}, 'used': {'b': 4688113664, '/': 0.55}, 'buffers': {'b': 258020000, '/': 0.03}, 'b': 8589934592, 'active': {'b': 2415570944, '/': 0.28}, 'cached': {'b': 241716800, '/': 0.03}, 'shared': {'b': 50448000, '/': 0.01}, 'free': {'b': 3279757312, '/': 0.38}, 'wired': {'b': 1303707648, '/': 0.15}}}, 'cpu': {0: {'busy': {'system': {'/': 0.02}, '/': 0.45, 'irq': {'/': 0.05}, 'iowait': {'/': 0.04}, 'softirq': {'/': 0.06}, 'user': {'/': 0.01}, 'nice': {'/': 0.03}, 'guest': {'/': 0.08}, 'guest_nice': {'/': 0.09}, 'steal': {'/': 0.07}}, 'idle': {'/': 0.55}}}, 'load': {'avg': {'15': 126.0, '1': 42.0, '5': 84.0}}, 'disk': {'/': {'flags': 1772, 'space': {'b': 100440011101, 'free': {'b': 98639472987, '/': 0.98}, 'used': {'b': 1800538114, '/': 0.02}}, 'inodes': {'n': 1769, 'avail': {'n': 3111696, '/': 1759.01}, 'free': {'n': 130691232, '/': 73878.59}}, 'block_size': {'b': 1764}, 'device': '/dev/disk42', 'filename_len_max': 1773, 'fragment_size': {'b': 1765}, 'blocks': {'n': 42, 'avail': {'n': 1768, '/': 42.1}, 'free': {'n': 74088, '/': 1764.0}}, 'fstype': 'zaphod'}}}

MongoDB

Collector for MongoDB (https://www.mongodb.org/).

{'server_status': {'uptimeEstimate': 22874, 'metrics': {'storage': {'freelist': {'search': {'scanned': 0, 'bucketExhausted': 0, 'requests': 0}}}, 'queryExecutor': {'scanned': 0, 'scannedObjects': 0}, 'cursor': {'open': {'pinned': 0, 'noTimeout': 0, 'total': 0}, 'timedOut': 0}, 'repl': {'buffer': {'sizeBytes': 0, 'maxSizeBytes': 268435456, 'count': 0}, 'apply': {'ops': 0, 'batches': {'num': 0, 'totalMillis': 0}}, 'preload': {'docs': {'num': 0, 'totalMillis': 0}, 'indexes': {'num': 0, 'totalMillis': 0}}, 'network': {'ops': 0, 'getmores': {'num': 0, 'totalMillis': 0}, 'bytes': 0, 'readersCreated': 0}}, 'commands': {'listDatabases': {'failed': 0, 'total': 1}, 'getLog': {'failed': 0, 'total': 8}, 'whatsmyuri': {'failed': 0, 'total': 26}, 'replSetGetStatus': {'failed': 16, 'total': 16}, 'top': {'failed': 0, 'total': 5}, 'ping': {'failed': 0, 'total': 4}, 'getnonce': {'failed': 0, 'total': 2}, 'serverStatus': {'failed': 0, 'total': 15}, 'dbStats': {'failed': 1, 'total': 4}, 'isMaster': {'failed': 0, 'total': 34}, '<UNKNOWN>': 4}, 'document': {'returned': 0, 'updated': 0, 'inserted': 0, 'deleted': 0}, 'ttl': {'deletedDocuments': 0, 'passes': 1883}, 'record': {'moves': 0}, 'getLastError': {'wtimeouts': 0, 'wtime': {'num': 0, 'totalMillis': 0}}, 'operation': {'fastmod': 0, 'scanAndOrder': 0, 'idhack': 0, 'writeConflicts': 0}}, 'asserts': {'warning': 0, 'regular': 0, 'rollovers': 0, 'msg': 0, 'user': 0}, 'host': 'tiredpixel.home', 'backgroundFlushing': {'average_ms': 4.1476367498672335, 'last_finished': '2015-05-16T15:45:53.346Z', 'total_ms': 7810, 'flushes': 1883, 'last_ms': 5}, 'dur': {'journaledMB': 0, 'commitsInWriteLock': 0, 'compression': 0, 'timeMs': {'remapPrivateView': 0, 'commitsInWriteLock': 0, 'prepLogBuffer': 0, 'writeToDataFiles': 0, 'writeToJournal': 0, 'commits': 0, 'dt': 3067}, 'writeToDataFilesMB': 0, 'earlyCommits': 0, 'commits': 10}, 'writeBacksQueued': False, 'network': {'bytesIn': 8422, 'bytesOut': 182467, 'numRequests': 120}, 'mem': {'bits': 64, 'resident': 50, 'virtual': 2973, 'mappedWithJournal': 480, 'mapped': 240, 'supported': True}, 'pid': 286, 'globalLock': {'currentQueue': {'readers': 0, 'writers': 0, 'total': 0}, 'totalTime': '618021658000', 'activeClients': {'readers': 0, 'writers': 0, 'total': 10}}, 'opcountersRepl': {'insert': 0, 'delete': 0, 'getmore': 0, 'update': 0, 'command': 0, 'query': 0}, 'extra_info': {'page_faults': 175428, 'note': 'fields vary by platform'}, 'opcounters': {'insert': 0, 'delete': 0, 'getmore': 0, 'update': 0, 'command': 115, 'query': 2}, 'storageEngine': {'name': 'mmapv1'}, 'connections': {'totalCreated': 28, 'current': 2, 'available': 202}, 'ok': 1, 'cursors': {'clientCursors_size': 0, 'pinned': 0, 'timedOut': 0, 'note': 'deprecated, use server status metrics', 'totalOpen': 0, 'totalNoTimeout': 0}, 'localTime': '2015-05-16T15:46:30.380Z', 'locks': {'Collection': {'acquireCount': {'R': 385078}}, 'Metadata': {'acquireCount': {'R': 1}}, 'MMAPV1Journal': {'timeAcquiringMicros': {'w': 131, 'R': 7163519}, 'acquireWaitCount': {'w': 3, 'R': 4}, 'acquireCount': {'w': 35, 'R': 354821, 'r': 366255}}, 'Database': {'acquireCount': {'R': 6, 'r': 366251, 'W': 15}}, 'Global': {'acquireCount': {'w': 15, 'r': 366257, 'W': 5}}}, 'version': '3.0.2', 'uptimeMillis': 618021690, 'process': 'mongod', 'uptime': 618022}, 'rs_status': {'members': [{'optime': 1404225575, '_id': 0, 'electionDate': '2014-05-01T14:39:46Z', 'state': 1, 'health': 1, 'stateStr': 'PRIMARY', 'name': 'm1.example.net:27017', 'optimeDate': '2014-05-01T14:39:35Z', 'uptime': 269, 'self': True, 'electionTime': 1404225586}, {'lastHeartbeat': '2014-05-01T14:44:03Z', 'optime': 1404225575, 'state': 2, 'pingMs': 0, 'stateStr': 'SECONDARY', 'syncingTo': 'm1.example.net:27017', '_id': 1, 'lastHeartbeatRecv': '2014-05-01T14:44:02Z', 'health': 1, 'name': 'm2.example.net:27017', 'optimeDate': '2014-05-01T14:39:35Z', 'uptime': 265}, {'lastHeartbeat': '2014-05-01T14:44:02Z', 'optime': 1404225575, 'state': 2, 'pingMs': 0, 'stateStr': 'SECONDARY', 'syncingTo': 'm1.example.net:27017', '_id': 2, 'lastHeartbeatRecv': '2014-05-01T14:44:02Z', 'health': 1, 'name': 'm3.example.net:27017', 'optimeDate': '2014-05-01T14:39:35Z', 'uptime': 265}], 'date': '2014-05-01T14:44:03Z', 'set': 'replset', 'ok': 1, 'myState': 1}}

MySQL

Collector for MySQL (https://www.mysql.com/).

{'slave_status': {'master_ssl_cert': None, 'until_log_file': None, 'seconds_behind_master': 0, 'relay_log_file': 'mysqld-relay-bin.000064', 'slave_io_state': 'Waiting for master to send event', 'slave_sql_running': True, 'exec_master_log_pos': 64707836, 'master_port': 3306, 'master_user': 'repl', 'last_sql_error': None, 'last_errno': 0, 'master_ssl_key': None, 'relay_master_log_file': 'mysql-bin.000024', 'master_ssl_allowed': False, 'read_master_log_pos': 64707836, 'master_host': 'i-00000000.example.com', 'last_sql_errno': 0, 'connect_retry': 60, 'master_ssl_cipher': None, 'replicate_ignore_db': None, 'master_log_file': 'mysql-bin.000024', 'master_ssl_ca_path': None, 'relay_log_pos': 64659963, 'until_condition': 'None', 'last_io_errno': 0, 'skip_counter': 0, 'replicate_wild_ignore_table': None, 'slave_io_running': True, 'replicate_wild_do_table': None, 'until_log_pos': 0, 'master_ssl_ca_file': None, 'replicate_ignore_table': None, 'master_ssl_verify_server_cert': False, 'replicate_do_table': None, 'replicate_ignore_server_ids': None, 'relay_log_space': 64660762, 'last_io_error': None, 'last_error': None, 'master_server_id': 1, 'replicate_do_db': None}, 'master_status': {'mysql-bin.000024': {'binlog_do_db': None, 'binlog_ignore_db': None, 'position': 64795006, 'file': 'mysql-bin.000024'}}, 'slave_hosts': {'2': {'host': 'i-00000000', 'port': 3306, 'server_id': 2, 'master_id': 1}}, 'status': {'innodb_buffer_pool_pages_free': 7736, 'com_alter_table': 0, 'com_uninstall_plugin': 0, 'key_writes': 0, 'aborted_connects': 13, 'innodb_buffer_pool_wait_free': 0, 'performance_schema_session_connect_attrs_lost': 0, 'sort_scan': 0, 'com_show_collations': 0, 'com_do': 0, 'com_show_fields': 0, 'slow_queries': 0, 'handler_rollback': 0, 'com_create_db': 0, 'com_show_slave_status': 0, 'com_insert_select': 0, 'performance_schema_cond_classes_lost': 0, 'com_show_slave_hosts': 0, 'ssl_server_not_before': None, 'innodb_buffer_pool_reads': 456, 'com_show_create_proc': 0, 'com_revoke': 0, 'performance_schema_hosts_lost': 0, 'key_read_requests': 0, 'com_show_variables': 0, 'handler_discover': 0, 'com_show_grants': 0, 'com_drop_function': 0, 'slave_last_heartbeat': None, 'ssl_accept_renegotiates': 0, 'performance_schema_thread_classes_lost': 0, 'com_show_function_code': 0, 'com_unlock_tables': 0, 'key_reads': 0, 'com_empty_query': 0, 'com_check': 0, 'com_rename_table': 0, 'threads_cached': 0, 'com_xa_recover': 0, 'innodb_data_written': 34304, 'innodb_available_undo_logs': 128, 'qcache_lowmem_prunes': 0, 'ssl_session_cache_overflows': 0, 'ssl_verify_depth': 0, 'com_update_multi': 0, 'questions': 2, 'delayed_writes': 0, 'performance_schema_file_handles_lost': 0, 'key_blocks_not_flushed': 0, 'innodb_row_lock_waits': 0, 'com_xa_end': 0, 'com_xa_prepare': 0, 'com_purge': 0, 'innodb_rows_deleted': 0, 'select_range_check': 0, 'connection_errors_internal': 0, 'com_rollback': 0, 'com_change_master': 0, 'com_show_engine_logs': 0, 'ssl_cipher_list': None, 'com_lock_tables': 0, 'select_full_join': 0, 'com_drop_index': 0, 'com_show_processlist': 0, 'ssl_accepts': 0, 'com_show_create_func': 0, 'innodb_buffer_pool_bytes_dirty': 0, 'performance_schema_locker_lost': 0, 'last_query_partial_plans': 0, 'com_show_tables': 0, 'com_alter_db': 0, 'innodb_dblwr_pages_written': 1, 'com_create_udf': 0, 'slave_running': False, 'com_ha_close': 0, 'handler_mrr_init': 0, 'ssl_session_cache_size': 0, 'performance_schema_accounts_lost': 0, 'com_drop_user': 0, 'connection_errors_max_connections': 0, 'com_set_option': 0, 'com_show_charsets': 0, 'open_files': 18, 'com_signal': 0, 'com_preload_keys': 0, 'com_create_table': 0, 'handler_write': 0, 'com_assign_to_keycache': 0, 'com_create_index': 0, 'com_savepoint': 0, 'handler_prepare': 0, 'opened_files': 118, 'innodb_row_lock_current_waits': 0, 'handler_read_first': 0, 'opened_tables': 0, 'performance_schema_mutex_instances_lost': 0, 'com_alter_tablespace': 0, 'handler_external_lock': 0, 'com_drop_db': 0, 'key_write_requests': 0, 'com_install_plugin': 0, 'com_create_trigger': 0, 'open_table_definitions': 68, 'com_drop_event': 0, 'handler_read_next': 0, 'ssl_session_cache_hits': 0, 'qcache_inserts': 0, 'ssl_used_session_cache_entries': 0, 'ssl_finished_connects': 0, 'com_reset': 0, 'select_range': 0, 'select_scan': 0, 'innodb_rows_read': 0, 'performance_schema_stage_classes_lost': 0, 'com_help': 0, 'innodb_buffer_pool_read_ahead_evicted': 0, 'com_checksum': 0, 'com_show_binlogs': 0, 'com_show_status': 1, 'innodb_buffer_pool_dump_status': 'not started', 'ssl_client_connects': 0, 'com_alter_event': 0, 'com_repair': 0, 'ssl_ctx_verify_mode': 0, 'ssl_callback_cache_hits': 0, 'handler_read_last': 0, 'innodb_os_log_pending_writes': 0, 'select_full_range_join': 0, 'performance_schema_rwlock_instances_lost': 0, 'table_locks_immediate': 74, 'ssl_finished_accepts': 0, 'com_rename_user': 0, 'innodb_buffer_pool_read_ahead_rnd': 0, 'innodb_data_read': 7540736, 'innodb_page_size': 16384, 'binlog_stmt_cache_use': 0, 'com_show_privileges': 0, 'com_execute_sql': 0, 'created_tmp_tables': 0, 'com_xa_commit': 0, 'connections': 148, 'ssl_ctx_verify_depth': 0, 'bytes_received': 224, 'com_show_create_event': 0, 'com_stmt_reprepare': 0, 'com_drop_server': 0, 'com_show_engine_status': 0, 'com_show_plugins': 0, 'connection_errors_accept': 0, 'innodb_have_atomic_builtins': True, 'com_alter_db_upgrade': 0, 'com_get_diagnostics': 0, 'handler_read_key': 0, 'com_show_profiles': 0, 'innodb_buffer_pool_pages_total': 8191, 'com_release_savepoint': 0, 'innodb_truncated_status_writes': 0, 'tc_log_max_pages_used': 0, 'binlog_stmt_cache_disk_use': 0, 'com_delete': 0, 'innodb_os_log_pending_fsyncs': 0, 'binlog_cache_disk_use': 0, 'performance_schema_rwlock_classes_lost': 0, 'delayed_errors': 0, 'innodb_num_open_files': 14, 'com_resignal': 0, 'com_drop_table': 0, 'com_truncate': 0, 'ssl_sessions_reused': 0, 'open_streams': 0, 'performance_schema_statement_classes_lost': 0, 'com_kill': 0, 'com_prepare_sql': 0, 'com_dealloc_sql': 0, 'innodb_buffer_pool_pages_flushed': 1, 'ssl_version': None, 'sort_range': 0, 'com_stmt_fetch': 0, 'performance_schema_socket_classes_lost': 0, 'slave_open_temp_tables': 0, 'com_drop_trigger': 0, 'com_xa_start': 0, 'com_show_keys': 0, 'com_xa_rollback': 0, 'ssl_cipher': None, 'handler_savepoint': 0, 'com_stmt_reset': 0, 'performance_schema_table_handles_lost': 0, 'open_tables': 61, 'handler_commit': 0, 'com_show_create_trigger': 0, 'com_alter_server': 0, 'com_stmt_execute': 0, 'com_select': 1, 'com_commit': 0, 'ssl_session_cache_misses': 0, 'handler_update': 0, 'com_show_create_db': 0, 'table_locks_waited': 0, 'com_stmt_send_long_data': 0, 'performance_schema_users_lost': 0, 'com_replace_select': 0, 'com_flush': 0, 'com_alter_user': 0, 'handler_read_rnd': 0, 'com_create_user': 0, 'com_create_function': 0, 'delayed_insert_threads': 0, 'connection_errors_tcpwrap': 0, 'com_call_procedure': 0, 'innodb_row_lock_time_max': 0, 'innodb_buffer_pool_load_status': 'not started', 'innodb_os_log_written': 512, 'com_show_procedure_status': 0, 'innodb_pages_created': 0, 'qcache_hits': 0, 'qcache_not_cached': 136, 'sort_rows': 0, 'com_admin_commands': 0, 'innodb_row_lock_time': 0, 'innodb_log_write_requests': 0, 'uptime_since_flush_status': 2616535, 'com_show_profile': 0, 'qcache_total_blocks': 1, 'com_stmt_close': 0, 'ssl_connect_renegotiates': 0, 'com_binlog': 0, 'com_alter_procedure': 0, 'compression': False, 'table_open_cache_misses': 0, 'com_update': 0, 'innodb_buffer_pool_pages_dirty': 0, 'created_tmp_files': 5, 'handler_delete': 0, 'com_begin': 0, 'com_create_event': 0, 'innodb_buffer_pool_pages_data': 455, 'innodb_buffer_pool_bytes_data': 7454720, 'slave_received_heartbeats': None, 'ssl_session_cache_timeouts': 0, 'com_purge_before_date': 0, 'com_drop_view': 0, 'performance_schema_mutex_classes_lost': 0, 'innodb_buffer_pool_read_requests': 8252, 'bytes_sent': 168, 'innodb_buffer_pool_write_requests': 1, 'com_ha_read': 0, 'table_open_cache_overflows': 0, 'key_blocks_used': 0, 'com_slave_start': 0, 'table_open_cache_hits': 0, 'com_slave_stop': 0, 'slave_retried_transactions': None, 'com_analyze': 0, 'com_rollback_to_savepoint': 0, 'slow_launch_threads': 0, 'com_show_create_table': 0, 'com_show_open_tables': 0, 'threads_created': 2, 'qcache_free_blocks': 1, 'opened_table_definitions': 0, 'com_optimize': 0, 'performance_schema_file_classes_lost': 0, 'innodb_data_reads': 477, 'com_delete_multi': 0, 'com_insert': 0, 'ssl_verify_mode': 0, 'com_create_procedure': 0, 'innodb_dblwr_writes': 1, 'com_show_procedure_code': 0, 'com_show_databases': 0, 'com_show_relaylog_events': 0, 'last_query_cost': 0.0, 'com_grant': 0, 'handler_read_prev': 0, 'com_alter_function': 0, 'connection_errors_select': 0, 'innodb_pages_read': 455, 'binlog_cache_use': 0, 'innodb_buffer_pool_read_ahead': 0, 'com_drop_procedure': 0, 'ssl_default_timeout': 0, 'queries': 410, 'handler_read_rnd_next': 0, 'innodb_buffer_pool_pages_misc': 0, 'com_show_events': 0, 'com_show_storage_engines': 0, 'com_revoke_all': 0, 'com_ha_open': 0, 'innodb_rows_updated': 0, 'created_tmp_disk_tables': 0, 'innodb_os_log_fsyncs': 3, 'innodb_data_fsyncs': 5, 'com_show_master_status': 0, 'innodb_data_writes': 5, 'flush_commands': 1, 'innodb_pages_written': 1, 'innodb_log_waits': 0, 'sort_merge_passes': 0, 'com_show_table_status': 0, 'tc_log_page_waits': 0, 'performance_schema_cond_instances_lost': 0, 'innodb_rows_inserted': 0, 'threads_running': 1, 'com_show_warnings': 0, 'com_show_binlog_events': 0, 'innodb_data_pending_writes': 0, 'com_show_triggers': 0, 'ssl_session_cache_mode': 'NONE', 'qcache_free_memory': 1031336, 'qcache_queries_in_cache': 0, 'com_show_errors': 0, 'innodb_log_writes': 1, 'com_create_server': 0, 'innodb_data_pending_fsyncs': 0, 'not_flushed_delayed_rows': 0, 'com_load': 0, 'com_change_db': 0, 'tc_log_page_size': 0, 'threads_connected': 2, 'performance_schema_digest_lost': 0, 'innodb_row_lock_time_avg': 0, 'performance_schema_file_instances_lost': 0, 'aborted_clients': 0, 'slave_heartbeat_period': None, 'uptime': 2616535, 'com_show_engine_mutex': 0, 'com_replace': 0, 'performance_schema_table_instances_lost': 0, 'handler_savepoint_rollback': 0, 'com_show_function_status': 0, 'performance_schema_socket_instances_lost': 0, 'com_create_view': 0, 'com_stmt_prepare': 0, 'key_blocks_unused': 6698, 'max_used_connections': 2, 'performance_schema_thread_instances_lost': 0, 'ssl_server_not_after': None, 'rsa_public_key': None, 'connection_errors_peer_address': 0, 'prepared_stmt_count': 0, 'innodb_data_pending_reads': 0}}

PostgreSQL

Collector for PostgreSQL (http://www.postgresql.org/).

{'status': {'pg_postmaster_start_time': '2015-05-09 13:06:08.858083+01', 'pg_last_xlog_replay_location': None, 'inet_client_port': 53666, 'pg_backup_start_time': None, 'uptime_s': 16227.606185, 'pg_is_in_backup': False, 'pg_current_xlog_location': '0/308A8CE0', 'pg_backend_pid': 28262, 'pg_is_xlog_replay_paused': None, 'version': 'PostgreSQL 9.4.1 on x86_64-apple-darwin14.3.0, compiled by Apple LLVM version 6.1.0 (clang-602.0.49) (based on LLVM 3.6.0svn), 64-bit', 'inet_server_addr': '127.0.0.1', 'pg_current_xlog_insert_location': '0/308A8CE0', 'pg_is_in_recovery': False, 'pg_last_xlog_receive_location': None, 'inet_server_port': 5432, 'pg_conf_load_time': '2015-05-09 13:06:08.690997+01', 'inet_client_addr': '127.0.0.1', 'pg_last_xact_replay_timestamp': None}, 'stat_replication': {'1114': {'application_name': 'walreceiver', 'sent_location': '0/290044C0', 'sync_priority': 0, 'backend_start': '15-MAY-14 19:54:05.535695 -04:00', 'client_addr': '127.0.0.1', 'backend_xmin': None, 'usename': 'repuser', 'usesysid': 16384, 'write_location': '0/290044C0', 'flush_location': '0/290044C0', 'pid': 1114, 'client_port': 52444, 'sync_state': 'async', 'client_hostname': None, 'replay_location': '0/290044C0', 'state': 'streaming'}}}

RabbitMQ

Collector for RabbitMQ (https://www.rabbitmq.com).

{'status': {'run_queue': 0, 'os': ['unix', 'darwin'], 'erlang_version': 'Erlang/OTP 17 [erts-6.1] [source] [64-bit] [smp:4:4] [async-threads:30] [hipe] [kernel-poll:true]\n', 'uptime': 699374, 'listeners': {'amqp': [5672, '127.0.0.1'], 'stomp': [61613, '::'], 'mqtt': [1883, '::'], 'clustering': [25672, '::']}, 'pid': 296, 'vm_memory_high_watermark': 0.4, 'alarms': 'memory', 'disk_free': 272057380864, 'running_applications': {'rabbitmq_amqp1_0': ['AMQP 1.0 support for RabbitMQ', '3.5.1'], 'os_mon': ['CPO  CXC 138 46', '2.2.15'], 'mnesia': ['MNESIA  CXC 138 12', '4.12.1'], 'webmachine': ['webmachine', '1.10.3-rmq3.5.1-gite9359c7'], 'stdlib': ['ERTS  CXC 138 10', '2.1'], 'rabbitmq_management': ['RabbitMQ Management Console', '3.5.1'], 'kernel': ['ERTS  CXC 138 10', '3.0.1'], 'rabbit': ['RabbitMQ', '3.5.1'], 'rabbitmq_stomp': ['Embedded Rabbit Stomp Adapter', '3.5.1'], 'rabbitmq_management_visualiser': ['RabbitMQ Visualiser', '3.5.1'], 'rabbitmq_web_dispatch': ['RabbitMQ Web Dispatcher', '3.5.1'], 'rabbitmq_mqtt': ['RabbitMQ MQTT Adapter', '3.5.1'], 'sasl': ['SASL  CXC 138 11', '2.4'], 'inets': ['INETS  CXC 138 49', '5.10.2'], 'rabbitmq_management_agent': ['RabbitMQ Management Agent', '3.5.1'], 'amqp_client': ['RabbitMQ AMQP Client', '3.5.1'], 'xmerl': ['XML parser', '1.3.7'], 'mochiweb': ['MochiMedia Web Server', '2.7.0-rmq3.5.1-git680dba8']}, 'disk_free_limit': 50000000, 'z_alarms': None, 'file_descriptors': {'total_used': 22, 'sockets_limit': 138, 'sockets_used': 3, 'total_limit': 156}, 'memory': {'connection_other': 5616, 'mnesia': 357864, 'other_ets': 1329520, 'other_proc': 13766976, 'other_system': 5281461, 'code': 20621106, 'mgmt_db': 413992, 'msg_index': 105184, 'total': 43976712, 'atom': 711569, 'connection_writers': 0, 'connection_readers': 0, 'queue_procs': 816560, 'plugins': 548176, 'binary': 18688, 'connection_channels': 0, 'queue_slave_procs': 0}, 'vm_memory_limit': 2804903116, 'processes': {'limit': 1048576, 'used': 274}}, 'cluster_status': {'partitions': {'hare@smacmullen': 'rabbit@smacmullen', 'rabbit@smacmullen': 'hare@smacmullen'}, 'running_nodes': ['rabbit@smacmullen', 'hare@smacmullen'], 'z_partitions': None, 'cluster_name': 'rabbit@tiredpixel.home', 'nodes': {'disc': ['hare@smacmullen', 'rabbit@smacmullen']}}}

Redis

Collector for Redis (http://redis.io/).

{'info': {'cpu': {'used_cpu_sys_children': 92.17, 'used_cpu_sys': 4209.35, 'used_cpu_user_children': 1159.56, 'used_cpu_user': 3376.71}, 'stats': {'sync_partial_ok': 0, 'sync_full': 0, 'evicted_keys': 0, 'instantaneous_input_kbps': 0.32, 'instantaneous_ops_per_sec': 4, 'total_net_output_bytes': 17899490797, 'keyspace_misses': 1852618, 'pubsub_patterns': 0, 'total_connections_received': 2574, 'pubsub_channels': 0, 'total_commands_processed': 24187020, 'total_net_input_bytes': 1826548361, 'expired_keys': 22482, 'latest_fork_usec': 3946, 'keyspace_hits': 6123971, 'sync_partial_err': 0, 'instantaneous_output_kbps': 0.08, 'migrate_cached_sockets': 0, 'rejected_connections': 0}, 'clients': {'blocked_clients': 0, 'client_longest_output_list': 0, 'connected_clients': 8, 'client_biggest_input_buf': 0}, 'server': {'redis_git_dirty': 1, 'redis_mode': 'standalone', 'config_file': '/etc/redis/6379.conf', 'multiplexing_api': 'epoll', 'tcp_port': 6379, 'arch_bits': 32, 'redis_git_sha1': '084a59c3', 'lru_clock': 4103598, 'redis_build_id': '4ee713e162d87771', 'uptime_in_days': 51, 'process_id': 3658, 'redis_version': '3.1.999', 'os': 'Linux 3.18.5-x86_64-linode52 x86_64', 'uptime_in_seconds': 4484973, 'gcc_version': '4.4.1', 'hz': 10, 'run_id': 'd2a51d884171bd123e36a03ffcc2d61db7a980d6'}, 'keyspace': {'db0': 'keys=3290,expires=1,avg_ttl=125388596277'}, 'replication': {'repl_backlog_first_byte_offset': 0, 'repl_backlog_size': 1048576, 'repl_backlog_active': 0, 'connected_slaves': 0, 'master_repl_offset': 0, 'role': 'master', 'repl_backlog_histlen': 0}, 'persistence': {'aof_last_write_status': 'ok', 'aof_rewrite_scheduled': 0, 'aof_current_rewrite_time_sec': -1, 'rdb_last_bgsave_time_sec': 0, 'rdb_changes_since_last_save': 481, 'loading': 0, 'aof_enabled': 0, 'rdb_bgsave_in_progress': 0, 'rdb_last_save_time': 1430166862, 'aof_rewrite_in_progress': 0, 'aof_last_rewrite_time_sec': -1, 'rdb_last_bgsave_status': 'ok', 'rdb_current_bgsave_time_sec': -1, 'aof_last_bgrewrite_status': 'ok'}, 'cluster': {'cluster_enabled': 0}, 'memory': {'maxmemory_policy': 'noeviction', 'maxmemory_human': '3.00G', 'total_system_memory_human': '3.91G', 'used_memory_lua_human': '23.00K', 'used_memory_peak': 13159480, 'total_system_memory': 4196720640, 'used_memory': 10752688, 'used_memory_peak_human': '12.55M', 'used_memory_human': '10.25M', 'used_memory_rss_human': '13.42M', 'used_memory_lua': 23552, 'mem_fragmentation_ratio': 1.31, 'used_memory_rss': 14069760, 'maxmemory': 3221225472, 'mem_allocator': 'jemalloc-3.6.0'}}, 'cluster_info': {'cluster_my_epoch': 2, 'cluster_slots_fail': 0, 'cluster_slots_pfail': 0, 'cluster_current_epoch': 6, 'cluster_stats_messages_sent': 1483972, 'cluster_size': 3, 'cluster_slots_assigned': 16384, 'cluster_stats_messages_received': 1483968, 'cluster_known_nodes': 6, 'cluster_state': 'ok', 'cluster_slots_ok': 16384}}

Design

One of the design goals of Pikka Bird is to enable production-suitable setup in a minimum of steps and configuration. To support this, Pikka Bird supports as many services as possible without requiring further dependencies, regardless of whether those services are installed or even compatible with the server.

For this reason, shelling out and using service executables directly is preferred to adding library dependencies (e.g. PostgreSQL psql to be used instead of using a nice library binding). This is slower, and can cause juggling with paths and different systems and shells, but enables the core dependencies to be kept small whilst allowing the supported services to grow into the tens or hundreds.

Pikka Bird is designed to gather and send as many metrics as can be found (with selective configuration at a collector level), even if that leads to large reports. Pikka Bird Collector has no concept of success or failure of individual metrics. Think more like a squirrel gathering nuts in a forest than asking the server whether it is okay by means of executed checks. This, at the expense of storage and some speed, does away with problems of remote execution privileges or installing and maintaining remote checks, as all interpretive dance occurs in the Server component on a fixed data structure.

Development

Copy the example configuration for development, adjusting to taste:

cp .env.example .env

Copy the example configuration for testing, adjusting to taste, adding the environment variable CI=true (the tests are destructive to the database):

cp .env.example .test.env

Install locally using Pip editable mode:

pip install -r requirements.txt
pip install -e .

Start a collector eternally using Honcho, which reads Procfile:

honcho start

Run the tests, which use py.test:

honcho run -e .test.env py.test

Stay Tuned

We have a Librelist mailing list! To subscribe, send an email to pikka.bird@librelist.com. To unsubscribe, send an email to pikka.bird-unsubscribe@librelist.com. There be archives.

You can also become a watcher on GitHub. And don't forget you can become a stargazer if you are so minded. :D

Contributions

Contributions are embraced with much love and affection! <3 Please fork the repository and wizard your magic, preferably with plenty of fairy-dust sprinkled over the tests. Then send me a pull request. :) If you're thinking about working on something involved, it would be great if you could wave via the issue tracker or mailing list; I'd hate for good effort to be wasted!

Do whatever makes you happy. We'll probably still like you. :)

Blessing

May you find peace, and help others to do likewise.

Licence

© tiredpixel 2015. It is free software, released under the MIT License, and may be redistributed under the terms specified in LICENSE.txt.