Software | Version |
---|---|
Python | 3.10.x |
- Add
python-ms-core
package as dependency in yourrequirements.txt
- or
pip install python-ms-core
- Start using the core packages in your code.
Contains all the abstract and Azure implementation classes for connecting to Azure components.
All the cloud connections are initialized with initialize
function of core which takes an optional parameter of type CoreConfig
. A CoreConfig
takes only one parameter called provider
which by default is set to Azure
. The Core.initialize method can be called without any parameter or with a constructed CoreConfig object.
Eg.
from python_ms_core import Core
core = Core() or Core(config='Local')
# To check the package version
Core.__version__ # 0.0.21
The method analyzes the .env
variables and does a health check on what components are available
Core
works with all the default options and wherever required, relies on environment variables for connecting. The environment variables can be accessed either by setting them in the local machine or by importing via python-dotenv
package which reads from a .env
file created in the source code. All the variables in the .env
file are optional. However, some of them will be needed inorder for the specific features to work.
Here is the structure
# Provider information. Only two options available Azure and Local
# Defaults to Azure if not provided
PROVIDER=Azure
# Connection string to queue.
# Optional. Logger functionality for Azure may not work
# if not provided
QUEUECONNECTION=
# connection string to Azure storage if the provider is Azure
# Same can be used for root folder in Local provider
STORAGECONNECTION=
# Name of the queue that the logger writes to.
# This is optional and defaults value tdei-ms-log
LOGGERQUEUE=
This file will have to be generated or shared offline as per the developer requirement.
Offers helper classes to help log the information. It is also used to record the audit messages as well as the analytics information required.
Use core.get_logger()
Eg.
from python_ms_core import Core
core = Core()
logger = core.get_logger()
# Record a metric
logger.record_metric(name='test', value='test') # Metric and value
Note:
- All the
debug
,info
,warn
,error
logs can be logged withconsole
and will be injected into appInsight traces. Eg.
from python_ms_core import Core
core = Core()
logger = core.get_logger()
# Record a metric
logger.debug('Debug Message')
logger.info('Info Message')
logger.warn('Warning Message')
logger.error('Error Message')
NOTE: In future, there will be other decorators in place based on the need. Eg. @Validate, @UUID, @Required
These will help in easily modelling the classes along with the required validation.
Topic is an advanced version of Queue where the messages are published and subscribed. Each message published to a topic can be subscribed
by multiple parties for their own analysis and purpose. The messages sent and received via a topic will still be of type QueueMessage
The configuration required by Queue and Topic is similar and will be handled via a connection string.
Topic can be accessed by the core method get_topic
. This method takes two parameters
- topic name (required)
from python_ms_core import Core
core = Core()
topic = core.get_topic(topic_name='topicName') # By default, process messages concurrently which are available CPU cores
topic = core.get_topic(topic_name='topicName', max_concurrent_messages=10) # Process 10 messages concurrently
Once the topic object is got, use publish
method to publish the message to topic.
from python_ms_core import Core
from python_ms_core.core.queue.models.queue_message import QueueMessage
core = Core()
topic = core.get_topic(topic_name='topicName')
topic.publish(QueueMessage.data_from(
{
'message': 'Test message'
}
))
An active subscription will listening to a subscription over the topic. This is achieved by using subscribe
method of Topic
.
It takes one parameter
- subscription name
from python_ms_core import Core
core = Core()
topic_object = topic = Core.get_topic(topic_name='topicName')
def process(message):
print(f'Message Received: {message}')
try:
topic_object.subscribe(subscription='subscriptionName',process)
except Exception as e:
print(e)
- Please note that
subscribe
is a blocking call which runs on a loop. Either use athread.Thread
to subscribe or an async method to continue with the other parts of the program
eg.
from python_ms_core import Core
import threading
core = Core()
topic_object = topic = Core.get_topic(topic_name='topicName')
def process(message):
print(f'Message Received: {message}')
try:
thread = threading.Thread(topic_object.subscribe, ['subscriptionName',process])
thread.start()
except Exception as e:
print(e)
For all the azure blobs and other storages, storage components will offer simple ways to upload/download and read the existing data.
from python_ms_core import Core
core = Core()
# Create storage client
azure_client = core.get_storage_client()
# Get a container in the storage client
container = azure_client.get_container(container_name='tdei-storage-test')
# To get the list of files
list_of_files = container.list_files()
There are two ways to fetch the content of the file.
- ReadStream - use
file.get_stream()
which gives astream
object - GetText - use
file.get_body_text()
which gives astring
object containing all the data of the file inutf-8
format.
File upload is done only through read stream.
from python_ms_core import Core
from io import BytesIO, StringIO
core = Core()
azure_client = core.get_storage_client()
container = azure_client.get_container(container_name='tdei-storage-test')
# Create an instance of `AzureFileEntity` with name and mime-type
txt = 'foo\nbar\nbaz'
file_like_io = StringIO(txt)
test_file = container.create_file('text.txt', 'text/plain')
# Call the upload method with the readstream.
test_file.upload(file_like_io.read())
Core offers a simple way of verifying the authorization of a user and their role.
Checking the permission involves three steps
- Preparing a permission request object
- Getting an authorizer object from core
- Requesting if the permission is valid/true
from python_ms_core.core.auth.models.permission_request import PermissionRequest
permission_request = PermissionRequest(
user_id='<userID>',
project_group_id='<projectGroupID>',
should_satisfy_all=False,
permissions=['permission1', 'permission2']
)
In the above example, should_satisfy_all
helps in figuring out if all the permissions are needed or any one of the permission is sufficient.
Core exposes get_authorizer
method with 2 parameters
-
request_params
parameter which is instance ofPermissionRequest
class (Mandatory Parameter). -
config
parameter (Optional)
There are two types of Authorizer
objects in core.
- HostedAuthorizer: checks the permissions against a hosted API
- SimulatedAuthorizer: makes a simulated authorizer used for local/non-hosted environment.
The following code demonstrates getting the simulated and hosted authorizer
from python_ms_core import Core
core = Core()
# HostedAuthorizer
hosted_authorizer = core.get_authorizer(config={'provider': 'Hosted', 'api_url': '<AUTH_API_URL>'})
simulated_authorizer = core.get_authorizer(config={'provider': 'Simulated'})
In case api_url
is not provided for Hosted
auth provider, the core will pick it up from environment variable AUTHURL
Use the method has_permission(request_params)
to know if the permission request is valid/not.
# Complete Example
from python_ms_core import Core
from python_ms_core.core.auth.models.permission_request import PermissionRequest
core = Core()
permission_request = PermissionRequest(
user_id='<userID>',
project_group_id='<projectGroupID>',
should_satisfy_all=False,
permissions=['permission1', 'permission2']
)
# With Hosted provider
auth_provider = core.get_authorizer(config={'provider': 'Hosted', 'api_url': '<AUTH_API_URL>'})
response = auth_provider.has_permission(request_params=permission_request)
# Response will be boolean
# With Simulated provider
auth_provider = core.get_authorizer(config={'provider': 'Simulated'})
response = auth_provider.has_permission(request_params=permission_request)
# Response will be boolean
With simulated authentication, the method has_permission
simply returns the value given in should_satisfy_all
property in the permission request.
The project is configured with python
to figure out the coverage of the unit tests. All the tests are in tests
folder.
-
To execute the tests, please follow the commands:
pip install -r requirements.txt
python -m unittest discover -v tests/unit_tests
-
To execute the code coverage, please follow the commands:
coverage run --source=src/python_ms_core -m unittest discover -v tests/unit_tests
coverage html
// Can be run after 1st commandcoverage report
// Can be run after 1st command -
After the commands are run, you can check the coverage report in
htmlcov/index.html
. Open the file in any browser, and it shows complete coverage details -
The terminal will show the output of coverage like this
> python -m coverage run --source=src/python_ms_core -m unittest discover -v tests/unit_tests
test_has_permission (test_auth.abstract.test_authorizer_abstraction.TestAuthorizerAbstract) ... ok
test_get_search_params (test_auth.models.test_permission_request.TestPermissionRequest) ... ok
test_has_permission_with_invalid_permissions (test_auth.provider.test_hosted_authorizer.TestHostedAuthorizer) ... ok
test_has_permission_with_missing_auth_url (test_auth.provider.test_hosted_authorizer.TestHostedAuthorizer) ... ok
test_has_permission_with_valid_permissions (test_auth.provider.test_hosted_authorizer.TestHostedAuthorizer) ... ok
test_has_permission_should_return_should_satisfy_all (test_auth.provider.test_simulated_authorizer.TestSimulatedAuthorizer) ... ok
test_has_permission_should_return_should_satisfy_none (test_auth.provider.test_simulated_authorizer.TestSimulatedAuthorizer) ... ok
test_core_config_auth (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_auth_url (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_auth_url_default (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_logger (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_provider (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_provider_default (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_provider_override (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_queue (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_queue_connection (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_queue_connection_default (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_queue_name (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_queue_name_default (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_storage (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_storage_connection (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_storage_connection_default (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_topic (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_topic_connection (test_config.test_cofig.TestCoreConfig) ... ok
test_core_config_topic_connection_default (test_config.test_cofig.TestCoreConfig) ... ok
test_local_config_auth (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_logger (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_provider (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_provider_default (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_queue (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_queue_connection (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_queue_connection_default (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_queue_name (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_queue_name_default (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_storage (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_storage_connection (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_storage_connection_default (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_topic (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_topic_connection (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_topic_connection_default (test_config.test_cofig.TestLocalConfig) ... ok
test_local_config_auth (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_logger (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_provider (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_queue (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_queue_connection_default (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_queue_name_default (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_storage (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_storage_connection_default (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_topic (test_config.test_cofig.TestUnknownConfig) ... ok
test_unknown_config_topic_connection_default (test_config.test_cofig.TestUnknownConfig) ... ok
test_get_authorizer_no_config (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: SIMULATED
ok
test_get_authorizer_with_config_hosted_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: AZURE
ok
test_get_authorizer_with_config_simulated_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: SIMULATED
ok
test_get_authorizer_with_config_unknown_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: UNKNOWN
ok
test_get_logger_azure_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: AZURE
ok
test_get_logger_local_provider (test_core.TestCore) ... ok
test_get_logger_unknown_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: UNKNOWN
ok
test_get_storage_client_azure_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: AZURE
ok
test_get_storage_client_local_provider (test_core.TestCore) ... ok
test_get_storage_client_unknown_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: UNKNOWN
ok
test_get_topic_azure_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: AZURE
ok
test_get_topic_local_provider (test_core.TestCore) ... ok
test_get_topic_unknown_provider (test_core.TestCore) ... 2023-07-03 18:07:41 ERROR Failed to initialize core.get_logger for provider: UNKNOWN
ok
test_add_request (test_logger.abstract.test_logger_abstract.TestLoggerAbstract) ... ok
test_debug (test_logger.abstract.test_logger_abstract.TestLoggerAbstract) ... ok
test_error (test_logger.abstract.test_logger_abstract.TestLoggerAbstract) ... ok
test_info (test_logger.abstract.test_logger_abstract.TestLoggerAbstract) ... ok
test_record_metric (test_logger.abstract.test_logger_abstract.TestLoggerAbstract) ... ok
test_warn (test_logger.abstract.test_logger_abstract.TestLoggerAbstract) ... ok
test_add_request (test_logger.test_local_logger.TestLocalLogger) ... ok
test_debug (test_logger.test_local_logger.TestLocalLogger) ... ok
test_error (test_logger.test_local_logger.TestLocalLogger) ... ok
test_info (test_logger.test_local_logger.TestLocalLogger) ... ok
test_record_metric (test_logger.test_local_logger.TestLocalLogger) ... ok
test_warn (test_logger.test_local_logger.TestLocalLogger) ... ok
test_add_request (test_logger.test_logger.TestLogger) ... ok
test_debug (test_logger.test_logger.TestLogger) ... ok
test_error (test_logger.test_logger.TestLogger) ... ok
test_info (test_logger.test_logger.TestLogger) ... ok
test_record_metric (test_logger.test_logger.TestLogger) ... ok
test_warn (test_logger.test_logger.TestLogger) ... ok
test_concrete_queue_has_abstract_methods_implemented (test_queue.abstract.test_queue_abstract.TestQueueAbstract) ... ok
test_concrete_queue_inherits_from_queue_abstract (test_queue.abstract.test_queue_abstract.TestQueueAbstract) ... ok
test_azure_provider_invalid_connection_string (test_queue.config.test_queue_config.TestConfig) ... ok
test_azure_provider_missing_queue_name (test_queue.config.test_queue_config.TestConfig) ... ok
test_azure_provider_valid_connection_string (test_queue.config.test_queue_config.TestConfig) ... ok
test_non_azure_provider (test_queue.config.test_queue_config.TestConfig) ... ok
test_add_message_to_queue (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_add_message_to_queue_no_data (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_data_from_invalid_string (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_data_from_valid_string (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_get_items (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_remove_message_from_empty_queue (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_remove_message_from_queue (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_send_queue (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_to_dict (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_validate_data (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_validate_data_invalid_type (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_validate_string (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_validate_string_invalid_type (test_queue.models.test_queue_message.QueueMessageTestCase) ... ok
test_add_message_to_queue (test_queue.test_local_queue.TestLocalQueue) ... ok
test_add_message_to_queue_with_no_data (test_queue.test_local_queue.TestLocalQueue) ... ok
test_empty_queue (test_queue.test_local_queue.TestLocalQueue) ... ok
test_get_items_from_queue (test_queue.test_local_queue.TestLocalQueue) ... ok
test_remove_message_from_queue (test_queue.test_local_queue.TestLocalQueue) ... ok
test_send_with_data (test_queue.test_local_queue.TestLocalQueue) ... 200
ok
test_send_without_data (test_queue.test_local_queue.TestLocalQueue) ... ok
test_add (test_queue.test_queue.TestQueue) ... ok
test_add_no_data (test_queue.test_queue.TestQueue) ... ok
test_empty (test_queue.test_queue.TestQueue) ... ok
test_get_items (test_queue.test_queue.TestQueue) ... ok
test_remove (test_queue.test_queue.TestQueue) ... ok
test_send_no_data (test_queue.test_queue.TestQueue) ... ok
test_send_with_data (test_queue.test_queue.TestQueue) ... ok
test_bad_request_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_conflict_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_forbidden_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_internal_server_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_not_found_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_service_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_timeout_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_too_many_request_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_unauthorized_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_unprocessable_error (test_resource_errors.test_errors.TestServiceErrors) ... ok
test_get_body_text (test_storage.abstract.test_file_entity.TestFileEntity) ... ok
test_get_remote_url (test_storage.abstract.test_file_entity.TestFileEntity) ... ok
test_get_stream (test_storage.abstract.test_file_entity.TestFileEntity) ... ok
test_init (test_storage.abstract.test_file_entity.TestFileEntity) ... ok
test_upload (test_storage.abstract.test_file_entity.TestFileEntity) ... ok
test_get_container (test_storage.abstract.test_storage_client.TestStorageClient) ... ok
test_get_file (test_storage.abstract.test_storage_client.TestStorageClient) ... ok
test_get_file_from_url (test_storage.abstract.test_storage_client.TestStorageClient) ... ok
test_create_file (test_storage.abstract.test_storage_container.TestStorageContainer) ... ok
test_init (test_storage.abstract.test_storage_container.TestStorageContainer) ... ok
test_list_files (test_storage.abstract.test_storage_container.TestStorageContainer) ... ok
test_azure_provider_invalid_connection_string (test_storage.config.test_storage_config.TestConfig) ... ok
test_azure_provider_missing_queue_name (test_storage.config.test_storage_config.TestConfig) ... ok
test_azure_provider_valid_connection_string (test_storage.config.test_storage_config.TestConfig) ... ok
test_non_azure_provider (test_storage.config.test_storage_config.TestConfig) ... ok
test_default_method (test_storage.models.test_config.TestCoreConfig) ... ok
test_init_with_custom_provider (test_storage.models.test_config.TestCoreConfig) ... ok
test_init_with_default_provider (test_storage.models.test_config.TestCoreConfig) ... ok
test_get_body_text (test_storage.test_azure_file_entity.TestAzureFileEntity) ... ok
test_get_remote_url (test_storage.test_azure_file_entity.TestAzureFileEntity) ... ok
test_get_stream (test_storage.test_azure_file_entity.TestAzureFileEntity) ... ok
test_upload (test_storage.test_azure_file_entity.TestAzureFileEntity) ... ok
test_get_container (test_storage.test_azure_storage_client.TestAzureStorageClient) ... ok
test_get_file (test_storage.test_azure_storage_client.TestAzureStorageClient) ... ok
test_get_file_from_url (test_storage.test_azure_storage_client.TestAzureStorageClient) ... ok
test_create_file (test_storage.test_azure_storage_container.TestAzureStorageContainer) ... ok
test_list_files (test_storage.test_azure_storage_container.TestAzureStorageContainer) ... ok
test_list_files_with_name_starts_with (test_storage.test_azure_storage_container.TestAzureStorageContainer) ... ok
test_get_body_text (test_storage.test_local_file_entity.TestLocalFileEntity) ... ok
test_get_remote_url (test_storage.test_local_file_entity.TestLocalFileEntity) ... ok
test_get_stream (test_storage.test_local_file_entity.TestLocalFileEntity) ... ok
test_upload (test_storage.test_local_file_entity.TestLocalFileEntity) ... ok
test_get_container (test_storage.test_local_storage_client.TestLocalStorageClient) ... ok
test_get_container_without_name (test_storage.test_local_storage_client.TestLocalStorageClient) ... ok
test_get_file (test_storage.test_local_storage_client.TestLocalStorageClient) ... ok
test_get_file_from_url (test_storage.test_local_storage_client.TestLocalStorageClient) ... ok
test_create_file (test_storage.test_local_storage_container.LocalStorageContainerTests) ... ok
test_list_files (test_storage.test_local_storage_container.LocalStorageContainerTests) ... ok
test_init (test_topic.abstract.test_topic_abstract.TestDerivedTopic) ... ok
test_publish (test_topic.abstract.test_topic_abstract.TestDerivedTopic) ... ok
test_subscribe (test_topic.abstract.test_topic_abstract.TestDerivedTopic) ... ok
test_azure_provider_invalid_connection_string (test_topic.config.test_topic_config.TestConfig) ... ok
test_azure_provider_logging_levels (test_topic.config.test_topic_config.TestConfig) ... ok
test_azure_provider_valid_connection_string (test_topic.config.test_topic_config.TestConfig) ... ok
test_non_azure_provider (test_topic.config.test_topic_config.TestConfig) ... ok
test_sender_creation (test_topic.config.test_topic_config.TestConfig) ... ok
test_publish (test_topic.test_local_topic.TestLocalTopic) ... ok
test_subscribe_with_subscription (test_topic.test_local_topic.TestLocalTopic) ... ok
test_publish (test_topic.test_topic.TestTopic) ... ok
test_subscribe_with_subscription (test_topic.test_topic.TestTopic) ... ok
----------------------------------------------------------------------
Ran 174 tests in 8.175s
OK