Easy S3

This package helps you use S3 easily.

📝 Table of Contents

• About
• Getting Started
• Usage
• Acknowledgments

🧐 About

This package helps you use S3 easily. You can use following functions.

Default

• Save
• Load
• List

Cache

• Save Cache
• Load Cache

Others

• List Directory Names

🏁 Getting Started

Installing

• If you want save as parquet format, install pandas and fastparquet.

pip install easy_s3

Prerequisites

1. Access Key required for S3 Authentication. If you don't have Access Key, Please follow below steps.

1. Click below URL.

   https://console.aws.amazon.com/iam/home#/users

2. Click Add user button.

   

3. Input User name and enable Programmatic access.

   

4. Click Attach existing policies directly and Search S3FullAccess and check AmazonS3FullAccess and click Next:Tags.

   

5. Click Next:Review

   

6. click Create user

7. copy Access Key ID and Secret access Key to user notepad.

   

8. complete!

2. Bucket required for store asset in S3. Please follow below steps.

1. create bucket. Please refer to URL below

   https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html

2. Click the bucket you just created.

   

3. click Permissions Tab and Uncheck Block all public access, and click Save button.

   

3. You need to know which region your bucket is in. If you don't know yet, Please follow below steps.

1. click HERE.

2. Your URL should been changed like this. Remember that region.

   • https://s3.console.aws.amazon.com/s3/home?`region=ap-northeast-2`

4. (Required) Create Handler

Use this code to create handler.

import easy_s3

bucket_name = "Your Bucket Name"

# The service name serves as a detailed classification within the bucket.
# In this situaion, accounts and orders and items are service name.
#   1. default/accounts/Your File Path
#   2. default/orders/Your File Path
#   3. default/items/Your File Path

service_name = "Your Service Name"
region_name = "Your Bucket Region"

# You don't need to use these two parameters if your authentication file is in ~/.aws/config.
aws_access_key_id = "YOUR AWS ACCESS KEY ID"
aws_secret_access_key = "YOUR AWS SECRET ACCESS KEY"

es = easy_s3.EasyS3(bucket_name, service_name, region_name, 
                    aws_access_key_id=aws_access_key_id,
                    aws_secret_access_key=aws_secret_access_key)

print(es)

result:

<easy_s3.EasyS3 object at 0x10d966278>

🎈 Usage

Please check Prerequisites before starting Usage.

🌱 Save

Use this function to save data into S3.

The saved path is as follows.

default/Your Service Name/Y-M-D/Your File Path

Examples

data = {"name": "apple", "price": "120"}
options = {
        "public": True,
        "ymd": False,
        "compress_type": "gzip"
    }
url = es.save("food/apple.json", data,
    options=options)

print(url)

result:

• If you want check file, check your bucket in S3 console.

https://test-bucket-725.s3.ap-northeast-2.amazonaws.com/default/items/food/apple.json

Parameters

• (required) path: str

  foo/bar/hello.json
  

• (required) value: dict | list | str | bytes | int | float | ...

  {"hello": "world", "yellow", "banana"}

• options: dict

  Parameters

  • public: bool (default: False)

    if this value is True, anyone can access it.

  • ymd: bool (default: False)

    if this value is True, ymd is entered automatically.

    default/Your Service Name/20-08-24/Your File Path
    

  • compress_type: str (default: None)

    currently only gzip is supported.

  {
      "public": True,
      "ymd": True,
      "compress_type": "gzip"
  }

Returns

• URL of saved file : str

🌱 Load

Use this function to load data from S3.

The loaded path is as follows.

default/Your Service Name/Your File Path

Examples

>>> data = es.load("food/apple.json")
>>> print(data)
{'name': 'apple', 'price': '120'}

Parameters

• (required) path: str

  foo/bar/hello.json
  

Returns

• loaded data : dict | list | str

🌱 List

Use this function to list directory from S3

The list path is as follows.

default/Your Service Name/Your Files Path

Examples

>>> print(es.list_objects("food/"))
['default/items/food/apple.json']

>>> print(es.list_objects("food/", True))
[{'key': 'default/items/food/apple.json', 'data': {'name': 'apple', 'price': '120'}}]

Parameters

• (required) path: str

  foo/bar/
  

• load: bool (default: False)

  if this value set True, the listed files are loaded.

Returns

• if load parameter is True, return list of paths: list

• if load parameter is False, return list of paths and loaded datas: list

🌱 Save Cache

Use this function to save data that uses the cache.

Use it when the cost to process is greater than the cost to save and load in S3. Click HERE for details.

The save cache path is as follows.

cache/Your Service Name/Your File Path

Examples

with cache_save you can see that file path starts with cache/...

>>> url = es.save_cache("food/apple.json", {"name": "apple", "price": "120"}, 10)

>>> print(url)

https://test-bucket-725.s3.ap-northeast-2.amazonaws.com/cache/items/food/apple.json

If open saved file, You can see that it is saved in the format below. Click HERE for more information.

{
    "value": {
        "name": "apple",
        "price": "120"
    },
    "cache_time": 10,
    "put_time": 1596727712.0505128
}

Parameters

• (required) path: str

  foo/bar/hello.json
  

• (required) value: dict | list | str | bytes | int | float | ...

  {"hello": "world", "yellow", "banana"}

• (required) cache_time: int

  Input the number of seconds you want to cache.

  10

Returns

• URL of saved file : str

🌱 Load Cache

Use this function to load cache data from S3.

Use it when the cost to process is greater than the cost to save and load in S3. See the example below for details.

The loaded path is as follows.

cache/Your Service Name/Your File Path

Examples

import time
import random

while True:
    print("\n=== Press any key to get started. ===")
    input()
    path = "food/apple.json"
    data = es.load_cache(path)

    if data == None:
        working_time = random.randint(0, 4)

        print(f"working for {working_time} seconds ...")

        time.sleep(working_time)

        data = {"name": "apple", "price": "120"}
        es.save_cache(path, data, cache_time=5)

        print("working complete!")
    else:
        print("cached!")

    print(data)

=== Press any key to get started. ===

working for 2 seconds ...
working complete!
{'name': 'apple', 'price': '120'}

=== Press any key to get started. ===

cached!
{'name': 'apple', 'price': '120'}

=== Press any key to get started. ===

cached!
{'name': 'apple', 'price': '120'}

=== Press any key to get started. ===

working for 1 seconds ...
working complete!
{'name': 'apple', 'price': '120'}

Parameters

• (required) path: str

  foo/bar/hello.json
  

Returns

• loaded data : dict | list | str | None

🌱 List Directory Names

Use this function to list directory names.

The list path is as follows.

default/Your Service Name/Your Directory Path

Examples

>>> print(es.listdir("/"))

['default/items/2020-08-06/food', 'default/items/2020-08-07/food', 'default/items/food']

Parameters

• (required) path: str

  foo/bar/
  

Returns

list of directory names: list

🎉 Acknowledgements

• Title icon made by Freepik.

• If you have a problem. please make issue.

• Please help develop this project 😀

• Thanks for reading 😄