API Reference

GetS3Handler

This module allows the user to interact with AWS S3 storage service.

>> from satsure_cloud_utils import GetS3Handler
>> s3_handler = GetS3Handler(config_file_path="<config-file-path>")

The module contains the following functions:

• list_buckets() - Returns List of all S3 buckets of an aws account.
• list_objects() - Returns List of all the objects in the bucket/bucket_path recursively.
• upload_objects() - Upload files/folders to S3 bucket.
• download_objects() - Download files/folders from S3 bucket.

list_buckets()

Lists all S3 buckets of an aws account

Returns:
    list: output/error list

>> output = s3_handler.list_buckets()
>> print(output)

$ ["project-data"]

list_objects()

Lists all the objects in the bucket/bucket_path recursively

        Args:
            bucket_name (string): Name of the bucket
            obj_path (string): Path of files in bucket (Default: '')
            include_filters (list): list of sub strings to include in filtering 
                                    Eg: ["20220101",".tif"] (No Wild card character allowed, only sub-strings)
            exclude_filters (list): list of sub strings to exclude in filtering
                                    Eg: ["20220101",".tif"] (No Wild card character allowed, only sub-strings)

        Returns:
            list: output/error list

>> output = s3_handler.list_objects(bucket_name = "project-data",
                                    obj_path = "folder_1/folder_2",
                                    include_filters = [".tif","20220101"])
>> print(output)

$ ['folder_1/folder_2/20120101_091007005000000_LCXX010.tif', 'folder_1/folder_2/20120101_091007007000000_LCXX010.tif']

upload_objects()

NOTE:

• This method will upload all files inside the local_path to s3_path.

Example: All the contents of Folder1 will be uploaded to s3_path.

Folder1 -->
          | - file_1.txt
          | - file_2.tif
          | - Folder2 -->
                        | - file3.py



>> s3_handler.upload_objects(bucket_name = "project-data",
                                    s3_path = "files",
                                    local_path = "./Folder1")

$  (dryrun) upload: Folder1/file_1.txt to s3://project-data/files/file_1.txt
$  (dryrun) upload: Folder1/file_2.tif to s3://project-data/files/file_2.tif
$  (dryrun) upload: Folder1/Folder2/file3.py to s3://project-data/files/Folder2/file3.py                

• Use dryrun = True to get a preview of actions without actually performing them

Upload files/folders to s3 bucket

        Args:
            bucket_name (string): Name of bucket 
            s3_path (string): Path on s3 bucket 
            local_path (string): Local path on your machine 
            include_filters (list): list of sub strings to include in filtering  
                                    Eg: ["20220101",".tif"] (No Wild card character allowed, only sub-strings) 
            exclude_filters (list): list of sub strings to exclude in filtering 
                                    Eg: ["20220101",".tif"] (No Wild card character allowed, only sub-strings) 
            dryrun (bool): Displays the operations that would be performed using the specified command without actually running them 

        Returns: 
            string: output/error string 

>> s3_handler.upload_objects(bucket_name = "project-data",
                                s3_path = "files",
                                local_path = "./files",
                                dryrun=True)

$  (dryrun) upload: files/file.txt to s3://project-data/files/file.txt
$  (dryrun) upload: files/file2.txt to s3://project-data/files/file2.txt
$  (dryrun) upload: files/folder1/file3.txt to s3://project-data/files/folder1/file3.txt

download_objects()

NOTE: By default downlod_objects method assumes you are downloading single file at a time but to bulk-download files, you need to pass the argument bulk = True

Download files/folders from s3 bucket

        Args:
            bucket_name (string): Name of bucket
            s3_path (string): path on s3 bucket
            local_path (string): Path on your machine
            bulk (bool): This allows to download files in bulk from bucket
            include_filters (list): list of strings to include in filtering 
                                    Eg: ["91021032*010_K021.tif","91021032*010_K019.tif"]
            exclude_filters (list): list of sub strings to exclude in filtering
                                    Eg: ["XX"]
            dryrun (bool): Displays the operations that would be performed using the specified command without actually running them
        Returns:
            string: output/error string

>> s3_handler.download_objects(bucket_name = "project-data",
                               s3_path = "files",
                               local_path = "./files",
                               include_filters = ["91021032*010_K021.tif","91021032*010_K019.tif"],
                               bulk=True,
                               dryrun = True)

$  (dryrun) download: s3://project-data/folder_1/folder_2/20120101_091021032000000_LCXX010.tif to ./files/20120101_091007005000000_LCXX010.tif
$  (dryrun) download: s3://project-data/folder_1/folder_2/20120101_091021032000000_LCXX010.tif to ./files/20120101_091007007000000_LCXX010.tif