The Globus Python SDK is a useful tool for using python code to initiate actions in Globus without using the web interface. The full documentation for the Globus Python SDK contains more detailed instructions on the full functionality of the software. These instructions are meant to be followed sequentially from section to section. The workflow example below is for transferring a file from Tempest to Blackmore, but can be altered for any two collections.

Creating Client in Globus Account

  1. Log in to Globus with your NetID
  2. Click the 'Setting' tab from the left panel and select the 'Developers' tab.

  3. Select “Register a thick client or script that will be installed and run by users on their devices.”

  4. Create or Select a Project, if you are creating a new project, this will take yo u to a new window to register the app.

  5. Click “Register App”. This will create your app and take you to a page describing it.

    Copy the “Project UUID” from the page. This Project UUID will be used as CLIENT_ID for authentication in the code in the next steps.

Selecting Source and Destination Collections

  1. Log in to Globus with your NetID

  2. On the landing page, i.e. File manager tab, enter “montana#tempest” in the Collection text bar.

  3. Hit enter and from the result list select the montana#tempest collection, and click 3 dots (at the right corner of the row).

  4. This will take you to the overview tab of the collection. Copy UUID. We will be using this as our source UUID

  5. Next search the “montana#blackmore” collection and click 3 dots (at the right corner of the row).

  6. This will take you to the overview tab of the collection. Copy UUID. We will be using this as our destination UUID.

Getting and Using Login Token for File Transfer

Use the python code below for transferring files from Tempest to Blackmore

import argparse
import os
import globus_sdk
from globus_sdk.scopes import TransferScopes
from globus_sdk.tokenstorage import SimpleJSONFileAdapter
from globus_sdk import (
    AuthClient,
    TransferClient,
    ConfidentialAppAuthClient,
    RefreshTokenAuthorizer,
)
my_file_adapter = SimpleJSONFileAdapter(os.path.expanduser("~/mytokens.json"))
my_file_adapterC = SimpleJSONFileAdapter(os.path.expanduser("~/mytokensC.json"))
parser = argparse.ArgumentParser()
parser.add_argument("SRC")
parser.add_argument("DST")
args = parser.parse_args()
CLIENT_ID = "c86c96ed-9eb5-415b-930b-72f75dd1a42e"
auth_client = globus_sdk.NativeAppAuthClient(CLIENT_ID)
# we will need to do the login flow potentially twice, so define it as a
# function
#
# we default to using the Transfer "all" scope, but it is settable here
# look at the ConsentRequired handler below for how this is used
def login_and_get_transfer_client(*, scopes=TransferScopes.all):
    # note that 'requested_scopes' can be a single scope or a list
    # this did not matter in previous examples but will be leveraged in
    # this one
    if not my_file_adapter.file_exists():
        auth_client.oauth2_start_flow(requested_scopes=scopes,refresh_tokens=True)
        authorize_url = auth_client.oauth2_get_authorize_url()
        print(f"Please go to this URL and login:\n\n{authorize_url}\n")
        auth_code = input("Please enter the code here: ").strip()
        tokens = auth_client.oauth2_exchange_code_for_tokens(auth_code)
        transfer_tokens = tokens.by_resource_server["transfer.api.globus.org"]
        transfer_rt = transfer_tokens["refresh_token"]
        transfer_at = transfer_tokens["access_token"]
        expires_at_s = transfer_tokens["expires_at_seconds"]
        my_file_adapter.store(tokens)
    else:
        transfer_tokens = my_file_adapter.get_token_data("transfer.api.globus.org")
        transfer_rt = transfer_tokens["refresh_token"]
        transfer_at = transfer_tokens["access_token"]
        expires_at_s = transfer_tokens["expires_at_seconds"]
    transfer_authorizer = RefreshTokenAuthorizer(
    transfer_rt, auth_client, access_token=transfer_at, expires_at=expires_at_s
    )   
    # return the TransferClient object, as the result of doing a login
    return globus_sdk.TransferClient(
        authorizer=transfer_authorizer)
# Duplicate function to save the token for consent           
def login_and_get_transfer_clientC(*, scopes=TransferScopes.all):
    # note that 'requested_scopes' can be a single scope or a list
    # this did not matter in previous examples but will be leveraged in
    # this one
    if not my_file_adapterC.file_exists():
        auth_client.oauth2_start_flow(requested_scopes=scopes,refresh_tokens=True)
        authorize_url = auth_client.oauth2_get_authorize_url()
        print(f"Please go to this URL and login:\n\n{authorize_url}\n")
        auth_code = input("Please enter the code here: ").strip()
        tokensC = auth_client.oauth2_exchange_code_for_tokens(auth_code)
        transfer_tokensC = tokensC.by_resource_server["transfer.api.globus.org"]
        transfer_rt = transfer_tokensC["refresh_token"]
        transfer_at = transfer_tokensC["access_token"]
        expires_at_s = transfer_tokensC["expires_at_seconds"]
        my_file_adapterC.store(tokensC)
    else:
        transfer_tokensC = my_file_adapterC.get_token_data("transfer.api.globus.org")
        transfer_rt = transfer_tokensC["refresh_token"]
        transfer_at = transfer_tokensC["access_token"]
        expires_at_s = transfer_tokensC["expires_at_seconds"]
    transfer_authorizer = RefreshTokenAuthorizer(
    transfer_rt, auth_client, access_token=transfer_at, expires_at=expires_at_s
    )   
    # return the TransferClient object, as the result of doing a login
    return globus_sdk.TransferClient(
        authorizer=transfer_authorizer)        
# get an initial client to try with, which requires a login flow
transfer_client = login_and_get_transfer_client()
# now, try an ls on the source and destination to see if ConsentRequired
# errors are raised
consent_required_scopes = []
def check_for_consent_required(target):
    try:
        transfer_client.operation_ls(target, path="/")
    # catch all errors and discard those other than ConsentRequired
    # e.g. ignore PermissionDenied errors as not relevant
    except globus_sdk.TransferAPIError as err:
        if err.info.consent_required:
            consent_required_scopes.extend(err.info.consent_required.required_scopes)
check_for_consent_required(args.SRC)
check_for_consent_required(args.DST)
# the block above may or may not populate this list
# but if it does, handle ConsentRequired with a new login
if consent_required_scopes:
    print(
        "One of your endpoints requires consent in order to be used.\n"
        "You must login a second time to grant consents.\n\n"
    )
    transfer_client = login_and_get_transfer_clientC(scopes=consent_required_scopes)
# from this point onwards, the example is exactly the same as the reactive
# case, including the behavior to retry on ConsentRequiredErrors. This is
# not obvious, but there are cases in which it is necessary -- for example,
# if a user consents at the start, but the process of building task_data is
# slow, they could revoke their consent before the submission step
#
# in the common case, a single submission with no retry would suffice
task_data = globus_sdk.TransferData(
    source_endpoint=args.SRC, destination_endpoint=args.DST
)
task_data.add_item(
    "/home/d81v711/helloWorld.java",  # source
    "/uit-rci/Nitasha-Test/helloWorld.java",  # dest
)
task_data.add_item(
    "/home/d81v711/helloWorld.m",  # source
    "/uit-rci/Nitasha-Test/helloWorld.m",  # dest
)
task_data.add_item(
    "/home/d81v711/helloWorld.class",  # source
    "/uit-rci/Nitasha-Test/helloWorld.class",  # dest
)
def do_submit(client):
    task_doc = client.submit_transfer(task_data)
    task_id = task_doc["task_id"]
    print(f"submitted transfer, task_id={task_id}")
try:
    do_submit(transfer_client)
except globus_sdk.TransferAPIError as err:
    if not err.info.consent_required:
        raise
    print(
        "Encountered a ConsentRequired error.\n"
        "You must login a second time to grant consents.\n\n"
    )
    transfer_client = login_and_get_transfer_clientC(
        scopes=err.info.consent_required.required_scopes
    )
    do_submit(transfer_client)

Altering Code for Usage

The code above first authenticates the client, and then transfers the file using file transfer service of Globus SDK. Moreover, it also creates a file for token storage. Lastly it creates a task and submits it.

File to store tokens are being created at the home directory with the names: mytokens.json and mytokensC.json

Do the following changes in the code.

  1. At line 14 of the above code, replace the CLIENT_ID = "0e5786c5-b867-4236-81d7-57902fdad922" with your client ID (created in step 1)

  2. Replace the source and destination files in the above code.

    1. Note you do not need to create a file in the destination folder, (you should just have a folder) but it is mandatory to specify the file name in the path for the destination folder.

task_data.add_item(
    "/home/d81v711/helloWorld.m",  # source
    "/uit-rci/Nitasha-Test/helloWorld.m",  # dest
)

If you want more files to transfer, just add more items in the task.

task_data.add_item(
    "/home/d81v711/helloWorld.java",  # source
    "/uit-rci/Nitasha-Test/helloWorld.java",  # dest
)
task_data.add_item(
    "/home/d81v711/helloWorld.m",  # source
    "/uit-rci/Nitasha-Test/helloWorld.m",  # dest
)
task_data.add_item(
    "/home/d81v711/helloWorld.class",  # source
    "/uit-rci/Nitasha-Test/helloWorld.class",  # dest
)

In case you need to transfer a large number of file, transfer the whole folder (after placing files into the folder)

task_data.add_item(
    "/home/d81v711/myProject",  # source
    "/uit-rci/Nitasha-Test/test",  # dest
)

NOTE: In case the source of destination collection is changed , we need a separate / new token file to store the authentication and consent of those collections.

Initiating Transfer Using SDK

  1. Create a Python file (e.g. globussdk.py) in your Tempest share, and copy the above code into this file .

  2. Now load Python module, using “module load Python” command, this will load the latest version of Python to your session.

  3. Next install Globus SDK using the following command.

 
pip install globus-sdk

  1. Change to the directory where you have created this python file “globussdk.py”. If it it is on root/home directory, just run the python code using the below command with the source and destination replaced with the IDs copied from step 2.

 
python globussdk.py <source-uuid> <destination-uuid>

  1. Upon execution of code, you will be presented with a URL in the terminal. Copy the URL and paste in the browser.

  2. Copy the code from the browser and place it back in the terminal.

  3. This is a one time step for one session. Once your SDK receives the token, it will save it in the file.

  4. If you are able to receive the task ID in the terminal, the transfer has been submitted. If the task fails, follow the error code to troubleshoot the origin of the issue.