Connectors Reference

Azure Data Lake

Azure Data Lake Store connector allows you to read and add data to an Azure Data Lake account.

 

Status: Production

Tier: Premium

Version: 1.0.0

 

Actions:

Name

Summary

ListFiles (string account, [internal]string op, [Optional]string path)

List Files

CreateFolder (string account, [internal]string op, string path)

Create Folder

AppendFile_Concurrent (string account, [internal]string op, [advanced][Optional]string appendMode, string filepath, string body)

Append File (concurrent)

ReadFile (string account, [internal]string op, [internal]string read, string filepath)

Read File

UploadFile (string account, [internal]string op, [internal]string write, string filepath, [advanced][Optional]boolean overwrite, string body)

Upload File

AppendFile_Sequential (string account, string filepath, string body, [internal]string op, [internal]string append, [advanced][Optional]integer offset)

Append File (sequential)

DeleteFile (string account, [internal]string op, string filepath)

Delete File

AppendChunk (string account, string filepath, [internal]string op, string body, [internal]string session_id)

Append Chunk

 

Triggers:

Name

Summary

Trigger_ListFiles (string account, [internal]string op, [Optional]string path)

When a file is created or modified (metadata only)

 

Objects:

Name

Summary

folderResponse

 

operationPerformed

 

Trigger_ListFilesResponseItem

 

 

Actions:

ListFiles

Summary: List Files

Description: List files and folders in a Data Lake Store account.

 

Syntax:

AzureDataLake.ListFiles (string account, [internal]string op, [Optional]string path)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

op

string

 

 

True

path

string

(Folder path)

Path of the folder (no leading '/').

False

 

Returns:

          Type:folderResponse

 

CreateFolder

Summary: Create Folder

Description: Create a folder in a Data Lake Store account.

 

Syntax:

AzureDataLake.CreateFolder (string account, [internal]string op, string path)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

op

string

 

 

True

path

string

(Folder path)

Path of the folder to create (no leading '/').

True

 

Returns:

          Type:operationPerformed

 

AppendFile_Concurrent

Summary: Append File (concurrent)

Description: Concurrent append to a file.

 

Syntax:

AzureDataLake.AppendFile_Concurrent (string account, [internal]string op, [advanced][Optional]string appendMode, string filepath, string body)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

op

string

 

 

True

appendMode

string

(Append mode)Values: [autocreate, ]

Set to auto-create if path doesn't exist.

False

filepath

string

(File path)

Path of the file (no leading '/').

True

body

string(binary)

 

 

True

 

Returns:

          Type:string

          Summary: Operation message

 

ReadFile

Summary: Read File

Description: Read data from a Data Lake Store account.

 

Syntax:

AzureDataLake.ReadFile (string account, [internal]string op, [internal]string read, string filepath)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

op

string

 

 

True

read

string

 

 

True

filepath

string

(File path)

Path of the file (no leading '/').

True

 

Returns:

          Type:string(binary)

          Summary: File content

 

UploadFile

Summary: Upload File

Description: Upload data into a Data Lake Store account.

 

Syntax:

AzureDataLake.UploadFile (string account, [internal]string op, [internal]string write, string filepath, [advanced][Optional]boolean overwrite, string body)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

op

string

 

 

True

write

string

 

 

True

filepath

string

(File path)

Path of the file (no leading '/').

True

overwrite

boolean

(Overwrite)

Overwrite if existing file.

False

body

string(binary)

 

 

True

 

Returns:

 

AppendFile_Sequential

Summary: Append File (sequential)

Description: Append to a file serially.

 

Syntax:

AzureDataLake.AppendFile_Sequential (string account, string filepath, string body, [internal]string op, [internal]string append, [advanced][Optional]integer offset)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

filepath

string

(File path)

Path of the file (no leading '/').

True

body

string(binary)

 

 

True

op

string

(Append type)

 

True

append

string

 

 

True

offset

integer(int64)

(Offset)

The optional offset in the stream to begin the append operation. Leave it empty or set value to file size in bytes. In both cases, the operation appends at the end of the stream.

False

 

Returns:

 

DeleteFile

Summary: Delete File

Description: Delete a file from a Data Lake Store account.

 

Syntax:

AzureDataLake.DeleteFile (string account, [internal]string op, string filepath)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

op

string

 

 

True

filepath

string

(File path)

Path of the file (no leading '/').

True

 

Returns:

          Type:operationPerformed

 

AppendChunk

Summary: Append Chunk

Description: Append chunk of data into a partial file in Azure Data Lake account.

 

Syntax:

AzureDataLake.AppendChunk (string account, string filepath, [internal]string op, string body, [internal]string session_id)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

filepath

string

(File path)

Path of the file (no leading '/').

True

op

string

 

 

True

body

string(binary)

 

 

True

session_id

string

 

 

True

 

Returns:

 

Triggers:

Trigger Trigger_ListFiles

Summary: When a file is created or modified (metadata only)

Description: Trigger whenever a new file is uploaded to a specific folder.  Only returns metadata of file and not content.

 

Syntax:

AzureDataLake.Trigger_ListFiles (string account, [internal]string op, [Optional]string path)

 

Parameters:

Name

Type

Summary

Required

Related Action

account

string

(Account name)

Account name for Azure Data Lake.

True

op

string

 

 

True

path

string

(Folder path)

Path of the folder (no leading '/').

False

 

Returns:

          Type:array of (Trigger_ListFilesResponseItem)

          Description: FileStatus

 


 

folderResponse

Summary:

Description:

 

          Properties:

Name

Type

Summary

FileStatuses

FileStatuses

 

FileStatuses

 

FileStatuses

Summary:

Description: FileStatuses

 

          Properties:

Name

Type

Summary

FileStatus

array of (Trigger_ListFilesResponseItem)

 

FileStatus

 


 

operationPerformed

Summary:

Description:

 

          Properties:

Name

Type

Summary

boolean

boolean

Is Successful

Flag if the last operation was successful.


 

Trigger_ListFilesResponseItem

Summary:

Description:

 

          Properties:

Name

Type

Summary

pathSuffix

string

File name

File or folder name.

type

string

Type

Type of item (directory or file).

blockSize

integer(int32)

Block Size

Block size of folder or file.

accessTime

integer(int64)

Access Time

Unix (Epoch) time when the item last accessed.

modificationTime

integer(int64)

Modification Time

Unix (Epoch) time when the item last modified.