Q:

How to Copy Files with Databricks dbutilis in particular order

A member from this group assisted me in copying files to a follow based on date
copy based on date
I would like to tweak the code to copy file based on certain characters in a filename – in the example that follows the characters are 1111, 1112, 1113 and 1114
So, if we have four files as follows:
File_Account_1111_exam1.csv
File_Account_1112_testxx.csv
File_Account_1113_pringle.csv
File_Account_1114_sam34.csv
I would like File_Account_1114_sam34.csv copied to the folder only if File_Account_1113_pringle.csv has already been copied to the folder.
Likewise I would only want File_Account_1113_pringle.csv copied  if File_Account_1112_testxx.csv has been already been copied to the folder and so on.
Therefore, if all files have been copied to a folder it would look something like the following:
dbutils.fs.put("/mnt/adls2/demo/files/file_Account_1111_exam1.csv", data, True)
dbutils.fs.put("/mnt/adls2/demo/files/file_Account_1112_testxx.csv", data, True)
dbutils.fs.put("/mnt/adls2/demo/files/file_Account_1113_pringle.csv", data, True)
dbutils.fs.put("/mnt/adls2/demo/files/file_Account_1114_sam34.csv", data, True)

A:

Instead of applying any business logic when uploading files to DBFS I would recommend uploading all available files, then read them using test = sc.wholeTextFiles("pathtofile") which will return the key/value RDD of the file name and the file content, here is a corresponding thread. Once it is done any sort or filtering business logic based on file name may be implemented and tested in Spark job. 
I hope it is helpful.

