My code accesses an FTP server, downloads a .zip file, pushes the file contents as .gz to an AWS S3 bucket.import boto3import ftplibimport gzipimport ioimport zipfiledef _move_to_s3(fname):    host = 'some_host'    user = 'some_user'    passwd = 'some_password'    tmp_dir = '/tmp/'    tmp_fname = fname.split('.')[0] + '.gz'    target_bucket = 'some_bucket'    s3 = boto3.resource('s3')    try:        ftp = ftplib.FTP(host, user, passwd)        with io.BytesIO() as data, gzip.open(tmp_dir + tmp_fname, 'wb') as gz:            ftp.retrbinary('RETR ' + fname, data.write)            gz.write(data.getvalue())        s3.Object(target_bucket, tmp_fname).put(Body=open(tmp_dir + tmp_fname, 'rb'))    except Exception as e:        print e    finally:        ftp.quit()if __name__ == '__main__':    _move_to_s3('some_file.zip')I welcome any comments but my main points of interest are:Assuming the source zip contains only one text file, is it OK towrite to the gzip file from a stream containing the contents of thedownloaded zip file?I think I should change the code to hold the gzip in memory, without generating temporary files.
Convert zip to gzip and upload to S3 bucket
python;python 2.7;compression;ftp;amazon s3
Looks okay to me in general.  I'd prefer to pass in configurationseparately, parsed from a config file or command line arguments, but ifit's a one-off script it's probably okay this way.The temporary files aren't deleted as far as I see; consider using thetempfile module toallocate and delete temporary files automatically.Also, while the string to bytes conversion works as is, it's alsobrittle when used with large files, e.g. larger then the amount of RAMavailable, because it will store the downloaded file in memory.  I'dsuggest passing in gz.write directly instead:with gzip.open(tmp_dir + tmp_fname, 'wb') as gz:    ftp.retrbinary('RETR ' + fname, gz.write)Unless I'm mistaken that should keep the bytes in order; if not youcould still pass in a function that converts only the downloaded chunkto bytes and calls gz.write on the converted value.Regarding your questions:Since there's no extraction of data from the zip file, I don't see howthat matters.  If you were to extract something (a single file) fromthe zip file, then yes, gzipping that will of course be okay.  Makesure that the gzipped files are how you expect them (i.e. a singlecompressed text file) and that you don't need the file name or otherattributes in the original zip archive.  At the moment you basicallyupload some_file.zip.gz to S3, as in, two compressions nested.Probably not what you want.As argued above that's probably not advisable unless you know that thedata fits into memory.  If it does, sure, why not.Some more remarks:The zipfile import is unused, as mentioned above.Preferos.path.jointo concatenating strings.fname/passwd instead of filename/password seems pointlesstruncation.print should be used as a function to be forwards compatible,i.e. print(e) instead.The result of tmp_dir + tmp_fname should be reused instead ofwriting the same expression twice.