I'm using wget to pipe straight from a url to S3 with the following command:
> wget -qO- '<url>' | aws s3 cp - s3://<bucket>/{data_file}The is to a file, which can be anywhere from 10 to 20Gb in size. I've ran the process a couple of times with a Python script with the following:
os.system('wget -qO- '<url>' | aws s3 cp - s3://<bucket>/{data_file}')When the process is complete I don't receive any errors, however the files in my s3 bucket are only ~4 or 5Gb in size, suggesting the entire file wasn't either read or written to the pipe.
I'm not sure what the problem is here. Has anyone had experience using wget to pipe big data files?
1 Reset to default
