How to pipe (/stream) big file (video,pdf,any…) from aws s3 to node.js express res?

0 votes

Currently i have tried many ways like

const S3FS = require('s3fs');

s3fsImpl = new S3FS(bucketName, {
  accessKeyId,
  secretAccessKey,
})

var range = 'undefined' !== typeof req.headers.range ? req.headers.range : 'bytes=0-';
var positions = range.replace(/bytes=/, "").split("-");
var start = parseInt(positions[0], 10);
var total = details.ContentLength;
var end = positions[1] ? parseInt(positions[1], 10) : total - 1;
var chunksize = (end - start) + 1;

res.setHeader("Content-Type", contentType);
res.status(206);
res.setHeader("Content-Range", "bytes " + start + "-" + end + "/" + total);
res.setHeader("Accept-Ranges", "bytes");
res.setHeader("Content-Length", chunksize);

s3fsImpl.createReadStream(key,{ start: start, end: end }).pipe(res);

( same with const AWS = require('aws-sdk') ) It just hangs up and no video occured. Currently i just do pipe without chunks, but it take video too long to occur, and seems i need to implement chunks stream from s3. So how can i pipe video, pddf, or other big files from the aws s3?

Sep 24, 2018 in AWS by bug_seeker
• 15,350 points
788 views

1 answer to this question.

0 votes

As far as I remember the limit for Amazon chubk size is somewhere around 5g's. And anyone sane would not implement this as a 5g chunk in node (not because it wouldn't make sense, but because it would be hard and wouldn't make sense). So unless you are over the size the issue doesn't seem to be in the code you posted

answered Sep 24, 2018 by Priyaj
• 56,520 points

Related Questions In AWS

0 votes
2 answers

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

Thanks for the answer. This should be clear ...READ MORE

answered Aug 14 in AWS by athenauserz
940 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,090 points
3,164 views
0 votes
1 answer

How to transfer file from ec2 to s3?

I'm using s3cmd to store nightly exported database backup ...READ MORE

answered Jan 14 in AWS by neha
133 views
0 votes
1 answer

AWS S3 uploading hidden files by default

versioning is enabled in your bucket. docs.aws.amazon.com/AmazonS3/latest/user-guide/….... the ...READ MORE

answered Oct 4, 2018 in AWS by Priyaj
• 56,520 points
200 views
0 votes
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 56,520 points
367 views
0 votes
1 answer

Import my AWS credentials using python script

Using AWS Cli  Configure your IAM user then ...READ MORE

answered Nov 16, 2018 in AWS by Jino
• 5,560 points
253 views
0 votes
2 answers
0 votes
1 answer

How to copy .csv file from Amazon S3 bucket?

Boto3 is the library to use for ...READ MORE

answered Jul 6, 2018 in AWS by Priyaj
• 56,520 points
362 views
0 votes
1 answer

how to access AWS S3 from Lambda in VPC

With boto3, the S3 urls are virtual by default, ...READ MORE

answered Sep 28, 2018 in AWS by Priyaj
• 56,520 points
1,484 views