How to pipe (/stream) big file (video,pdf,any…) from aws s3 to node.js express res?

0 votes

Currently i have tried many ways like

const S3FS = require('s3fs');

s3fsImpl = new S3FS(bucketName, {
  accessKeyId,
  secretAccessKey,
})

var range = 'undefined' !== typeof req.headers.range ? req.headers.range : 'bytes=0-';
var positions = range.replace(/bytes=/, "").split("-");
var start = parseInt(positions[0], 10);
var total = details.ContentLength;
var end = positions[1] ? parseInt(positions[1], 10) : total - 1;
var chunksize = (end - start) + 1;

res.setHeader("Content-Type", contentType);
res.status(206);
res.setHeader("Content-Range", "bytes " + start + "-" + end + "/" + total);
res.setHeader("Accept-Ranges", "bytes");
res.setHeader("Content-Length", chunksize);

s3fsImpl.createReadStream(key,{ start: start, end: end }).pipe(res);

( same with const AWS = require('aws-sdk') ) It just hangs up and no video occured. Currently i just do pipe without chunks, but it take video too long to occur, and seems i need to implement chunks stream from s3. So how can i pipe video, pddf, or other big files from the aws s3?

Sep 24, 2018 in AWS by bug_seeker
• 15,310 points
682 views

1 answer to this question.

0 votes

As far as I remember the limit for Amazon chubk size is somewhere around 5g's. And anyone sane would not implement this as a 5g chunk in node (not because it wouldn't make sense, but because it would be hard and wouldn't make sense). So unless you are over the size the issue doesn't seem to be in the code you posted

answered Sep 24, 2018 by Priyaj
• 56,160 points

Related Questions In AWS

0 votes
1 answer

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

This is a known deficiency. The best ...READ MORE

answered Sep 4, 2018 in AWS by Archana
• 4,090 points
749 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,090 points
2,404 views
0 votes
1 answer

How to transfer file from ec2 to s3?

I'm using s3cmd to store nightly exported database backup ...READ MORE

answered Jan 14 in AWS by neha
112 views
0 votes
1 answer

AWS S3 uploading hidden files by default

versioning is enabled in your bucket. docs.aws.amazon.com/AmazonS3/latest/user-guide/….... the ...READ MORE

answered Oct 4, 2018 in AWS by Priyaj
• 56,160 points
148 views
0 votes
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 56,160 points
277 views
0 votes
1 answer

Import my AWS credentials using python script

Using AWS Cli  Configure your IAM user then ...READ MORE

answered Nov 16, 2018 in AWS by Jino
• 5,550 points
198 views
0 votes
2 answers
0 votes
1 answer

How to copy .csv file from Amazon S3 bucket?

Boto3 is the library to use for ...READ MORE

answered Jul 6, 2018 in AWS by Priyaj
• 56,160 points
328 views
0 votes
1 answer

how to access AWS S3 from Lambda in VPC

With boto3, the S3 urls are virtual by default, ...READ MORE

answered Sep 28, 2018 in AWS by Priyaj
• 56,160 points
1,233 views