Upload Video to GCP Cloud Storage in Nodejs

Question

I am trying to upload a video from my flutter client to my Node API. I am sending the file from the client as a stream which is being received by my server as an application/octet-stream.

How do I translate the octet-stream into .mp4 format on my server?

When I was just doing a multipart/form-data upload I had no problem uploading to GCP Storage in .mp4 format. But since the videos can get quick large I wanted to send it in chunks to my API.

Here is my current implementation:

  const newFileName = 'newFile';
  const bucket = storage.bucket('dev.appspot.com');
  const blob = bucket.file(newFileName);
  const blobStream = blob.createWriteStream({
    contentType: 'video/mp4',
    metadata: {
      contentType: 'video/mp4',
    },
  });
  blobStream.on('error', (err) => console.log(err));
  blobStream.on('finish', () => {
    console.log('finished');
  });

  req.pipe(blobStream);
  req.on('end', () => {
    console.log('upload complete');
  });

Update / Revision

req.pipe(blobStream); did not work because I was piping the entire request body into CGS which was causing the issue.

I now have

const newFileName = 'newFile';
          const bucket = storage.bucket('dev.appspot.com/postVideos');
          const blob = bucket.file(newFileName);
          const blobStream = blob.createWriteStream({
            metadata: { contentType: 'video/mp4' },
          });
          blobStream.on('error', (err) => {
            reject(err);
          });
          blobStream.on('finish', () => {
            resolve();
          });

          req.on('data', function(chunks) {
            // this is recieving video from the client in buffer chunks, but I have not figured out how to pipe those chunks into GCS as one file
          })

Now that I am receiving buffer chunks from the client, I am still unable to figure out how to stream those chunks into GCS to get one continuous file upload.

Answer 1

I hope this answers your question.

The problem you were facing was due to the fact that the multiparty module was creating content-type: octet stream headers on the part object that I was passing into the pipe to GCS. This caused GCS to receive two content-types of which the octet part was last. As a result GCS was using this for the inbound file.