▁ ▂ ▄ ｕ𝕟𝔻Ⓔ𝐫Ć𝔬𝓓ⓔ ▄ ▂ ▁

🦑the whole idea

1) The first step is to combine the project background and investigate more optimized solutions.

2) File upload failure is a common problem.

3) The common solution is to slice a large file into multiple small files and upload them in parallel to the request interface. After all requests are answered, all the fragmented files are merged on the server side. When the upload of a fragment fails, you can judge when re-uploading, upload only the part that failed last time, reduce the user's waiting time, and ease the pressure on the server. This is the shard upload file.

🦑Large file upload

So how do you upload large files in pieces?

The flow chart is as follows:

🦑SO Divided into the following steps to achieve:

1. File MD5 encryption

MD5 is the unique identifier of the file. You can use the MD5 of the file to query the upload status of the file.

According to the file modification time, file name, last modification time and other information, generate MD5 of the file through spark-md5 . It should be noted that large-format files need to be read in pieces, and the contents of the read files are added to the hash calculation of spark-md5 until the file is read, and finally the final hash code is returned to the callback callback function. Here you can add a progress bar for file reading as needed.

🦑 The implementation method is as follows:

> md5File (file) {
return new Promise((resolve, reject) => {
let blobSlice =
File.prototype.slice ||
File.prototype.mozSlice ||
File.prototype.webkitSlice
let chunkSize = file.size / 100
let chunks = 100
let currentChunk = 0
let spark = new SparkMD5.ArrayBuffer()
let fileReader = new FileReader()
fileReader.onload = function (e) {
console.log('read chunk nr', currentChunk + 1, 'of', chunks)
spark.append(e.target.result) // Append array buffer
currentChunk++
if (currentChunk < chunks) {
loadNext()
} else {
let cur = +new Date()
console.log('finished loading')
// alert(spark.end() + '---' + (cur - pre)); // Compute hash
let result = spark.end()
resolve(result)
}
}
fileReader.onerror = function (err) {
console.warn('oops, something went wrong.')
reject(err)
}
function loadNext () {
let start = currentChunk * chunkSize
let end =
start + chunkSize >= file.size ? file.size : start + chunkSize
fileReader.readAsArrayBuffer(blobSlice.call(file, start, end))
}
loadNext()
})
}

2. Query file status

After the front end to get MD5 file, query whether there is a name from the background MD5of the folder, if present, are listed in the folder and all files, get a list of sections have been uploaded, and if not, the list of already uploaded slice is empty.

checkFileMD5 (file, fileName, fileMd5Value, onError) {
const fileSize = file.size
const { chunkSize, uploadProgress } = this
this.chunks = Math.ceil(fileSize / chunkSize)
return new Promise(async (resolve, reject) => {
const params = {
fileName: fileName,
fileMd5Value: fileMd5Value,
}
const { ok, data } = await services.checkFile(params)
if (ok) {
this.hasUploaded = data.chunkList.length
uploadProgress(file)
resolve(data)
} else {
reject(ok)
onError()
}
})
}

3. File fragmentation

The core of file upload optimization is file fragmentation. The slice method in the Blob object can cut the file. The File object inherits the Blob object, so the File object also has a slice method.

Define the size variable of each slice file as chunkSize, get the number of slice chunks from the file size FileSize and the slice size chunkSize, use the for loop and file.slice () method to slice the file, the sequence number is 0-n, and The uploaded slice list is compared to get all unuploaded slices and pushed to the request list requestList.

async checkAndUploadChunk (file, fileMd5Value, chunkList) {
let { chunks, upload } = this
const requestList = []
for (let i = 0; i < chunks; i++) {
let exit = chunkList.indexOf(i + '') > -1
if (!exit) {
requestList.push(upload(i, fileMd5Value, file))
}
}
console.log({ requestList })
const result =
requestList.length > 0
? await Promise.all(requestList)
.then(result => {
console.log({ result })
return result.every(i => i.ok)
})
.catch(err => {
return err
})
: true
console.log({ result })
return result === true
}

4. Upload fragments

Call Promise.all to upload all the slices concurrently, and pass the slice number, slice file, and file MD5 to the background.

Backstage after receiving the upload request, first check the name 文件 MD5of the folder exists, does not exist, create the folder, and then fs-extrarename the method will move slice sliced from the temporary folder path, as follows:

🦑When all the pieces are uploaded successfully, the server is notified to merge, and when one piece fails to upload, it prompts "Upload failed". When re-uploading, the file upload status is obtained through the file MD5. When the server already has a slice corresponding to the MD5, it means that the slice has already been uploaded, and there is no need to upload again. When the server cannot find the slice corresponding to the MD5, it means that The slice needs to be uploaded. The user only needs to upload this part of the slice, and the entire file can be uploaded in full. This is the breakpoint resume of the file.

upload (i, fileMd5Value, file) {
const { uploadProgress, chunks } = this
return new Promise((resolve, reject) => {
let { chunkSize } = this
////FormData HTML5
let end =
(i + 1) * chunkSize >= file.size ? file.size : (i + 1) * chunkSize
let form = new FormData()
form.append('data', file.slice(i * chunkSize, end)) //
form.append('total', chunks) //
form.append('index', i) //
form.append('fileMd5Value', fileMd5Value)
services
.uploadLarge(form)
.then(data => {
if (data.ok) {
this.hasUploaded++
uploadProgress(file)
}
console.log({ data })
resolve(data)
})
.catch(err => {
reject(err)
})
})
}

5. Upload progress

Although the bulk upload of shards is much faster than the single upload of large files, there is still a period of loading time. At this time, a prompt of the upload progress should be added to display the progress of the file upload in real time.

The native Javascript XMLHttpRequest provides a progress event, which returns the uploaded size and total size of the file. Project uses axios of ajax encapsulated may increase in the config onUploadProgressmethod, monitor file upload progress.

const config = {
onUploadProgress: progressEvent => {
var complete = (progressEvent.loaded / progressEvent.total * 100 | 0) + '%'
}
}
services.uploadChunk(form, config)

6. Merge shards

After uploading all the file fragments, the front end actively informs the server to merge. When the server receives this request, it actively merges the slices and finds the folder with the same name in the file upload path of the server through the file MD5. As can be seen from the above, the file fragments are named according to the fragment sequence number, and the fragment upload interface is asynchronous, and there is no guarantee that the slices received by the server are spliced ​​in the order requested. So it should be before the segment file merge folder, and sorted according to file name, and then by concat-filesmerging fragmented files get uploaded files from users. So far, the large file upload is complete.