Introduction to Shionlib's Large-File Upload System
Why design a large-file upload system?
One of Shionlib’s goals is to build and maintain a complete, controllable archive of visual novels/galgames across all eras. Achieving this is anything but easy. From our experience, only a community-driven model can make it possible.
Therefore, we designed an upload system that grants all users large-file upload permissions.
Upload workflow
After you select a local file, the system slices it into chunks (each no larger than 5 MB) and computes the file’s SHA-256 hash.
Once the upload starts, the browser sends all chunks and the hash to Shionlib’s servers. The server merges the finished chunks, recomputes the SHA-256 hash, and compares it with the hash calculated by the browser. If they match, the upload is marked as successful.
Next, Shionlib’s servers scan the uploaded file to check whether it meets requirements (including file integrity, corruption, encryption, and potential malware). If the scan passes, the file enters a waiting queue to be uploaded to Shionlib’s S3 bucket.
Finally, the file is uploaded to Shionlib’s S3 bucket and a download link is generated. The upload process is then marked as successfully completed.
Challenges
It’s obvious we must design the entire system with great care. If anyone attempts to disrupt it in any way, the consequences would be unacceptable.
- Maliciously uploading a large number of huge files. This would occupy far more storage than planned, potentially forcing us to pay excessive storage fees.
- Maliciously uploading files that contain malware. Although we never attempt to extract or execute files at any point, such files could still pose risks to users who download them.
To ensure the system can operate normally and sustainably, we designed an upload quota system.
Each user receives a 5 GB upload quota fifteen days after registration, which resets on the first day of each month. If you consume your quota in a given month by uploading compliant files to Shionlib, the system will dynamically increase your quota, up to a maximum of 20 GB.
If you need to upload larger files, you can request a quota increase from an administrator.
Note that the system may clear/deduct your upload quota, or even ban your account, under any of the following circumstances:
- The files you upload are determined to contain malware.
- The files you upload are meaningless junk.
- Your account remains inactive for an extended period.
- Other users report your uploads as violating content.
Uploading Files
Format and file-naming guidelines
Your upload must be packaged as a single zip/rar/7z/tar/gzip file, and must not be password-protected.
The filename can be the game’s title/alias and there is no hard requirement. However, we do not recommend stuffing too much information into the filename (e.g., language, translation group/translator, version). Such details can be placed in the resource metadata you’re about to create (explained below).
Start the upload
On the game entry page, click Upload to open the upload dialog. Then select the file you want to upload and click Start to begin.
During the upload, you can track progress via the progress bar in the upload dialog. Please keep your browser and the current tab in the foreground until the upload completes.
You can click Pause at any time to pause the upload. We’ll preserve the upload session; later, click Resume to continue.
We designed for possible interruptions and support resumable uploads. If an upload is interrupted, reopen the upload dialog. You should see Ongoing Uploads; click Continue for the corresponding session and select the same file to resume.
After the file finishes uploading, fill in the file’s metadata below, including platform, languages, and notes. When you’re done, click Submit to finalize the upload.
After submission, the system will process the file: scanning, uploading to S3, and more. You can track the processing progress in the Latest Activity section on the home page.