MD5/SHA images and cacheing

[KazuyaKan]

A good number of groups use the same opening and/or end pages for every chapter of a series they are translating, some groups even use multiple pages on every release. To help alleviate bandwidth and storage usage, it may help to create a MD5/SHA hash of the images uploaded, and those images that have been identified as the same, to automatically be served from a single image. In this manor, you save server space by having only 1 image instead of dozens and dozens, and bandwidth on the user end by the user already having the image in their browser cache.

 

[cmertb]

That will save like 1% of space, or even less.

 

[KazuyaKan]

Space isn't the only consideration. It's also the potential bandwidth that could be saved which can add up over time.

 

[ItsQuote]

my two cents on this one:

- the bandwidth savings would probably be lower than you'd think, since browser caches these days seem to be a lot more ephemeral and volatile with multimedia than with code and markup. not everyone keeps a local cache either (i certainly don't!).
- i know nothing about mangadex's backend, but if we go with the assumption that the architecture is mostly IO-bound (which i think is a safe assumption since the whole site is geared around sharing images) then there may be costs associated with adding a somewhat expensive compute-bound operation to every image that is added.
- it'd be a one-time operation, but every image that's ever been uploaded would also have to be hashed for this to work.
- MD5 is utterly broken and SHA-1 ain't far behind - better algorithms exist, but that adds to the computation requirements.
- however remote, the possibility of collisions would still exist. if an especially weak hash algorithm were chosen, collisions could be engineered by trolls.

for this particular one, i think a better implementation would be allowing group leaders to upload intro/outro pages for their groups, which would be served before and after the chapter within the reader. but that opens up the possibility of getting them twice if the uploader goes off muscle memory and includes the intro/outro in the chapter anyway.