was messing around with streaming files straight from object storage, mostly because i wanted to know how stuff like arXiv or X shows you a giant pdf or video without the server downloading the whole thing first
my first try was the dumb version. download the whole object to a volume, then read it. fine for small files. for big ones you're paying for disk, waiting for the last byte before anything happens, and if the connection dies at 90% you start over
turns out the sdk already hands you a stream and i was literally writing it to a file for no reason. so i just started reading the body as it comes in
then byte ranges. s3, gcs, azure blob, r2 all support the Range header, so you grab a few MB at a time and keep a few requests going at once. if one chunk fails you retry that chunk, not the whole file
the part that surprised me ngl, pdfs, zips and parquet keep their index at the end, so you read the last few KB first and only fetch the bits you actually need
this is how i ended up picturing it. each lift is one GET, the 3 pockets are basically all the memory the worker has, and if a pocket's still full the lift just sits there and waits
tldr stop treating the object like a file, it's just bytes you can ask for in pieces
haven't figured out how they cache all of this closer to users yet, that's next
if you've done this in prod or know a better way, hit me up on LinkedIn