逃出七牛云

2022-04-23

备份云中数据

由于众所周知的原因，国内的云存储服务不再可靠。但是我在过往中大量的使用了七牛云的外链，所以需要将七牛中的文件保存起来。

在海外vps或者国内服务仍然可用的情况下，使用七牛的SDK批量备份文件，代码如下：

# -*- coding: utf-8 -*-
# RUN TIME: Python 3.6.5
# CREATION ON: 7/30, 2018
from qiniu import Auth, BucketManager
from secret import key, secret
import urllib.request
import threading
import time,os,shutil

# 七牛相关配置
host = "rs.qbox.me"
q = Auth(key, secret)
bucket = BucketManager(q)

cdn_prefix = 'http://r9rqga72n.bkt.clouddn.com/'
bucket_name = "evink-southerneast"
prefix = 'images'

threads_count = 0


def get_file_list():
    bucket = BucketManager(q)
    # 前缀
    prefix = f"/{prefix}"
    # 列举条目
    limit = 1000
    # 列举出除'/'的所有文件以及以'/'为分隔的所有前缀
    delimiter = None
    # 标记
    marker = None
    ret, eof, info = bucket.list(bucket_name, prefix, marker, limit, delimiter)
    # print(info)
    items: list = ret.get('items')
    # 所有百分号(%25)都被转义为了百分号(%), 将 25 补充回去
    keys = []
    for item in items:
        key = item.get('key')
        idx = key.find('%')
        if idx >= 0:
            key = f'{key[:idx+1]}25{key[idx+1:]}'
        keys.append(key)
    print(keys)
    return keys

def download(key, cb):
    try:
        urllib.request.urlretrieve(cdn_prefix+key, key)
        cb(None, None)
    except Exception as e:
        cb(key, e)
        return False

def download_callback(key, e):
    global threads_count
    if key:
        print(e)
        print(f'Failed: {key}')

    threads_count -= 1



def download_task(key):
    global threads_count
    while True:
        if threads_count < 10:
            break
        time.sleep(1)

    t = threading.Thread(target=download, args=(key, download_callback))
    threads_count += 1
    t.start()

if __name__ == '__main__':
    keys = get_file_list()
    dp = __file__[:-7]
    shutil.rmtree(f'{dp}{imags}')
    os.mkdir(f'{dp}{imags}')
    for key in keys:
        download_task(key)

由于使用了多线程，大量访问可能会导致七牛云拒绝响应，或者下载超时，控制台会打印出失败的文件，需要手动处理，例如:

<urlopen error retrieval incomplete: got only 1048576 out of 1592895 bytes>
Failed: images/2017-01-26/IMG_3434.JPG
<urlopen error retrieval incomplete: got only 0 out of 2400540 bytes>
Failed: images/2021-11-07/2021_11_07_07_16_IMG_4437.jpg
<urlopen error retrieval incomplete: got only 0 out of 3448056 bytes>
Failed: images/2021-11-07/2021_11_06_12_47_IMG_4407.jpg
<urlopen error retrieval incomplete: got only 0 out of 2354957 bytes>
Failed: images/2021-11-07/2021_11_07_11_24_IMG_4438.jpg
<urlopen error retrieval incomplete: got only 0 out of 3071984 bytes>
Failed: images/IMG_0747.JPG

注：如果大量失败，需要手动调整代码以适用于不同的需求。