V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Distributions
Ubuntu
Fedora
CentOS
中文资源站
网易开源镜像站
islujw
V2EX  ›  Linux

Linux 批量下载 json 中的多个 url 的文件

  •  
  •   islujw · 2018-08-29 14:39:04 +08:00 · 4015 次点击
    这是一个创建于 2313 天前的主题,其中的信息可能已经有所发展或是发生改变。

    现有一个 json 文件,希望下载其中所有的 “ url ” 的文件:

    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json

    如何在 Linux 上批量下载呢?使用 wget 一个个复制粘贴效率太低了。

    13 条回复    2018-09-23 23:50:10 +08:00
    hanxiV2EX
        1
    hanxiV2EX  
       2018-08-29 14:40:57 +08:00 via Android
    解析 json,对每个 url 执行 wget 命令
    exkernel
        2
    exkernel  
       2018-08-29 14:41:08 +08:00
    sed 一下就出来了, 不会的话用[jq]( https://stedolan.github.io/jq/)
    omph
        3
    omph  
       2018-08-29 14:42:41 +08:00
    优雅一点的,用程序解析 json 中的 url,然后下载
    暴力点的,正则出 url
    fireapp
        4
    fireapp  
       2018-08-29 14:57:02 +08:00 via iPhone
    sed | awk | wget 搞定
    JHerschel
        5
    JHerschel  
       2018-08-29 15:32:13 +08:00
    import json
    file = open('pathToFile')
    file_to_json = json.load(file)
    for a in file_to_json:
    for b in a['assets']:
    print(b['url'])

    wget 或者 requests 搞定。
    hasdream
        6
    hasdream  
       2018-08-29 15:32:39 +08:00
    ```
    curl http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json | grep url | sed -e 's/"url"\ ://;s/\,//g' | awk '{print "curl -L -O "$1}' | sh
    ```
    UnknownR
        7
    UnknownR  
       2018-08-29 16:00:33 +08:00
    作为 windows 党,推荐.net core 和 powershell,先复制 json 内容至剪切板
    [string]$a = get-clipboard
    $b = ConvertFrom-Json -InputObject $a
    $b.assets.url
    就拿到了所有 url,再当成数组一个个遍历,多线程用 wget 下载就行了

    [URL]
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_GL_G004_C010_v03_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D011_D009_SIGNCMP_v15_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LA_A005_C009_v05_t9_6M.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_HK_B005_C011_t9_6M_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_HK_H004_C008_v10_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LA_A009_C009_t9_6M_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_C002_C005_0818SC_001_v01_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_GL_G010_C006_v08_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LW_L001_C006_t9_6M_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D011_C010_v10_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/plate_G002_C002_BG_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_C007_C011_08244D_001_v01_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LA_A006_C008_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D001_C001_v03_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_HK_H004_C010_4k_v01_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LA_A008_C004_ALT_v33_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D002_C003_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_C007_C004_0824AJ_001_v01_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D001_C005_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_HK_H004_C013_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D008_C010_v04_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b1-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b1-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b1-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b1-4.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b2-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b2-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b2-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b2-4.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b3-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b3-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b3-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b4-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b4-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b4-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b5-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b5-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b5-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b6-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b6-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b6-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b6-4.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b7-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b7-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b7-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b10-4.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b8-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b8-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b9-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b9-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b9-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b10-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b10-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b10-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b8-1.mov
    ps1aniuge
        8
    ps1aniuge  
       2018-08-30 10:56:14 +08:00
    win linux 通用 powershell:
    ps1aniuge
        9
    ps1aniuge  
       2018-08-30 10:58:17 +08:00
    $a = Invoke-RestMethod http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json |ConvertFrom-Json
    $a.assets.url

    和楼上的道理一样。只不过这个网址现在好像,啥命令都不行了,502 Bad Gateway 了。
    mingyun
        10
    mingyun  
       2018-09-01 12:24:16 +08:00
    @hasdream 命令行玩的 6
    ps1aniuge
        11
    ps1aniuge  
       2018-09-01 17:19:07 +08:00
    $url = 'http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json'
    $a = Invoke-RestMethod -Uri $url
    $a.assets.url

    测试通过

    我不知道网址内容是否变了,反正现在不需要搞 json 了。
    iwtbauh
        12
    iwtbauh  
       2018-09-05 08:21:35 +08:00 via Android
    jshon 了解一下

    #!/bin/bash

    url=`jshon -e xxxx -e url1 -u`
    curl -s ...
    abcbc
        13
    abcbc  
       2018-09-23 23:50:10 +08:00
    用 jq 来处理很简单:

    curl -L http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json | jq -r '.[].assets | .[].url' | xargs -I {} curl -LO {}
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   2581 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 19ms · UTC 03:09 · PVG 11:09 · LAX 19:09 · JFK 22:09
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.