Menu
Zero One Labs
  • Home
  • About
  • Scripts
  • Snippets
Zero One Labs

How I downloaded a video from Twitter using Curl

Posted on August 22, 2017February 17, 2019 by Zan

Update 1: It appears that Twitter has changed their scheme in how they display videos. They are now using a JavaScript method that obfuscates the video URL via a blob mechanic. The URL is only able to be decoded by the user’s web browsing device and cannot be “CURL’d” by a command line. For reasons of which one would still like to learn about downloading m3u8 videos and stitching them together, I’m leaving this up. Cheers.

Update 2: Twitter has made more changes to their m3u8 file format. I’ve updated my script to be more concise.

Hello my bits. Today I will share with you how I downloaded a video from Twitter. But first, let’s try to understand what happens when you load a Tweet that contains a video player, or rather, a thumbnail with a “Play” button on top.

When you click “Play” on a video on Twitter, your web browser (on your phone or desktop) essentially asks Twitter for a small file that contains addresses to small, medium, and large versions of the video you’d like to play. These files are in an m3u8 format. See example 1:

$ curl --silent https://video.twimg.com/amplify_video/123456789010111213/pl/jsuND9234KAOdks.m3u8
#EXTM3U
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=320000,RESOLUTION=320x180,CODECS="mp4a.40.2,avc1.4d0015"
/amplify_video/123456789010111213/pl/320x180/jsuND9234KAOdks.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=832000,RESOLUTION=640x360,CODECS="mp4a.40.2,avc1.4d001f"
/amplify_video/123456789010111213/pl/640x360/jsuND9234KAOdks.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2176000,RESOLUTION=1280x720,CODECS="mp4a.40.2,avc1.640020"
/amplify_video/123456789010111213/pl/1280x720/jsuND9234KAOdks.m3u8

As you can see, the file is listing multiple dimensions (resolutions) and network speeds (bandwidth) which my device will automatically choose from. From there on, I chose the URI I want to use to download the video. In this case I chose “/amplify_video/123123123/etc/m3u8”. Upon inspecting this file, I saw that there were several relative paths to “.ts” files, which are segments of the complete video (see an abridged example below)

$ curl --silent https://video.twimg.com/amplify_video/1234567/pl/1280x720/youGottaDerpTheDerp.m3u8
#EXTM3U
#EXT-X-VERSION:6
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:3
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:3.000,
/amplify_video/1234567/vid/0/3000/1280x720/Srm-jsuneusl.ts
#EXTINF:3.000,
/amplify_video/1234567/vid/3000/6000/1280x720/jsuneusl23.ts
#EXTINF:3.000,
/amplify_video/1234567/vid/6000/9000/1280x720/jsuneusldsa.ts
#EXTINF:3.000,
/amplify_video/1234567/vid/9000/12000/1280x720/jsuneusl-QPL.ts
#EXTINF:3.000,
/amplify_video/1234567/vid/33000/36000/1280x720/jsuneusl23ggg.ts
#EXTINF:3.000,

At this point I wrote a script to accept a URL to a Twitter m3u8 file (which I usually get via Chrome web developer tools) as an argument, then proceed to run a subshell and download each “.ts” file to a download directory, and then run another subshell to concatenate all video files into a “.mp4” file and move it to a folder in my user’s “Downloads” directory.

Here’s a short script I wrote that should do it all in one shot. This “works” on MacOS:

#!/bin/bash
#
# Hi there. Zan here.
# This works on most newer Macs. USE WITH CAUTION. 
# I am not liable if this script ruins or damages your computer, etc.
#
#  I DID NOT WRITE ANY SAFETY CHECKS INTO THIS SCRIPT. THIS IS ONLY FOR 
#+ EDUCATIONAL PURPOSES
#
# YOU HAVE BEEN WARNED.

# Change this path to whatever you want.
saveDir="~/Downloads/_TWTRvids/"

#  Generally, you can add the URL after invoking the script (e.g. './dl_twtr.sh URL'),
#+ but you can also uncomment the next line and add your own URL.
twM3U8="${1}"
# twM3U8="https://vidz.twtr.com/ext_tw_video/SomethingSomethingDARKSIDE.m3u8"

# # # Don't edit the rest of this. It's for your own protection.
#
twID="$(echo "${twM3U8}" | grep -oE "(\w+|-|_){1,}\.m3u8" | cut -d. -f1)"
tmpDir="/private/tmp/"${twID}""

if [[ ! -d "${tmpDir}" ]];then mkdir -p "${tmpDir}";fi
if [[ ! -d "${saveDir}" ]];then mkdir -p "${saveDir}";fi
cd "${tmpDir}";vtw="tw"

# Get relative paths to '.ts' video segments and download them to the current directory
for tVid in $(curl --silent "${twM3U8}" | grep -E "^/.*\.ts$");do 
  curl -O "https://video.${vtw}img.com${tVid}"
done

for dlVid in $(curl --silent "${twM3U8}" | grep -oE "(\w+|-|_){1,}\.ts");do 
  cat ${dlVid} >> "${twID}".mp4
done
mv "${twID}".mp4 "${saveDir}"

## Normally macOS will clean up any files in /private/tmp/ after reboots, and such.
#+ However, if you want this script to delete the previously downloaded ".ts" files, uncomment the next line:
# rm -Rf "${tmpDir}"
exit

Easy peezy.

I hope you enjoyed this article!

  • bash
  • concatenate
  • curl
  • export
  • ts
  • twitter
  • video files
  • Categories

    • Articles
    • Bash
    • Mac
    • PackageMaker
    • Python
    • Scripts
    • Terminal
    • Uncategorized
    My LinkedIn
    ©2023 Zero One Labs | Powered by WordPress & Superb Themes