Update 1: It appears that Twitter has changed their scheme in how they display videos. They are now using a JavaScript method that obfuscates the video URL via a blob mechanic. The URL is only able to be decoded by the user’s web browsing device and cannot be “CURL’d” by a command line. For reasons of which one would still like to learn about downloading m3u8 videos and stitching them together, I’m leaving this up. Cheers.
Update 2: Twitter has made more changes to their m3u8 file format. I’ve updated my script to be more concise.
Hello my bits. Today I will share with you how I downloaded a video from Twitter. But first, let’s try to understand what happens when you load a Tweet that contains a video player, or rather, a thumbnail with a “Play” button on top.
When you click “Play” on a video on Twitter, your web browser (on your phone or desktop) essentially asks Twitter for a small file that contains addresses to small, medium, and large versions of the video you’d like to play. These files are in an m3u8 format. See example 1:
$ curl --silent https://video.twimg.com/amplify_video/123456789010111213/pl/jsuND9234KAOdks.m3u8 #EXTM3U #EXT-X-INDEPENDENT-SEGMENTS #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=320000,RESOLUTION=320x180,CODECS="mp4a.40.2,avc1.4d0015" /amplify_video/123456789010111213/pl/320x180/jsuND9234KAOdks.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=832000,RESOLUTION=640x360,CODECS="mp4a.40.2,avc1.4d001f" /amplify_video/123456789010111213/pl/640x360/jsuND9234KAOdks.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2176000,RESOLUTION=1280x720,CODECS="mp4a.40.2,avc1.640020" /amplify_video/123456789010111213/pl/1280x720/jsuND9234KAOdks.m3u8
As you can see, the file is listing multiple dimensions (resolutions) and network speeds (bandwidth) which my device will automatically choose from. From there on, I chose the URI I want to use to download the video. In this case I chose “/amplify_video/123123123/etc/m3u8”. Upon inspecting this file, I saw that there were several relative paths to “.ts” files, which are segments of the complete video (see an abridged example below)
$ curl --silent https://video.twimg.com/amplify_video/1234567/pl/1280x720/youGottaDerpTheDerp.m3u8 #EXTM3U #EXT-X-VERSION:6 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:3 #EXT-X-PLAYLIST-TYPE:VOD #EXTINF:3.000, /amplify_video/1234567/vid/0/3000/1280x720/Srm-jsuneusl.ts #EXTINF:3.000, /amplify_video/1234567/vid/3000/6000/1280x720/jsuneusl23.ts #EXTINF:3.000, /amplify_video/1234567/vid/6000/9000/1280x720/jsuneusldsa.ts #EXTINF:3.000, /amplify_video/1234567/vid/9000/12000/1280x720/jsuneusl-QPL.ts #EXTINF:3.000, /amplify_video/1234567/vid/33000/36000/1280x720/jsuneusl23ggg.ts #EXTINF:3.000,
At this point I wrote a script to accept a URL to a Twitter m3u8 file (which I usually get via Chrome web developer tools) as an argument, then proceed to run a subshell and download each “.ts” file to a download directory, and then run another subshell to concatenate all video files into a “.mp4” file and move it to a folder in my user’s “Downloads” directory.
Here’s a short script I wrote that should do it all in one shot. This “works” on MacOS:
#!/bin/bash
#
# Hi there. Zan here.
# This works on most newer Macs. USE WITH CAUTION.
# I am not liable if this script ruins or damages your computer, etc.
#
# I DID NOT WRITE ANY SAFETY CHECKS INTO THIS SCRIPT. THIS IS ONLY FOR
#+ EDUCATIONAL PURPOSES
#
# YOU HAVE BEEN WARNED.
# Change this path to whatever you want.
saveDir="~/Downloads/_TWTRvids/"
# Generally, you can add the URL after invoking the script (e.g. './dl_twtr.sh URL'),
#+ but you can also uncomment the next line and add your own URL.
twM3U8="${1}"
# twM3U8="https://vidz.twtr.com/ext_tw_video/SomethingSomethingDARKSIDE.m3u8"
# # # Don't edit the rest of this. It's for your own protection.
#
twID="$(echo "${twM3U8}" | grep -oE "(\w+|-|_){1,}\.m3u8" | cut -d. -f1)"
tmpDir="/private/tmp/"${twID}""
if [[ ! -d "${tmpDir}" ]];then mkdir -p "${tmpDir}";fi
if [[ ! -d "${saveDir}" ]];then mkdir -p "${saveDir}";fi
cd "${tmpDir}";vtw="tw"
# Get relative paths to '.ts' video segments and download them to the current directory
for tVid in $(curl --silent "${twM3U8}" | grep -E "^/.*\.ts$");do
curl -O "https://video.${vtw}img.com${tVid}"
done
for dlVid in $(curl --silent "${twM3U8}" | grep -oE "(\w+|-|_){1,}\.ts");do
cat ${dlVid} >> "${twID}".mp4
done
mv "${twID}".mp4 "${saveDir}"
## Normally macOS will clean up any files in /private/tmp/ after reboots, and such.
#+ However, if you want this script to delete the previously downloaded ".ts" files, uncomment the next line:
# rm -Rf "${tmpDir}"
exit
Easy peezy.
I hope you enjoyed this article!