以前给客户写的一个批量采集软件失效了,今天又过来找我我就看了一下,作者采集那里没有问题,主要是出现在视频解析那里,以前的解析接口失效了。
客户那边是做视频采集搬运的,UC大鱼的视频质量还不错,没有水印,记录一下过程。
首先从手机UC浏览器分享了一个作者链接过来:
http://a.mp.uc.cn/media.html?mid=eafce118a8d04a7fb49edb26e67a16f4&client=ucweb&uc_param_str=frdnsnpfvecpntnwprdsssnikt&uc_biz_str=S:custom%7CC:iflow_ncmt
翻了一下页面是分页加载的,这种一般都是以Json的形式返回数据。
打开控制台切换到network页看一下网页请求,这里我有一个小技巧,一般这种分页加载的网页我会先尝试过滤一下“page=”字符看一下。
过滤”page=”字符接口一下子就出来了:
http://ff.dayu.com/contents/author/eafce118a8d04a7fb49edb26e67a16f4?biz_id=1002&_size=8&_page=1&_order_type=published_at&status=1&_fetch=1&uc_param_str=frdnsnpfvecpntnwprdsssnikt&_=1547577957727
接口请求的时候主要注意一下下面两个参数,其他默认就好。
eafce118a8d04a7fb49edb26e67a16f4 作者ID
page= 分页
以上UC大鱼作者的所有视频链接都采集出来了,接下来就是把视频直链解析出来。
我们先从作者页打开一个视频播放页面,刷新拦截一下看视频链接是从哪个接口反回来的。
http://v.mp.uc.cn/video.html?uc_param_str=frdnsnpfvecpntnwprdssskt&from=media#!wm_cid=267399668026835968!!wm_aid=26fa9499c6dd453ebbfc4880b71aaa75!!wm_id=eafce118a8d04a7fb49edb26e67a16f4
我翻了一下请求发现他的视频链接是从下面接口反回来的:
http://v.mp.uc.cn/api/vps?token=Hh%2FdqhK4JBpmDIcJ9CLIWQ%3D%3D&ums_id=97aff448e4250d3d&wm_cid=267399668026835968&wm_id=eafce118a8d04a7fb49edb26e67a16f4
有几个乱码看着不方便,Url解码一下得到下面数据:
http://v.mp.uc.cn/api/vps?token=Hh/dqhK4JBpmDIcJ9CLIWQ==&ums_id=97aff448e4250d3d&wm_cid=267399668026835968&wm_id=eafce118a8d04a7fb49edb26e67a16f4
接口返回了以下数据:
{"status":0,"message":"","data":{"url":"http://dy-frontend.video.ums.uc.cn/w/1547547793/video/wemedia/eafce118a8d04a7fb49edb26e67a16f4/444065d12d690e266b5493b90e514f99-1337752270-6-0-2.mp4?auth_key=1547581955-b0bca68f99dc4b42a0c79e978259fea2-0-504c180f7f177d2423bb13078f54a940","duration":196,"size":"21057680","sign":"db4c7d3ccc6a5ee0729a7c24da432ac3"}}
这个接口有四个参数:
token=Hh/dqhK4JBpmDIcJ9CLIWQ==
ums_id=97aff448e4250d3d
wm_cid=267399668026835968
wm_id=eafce118a8d04a7fb49edb26e67a16f4
这些参数是从哪来的一般来说我都会先尝试搜索一下网页源码,这里我没有搜索到,还是得往前面看一下是不是从别的接口返回来的。(这里我说一下抓包我推荐用Fiddler,使用Fiddler全局搜索很容易能搜索到这些参数是哪个接口返回的,就不用像我这样子一个接口一个接口去看,我因为是在老家破电脑用不了Fiddler所以就只能用浏览器的开发工具。)
往前面翻了一下发现除了token,其他几个参数都在下面这个接口:
http://ff.dayu.com/contents/267399668026835968?biz_id=1002&_fetch_author=1&_fetch_incrs=1
访问后返回了下面数据:
{"data":{"created_at":"2019-01-15T18:21:37.806+0800","updated_at":"2019-01-16T02:26:22.759+0800","content_id":"267399668026835968","format_type":1002,"biz_id":1002,"category":"游戏","content_type":1001,"origin_id":"26fa9499c6dd453ebbfc4880b71aaa75","title":"呆妹儿教你吃鸡没有小姐姐: 女人的嘴, 骗人的鬼","sub_title":"呆妹儿教你吃鸡没有小姐姐: 女人的嘴, 骗人的鬼","status":1,"audit_status":1,"author_id":"eafce118a8d04a7fb49edb26e67a16f4","shard_id":"eafce118a8d04a7fb49edb26e67a16f4","source":"1","cover_url":"http://image.uc.cn/s/wemedia/s/upload/2019/6768873b289bf9771a40f711e3faab2e.jpg","published_at":"2019-01-15T18:21:37.780+0800","body":{"videos":[{"file_name":"呆妹儿教你吃鸡没有小姐姐:女人的嘴,骗人的鬼.mp4","ums_id":"97aff448e4250d3d","page_url":"http://v.ums.uc.cn/video/v_97aff448e4250d3d.html","thumbnail":"http://img.ums.uc.cn/smart_snapshot/1547547811-1337752270-1750100306","duration":196,"width":1920,"height":1080,"bitrate":"6217","size":152837816,"status":"1","description":"呆妹儿教你吃鸡没有小姐姐: 女人的嘴, 骗人的鬼","yk_vid":"1003595284","extra":{"material_id":"","cover_from":1,"video_from":""}}]},"_extra":{"is_original":1,"covers":[],"vertical_cover_url":"","origin_cover_url":"http://image.uc.cn/s/wemedia/s/upload/2019/1b917374febb8eb03bec32147fd87c5d.jpg","customize_tags":["呆妹儿","绝地求生","吃鸡","主播","搞笑"],"second_title":"","xss_item_id":"12803331104905082604","origin_author":"","quality_score":5},"_product_info":{"settings":{"weixin_promote":false,"article_activity_id":""},"reproduce":{"open_reproduce":0},"award":{"open_award":false},"ad":{"is_show_ad":true,"video_share_status":1}},"_author":{"created_at":"2017-05-20T12:42:42.000+0800","updated_at":"2019-01-15T10:59:12.000+0800","author_id":"eafce118a8d04a7fb49edb26e67a16f4","author_name":"侠游","author_alias":"财务学长","introduction":"游戏视频创作者,玩好玩的游戏,做有趣的视频","special_introduction":"","author_type":"0","active":1,"author_level":"5","category":"游戏","avatar_url":"http://image.uc.cn/s/wemedia/s/upload/2018/27095d1b0a1d36a474233d9d5409c3e6x200x200x27.png","source":"3","biz_id":1001,"open_award":"false","open_tbk":"0","super_v":"0","xss_seed_id":"594528","extra_map":{"is_original":0,"has_opened_ad":true,"wm_weixin_id":"xiayousp"}},"_incrs":{"play":0,"A":0,"B":0,"cover_return":0,"C":0,"D":0,"dislike":0,"release":0,"buy":0,"show":0,"fav_total":0,"click_7day":0,"recommend":199,"click_total":0,"vote_keng":0,"fav":0,"share":0,"like":1,"vote_jian":0,"vote_diao":0,"click":0,"play_total":0,"click2":7,"click3":0,"comment":0,"comment_total":0,"click1":0}}}
现在我们就差一个token参数
翻了好一会才发现token原来是下面这个接口服务器响应头返回的Set-Cookie:
http://v.mp.uc.cn/video.html?uc_param_str=frdnsnpfvecpntnwprdssskt&from=media
以上有点乱,怎么获取作者全部视频就不说了,我们来盘一下UC大鱼获取视频链接的过程。
首先访问一个视频链接:http://v.mp.uc.cn/video.html?uc_param_str=frdnsnpfvecpntnwprdssskt&from=media#!wm_cid=267399668026835968!!wm_aid=26fa9499c6dd453ebbfc4880b71aaa75!!wm_id=eafce118a8d04a7fb49edb26e67a16f4
然后取上面视频链接的wm_aid参数拼接出下面链接:http://ff.dayu.com/contents/267399668026835968?biz_id=1002&_fetch_author=1&_fetch_incrs=1
访问后返回数据如下:
{"data":{"created_at":"2019-01-15T18:21:37.806+0800","updated_at":"2019-01-16T04:09:27.721+0800","content_id":"267399668026835968","format_type":1002,"biz_id":1002,"category":"游戏","content_type":1001,"origin_id":"26fa9499c6dd453ebbfc4880b71aaa75","title":"呆妹儿教你吃鸡没有小姐姐: 女人的嘴, 骗人的鬼","sub_title":"呆妹儿教你吃鸡没有小姐姐: 女人的嘴, 骗人的鬼","status":1,"audit_status":1,"author_id":"eafce118a8d04a7fb49edb26e67a16f4","shard_id":"eafce118a8d04a7fb49edb26e67a16f4","source":"1","cover_url":"http://image.uc.cn/s/wemedia/s/upload/2019/6768873b289bf9771a40f711e3faab2e.jpg","published_at":"2019-01-15T18:21:37.780+0800","body":{"videos":[{"file_name":"呆妹儿教你吃鸡没有小姐姐:女人的嘴,骗人的鬼.mp4","ums_id":"97aff448e4250d3d","page_url":"http://v.ums.uc.cn/video/v_97aff448e4250d3d.html","thumbnail":"http://img.ums.uc.cn/smart_snapshot/1547547811-1337752270-1750100306","duration":196,"width":1920,"height":1080,"bitrate":"6217","size":152837816,"status":"1","description":"呆妹儿教你吃鸡没有小姐姐: 女人的嘴, 骗人的鬼","yk_vid":"1003595284","extra":{"material_id":"","cover_from":1,"video_from":""}}]},"_extra":{"is_original":1,"covers":[],"vertical_cover_url":"","origin_cover_url":"http://image.uc.cn/s/wemedia/s/upload/2019/1b917374febb8eb03bec32147fd87c5d.jpg","customize_tags":["呆妹儿","绝地求生","吃鸡","主播","搞笑"],"second_title":"","xss_item_id":"12803331104905082604","origin_author":"","quality_score":5},"_product_info":{"settings":{"weixin_promote":false,"article_activity_id":""},"reproduce":{"open_reproduce":0},"award":{"open_award":false},"ad":{"is_show_ad":true,"video_share_status":1}},"_author":{"created_at":"2017-05-20T12:42:42.000+0800","updated_at":"2019-01-15T10:59:12.000+0800","author_id":"eafce118a8d04a7fb49edb26e67a16f4","author_name":"侠游","author_alias":"财务学长","introduction":"游戏视频创作者,玩好玩的游戏,做有趣的视频","special_introduction":"","author_type":"0","active":1,"author_level":"5","category":"游戏","avatar_url":"http://image.uc.cn/s/wemedia/s/upload/2018/27095d1b0a1d36a474233d9d5409c3e6x200x200x27.png","source":"3","biz_id":1001,"open_award":"false","open_tbk":"0","super_v":"0","xss_seed_id":"594528","extra_map":{"is_original":0,"has_opened_ad":true,"wm_weixin_id":"xiayousp"}},"_incrs":{"play":0,"A":0,"B":0,"cover_return":0,"C":0,"D":0,"dislike":0,"release":0,"buy":0,"show":0,"fav_total":0,"click_7day":0,"recommend":208,"click_total":0,"vote_keng":0,"fav":0,"share":0,"like":1,"vote_jian":0,"vote_diao":0,"click":0,"play_total":0,"click2":17,"click3":0,"comment":0,"comment_total":0,"click1":0}}}
然后我们从上面返回数据分别取出:
ums_id":"97aff448e4250d3d"; 对应视频直链接口的ums_id参数
content_id":"267399668026835968" 对应视频直链接口的wm_cid参数
author_id":"eafce118a8d04a7fb49edb26e67a16f4" 对应视频直链接口的wm_id参数
然后再访问下面链接:
http://v.mp.uc.cn/video.html?uc_param_str=frdnsnpfvecpntnwprdssskt&from=media
取出服务器响应头返回的Set-Cookie:
vpstoken=yqgmEDSvsJJeQMbupNTAVA%3D%3D; Path=/ 对应视频直链接口的token参数
拼接出视频直链的接口:
http://v.mp.uc.cn/api/vps?token=yqgmEDSvsJJeQMbupNTAVA%3D%3D&ums_id=97aff448e4250d3d&wm_cid=267399668026835968&wm_id=eafce118a8d04a7fb49edb26e67a16f4