app抓包

环境搭建

Android模拟器安装

image

官网下载夜神模拟器安装

抓包工具安装

image

appium安装

https://github.com/appium/appium-desktop/releases/tag/v1.11.0

mitmproxy安装

下载安装包, 直接点击下一步即可安装
https://github.com/mitmproxy/mitmproxy/releases/

装好之后配置一下环境变量就行了

也可以直接使用pip install mitmproxy

安装证书

在cmd中输入mitmdump, 可以看到mitmdump已经启动了, 在监听8080端口

C:\Users\IIce>mitmdump
Proxy server listening at http://*:8080

打开模拟器, 配置代理

查看一下pc的ip

以太网适配器 以太网:

   连接特定的 DNS 后缀 . . . . . . . : North-Class.com
   本地链接 IPv6 地址. . . . . . . . : fe80::68d7:38a8:2729:4d97%6
   IPv4 地址 . . . . . . . . . . . . : 192.168.100.243
   子网掩码  . . . . . . . . . . . . : 255.255.255.0
   默认网关. . . . . . . . . . . . . : 192.168.100.250

image

配置好以后, 打开浏览器, 输入baidu.com进行查看
此时会弹出证书问题, 点继续即可

输入mitm.it
选择相应的版本进行安装

image

此时再访问网站就不会有证书问题了

docker安装

根据配置, 二选一

docker-toolbox

https://docs.docker.com/toolbox/toolbox_install_windows/
下载docker-toolbox, 双击进行安装即可

如果安装快要结束时报错
IPersistFile：Save 失败，代码0x80070005 拒绝访问
需要检查杀毒软件

安装成功后会出现三个图标

image

双击Docker Quickstart Terminal图标，启动一个终端
会下载一个boot2docker.iso文件,如果下载较慢的话,可以复制链接自行下载,
下载完成后复制到目录中即可

image

如果出现Unable to start the VM: C:\Program Files\Oracle\VirtualBox\VBoxManage.exe startvm default --type headless failed:卸载掉Oracle VM Virtualbox安装最新版即可
https://www.virtualbox.org/wiki/Downloads

完成后会出现

image

输入docker run hello-world

image

Docker for Windows

https://docs.docker.com/docker-for-windows/install/

下载后双击安装即可

如果安装卡顿, 需要检查杀毒软件, 因为会修改注册表和启动项等

启动时如果报错
“Hardware assisted virtualization and data execution protection must be enabled in the BIOS”

需要开启虚拟化和Hyper-V

如果都开启还如法启动
参考https://www.e-learn.cn/content/wangluowenzhang/589447

两个混装可能出现的错误
https://blog.csdn.net/qq_35852248/article/details/80925154

设置加速器
https://cr.console.aliyun.com/cn-hangzhou/instances/mirrors

fiddler设置

image

手机连接配置
查看pc端IP

...

以太网适配器 以太网:

   连接特定的 DNS 后缀 . . . . . . . : North-Class.com
   本地链接 IPv6 地址. . . . . . . . : fe80::f44c:fb33:30bf:5c57%18
   IPv4 地址 . . . . . . . . . . . . : 192.168.100.248
   子网掩码  . . . . . . . . . . . . : 255.255.255.0
   默认网关. . . . . . . . . . . . . : 192.168.100.250
...

设置代理,服务器主机名是pc端IPv4地址

image

设置完成后浏览器访问主机IP+端口

image

App应用在开启抓包工具后无法联网问题

http://www.imooc.com/article/251500

fiddler 不能抓包的方法

https://testerhome.com/topics/11462?from=singlemessage

豆果美食菜谱抓取

在模拟器中下载并安装豆果美食
设置代理准备进行数据抓包
打开fiddler和豆果美食

image

点击菜谱分类

image

点击标签进入详情

image

抓包分析,http://api.douguo.net/recipe/flatcatalogs这个url返回的是菜谱分类

image

http://api.douguo.net/recipe/v2/search/0/20返回的是详情
综合最佳,收藏最多和做过最多使用的都是这一个url,只是提交的参数不同

# 0:综合最佳   2: 收藏最多   3:做过最多
"order": "0",

编码实现

请求头

首先将请求头共用的部分提取出来,注释掉的都是可以不用提交的

def handle_reques(url, data):
    header = {
        "client": "4",
        "version": "6934.2",
        "device": "OPPO R11",
        "sdk": "22,5.1.1",
        "imei": "866174010942858",
        "channel": "baidu",
        # "mac": " 6A:07:15:F0:34:85",
        "resolution": "1280*720",
        "dpi": "1.5",
        # "android-id": "6a0715f034851883",
        # "pseudo - id": "5f0348518836a071",
        "brand": "OPPO",
        "scale": "1.5",
        "timezone": "28800",
        "language": "zh",
        "cns": "3",
        "carrier": "CHINA+MOBILE",
        # "imsi": "460071060715240",
        "user-agent": "Mozilla/5.0 (Linux; Android 5.1.1; OPPO R11 Build/NMF26X) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/39.0.0.0 Mobile Safari/537.36",
        "reach": "1",
        "newbie": "1",
        "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
        "Accept-Encoding": "gzip, deflate",
        "Connection": "Keep-Alive",
        # "Cookie": "duid=59159842",
        "Host": "api.douguo.net",
        # "Content-Length": "74",
    }

    response = requests.post(url=url, headers=header, data=data)
    return response

菜谱分类


from multiprocessing import Queue

queue_list = Queue()

def handle_index():
    url = "http://api.douguo.net/recipe/flatcatalogs"
    data = {
        "client": "4",
        # "_session": "1552715432169866174010942858",
        # "v": "1503650468",
        # "_vs": "0",   0 和 2305都可以
        "_vs": "2305",

    }

    response = handle_reques(url=url, data=data)
    # print(response.text)
    response_to_dict = json.loads(response.text)

    for item in response_to_dict['result']['cs']:
        for item_1 in item['cs']:
            for item_2 in item_1['cs']:
                data_2 = {
                    "client": "4",
                    # "_session": "1552715831226866174010942858",
                    "keyword": item_2['name'],
                    # 0:综合最佳   2: 收藏最多   3:做过最多
                    "order": "0",
                    "_vs": "400",
                }
                queue_list.put(data_2)

详情

def handle_caipu_list(data):
    print("当前处理:", data['keyword'])
    caipu_list_url = 'http://api.douguo.net/recipe/v2/search/0/20'
    caipu_list_response = handle_reques(url=caipu_list_url, data=data)
    response_to_dict = json.loads(caipu_list_response.text)
    handle_caipu_detail(data, response_to_dict)

    count=0
    while response_to_dict['result']['end'] == 0:
        count+=1
        caipu_list_url = 'http://api.douguo.net/recipe/v2/search/{}/20'.format(count*20)
        caipu_list_response = handle_reques(url=caipu_list_url, data=data)
        response_to_dict = json.loads(caipu_list_response.text)
        handle_caipu_detail(data, response_to_dict)

具体做法

def handle_caipu_detail(data, response_to_dict):

    for item in response_to_dict['result']['list']:
        caipu_info = {}
        caipu_info['shicai'] = data['keyword']

        if item['type'] == 13:
            caipu_info['author'] = item['r']['an']
            caipu_info['shicai_id'] = item['r']['id']  # 查看详细操作步骤时使用
            caipu_info['describe'] = item['r']['cookstory']
            caipu_info['caipu_name'] = item['r']['n']
            caipu_info['zuoliao_list'] = item['r']['major']

            detail_url = 'http://api.douguo.net/recipe/detail/' + str(caipu_info['shicai_id'])
            detail_data = {
                "client": "4",
                # "_session": "1552715831226866174010942858",
                "author_id": "0",
                "_vs": "2803",
                "_ext": '{"query":{"kw":' + data["keyword"] + ',"src":"2803","type":"13","id":' + str(
                    caipu_info["shicai_id"]) + '}}',
            }

            detail_response = handle_reques(url=detail_url, data=detail_data)
            # print(detail_response.text)
            detail_response_to_dict = json.loads(detail_response.text)

            caipu_info['tips'] = detail_response_to_dict['result']['recipe']['tips']
            caipu_info['cook_step'] = detail_response_to_dict['result']['recipe']['cookstep']

            print('当前入库:', caipu_info['caipu_name'])
            mongo_info.insert_item(caipu_info)

        else:
            continue

入库

import pymongo

from pymongo.collection import Collection


class Connect_Mongo:
    def __init__(self):
        self.client = pymongo.MongoClient()
        self.db_data = self.client['dou_guo_mei_shi']

    def insert_item(self, item):
        db_collection = Collection(self.db_data, 'mei_shi')
        db_collection.insert_one(item)


mongo_info = Connect_Mongo()

多线程测试

if __name__ == '__main__':
    handle_index()
    # print(queue_list.qsize())
    # handle_caipu_list(queue_list.get())
    pool = ThreadPoolExecutor()

    while queue_list.qsize() > 0:
        pool.submit(handle_caipu_list, queue_list.get())

安装android-sdk

http://tools.android-studio.org/index.php/sdk
下载安装即可

配置环境变量

变量	值
ANDROID_HOME(新建)	G:\Program Files (x86)\Android\android-sdk
Path(添加)	%ANDROID_HOME%\tools
Path(添加)	%ANDROID_HOME%\platform-tools

运行SDK Manager.exe

image

安卓版本勾选最新版的即可,兼容旧版本

image

安装完成后打开cmd,输入adb,可以看到adb版本

Android Debug Bridge version 1.0.40
Version 28.0.2-5303910
Installed as G:\Program Files (x86)\Android\android-sdk\platform-tools\adb.exe

global options:
 -a         listen on all network interfaces, not just localhost
 -d         use USB device (error if multiple devices connected)
 -e         use TCP/IP device (error if multiple TCP/IP devices available)
 -s SERIAL  use device with given serial (overrides $ANDROID_SERIAL)
 -t ID      use device with given transport id
 -H         name of adb server host [default=localhost]
 -P         port of adb server [default=5037]
 -L SOCKET  listen on given socket for adb server [default=tcp:localhost:5037]

升级夜神模拟器的adb

将android-sdk\platform-tools中的三个adb文件拷贝到模拟器安装目录下

image

将adb.exe复制一份,覆盖掉原来的nox_adb.exe,
开启模拟器的开发者选项
重启模拟器,打开cmd

C:\Users\lenovo>adb devices
List of devices attached
127.0.0.1:52001 device

模拟器已连接上了

uiautomatorviewer

文件位置D:\Program Files (x86)\Android\android-sdk\tools\uiautomatorviewer.bat

双击运行, 将黑窗口最小化,不要关闭

点击生成屏幕快照, 可以使用鼠标查看元素的信息

appium

启动参数配置
http://www.testclass.net/appium

{
  "platformName": "Android",
  "deviceName": "127.0.0.1:52001",
  "platformVersion": "5.1.1",
  "appPackage": "com.tal.kaoyan",
  "appActivity": "com.tal.kaoyan.ui.activity.SplashActivity"
}

appPackage和appActivity 获取
使用aapt.exe dump badging来获取

D:\Program Files (x86)\Android\android-sdk\build-tools\28.0.3>aapt.exe dump badging F:\BrowserDownload\kaoyanbang_3.3.7beta.243.apk

package: name='com.tal.kaoyan' versionCode='92' versionName='3.3.7beta' compileSdkVersion='28' compileSdkVersionCodename='9'
sdkVersion:'16'
...

launchable-activity: name='com.tal.kaoyan.ui.activity.SplashActivity'  label='' icon=''

...

考研帮测试

pip install Appium-Python-Client

{
  "platformName": "Android",
  "deviceName": "127.0.0.1:52001",
  "platformVersion": "5.1.1",
  "appPackage": "com.tal.kaoyan",
  "appActivity": "com.tal.kaoyan.ui.activity.SplashActivity",
  "noReset": true
}

import time

from appium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

cap = {
    "platformName": "Android",
    "deviceName": "127.0.0.1:52001",
    "platformVersion": "5.1.1",
    "appPackage": "com.tal.kaoyan",
    "appActivity": "com.tal.kaoyan.ui.activity.SplashActivity",
    "noReset": True
}

name = ""
pwd = ""

driver = webdriver.Remote("http://localhost:4723/wd/hub", cap)


def get_size():
    x = driver.get_window_size()['width']
    y = driver.get_window_size()['height']
    return (x, y)


try:
    # 是否跳过
    if WebDriverWait(driver, 3).until(
            lambda x: x.find_element_by_xpath("//android.widget.TextView[@resource-id='com.tal.kaoyan:id/tv_skip']")):
        driver.find_element_by_xpath("//android.widget.TextView[@resource-id='com.tal.kaoyan:id/tv_skip']").click()
except:
    pass

try:
    if WebDriverWait(driver, 3).until(lambda x: x.find_element_by_xpath(
            "//android.widget.EditText[@resource-id='com.tal.kaoyan:id/login_email_edittext']")):
        driver.find_element_by_xpath(
            "//android.widget.EditText[@resource-id='com.tal.kaoyan:id/login_email_edittext']").send_keys(name)
        driver.find_element_by_xpath(
            "//android.widget.EditText[@resource-id='com.tal.kaoyan:id/login_password_edittext']").send_keys(pwd)
        driver.find_element_by_xpath(
            "//android.widget.Button[@resource-id='com.tal.kaoyan:id/login_login_btn']").click()
except:
    pass

try:
    # 隐私协议
    if WebDriverWait(driver, 3).until(
            lambda x: x.find_element_by_xpath("//android.widget.TextView[@resource-id='com.tal.kaoyan:id/tv_title']")):
        driver.find_element_by_xpath("//android.widget.TextView[@resource-id='com.tal.kaoyan:id/tv_agree']").click()
        driver.find_element_by_xpath(
            "//android.support.v7.widget.RecyclerView[@resource-id='com.tal.kaoyan:id/date_fix']/android.widget.RelativeLayout[3]").click()
except:
    pass

# 点击研讯
if WebDriverWait(driver, 3).until(lambda x: x.find_element_by_xpath(
        "//android.support.v7.widget.RecyclerView[@resource-id='com.tal.kaoyan:id/date_fix']/android.widget.RelativeLayout[3]/android.widget.LinearLayout[1]/android.widget.ImageView[1]")):
    driver.find_element_by_xpath(
        "//android.support.v7.widget.RecyclerView[@resource-id='com.tal.kaoyan:id/date_fix']/android.widget.RelativeLayout[3]/android.widget.LinearLayout[1]/android.widget.ImageView[1]").click()

    l = get_size()

    x1 = int(l[0] * 0.5)
    y1 = int(l[1] * 0.75)
    y2 = int(l[1] * 0.25)

    # 滑动操作
    while True:
        driver.swipe(x1, y1, x1, y2)
        time.sleep(0.5)

整体操作和selenium差不多

抖音粉丝抓取

先找一个分享链接
https://www.douyin.com/share/user/96578108671

浏览器打开, 进行查看, 可以看到数字被进行了混淆,
字符文件链接https://s3.bytecdn.cn/ies/resource/falcon/douyin_falcon/static/font/iconfont_9eb9a50.woff

在线字体查看http://fontstore.baidu.com/static/editor/index.html
将下载的字体文件上传到网站, 就能看到字符和数字之间的关系了

分享页面内容抓取

import re
import requests
import time
from lxml import etree

from douyin.handle_mongo import get_task


def handle_decode(input_data, share_web_url, task):
    search_douyin_str = re.compile('抖音ID：')
    regex_list = [
        {'name': [' &#xe603; ', ' &#xe60d; ', ' &#xe616; '], 'value': 0},
        {'name': [' &#xe602; ', ' &#xe60e; ', ' &#xe618; '], 'value': 1},
        {'name': [' &#xe605; ', ' &#xe610; ', ' &#xe617; '], 'value': 2},
        {'name': [' &#xe604; ', ' &#xe611; ', ' &#xe61a; '], 'value': 3},
        {'name': [' &#xe606; ', ' &#xe60c; ', ' &#xe619; '], 'value': 4},
        {'name': [' &#xe607; ', ' &#xe60f; ', ' &#xe61b; '], 'value': 5},
        {'name': [' &#xe608; ', ' &#xe612; ', ' &#xe61f; '], 'value': 6},
        {'name': [' &#xe60a; ', ' &#xe613; ', ' &#xe61c; '], 'value': 7},
        {'name': [' &#xe60b; ', ' &#xe614; ', ' &#xe61d; '], 'value': 8},
        {'name': [' &#xe609; ', ' &#xe615; ', ' &#xe61e; '], 'value': 9},
    ]

    for i1 in regex_list:
        for i2 in i1['name']:
            input_data = re.sub(i2, str(i1['value']), input_data)
    share_web_html = etree.HTML(input_data)
    douyin_info = {}
    douyin_info['nick_name'] = \
    share_web_html.xpath("//div[@class='personal-card']/div[@class='info1']//p[@class='nickname']/text()")[0]
    if 'douyin_id' in task:
        douyin_info['douyin_id'] = task['douyin_id']
    else:
        douyin_id = ''.join(
            share_web_html.xpath("//div[@class='personal-card']/div[@class='info1']/p[@class='shortid']/i/text()"))
        if douyin_id == '':
            try:
                douyin_info['douyin_id'] = re.sub(search_douyin_str, '', share_web_html.xpath(
                    "//div[@class='personal-card']/div[@class='info1']/p[@class='shortid']/text()")[0]).strip()
            except:
                douyin_info['douyin_id'] = '无数据'
        else:
            douyin_info['douyin_id'] = douyin_id

    try:
        douyin_info['job'] = share_web_html.xpath(
            "//div[@class='personal-card']/div[@class='info2']/div[@class='verify-info']/span[@class='info']/text()")[
            0].strip()
    except:
        pass
    douyin_info['describe'] = \
    share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='signature']/text()")[0].replace(
        '\n', ',')
    douyin_info['location'] = \
    share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='extra-info']/span[1]/text()")
    douyin_info['xingzuo'] = \
    share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='extra-info']/span[2]/text()")
    douyin_info['follow_count'] = share_web_html.xpath(
        "//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='focus block']//i[@class='icon iconfont follow-num']/text()")[
        0].strip()
    fans_value = ''.join(share_web_html.xpath(
        "//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']//i[@class='icon iconfont follow-num']/text()"))
    unit = share_web_html.xpath(
        "//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']/span[@class='num']/text()")
    if unit[-1].strip() == 'w':
        douyin_info['fans'] = str((int(fans_value) / 10)) + 'w'
    like = ''.join(share_web_html.xpath(
        "//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='liked-num block']//i[@class='icon iconfont follow-num']/text()"))
    unit = share_web_html.xpath(
        "//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='liked-num block']/span[@class='num']/text()")
    if unit[-1].strip() == 'w':
        douyin_info['like'] = str(int(like) / 10) + 'w'
    douyin_info['from_url'] = share_web_url

    print(douyin_info)



def handle_douyin_web_share(task):
    share_web_url = 'https://www.douyin.com/share/user/' + task
    print(share_web_url)
    share_web_header = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36'
    }
    share_web_response = requests.get(url=share_web_url, headers=share_web_header)
    handle_decode(share_web_response.text, share_web_url, task)

if __name__ == '__main__':
    # task = get_task("share_id")
    handle_douyin_web_share("88445518961")

https://www.douyin.com/share/user/88445518961
{'nick_name': 'Dear-迪丽热巴', 'douyin_id': '274110380', 'job': '演员', 'describe': '先定一个能达到的小目标，比方说来句签名', 'location': [], 'xingzuo': [], 'follow_count': '0', 'fans': '5046.8w', 'like': '13527.7w', 'from_url': 'https://www.douyin.com/share/user/88445518961'}

粉丝抓取

前提: 登录状态, 最新版本

抓取个人的粉丝

import sys
import time
from selenium.webdriver.support.ui import WebDriverWait
from appium import webdriver

desired_caps = {}
desired_caps['platformName'] = 'Android'
desired_caps['deviceName'] = '127.0.0.1:62001'
desired_caps['platformVersion'] = '5.1.1'
desired_caps['appPackage'] = 'com.ss.android.ugc.aweme'
desired_caps['appActivity'] = 'com.ss.android.ugc.aweme.splash.SplashActivity'
desired_caps['noReset'] = True
desired_caps['unicodeKeyboard'] = True
desired_caps['resetKeyboard'] = True

driver = webdriver.Remote('http://localhost:4723/wd/hub', desired_caps)


def get_size(driver):
    x = driver.get_window_size()['width']
    y = driver.get_window_size()['height']
    return (x, y)


def handle_douyin(driver):
    try:
        # 点击搜索
        while WebDriverWait(driver, 10).until(lambda x: x.find_element_by_xpath(
                "//android.widget.LinearLayout[@resource-id='com.ss.android.ugc.aweme:id/aps']")):
            driver.find_element_by_xpath(
                "//android.widget.LinearLayout[@resource-id='com.ss.android.ugc.aweme:id/aps']").click()
            break
    except:
        print("找不到搜索按钮")

    # 定位搜索框
    if WebDriverWait(driver, 3).until(lambda x: x.find_element_by_xpath(
            "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']")):
        # 获取douyin_id进行搜索
        driver.find_element_by_xpath(
            "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']").send_keys('706942127')
        while driver.find_element_by_xpath(
                "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']").text != '706942127':
            driver.find_element_by_xpath(
                "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']").send_keys('706942127')
            time.sleep(0.1)
    # 点击搜索
    driver.find_element_by_xpath("//android.widget.TextView[@resource-id='com.ss.android.ugc.aweme:id/afr']").click()

    # 点击用户标签
    if WebDriverWait(driver, 3).until(lambda x: x.find_element_by_xpath("//android.widget.TextView[@text='用户']")):
        driver.find_element_by_xpath("//android.widget.TextView[@text='用户']").click()

    # 点击头像
    if WebDriverWait(driver, 3).until(lambda x: x.find_element_by_xpath(
            "/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout[2]/android.widget.RelativeLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.support.v4.view.ViewPager/android.widget.LinearLayout/android.widget.FrameLayout/android.view.View/android.support.v7.widget.RecyclerView/android.widget.RelativeLayout[1]/android.widget.RelativeLayout[1]/android.widget.ImageView[2]")):
        driver.find_element_by_xpath(
            "/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout[2]/android.widget.RelativeLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.support.v4.view.ViewPager/android.widget.LinearLayout/android.widget.FrameLayout/android.view.View/android.support.v7.widget.RecyclerView/android.widget.RelativeLayout[1]/android.widget.RelativeLayout[1]/android.widget.ImageView[2]").click()
    # 点击粉丝按钮
    if WebDriverWait(driver, 3).until(lambda x: x.find_element_by_id("com.ss.android.ugc.aweme:id/aj1")):
        driver.find_element_by_id("com.ss.android.ugc.aweme:id/aj1").click()

    l = get_size(driver)
    x1 = int(l[0] * 0.5)
    y1 = int(l[1] * 0.75)
    y2 = int(l[1] * 0.25)
    while True:
        if '没有更多了' in driver.page_source:
            break
        driver.swipe(x1, y1, x1, y2)
        time.sleep(0.5)


if __name__ == '__main__':
    handle_douyin(driver)

Appium会先打开抖音, 然后点击搜索图标, 获取搜索栏进行输入, 点击搜索按钮, 点击用户, 点击头像, 点击粉丝, 模拟滑动, 直到没有粉丝了

粉丝入库

使用mitmdump来将数据存入数据库
mitmdump -s xxx.py

import json

from douyin.handle_mongo import save_task


def response(flow):
    if 'aweme/v1/user/follower/list/' in flow.request.url:
        for user in json.loads(flow.response.text)['followers']:
            douyin_info = {}
            douyin_info['share_id'] = user['uid']
            douyin_info['douyin_id'] = user['short_id']
            douyin_info['nickname'] = user['nickname']
            save_task(douyin_info)

这样在滑动粉丝时, 就会将粉丝的信息添加进数据库

多设备抓取

设置一下appium
appium客户端设置 udid
appium服务端设置 bootstrapPort

image

需要开启多个模拟器或者多台真机

import multiprocessing
import sys
import time
from selenium.webdriver.support.ui import WebDriverWait
from appium import webdriver

# desired_caps = {}
# desired_caps['platformName'] = 'Android'
# desired_caps['deviceName'] = '127.0.0.1:62001'
# desired_caps['platformVersion'] = '5.1.1'
# desired_caps['appPackage'] = 'com.ss.android.ugc.aweme'
# desired_caps['appActivity'] = 'com.ss.android.ugc.aweme.splash.SplashActivity'
# desired_caps['noReset'] = True
# desired_caps['unicodeKeyboard'] = True
# desired_caps['resetKeyboard'] = True
#
# driver = webdriver.Remote('http://localhost:4723/wd/hub', desired_caps)


def get_size(driver):
    x = driver.get_window_size()['width']
    y = driver.get_window_size()['height']
    return (x, y)


def handle_douyin(driver):
    while True:
        # 定位搜索框
        while WebDriverWait(driver, 10).until(lambda x: x.find_element_by_xpath(
                "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']")):
            # 获取douyin_id进行搜索
            driver.find_element_by_xpath(
                "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']").send_keys('706942127')
            while driver.find_element_by_xpath(
                    "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']").text != '706942127':
                driver.find_element_by_xpath(
                    "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']").send_keys('706942127')
                time.sleep(0.1)
                break
            break
        # 点击搜索
        driver.find_element_by_xpath("//android.widget.TextView[@resource-id='com.ss.android.ugc.aweme:id/afr']").click()

        # 点击用户标签
        if WebDriverWait(driver, 10).until(lambda x: x.find_element_by_xpath("//android.widget.TextView[@text='用户']")):
            driver.find_element_by_xpath("//android.widget.TextView[@text='用户']").click()

        # 点击头像
        if WebDriverWait(driver, 10).until(lambda x: x.find_element_by_xpath(
                "/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout[2]/android.widget.RelativeLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.support.v4.view.ViewPager/android.widget.LinearLayout/android.widget.FrameLayout/android.view.View/android.support.v7.widget.RecyclerView/android.widget.RelativeLayout[1]/android.widget.RelativeLayout[1]/android.widget.ImageView[2]")):
            driver.find_element_by_xpath(
                "/hierarchy/android.widget.FrameLayout/android.widget.LinearLayout/android.widget.FrameLayout/android.widget.FrameLayout/android.widget.FrameLayout[2]/android.widget.RelativeLayout/android.widget.FrameLayout/android.widget.RelativeLayout/android.support.v4.view.ViewPager/android.widget.LinearLayout/android.widget.FrameLayout/android.view.View/android.support.v7.widget.RecyclerView/android.widget.RelativeLayout[1]/android.widget.RelativeLayout[1]/android.widget.ImageView[2]").click()
        # 点击粉丝按钮
        if WebDriverWait(driver, 10).until(lambda x: x.find_element_by_id("com.ss.android.ugc.aweme:id/aj1")):
            driver.find_element_by_id("com.ss.android.ugc.aweme:id/aj1").click()

        l = get_size(driver)
        x1 = int(l[0] * 0.5)
        y1 = int(l[1] * 0.75)
        y2 = int(l[1] * 0.25)
        while True:
            if '没有更多了' in driver.page_source:
                break
            elif '还没有粉丝' in driver.page_source:
                break
            else:
                driver.swipe(x1, y1, x1, y2)
                time.sleep(0.5)

        driver.find_element_by_id("com.ss.android.ugc.aweme:id/n7").click()
        driver.find_element_by_id("com.ss.android.ugc.aweme:id/n7").click()
        driver.find_element_by_xpath(
            "//android.widget.EditText[@resource-id='com.ss.android.ugc.aweme:id/afo']").clear()


def handle_appium(device, port):
    caps = {}
    caps["platformName"] = "Android"
    caps["deviceName"] = device
    caps["platformVersion"] = "5.1.1"
    caps["appPackage"] = "com.ss.android.ugc.aweme"
    caps["appActivity"] = "com.ss.android.ugc.aweme.splash.SplashActivity"
    caps["noReset"] = True
    caps["unicodeKeyboard"] = True
    caps["resetKeyboard"] = True
    caps["udid"] = device

    driver = webdriver.Remote('http://localhost:'+str(port)+'/wd/hub', caps)

    try:
        # 点击搜索图标
        while WebDriverWait(driver, 10).until(lambda x: x.find_element_by_xpath(
                "//android.widget.LinearLayout[@resource-id='com.ss.android.ugc.aweme:id/aps']")):
            driver.find_element_by_xpath(
                "//android.widget.LinearLayout[@resource-id='com.ss.android.ugc.aweme:id/aps']").click()
            break
    except:
        print("找不到搜索按钮")

    handle_douyin(driver)

if __name__ == '__main__':
    m_list = []

    devices_list = ['127.0.0.1:62001', '127.0.0.1:62025']
    for device in range(len(devices_list)):
        port = 4723+2*device
        m_list.append(multiprocessing.Process(target=handle_appium, args=(devices_list[device], port)))

    for m in m_list:
        m.start()

    for m in m_list:
        m.join()

devices_list 里的数据可以通过adb devices查看

C:\Users\IIce>adb devices
List of devices attached
127.0.0.1:62001 device
127.0.0.1:62025 device

抖音视频抓取

从抖音 APP 分享个人信息，复制链接，获得个人主页地址，示例：
https://www.iesdouyin.com/share/user/58862693224

视频接口解析

使用 Chrome 抓包，获取视频列表接口的请求信息

image

链接参数解析

https://www.iesdouyin.com/web/api/v2/aweme/post/?
user_id=58862693224 #   分享链接中的id
count=21            #   视频个数
max_cursor=0        #   翻页使用的参数, 第一次是0, 往后会根据上次的返回结果进行变化
aid=1128            #   固定值
_signature=laPLvBAVyX-c77Gpje7Ys5Wjy6   #   签名值，由签名算法计算
dytk=66cb5d220e0e48ed9195a7f62ac32764   #   不知道是啥, 网页中可直接提取

获取签名算法

打开控制台, 搜索_signature

image

定位_bytedAcrawler

image

定位 douyin_falcon:node_modules/byted-acrawler/dist/runtime

image

定位 __M.define

image

分析签名算法的执行逻辑

① 定义 __M对象，及其define 和require 函数
② 执行 __M.define("douyin_falcon:node_modules/byted-acrawler/dist/runtime......" 这段代码
③ 执行_bytedAcrawler = require("douyin_falcon:node_modules/byted-acrawler/dist/runtime")

④ 计算签名值 _signature = _bytedAcrawler.sign(user_id)

我们可以自己编写一个html文件, 访问这个文件来得到_signature
淘宝chromedriver镜像

源码地址

关于水印

视频链接的url分两种

https://aweme.snssdk.com/aweme/v1/play/?video_id=v0300f6d0000bj81rdqrh6f3j18kvnpg&line=0&ratio=540p&media_type=4&vr_type=0&improve_bitrate=0
https://aweme.snssdk.com/aweme/v1/playwm/?video_id=v0300f6d0000bj81rdqrh6f3j18kvnpg&line=0&ratio=540p&media_type=4&vr_type=0&improve_bitrate=0

区别:

第一个请求的是play,第二个请求的是playwm
第一个网站是打不开的, 第二个可以打开
都可以使用requests来获取
第一个是无水印的!!!

通过Postman测试, 发现只保留一个video_id即可

image

参数说明

image

has_more来判断是否需要翻页
max_cursor下次请求时需要携带的, 首次为 0

参考

使用 NodeJS 提供抖音签名算法服务

无水印解析

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 203,098评论 5赞 476
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 85,213评论 2赞 380
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 149,960评论 0赞 336
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 54,519评论 1赞 273
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 63,512评论 5赞 364
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 48,533评论 1赞 281
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 37,914评论 3赞 395
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 36,574评论 0赞 256
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 40,804评论 1赞 296
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 35,563评论 2赞 319
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 37,644评论 1赞 329
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 33,350评论 4赞 318
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 38,933评论 3赞 307
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 29,908评论 0赞 19
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 31,146评论 1赞 259
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 42,847评论 2赞 349
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 42,361评论 2赞 342

app抓包

环境搭建

Android模拟器安装

抓包工具安装

appium安装

mitmproxy安装

安装证书

docker安装

docker-toolbox

Docker for Windows

fiddler设置

App应用在开启抓包工具后无法联网问题

fiddler 不能抓包的方法

豆果美食菜谱抓取

编码实现

请求头

菜谱分类

详情

具体做法

入库

多线程测试

安装android-sdk

升级夜神模拟器的adb

uiautomatorviewer

appium

考研帮测试

抖音粉丝抓取

分享页面内容抓取

粉丝抓取

粉丝入库

多设备抓取

抖音视频抓取

视频接口解析

获取签名算法

分析签名算法的执行逻辑

关于水印

参数说明

参考

推荐阅读更多精彩内容