记一次压测

前言

先说一下测试的背景，本次测试是关于一个回调接口，没有业务逻辑，期望系统的处理能力可以达到3000TPS。

大概的业务调用关系是这样的：

image.png

问题

在压测开始，没觉得的会有什么问题，首先脚本很简单，一个post接口，没有进行任何的参数化，只是单纯的向服务器发送请求，且接口没有任何返回值。

接口的压测策略，设置的梯度加压，最大并发用户就只有4个

这个是TPS，从下图可以看出，是很有规律的波动

image.png

这个是响应时间，同TPS一样，很有规律的波动

image.png

下图为聚合报告，由此可以看此，99%的响应时间均小于4ms，最大响应时间为3s多。

image.png

根据以上结果，开始就觉得，是不是服务器的问题，但是监控服务器，cpu、内存、io等都很正常，这时又把目光转向压力机和jmeter本身。

这时，就想到是不是压力机本身出现了瓶颈，虽说考虑到其配置觉得可能性比较小（32核64G），通过win任务管理器查看机器的性能数据如下：

压力机：

可以看到，在压测时，机器用开启了1130个线程数，71个进程数

image.png

根据测试结果，响应时间99%小于4ms，于是就设置了一个超时时间，设置为30ms，再次执行脚本。这时，得到的结果可以看到大多数为链接失败，但有两种链接失败的情况，于是上网查找以下问题可能的原因：

image.png

最后，看到一个教程说，需要设置以下Advanced-Implementation，选择“java”。于是乎，就尝试的设置一下，结果，后续请求一切正常。

image.png

Abvanced：

java、HttpClint4

jmeter官方文档给出的说明我复制了下来：

HTTP Request
This sampler lets you send an HTTP/HTTPS request to a web server.
It also lets you control whether or not JMeter parses HTML files for images 
and other embedded resources and sends HTTP requests to retrieve them. 
The following types of embedded resource are retrieved:
      images
      applets
      stylesheets (CSS) and resources referenced from those files
      external scripts
      frames, iframes
      background images (body, table, TD, TR)
      background sound
      
      
The default parser is org.apache.jmeter.protocol.http.parser.LagartoBasedHtmlParser.
This can be changed by using the property "htmlparser.className" - see jmeter.properties for details.

If you are going to send multiple requests to the same web server, 
consider using an HTTP Request Defaults Configuration Element so you do not have to enter the same information for each HTTP Request.

Or, instead of manually adding HTTP Requests, you may want to use JMeter's HTTP(S) Test Script Recorder to create them.
This can save you time if you have a lot of HTTP requests or requests with many parameters.

There are two different test elements used to define the samplers:

AJP/1.3 Sampler
    uses the Tomcat mod_jk protocol (allows testing of Tomcat in AJP mode without needing Apache httpd) 
  The AJP Sampler does not support multiple file upload; only the first file will be used.
HTTP Request
    this has an implementation drop-down box, which selects the HTTP protocol implementation to be used:
Java
    uses the HTTP implementation provided by the JVM. 
  This has some limitations in comparison with the HttpClient implementations - see below.
HTTPClient4
    uses Apache HttpComponents HttpClient 4.x.
Blank Value
    does not set implementation on HTTP Samplers, 
  so relies on HTTP Request Defaults if present or on jmeter.httpsampler property defined in jmeter.properties

The Java HTTP implementation has some limitations:

There is no control over how connections are re-used. When a connection is released by JMeter, it may or may not be re-used by the same thread.
The API is best suited to single-threaded usage - various settings are defined via system properties, and therefore apply to all connections.
No support of Kerberos authentication
It does not support client based certificate testing with Keystore Config.
Better control of Retry mechanism
It does not support virtual hosts.
It supports only the following methods: GET, POST, HEAD, OPTIONS, PUT, DELETE and TRACE
Better control on DNS Caching with DNS Cache Manager

个人理解--待完善

关于此处，如何去理解？

问题2

在压测过程中，当线程数和请求数比较少的时候，一切正常，但是随着线程数增加，请求量上升，就开始一直报错。
线程数比较少时，正常；
当线程说多时，报错信息如下：

2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Unexpected end of file from server
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Unexpected end of file from server
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed

解决思路

在网上查询一下报错信息，是说重定向的次数被限制了，但因为自己压测的接口没有重定向，而且前期一直正常，到后面才出现这个问题，这不科学。所以就继续寻找问题。
首先，因为我压测的链路是这样的

SLB --- nginx--kong---docker swarm --具体应用

这时，尝试性的跳过SLB ，直接到nginx，发现压测结果一切正常，这时可以初步判断问题在SLB。

经过查询，发现是因为SLB有一个防攻击的配置，将后面大量的请求，当作了攻击，所以才不行。

至此，问题解决。

以下是重定向请求的一些响应信息，仅作为记录：

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<h1>302 Found</h1>
<p>The requested resource resides temporarily under a different URI.</p>
<hr/>Powered by Tengine</body>
</html>

HTTP/1.1 302 Found
Server: Tengine
Date: Tue, 18 Feb 2020 10:14:20 GMT
Content-Type: text/html
Content-Length: 258
Connection: keep-alive
Set-Cookie: acw_sc__v1=5e4bb8fc37967fa0c859c94a7f9dafd851d8bb95;path=/;Max-Age=1800
cache-control: no-cache, no-store
Pragma: no-cache
Set-Cookie: acw_tc=784c10e715820208607852086e610bc1cb505dd898c94daf023389aca8c4d8;path=/;HttpOnly;Max-Age=2678401
Location: /application-center/api/v1/xxx/xxx/20000001103