大家好，又见面了，我是你们的朋友全栈君。如果您正在找激活码,请点击查看最新教程,关注关注公众号 “全栈程序员社区” 获取激活教程,可能之前旧版本教程已经失效.最新Idea2022.1教程亲测有效,一键激活。
Jetbrains全系列IDE使用 1年只要46元 售后保障 童叟无欺
记录一次正式环境服务报错排查记录。
某日被通知线上服务告警，错误日志全是 Timeout waiting for connection
首先梳理项目架构，项目很简单，就是一个使用axis2构建的webserice的客户端， Axis2版本为1.5.5
开始从此段报错入手排查，定位到 MultiThreadedHttpConnectionManager 这个类的 doGetConnection 方法
private HttpConnection doGetConnection(HostConfiguration hostConfiguration,
long timeout) throws ConnectionPoolTimeoutException {
HttpConnection connection = null;
int maxHostConnections = this.params.getMaxConnectionsPerHost(hostConfiguration);
int maxTotalConnections = this.params.getMaxTotalConnections();
synchronized (connectionPool) {
// we clone the hostConfiguration
// so that it cannot be changed once the connection has been retrieved
hostConfiguration = new HostConfiguration(hostConfiguration);
HostConnectionPool hostPool = connectionPool.getHostPool(hostConfiguration, true);
WaitingThread waitingThread = null;
boolean useTimeout = (timeout > 0);
long timeToWait = timeout;
long startWait = 0;
long endWait = 0;
while (connection == null) {
if (shutdown) {
throw new IllegalStateException("Connection factory has been shutdown.");
}
// happen to have a free connection with the right specs
//
if (hostPool.freeConnections.size() > 0) {
connection = connectionPool.getFreeConnection(hostConfiguration);
// have room to make more
//
} else if ((hostPool.numConnections < maxHostConnections)
&& (connectionPool.numConnections < maxTotalConnections)) {
connection = connectionPool.createConnection(hostConfiguration);
// have room to add host connection, and there is at least one free
// connection that can be liberated to make overall room
//
} else if ((hostPool.numConnections < maxHostConnections)
&& (connectionPool.freeConnections.size() > 0)) {
connectionPool.deleteLeastUsedConnection();
connection = connectionPool.createConnection(hostConfiguration);
// otherwise, we have to wait for one of the above conditions to
// become true
//
} else {
// TODO: keep track of which hostConfigurations have waiting
// threads, so they avoid being sacrificed before necessary
try {
if (useTimeout && timeToWait <= 0) {
throw new ConnectionPoolTimeoutException("Timeout waiting for connection");
}
if (LOG.isDebugEnabled()) {
LOG.debug("Unable to get a connection, waiting..., hostConfig=" + hostConfiguration);
}
if (waitingThread == null) {
waitingThread = new WaitingThread();
waitingThread.hostConnectionPool = hostPool;
waitingThread.thread = Thread.currentThread();
} else {
waitingThread.interruptedByConnectionPool = false;
}
if (useTimeout) {
startWait = System.currentTimeMillis();
}
hostPool.waitingThreads.addLast(waitingThread);
connectionPool.waitingThreads.addLast(waitingThread);
connectionPool.wait(timeToWait);
} catch (InterruptedException e) {
if (!waitingThread.interruptedByConnectionPool) {
LOG.debug("Interrupted while waiting for connection", e);
throw new IllegalThreadStateException(
"Interrupted while waiting in MultiThreadedHttpConnectionManager");
}
// Else, do nothing, we were interrupted by the connection pool
// and should now have a connection waiting for us, continue
// in the loop and let's get it.
} finally {
if (!waitingThread.interruptedByConnectionPool) {
// Either we timed out, experienced a "spurious wakeup", or were
// interrupted by an external thread. Regardless we need to
// cleanup for ourselves in the wait queue.
hostPool.waitingThreads.remove(waitingThread);
connectionPool.waitingThreads.remove(waitingThread);
}
if (useTimeout) {
endWait = System.currentTimeMillis();
timeToWait -= (endWait - startWait);
}
}
}
}
}
return connection;
}得到几个关键变量：
连接池最大连接数 maxTotalConnections=20；
当前连接主机（或相同主机）最大连接数 maxHostConnections=2；
连接池总空闲连接数 connectionPool.freeConnections；
当前连接主机空闲连接数 hostPool.freeConnections；
得到结论一：
当前主机连接数达到最大值，且当前连接主机空闲连接数为0时，获取连接超时时会抛出 Timeout waiting for connection 异常。
到此，初步怀疑某次请求时未正常释放占用的连接，或未回收到空闲连接池中。
继续排查日志，一直找到第一次抛出 Timeout waiting for connection 异常的时候再往上查看还有没有其他错误日志。于是发现了 Transport error: 503 Error: Service Unavailable 这个异常
直接进入源码
/**
* Used to handle the HTTP Response
*
* @param msgContext - The MessageContext of the message
* @param method - The HTTP method used
* @throws IOException - Thrown in case an exception occurs
*/
private void handleResponse(MessageContext msgContext,
HttpMethodBase method) throws IOException {
int statusCode = method.getStatusCode();
log.trace("Handling response - " + statusCode);
if (statusCode == HttpStatus.SC_OK) {
// Save the HttpMethod so that we can release the connection when cleaning up
msgContext.setProperty(HTTPConstants.HTTP_METHOD, method);
processResponse(method, msgContext);
} else if (statusCode == HttpStatus.SC_ACCEPTED) {
// Since we don't expect any content with a 202 response, we must release the connection
method.releaseConnection();
} else if (statusCode == HttpStatus.SC_INTERNAL_SERVER_ERROR ||
statusCode == HttpStatus.SC_BAD_REQUEST) {
// Save the HttpMethod so that we can release the connection when cleaning up
msgContext.setProperty(HTTPConstants.HTTP_METHOD, method);
Header contenttypeHeader =
method.getResponseHeader(HTTPConstants.HEADER_CONTENT_TYPE);
String value = null;
if (contenttypeHeader != null) {
value = contenttypeHeader.getValue();
}
OperationContext opContext = msgContext.getOperationContext();
if(opContext!=null){
MessageContext inMessageContext =
opContext.getMessageContext(WSDLConstants.MESSAGE_LABEL_IN_VALUE);
if(inMessageContext!=null){
inMessageContext.setProcessingFault(true);
}
}
if (value != null) {
processResponse(method, msgContext);
}
Object isTransportNonBlocking = msgContext.getProperty(
MessageContext.TRANSPORT_NON_BLOCKING);
if (isTransportNonBlocking != null && (Boolean)isTransportNonBlocking) {
throw new AxisFault(Messages.getMessage("transportError",
String.valueOf(statusCode),
method.getStatusText()));
}
} else {
throw new AxisFault(Messages.getMessage("transportError",
String.valueOf(statusCode),
method.getStatusText()));
}
}根据我之前读的源码我知道 method.releaseConnection(); 这个方法将会最终调用 connectionPool.freeConnection(conn); 即生成空闲连接放入总连接池中。
而 msgContext.setProperty(HTTPConstants.HTTP_METHOD, method); 这个方法将由其他地方获取调用并最终调用的仍是 method.releaseConnection();
综上可知 返回码为503或者任意其他上述代码中未处理的返回码时都不会将占用连接重置为空闲连接放入总连接池中。
到此可知服务告警原因：
因为服务端发生了我们未知的操作，导致客户端获取请求时的返回码为503而占用了所有的当前主机连接数（2个）而不会释放重置为空闲连接放入连接池
于是后续连接从连接池获取连接时均超时抛出异常 Timeout waiting for connection
发布者：全栈程序员-用户IM，转载请注明出处：https://javaforall.cn/194542.html原文链接：https://javaforall.cn
【正版授权，激活自己账号】： Jetbrains全家桶Ide使用，1年售后保障，每天仅需1毛
【官方授权 正版激活】： 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...
