Linux作为服务器针对使用场景, 有许多需要调优的地方, 本文记录常用优化项.
性能基本调优
通常涉及到/etc/sysctl.conf和/etc/security/limits.conf配置文件的修改.
也可使用命令修改,使用sysctl -p 立即生效.
内核参数设置
sysctl命令可以用来实时的读取/修改内核参数
# 显示所有可用内核参数
sysctl -a
# 加载/etc/sysctl.conf的参数
sysctl -p提高文件描述符限制
soft limit类似于warning, hard limit是真实的最大值限制.
默认的1024偏小, 有两种方式修改:
临时设置
# 查看当前值
ulimit -n
# 临时增加
ulimit -n 65535
# 单个进程的限制为soft limit
# hard limit应小于当前系统打开的文件描述符
# 相应增加nr_open
echo 2000000 > /proc/sys/fs/nr_open
# 系统级限制, 上限为nr_open
echo 1000000 > /proc/sys/fs/file-maxSystemD服务设置
在CentOS7使用SystemD启动的服务不同于CentOS6, 对/etc/security/limits.conf的设置不会生效.
需要修改/etc/systemd/system.conf或/etc/systemd/system.conf.d/tomcat.conf.
# 对应下面几项
DefaultLimitCORE=infinity
DefaultLimitNOFILE=102400
DefaultLimitNPROC=102400常用的命令:
# 查看当前进程的限制
cat /proc/$(pgrep java)/limits
# 查看当前进程句柄数
lsof -p $(pgrep java)|wc -l
## 或者使用
ls /proc/$(pgrep java)/fd|wc -l永久设置
修改 /etc/security/limits.conf,重启以生效.
* soft nofile 65535
* hard nofile 65535
root soft nofile 65535
root hard nofile 65535
# 系统级内核句柄限制
fs.file-max = 1000000查看当前描述符情况
# 列出打开/占用的文件描述符
cat /proc/sys/fs/file-nr
# 三个值分别代表 占用/未使用/最大可用值
# 注: lsof只会列出进程占用
lsof | wc -l
# 要得到线程占用,需要使用
ps -eLf
# 查看某进程限制
cat /proc/[Process ID]/limits增加可用端口数
默认28000, net.ipv4.ip_local_port_range
如果Nginx作代理,需要增加端口范围,否则会出现错误: Cannot assign requested address .
IPv4端口可用数:端口号是16位无符号整数,即65535
# 实时生效
echo 12000 64000 > /proc/sys/net/ipv4/ip_local_port_range
# 永久生效
sysctl -w net.ipv4.ip_local_port_range="12000 64000"可选:最大线程数
一般不需要设置
# 默认31299
echo 100000 > /proc/sys/kernel/threads-max查看当前线程数:
- top, then hit H to view threads
- top -H
- htop
网络调优
通用网络参数
/etc/sysctl.conf
# 系统网络设置
# 生效值取系统和下面TCP设置值的最大值
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216TCP/IP调优
Backlog Queue
最大连接数队列. 可选, 查看kernel日志决定是否需要调整
net.core.somaxconn
net.core.netdev_max_backlog = 300000网络缓冲区大小
# TCP读取缓冲区
# 格式: 最小值/默认值/最大值 字节数
# cat /proc/sys/net/ipv4/tcp_rmem
net.ipv4.tcp_rmem = 4096 87380 16777216
# 发送缓冲区
net.ipv4.tcp_wmem = 4096 65536 16777216
# TCP内存, 对应 low/pressure/high 页大小(4K)
net.ipv4.tcp_mem = 786432 2097152 3145728UDP调优
默认比较受限
#改成8M
sysctl -w net.core.rmem_max=8388608主要参数
Receive-Side Scaling (RSS)
also known as multi-queue receive, distributes network receive processing across several hardware-based receive queues, allowing inbound network traffic to be processed by multiple CPUs.
cat /sys/class/net/eth1/queues/<rx-0>/
ethtool --show-rxfh-indir eth1CLOSE_WAIT 和 TIME_WAIT 解释
TCP是全双工的,任何一端可以是source或destination.
Due to the way TCP/IP works, connections can not be closed immediately. Packets may arrive out of order or be retransmitted after the connection has been closed.
CLOSE_WAIT indicates that the remote endpoint (other side of the connection) has closed the connection.
TIME_WAIT indicates that local endpoint (this side) has closed the connection. The connection is being kept around so that any delayed packets can be matched to the connection and handled appropriately.
The connections will be removed when they time out within four minutes.
tcp_tw_reuse和tcp_tw_recycle
不用开启 net.ipv4.tcp_tw_recycle, 最新内核4.12已结去掉该参数.
连接有incoming和outgoing之分，tcp_tw_reuse仅仅对outgoing有效.
设计协议时,尽量不用让客户端先关闭连接,应该让服务端控制.
TCP/UDP参数
- Socket receive buffer size: Socket send and receive sizes are dynamically adjusted, so they rarely need to be manually edited.
- rmem_default : A kernel parameter that controls the default size of receive buffers used by sockets.
调优常用指标
- Ping 100以下
- 网络延迟50ms以下
- Dns解析尽量快
- 尽量少丢包
- 反向代理优化
调优辅助工具
perf-tools
开源的性能分析工具,基于perf和ftrace.