Compare commits

...

166 Commits

Author SHA1 Message Date
刘祥超
77bb1cf14e 版本更改为0.4.8.1 2022-06-20 09:34:06 +08:00
刘祥超
274284dbe1 静态文件分发也支持压缩、WebP转换 2022-06-19 11:39:21 +08:00
刘祥超
cd6d7221e8 不限制206 Partial Content两次写入文件的时间差 2022-06-18 20:05:09 +08:00
刘祥超
b1d0c8852e 如果缓存条件支持206 Partial Content,则第一次加载时自动尝试从分片缓存中读取内容 2022-06-18 19:31:10 +08:00
刘祥超
4c4033bb56 TCP负载均衡实现流量限制,达到限制后,关闭连接 2022-06-17 21:49:15 +08:00
刘祥超
6d642b75f6 延长预热超时时间 2022-06-16 20:32:11 +08:00
刘祥超
eb47e3a08c 删除缓存的时同时删除相关的缓存(压缩格式、WebP格式、http和https互换URL) 2022-06-15 12:54:56 +08:00
刘祥超
d82d16e28d 修复内容为空时无法缓存的Bug 2022-06-09 20:26:36 +08:00
刘祥超
b2fc785543 WAF规则中增加${requestURL}参数 2022-06-09 19:44:11 +08:00
刘祥超
189e3342ce 将缓存maxOpenFiles最小值从2改为4 2022-06-09 19:12:29 +08:00
刘祥超
885defbf31 优化nftables相关代码 2022-06-09 19:12:10 +08:00
刘祥超
74f1bf330d DNS解析库默认使用Go原生库 2022-06-07 11:49:38 +08:00
刘祥超
ad843d9d10 修复源站从http跳转到https导致无限循环的问题 2022-06-07 11:25:09 +08:00
刘祥超
13e718742d 节点状态上报时增加时间戳字段 2022-06-07 11:23:40 +08:00
刘祥超
771eff8fb1 增加刷新、预热缓存任务管理 2022-06-05 17:15:02 +08:00
刘祥超
20d7e0b1bf ACME申请证书时如果找不到Token,则直接跳过执行后面请求 2022-06-02 15:34:14 +08:00
刘祥超
e6c7bbec06 修复一个源站主备切换不灵敏的问题/WebSocket也支持源站主备自动切换 2022-05-23 20:01:26 +08:00
刘祥超
be61ef89fe fix typo 2022-05-23 16:15:02 +08:00
刘祥超
3d7d8f1e63 优化代码 2022-05-23 11:34:58 +08:00
刘祥超
a4fb465a19 在严格匹配域名模式下仍然可以通过节点IP进行健康检查 2022-05-23 11:17:53 +08:00
刘祥超
96c725c13d 增加LICENSE和README 2022-05-22 11:36:43 +08:00
刘祥超
7635def2fa 修正自动使用本地防火墙延长封禁时间逻辑 2022-05-21 22:15:11 +08:00
刘祥超
b704a73338 修复一个日志typo 2022-05-21 22:04:23 +08:00
刘祥超
123b5f5969 自动将同集群节点IP加入白名单/尝试使用本地防火墙提升黑名单连接封锁效率 2022-05-21 21:32:10 +08:00
刘祥超
eea2037444 优化验证码失败次数统计 2022-05-21 20:02:35 +08:00
刘祥超
4e6d2fa5ea WAF策略中增加验证码相关定制设置 2022-05-21 11:17:53 +08:00
刘祥超
14bb131e8d WAF CAPTCHA:刷新验证码页面也算入校验失败次数 2022-05-20 11:56:06 +08:00
刘祥超
31814bb54c 忽略301, 302, 303, 307, 308响应中没有Location的错误提示 2022-05-19 20:16:40 +08:00
刘祥超
49b8fd6e97 健康检查增加是否记录访问日志选项 2022-05-19 17:13:20 +08:00
刘祥超
a9d31a2e35 增加edge-node accesslog命令,用来在本地查看访问日志 2022-05-18 23:14:57 +08:00
刘祥超
298cef7f05 缩短指标统计队列长度 2022-05-18 21:41:34 +08:00
刘祥超
9bdd9a433c 实现基础的DDoS防护 2022-05-18 21:03:51 +08:00
刘祥超
45620dcdb7 计算CC的时候不再跨时间范围累积 2022-05-12 21:48:33 +08:00
刘祥超
84a5d38b0b 在启动时检查节点时间戳是否和API节点一致,如果不一致则上报 2022-05-12 21:07:45 +08:00
刘祥超
e812b3fcf6 X-Forwarded-For中包含当前客户端的IP 2022-05-08 16:56:10 +08:00
刘祥超
1bd16fa1d3 往硬盘刷数据时不统计maxOpenFiles 2022-05-07 22:02:41 +08:00
刘祥超
f3ea4957be fix typo 2022-05-05 11:01:03 +08:00
刘祥超
04da107c94 路由规则可以单独设置UAM(仅企业版可用) 2022-05-04 20:32:25 +08:00
刘祥超
e77de69a15 节点增加DNS解析库类型设置 2022-05-04 16:40:25 +08:00
刘祥超
e88eda56f5 自动替换Location中的地址时检查域名是否为当前域名 2022-05-04 10:48:14 +08:00
刘祥超
d6ceccc52e 增加基准测试 2022-05-01 10:40:19 +08:00
刘祥超
cd948ac68c 实现新的gzip库提升gzip性能 2022-04-30 22:22:30 +08:00
刘祥超
9eac8afa3d 停止节点时systemctl命令不阻塞当前进程 2022-04-26 12:22:00 +08:00
刘祥超
fb3610966a 白名单中的IP不受请求限制的影响 2022-04-25 11:11:25 +08:00
刘祥超
a6673449db 修改版本为0.4.8 2022-04-25 11:11:03 +08:00
刘祥超
1d33284b2e 节点状态中增加本地防火墙名称 2022-04-22 14:59:00 +08:00
刘祥超
09ec4507be 优化WAF日志逻辑 2022-04-21 19:44:19 +08:00
刘祥超
a15f0e3051 默认记录WAF的条件从检测到攻击改为所有匹配WAF规则集的请求 2022-04-21 19:02:17 +08:00
刘祥超
67b982c67a 节点状态记录是否检查到本地防火墙 2022-04-21 18:14:53 +08:00
刘祥超
5e1f8f305c 调整默认缓存容量 2022-04-21 09:40:20 +08:00
刘祥超
adfdd5f1b6 强制记录攻击日志 2022-04-21 09:40:05 +08:00
刘祥超
553deda20b 优化代码 2022-04-20 20:05:16 +08:00
刘祥超
7225cef18e 修正文件缓存“慢”打开文件耗时阈值 2022-04-20 18:41:53 +08:00
刘祥超
0e5aef923d 文件缓存增加自动限速/提升本地缓存数据库写入和查询速度 2022-04-20 18:23:26 +08:00
刘祥超
5c54a47587 优化节点停止逻辑 2022-04-20 10:32:40 +08:00
刘祥超
6c294d0282 更新相关库 2022-04-20 10:22:25 +08:00
刘祥超
d6eec0fa52 去除错误提示中的程序文件名中的Workspace目录 2022-04-18 16:31:48 +08:00
刘祥超
9c79152efe 修复UDP服务端口变化时导致的死循环 2022-04-18 15:39:42 +08:00
刘祥超
8765f3a0c6 修复服务配置变化可能导致的死锁 2022-04-18 15:39:02 +08:00
刘祥超
8272fe7fa5 优化缓存相关代码 2022-04-15 14:23:06 +08:00
刘祥超
1636ef8891 优化缓存相关代码 2022-04-14 10:25:34 +08:00
刘祥超
ed0c562b2e 优化缓存相关代码 2022-04-14 09:36:02 +08:00
刘祥超
08a307d8d8 优化代码 2022-04-13 19:24:23 +08:00
刘祥超
5e2bf493b2 优化代码 2022-04-12 21:43:19 +08:00
刘祥超
065dda1dbf 优化基准测试 2022-04-10 21:21:45 +08:00
刘祥超
715e79c3e1 优化代码 2022-04-10 18:46:44 +08:00
刘祥超
2490d6f9d8 优化本地日志 2022-04-10 15:54:30 +08:00
刘祥超
8c4e7129f3 如果Header中Location字段含有跟源站一样的地址,则自动修改为当前域名 2022-04-09 20:37:05 +08:00
刘祥超
3f621c5af0 优化ttlcache 2022-04-09 18:49:52 +08:00
刘祥超
dba9c2c47d 优化ttlcache 2022-04-09 18:44:51 +08:00
刘祥超
ded2f98cce 优化ttlcache 2022-04-09 18:28:22 +08:00
刘祥超
02469f2886 增加基准测试 2022-04-09 10:02:09 +08:00
刘祥超
2121ebe2e0 优化代码 2022-04-08 16:11:05 +08:00
刘祥超
51b9ce6f48 修改个别错误级别 2022-04-08 15:25:53 +08:00
刘祥超
bed19bf844 优化代码/CPU监控信息增加CPU逻辑核数 2022-04-07 09:45:55 +08:00
刘祥超
20a77021f6 将RPC Canceled错误级别调整为警告 2022-04-07 09:45:27 +08:00
刘祥超
c5d061dbe2 优化代码 2022-04-05 15:10:32 +08:00
刘祥超
bd9c8b3d0e 保存L2节点数据时同时记录缓存时间 2022-04-05 11:00:55 +08:00
刘祥超
221d7e6434 缓存文件实现Sendfile 2022-04-04 19:45:57 +08:00
刘祥超
a59007a249 优化代码 2022-04-04 18:25:54 +08:00
刘祥超
6998d3468b 优化代码/商业版支持L2节点 2022-04-04 12:06:53 +08:00
刘祥超
d7ec36b12c 修改一处日志级别 2022-04-02 15:34:00 +08:00
刘祥超
79f4db5a6c response body buffer默认改为16k 2022-04-02 10:41:02 +08:00
刘祥超
048c6f213b 集群可以设置WebP策略 2022-04-01 16:18:15 +08:00
刘祥超
59faf95885 限制WebP可转换的最大长度为128M(非ChunkEncoding下) 2022-03-31 16:30:15 +08:00
刘祥超
7b0b9ac3b4 只有满足缓存条件的图片内容才会被转换 2022-03-31 16:22:23 +08:00
刘祥超
f7ed942779 修复临时关闭页面内容切换到HTML无法使用的问题 2022-03-31 15:17:30 +08:00
刘祥超
6e6be5d8d1 优化OpenFileCache fsnotify事件 2022-03-31 13:30:52 +08:00
刘祥超
221d00b450 修复OpenFileCache可能无法更新的Bug 2022-03-31 11:47:31 +08:00
刘祥超
9689baccf7 修复编译386时可能出现的错误 2022-03-30 22:53:06 +08:00
刘祥超
21de83d31e 反向代理错误增加URL显示 2022-03-30 17:31:47 +08:00
刘祥超
f925d86107 商业版增加UAM功能 2022-03-29 21:24:57 +08:00
刘祥超
db6e20ba37 支持路由定义请求脚本 2022-03-26 22:03:40 +08:00
刘祥超
33a8bae7c5 服务相关流量统计增加Header 2022-03-26 12:29:34 +08:00
刘祥超
b243d6c92e 攻击流量统计使用上行流量 2022-03-26 12:16:25 +08:00
刘祥超
050dd02fa5 优化编译脚本 2022-03-26 11:47:25 +08:00
刘祥超
e5b44ce178 优化代码 2022-03-25 14:11:34 +08:00
刘祥超
f6a983e683 HttpWriter暴露两个方法/默认Buffer为4K 2022-03-24 21:42:03 +08:00
刘祥超
99227bb4f2 IP找不到不再提示错误 2022-03-24 21:41:32 +08:00
刘祥超
d1ea67581e 版本号改为0.4.7 2022-03-23 14:44:43 +08:00
刘祥超
eb6cc9d781 Age改为在缓存中的已存活时间 2022-03-20 21:17:54 +08:00
刘祥超
3c75d860e4 优化代码 2022-03-20 21:15:25 +08:00
刘祥超
ad59c28200 优化代码 2022-03-20 20:58:34 +08:00
刘祥超
130519be71 增加测试用例 2022-03-20 18:33:51 +08:00
刘祥超
da0fdf6485 优化代码 2022-03-20 16:20:40 +08:00
刘祥超
37f3e6fa2d 取消查询缓存超时的设置 2022-03-20 16:12:49 +08:00
刘祥超
c2151e895d GRPC通讯支持gzip压缩 2022-03-20 11:28:34 +08:00
刘祥超
e15afc0e0c 修复一个测试用例 2022-03-20 10:53:35 +08:00
刘祥超
c86f2d5a44 优化代码 2022-03-20 10:48:11 +08:00
刘祥超
d45254b46d 更新相关库 2022-03-20 10:48:06 +08:00
刘祥超
83df9d0c29 内容压缩算法使用Pool管理 2022-03-20 00:05:47 +08:00
刘祥超
856f9d4a3b 调整Brotli压缩参数 2022-03-19 19:28:20 +08:00
刘祥超
14eda8c276 优化代码 2022-03-19 12:17:28 +08:00
刘祥超
0e732e4821 OCSP支持过期时间 2022-03-18 20:20:47 +08:00
刘祥超
b5b7ab99d3 动态更新OCSP,防止过期 2022-03-18 17:09:15 +08:00
刘祥超
134b56c87a 优化代码 2022-03-17 19:48:04 +08:00
刘祥超
a381850dbc 源站可以自定义回源主机名 2022-03-17 17:03:52 +08:00
刘祥超
14abce08c3 IPSet支持IPv6 2022-03-16 20:47:55 +08:00
刘祥超
469a3ea979 优化WAF性能 2022-03-16 17:06:26 +08:00
刘祥超
9b85487a70 提升缓存效率 2022-03-16 16:20:53 +08:00
刘祥超
06ef206c9f 节点可以单独设置缓存目录 2022-03-16 15:24:35 +08:00
刘祥超
1c2ca73208 缓存策略修改时尽可能不重新加载 2022-03-15 21:33:44 +08:00
刘祥超
b9abc55728 本地数据库升级之后增加日志 2022-03-15 18:58:56 +08:00
刘祥超
21e206061d 优化本地数据库性能 2022-03-15 18:32:39 +08:00
刘祥超
ddebc0e4a8 增加回源跟随功能 2022-03-14 15:07:18 +08:00
刘祥超
31b16ad67a 正在退出时不上报错误 2022-03-14 12:23:28 +08:00
刘祥超
d0b86af4ef 程序意外退出时关闭sqlite指针 2022-03-14 11:47:34 +08:00
刘祥超
ce87fa25e7 sqlite添加参数_sync=OFF 2022-03-14 11:08:02 +08:00
刘祥超
ecbb11eee2 增加对数据库操作的统计命令:edge-node dbstat/减少几个不必要的查询操作 2022-03-13 19:27:38 +08:00
刘祥超
9849f14044 自动为热点数据设置合适的过期时间 2022-03-12 20:50:05 +08:00
刘祥超
34fae1c2a3 内存缓存刷到磁盘后自动从内存中删除 2022-03-12 20:01:28 +08:00
刘祥超
8b22d00ce4 优化代码 2022-03-12 19:45:22 +08:00
刘祥超
1e64386744 更新相关库 2022-03-12 19:11:29 +08:00
刘祥超
8dfeda0d80 优化代码 2022-03-12 18:00:22 +08:00
刘祥超
0ece9e0897 优化代码 2022-03-11 17:21:23 +08:00
刘祥超
79ffda4706 使用异步IO写入缓存文件 2022-03-11 15:29:18 +08:00
刘祥超
0d2d7591e5 自动清理本地IP名单过期条目/修复白名单可能不起作用的Bug 2022-03-06 19:40:26 +08:00
刘祥超
577a5618a1 修复纯内存缓存可能启动多个flush内容的goroutine的Bug 2022-03-06 18:09:33 +08:00
刘祥超
49822ab7e9 分块传输内容可以写入到内存中/分块传输内容可以判断最大尺寸 2022-03-06 17:18:06 +08:00
刘祥超
421cb82505 增加Partial Content大尺寸测试用例 2022-03-06 14:40:31 +08:00
刘祥超
1c0192d773 优化Partial Content缓存 2022-03-05 19:31:50 +08:00
刘祥超
10443dfb15 优化Partial Content缓存 2022-03-05 16:47:17 +08:00
刘祥超
f34ad57e12 优化Partial Content缓存 2022-03-04 22:42:03 +08:00
刘祥超
21c80aea78 不支持对GET以外的方法返回的Partial内容的缓存 2022-03-04 17:09:12 +08:00
刘祥超
16c123c400 当缓存条件状态码为206时,自动支持区间缓存 2022-03-04 17:00:48 +08:00
刘祥超
18d301e17f 更新TeaGo 2022-03-04 15:11:18 +08:00
刘祥超
dc9a399ff8 升级相关依赖库 2022-03-04 12:30:06 +08:00
刘祥超
269e33b9a0 删除Partial缓存时,同时删除区间范围相关文件 2022-03-04 11:51:59 +08:00
刘祥超
581a3d49fc 实现基础的206 partial content缓存 2022-03-03 19:36:28 +08:00
刘祥超
8aeb5eb1b6 WAF有允许(allow)动作出现的时候不再继续往下执行 2022-02-26 17:13:27 +08:00
刘祥超
4d79ce7659 修改Partial Content的Bounary值长度(从60字节修改为16字节) 2022-02-25 19:09:09 +08:00
刘祥超
9fda8c425a 增加 edge-node reload 命令/优化命令行帮助 2022-02-25 11:23:32 +08:00
刘祥超
14259b2ef5 Update http_request_cache.go 2022-02-24 20:39:43 +08:00
刘祥超
211750eb8e 默认不同步写入压缩缓存/增加是否同步写入压缩缓存设置/去除默认content-type类型设置 2022-02-24 20:13:05 +08:00
刘祥超
1050231086 修改版本为0.4.5 2022-02-24 19:26:57 +08:00
刘祥超
034965a697 优化代码 2022-02-24 16:44:28 +08:00
刘祥超
239acf2d17 修复WAF提取正则表达式关键词时可能会使程序崩溃的Bug 2022-02-23 17:35:35 +08:00
刘祥超
6260fd1009 sqlite缓存表增加tag字段 2022-02-23 16:06:18 +08:00
刘祥超
f10ffbaebd 修改版本为0.4.4 2022-02-23 14:49:42 +08:00
刘祥超
83f581f7f0 清理缓存时也清理HEAD缓存 2022-02-22 21:52:28 +08:00
刘祥超
03ce082926 支持对GET/POST之外的请求方法独立缓存 2022-02-22 21:43:47 +08:00
刘祥超
ebafe458ff 支持缓存压缩后的内容 2022-02-22 19:29:27 +08:00
刘祥超
ff30b8d15c 修改版本为v0.4.3 2022-02-21 18:31:17 +08:00
刘祥超
51dd778ca7 更改为v0.4.2 2022-02-21 17:52:02 +08:00
刘祥超
b5f706686c 修复热点数据从文件系统转移到内存时可能不完整的Bug/实现部分Partial Content功能 2022-02-21 17:33:58 +08:00
刘祥超
b67c2ec39c 缓存关闭X-Cache显示时从Header中删除X-Cache 2022-02-21 16:55:25 +08:00
243 changed files with 11391 additions and 2596 deletions

29
LICENSE Normal file
View File

@@ -0,0 +1,29 @@
BSD 3-Clause License
Copyright (c) 2020, LiuXiangChao
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

1
README.md Normal file
View File

@@ -0,0 +1 @@
GoEdge边缘节点源码

View File

@@ -42,6 +42,11 @@ function build() {
mkdir $DIST/configs
mkdir $DIST/logs
mkdir $DIST/data
if [ "$TAG" = "plus" ]; then
mkdir $DIST/scripts
mkdir $DIST/scripts/js
fi
fi
cp $ROOT/configs/api.template.yaml $DIST/configs

View File

@@ -22,7 +22,9 @@ func main() {
app := apps.NewAppCmd().
Version(teaconst.Version).
Product(teaconst.ProductName).
Usage(teaconst.ProcessName + " [-v|start|stop|restart|status|quit|test|service|daemon|pprof]")
Usage(teaconst.ProcessName + " [-v|start|stop|restart|status|quit|test|reload|service|daemon|pprof|accesslog]").
Usage(teaconst.ProcessName + " [trackers|goman|conns|gc]").
Usage(teaconst.ProcessName + " [ip.drop|ip.reject|ip.remove] IP")
app.On("test", func() {
err := nodes.NewNode().Test()
@@ -30,6 +32,20 @@ func main() {
_, _ = os.Stderr.WriteString(err.Error())
}
})
app.On("reload", func() {
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{Code: "reload"})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
var params = maps.NewMap(reply.Params)
if params.Has("error") {
fmt.Println("[ERROR]" + params.GetString("error"))
} else {
fmt.Println("ok")
}
}
})
app.On("daemon", func() {
nodes.NewNode().Daemon()
})
@@ -65,6 +81,12 @@ func main() {
node := nodes.NewNode()
node.Start()
})
app.On("dbstat", func() {
teaconst.EnableDBStat = true
node := nodes.NewNode()
node.Start()
})
app.On("trackers", func() {
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{Code: "trackers"})
@@ -236,6 +258,53 @@ func main() {
}
}
})
app.On("accesslog", func() {
// local sock
var tmpDir = os.TempDir()
var sockFile = tmpDir + "/" + teaconst.AccessLogSockName
_, err := os.Stat(sockFile)
if err != nil {
if !os.IsNotExist(err) {
fmt.Println("[ERROR]" + err.Error())
return
}
}
var processSock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := processSock.Send(&gosock.Command{
Code: "accesslog",
})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
return
}
if reply.Code == "error" {
var errString = maps.NewMap(reply.Params).GetString("error")
if len(errString) > 0 {
fmt.Println("[ERROR]" + errString)
return
}
}
conn, err := net.Dial("unix", sockFile)
if err != nil {
fmt.Println("[ERROR]start reading access log failed: " + err.Error())
return
}
defer func() {
_ = conn.Close()
}()
var buf = make([]byte, 1024)
for {
n, err := conn.Read(buf)
if n > 0 {
fmt.Print(string(buf[:n]))
}
if err != nil {
break
}
}
})
app.Run(func() {
node := nodes.NewNode()
node.Start()

25
go.mod
View File

@@ -8,33 +8,32 @@ replace (
require (
github.com/TeaOSLab/EdgeCommon v0.0.0-00010101000000-000000000000
github.com/andybalholm/brotli v1.0.3
github.com/andybalholm/brotli v1.0.4
github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670
github.com/cespare/xxhash v1.1.0
github.com/chai2010/webp v1.1.0 // indirect
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f
github.com/fsnotify/fsnotify v1.5.1 // indirect
github.com/go-yaml/yaml v2.1.0+incompatible
github.com/fsnotify/fsnotify v1.5.1
github.com/go-redis/redis/v8 v8.11.5
github.com/golang/protobuf v1.5.2
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6
github.com/iwind/TeaGo v0.0.0-20220304043459-0dd944a5b475
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3
github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e // indirect
github.com/klauspost/compress v1.15.2 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mattn/go-sqlite3 v1.14.9
github.com/miekg/dns v1.1.43
github.com/mssola/user_agent v0.5.3
github.com/pires/go-proxyproto v0.6.1
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/tklauser/go-sysconf v0.3.6 // indirect
github.com/yusufpapurcu/wmi v1.2.2 // indirect
github.com/shirou/gopsutil/v3 v3.22.2
golang.org/x/image v0.0.0-20211028202545-6944b10bf410
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9
golang.org/x/sys v0.0.0-20220111092808-5a964db01320
golang.org/x/net v0.0.0-20220225172249-27dd8689420f
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad
golang.org/x/text v0.3.7
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa // indirect
google.golang.org/grpc v1.43.0
google.golang.org/grpc v1.45.0
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
)

198
go.sum
View File

@@ -2,12 +2,14 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMT
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/toml v0.4.1 h1:GaI7EiDXDRfa8VshkTj7Fym7ha+y8/XxIgD2okUIjLw=
github.com/BurntSushi/toml v0.4.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/DataDog/sketches-go v0.0.0-20190923095040-43f19ad77ff7/go.mod h1:Q5DbzQ+3AkgGwymQO7aZFNP7ns2lZKGtvRBzRXfdi60=
github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/andybalholm/brotli v1.0.3 h1:fpcw+r1N1h0Poc1F/pHbW40cUm/lMEQslZtCkBQ0UnM=
github.com/andybalholm/brotli v1.0.3/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY=
github.com/andybalholm/brotli v1.0.4/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/benbjohnson/clock v1.0.3/go.mod h1:bGMdMPoPVvcYyt1gHDf4J2KE153Yf9BuiUKYMaxlTDM=
github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670 h1:FQPKKjDhzG0T4ew6dm6MGrXb4PRAi8ZmTuYuxcF62BM=
@@ -15,15 +17,21 @@ github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670/go.mod h1:iRWAF
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chai2010/webp v1.1.0 h1:4Ei0/BRroMF9FaXDG2e4OxwFcuW2vcXd+A6tyqTJUQQ=
github.com/chai2010/webp v1.1.0/go.mod h1:LP12PG5IFmLGHUU26tBiCBKnghxx3toZFwDjOYvd3Ow=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/cilium/ebpf v0.5.0/go.mod h1:4tRaxcgiL706VnOzHOdBlY8IEAIdxINsQBcU4xJJXRs=
github.com/cilium/ebpf v0.7.0 h1:1k/q3ATgxSXRdrmPfH8d7YK0GfqVsEKZAX9dQZvs56k=
github.com/cilium/ebpf v0.7.0/go.mod h1:/oI2+1shJiTGAMgl6/RgJr36Eo1jzrRcAWbcXO2usCA=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
@@ -34,13 +42,16 @@ github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f h1:q/DpyjJjZs94bziQ7YkBmIlpqbVP7yw179rnzoNVX1M=
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f/go.mod h1:QGrK8vMWWHQYQ3QU9bw9Y9OPNfxccGzfb41qjvVeXtY=
github.com/dgryski/go-rendezvous v0.0.0-20200624174652-8d2f3be8b2d9/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/frankban/quicktest v1.11.3 h1:8sXhOn0uLys67V8EsXLc6eszDs8VXWxL3iRvebPhedY=
github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsnotify/fsnotify v1.5.1 h1:mZcQUHVQUQWoPXXtuf9yuEXKudkV2sx1E06UadKWpgI=
@@ -50,10 +61,11 @@ github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-redis/redis/v8 v8.0.0-beta.7/go.mod h1:FGJAWDWFht1sQ4qxyJHZZbVyvnVcKQN0E3u5/5lRz+g=
github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-yaml/yaml v2.1.0+incompatible h1:RYi2hDdss1u4YE7GwixGzWwVo47T8UQwnTLB6vQiq+o=
github.com/go-yaml/yaml v2.1.0+incompatible/go.mod h1:w2MrLa16VYP0jy6N7M5kHaCkaLENm+P+Tv+MfurjSw0=
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -75,24 +87,47 @@ github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMyw
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6 h1:btadZscaRmsi/+fOhkyUguRpSnrf6dykNEWxDeUCj9I=
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6/go.mod h1:0F8on3JWMkm+xahTHItkiu/E1SPqMd0TOxNweQv8ptE=
github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060/go.mod h1:KU4mS7QNiZ7QWEuDBk1zw0/Q2LrAPZv3tycEFBsuUwc=
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24 h1:1cGulkD2SNJJRok5OKwyhP/Ddm+PgSWKOupn0cR36/A=
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24/go.mod h1:KU4mS7QNiZ7QWEuDBk1zw0/Q2LrAPZv3tycEFBsuUwc=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/iwind/TeaGo v0.0.0-20220304043459-0dd944a5b475 h1:EseyfFaQOjWanGiby9KMw7PjDBMg/95tLDgIw/ns0Cw=
github.com/iwind/TeaGo v0.0.0-20220304043459-0dd944a5b475/go.mod h1:HRHK0zoC/og3c9/hKosD9yYVMTnnzm3PgXUdhRYHaLc=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11 h1:DaQjoWZhLNxjhIXedVg4/vFEtHkZhK4IjIwsWdyzBLg=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11/go.mod h1:JtbX20untAjUVjZs1ZBtq80f5rJWvwtQNRL6EnuYRnY=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3 h1:aBSonas7vFcgTj9u96/bWGILGv1ZbUSTLiOzcI1ZT6c=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3/go.mod h1:H5Q7SXwbx3a97ecJkaS2sD77gspzE7HFUafBO0peEyA=
github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4 h1:VWGsCqTzObdlbf7UUE3oceIpcEKi4C/YBUszQXk118A=
github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4/go.mod h1:H5Q7SXwbx3a97ecJkaS2sD77gspzE7HFUafBO0peEyA=
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8 h1:AojsHz9Es9B3He2MQQxeRq3TyD//o9huxUo7r1wh44g=
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8/go.mod h1:QJBY2txYhLMzwLO29iB5ujDJ3s3V7DsZ582nw4Ss+tM=
github.com/josharian/native v0.0.0-20200817173448-b6b71def0850 h1:uhL5Gw7BINiiPAo24A2sxkcDI0Jt/sqp1v5xQCniEFA=
github.com/josharian/native v0.0.0-20200817173448-b6b71def0850/go.mod h1:7X/raswPFr05uY3HiLlYeyQntB6OO7E/d2Cu7qoaN2w=
github.com/jsimonetti/rtnetlink v0.0.0-20190606172950-9527aa82566a/go.mod h1:Oz+70psSo5OFh8DBl0Zv2ACw7Esh6pPUphlvZG9x7uw=
github.com/jsimonetti/rtnetlink v0.0.0-20200117123717-f846d4f6c1f4/go.mod h1:WGuG/smIU4J/54PblvSbh+xvCZmpJnFgr3ds6Z55XMQ=
github.com/jsimonetti/rtnetlink v0.0.0-20201009170750-9c6f07d100c1/go.mod h1:hqoO/u39cqLeBLebZ8fWdE96O7FxrAsRYhnVOdgHxok=
github.com/jsimonetti/rtnetlink v0.0.0-20201216134343-bde56ed16391/go.mod h1:cR77jAZG3Y3bsb8hF6fHJbFoyFukLFOkQ98S0pQz3xw=
github.com/jsimonetti/rtnetlink v0.0.0-20201220180245-69540ac93943/go.mod h1:z4c53zj6Eex712ROyh8WI0ihysb5j2ROyV42iNogmAs=
github.com/jsimonetti/rtnetlink v0.0.0-20210122163228-8d122574c736/go.mod h1:ZXpIyOK59ZnN7J0BV99cZUPmsqDRZ3eq5X+st7u/oSA=
github.com/jsimonetti/rtnetlink v0.0.0-20210212075122-66c871082f2b/go.mod h1:8w9Rh8m+aHZIG69YPGGem1i5VzoyRC8nw2kA8B+ik5U=
github.com/jsimonetti/rtnetlink v0.0.0-20210525051524-4cc836578190/go.mod h1:NmKSdU4VGSiv1bMsdqNALI4RSvvjtz65tTMCnD05qLo=
github.com/jsimonetti/rtnetlink v0.0.0-20211022192332-93da33804786 h1:N527AHMa793TP5z5GNAn/VLPzlc0ewzWdeP/25gDfgQ=
github.com/jsimonetti/rtnetlink v0.0.0-20211022192332-93da33804786/go.mod h1:v4hqbTdfQngbVSZJVWUhGE/lbTFf9jb+ygmNUDQMuOs=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e h1:LvL4XsI70QxOGHed6yhQtAU34Kx3Qq2wwBzGFKY8zKk=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e/go.mod h1:kLgvv7o6UM+0QSf0QjAse3wReFDsb9qbZJdfexWlrQw=
github.com/klauspost/compress v1.15.2 h1:3WH+AG7s2+T8o3nrM/8u2rdqUEcQhmga7smjrT41nAw=
github.com/klauspost/compress v1.15.2/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@@ -100,8 +135,31 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/mattn/go-sqlite3 v1.14.9 h1:10HX2Td0ocZpYEjhilsuo6WWtUqttj2Kb0KtD86/KYA=
github.com/mattn/go-sqlite3 v1.14.9/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mdlayher/ethtool v0.0.0-20210210192532-2b88debcdd43/go.mod h1:+t7E0lkKfbBsebllff1xdTmyJt8lH37niI6kwFk9OTo=
github.com/mdlayher/ethtool v0.0.0-20211028163843-288d040e9d60 h1:tHdB+hQRHU10CfcK0furo6rSNgZ38JT8uPh70c/pFD8=
github.com/mdlayher/ethtool v0.0.0-20211028163843-288d040e9d60/go.mod h1:aYbhishWc4Ai3I2U4Gaa2n3kHWSwzme6EsG/46HRQbE=
github.com/mdlayher/genetlink v1.0.0 h1:OoHN1OdyEIkScEmRgxLEe2M9U8ClMytqA5niynLtfj0=
github.com/mdlayher/genetlink v1.0.0/go.mod h1:0rJ0h4itni50A86M2kHcgS85ttZazNt7a8H2a2cw0Gc=
github.com/mdlayher/netlink v0.0.0-20190409211403-11939a169225/go.mod h1:eQB3mZE4aiYnlUsyGGCOpPETfdQq4Jhsgf1fk3cwQaA=
github.com/mdlayher/netlink v1.0.0/go.mod h1:KxeJAFOFLG6AjpyDkQ/iIhxygIUKD+vcwqcnu43w/+M=
github.com/mdlayher/netlink v1.1.0/go.mod h1:H4WCitaheIsdF9yOYu8CFmCgQthAPIWZmcKp9uZHgmY=
github.com/mdlayher/netlink v1.1.1/go.mod h1:WTYpFb/WTvlRJAyKhZL5/uy69TDDpHHu2VZmb2XgV7o=
github.com/mdlayher/netlink v1.2.0/go.mod h1:kwVW1io0AZy9A1E2YYgaD4Cj+C+GPkU6klXCMzIJ9p8=
github.com/mdlayher/netlink v1.2.1/go.mod h1:bacnNlfhqHqqLo4WsYeXSqfyXkInQ9JneWI68v1KwSU=
github.com/mdlayher/netlink v1.2.2-0.20210123213345-5cc92139ae3e/go.mod h1:bacnNlfhqHqqLo4WsYeXSqfyXkInQ9JneWI68v1KwSU=
github.com/mdlayher/netlink v1.3.0/go.mod h1:xK/BssKuwcRXHrtN04UBkwQ6dY9VviGGuriDdoPSWys=
github.com/mdlayher/netlink v1.4.0/go.mod h1:dRJi5IABcZpBD2A3D0Mv/AiX8I9uDEu5oGkAVrekmf8=
github.com/mdlayher/netlink v1.4.1/go.mod h1:e4/KuJ+s8UhfUpO9z00/fDZZmhSrs+oxyqAS9cNgn6Q=
github.com/mdlayher/netlink v1.4.2 h1:3sbnJWe/LETovA7yRZIX3f9McVOWV3OySH6iIBxiFfI=
github.com/mdlayher/netlink v1.4.2/go.mod h1:13VaingaArGUTUxFLf/iEovKxXji32JAtF858jZYEug=
github.com/mdlayher/socket v0.0.0-20210307095302-262dc9984e00/go.mod h1:GAFlyu4/XV68LkQKYzKhIo/WW7j3Zi0YRAz/BOoanUc=
github.com/mdlayher/socket v0.0.0-20211007213009-516dcbdf0267/go.mod h1:nFZ1EtZYK8Gi/k6QNu7z7CgO20i/4ExeQswwWuPmG/g=
github.com/mdlayher/socket v0.0.0-20211102153432-57e3fa563ecb h1:2dC7L10LmTqlyMVzFJ00qM25lqESg9Z4u3GuEXN5iHY=
github.com/mdlayher/socket v0.0.0-20211102153432-57e3fa563ecb/go.mod h1:nFZ1EtZYK8Gi/k6QNu7z7CgO20i/4ExeQswwWuPmG/g=
github.com/miekg/dns v1.1.43 h1:JKfpVSCB84vrAmHzyrsxB5NAr5kLoMXZArPSw7Qlgyg=
github.com/miekg/dns v1.1.43/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -110,22 +168,34 @@ github.com/mssola/user_agent v0.5.3 h1:lBRPML9mdFuIZgI2cmlQ+atbpJdLdeVl2IDodjBR5
github.com/mssola/user_agent v0.5.3/go.mod h1:TTPno8LPY3wAIEKRpAtkdMT0f8SE24pLRGPahjCH4uw=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
github.com/onsi/ginkgo/v2 v2.0.0 h1:CcuG/HvWNkkaqCUpJifQY8z7qEMBJya6aLPx6ftGyjQ=
github.com/onsi/ginkgo/v2 v2.0.0/go.mod h1:vw5CSIxN1JObi/U8gcbwft7ZxR2dgaR70JSE3/PpL4c=
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/onsi/gomega v1.17.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY=
github.com/onsi/gomega v1.18.1 h1:M1GfJqGRrBrrGGsbxzV5dqM2U2ApXefZCQpkukxYRLE=
github.com/onsi/gomega v1.18.1/go.mod h1:0q+aL8jAiMXy9hbwj2mr5GziHiwhAIQpFmmtT5hitRs=
github.com/opentracing/opentracing-go v1.1.1-0.20190913142402-a7454ce5950e/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/pires/go-proxyproto v0.6.1 h1:EBupykFmo22SDjv4fQVQd2J9NOoLPmyZA/15ldOGkPw=
github.com/pires/go-proxyproto v0.6.1/go.mod h1:Odh9VFOZJCf9G8cLW5o435Xf1J95Jw9Gw5rnCjcwzAY=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF4JjgDlrVEn3C11VoGHZN7m8qihwgMEtzYw=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/shirou/gopsutil v3.21.11+incompatible h1:+1+c1VGhc88SSonWP6foOcLhvnKlUeu/erjjvaPEYiI=
github.com/shirou/gopsutil v3.21.11+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shirou/gopsutil/v3 v3.22.2 h1:wCrArWFkHYIdDxx/FSfF5RB4dpJYW6t7rcp3+zL8uks=
github.com/shirou/gopsutil/v3 v3.22.2/go.mod h1:WapW1AOOPlHyXr+yOyw3uYx36enocrtSoSBy0L5vUHY=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72 h1:qLC7fQah7D6K1B0ujays3HV9gkFtllcxhzImRR7ArPQ=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -134,10 +204,15 @@ github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/tklauser/go-sysconf v0.3.6 h1:oc1sJWvKkmvIxhDHeKWvZS4f6AW+YcoguSfRF2/Hmo4=
github.com/tklauser/go-sysconf v0.3.6/go.mod h1:MkWzOF4RMCshBAMXuhXJs64Rte09mITnppBXY/rYEFI=
github.com/tklauser/numcpus v0.2.2 h1:oyhllyrScuYI6g+h/zUvNXNp1wy7x8qQy3t/piefldA=
github.com/tklauser/numcpus v0.2.2/go.mod h1:x3qojaO3uyYt0i56EW/VUYs7uBvdl2fkfZFu0T9wgjM=
github.com/tklauser/go-sysconf v0.3.9 h1:JeUVdAOWhhxVcU6Eqr/ATFHgXk/mmiItdKeJPev3vTo=
github.com/tklauser/go-sysconf v0.3.9/go.mod h1:11DU/5sG7UexIrp/O6g35hrWzu0JxlwQ3LSFUzyeuhs=
github.com/tklauser/numcpus v0.3.0 h1:ILuRUQBtssgnxw0XXIjKUC56fgnOrFoQQ/4+DeU2biQ=
github.com/tklauser/numcpus v0.3.0/go.mod h1:yFGUr7TUHQRAhyqBcEg0Ge34zDBAsIvJJcyE6boqnA8=
github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc h1:R83G5ikgLMxrBvLh22JhdfI8K6YXEPHx5P03Uu3DRs4=
github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg=
github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.opentelemetry.io/otel v0.7.0/go.mod h1:aZMyHG5TqDOXEgH2tyLiXSUKly1jT3yqE9PmrzIeCdo=
@@ -158,6 +233,10 @@ golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHl
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38=
golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -166,46 +245,93 @@ golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191007182048-72f939374954/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201010224723-4f7140c49acb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201216054612-986b41b23924/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9 h1:kmreh1vGI63l2FxOAYS3Yv6ATsi7lSTuwNSVbGfJV9I=
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210928044308-7d9f5e0b762b/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211020060615-d418f374d309/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211201190559-0a0e4e1bb54c/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211209124913-491a49abca63/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f h1:oA4XRj0qtSt8Yo1Zms0CUlsT3KG69V2UGQWPBxujDmc=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190411185658-b44545bcd369/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201009025420-dfb3f7c4e634/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201118182958-a01c418693c7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201218084310-7d0127a74742/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210110051926-789bb1bd4061/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210123111255-9b0068b26619/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210216163648-f7da38b97c65/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210316164454-77fc1eacc6aa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210305230114-8fe3ee5dd75b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210525143221-35b2ab0089ea/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211214234402-4825e8c3871d h1:1oIt9o40TWWI9FUaveVpUvBe13FNqBNVXy3ue2fcfkw=
golang.org/x/sys v0.0.0-20211214234402-4825e8c3871d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220111092808-5a964db01320 h1:0jf+tOCoZ3LyutmCOWpVni1chK4VfFLhRsDK7MhqGRY=
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210816074244-15123e1e1f71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211205182925-97ca703d548d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220111092808-5a964db01320/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220319134239-a9b59b0215f8 h1:OH54vjqzRWmbJ62fjuhxy7AxFFgoHN0/DPc/UrL8cAs=
golang.org/x/sys v0.0.0-20220319134239-a9b59b0215f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad h1:ntjMns5wyP/fN65tdBD4g8J5w8n015+iIIs9rtjXkY0=
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
@@ -218,7 +344,14 @@ golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGm
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo=
golang.org/x/tools v0.1.8 h1:P1HhGGuLW4aAclzjtmJdf0mJOjVUZUzOTqkAkWL+l6w=
golang.org/x/tools v0.1.8/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
@@ -230,19 +363,17 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98
google.golang.org/genproto v0.0.0-20191009194640-548a555dbc03/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa h1:I0YcKz0I7OAhddo7ya8kMnvprhcWM045PmkBdMO9zN0=
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
google.golang.org/genproto v0.0.0-20220317150908-0efb43f6373e h1:fNKDNuUyC4WH+inqDMpfXDdfvwfYILbsX+oskGZ8hxg=
google.golang.org/genproto v0.0.0-20220317150908-0efb43f6373e/go.mod h1:hAL49I2IFola2sVEjAn7MEwsja0xp51I0tlGAf9hz4E=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.32.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
google.golang.org/grpc v1.43.0 h1:Eeu7bZtDZ2DpRCsLhUlcrLnvYaMK1Gz86a+hMVvELmM=
google.golang.org/grpc v1.43.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU=
google.golang.org/grpc v1.45.0 h1:NEpgUqV3Z+ZjkqMsxMg11IaDrXY4RY6CQukSGK0uI1M=
google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -262,6 +393,7 @@ gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -272,7 +404,11 @@ gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.2.1/go.mod h1:lPVVZ2BS5TfnjLyizF7o7hv7j9/L+8cZY2hLyjP9cGY=
honnef.co/go/tools v0.2.2 h1:MNh1AVMyVX23VUHE2O27jm6lNj3vjO5DexS4A1xvnzk=
honnef.co/go/tools v0.2.2/go.mod h1:lPVVZ2BS5TfnjLyizF7o7hv7j9/L+8cZY2hLyjP9cGY=
rogchap.com/v8go v0.7.0/go.mod h1:MxgP3pL2MW4dpme/72QRs8sgNMmM0pRc8DPhcuLWPAs=

View File

@@ -20,7 +20,7 @@ import (
type AppCmd struct {
product string
version string
usage string
usages []string
options []*CommandHelpOption
appendStrings []string
@@ -54,7 +54,7 @@ func (this *AppCmd) Version(version string) *AppCmd {
// Usage 使用方法
func (this *AppCmd) Usage(usage string) *AppCmd {
this.usage = usage
this.usages = append(this.usages, usage)
return this
}
@@ -77,8 +77,10 @@ func (this *AppCmd) Append(appendString string) *AppCmd {
func (this *AppCmd) Print() {
fmt.Println(this.product + " v" + this.version)
usage := this.usage
fmt.Println("Usage:", "\n "+usage)
fmt.Println("Usage:")
for _, usage := range this.usages {
fmt.Println(" " + usage)
}
if len(this.options) > 0 {
fmt.Println("")
@@ -211,7 +213,19 @@ func (this *AppCmd) runStop() {
return
}
_, _ = this.sock.Send(&gosock.Command{Code: "stop"})
// 从systemd中停止
if runtime.GOOS == "linux" {
systemctl, _ := exec.LookPath("systemctl")
if len(systemctl) > 0 {
go func() {
// 有可能会长时间执行,这里不阻塞进程
_ = exec.Command(systemctl, "stop", teaconst.SystemdServiceName).Run()
}()
}
}
// 如果仍在运行,则发送停止指令
_, _ = this.sock.SendTimeout(&gosock.Command{Code: "stop"}, 1*time.Second)
fmt.Println(this.product+" stopped ok, pid:", types.String(pid))
}

View File

@@ -1,10 +1,12 @@
package apps
import (
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/sizes"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/files"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/utils/time"
timeutil "github.com/iwind/TeaGo/utils/time"
"log"
"os"
"runtime"
@@ -13,27 +15,55 @@ import (
)
type LogWriter struct {
fileAppender *files.Appender
fp *os.File
c chan string
}
func (this *LogWriter) Init() {
// 创建目录
dir := files.NewFile(Tea.LogDir())
var dir = files.NewFile(Tea.LogDir())
if !dir.Exists() {
err := dir.Mkdir()
if err != nil {
log.Println("[error]" + err.Error())
log.Println("[LOG]create log dir failed: " + err.Error())
}
}
logFile := files.NewFile(Tea.LogFile("run.log"))
// 打开要写入的日志文件
appender, err := logFile.Appender()
var logPath = Tea.LogFile("run.log")
fp, err := os.OpenFile(logPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
logs.Error(err)
log.Println("[LOG]open log file failed: " + err.Error())
} else {
this.fileAppender = appender
this.fp = fp
}
this.c = make(chan string, 1024)
// 异步写入文件
var maxFileSize = 2 * sizes.G // 文件最大尺寸,超出此尺寸则清空
if fp != nil {
goman.New(func() {
var totalSize int64 = 0
stat, err := fp.Stat()
if err == nil {
totalSize = stat.Size()
}
for message := range this.c {
totalSize += int64(len(message))
_, err := fp.WriteString(timeutil.Format("Y/m/d H:i:s ") + message + "\n")
if err != nil {
log.Println("[LOG]write log failed: " + err.Error())
} else {
// 如果太大则Truncate
if totalSize > maxFileSize {
_ = fp.Truncate(0)
totalSize = 0
}
}
}
})
}
}
@@ -48,7 +78,7 @@ func (this *LogWriter) Write(message string) {
var ok bool
_, file, line, ok = runtime.Caller(callDepth)
if ok {
file = this.packagePath(file)
file = utils.RemoveWorkspace(this.packagePath(file))
}
}
@@ -59,18 +89,15 @@ func (this *LogWriter) Write(message string) {
}
}
if this.fileAppender != nil {
_, err := this.fileAppender.AppendString(timeutil.Format("Y/m/d H:i:s ") + message + "\n")
if err != nil {
log.Println("[error]" + err.Error())
}
}
this.c <- message
}
func (this *LogWriter) Close() {
if this.fileAppender != nil {
_ = this.fileAppender.Close()
if this.fp != nil {
_ = this.fp.Close()
}
close(this.c)
}
func (this *LogWriter) packagePath(path string) string {

10
internal/caches/consts.go Normal file
View File

@@ -0,0 +1,10 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
const (
SuffixWebP = "@GOEDGE_WEBP" // WebP后缀
SuffixCompression = "@GOEDGE_" // 压缩后缀 SuffixCompression + Encoding
SuffixMethod = "@GOEDGE_" // 请求方法后缀 SuffixMethod + RequestMethod
SuffixPartial = "@GOEDGE_partial" // 分区缓存后缀
)

View File

@@ -6,9 +6,13 @@ import "errors"
// 常用的几个错误
var (
ErrNotFound = errors.New("cache not found")
ErrFileIsWriting = errors.New("the file is writing")
ErrInvalidRange = errors.New("invalid range")
ErrNotFound = errors.New("cache not found")
ErrFileIsWriting = errors.New("the file is writing")
ErrInvalidRange = errors.New("invalid range")
ErrEntityTooLarge = errors.New("entity too large")
ErrWritingUnavailable = errors.New("writing unavailable")
ErrWritingQueueFull = errors.New("writing queue full")
ErrTooManyOpenFiles = errors.New("too many open files")
)
// CapacityError 容量错误
@@ -30,7 +34,11 @@ func CanIgnoreErr(err error) bool {
if err == nil {
return true
}
if err == ErrFileIsWriting {
if err == ErrFileIsWriting ||
err == ErrEntityTooLarge ||
err == ErrWritingUnavailable ||
err == ErrWritingQueueFull ||
err == ErrTooManyOpenFiles {
return true
}
_, ok := err.(*CapacityError)

View File

@@ -3,8 +3,6 @@
package caches
type HotItem struct {
Key string
ExpiresAt int64
Hits uint32
Status int
Key string
Hits uint32
}

View File

@@ -4,52 +4,32 @@ package caches
import (
"database/sql"
"errors"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/ttlcache"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/lists"
"github.com/TeaOSLab/EdgeNode/internal/utils/fnv"
"github.com/iwind/TeaGo/types"
timeutil "github.com/iwind/TeaGo/utils/time"
_ "github.com/mattn/go-sqlite3"
"os"
"sync/atomic"
"time"
)
const CountFileDB = 20
// FileList 文件缓存列表管理
type FileList struct {
dir string
db *sql.DB
total int64
dir string
dbList [CountFileDB]*FileListDB
total int64
onAdd func(item *Item)
onRemove func(item *Item)
// cacheItems
existsByHashStmt *sql.Stmt // 根据hash检查是否存在
insertStmt *sql.Stmt // 写入数据
selectByHashStmt *sql.Stmt // 使用hash查询数据
deleteByHashStmt *sql.Stmt // 根据hash删除数据
statStmt *sql.Stmt // 统计
purgeStmt *sql.Stmt // 清理
deleteAllStmt *sql.Stmt // 删除所有数据
// hits
insertHitStmt *sql.Stmt // 写入数据
increaseHitStmt *sql.Stmt // 增加点击量
deleteHitByHashStmt *sql.Stmt // 根据hash删除数据
lfuHitsStmt *sql.Stmt // 读取老的数据
oldTables []string
itemsTableName string
hitsTableName string
isClosed bool
isReady bool
memoryCache *ttlcache.Cache
// 老数据库地址
oldDir string
}
func NewFileList(dir string) ListInterface {
@@ -59,6 +39,10 @@ func NewFileList(dir string) ListInterface {
}
}
func (this *FileList) SetOldDir(oldDir string) {
this.oldDir = oldDir
}
func (this *FileList) Init() error {
// 检查目录是否存在
_, err := os.Stat(this.dir)
@@ -70,114 +54,38 @@ func (this *FileList) Init() error {
remotelogs.Println("CACHE", "create cache dir '"+this.dir+"'")
}
this.itemsTableName = "cacheItems_v3"
this.hitsTableName = "hits"
var dir = this.dir
if dir == "/" {
// 防止sqlite提示authority错误
dir = ""
}
var dbPath = dir + "/index.db"
remotelogs.Println("CACHE", "loading database '"+dbPath+"'")
db, err := sql.Open("sqlite3", "file:"+dbPath+"?cache=shared&mode=rwc&_journal_mode=WAL")
if err != nil {
return errors.New("open database failed: " + err.Error())
}
db.SetMaxOpenConns(1)
remotelogs.Println("CACHE", "loading database from '"+dir+"' ...")
for i := 0; i < CountFileDB; i++ {
var db = NewFileListDB()
err = db.Open(dir + "/db-" + types.String(i) + ".db")
if err != nil {
return err
}
this.db = db
err = db.Init()
if err != nil {
return err
}
// TODO 耗时过长,暂时不整理数据库
/**_, err = db.Exec("VACUUM")
if err != nil {
return err
}**/
// 创建
err = this.initTables(db, 1)
if err != nil {
return errors.New("init tables failed: " + err.Error())
}
// 清除旧表
// 这个一定要在initTables()之后,因为老的数据需要转移
this.oldTables = []string{
"cacheItems",
"cacheItems_v2",
}
err = this.removeOldTables()
if err != nil {
remotelogs.Warn("CACHE", "clean old tables failed: "+err.Error())
this.dbList[i] = db
}
// 读取总数量
row := this.db.QueryRow(`SELECT COUNT(*) FROM "` + this.itemsTableName + `"`)
if row.Err() != nil {
return row.Err()
}
var total int64
err = row.Scan(&total)
if err != nil {
return err
}
this.total = total
// 常用语句
this.existsByHashStmt, err = this.db.Prepare(`SELECT "expiredAt" FROM "` + this.itemsTableName + `" WHERE "hash"=? AND expiredAt>? LIMIT 1`)
if err != nil {
return err
this.total = 0
for _, db := range this.dbList {
this.total += db.total
}
this.insertStmt, err = this.db.Prepare(`INSERT INTO "` + this.itemsTableName + `" ("hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "staleAt", "host", "serverId", "createdAt") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
return err
}
this.selectByHashStmt, err = this.db.Prepare(`SELECT "key", "headerSize", "bodySize", "metaSize", "expiredAt" FROM "` + this.itemsTableName + `" WHERE "hash"=? LIMIT 1`)
if err != nil {
return err
}
this.deleteByHashStmt, err = this.db.Prepare(`DELETE FROM "` + this.itemsTableName + `" WHERE "hash"=?`)
if err != nil {
return err
}
this.statStmt, err = this.db.Prepare(`SELECT COUNT(*), IFNULL(SUM(headerSize+bodySize+metaSize), 0), IFNULL(SUM(headerSize+bodySize), 0) FROM "` + this.itemsTableName + `"`)
if err != nil {
return err
}
this.purgeStmt, err = this.db.Prepare(`SELECT "hash" FROM "` + this.itemsTableName + `" WHERE staleAt<=? LIMIT ?`)
if err != nil {
return err
}
this.deleteAllStmt, err = this.db.Prepare(`DELETE FROM "` + this.itemsTableName + `"`)
if err != nil {
return err
}
this.insertHitStmt, err = this.db.Prepare(`INSERT INTO "` + this.hitsTableName + `" ("hash", "week2Hits", "week") VALUES (?, 1, ?)`)
this.increaseHitStmt, err = this.db.Prepare(`INSERT INTO "` + this.hitsTableName + `" ("hash", "week2Hits", "week") VALUES (?, 1, ?) ON CONFLICT("hash") DO UPDATE SET "week1Hits"=IIF("week"=?, "week1Hits", "week2Hits"), "week2Hits"=IIF("week"=?, "week2Hits"+1, 1), "week"=?`)
if err != nil {
return err
}
this.deleteHitByHashStmt, err = this.db.Prepare(`DELETE FROM "` + this.hitsTableName + `" WHERE "hash"=?`)
if err != nil {
return err
}
this.lfuHitsStmt, err = this.db.Prepare(`SELECT "hash" FROM "` + this.hitsTableName + `" ORDER BY "week" ASC, "week1Hits"+"week2Hits" ASC LIMIT ?`)
if err != nil {
return err
}
this.isReady = true
// 升级老版本数据库
goman.New(func() {
this.upgradeOldDB()
})
return nil
}
@@ -188,26 +96,22 @@ func (this *FileList) Reset() error {
}
func (this *FileList) Add(hash string, item *Item) error {
if !this.isReady {
var db = this.getDB(hash)
if !db.IsReady() {
return nil
}
if item.StaleAt == 0 {
item.StaleAt = item.ExpiredAt
}
_, err := this.insertStmt.Exec(hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.StaleAt, item.Host, item.ServerId, utils.UnixTime())
err := db.Add(hash, item)
if err != nil {
return err
}
_, err = this.insertHitStmt.Exec(hash, timeutil.Format("YW"))
if err != nil {
return err
}
atomic.AddInt64(&this.total, 1)
// 这里不增加点击量,以减少对数据库的操作次数
this.memoryCache.Write(hash, 1, item.ExpiredAt)
atomic.AddInt64(&this.total, 1)
if this.onAdd != nil {
this.onAdd(item)
@@ -216,40 +120,36 @@ func (this *FileList) Add(hash string, item *Item) error {
}
func (this *FileList) Exist(hash string) (bool, error) {
if !this.isReady {
var db = this.getDB(hash)
if !db.IsReady() {
return false, nil
}
item := this.memoryCache.Read(hash)
var item = this.memoryCache.Read(hash)
if item != nil {
return true, nil
}
rows, err := this.existsByHashStmt.Query(hash, time.Now().Unix())
var row = db.existsByHashStmt.QueryRow(hash, time.Now().Unix())
if row.Err() != nil {
return false, nil
}
var expiredAt int64
err := row.Scan(&expiredAt)
if err != nil {
if err == sql.ErrNoRows {
err = nil
}
return false, err
}
defer func() {
_ = rows.Close()
}()
if rows.Next() {
var expiredAt int64
err = rows.Scan(&expiredAt)
if err != nil {
return false, nil
}
this.memoryCache.Write(hash, 1, expiredAt)
return true, nil
}
return false, nil
this.memoryCache.Write(hash, 1, expiredAt)
return true, nil
}
// CleanPrefix 清理某个前缀的缓存数据
func (this *FileList) CleanPrefix(prefix string) error {
if !this.isReady {
return nil
}
if len(prefix) == 0 {
return nil
}
@@ -258,106 +158,48 @@ func (this *FileList) CleanPrefix(prefix string) error {
this.memoryCache.Clean()
}()
var count = int64(10000)
var staleLife = 600 // TODO 需要可以设置
var unixTime = utils.UnixTime() // 只删除当前的,不删除新的
for {
result, err := this.db.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0,staleAt=? WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)=1 LIMIT `+types.String(count)+`)`, unixTime+int64(staleLife), unixTime, prefix)
for _, db := range this.dbList {
err := db.CleanPrefix(prefix)
if err != nil {
return err
}
affectedRows, err := result.RowsAffected()
if err != nil {
return err
}
if affectedRows < count {
return nil
}
}
return nil
}
func (this *FileList) Remove(hash string) error {
if !this.isReady {
return nil
}
// 从缓存中删除
this.memoryCache.Delete(hash)
row := this.selectByHashStmt.QueryRow(hash)
if row.Err() != nil {
return row.Err()
}
var item = &Item{Type: ItemTypeFile}
err := row.Scan(&item.Key, &item.HeaderSize, &item.BodySize, &item.MetaSize, &item.ExpiredAt)
if err != nil {
if err == sql.ErrNoRows {
return nil
}
return err
}
_, err = this.deleteByHashStmt.Exec(hash)
if err != nil {
return err
}
_, err = this.deleteHitByHashStmt.Exec(hash)
if err != nil {
return err
}
atomic.AddInt64(&this.total, -1)
if this.onRemove != nil {
this.onRemove(item)
}
return nil
_, err := this.remove(hash)
return err
}
// Purge 清理过期的缓存
// count 每次遍历的最大数量,控制此数字可以保证每次清理的时候不用花太多时间
// callback 每次发现过期key的调用
func (this *FileList) Purge(count int, callback func(hash string) error) (int, error) {
if !this.isReady {
return 0, nil
}
count /= CountFileDB
if count <= 0 {
count = 1000
count = 100
}
rows, err := this.purgeStmt.Query(time.Now().Unix(), count)
if err != nil {
return 0, err
}
hashStrings := []string{}
var countFound = 0
for rows.Next() {
var hash string
err = rows.Scan(&hash)
for _, db := range this.dbList {
hashStrings, err := db.ListExpiredItems(count)
if err != nil {
_ = rows.Close()
return 0, err
return 0, nil
}
hashStrings = append(hashStrings, hash)
countFound++
}
_ = rows.Close() // 不能使用defer防止读写冲突
countFound += len(hashStrings)
// 不在 rows.Next() 循环中操作是为了避免死锁
for _, hash := range hashStrings {
err = this.Remove(hash)
if err != nil {
return 0, err
}
// 不在 rows.Next() 循环中操作是为了避免死锁
for _, hash := range hashStrings {
err = this.Remove(hash)
if err != nil {
return 0, err
}
err = callback(hash)
if err != nil {
return 0, err
err = callback(hash)
if err != nil {
return 0, err
}
}
}
@@ -365,80 +207,80 @@ func (this *FileList) Purge(count int, callback func(hash string) error) (int, e
}
func (this *FileList) PurgeLFU(count int, callback func(hash string) error) error {
if !this.isReady {
return nil
}
count /= CountFileDB
if count <= 0 {
return nil
count = 100
}
rows, err := this.lfuHitsStmt.Query(count)
if err != nil {
return err
}
hashStrings := []string{}
var countFound = 0
for rows.Next() {
var hash string
err = rows.Scan(&hash)
if err != nil {
_ = rows.Close()
return err
}
hashStrings = append(hashStrings, hash)
countFound++
}
_ = rows.Close() // 不能使用defer防止读写冲突
// 不在 rows.Next() 循环中操作是为了避免死锁
for _, hash := range hashStrings {
err = this.Remove(hash)
for _, db := range this.dbList {
hashStrings, err := db.ListLFUItems(count)
if err != nil {
return err
}
err = callback(hash)
if err != nil {
return err
// 不在 rows.Next() 循环中操作是为了避免死锁
for _, hash := range hashStrings {
notFound, err := this.remove(hash)
if err != nil {
return err
}
if notFound {
_, err = db.deleteHitByHashStmt.Exec(hash)
if err != nil {
return err
}
}
err = callback(hash)
if err != nil {
return err
}
}
}
return nil
}
func (this *FileList) CleanAll() error {
if !this.isReady {
return nil
defer this.memoryCache.Clean()
for _, db := range this.dbList {
err := db.CleanAll()
if err != nil {
return err
}
}
this.memoryCache.Clean()
_, err := this.deleteAllStmt.Exec()
if err != nil {
return err
}
atomic.StoreInt64(&this.total, 0)
return nil
}
func (this *FileList) Stat(check func(hash string) bool) (*Stat, error) {
if !this.isReady {
return &Stat{}, nil
var result = &Stat{}
for _, db := range this.dbList {
if !db.IsReady() {
return &Stat{}, nil
}
// 这里不设置过期时间、不使用 check 函数,目的是让查询更快速一些
_ = check
row := db.statStmt.QueryRow()
if row.Err() != nil {
return nil, row.Err()
}
var stat = &Stat{}
err := row.Scan(&stat.Count, &stat.Size, &stat.ValueSize)
if err != nil {
return nil, err
}
result.Count += stat.Count
result.Size += stat.Size
result.ValueSize += stat.ValueSize
}
// 这里不设置过期时间、不使用 check 函数,目的是让查询更快速一些
row := this.statStmt.QueryRow()
if row.Err() != nil {
return nil, row.Err()
}
stat := &Stat{}
err := row.Scan(&stat.Count, &stat.Size, &stat.ValueSize)
if err != nil {
return nil, err
}
return stat, nil
return result, nil
}
// Count 总数量
@@ -449,9 +291,13 @@ func (this *FileList) Count() (int64, error) {
// IncreaseHit 增加点击量
func (this *FileList) IncreaseHit(hash string) error {
var week = timeutil.Format("YW")
_, err := this.increaseHitStmt.Exec(hash, week, week, week, week)
return err
var db = this.getDB(hash)
if !db.IsReady() {
return nil
}
return db.IncreaseHit(hash)
}
// OnAdd 添加事件
@@ -465,162 +311,177 @@ func (this *FileList) OnRemove(f func(item *Item)) {
}
func (this *FileList) Close() error {
this.isClosed = true
this.isReady = false
this.memoryCache.Destroy()
if this.db != nil {
_ = this.existsByHashStmt.Close()
_ = this.insertStmt.Close()
_ = this.selectByHashStmt.Close()
_ = this.deleteByHashStmt.Close()
_ = this.statStmt.Close()
_ = this.purgeStmt.Close()
_ = this.deleteAllStmt.Close()
_ = this.insertHitStmt.Close()
_ = this.increaseHitStmt.Close()
_ = this.deleteHitByHashStmt.Close()
_ = this.lfuHitsStmt.Close()
return this.db.Close()
for _, db := range this.dbList {
if db != nil {
_ = db.Close()
}
}
return nil
}
// 初始化
func (this *FileList) initTables(db *sql.DB, times int) error {
// 检查是否存在
_, err := db.Exec(`SELECT id FROM "` + this.itemsTableName + `" LIMIT 1`)
var notFound = false
func (this *FileList) GetDBIndex(hash string) uint64 {
return fnv.HashString(hash) % CountFileDB
}
func (this *FileList) getDB(hash string) *FileListDB {
return this.dbList[fnv.HashString(hash)%CountFileDB]
}
func (this *FileList) remove(hash string) (notFound bool, err error) {
var db = this.getDB(hash)
if !db.IsReady() {
return false, nil
}
// 从缓存中删除
this.memoryCache.Delete(hash)
var row = db.selectByHashStmt.QueryRow(hash)
if row.Err() != nil {
if row.Err() == sql.ErrNoRows {
return true, nil
}
return false, row.Err()
}
var item = &Item{Type: ItemTypeFile}
err = row.Scan(&item.Key, &item.HeaderSize, &item.BodySize, &item.MetaSize, &item.ExpiredAt)
if err != nil {
notFound = true
}
{
// expiredAt - 过期时间,用来判断有无过期
// staleAt - 陈旧最大时间,用来清理缓存
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"key" varchar(1024),
"headerSize" integer DEFAULT 0,
"bodySize" integer DEFAULT 0,
"metaSize" integer DEFAULT 0,
"expiredAt" integer DEFAULT 0,
"staleAt" integer DEFAULT 0,
"createdAt" integer DEFAULT 0,
"host" varchar(128),
"serverId" integer
);
CREATE INDEX IF NOT EXISTS "createdAt"
ON "` + this.itemsTableName + `" (
"createdAt" ASC
);
CREATE INDEX IF NOT EXISTS "expiredAt"
ON "` + this.itemsTableName + `" (
"expiredAt" ASC
);
CREATE INDEX IF NOT EXISTS "staleAt"
ON "` + this.itemsTableName + `" (
"staleAt" ASC
);
CREATE UNIQUE INDEX IF NOT EXISTS "hash"
ON "` + this.itemsTableName + `" (
"hash" ASC
);
CREATE INDEX IF NOT EXISTS "serverId"
ON "` + this.itemsTableName + `" (
"serverId" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.itemsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
}
return err
}
return err
if err == sql.ErrNoRows {
return true, nil
}
return false, err
}
// 如果数据为空,从老数据中加载数据
if notFound {
// v2 => v3
remotelogs.Println("CACHE", "transferring old data from v2 to v3 ...")
result, err := db.Exec(`INSERT INTO "` + this.itemsTableName + `" ("id", "hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "createdAt", "host", "serverId", "staleAt") SELECT "id", "hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "createdAt", "host", "serverId", "expiredAt"+600 FROM cacheItems_v2`)
if err == nil {
count, _ := result.RowsAffected()
remotelogs.Println("CACHE", "transfer old data from v2 to v3 finished, "+types.String(count)+" rows transferred")
}
_, err = db.deleteByHashStmt.Exec(hash)
if err != nil {
return false, err
}
{
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.hitsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"week1Hits" integer DEFAULT 0,
"week2Hits" integer DEFAULT 0,
"week" varchar(6)
);
atomic.AddInt64(&this.total, -1)
CREATE UNIQUE INDEX IF NOT EXISTS "hits_hash"
ON "` + this.hitsTableName + `" (
"hash" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.hitsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
}
return err
}
return err
}
_, err = db.deleteHitByHashStmt.Exec(hash)
if err != nil {
return false, err
}
return nil
if this.onRemove != nil {
this.onRemove(item)
}
return false, nil
}
// 删除过期不用的表格
func (this *FileList) removeOldTables() error {
rows, err := this.db.Query(`SELECT "name" FROM sqlite_master WHERE "type"='table'`)
// 升级老版本数据库
func (this *FileList) upgradeOldDB() {
if len(this.oldDir) == 0 {
return
}
_ = this.UpgradeV3(this.oldDir, false)
}
func (this *FileList) UpgradeV3(oldDir string, brokenOnError bool) error {
// index.db
var indexDBPath = oldDir + "/index.db"
_, err := os.Stat(indexDBPath)
if err != nil {
return nil
}
remotelogs.Println("CACHE", "upgrading local database from '"+oldDir+"' ...")
defer func() {
_ = os.Remove(indexDBPath)
remotelogs.Println("CACHE", "upgrading local database finished")
}()
db, err := sql.Open("sqlite3", "file:"+indexDBPath+"?cache=shared&mode=rwc&_journal_mode=WAL&_sync=OFF")
if err != nil {
return err
}
defer func() {
_ = rows.Close()
_ = db.Close()
}()
for rows.Next() {
var name string
err = rows.Scan(&name)
var isFinished = false
var offset = 0
var count = 10000
for {
if isFinished {
break
}
err = func() error {
defer func() {
offset += count
}()
rows, err := db.Query(`SELECT "hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "staleAt", "createdAt", "host", "serverId" FROM "cacheItems_v3" ORDER BY "id" ASC LIMIT ?, ?`, offset, count)
if err != nil {
return err
}
defer func() {
_ = rows.Close()
}()
var hash = ""
var key = ""
var headerSize int64
var bodySize int64
var metaSize int64
var expiredAt int64
var staleAt int64
var createdAt int64
var host string
var serverId int64
isFinished = true
for rows.Next() {
isFinished = false
err = rows.Scan(&hash, &key, &headerSize, &bodySize, &metaSize, &expiredAt, &staleAt, &createdAt, &host, &serverId)
if err != nil {
if brokenOnError {
return err
}
return nil
}
err = this.Add(hash, &Item{
Type: ItemTypeFile,
Key: key,
ExpiredAt: expiredAt,
StaleAt: staleAt,
HeaderSize: headerSize,
BodySize: bodySize,
MetaSize: metaSize,
Host: host,
ServerId: serverId,
Week1Hits: 0,
Week2Hits: 0,
Week: 0,
})
if err != nil {
if brokenOnError {
return err
}
}
}
return nil
}()
if err != nil {
return err
}
if lists.ContainsString(this.oldTables, name) {
// 异步执行
goman.New(func() {
remotelogs.Println("CACHE", "remove old table '"+name+"' ...")
_, _ = this.db.Exec(`DROP TABLE "` + name + `"`)
remotelogs.Println("CACHE", "remove old table '"+name+"' done")
})
}
time.Sleep(1 * time.Second)
}
return nil
}

View File

@@ -0,0 +1,474 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
import (
"database/sql"
"errors"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/dbs"
"github.com/iwind/TeaGo/types"
timeutil "github.com/iwind/TeaGo/utils/time"
"runtime"
"strings"
"time"
)
type FileListDB struct {
readDB *dbs.DB
writeDB *dbs.DB
itemsTableName string
hitsTableName string
total int64
isClosed bool
isReady bool
// cacheItems
existsByHashStmt *dbs.Stmt // 根据hash检查是否存在
insertStmt *dbs.Stmt // 写入数据
selectByHashStmt *dbs.Stmt // 使用hash查询数据
deleteByHashStmt *dbs.Stmt // 根据hash删除数据
statStmt *dbs.Stmt // 统计
purgeStmt *dbs.Stmt // 清理
deleteAllStmt *dbs.Stmt // 删除所有数据
listOlderItemsStmt *dbs.Stmt // 读取较早存储的缓存
// hits
insertHitStmt *dbs.Stmt // 写入数据
increaseHitStmt *dbs.Stmt // 增加点击量
deleteHitByHashStmt *dbs.Stmt // 根据hash删除数据
lfuHitsStmt *dbs.Stmt // 读取老的数据
}
func NewFileListDB() *FileListDB {
return &FileListDB{}
}
func (this *FileListDB) Open(dbPath string) error {
// write db
writeDB, err := sql.Open("sqlite3", "file:"+dbPath+"?cache=private&mode=rwc&_journal_mode=WAL&_sync=OFF&_cache_size=32000&_secure_delete=FAST")
if err != nil {
return errors.New("open write database failed: " + err.Error())
}
writeDB.SetMaxOpenConns(1)
// TODO 耗时过长,暂时不整理数据库
// TODO 需要根据行数来判断是否VACUUM
/**_, err = db.Exec("VACUUM")
if err != nil {
return err
}**/
this.writeDB = dbs.NewDB(writeDB)
if teaconst.EnableDBStat {
this.writeDB.EnableStat(true)
}
// read db
readDB, err := sql.Open("sqlite3", "file:"+dbPath+"?cache=private&mode=ro&_journal_mode=WAL&_sync=OFF&_cache_size=32000")
if err != nil {
return errors.New("open read database failed: " + err.Error())
}
readDB.SetMaxOpenConns(runtime.NumCPU())
this.readDB = dbs.NewDB(readDB)
if teaconst.EnableDBStat {
this.readDB.EnableStat(true)
}
return nil
}
func (this *FileListDB) Init() error {
this.itemsTableName = "cacheItems"
this.hitsTableName = "hits"
// 创建
var err = this.initTables(1)
if err != nil {
return errors.New("init tables failed: " + err.Error())
}
// 读取总数量
row := this.readDB.QueryRow(`SELECT COUNT(*) FROM "` + this.itemsTableName + `"`)
if row.Err() != nil {
return row.Err()
}
var total int64
err = row.Scan(&total)
if err != nil {
return err
}
this.total = total
// 常用语句
this.existsByHashStmt, err = this.readDB.Prepare(`SELECT "expiredAt" FROM "` + this.itemsTableName + `" INDEXED BY "hash" WHERE "hash"=? AND expiredAt>? LIMIT 1`)
if err != nil {
return err
}
this.insertStmt, err = this.writeDB.Prepare(`INSERT INTO "` + this.itemsTableName + `" ("hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "staleAt", "host", "serverId", "createdAt") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
return err
}
this.selectByHashStmt, err = this.readDB.Prepare(`SELECT "key", "headerSize", "bodySize", "metaSize", "expiredAt" FROM "` + this.itemsTableName + `" WHERE "hash"=? LIMIT 1`)
if err != nil {
return err
}
this.deleteByHashStmt, err = this.writeDB.Prepare(`DELETE FROM "` + this.itemsTableName + `" WHERE "hash"=?`)
if err != nil {
return err
}
this.statStmt, err = this.readDB.Prepare(`SELECT COUNT(*), IFNULL(SUM(headerSize+bodySize+metaSize), 0), IFNULL(SUM(headerSize+bodySize), 0) FROM "` + this.itemsTableName + `"`)
if err != nil {
return err
}
this.purgeStmt, err = this.readDB.Prepare(`SELECT "hash" FROM "` + this.itemsTableName + `" WHERE staleAt<=? LIMIT ?`)
if err != nil {
return err
}
this.deleteAllStmt, err = this.writeDB.Prepare(`DELETE FROM "` + this.itemsTableName + `"`)
if err != nil {
return err
}
this.listOlderItemsStmt, err = this.readDB.Prepare(`SELECT "hash" FROM "` + this.itemsTableName + `" ORDER BY "id" ASC LIMIT ?`)
this.insertHitStmt, err = this.writeDB.Prepare(`INSERT INTO "` + this.hitsTableName + `" ("hash", "week2Hits", "week") VALUES (?, 1, ?)`)
this.increaseHitStmt, err = this.writeDB.Prepare(`INSERT INTO "` + this.hitsTableName + `" ("hash", "week2Hits", "week") VALUES (?, 1, ?) ON CONFLICT("hash") DO UPDATE SET "week1Hits"=IIF("week"=?, "week1Hits", "week2Hits"), "week2Hits"=IIF("week"=?, "week2Hits"+1, 1), "week"=?`)
if err != nil {
return err
}
this.deleteHitByHashStmt, err = this.writeDB.Prepare(`DELETE FROM "` + this.hitsTableName + `" WHERE "hash"=?`)
if err != nil {
return err
}
this.lfuHitsStmt, err = this.readDB.Prepare(`SELECT "hash" FROM "` + this.hitsTableName + `" ORDER BY "week" ASC, "week1Hits"+"week2Hits" ASC LIMIT ?`)
if err != nil {
return err
}
this.isReady = true
return nil
}
func (this *FileListDB) IsReady() bool {
return this.isReady
}
func (this *FileListDB) Total() int64 {
return this.total
}
func (this *FileListDB) Add(hash string, item *Item) error {
if item.StaleAt == 0 {
item.StaleAt = item.ExpiredAt
}
// 放入队列
_, err := this.insertStmt.Exec(hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.StaleAt, item.Host, item.ServerId, utils.UnixTime())
if err != nil {
return err
}
return nil
}
func (this *FileListDB) ListExpiredItems(count int) (hashList []string, err error) {
if !this.isReady {
return nil, nil
}
if count <= 0 {
count = 100
}
rows, err := this.purgeStmt.Query(time.Now().Unix(), count)
if err != nil {
return nil, err
}
defer func() {
_ = rows.Close()
}()
for rows.Next() {
var hash string
err = rows.Scan(&hash)
if err != nil {
return nil, err
}
hashList = append(hashList, hash)
}
return hashList, nil
}
func (this *FileListDB) ListLFUItems(count int) (hashList []string, err error) {
if !this.isReady {
return nil, nil
}
if count <= 0 {
count = 100
}
hashList, err = this.listLFUItems(count)
if err != nil {
return
}
if len(hashList) > count/2 {
return
}
// 不足补齐
olderHashList, err := this.listOlderItems(count - len(hashList))
if err != nil {
return nil, err
}
hashList = append(hashList, olderHashList...)
return
}
func (this *FileListDB) IncreaseHit(hash string) error {
var week = timeutil.Format("YW")
_, err := this.increaseHitStmt.Exec(hash, week, week, week, week)
return err
}
func (this *FileListDB) CleanPrefix(prefix string) error {
if !this.isReady {
return nil
}
var count = int64(10000)
var staleLife = 600 // TODO 需要可以设置
var unixTime = utils.UnixTime() // 只删除当前的,不删除新的
for {
result, err := this.writeDB.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0,staleAt=? WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)=1 LIMIT `+types.String(count)+`)`, unixTime+int64(staleLife), unixTime, prefix)
if err != nil {
return err
}
affectedRows, err := result.RowsAffected()
if err != nil {
return err
}
if affectedRows < count {
return nil
}
}
}
func (this *FileListDB) CleanAll() error {
if !this.isReady {
return nil
}
_, err := this.deleteAllStmt.Exec()
if err != nil {
return err
}
return nil
}
func (this *FileListDB) Close() error {
this.isClosed = true
this.isReady = false
if this.existsByHashStmt != nil {
_ = this.existsByHashStmt.Close()
}
if this.insertStmt != nil {
_ = this.insertStmt.Close()
}
if this.selectByHashStmt != nil {
_ = this.selectByHashStmt.Close()
}
if this.deleteByHashStmt != nil {
_ = this.deleteByHashStmt.Close()
}
if this.statStmt != nil {
_ = this.statStmt.Close()
}
if this.purgeStmt != nil {
_ = this.purgeStmt.Close()
}
if this.deleteAllStmt != nil {
_ = this.deleteAllStmt.Close()
}
if this.listOlderItemsStmt != nil {
_ = this.listOlderItemsStmt.Close()
}
if this.insertHitStmt != nil {
_ = this.insertHitStmt.Close()
}
if this.increaseHitStmt != nil {
_ = this.increaseHitStmt.Close()
}
if this.deleteHitByHashStmt != nil {
_ = this.deleteHitByHashStmt.Close()
}
if this.lfuHitsStmt != nil {
_ = this.lfuHitsStmt.Close()
}
var errStrings []string
if this.readDB != nil {
err := this.readDB.Close()
if err != nil {
errStrings = append(errStrings, err.Error())
}
}
if this.writeDB != nil {
err := this.writeDB.Close()
if err != nil {
errStrings = append(errStrings, err.Error())
}
}
if len(errStrings) == 0 {
return nil
}
return errors.New("close database failed: " + strings.Join(errStrings, ", "))
}
// 初始化
func (this *FileListDB) initTables(times int) error {
{
// expiredAt - 过期时间,用来判断有无过期
// staleAt - 过时缓存最大时间,用来清理缓存
_, err := this.writeDB.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"key" varchar(1024),
"tag" varchar(64),
"headerSize" integer DEFAULT 0,
"bodySize" integer DEFAULT 0,
"metaSize" integer DEFAULT 0,
"expiredAt" integer DEFAULT 0,
"staleAt" integer DEFAULT 0,
"createdAt" integer DEFAULT 0,
"host" varchar(128),
"serverId" integer
);
DROP INDEX IF EXISTS "createdAt";
DROP INDEX IF EXISTS "expiredAt";
DROP INDEX IF EXISTS "serverId";
CREATE INDEX IF NOT EXISTS "staleAt"
ON "` + this.itemsTableName + `" (
"staleAt" ASC
);
CREATE UNIQUE INDEX IF NOT EXISTS "hash"
ON "` + this.itemsTableName + `" (
"hash" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := this.writeDB.Exec(`DROP TABLE "` + this.itemsTableName + `"`)
if dropErr == nil {
return this.initTables(times + 1)
}
return err
}
return err
}
}
{
_, err := this.writeDB.Exec(`CREATE TABLE IF NOT EXISTS "` + this.hitsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"week1Hits" integer DEFAULT 0,
"week2Hits" integer DEFAULT 0,
"week" varchar(6)
);
CREATE UNIQUE INDEX IF NOT EXISTS "hits_hash"
ON "` + this.hitsTableName + `" (
"hash" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := this.writeDB.Exec(`DROP TABLE "` + this.hitsTableName + `"`)
if dropErr == nil {
return this.initTables(times + 1)
}
return err
}
return err
}
}
return nil
}
func (this *FileListDB) listLFUItems(count int) (hashList []string, err error) {
rows, err := this.lfuHitsStmt.Query(count)
if err != nil {
return nil, err
}
defer func() {
_ = rows.Close()
}()
for rows.Next() {
var hash string
err = rows.Scan(&hash)
if err != nil {
return nil, err
}
hashList = append(hashList, hash)
}
return hashList, nil
}
func (this *FileListDB) listOlderItems(count int) (hashList []string, err error) {
rows, err := this.listOlderItemsStmt.Query(count)
if err != nil {
return nil, err
}
defer func() {
_ = rows.Close()
}()
for rows.Next() {
var hash string
err = rows.Scan(&hash)
if err != nil {
return nil, err
}
hashList = append(hashList, hash)
}
return hashList, nil
}

View File

@@ -1,8 +1,9 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
package caches_test
import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/rands"
@@ -15,23 +16,33 @@ import (
)
func TestFileList_Init(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
t.Log("ok")
}
func TestFileList_Add(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1").(*caches.FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
err = list.Add(stringutil.Md5("123456"), &Item{
defer func() {
_ = list.Close()
}()
var hash = stringutil.Md5("123456")
t.Log("db index:", list.GetDBIndex(hash))
err = list.Add(hash, &caches.Item{
Key: "123456",
ExpiredAt: time.Now().Unix(),
ExpiredAt: time.Now().Unix() + 1,
HeaderSize: 1,
MetaSize: 2,
BodySize: 3,
@@ -41,19 +52,27 @@ func TestFileList_Add(t *testing.T) {
if err != nil {
t.Fatal(err)
}
t.Log(list.Exist(hash))
t.Log("ok")
}
func TestFileList_Add_Many(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
before := time.Now()
for i := 0; i < 2000_0000; i++ {
for i := 0; i < 100_000; i++ {
u := "https://edge.teaos.cn/123456" + strconv.Itoa(i)
_ = list.Add(stringutil.Md5(u), &Item{
_ = list.Add(stringutil.Md5(u), &caches.Item{
Key: u,
ExpiredAt: time.Now().Unix() + 3600,
HeaderSize: 1,
@@ -72,36 +91,46 @@ func TestFileList_Add_Many(t *testing.T) {
}
func TestFileList_Exist(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1").(*caches.FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
before := time.Now()
defer func() {
_ = list.Close()
}()
total, _ := list.Count()
t.Log("total:", total)
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
{
exists, err := list.Exist(stringutil.Md5("123456"))
var hash = stringutil.Md5("123456")
exists, err := list.Exist(hash)
if err != nil {
t.Fatal(err)
}
t.Log("exists:", exists)
t.Log(hash, "exists:", exists)
}
{
exists, err := list.Exist(stringutil.Md5("http://edge.teaos.cn/1234561"))
var hash = stringutil.Md5("http://edge.teaos.cn/1234561")
exists, err := list.Exist(hash)
if err != nil {
t.Fatal(err)
}
t.Log("exists:", exists)
t.Log(hash, "exists:", exists)
}
}
func TestFileList_Exist_Many_DB(t *testing.T) {
// 测试在多个数据库下的性能
var listSlice = []ListInterface{}
var listSlice = []caches.ListInterface{}
for i := 1; i <= 10; i++ {
list := NewFileList(Tea.Root + "/data/data" + strconv.Itoa(i))
list := caches.NewFileList(Tea.Root + "/data/data" + strconv.Itoa(i))
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -151,13 +180,18 @@ func TestFileList_Exist_Many_DB(t *testing.T) {
}
func TestFileList_CleanPrefix(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
before := time.Now()
err = list.CleanPrefix("1234")
err = list.CleanPrefix("123")
if err != nil {
t.Fatal(err)
}
@@ -165,12 +199,17 @@ func TestFileList_CleanPrefix(t *testing.T) {
}
func TestFileList_Remove(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1").(*caches.FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
list.OnRemove(func(item *Item) {
defer func() {
_ = list.Close()
}()
list.OnRemove(func(item *caches.Item) {
t.Logf("remove %#v", item)
})
err = list.Remove(stringutil.Md5("123456"))
@@ -178,30 +217,71 @@ func TestFileList_Remove(t *testing.T) {
t.Fatal(err)
}
t.Log("ok")
t.Log("===count===")
t.Log(list.Count())
}
func TestFileList_Purge(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
_, err = list.Purge(2, func(hash string) error {
defer func() {
_ = list.Close()
}()
var count = 0
_, err = list.Purge(caches.CountFileDB*2, func(hash string) error {
t.Log(hash)
count++
return nil
})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
t.Log("ok, purged", count, "items")
}
func TestFileList_Stat(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
func TestFileList_PurgeLFU(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
err = list.IncreaseHit(stringutil.Md5("123456"))
if err != nil {
t.Fatal(err)
}
var count = 0
err = list.PurgeLFU(caches.CountFileDB*2, func(hash string) error {
t.Log(hash)
count++
return nil
})
if err != nil {
t.Fatal(err)
}
t.Log("ok, purged", count, "items")
}
func TestFileList_Stat(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
stat, err := list.Stat(nil)
if err != nil {
t.Fatal(err)
@@ -210,7 +290,7 @@ func TestFileList_Stat(t *testing.T) {
}
func TestFileList_Count(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
list := caches.NewFileList(Tea.Root + "/data")
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -225,7 +305,7 @@ func TestFileList_Count(t *testing.T) {
}
func TestFileList_CleanAll(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
list := caches.NewFileList(Tea.Root + "/data")
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -238,58 +318,17 @@ func TestFileList_CleanAll(t *testing.T) {
t.Log(list.Count())
}
func TestFileList_Conflict(t *testing.T) {
list := NewFileList(Tea.Root + "/data").(*FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
rows, err := list.purgeStmt.Query(time.Now().Unix(), 1000)
if err != nil {
t.Fatal(err)
}
go func() {
time.Sleep(5 * time.Second)
_ = rows.Close()
}()
t.Log("before exists")
t.Log(list.Exist("123456"))
t.Log("after exists")
}
func TestFileList_IIF(t *testing.T) {
list := NewFileList(Tea.Root + "/data").(*FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
rows, err := list.db.Query("SELECT IIF(0, 2, 3)")
if err != nil {
t.Fatal(err)
}
defer func() {
_ = rows.Close()
}()
if rows.Next() {
var result int
err = rows.Scan(&result)
if err != nil {
t.Fatal(err)
}
t.Log("result:", result)
}
}
func TestFileList_IncreaseHit(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
@@ -303,13 +342,32 @@ func TestFileList_IncreaseHit(t *testing.T) {
t.Log("ok")
}
func TestFileList_UpgradeV3(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p43").(*caches.FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
err = list.UpgradeV3("/Users/WorkSpace/EdgeProject/EdgeCache/p43", false)
if err != nil {
t.Log(err)
return
}
t.Log("ok")
}
func BenchmarkFileList_Exist(b *testing.B) {
list := NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
b.Fatal(err)
}
for i := 0; i < b.N; i++ {
_, _ = list.Exist("f0eb5b87e0b0041f3917002c0707475f")
_, _ = list.Exist("f0eb5b87e0b0041f3917002c0707475f" + types.String(i))
}
}

View File

@@ -25,6 +25,7 @@ func init() {
type Manager struct {
// 全局配置
MaxDiskCapacity *shared.SizeCapacity
DiskDir string
MaxMemoryCapacity *shared.SizeCapacity
policyMap map[int64]*serverconfigs.HTTPCachePolicy // policyId => []*Policy
@@ -49,13 +50,18 @@ func (this *Manager) UpdatePolicies(newPolicies []*serverconfigs.HTTPCachePolicy
newPolicyIds := []int64{}
for _, policy := range newPolicies {
// 使用节点单独的缓存目录
if len(this.DiskDir) > 0 {
policy.UpdateDiskDir(this.DiskDir)
}
newPolicyIds = append(newPolicyIds, policy.Id)
}
// 停止旧有的
for _, oldPolicy := range this.policyMap {
if !lists.ContainsInt64(newPolicyIds, oldPolicy.Id) {
remotelogs.Error("CACHE", "remove policy "+strconv.FormatInt(oldPolicy.Id, 10))
remotelogs.Println("CACHE", "remove policy "+strconv.FormatInt(oldPolicy.Id, 10))
delete(this.policyMap, oldPolicy.Id)
storage, ok := this.storageMap[oldPolicy.Id]
if ok {
@@ -99,7 +105,19 @@ func (this *Manager) UpdatePolicies(newPolicies []*serverconfigs.HTTPCachePolicy
} else {
// 检查policy是否有变化
if !storage.Policy().IsSame(policy) {
remotelogs.Println("CACHE", "policy "+strconv.FormatInt(policy.Id, 10)+" changed")
// 检查是否可以直接修改
if storage.CanUpdatePolicy(policy) {
err := policy.Init()
if err != nil {
remotelogs.Error("CACHE", "reload policy '"+types.String(policy.Id)+"' failed: init policy failed: "+err.Error())
continue
}
remotelogs.Println("CACHE", "reload policy '"+types.String(policy.Id)+"'")
storage.UpdatePolicy(policy)
continue
}
remotelogs.Println("CACHE", "restart policy '"+types.String(policy.Id)+"'")
// 停止老的
storage.Stop()
@@ -196,3 +214,15 @@ func (this *Manager) FindAllCachePaths() []string {
}
return result
}
// FindAllStorages 读取所有缓存存储
func (this *Manager) FindAllStorages() []StorageInterface {
this.locker.Lock()
defer this.locker.Unlock()
var storages = []StorageInterface{}
for _, storage := range this.storageMap {
storages = append(storages, storage)
}
return storages
}

View File

@@ -2,19 +2,20 @@ package caches
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
"github.com/iwind/TeaGo/Tea"
"testing"
)
func TestManager_UpdatePolicies(t *testing.T) {
{
policies := []*serverconfigs.HTTPCachePolicy{}
var policies = []*serverconfigs.HTTPCachePolicy{}
SharedManager.UpdatePolicies(policies)
printManager(t)
}
{
policies := []*serverconfigs.HTTPCachePolicy{
var policies = []*serverconfigs.HTTPCachePolicy{
{
Id: 1,
Type: serverconfigs.CachePolicyStorageFile,
@@ -42,7 +43,7 @@ func TestManager_UpdatePolicies(t *testing.T) {
}
{
policies := []*serverconfigs.HTTPCachePolicy{
var policies = []*serverconfigs.HTTPCachePolicy{
{
Id: 1,
Type: serverconfigs.CachePolicyStorageFile,
@@ -71,6 +72,50 @@ func TestManager_UpdatePolicies(t *testing.T) {
}
}
func TestManager_ChangePolicy_Memory(t *testing.T) {
var policies = []*serverconfigs.HTTPCachePolicy{
{
Id: 1,
Type: serverconfigs.CachePolicyStorageMemory,
Options: map[string]interface{}{},
Capacity: &shared.SizeCapacity{Count: 1, Unit: shared.SizeCapacityUnitGB},
},
}
SharedManager.UpdatePolicies(policies)
SharedManager.UpdatePolicies([]*serverconfigs.HTTPCachePolicy{
{
Id: 1,
Type: serverconfigs.CachePolicyStorageMemory,
Options: map[string]interface{}{},
Capacity: &shared.SizeCapacity{Count: 2, Unit: shared.SizeCapacityUnitGB},
},
})
}
func TestManager_ChangePolicy_File(t *testing.T) {
var policies = []*serverconfigs.HTTPCachePolicy{
{
Id: 1,
Type: serverconfigs.CachePolicyStorageFile,
Options: map[string]interface{}{
"dir": Tea.Root + "/data/cache-index/p1",
},
Capacity: &shared.SizeCapacity{Count: 1, Unit: shared.SizeCapacityUnitGB},
},
}
SharedManager.UpdatePolicies(policies)
SharedManager.UpdatePolicies([]*serverconfigs.HTTPCachePolicy{
{
Id: 1,
Type: serverconfigs.CachePolicyStorageFile,
Options: map[string]interface{}{
"dir": Tea.Root + "/data/cache-index/p1",
},
Capacity: &shared.SizeCapacity{Count: 2, Unit: shared.SizeCapacityUnitGB},
},
})
}
func printManager(t *testing.T) {
t.Log("===manager==")
t.Log("storage:")

View File

@@ -0,0 +1,98 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package caches
import (
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"sync/atomic"
"time"
)
const (
minOpenFilesValue int32 = 4
maxOpenFilesValue int32 = 65535
modeSlow int32 = 1
modeFast int32 = 2
)
// MaxOpenFiles max open files manager
type MaxOpenFiles struct {
step int32
maxOpenFiles int32
ptr *int32
ticker *time.Ticker
mode int32
lastOpens int32
currentOpens int32
}
func NewMaxOpenFiles(step int32) *MaxOpenFiles {
if step <= 0 {
step = 2
}
var f = &MaxOpenFiles{
step: step,
maxOpenFiles: minOpenFilesValue,
}
if teaconst.DiskIsFast {
f.maxOpenFiles = 32
}
f.ptr = &f.maxOpenFiles
f.ticker = time.NewTicker(1 * time.Second)
f.init()
return f
}
func (this *MaxOpenFiles) init() {
goman.New(func() {
for range this.ticker.C {
var mod = atomic.LoadInt32(&this.mode)
switch mod {
case modeSlow:
// we decrease more quickly, with more steps
if atomic.AddInt32(this.ptr, -this.step*2) <= 0 {
atomic.StoreInt32(this.ptr, minOpenFilesValue)
}
case modeFast:
// we increase only when file opens increases
var currentOpens = atomic.LoadInt32(&this.currentOpens)
if currentOpens > this.lastOpens {
if atomic.AddInt32(this.ptr, this.step) >= maxOpenFilesValue {
atomic.StoreInt32(this.ptr, maxOpenFilesValue)
}
}
this.lastOpens = currentOpens
atomic.StoreInt32(&this.currentOpens, 0)
}
// reset mode
atomic.StoreInt32(&this.mode, 0)
}
})
}
func (this *MaxOpenFiles) Fast() {
if atomic.LoadInt32(&this.mode) == 0 {
this.mode = modeFast
}
atomic.AddInt32(&this.currentOpens, 1)
}
func (this *MaxOpenFiles) Slow() {
atomic.StoreInt32(&this.mode, modeSlow)
}
func (this *MaxOpenFiles) Max() int32 {
if atomic.LoadInt32(&this.mode) == modeSlow {
return 0
}
var v = atomic.LoadInt32(this.ptr)
if v <= minOpenFilesValue {
return minOpenFilesValue
}
return v
}

View File

@@ -0,0 +1,28 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package caches_test
import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
"testing"
"time"
)
func TestNewMaxOpenFiles(t *testing.T) {
var maxOpenFiles = caches.NewMaxOpenFiles(2)
maxOpenFiles.Fast()
t.Log(maxOpenFiles.Max())
maxOpenFiles.Fast()
time.Sleep(1 * time.Second)
t.Log(maxOpenFiles.Max())
maxOpenFiles.Slow()
t.Log(maxOpenFiles.Max())
maxOpenFiles.Slow()
t.Log(maxOpenFiles.Max())
maxOpenFiles.Slow()
t.Log(maxOpenFiles.Max())
}

View File

@@ -14,11 +14,12 @@ type OpenFile struct {
version int64
}
func NewOpenFile(fp *os.File, meta []byte, header []byte) *OpenFile {
func NewOpenFile(fp *os.File, meta []byte, header []byte, version int64) *OpenFile {
return &OpenFile{
fp: fp,
meta: meta,
header: header,
fp: fp,
meta: meta,
header: header,
version: version,
}
}

View File

@@ -9,6 +9,7 @@ import (
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/types"
"path/filepath"
"runtime"
"sync"
"time"
)
@@ -43,7 +44,7 @@ func NewOpenFileCache(maxSize int) (*OpenFileCache, error) {
goman.New(func() {
for event := range watcher.Events {
if event.Op&fsnotify.Chmod != fsnotify.Chmod {
if runtime.GOOS == "linux" || event.Op&fsnotify.Chmod != fsnotify.Chmod {
cache.Close(event.Name)
}
}
@@ -77,6 +78,7 @@ func (this *OpenFileCache) Put(filename string, file *OpenFile) {
} else {
_ = this.watcher.Add(filename)
pool = NewOpenFilePool(filename)
pool.version = file.version
this.poolMap[filename] = pool
success = pool.Put(file)
}

View File

@@ -45,7 +45,7 @@ func (this *OpenFilePool) Get() (*OpenFile, bool) {
}
func (this *OpenFilePool) Put(file *OpenFile) bool {
if file.version > 0 && file.version != this.version {
if this.version > 0 && file.version > 0 && file.version != this.version {
_ = file.Close()
return false
}

View File

@@ -11,7 +11,7 @@ func TestOpenFilePool_Get(t *testing.T) {
var pool = caches.NewOpenFilePool("a")
t.Log(pool.Filename())
t.Log(pool.Get())
t.Log(pool.Put(caches.NewOpenFile(nil, nil, []byte{})))
t.Log(pool.Put(caches.NewOpenFile(nil, nil, nil, 0)))
t.Log(pool.Get())
t.Log(pool.Get())
}

View File

@@ -0,0 +1,189 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
import (
"encoding/json"
"errors"
"io/ioutil"
)
// PartialRanges 内容分区范围定义
type PartialRanges struct {
Ranges [][2]int64 `json:"ranges"`
}
// NewPartialRanges 获取新对象
func NewPartialRanges() *PartialRanges {
return &PartialRanges{Ranges: [][2]int64{}}
}
// NewPartialRangesFromJSON 从JSON中解析范围
func NewPartialRangesFromJSON(data []byte) (*PartialRanges, error) {
var rs = NewPartialRanges()
err := json.Unmarshal(data, &rs)
if err != nil {
return nil, err
}
return rs, nil
}
func NewPartialRangesFromFile(path string) (*PartialRanges, error) {
data, err := ioutil.ReadFile(path)
if err != nil {
return nil, err
}
return NewPartialRangesFromJSON(data)
}
// Add 添加新范围
func (this *PartialRanges) Add(begin int64, end int64) {
if begin > end {
begin, end = end, begin
}
var nr = [2]int64{begin, end}
var count = len(this.Ranges)
if count == 0 {
this.Ranges = [][2]int64{nr}
return
}
// insert
// TODO 将来使用二分法改进
var index = -1
for i, r := range this.Ranges {
if r[0] > begin || (r[0] == begin && r[1] >= end) {
index = i
this.Ranges = append(this.Ranges, [2]int64{})
copy(this.Ranges[index+1:], this.Ranges[index:])
this.Ranges[index] = nr
break
}
}
if index == -1 {
index = count
this.Ranges = append(this.Ranges, nr)
}
this.merge(index)
}
// Contains 检查是否包含某个范围
func (this *PartialRanges) Contains(begin int64, end int64) bool {
if len(this.Ranges) == 0 {
return false
}
// TODO 使用二分法查找改进性能
for _, r2 := range this.Ranges {
if r2[0] <= begin && r2[1] >= end {
return true
}
}
return false
}
// Nearest 查找最近的某个范围
func (this *PartialRanges) Nearest(begin int64, end int64) (r [2]int64, ok bool) {
if len(this.Ranges) == 0 {
return
}
// TODO 使用二分法查找改进性能
for _, r2 := range this.Ranges {
if r2[0] <= begin && r2[1] > begin {
r = [2]int64{begin, this.min(end, r2[1])}
ok = true
return
}
}
return
}
// AsJSON 转换为JSON
func (this *PartialRanges) AsJSON() ([]byte, error) {
return json.Marshal(this)
}
// WriteToFile 写入到文件中
func (this *PartialRanges) WriteToFile(path string) error {
data, err := this.AsJSON()
if err != nil {
return errors.New("convert to json failed: " + err.Error())
}
return ioutil.WriteFile(path, data, 0666)
}
// ReadFromFile 从文件中读取
func (this *PartialRanges) ReadFromFile(path string) (*PartialRanges, error) {
data, err := ioutil.ReadFile(path)
if err != nil {
return nil, err
}
return NewPartialRangesFromJSON(data)
}
func (this *PartialRanges) Max() int64 {
if len(this.Ranges) > 0 {
return this.Ranges[len(this.Ranges)-1][1]
}
return 0
}
func (this *PartialRanges) merge(index int) {
// forward
var lastIndex = index
for i := index; i >= 1; i-- {
var curr = this.Ranges[i]
var prev = this.Ranges[i-1]
var w1 = this.w(curr)
var w2 = this.w(prev)
if w1+w2 >= this.max(curr[1], prev[1])-this.min(curr[0], prev[0])-1 {
prev = [2]int64{this.min(curr[0], prev[0]), this.max(curr[1], prev[1])}
this.Ranges[i-1] = prev
this.Ranges = append(this.Ranges[:i], this.Ranges[i+1:]...)
lastIndex = i - 1
} else {
break
}
}
// backward
index = lastIndex
for index < len(this.Ranges)-1 {
var curr = this.Ranges[index]
var next = this.Ranges[index+1]
var w1 = this.w(curr)
var w2 = this.w(next)
if w1+w2 >= this.max(curr[1], next[1])-this.min(curr[0], next[0])-1 {
curr = [2]int64{this.min(curr[0], next[0]), this.max(curr[1], next[1])}
this.Ranges = append(this.Ranges[:index], this.Ranges[index+1:]...)
this.Ranges[index] = curr
} else {
break
}
}
}
func (this *PartialRanges) w(r [2]int64) int64 {
return r[1] - r[0]
}
func (this *PartialRanges) min(n1 int64, n2 int64) int64 {
if n1 <= n2 {
return n1
}
return n2
}
func (this *PartialRanges) max(n1 int64, n2 int64) int64 {
if n1 >= n2 {
return n1
}
return n2
}

View File

@@ -0,0 +1,175 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches_test
import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/iwind/TeaGo/assert"
"github.com/iwind/TeaGo/logs"
"testing"
)
func TestNewPartialRanges(t *testing.T) {
var r = caches.NewPartialRanges()
r.Add(1, 100)
r.Add(50, 300)
r.Add(30, 80)
r.Add(30, 100)
r.Add(30, 400)
r.Add(1000, 10000)
r.Add(200, 1000)
r.Add(200, 10040)
logs.PrintAsJSON(r.Ranges, t)
t.Log("max:", r.Max())
}
func TestNewPartialRanges1(t *testing.T) {
var a = assert.NewAssertion(t)
var r = caches.NewPartialRanges()
r.Add(1, 100)
r.Add(1, 101)
r.Add(1, 102)
r.Add(2, 103)
r.Add(200, 300)
r.Add(1, 1000)
var rs = r.Ranges
logs.PrintAsJSON(rs, t)
a.IsTrue(len(rs) == 1)
if len(rs) == 1 {
a.IsTrue(rs[0][0] == 1)
a.IsTrue(rs[0][1] == 1000)
}
}
func TestNewPartialRanges2(t *testing.T) {
// low -> high
var r = caches.NewPartialRanges()
r.Add(1, 100)
r.Add(1, 101)
r.Add(1, 102)
r.Add(2, 103)
r.Add(200, 300)
r.Add(301, 302)
r.Add(303, 304)
r.Add(250, 400)
var rs = r.Ranges
logs.PrintAsJSON(rs, t)
}
func TestNewPartialRanges3(t *testing.T) {
// high -> low
var r = caches.NewPartialRanges()
r.Add(301, 302)
r.Add(303, 304)
r.Add(200, 300)
r.Add(250, 400)
var rs = r.Ranges
logs.PrintAsJSON(rs, t)
}
func TestNewPartialRanges4(t *testing.T) {
// nearby
var r = caches.NewPartialRanges()
r.Add(301, 302)
r.Add(303, 304)
r.Add(305, 306)
r.Add(417, 417)
r.Add(410, 415)
r.Add(400, 409)
var rs = r.Ranges
logs.PrintAsJSON(rs, t)
t.Log(r.Contains(400, 416))
}
func TestNewPartialRanges5(t *testing.T) {
var r = caches.NewPartialRanges()
for j := 0; j < 1000; j++ {
r.Add(int64(j), int64(j+100))
}
logs.PrintAsJSON(r.Ranges, t)
}
func TestNewPartialRanges_Nearest(t *testing.T) {
{
// nearby
var r = caches.NewPartialRanges()
r.Add(301, 400)
r.Add(401, 500)
r.Add(501, 600)
t.Log(r.Nearest(100, 200))
t.Log(r.Nearest(300, 350))
t.Log(r.Nearest(302, 350))
}
{
// nearby
var r = caches.NewPartialRanges()
r.Add(301, 400)
r.Add(450, 500)
r.Add(550, 600)
t.Log(r.Nearest(100, 200))
t.Log(r.Nearest(300, 350))
t.Log(r.Nearest(302, 350))
t.Log(r.Nearest(302, 440))
t.Log(r.Nearest(302, 1000))
}
}
func TestNewPartialRanges_Large_Range(t *testing.T) {
var a = assert.NewAssertion(t)
var largeSize int64 = 10000000000000
t.Log(largeSize/1024/1024/1024, "G")
var r = caches.NewPartialRanges()
r.Add(1, largeSize)
jsonData, err := r.AsJSON()
if err != nil {
t.Fatal(err)
}
t.Log(string(jsonData))
r2, err := caches.NewPartialRangesFromJSON(jsonData)
if err != nil {
t.Fatal(err)
}
a.IsTrue(largeSize == r2.Ranges[0][1])
}
func TestNewPartialRanges_AsJSON(t *testing.T) {
var r = caches.NewPartialRanges()
for j := 0; j < 1000; j++ {
r.Add(int64(j), int64(j+100))
}
data, err := r.AsJSON()
if err != nil {
t.Fatal(err)
}
t.Log(string(data))
r2, err := caches.NewPartialRangesFromJSON(data)
if err != nil {
t.Fatal(err)
}
t.Log(r2.Ranges)
}
func BenchmarkNewPartialRanges(b *testing.B) {
for i := 0; i < b.N; i++ {
var r = caches.NewPartialRanges()
for j := 0; j < 1000; j++ {
r.Add(int64(j), int64(j+100))
}
}
}

View File

@@ -1,5 +1,7 @@
package caches
import "github.com/TeaOSLab/EdgeNode/internal/utils/ranges"
type ReaderFunc func(n int) (goNext bool, err error)
type Reader interface {
@@ -36,6 +38,9 @@ type Reader interface {
// BodySize Body Size
BodySize() int64
// ContainsRange 是否包含某个区间内容
ContainsRange(r rangeutils.Range) (r2 rangeutils.Range, ok bool)
// Close 关闭
Close() error
}

View File

@@ -3,6 +3,7 @@ package caches
import (
"encoding/binary"
"errors"
rangeutils "github.com/TeaOSLab/EdgeNode/internal/utils/ranges"
"github.com/iwind/TeaGo/types"
"io"
"os"
@@ -32,6 +33,10 @@ func NewFileReader(fp *os.File) *FileReader {
}
func (this *FileReader) Init() error {
return this.InitAutoDiscard(true)
}
func (this *FileReader) InitAutoDiscard(autoDiscard bool) error {
if this.openFile != nil {
this.meta = this.openFile.meta
this.header = this.openFile.header
@@ -39,11 +44,13 @@ func (this *FileReader) Init() error {
isOk := false
defer func() {
if !isOk {
_ = this.discard()
}
}()
if autoDiscard {
defer func() {
if !isOk {
_ = this.discard()
}
}()
}
var buf = this.meta
if len(buf) == 0 {
@@ -60,17 +67,17 @@ func (this *FileReader) Init() error {
this.expiresAt = int64(binary.BigEndian.Uint32(buf[:SizeExpiresAt]))
status := types.Int(string(buf[SizeExpiresAt : SizeExpiresAt+SizeStatus]))
status := types.Int(string(buf[OffsetStatus : OffsetStatus+SizeStatus]))
if status < 100 || status > 999 {
return errors.New("invalid status")
}
this.status = status
// URL
urlLength := binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus : SizeExpiresAt+SizeStatus+SizeURLLength])
urlLength := binary.BigEndian.Uint32(buf[OffsetURLLength : OffsetURLLength+SizeURLLength])
// header
headerSize := int(binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus+SizeURLLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength]))
headerSize := int(binary.BigEndian.Uint32(buf[OffsetHeaderLength : OffsetHeaderLength+SizeHeaderLength]))
if headerSize == 0 {
return nil
}
@@ -78,13 +85,13 @@ func (this *FileReader) Init() error {
this.headerOffset = int64(SizeMeta) + int64(urlLength)
// body
bodySize := int(binary.BigEndian.Uint64(buf[SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength+SizeBodyLength]))
this.bodyOffset = this.headerOffset + int64(headerSize)
bodySize := int(binary.BigEndian.Uint64(buf[OffsetBodyLength : OffsetBodyLength+SizeBodyLength]))
if bodySize == 0 {
isOk = true
return nil
}
this.bodySize = int64(bodySize)
this.bodyOffset = this.headerOffset + int64(headerSize)
// read header
if this.openFileCache != nil && len(this.header) == 0 {
@@ -326,6 +333,16 @@ func (this *FileReader) ReadBodyRange(buf []byte, start int64, end int64, callba
return nil
}
// ContainsRange 是否包含某些区间内容
func (this *FileReader) ContainsRange(r rangeutils.Range) (r2 rangeutils.Range, ok bool) {
return r, true
}
// FP 原始的文件句柄
func (this *FileReader) FP() *os.File {
return this.fp
}
func (this *FileReader) Close() error {
if this.openFileCache != nil {
if this.isClosed {
@@ -336,7 +353,9 @@ func (this *FileReader) Close() error {
if this.openFile != nil {
this.openFileCache.Put(this.fp.Name(), this.openFile)
} else {
this.openFileCache.Put(this.fp.Name(), NewOpenFile(this.fp, this.meta, this.header))
var cacheMeta = make([]byte, len(this.meta))
copy(cacheMeta, this.meta)
this.openFileCache.Put(this.fp.Name(), NewOpenFile(this.fp, cacheMeta, this.header, this.LastModified()))
}
return nil
}
@@ -355,5 +374,12 @@ func (this *FileReader) readToBuff(fp *os.File, buf []byte) (ok bool, err error)
func (this *FileReader) discard() error {
_ = this.fp.Close()
this.isClosed = true
// close open file cache
if this.openFileCache != nil {
this.openFileCache.Close(this.fp.Name())
}
// remove file
return os.Remove(this.fp.Name())
}

View File

@@ -54,6 +54,30 @@ func TestFileReader(t *testing.T) {
}
}
func TestFileReader_ReadHeader(t *testing.T) {
var path = "/Users/WorkSpace/EdgeProject/EdgeCache/p43/12/6b/126bbed90fc80f2bdfb19558948b0d49.cache"
fp, err := os.Open(path)
if err != nil {
t.Fatal(err)
}
defer func() {
_ = fp.Close()
}()
var reader = NewFileReader(fp)
err = reader.Init()
if err != nil {
t.Fatal(err)
}
var buf = make([]byte, 16*1024)
err = reader.ReadHeader(buf, func(n int) (goNext bool, err error) {
t.Log("header:", string(buf[:n]))
return
})
if err != nil {
t.Fatal(err)
}
}
func TestFileReader_Range(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,

View File

@@ -2,6 +2,7 @@ package caches
import (
"errors"
rangeutils "github.com/TeaOSLab/EdgeNode/internal/utils/ranges"
"io"
)
@@ -24,7 +25,7 @@ func (this *MemoryReader) TypeName() string {
}
func (this *MemoryReader) ExpiresAt() int64 {
return this.item.ExpiredAt
return this.item.ExpiresAt
}
func (this *MemoryReader) Status() int {
@@ -197,6 +198,11 @@ func (this *MemoryReader) ReadBodyRange(buf []byte, start int64, end int64, call
return nil
}
// ContainsRange 是否包含某些区间内容
func (this *MemoryReader) ContainsRange(r rangeutils.Range) (r2 rangeutils.Range, ok bool) {
return r, true
}
func (this *MemoryReader) Close() error {
return nil
}

View File

@@ -4,7 +4,7 @@ import "testing"
func TestMemoryReader_Header(t *testing.T) {
item := &MemoryItem{
ExpiredAt: 0,
ExpiresAt: 0,
HeaderValue: []byte("0123456789"),
BodyValue: nil,
Status: 2000,
@@ -22,7 +22,7 @@ func TestMemoryReader_Header(t *testing.T) {
func TestMemoryReader_Body(t *testing.T) {
item := &MemoryItem{
ExpiredAt: 0,
ExpiresAt: 0,
HeaderValue: nil,
BodyValue: []byte("0123456789"),
Status: 2000,
@@ -40,7 +40,7 @@ func TestMemoryReader_Body(t *testing.T) {
func TestMemoryReader_Body_Range(t *testing.T) {
item := &MemoryItem{
ExpiredAt: 0,
ExpiresAt: 0,
HeaderValue: nil,
BodyValue: []byte("0123456789"),
Status: 2000,

View File

@@ -0,0 +1,144 @@
package caches
import (
"encoding/binary"
"errors"
rangeutils "github.com/TeaOSLab/EdgeNode/internal/utils/ranges"
"github.com/iwind/TeaGo/types"
"io"
"os"
)
type PartialFileReader struct {
*FileReader
ranges *PartialRanges
rangePath string
}
func NewPartialFileReader(fp *os.File) *PartialFileReader {
return &PartialFileReader{
FileReader: NewFileReader(fp),
rangePath: partialRangesFilePath(fp.Name()),
}
}
func (this *PartialFileReader) Init() error {
return this.InitAutoDiscard(true)
}
func (this *PartialFileReader) InitAutoDiscard(autoDiscard bool) error {
if this.openFile != nil {
this.meta = this.openFile.meta
this.header = this.openFile.header
}
isOk := false
if autoDiscard {
defer func() {
if !isOk {
_ = this.discard()
}
}()
}
// 读取Range
ranges, err := NewPartialRangesFromFile(this.rangePath)
if err != nil {
return errors.New("read ranges failed: " + err.Error())
}
this.ranges = ranges
var buf = this.meta
if len(buf) == 0 {
buf = make([]byte, SizeMeta)
ok, err := this.readToBuff(this.fp, buf)
if err != nil {
return err
}
if !ok {
return ErrNotFound
}
this.meta = buf
}
this.expiresAt = int64(binary.BigEndian.Uint32(buf[:SizeExpiresAt]))
status := types.Int(string(buf[SizeExpiresAt : SizeExpiresAt+SizeStatus]))
if status < 100 || status > 999 {
return errors.New("invalid status")
}
this.status = status
// URL
urlLength := binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus : SizeExpiresAt+SizeStatus+SizeURLLength])
// header
headerSize := int(binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus+SizeURLLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength]))
if headerSize == 0 {
return nil
}
this.headerSize = headerSize
this.headerOffset = int64(SizeMeta) + int64(urlLength)
// body
this.bodyOffset = this.headerOffset + int64(headerSize)
bodySize := int(binary.BigEndian.Uint64(buf[SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength+SizeBodyLength]))
if bodySize == 0 {
isOk = true
return nil
}
this.bodySize = int64(bodySize)
// read header
if this.openFileCache != nil && len(this.header) == 0 {
if headerSize > 0 && headerSize <= 512 {
this.header = make([]byte, headerSize)
_, err := this.fp.Seek(this.headerOffset, io.SeekStart)
if err != nil {
return err
}
_, err = this.readToBuff(this.fp, this.header)
if err != nil {
return err
}
}
}
isOk = true
return nil
}
// ContainsRange 是否包含某些区间内容
// 这里的 r 是已经经过格式化的
func (this *PartialFileReader) ContainsRange(r rangeutils.Range) (r2 rangeutils.Range, ok bool) {
r2, ok = this.ranges.Nearest(r.Start(), r.End())
if ok && this.bodySize > 0 {
// 考虑可配置
var span int64 = 512 * 1024
if this.bodySize > 1<<30 {
span = 1 << 20
}
// 这里限制返回的最小缓存,防止因为返回的内容过小而导致请求过多
if r2.Length() < r.Length() && r2.Length() < span {
ok = false
}
}
return
}
// MaxLength 获取区间最大长度
func (this *PartialFileReader) MaxLength() int64 {
if this.bodySize > 0 {
return this.bodySize
}
return this.ranges.Max() + 1
}
func (this *PartialFileReader) discard() error {
_ = os.Remove(this.rangePath)
return this.FileReader.discard()
}

File diff suppressed because it is too large Load Diff

View File

@@ -62,7 +62,7 @@ func TestFileStorage_OpenWriter(t *testing.T) {
header := []byte("Header")
body := []byte("This is Body")
writer, err := storage.OpenWriter("my-key", time.Now().Unix()+86400, 200, -1)
writer, err := storage.OpenWriter("my-key", time.Now().Unix()+86400, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -87,6 +87,41 @@ func TestFileStorage_OpenWriter(t *testing.T) {
t.Log("ok")
}
func TestFileStorage_OpenWriter_Partial(t *testing.T) {
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 2,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
err := storage.Init()
if err != nil {
t.Fatal(err)
}
writer, err := storage.OpenWriter("my-key", time.Now().Unix()+86400, 200, -1, -1, true)
if err != nil {
t.Fatal(err)
}
_, err = writer.WriteHeader([]byte("Content-Type:text/html; charset=utf-8"))
if err != nil {
t.Fatal(err)
}
err = writer.WriteAt(0, []byte("Hello, World"))
if err != nil {
t.Fatal(err)
}
err = writer.Close()
if err != nil {
t.Fatal(err)
}
t.Log(writer)
}
func TestFileStorage_OpenWriter_HTTP(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
@@ -104,7 +139,7 @@ func TestFileStorage_OpenWriter_HTTP(t *testing.T) {
t.Log(time.Since(now).Seconds()*1000, "ms")
}()
writer, err := storage.OpenWriter("my-http-response", time.Now().Unix()+86400, 200, -1)
writer, err := storage.OpenWriter("my-http-response", time.Now().Unix()+86400, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -177,10 +212,11 @@ func TestFileStorage_Concurrent_Open_DifferentFile(t *testing.T) {
go func(i int) {
defer wg.Done()
writer, err := storage.OpenWriter("abc"+strconv.Itoa(i), time.Now().Unix()+3600, 200, -1)
writer, err := storage.OpenWriter("abc"+strconv.Itoa(i), time.Now().Unix()+3600, 200, -1, -1, false)
if err != nil {
if err != ErrFileIsWriting {
t.Fatal(err)
t.Error(err)
return
}
return
}
@@ -188,7 +224,8 @@ func TestFileStorage_Concurrent_Open_DifferentFile(t *testing.T) {
_, err = writer.Write([]byte("Hello,World"))
if err != nil {
t.Fatal(err)
t.Error(err)
return
}
// 故意造成慢速写入
@@ -196,7 +233,8 @@ func TestFileStorage_Concurrent_Open_DifferentFile(t *testing.T) {
err = writer.Close()
if err != nil {
t.Fatal(err)
t.Error(err)
return
}
}(i)
}
@@ -229,10 +267,11 @@ func TestFileStorage_Concurrent_Open_SameFile(t *testing.T) {
go func(i int) {
defer wg.Done()
writer, err := storage.OpenWriter("abc"+strconv.Itoa(0), time.Now().Unix()+3600, 200, -1)
writer, err := storage.OpenWriter("abc"+strconv.Itoa(0), time.Now().Unix()+3600, 200, -1, -1, false)
if err != nil {
if err != ErrFileIsWriting {
t.Fatal(err)
t.Error(err)
return
}
return
}
@@ -241,7 +280,8 @@ func TestFileStorage_Concurrent_Open_SameFile(t *testing.T) {
t.Log("writing")
_, err = writer.Write([]byte("Hello,World"))
if err != nil {
t.Fatal(err)
t.Error(err)
return
}
// 故意造成慢速写入
@@ -249,7 +289,8 @@ func TestFileStorage_Concurrent_Open_SameFile(t *testing.T) {
err = writer.Close()
if err != nil {
t.Fatal(err)
t.Error(err)
return
}
}(i)
}
@@ -270,7 +311,7 @@ func TestFileStorage_Read(t *testing.T) {
t.Fatal(err)
}
now := time.Now()
reader, err := storage.OpenReader("my-key", false)
reader, err := storage.OpenReader("my-key", false, false)
if err != nil {
t.Fatal(err)
}
@@ -306,7 +347,7 @@ func TestFileStorage_Read_HTTP_Response(t *testing.T) {
t.Fatal(err)
}
now := time.Now()
reader, err := storage.OpenReader("my-http-response", false)
reader, err := storage.OpenReader("my-http-response", false, false)
if err != nil {
t.Fatal(err)
}
@@ -360,7 +401,7 @@ func TestFileStorage_Read_NotFound(t *testing.T) {
}
now := time.Now()
buf := make([]byte, 6)
reader, err := storage.OpenReader("my-key-10000", false)
reader, err := storage.OpenReader("my-key-10000", false, false)
if err != nil {
if err == ErrNotFound {
t.Log("cache not fund")
@@ -470,7 +511,7 @@ func TestFileStorage_Stop(t *testing.T) {
}
func TestFileStorage_DecodeFile(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
@@ -485,6 +526,11 @@ func TestFileStorage_DecodeFile(t *testing.T) {
t.Log(path)
}
func TestFileStorage_RemoveCacheFile(t *testing.T) {
var storage = NewFileStorage(nil)
t.Log(storage.removeCacheFile("/Users/WorkSpace/EdgeProject/EdgeCache/p43/15/7e/157eba0dfc6dfb6fbbf20b1f9e584674.cache"))
}
func BenchmarkFileStorage_Read(b *testing.B) {
runtime.GOMAXPROCS(1)
@@ -502,7 +548,7 @@ func BenchmarkFileStorage_Read(b *testing.B) {
b.Fatal(err)
}
for i := 0; i < b.N; i++ {
reader, err := storage.OpenReader("my-key", false)
reader, err := storage.OpenReader("my-key", false, false)
if err != nil {
b.Fatal(err)
}
@@ -518,8 +564,8 @@ func BenchmarkFileStorage_KeyPath(b *testing.B) {
runtime.GOMAXPROCS(1)
var storage = &FileStorage{
cacheConfig: &serverconfigs.HTTPFileCacheStorage{},
policy: &serverconfigs.HTTPCachePolicy{Id: 1},
options: &serverconfigs.HTTPFileCacheStorage{},
policy: &serverconfigs.HTTPCachePolicy{Id: 1},
}
for i := 0; i < b.N; i++ {

View File

@@ -10,10 +10,14 @@ type StorageInterface interface {
Init() error
// OpenReader 读取缓存
OpenReader(key string, useStale bool) (reader Reader, err error)
OpenReader(key string, useStale bool, isPartial bool) (reader Reader, err error)
// OpenWriter 打开缓存写入器等待写入
OpenWriter(key string, expiredAt int64, status int, size int64) (Writer, error)
// size 和 maxSize 可能为-1
OpenWriter(key string, expiresAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error)
// OpenFlushWriter 打开从其他媒介直接刷入的写入器
OpenFlushWriter(key string, expiresAt int64, status int) (Writer, error)
// Delete 删除某个键值对应的缓存
Delete(key string) error
@@ -31,6 +35,7 @@ type StorageInterface interface {
CleanAll() error
// Purge 批量删除缓存
// urlType 值为file|dir
Purge(keys []string, urlType string) error
// Stop 停止缓存策略
@@ -39,6 +44,18 @@ type StorageInterface interface {
// Policy 获取当前存储的Policy
Policy() *serverconfigs.HTTPCachePolicy
// UpdatePolicy 修改策略
UpdatePolicy(newPolicy *serverconfigs.HTTPCachePolicy)
// CanUpdatePolicy 检查策略是否可以更新
CanUpdatePolicy(newPolicy *serverconfigs.HTTPCachePolicy) bool
// AddToList 将缓存添加到列表
AddToList(item *Item)
// IgnoreKey 忽略某个Key即不缓存某个Key
IgnoreKey(key string)
// CanSendfile 是否支持Sendfile
CanSendfile() bool
}

View File

@@ -7,10 +7,13 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
setutils "github.com/TeaOSLab/EdgeNode/internal/utils/sets"
"github.com/TeaOSLab/EdgeNode/internal/utils/sizes"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/cespare/xxhash"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
"github.com/shirou/gopsutil/v3/load"
"math"
"runtime"
"strconv"
@@ -20,7 +23,7 @@ import (
)
type MemoryItem struct {
ExpiredAt int64
ExpiresAt int64
HeaderValue []byte
BodyValue []byte
Status int
@@ -29,7 +32,7 @@ type MemoryItem struct {
}
func (this *MemoryItem) IsExpired() bool {
return this.ExpiredAt < utils.UnixTime()
return this.ExpiresAt < utils.UnixTime()
}
type MemoryStorage struct {
@@ -40,31 +43,39 @@ type MemoryStorage struct {
locker *sync.RWMutex
valuesMap map[uint64]*MemoryItem // hash => item
dirtyChan chan string // hash chan
dirtyChan chan string // hash chan
dirtyQueueSize int
purgeTicker *utils.Ticker
totalSize int64
writingKeyMap map[string]zero.Zero // key => bool
ignoreKeys *setutils.FixedSet
}
func NewMemoryStorage(policy *serverconfigs.HTTPCachePolicy, parentStorage StorageInterface) *MemoryStorage {
var dirtyChan chan string
var queueSize = policy.MemoryAutoFlushQueueSize
if parentStorage != nil {
var queueSize = policy.MemoryAutoFlushQueueSize
if queueSize <= 0 {
queueSize = 2048
queueSize = 2048 + int(policy.CapacityBytes()/sizes.G)*2048
}
dirtyChan = make(chan string, queueSize)
}
return &MemoryStorage{
parentStorage: parentStorage,
policy: policy,
list: NewMemoryList(),
locker: &sync.RWMutex{},
valuesMap: map[uint64]*MemoryItem{},
dirtyChan: dirtyChan,
writingKeyMap: map[string]zero.Zero{},
parentStorage: parentStorage,
policy: policy,
list: NewMemoryList(),
locker: &sync.RWMutex{},
valuesMap: map[uint64]*MemoryItem{},
dirtyChan: dirtyChan,
dirtyQueueSize: queueSize,
writingKeyMap: map[string]zero.Zero{},
ignoreKeys: setutils.NewFixedSet(32768),
}
}
@@ -79,33 +90,25 @@ func (this *MemoryStorage) Init() error {
atomic.AddInt64(&this.totalSize, -item.TotalSize())
})
var autoPurgeInterval = this.policy.MemoryAutoPurgeInterval
if autoPurgeInterval <= 0 {
autoPurgeInterval = 5
}
// 启动定时清理任务
this.purgeTicker = utils.NewTicker(time.Duration(autoPurgeInterval) * time.Second)
goman.New(func() {
for this.purgeTicker.Next() {
var tr = trackers.Begin("MEMORY_CACHE_STORAGE_PURGE_LOOP")
this.purgeLoop()
tr.End()
}
})
this.initPurgeTicker()
// 启动定时Flush memory to disk任务
goman.New(func() {
for hash := range this.dirtyChan {
this.flushItem(hash)
if this.parentStorage != nil {
// TODO 应该根据磁盘性能决定线程数
var threads = 1
for i := 0; i < threads; i++ {
goman.New(func() {
this.startFlush()
})
}
})
}
return nil
}
// OpenReader 读取缓存
func (this *MemoryStorage) OpenReader(key string, useStale bool) (Reader, error) {
func (this *MemoryStorage) OpenReader(key string, useStale bool, isPartial bool) (Reader, error) {
hash := this.hash(key)
this.locker.RLock()
@@ -115,7 +118,7 @@ func (this *MemoryStorage) OpenReader(key string, useStale bool) (Reader, error)
return nil, ErrNotFound
}
if useStale || (item.ExpiredAt > utils.UnixTime()) {
if useStale || (item.ExpiresAt > utils.UnixTime()) {
reader := NewMemoryReader(item)
err := reader.Init()
if err != nil {
@@ -145,11 +148,32 @@ func (this *MemoryStorage) OpenReader(key string, useStale bool) (Reader, error)
}
// OpenWriter 打开缓存写入器等待写入
func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int, size int64) (Writer, error) {
return this.openWriter(key, expiredAt, status, size, true)
func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error) {
if this.ignoreKeys.Has(key) {
return nil, ErrEntityTooLarge
}
// TODO 内存缓存暂时不支持分块内容存储
if isPartial {
return nil, ErrFileIsWriting
}
return this.openWriter(key, expiredAt, status, size, maxSize, true)
}
func (this *MemoryStorage) openWriter(key string, expiredAt int64, status int, size int64, isDirty bool) (Writer, error) {
// OpenFlushWriter 打开从其他媒介直接刷入的写入器
func (this *MemoryStorage) OpenFlushWriter(key string, expiresAt int64, status int) (Writer, error) {
return this.openWriter(key, expiresAt, status, -1, -1, true)
}
func (this *MemoryStorage) openWriter(key string, expiresAt int64, status int, size int64, maxSize int64, isDirty bool) (Writer, error) {
// 待写入队列是否已满
if isDirty &&
this.parentStorage != nil &&
this.dirtyQueueSize > 0 &&
len(this.dirtyChan) == this.dirtyQueueSize { // 缓存时间过长
return nil, ErrWritingQueueFull
}
this.locker.Lock()
defer this.locker.Unlock()
@@ -196,7 +220,7 @@ func (this *MemoryStorage) openWriter(key string, expiredAt int64, status int, s
}
isWriting = true
return NewMemoryWriter(this, key, expiredAt, status, isDirty, func() {
return NewMemoryWriter(this, key, expiresAt, status, isDirty, maxSize, func() {
this.locker.Lock()
delete(this.writingKeyMap, key)
this.locker.Unlock()
@@ -243,8 +267,10 @@ func (this *MemoryStorage) Purge(keys []string, urlType string) error {
return err
}
}
return nil
}
// URL
for _, key := range keys {
err := this.Delete(key)
if err != nil {
@@ -265,7 +291,7 @@ func (this *MemoryStorage) Stop() {
this.purgeTicker.Stop()
}
if this.parentStorage != nil && this.dirtyChan != nil {
if this.dirtyChan != nil {
close(this.dirtyChan)
}
@@ -273,6 +299,8 @@ func (this *MemoryStorage) Stop() {
this.locker.Unlock()
this.ignoreKeys.Reset()
// 回收内存
runtime.GC()
@@ -284,6 +312,26 @@ func (this *MemoryStorage) Policy() *serverconfigs.HTTPCachePolicy {
return this.policy
}
// UpdatePolicy 修改策略
func (this *MemoryStorage) UpdatePolicy(newPolicy *serverconfigs.HTTPCachePolicy) {
var oldPolicy = this.policy
this.policy = newPolicy
if oldPolicy.MemoryAutoPurgeInterval != newPolicy.MemoryAutoPurgeInterval {
this.initPurgeTicker()
}
// 如果是空的,则清空
if newPolicy.CapacityBytes() == 0 {
_ = this.CleanAll()
}
}
// CanUpdatePolicy 检查策略是否可以更新
func (this *MemoryStorage) CanUpdatePolicy(newPolicy *serverconfigs.HTTPCachePolicy) bool {
return true
}
// AddToList 将缓存添加到列表
func (this *MemoryStorage) AddToList(item *Item) {
item.MetaSize = int64(len(item.Key)) + 128 /** 128是我们评估的数据结构的长度 **/
@@ -301,6 +349,16 @@ func (this *MemoryStorage) TotalMemorySize() int64 {
return atomic.LoadInt64(&this.totalSize)
}
// IgnoreKey 忽略某个Key即不缓存某个Key
func (this *MemoryStorage) IgnoreKey(key string) {
this.ignoreKeys.Push(key)
}
// CanSendfile 是否支持Sendfile
func (this *MemoryStorage) CanSendfile() bool {
return false
}
// 计算Key Hash
func (this *MemoryStorage) hash(key string) uint64 {
return xxhash.Sum64String(key)
@@ -369,7 +427,40 @@ func (this *MemoryStorage) purgeLoop() {
}
}
// Flush任务
// 开始Flush任务
func (this *MemoryStorage) startFlush() {
var statCount = 0
var writeDelayMS float64 = 0
for hash := range this.dirtyChan {
statCount++
if statCount == 100 {
statCount = 0
loadStat, err := load.Avg()
if err == nil && loadStat != nil {
if loadStat.Load1 > 10 {
writeDelayMS = 100
} else if loadStat.Load1 > 3 {
writeDelayMS = 50
} else if loadStat.Load1 > 2 {
writeDelayMS = 10
} else {
writeDelayMS = 0
}
}
}
this.flushItem(hash)
if writeDelayMS > 0 {
time.Sleep(time.Duration(writeDelayMS) * time.Millisecond)
}
}
}
// 单次Flush任务
func (this *MemoryStorage) flushItem(key string) {
if this.parentStorage == nil {
return
@@ -387,7 +478,7 @@ func (this *MemoryStorage) flushItem(key string) {
return
}
writer, err := this.parentStorage.OpenWriter(key, item.ExpiredAt, item.Status, -1)
writer, err := this.parentStorage.OpenFlushWriter(key, item.ExpiresAt, item.Status)
if err != nil {
if !CanIgnoreErr(err) {
remotelogs.Error("CACHE", "flush items failed: open writer failed: "+err.Error())
@@ -413,16 +504,20 @@ func (this *MemoryStorage) flushItem(key string) {
if err != nil {
_ = writer.Discard()
remotelogs.Error("CACHE", "flush items failed: close writer failed: "+err.Error())
return
}
this.parentStorage.AddToList(&Item{
Type: writer.ItemType(),
Key: key,
ExpiredAt: item.ExpiredAt,
ExpiredAt: item.ExpiresAt,
HeaderSize: writer.HeaderSize(),
BodySize: writer.BodySize(),
})
// 从内存中移除
_ = this.Delete(key)
return
}
@@ -449,3 +544,25 @@ func (this *MemoryStorage) deleteWithoutLocker(key string) error {
_ = this.list.Remove(fmt.Sprintf("%d", hash))
return nil
}
func (this *MemoryStorage) initPurgeTicker() {
var autoPurgeInterval = this.policy.MemoryAutoPurgeInterval
if autoPurgeInterval <= 0 {
autoPurgeInterval = 5
}
// 启动定时清理任务
if this.purgeTicker != nil {
this.purgeTicker.Stop()
}
this.purgeTicker = utils.NewTicker(time.Duration(autoPurgeInterval) * time.Second)
goman.New(func() {
for this.purgeTicker.Next() {
var tr = trackers.Begin("MEMORY_CACHE_STORAGE_PURGE_LOOP")
this.purgeLoop()
tr.End()
}
})
}

View File

@@ -15,7 +15,7 @@ import (
func TestMemoryStorage_OpenWriter(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1)
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -25,7 +25,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
t.Log(storage.valuesMap)
{
reader, err := storage.OpenReader("abc", false)
reader, err := storage.OpenReader("abc", false, false)
if err != nil {
if err == ErrNotFound {
t.Log("not found: abc")
@@ -52,7 +52,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
}
{
_, err := storage.OpenReader("abc 2", false)
_, err := storage.OpenReader("abc 2", false, false)
if err != nil {
if err == ErrNotFound {
t.Log("not found: abc2")
@@ -62,13 +62,13 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
}
}
writer, err = storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1)
writer, err = storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello123"))
{
reader, err := storage.OpenReader("abc", false)
reader, err := storage.OpenReader("abc", false, false)
if err != nil {
if err == ErrNotFound {
t.Log("not found: abc")
@@ -97,13 +97,13 @@ func TestMemoryStorage_OpenReaderLock(t *testing.T) {
IsDone: true,
},
}
_, _ = storage.OpenReader("test", false)
_, _ = storage.OpenReader("test", false, false)
}
func TestMemoryStorage_Delete(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
{
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1)
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -111,7 +111,7 @@ func TestMemoryStorage_Delete(t *testing.T) {
t.Log(len(storage.valuesMap))
}
{
writer, err := storage.OpenWriter("abc1", time.Now().Unix()+60, 200, -1)
writer, err := storage.OpenWriter("abc1", time.Now().Unix()+60, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -126,7 +126,7 @@ func TestMemoryStorage_Stat(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1)
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -139,7 +139,7 @@ func TestMemoryStorage_Stat(t *testing.T) {
})
}
{
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1)
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -163,7 +163,7 @@ func TestMemoryStorage_CleanAll(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1)
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -175,7 +175,7 @@ func TestMemoryStorage_CleanAll(t *testing.T) {
})
}
{
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1)
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -198,7 +198,7 @@ func TestMemoryStorage_Purge(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1)
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -210,7 +210,7 @@ func TestMemoryStorage_Purge(t *testing.T) {
})
}
{
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1)
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -241,7 +241,7 @@ func TestMemoryStorage_Expire(t *testing.T) {
for i := 0; i < 1000; i++ {
expiredAt := time.Now().Unix() + int64(rands.Int(0, 60))
key := "abc" + strconv.Itoa(i)
writer, err := storage.OpenWriter(key, expiredAt, 200, -1)
writer, err := storage.OpenWriter(key, expiredAt, 200, -1, -1, false)
if err != nil {
t.Fatal(err)
}

View File

@@ -0,0 +1,18 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
import "strings"
// 获取 ranges 文件路径
func partialRangesFilePath(path string) string {
// ranges路径
var dotIndex = strings.LastIndex(path, ".")
var rangePath = ""
if dotIndex < 0 {
rangePath = path + "@ranges.cache"
} else {
rangePath = path[:dotIndex] + "@ranges" + path[dotIndex:]
}
return rangePath
}

View File

@@ -8,6 +8,9 @@ type Writer interface {
// Write 写入Body数据
Write(data []byte) (n int, err error)
// WriteAt 在指定位置写入数据
WriteAt(offset int64, data []byte) error
// HeaderSize 写入的Header数据大小
HeaderSize() int64

View File

@@ -2,6 +2,7 @@ package caches
import (
"encoding/binary"
"errors"
"github.com/iwind/TeaGo/types"
"io"
"os"
@@ -10,20 +11,24 @@ import (
)
type FileWriter struct {
storage StorageInterface
rawWriter *os.File
key string
headerSize int64
bodySize int64
expiredAt int64
maxSize int64
endFunc func()
once sync.Once
}
func NewFileWriter(rawWriter *os.File, key string, expiredAt int64, endFunc func()) *FileWriter {
func NewFileWriter(storage StorageInterface, rawWriter *os.File, key string, expiredAt int64, maxSize int64, endFunc func()) *FileWriter {
return &FileWriter{
storage: storage,
key: key,
rawWriter: rawWriter,
expiredAt: expiredAt,
maxSize: maxSize,
endFunc: endFunc,
}
}
@@ -59,12 +64,28 @@ func (this *FileWriter) WriteHeaderLength(headerLength int) error {
func (this *FileWriter) Write(data []byte) (n int, err error) {
n, err = this.rawWriter.Write(data)
this.bodySize += int64(n)
if this.maxSize > 0 && this.bodySize > this.maxSize {
err = ErrEntityTooLarge
if this.storage != nil {
this.storage.IgnoreKey(this.key)
}
}
if err != nil {
_ = this.Discard()
}
return
}
// WriteAt 在指定位置写入数据
func (this *FileWriter) WriteAt(offset int64, data []byte) error {
_ = data
_ = offset
return errors.New("not supported")
}
// WriteBodyLength 写入Body长度数据
func (this *FileWriter) WriteBodyLength(bodyLength int64) error {
bytes8 := make([]byte, 8)
@@ -106,8 +127,8 @@ func (this *FileWriter) Close() error {
err = this.rawWriter.Close()
if err != nil {
_ = os.Remove(path)
} else {
err = os.Rename(path, strings.Replace(path, ".tmp", "", 1))
} else if strings.HasSuffix(path, FileTmpSuffix) {
err = os.Rename(path, strings.Replace(path, FileTmpSuffix, "", 1))
if err != nil {
_ = os.Remove(path)
}

View File

@@ -1,6 +1,7 @@
package caches
import (
"errors"
"github.com/cespare/xxhash"
"sync"
"time"
@@ -15,6 +16,7 @@ type MemoryWriter struct {
bodySize int64
status int
isDirty bool
maxSize int64
hash uint64
item *MemoryItem
@@ -22,18 +24,19 @@ type MemoryWriter struct {
once sync.Once
}
func NewMemoryWriter(memoryStorage *MemoryStorage, key string, expiredAt int64, status int, isDirty bool, endFunc func()) *MemoryWriter {
func NewMemoryWriter(memoryStorage *MemoryStorage, key string, expiredAt int64, status int, isDirty bool, maxSize int64, endFunc func()) *MemoryWriter {
w := &MemoryWriter{
storage: memoryStorage,
key: key,
expiredAt: expiredAt,
item: &MemoryItem{
ExpiredAt: expiredAt,
ExpiresAt: expiredAt,
ModifiedAt: time.Now().Unix(),
Status: status,
},
status: status,
isDirty: isDirty,
maxSize: maxSize,
endFunc: endFunc,
}
w.hash = w.calculateHash(key)
@@ -52,9 +55,24 @@ func (this *MemoryWriter) WriteHeader(data []byte) (n int, err error) {
func (this *MemoryWriter) Write(data []byte) (n int, err error) {
this.bodySize += int64(len(data))
this.item.BodyValue = append(this.item.BodyValue, data...)
// 检查尺寸
if this.maxSize > 0 && this.bodySize > this.maxSize {
err = ErrEntityTooLarge
this.storage.IgnoreKey(this.key)
return len(data), err
}
return len(data), nil
}
// WriteAt 在指定位置写入数据
func (this *MemoryWriter) WriteAt(offset int64, b []byte) error {
_ = b
_ = offset
return errors.New("not supported")
}
// HeaderSize 数据尺寸
func (this *MemoryWriter) HeaderSize() int64 {
return this.headerSize

View File

@@ -0,0 +1,224 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
import (
"encoding/binary"
"github.com/iwind/TeaGo/types"
"io"
"os"
"sync"
)
type PartialFileWriter struct {
rawWriter *os.File
key string
headerSize int64
bodySize int64
expiredAt int64
endFunc func()
once sync.Once
isNew bool
isPartial bool
bodyOffset int64
ranges *PartialRanges
rangePath string
}
func NewPartialFileWriter(rawWriter *os.File, key string, expiredAt int64, isNew bool, isPartial bool, bodyOffset int64, ranges *PartialRanges, endFunc func()) *PartialFileWriter {
return &PartialFileWriter{
key: key,
rawWriter: rawWriter,
expiredAt: expiredAt,
endFunc: endFunc,
isNew: isNew,
isPartial: isPartial,
bodyOffset: bodyOffset,
ranges: ranges,
rangePath: partialRangesFilePath(rawWriter.Name()),
}
}
// WriteHeader 写入数据
func (this *PartialFileWriter) WriteHeader(data []byte) (n int, err error) {
if !this.isNew {
return
}
n, err = this.rawWriter.Write(data)
this.headerSize += int64(n)
if err != nil {
_ = this.Discard()
}
return
}
func (this *PartialFileWriter) AppendHeader(data []byte) error {
_, err := this.rawWriter.Write(data)
if err != nil {
_ = this.Discard()
} else {
var c = len(data)
this.headerSize += int64(c)
err = this.WriteHeaderLength(int(this.headerSize))
if err != nil {
_ = this.Discard()
}
}
return err
}
// WriteHeaderLength 写入Header长度数据
func (this *PartialFileWriter) WriteHeaderLength(headerLength int) error {
bytes4 := make([]byte, 4)
binary.BigEndian.PutUint32(bytes4, uint32(headerLength))
_, err := this.rawWriter.Seek(SizeExpiresAt+SizeStatus+SizeURLLength, io.SeekStart)
if err != nil {
_ = this.Discard()
return err
}
_, err = this.rawWriter.Write(bytes4)
if err != nil {
_ = this.Discard()
return err
}
return nil
}
// Write 写入数据
func (this *PartialFileWriter) Write(data []byte) (n int, err error) {
n, err = this.rawWriter.Write(data)
this.bodySize += int64(n)
if err != nil {
_ = this.Discard()
}
return
}
// WriteAt 在指定位置写入数据
func (this *PartialFileWriter) WriteAt(offset int64, data []byte) error {
var c = int64(len(data))
if c == 0 {
return nil
}
var end = offset + c - 1
// 是否已包含在内
if this.ranges.Contains(offset, end) {
return nil
}
if this.bodyOffset == 0 {
this.bodyOffset = SizeMeta + int64(len(this.key)) + this.headerSize
}
_, err := this.rawWriter.WriteAt(data, this.bodyOffset+offset)
if err != nil {
return err
}
this.ranges.Add(offset, end)
return nil
}
// SetBodyLength 设置内容总长度
func (this *PartialFileWriter) SetBodyLength(bodyLength int64) {
this.bodySize = bodyLength
}
// WriteBodyLength 写入Body长度数据
func (this *PartialFileWriter) WriteBodyLength(bodyLength int64) error {
bytes8 := make([]byte, 8)
binary.BigEndian.PutUint64(bytes8, uint64(bodyLength))
_, err := this.rawWriter.Seek(SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength, io.SeekStart)
if err != nil {
_ = this.Discard()
return err
}
_, err = this.rawWriter.Write(bytes8)
if err != nil {
_ = this.Discard()
return err
}
return nil
}
// Close 关闭
func (this *PartialFileWriter) Close() error {
defer this.once.Do(func() {
this.endFunc()
})
err := this.ranges.WriteToFile(this.rangePath)
if err != nil {
return err
}
// 关闭当前writer
if this.isNew {
err = this.WriteHeaderLength(types.Int(this.headerSize))
if err != nil {
_ = this.rawWriter.Close()
this.remove()
return err
}
err = this.WriteBodyLength(this.bodySize)
if err != nil {
_ = this.rawWriter.Close()
this.remove()
return err
}
}
err = this.rawWriter.Close()
if err != nil {
this.remove()
}
return err
}
// Discard 丢弃
func (this *PartialFileWriter) Discard() error {
defer this.once.Do(func() {
this.endFunc()
})
_ = this.rawWriter.Close()
_ = os.Remove(this.rangePath)
err := os.Remove(this.rawWriter.Name())
return err
}
func (this *PartialFileWriter) HeaderSize() int64 {
return this.headerSize
}
func (this *PartialFileWriter) BodySize() int64 {
return this.bodySize
}
func (this *PartialFileWriter) ExpiredAt() int64 {
return this.expiredAt
}
func (this *PartialFileWriter) Key() string {
return this.key
}
// ItemType 获取内容类型
func (this *PartialFileWriter) ItemType() ItemType {
return ItemTypeFile
}
func (this *PartialFileWriter) IsNew() bool {
return this.isNew && len(this.ranges.Ranges) == 0
}
func (this *PartialFileWriter) remove() {
_ = os.Remove(this.rawWriter.Name())
_ = os.Remove(this.rangePath)
}

View File

@@ -0,0 +1,51 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches_test
import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/iwind/TeaGo/types"
"io/ioutil"
"os"
"testing"
"time"
)
func TestPartialFileWriter_Write(t *testing.T) {
var path = "/tmp/test_partial.cache"
_ = os.Remove(path)
var reader = func() {
data, err := ioutil.ReadFile(path)
if err != nil {
t.Fatal(err)
}
t.Log("["+types.String(len(data))+"]", string(data))
}
fp, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, 0666)
if err != nil {
t.Fatal(err)
}
var ranges = caches.NewPartialRanges()
var writer = caches.NewPartialFileWriter(fp, "test", time.Now().Unix()+86500, true, true, 0, ranges, func() {
t.Log("end")
})
_, err = writer.WriteHeader([]byte("header"))
if err != nil {
t.Fatal(err)
}
// 移动位置
err = writer.WriteAt(100, []byte("HELLO"))
if err != nil {
t.Fatal(err)
}
err = writer.Close()
if err != nil {
t.Fatal(err)
}
reader()
}

View File

@@ -2,7 +2,14 @@
package compressions
import "io"
type Reader interface {
Read(p []byte) (n int, err error)
Reset(reader io.Reader) error
RawClose() error
Close() error
SetPool(pool *ReaderPool)
ResetFinish()
}

View File

@@ -0,0 +1,26 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
type BaseReader struct {
pool *ReaderPool
isFinished bool
}
func (this *BaseReader) SetPool(pool *ReaderPool) {
this.pool = pool
}
func (this *BaseReader) Finish(obj Reader) error {
err := obj.RawClose()
if err == nil && this.pool != nil && !this.isFinished {
this.pool.Put(obj)
}
this.isFinished = true
return err
}
func (this *BaseReader) ResetFinish() {
this.isFinished = false
}

View File

@@ -9,10 +9,16 @@ import (
)
type BrotliReader struct {
BaseReader
reader *brotli.Reader
}
func NewBrotliReader(reader io.Reader) (Reader, error) {
return sharedBrotliReaderPool.Get(reader)
}
func newBrotliReader(reader io.Reader) (Reader, error) {
return &BrotliReader{reader: brotli.NewReader(reader)}, nil
}
@@ -24,6 +30,14 @@ func (this *BrotliReader) Read(p []byte) (n int, err error) {
return
}
func (this *BrotliReader) Close() error {
func (this *BrotliReader) Reset(reader io.Reader) error {
return this.reader.Reset(reader)
}
func (this *BrotliReader) RawClose() error {
return nil
}
func (this *BrotliReader) Close() error {
return this.Finish(this)
}

View File

@@ -0,0 +1,51 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions_test
import (
"bytes"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"io"
"testing"
)
func TestBrotliReader(t *testing.T) {
for _, testString := range []string{"Hello", "World", "Ni", "Hao"} {
t.Log("===", testString, "===")
var buf = &bytes.Buffer{}
writer, err := compressions.NewBrotliWriter(buf, 5)
if err != nil {
t.Fatal(err)
}
_, err = writer.Write([]byte(testString))
if err != nil {
t.Fatal(err)
}
err = writer.Close()
if err != nil {
t.Fatal(err)
}
reader, err := compressions.NewBrotliReader(buf)
if err != nil {
t.Fatal(err)
}
var data = make([]byte, 4096)
for {
n, err := reader.Read(data)
if n > 0 {
t.Log(string(data[:n]))
}
if err != nil {
if err == io.EOF {
break
}
t.Fatal(err)
}
}
err = reader.Close()
if err != nil {
t.Fatal(err)
}
}
}

View File

@@ -8,10 +8,16 @@ import (
)
type DeflateReader struct {
BaseReader
reader io.ReadCloser
}
func NewDeflateReader(reader io.Reader) (Reader, error) {
return sharedDeflateReaderPool.Get(reader)
}
func newDeflateReader(reader io.Reader) (Reader, error) {
return &DeflateReader{reader: flate.NewReader(reader)}, nil
}
@@ -19,6 +25,15 @@ func (this *DeflateReader) Read(p []byte) (n int, err error) {
return this.reader.Read(p)
}
func (this *DeflateReader) Close() error {
func (this *DeflateReader) Reset(reader io.Reader) error {
this.reader = flate.NewReader(reader)
return nil
}
func (this *DeflateReader) RawClose() error {
return this.reader.Close()
}
func (this *DeflateReader) Close() error {
return this.Finish(this)
}

View File

@@ -0,0 +1,51 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions_test
import (
"bytes"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"io"
"testing"
)
func TestDeflateReader(t *testing.T) {
for _, testString := range []string{"Hello", "World", "Ni", "Hao"} {
t.Log("===", testString, "===")
var buf = &bytes.Buffer{}
writer, err := compressions.NewDeflateWriter(buf, 5)
if err != nil {
t.Fatal(err)
}
_, err = writer.Write([]byte(testString))
if err != nil {
t.Fatal(err)
}
err = writer.Close()
if err != nil {
t.Fatal(err)
}
reader, err := compressions.NewDeflateReader(buf)
if err != nil {
t.Fatal(err)
}
var data = make([]byte, 4096)
for {
n, err := reader.Read(data)
if n > 0 {
t.Log(string(data[:n]))
}
if err != nil {
if err == io.EOF {
break
}
t.Fatal(err)
}
}
err = reader.Close()
if err != nil {
t.Fatal(err)
}
}
}

View File

@@ -3,15 +3,21 @@
package compressions
import (
"compress/gzip"
"github.com/klauspost/compress/gzip"
"io"
)
type GzipReader struct {
BaseReader
reader *gzip.Reader
}
func NewGzipReader(reader io.Reader) (Reader, error) {
return sharedGzipReaderPool.Get(reader)
}
func newGzipReader(reader io.Reader) (Reader, error) {
r, err := gzip.NewReader(reader)
if err != nil {
return nil, err
@@ -25,6 +31,14 @@ func (this *GzipReader) Read(p []byte) (n int, err error) {
return this.reader.Read(p)
}
func (this *GzipReader) Close() error {
func (this *GzipReader) Reset(reader io.Reader) error {
return this.reader.Reset(reader)
}
func (this *GzipReader) RawClose() error {
return this.reader.Close()
}
func (this *GzipReader) Close() error {
return this.Finish(this)
}

View File

@@ -1,68 +1,106 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
package compressions_test
import (
"bytes"
"errors"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
"io"
"os"
"strings"
"testing"
)
func TestGzipReader(t *testing.T) {
fp, err := os.Open("/Users/WorkSpace/EdgeProject/EdgeCache/p43/36/7e/367e02720713fe05b66573a1d69b4f0a.cache")
if err != nil {
// not fatal
t.Log(err)
return
}
defer func() {
_ = fp.Close()
}()
var buf = make([]byte, 32*1024)
cacheReader := caches.NewFileReader(fp)
err = cacheReader.Init()
if err != nil {
t.Fatal(err)
}
var headerBuf = []byte{}
err = cacheReader.ReadHeader(buf, func(n int) (goNext bool, err error) {
headerBuf = append(headerBuf, buf[:n]...)
for {
nIndex := bytes.Index(headerBuf, []byte{'\n'})
if nIndex >= 0 {
row := headerBuf[:nIndex]
spaceIndex := bytes.Index(row, []byte{':'})
if spaceIndex <= 0 {
return false, errors.New("invalid header '" + string(row) + "'")
}
headerBuf = headerBuf[nIndex+1:]
} else {
break
}
}
return true, nil
})
reader, err := NewGzipReader(cacheReader)
if err != nil {
t.Fatal(err)
}
for {
n, err := reader.Read(buf)
for _, testString := range []string{"Hello", "World", "Ni", "Hao"} {
t.Log("===", testString, "===")
var buf = &bytes.Buffer{}
writer, err := compressions.NewGzipWriter(buf, 5)
if err != nil {
if err != io.EOF {
t.Fatal(err)
}
_, err = writer.Write([]byte(testString))
if err != nil {
t.Fatal(err)
}
err = writer.Close()
if err != nil {
t.Fatal(err)
}
reader, err := compressions.NewGzipReader(buf)
if err != nil {
t.Fatal(err)
}
var data = make([]byte, 4096)
for {
n, err := reader.Read(data)
if n > 0 {
t.Log(string(data[:n]))
}
if err != nil {
if err == io.EOF {
break
}
t.Fatal(err)
} else {
break
}
}
t.Log(string(buf[:n]))
_ = n
err = reader.Close()
if err != nil {
t.Fatal(err)
}
}
}
func BenchmarkGzipReader(b *testing.B) {
var randomData = func() []byte {
var b = strings.Builder{}
for i := 0; i < 1024; i++ {
b.WriteString(types.String(rands.Int64() % 10))
}
return []byte(b.String())
}
var buf = &bytes.Buffer{}
writer, err := compressions.NewGzipWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
_, err = writer.Write(randomData())
if err != nil {
b.Fatal(err)
}
err = writer.Close()
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
var newBytes = make([]byte, buf.Len())
copy(newBytes, buf.Bytes())
reader, err := compressions.NewGzipReader(bytes.NewReader(newBytes))
if err != nil {
b.Fatal(err)
}
var data = make([]byte, 4096)
for {
n, err := reader.Read(data)
if n > 0 {
_ = data[:n]
}
if err != nil {
if err == io.EOF {
break
}
b.Fatal(err)
}
}
err = reader.Close()
if err != nil {
b.Fatal(err)
}
}
}

View File

@@ -0,0 +1,56 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"io"
)
type ReaderPool struct {
c chan Reader
newFunc func(reader io.Reader) (Reader, error)
}
func NewReaderPool(maxSize int, newFunc func(reader io.Reader) (Reader, error)) *ReaderPool {
if maxSize <= 0 {
maxSize = 1024
}
return &ReaderPool{
c: make(chan Reader, maxSize),
newFunc: newFunc,
}
}
func (this *ReaderPool) Get(parentReader io.Reader) (Reader, error) {
select {
case reader := <-this.c:
err := reader.Reset(parentReader)
if err != nil {
// create new
reader, err = this.newFunc(parentReader)
if err != nil {
return nil, err
}
reader.SetPool(this)
return reader, nil
}
reader.ResetFinish()
return reader, nil
default:
// create new
reader, err := this.newFunc(parentReader)
if err != nil {
return nil, err
}
reader.SetPool(this)
return reader, nil
}
}
func (this *ReaderPool) Put(reader Reader) {
select {
case this.c <- reader:
default:
}
}

View File

@@ -0,0 +1,20 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
)
var sharedBrotliReaderPool *ReaderPool
func init() {
var maxSize = utils.SystemMemoryGB() * 256
if maxSize == 0 {
maxSize = 256
}
sharedBrotliReaderPool = NewReaderPool(maxSize, func(reader io.Reader) (Reader, error) {
return newBrotliReader(reader)
})
}

View File

@@ -0,0 +1,20 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
)
var sharedDeflateReaderPool *ReaderPool
func init() {
var maxSize = utils.SystemMemoryGB() * 256
if maxSize == 0 {
maxSize = 256
}
sharedDeflateReaderPool = NewReaderPool(maxSize, func(reader io.Reader) (Reader, error) {
return newDeflateReader(reader)
})
}

View File

@@ -0,0 +1,20 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
)
var sharedGzipReaderPool *ReaderPool
func init() {
var maxSize = utils.SystemMemoryGB() * 256
if maxSize == 0 {
maxSize = 256
}
sharedGzipReaderPool = NewReaderPool(maxSize, func(reader io.Reader) (Reader, error) {
return newGzipReader(reader)
})
}

View File

@@ -18,6 +18,11 @@ const (
var ErrNotSupportedContentEncoding = errors.New("not supported content encoding")
// AllEncodings 当前支持的所有编码
func AllEncodings() []ContentEncoding {
return []ContentEncoding{ContentEncodingBr, ContentEncodingGzip, ContentEncodingDeflate}
}
// NewReader 获取Reader
func NewReader(reader io.Reader, contentEncoding ContentEncoding) (Reader, error) {
switch contentEncoding {
@@ -32,7 +37,6 @@ func NewReader(reader io.Reader, contentEncoding ContentEncoding) (Reader, error
}
// NewWriter 获取Writer
// TODO 考虑重用Writer
func NewWriter(writer io.Writer, compressType serverconfigs.HTTPCompressionType, level int) (Writer, error) {
switch compressType {
case serverconfigs.HTTPCompressionTypeGzip:

View File

@@ -2,9 +2,16 @@
package compressions
import "io"
type Writer interface {
Write(p []byte) (int, error)
Flush() error
Reset(writer io.Writer)
RawClose() error
Close() error
Level() int
SetPool(pool *WriterPool)
ResetFinish()
}

View File

@@ -0,0 +1,26 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
type BaseWriter struct {
pool *WriterPool
isFinished bool
}
func (this *BaseWriter) SetPool(pool *WriterPool) {
this.pool = pool
}
func (this *BaseWriter) Finish(obj Writer) error {
err := obj.RawClose()
if err == nil && this.pool != nil && !this.isFinished {
this.pool.Put(obj)
}
this.isFinished = true
return err
}
func (this *BaseWriter) ResetFinish() {
this.isFinished = false
}

View File

@@ -8,19 +8,28 @@ import (
)
type BrotliWriter struct {
BaseWriter
writer *brotli.Writer
level int
}
func NewBrotliWriter(writer io.Writer, level int) (Writer, error) {
return sharedBrotliWriterPool.Get(writer, level)
}
func newBrotliWriter(writer io.Writer, level int) (*BrotliWriter, error) {
if level <= 0 {
level = brotli.BestSpeed
} else if level > brotli.BestCompression {
level = brotli.BestCompression
}
return &BrotliWriter{
writer: brotli.NewWriterLevel(writer, level),
level: level,
writer: brotli.NewWriterOptions(writer, brotli.WriterOptions{
Quality: level,
LGWin: 13, // TODO 在全局设置里可以设置此值
}),
level: level,
}, nil
}
@@ -32,10 +41,18 @@ func (this *BrotliWriter) Flush() error {
return this.writer.Flush()
}
func (this *BrotliWriter) Close() error {
func (this *BrotliWriter) Reset(newWriter io.Writer) {
this.writer.Reset(newWriter)
}
func (this *BrotliWriter) RawClose() error {
return this.writer.Close()
}
func (this *BrotliWriter) Close() error {
return this.Finish(this)
}
func (this *BrotliWriter) Level() int {
return this.level
}

View File

@@ -0,0 +1,116 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions_test
import (
"bytes"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"strings"
"testing"
)
func BenchmarkBrotliWriter_Write(b *testing.B) {
var data = []byte(strings.Repeat("A", 1024))
for i := 0; i < b.N; i++ {
var buf = &bytes.Buffer{}
writer, err := compressions.NewBrotliWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
for j := 0; j < 100; j++ {
_, err = writer.Write(data)
if err != nil {
b.Fatal(err)
}
/**err = writer.Flush()
if err != nil {
b.Fatal(err)
}**/
}
_ = writer.Close()
}
}
func BenchmarkBrotliWriter_Write_Parallel(b *testing.B) {
var data = []byte(strings.Repeat("A", 1024))
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
var buf = &bytes.Buffer{}
writer, err := compressions.NewBrotliWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
for j := 0; j < 100; j++ {
_, err = writer.Write(data)
if err != nil {
b.Fatal(err)
}
/**err = writer.Flush()
if err != nil {
b.Fatal(err)
}**/
}
_ = writer.Close()
}
})
}
func BenchmarkBrotliWriter_Write_Small(b *testing.B) {
var data = []byte(strings.Repeat("A", 16))
for i := 0; i < b.N; i++ {
var buf = &bytes.Buffer{}
writer, err := compressions.NewBrotliWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
for j := 0; j < 100; j++ {
_, err = writer.Write(data)
if err != nil {
b.Fatal(err)
}
/**err = writer.Flush()
if err != nil {
b.Fatal(err)
}**/
}
_ = writer.Close()
}
}
func BenchmarkBrotliWriter_Write_Large(b *testing.B) {
var data = []byte(strings.Repeat("A", 4096))
for i := 0; i < b.N; i++ {
var buf = &bytes.Buffer{}
writer, err := compressions.NewBrotliWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
for j := 0; j < 100; j++ {
_, err = writer.Write(data)
if err != nil {
b.Fatal(err)
}
/**err = writer.Flush()
if err != nil {
b.Fatal(err)
}**/
}
_ = writer.Close()
}
}

View File

@@ -8,11 +8,17 @@ import (
)
type DeflateWriter struct {
BaseWriter
writer *flate.Writer
level int
}
func NewDeflateWriter(writer io.Writer, level int) (Writer, error) {
return sharedDeflateWriterPool.Get(writer, level)
}
func newDeflateWriter(writer io.Writer, level int) (Writer, error) {
if level <= 0 {
level = flate.BestSpeed
} else if level > flate.BestCompression {
@@ -38,10 +44,18 @@ func (this *DeflateWriter) Flush() error {
return this.writer.Flush()
}
func (this *DeflateWriter) Close() error {
func (this *DeflateWriter) Reset(writer io.Writer) {
this.writer.Reset(writer)
}
func (this *DeflateWriter) RawClose() error {
return this.writer.Close()
}
func (this *DeflateWriter) Close() error {
return this.Finish(this)
}
func (this *DeflateWriter) Level() int {
return this.level
}

View File

@@ -0,0 +1,36 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions_test
import (
"bytes"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"strings"
"testing"
)
func BenchmarkDeflateWriter_Write(b *testing.B) {
var data = []byte(strings.Repeat("A", 1024))
for i := 0; i < b.N; i++ {
var buf = &bytes.Buffer{}
writer, err := compressions.NewDeflateWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
for j := 0; j < 100; j++ {
_, err = writer.Write(data)
if err != nil {
b.Fatal(err)
}
/**err = writer.Flush()
if err != nil {
b.Fatal(err)
}**/
}
_ = writer.Close()
}
}

View File

@@ -1,51 +0,0 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"bytes"
"io"
)
type EncodingWriter struct {
contentEncoding ContentEncoding
writer Writer
buf *bytes.Buffer
}
func NewEncodingWriter(contentEncoding ContentEncoding, writer Writer) (Writer, error) {
return &EncodingWriter{
contentEncoding: contentEncoding,
writer: writer,
buf: &bytes.Buffer{},
}, nil
}
func (this *EncodingWriter) Write(p []byte) (int, error) {
return this.buf.Write(p)
}
func (this *EncodingWriter) Flush() error {
return this.writer.Flush()
}
func (this *EncodingWriter) Close() error {
reader, err := NewReader(this.buf, this.contentEncoding)
if err != nil {
_ = this.writer.Close()
return err
}
_, err = io.Copy(this.writer, reader)
if err != nil {
_ = reader.Close()
_ = this.writer.Close()
return err
}
_ = reader.Close()
return this.writer.Close()
}
func (this *EncodingWriter) Level() int {
return this.writer.Level()
}

View File

@@ -1,47 +0,0 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"testing"
)
func TestNewEncodingWriter(t *testing.T) {
var buf = &bytes.Buffer{}
subWriter, err := NewWriter(buf, serverconfigs.HTTPCompressionTypeGzip, 5)
if err != nil {
t.Fatal(err)
}
writer, err := NewEncodingWriter(ContentEncodingGzip, subWriter)
if err != nil {
t.Fatal(err)
}
gzipBuf := &bytes.Buffer{}
gzipWriter, err := NewGzipWriter(gzipBuf, 5)
if err != nil {
t.Fatal(err)
}
_, err = gzipWriter.Write([]byte("Hello"))
if err != nil {
t.Fatal(err)
}
_, err = gzipWriter.Write([]byte("World"))
if err != nil {
t.Fatal(err)
}
_ = gzipWriter.Close()
_, err = writer.Write(gzipBuf.Bytes())
if err != nil {
t.Fatal(err)
}
_ = writer.Close()
t.Log(buf.String())
}

View File

@@ -3,16 +3,22 @@
package compressions
import (
"compress/gzip"
"github.com/klauspost/compress/gzip"
"io"
)
type GzipWriter struct {
BaseWriter
writer *gzip.Writer
level int
}
func NewGzipWriter(writer io.Writer, level int) (Writer, error) {
return sharedGzipWriterPool.Get(writer, level)
}
func newGzipWriter(writer io.Writer, level int) (Writer, error) {
if level <= 0 {
level = gzip.BestSpeed
} else if level > gzip.BestCompression {
@@ -38,10 +44,18 @@ func (this *GzipWriter) Flush() error {
return this.writer.Flush()
}
func (this *GzipWriter) Close() error {
func (this *GzipWriter) Reset(writer io.Writer) {
this.writer.Reset(writer)
}
func (this *GzipWriter) RawClose() error {
return this.writer.Close()
}
func (this *GzipWriter) Close() error {
return this.Finish(this)
}
func (this *GzipWriter) Level() int {
return this.level
}

View File

@@ -0,0 +1,64 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions_test
import (
"bytes"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"strings"
"testing"
)
func BenchmarkGzipWriter_Write(b *testing.B) {
var data = []byte(strings.Repeat("A", 1024))
for i := 0; i < b.N; i++ {
var buf = &bytes.Buffer{}
writer, err := compressions.NewGzipWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
for j := 0; j < 100; j++ {
_, err = writer.Write(data)
if err != nil {
b.Fatal(err)
}
/**err = writer.Flush()
if err != nil {
b.Fatal(err)
}**/
}
_ = writer.Close()
}
}
func BenchmarkGzipWriter_Write_Parallel(b *testing.B) {
var data = []byte(strings.Repeat("A", 1024))
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
var buf = &bytes.Buffer{}
writer, err := compressions.NewGzipWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
for j := 0; j < 100; j++ {
_, err = writer.Write(data)
if err != nil {
b.Fatal(err)
}
/**err = writer.Flush()
if err != nil {
b.Fatal(err)
}**/
}
_ = writer.Close()
}
})
}

View File

@@ -0,0 +1,61 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"io"
)
type WriterPool struct {
m map[int]chan Writer // level => chan Writer
newFunc func(writer io.Writer, level int) (Writer, error)
}
func NewWriterPool(maxSize int, maxLevel int, newFunc func(writer io.Writer, level int) (Writer, error)) *WriterPool {
if maxSize <= 0 {
maxSize = 1024
}
var m = map[int]chan Writer{}
for i := 0; i <= maxLevel; i++ {
m[i] = make(chan Writer, maxSize)
}
return &WriterPool{
m: m,
newFunc: newFunc,
}
}
func (this *WriterPool) Get(parentWriter io.Writer, level int) (Writer, error) {
c, ok := this.m[level]
if !ok {
c = this.m[0]
}
select {
case writer := <-c:
writer.Reset(parentWriter)
writer.ResetFinish()
return writer, nil
default:
writer, err := this.newFunc(parentWriter, level)
if err != nil {
return nil, err
}
writer.SetPool(this)
return writer, nil
}
}
func (this *WriterPool) Put(writer Writer) {
var level = writer.Level()
c, ok := this.m[level]
if !ok {
c = this.m[0]
}
select {
case c <- writer:
default:
}
}

View File

@@ -0,0 +1,21 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/andybalholm/brotli"
"io"
)
var sharedBrotliWriterPool *WriterPool
func init() {
var maxSize = utils.SystemMemoryGB() * 256
if maxSize == 0 {
maxSize = 256
}
sharedBrotliWriterPool = NewWriterPool(maxSize, brotli.BestCompression, func(writer io.Writer, level int) (Writer, error) {
return newBrotliWriter(writer, level)
})
}

View File

@@ -0,0 +1,21 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"compress/flate"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
)
var sharedDeflateWriterPool *WriterPool
func init() {
var maxSize = utils.SystemMemoryGB() * 256
if maxSize == 0 {
maxSize = 256
}
sharedDeflateWriterPool = NewWriterPool(maxSize, flate.BestCompression, func(writer io.Writer, level int) (Writer, error) {
return newDeflateWriter(writer, level)
})
}

View File

@@ -0,0 +1,21 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"compress/gzip"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
)
var sharedGzipWriterPool *WriterPool
func init() {
var maxSize = utils.SystemMemoryGB() * 256
if maxSize == 0 {
maxSize = 256
}
sharedGzipWriterPool = NewWriterPool(maxSize, gzip.BestCompression, func(writer io.Writer, level int) (Writer, error) {
return newGzipWriter(writer, level)
})
}

View File

@@ -1,12 +1,12 @@
package configs
import (
"github.com/go-yaml/yaml"
"github.com/iwind/TeaGo/Tea"
"gopkg.in/yaml.v3"
"io/ioutil"
)
// 节点API配置
// APIConfig 节点API配置
type APIConfig struct {
RPC struct {
Endpoints []string `yaml:"endpoints"`

View File

@@ -1,6 +1,6 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build community
// +build community
//go:build !plus
// +build !plus
package teaconst

View File

@@ -1,7 +1,7 @@
package teaconst
const (
Version = "0.4.1"
Version = "0.4.8.1"
ProductName = "Edge Node"
ProcessName = "edge-node"
@@ -13,4 +13,6 @@ const (
// SystemdServiceName systemd
SystemdServiceName = "edge-node"
AccessLogSockName = "edge-node.accesslog.sock"
)

View File

@@ -14,4 +14,9 @@ var (
NodeIdString = ""
GlobalProductName = nodeconfigs.DefaultProductName
IsQuiting = false // 是否正在退出
EnableDBStat = false // 是否开启本地数据库统计
DiskIsFast = false // 是否为高速硬盘
)

View File

@@ -1,12 +1,12 @@
package encrypt
type MethodInterface interface {
// 初始化
// Init 初始化
Init(key []byte, iv []byte) error
// 加密
// Encrypt 加密
Encrypt(src []byte) (dst []byte, err error)
// 解密
// Decrypt 解密
Decrypt(dst []byte) (src []byte, err error)
}

View File

@@ -2,6 +2,7 @@ package errors
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"path/filepath"
"runtime"
"strconv"
@@ -15,7 +16,7 @@ type errorObj struct {
}
func (this *errorObj) Error() string {
s := this.err.Error() + "\n " + this.file
s := this.err.Error() + "\n " + utils.RemoveWorkspace(this.file)
if len(this.funcName) > 0 {
s += ":" + this.funcName + "()"
}
@@ -23,7 +24,7 @@ func (this *errorObj) Error() string {
return s
}
// 新错误
// New 新错误
func New(errText string) error {
ptr, file, line, ok := runtime.Caller(1)
funcName := ""
@@ -39,7 +40,7 @@ func New(errText string) error {
}
}
// 包装已有错误
// Wrap 包装已有错误
func Wrap(err error) error {
if err == nil {
return nil

View File

@@ -3,8 +3,10 @@ package events
type Event = string
const (
EventStart Event = "start" // start loading
EventLoaded Event = "loaded" // first load
EventQuit Event = "quit" // quit node gracefully
EventReload Event = "reload" // reload config
EventStart Event = "start" // start loading
EventLoaded Event = "loaded" // first load
EventQuit Event = "quit" // quit node gracefully
EventReload Event = "reload" // reload config
EventTerminated Event = "terminated" // process terminated
EventNFTablesReady Event = "nftablesReady" // nftables ready
)

View File

@@ -1,6 +1,7 @@
package events
import (
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"sync"
)
@@ -59,6 +60,14 @@ func Remove(key interface{}) {
// Notify 通知事件
func Notify(event Event) {
// 特殊事件
switch event {
case EventQuit:
teaconst.IsQuiting = true
case EventTerminated:
teaconst.IsQuiting = true
}
locker.Lock()
m := eventsMap[event]
locker.Unlock()

View File

@@ -0,0 +1,502 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package firewalls
import (
"bytes"
"encoding/json"
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/ddosconfigs"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
stringutil "github.com/iwind/TeaGo/utils/string"
"net"
"os/exec"
"strings"
)
var SharedDDoSProtectionManager = NewDDoSProtectionManager()
func init() {
events.On(events.EventReload, func() {
if nftablesInstance == nil {
return
}
nodeConfig, _ := nodeconfigs.SharedNodeConfig()
if nodeConfig != nil {
err := SharedDDoSProtectionManager.Apply(nodeConfig.DDOSProtection)
if err != nil {
remotelogs.Error("FIREWALL", "apply DDoS protection failed: "+err.Error())
}
}
})
events.On(events.EventNFTablesReady, func() {
nodeConfig, _ := nodeconfigs.SharedNodeConfig()
if nodeConfig != nil {
err := SharedDDoSProtectionManager.Apply(nodeConfig.DDOSProtection)
if err != nil {
remotelogs.Error("FIREWALL", "apply DDoS protection failed: "+err.Error())
}
}
})
}
// DDoSProtectionManager DDoS防护
type DDoSProtectionManager struct {
nftPath string
lastAllowIPList []string
lastConfig []byte
}
// NewDDoSProtectionManager 获取新对象
func NewDDoSProtectionManager() *DDoSProtectionManager {
nftPath, _ := exec.LookPath("nft")
return &DDoSProtectionManager{
nftPath: nftPath,
}
}
// Apply 应用配置
func (this *DDoSProtectionManager) Apply(config *ddosconfigs.ProtectionConfig) error {
// 同集群节点IP白名单
var allowIPListChanged = false
nodeConfig, _ := nodeconfigs.SharedNodeConfig()
if nodeConfig != nil {
var allowIPList = nodeConfig.AllowedIPs
if !utils.ContainsSameStrings(allowIPList, this.lastAllowIPList) {
allowIPListChanged = true
this.lastAllowIPList = allowIPList
}
}
// 对比配置
configJSON, err := json.Marshal(config)
if err != nil {
return errors.New("encode config to json failed: " + err.Error())
}
if !allowIPListChanged && bytes.Equal(this.lastConfig, configJSON) {
return nil
}
remotelogs.Println("FIREWALL", "change DDoS protection config")
if len(this.nftPath) == 0 {
return errors.New("can not find nft command")
}
if nftablesInstance == nil {
return errors.New("nftables instance should not be nil")
}
if config == nil {
// TCP
err := this.removeTCPRules()
if err != nil {
return err
}
// TODO other protocols
return nil
}
// TCP
if config.TCP == nil {
err := this.removeTCPRules()
if err != nil {
return err
}
} else {
// allow ip list
var allowIPList = []string{}
for _, ipConfig := range config.TCP.AllowIPList {
allowIPList = append(allowIPList, ipConfig.IP)
}
for _, ip := range this.lastAllowIPList {
if !lists.ContainsString(allowIPList, ip) {
allowIPList = append(allowIPList, ip)
}
}
err = this.updateAllowIPList(allowIPList)
if err != nil {
return err
}
// tcp
if config.TCP.IsOn {
err := this.addTCPRules(config.TCP)
if err != nil {
return err
}
} else {
err := this.removeTCPRules()
if err != nil {
return err
}
}
}
this.lastConfig = configJSON
return nil
}
// 添加TCP规则
func (this *DDoSProtectionManager) addTCPRules(tcpConfig *ddosconfigs.TCPConfig) error {
// 检查nft版本不能小于0.9
if len(nftablesInstance.version) > 0 && stringutil.VersionCompare("0.9", nftablesInstance.version) > 0 {
return nil
}
var ports = []int32{}
for _, portConfig := range tcpConfig.Ports {
if !lists.ContainsInt32(ports, portConfig.Port) {
ports = append(ports, portConfig.Port)
}
}
if len(ports) == 0 {
ports = []int32{80, 443}
}
for _, filter := range nftablesFilters {
chain, oldRules, err := this.getRules(filter)
if err != nil {
return errors.New("get old rules failed: " + err.Error())
}
var protocol = filter.protocol()
// max connections
var maxConnections = tcpConfig.MaxConnections
if maxConnections <= 0 {
maxConnections = nodeconfigs.DefaultTCPMaxConnections
if maxConnections <= 0 {
maxConnections = 100000
}
}
// max connections per ip
var maxConnectionsPerIP = tcpConfig.MaxConnectionsPerIP
if maxConnectionsPerIP <= 0 {
maxConnectionsPerIP = nodeconfigs.DefaultTCPMaxConnectionsPerIP
if maxConnectionsPerIP <= 0 {
maxConnectionsPerIP = 100000
}
}
// new connections rate
var newConnectionsRate = tcpConfig.NewConnectionsRate
if newConnectionsRate <= 0 {
newConnectionsRate = nodeconfigs.DefaultTCPNewConnectionsRate
if newConnectionsRate <= 0 {
newConnectionsRate = 100000
}
}
// 检查是否有变化
var hasChanges = false
for _, port := range ports {
if !this.existsRule(oldRules, []string{"tcp", types.String(port), "maxConnections", types.String(maxConnections)}) {
hasChanges = true
break
}
if !this.existsRule(oldRules, []string{"tcp", types.String(port), "maxConnectionsPerIP", types.String(maxConnectionsPerIP)}) {
hasChanges = true
break
}
if !this.existsRule(oldRules, []string{"tcp", types.String(port), "newConnectionsRate", types.String(newConnectionsRate)}) {
hasChanges = true
break
}
}
if !hasChanges {
// 检查是否有多余的端口
var oldPorts = this.getTCPPorts(oldRules)
if !this.eqPorts(ports, oldPorts) {
hasChanges = true
}
}
if !hasChanges {
return nil
}
// 先清空所有相关规则
err = this.removeOldTCPRules(chain, oldRules)
if err != nil {
return errors.New("delete old rules failed: " + err.Error())
}
// 添加新规则
for _, port := range ports {
if maxConnections > 0 {
var cmd = exec.Command(this.nftPath, "add", "rule", protocol, filter.Name, nftablesChainName, "tcp", "dport", types.String(port), "ct", "count", "over", types.String(maxConnections), "counter", "drop", "comment", this.encodeUserData([]string{"tcp", types.String(port), "maxConnections", types.String(maxConnections)}))
var stderr = &bytes.Buffer{}
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
return errors.New("add nftables rule '" + cmd.String() + "' failed: " + err.Error() + " (" + stderr.String() + ")")
}
}
if maxConnectionsPerIP > 0 {
var cmd = exec.Command(this.nftPath, "add", "rule", protocol, filter.Name, nftablesChainName, "tcp", "dport", types.String(port), "meter", "meter-"+protocol+"-"+types.String(port)+"-max-connections", "{ "+protocol+" saddr ct count over "+types.String(maxConnectionsPerIP)+" }", "counter", "drop", "comment", this.encodeUserData([]string{"tcp", types.String(port), "maxConnectionsPerIP", types.String(maxConnectionsPerIP)}))
var stderr = &bytes.Buffer{}
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
return errors.New("add nftables rule '" + cmd.String() + "' failed: " + err.Error() + " (" + stderr.String() + ")")
}
}
if newConnectionsRate > 0 {
// TODO 思考是否有惩罚机制
var cmd = exec.Command(this.nftPath, "add", "rule", protocol, filter.Name, nftablesChainName, "tcp", "dport", types.String(port), "ct", "state", "new", "meter", "meter-"+protocol+"-"+types.String(port)+"-new-connections-rate", "{ "+protocol+" saddr limit rate over "+types.String(newConnectionsRate)+"/minute burst "+types.String(newConnectionsRate+3)+" packets }" /**"add", "@deny_set", "{"+protocol+" saddr}",**/, "counter", "drop", "comment", this.encodeUserData([]string{"tcp", types.String(port), "newConnectionsRate", types.String(newConnectionsRate)}))
var stderr = &bytes.Buffer{}
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
return errors.New("add nftables rule '" + cmd.String() + "' failed: " + err.Error() + " (" + stderr.String() + ")")
}
}
}
}
return nil
}
// 删除TCP规则
func (this *DDoSProtectionManager) removeTCPRules() error {
for _, filter := range nftablesFilters {
chain, rules, err := this.getRules(filter)
// TCP
err = this.removeOldTCPRules(chain, rules)
if err != nil {
return err
}
}
return nil
}
// 组合user data
// 数据中不能包含字母、数字、下划线以外的数据
func (this *DDoSProtectionManager) encodeUserData(attrs []string) string {
if attrs == nil {
return ""
}
return "ZZ" + strings.Join(attrs, "_") + "ZZ"
}
// 解码user data
func (this *DDoSProtectionManager) decodeUserData(data []byte) []string {
if len(data) == 0 {
return nil
}
var dataCopy = make([]byte, len(data))
copy(dataCopy, data)
var separatorLen = 2
var index1 = bytes.Index(dataCopy, []byte{'Z', 'Z'})
if index1 < 0 {
return nil
}
dataCopy = dataCopy[index1+separatorLen:]
var index2 = bytes.LastIndex(dataCopy, []byte{'Z', 'Z'})
if index2 < 0 {
return nil
}
var s = string(dataCopy[:index2])
var pieces = strings.Split(s, "_")
for index, piece := range pieces {
pieces[index] = strings.TrimSpace(piece)
}
return pieces
}
// 清除规则
func (this *DDoSProtectionManager) removeOldTCPRules(chain *nftables.Chain, rules []*nftables.Rule) error {
for _, rule := range rules {
var pieces = this.decodeUserData(rule.UserData())
if len(pieces) != 4 {
continue
}
if pieces[0] != "tcp" {
continue
}
switch pieces[2] {
case "maxConnections", "maxConnectionsPerIP", "newConnectionsRate":
err := chain.DeleteRule(rule)
if err != nil {
return err
}
}
}
return nil
}
// 根据参数检查规则是否存在
func (this *DDoSProtectionManager) existsRule(rules []*nftables.Rule, attrs []string) (exists bool) {
if len(attrs) == 0 {
return false
}
for _, oldRule := range rules {
var pieces = this.decodeUserData(oldRule.UserData())
if len(attrs) != len(pieces) {
continue
}
var isSame = true
for index, piece := range pieces {
if strings.TrimSpace(piece) != attrs[index] {
isSame = false
break
}
}
if isSame {
return true
}
}
return false
}
// 获取规则中的端口号
func (this *DDoSProtectionManager) getTCPPorts(rules []*nftables.Rule) []int32 {
var ports = []int32{}
for _, rule := range rules {
var pieces = this.decodeUserData(rule.UserData())
if len(pieces) != 4 {
continue
}
if pieces[0] != "tcp" {
continue
}
var port = types.Int32(pieces[1])
if port > 0 && !lists.ContainsInt32(ports, port) {
ports = append(ports, port)
}
}
return ports
}
// 检查端口是否一样
func (this *DDoSProtectionManager) eqPorts(ports1 []int32, ports2 []int32) bool {
if len(ports1) != len(ports2) {
return false
}
var portMap = map[int32]bool{}
for _, port := range ports2 {
portMap[port] = true
}
for _, port := range ports1 {
_, ok := portMap[port]
if !ok {
return false
}
}
return true
}
// 查找Table
func (this *DDoSProtectionManager) getTable(filter *nftablesTableDefinition) (*nftables.Table, error) {
var family nftables.TableFamily
if filter.IsIPv4 {
family = nftables.TableFamilyIPv4
} else if filter.IsIPv6 {
family = nftables.TableFamilyIPv6
} else {
return nil, errors.New("table '" + filter.Name + "' should be IPv4 or IPv6")
}
return nftablesInstance.conn.GetTable(filter.Name, family)
}
// 查找所有规则
func (this *DDoSProtectionManager) getRules(filter *nftablesTableDefinition) (*nftables.Chain, []*nftables.Rule, error) {
table, err := this.getTable(filter)
if err != nil {
return nil, nil, errors.New("get table failed: " + err.Error())
}
chain, err := table.GetChain(nftablesChainName)
if err != nil {
return nil, nil, errors.New("get chain failed: " + err.Error())
}
rules, err := chain.GetRules()
return chain, rules, err
}
// 更新白名单
func (this *DDoSProtectionManager) updateAllowIPList(allIPList []string) error {
if nftablesInstance == nil {
return nil
}
var allMap = map[string]zero.Zero{}
for _, ip := range allIPList {
allMap[ip] = zero.New()
}
for _, set := range []*nftables.Set{nftablesInstance.allowIPv4Set, nftablesInstance.allowIPv6Set} {
var isIPv4 = set == nftablesInstance.allowIPv4Set
var isIPv6 = !isIPv4
// 现有的
oldList, err := set.GetIPElements()
if err != nil {
return err
}
var oldMap = map[string]zero.Zero{} // ip=> zero
for _, ip := range oldList {
oldMap[ip] = zero.New()
if (utils.IsIPv4(ip) && isIPv4) || (utils.IsIPv6(ip) && isIPv6) {
_, ok := allMap[ip]
if !ok {
// 不存在则删除
err = set.DeleteIPElement(ip)
if err != nil {
return errors.New("delete ip element '" + ip + "' failed: " + err.Error())
}
}
}
}
// 新增的
for _, ip := range allIPList {
var ipObj = net.ParseIP(ip)
if ipObj == nil {
continue
}
if (utils.IsIPv4(ip) && isIPv4) || (utils.IsIPv6(ip) && isIPv6) {
_, ok := oldMap[ip]
if !ok {
// 不存在则添加
err = set.AddIPElement(ip, nil)
if err != nil {
return errors.New("add ip '" + ip + "' failed: " + err.Error())
}
}
}
}
}
return nil
}

View File

@@ -0,0 +1,23 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build !linux
// +build !linux
package firewalls
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/ddosconfigs"
)
var SharedDDoSProtectionManager = NewDDoSProtectionManager()
type DDoSProtectionManager struct {
nftPath string
}
func NewDDoSProtectionManager() *DDoSProtectionManager {
return &DDoSProtectionManager{}
}
func (this *DDoSProtectionManager) Apply(config *ddosconfigs.ProtectionConfig) error {
return nil
}

View File

@@ -5,17 +5,18 @@ package firewalls
import (
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"runtime"
"sync"
)
var currentFirewall FirewallInterface
var firewallLocker = &sync.Mutex{}
// 初始化
func init() {
events.On(events.EventLoaded, func() {
var firewall = Firewall()
if firewall.Name() == "mock" {
remotelogs.Warn("FIREWALL", "'firewalld' on this system should be enabled to block attackers more effectively")
} else {
if firewall.Name() != "mock" {
remotelogs.Println("FIREWALL", "found local firewall '"+firewall.Name()+"'")
}
})
@@ -23,12 +24,30 @@ func init() {
// Firewall 查找当前系统中最适合的防火墙
func Firewall() FirewallInterface {
firewallLocker.Lock()
defer firewallLocker.Unlock()
if currentFirewall != nil {
return currentFirewall
}
// nftables
if runtime.GOOS == "linux" {
nftables, err := NewNFTablesFirewall()
if err != nil {
remotelogs.Warn("FIREWALL", "'nftables' should be installed on the system to enhance security (init failed: "+err.Error()+")")
} else {
if nftables.IsReady() {
currentFirewall = nftables
events.Notify(events.EventNFTablesReady)
return nftables
} else {
remotelogs.Warn("FIREWALL", "'nftables' should be enabled on the system to enhance security")
}
}
}
// firewalld
{
if runtime.GOOS == "linux" {
var firewalld = NewFirewalld()
if firewalld.IsReady() {
currentFirewall = firewalld

View File

@@ -23,10 +23,13 @@ func NewFirewalld() *Firewalld {
path, err := exec.LookPath("firewall-cmd")
if err == nil && len(path) > 0 {
var cmd = exec.Command(path, "-V")
var cmd = exec.Command(path, "--state")
err := cmd.Run()
if err == nil {
firewalld.exe = path
// TODO check firewalld status with 'firewall-cmd --state' (running or not running),
// but we should recover the state when firewalld state changes, maybe check it every minutes
firewalld.isReady = true
firewalld.init()
}
@@ -58,6 +61,11 @@ func (this *Firewalld) IsReady() bool {
return this.isReady
}
// IsMock 是否为模拟
func (this *Firewalld) IsMock() bool {
return false
}
func (this *Firewalld) AllowPort(port int, protocol string) error {
if !this.isReady {
return nil

View File

@@ -10,6 +10,9 @@ type FirewallInterface interface {
// IsReady 是否已准备被调用
IsReady() bool
// IsMock 是否为模拟
IsMock() bool
// AllowPort 允许端口
AllowPort(port int, protocol string) error

View File

@@ -20,6 +20,11 @@ func (this *MockFirewall) IsReady() bool {
return true
}
// IsMock 是否为模拟
func (this *MockFirewall) IsMock() bool {
return true
}
// AllowPort 允许端口
func (this *MockFirewall) AllowPort(port int, protocol string) error {
_ = port

View File

@@ -0,0 +1,395 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package firewalls
import (
"bytes"
"errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/types"
"net"
"os/exec"
"regexp"
"runtime"
"strings"
"time"
)
// check nft status, if being enabled we load it automatically
func init() {
if runtime.GOOS == "linux" {
var ticker = time.NewTicker(3 * time.Minute)
go func() {
for range ticker.C {
// if already ready, we break
if nftablesIsReady {
ticker.Stop()
break
}
_, err := exec.LookPath("nft")
if err == nil {
nftablesFirewall, err := NewNFTablesFirewall()
if err != nil {
continue
}
currentFirewall = nftablesFirewall
remotelogs.Println("FIREWALL", "nftables is ready")
// fire event
if nftablesFirewall.IsReady() {
events.Notify(events.EventNFTablesReady)
}
ticker.Stop()
break
}
}
}()
}
}
var nftablesInstance *NFTablesFirewall
var nftablesIsReady = false
var nftablesFilters = []*nftablesTableDefinition{
// we shorten the name for table name length restriction
{Name: "edge_dft_v4", IsIPv4: true},
{Name: "edge_dft_v6", IsIPv6: true},
}
var nftablesChainName = "input"
type nftablesTableDefinition struct {
Name string
IsIPv4 bool
IsIPv6 bool
}
func (this *nftablesTableDefinition) protocol() string {
if this.IsIPv6 {
return "ip6"
}
return "ip"
}
func NewNFTablesFirewall() (*NFTablesFirewall, error) {
var firewall = &NFTablesFirewall{
conn: nftables.NewConn(),
}
err := firewall.init()
if err != nil {
return nil, err
}
return firewall, nil
}
type NFTablesFirewall struct {
conn *nftables.Conn
isReady bool
version string
allowIPv4Set *nftables.Set
allowIPv6Set *nftables.Set
denyIPv4Set *nftables.Set
denyIPv6Set *nftables.Set
firewalld *Firewalld
}
func (this *NFTablesFirewall) init() error {
// check nft
nftPath, err := exec.LookPath("nft")
if err != nil {
return errors.New("nft not found")
}
this.version = this.readVersion(nftPath)
// table
for _, tableDef := range nftablesFilters {
var family nftables.TableFamily
if tableDef.IsIPv4 {
family = nftables.TableFamilyIPv4
} else if tableDef.IsIPv6 {
family = nftables.TableFamilyIPv6
} else {
return errors.New("invalid table family: " + types.String(tableDef))
}
table, err := this.conn.GetTable(tableDef.Name, family)
if err != nil {
if nftables.IsNotFound(err) {
if tableDef.IsIPv4 {
table, err = this.conn.AddIPv4Table(tableDef.Name)
} else if tableDef.IsIPv6 {
table, err = this.conn.AddIPv6Table(tableDef.Name)
}
if err != nil {
return errors.New("create table '" + tableDef.Name + "' failed: " + err.Error())
}
} else {
return errors.New("get table '" + tableDef.Name + "' failed: " + err.Error())
}
}
if table == nil {
return errors.New("can not create table '" + tableDef.Name + "'")
}
// chain
var chainName = nftablesChainName
chain, err := table.GetChain(chainName)
if err != nil {
if nftables.IsNotFound(err) {
chain, err = table.AddAcceptChain(chainName)
if err != nil {
return errors.New("create chain '" + chainName + "' failed: " + err.Error())
}
} else {
return errors.New("get chain '" + chainName + "' failed: " + err.Error())
}
}
if chain == nil {
return errors.New("can not create chain '" + chainName + "'")
}
// allow lo
var loRuleName = []byte("lo")
_, err = chain.GetRuleWithUserData(loRuleName)
if err != nil {
if nftables.IsNotFound(err) {
_, err = chain.AddAcceptInterfaceRule("lo", loRuleName)
}
if err != nil {
return errors.New("add 'lo' rule failed: " + err.Error())
}
}
// allow set
// "allow" should be always first
for _, setAction := range []string{"allow", "deny"} {
var setName = setAction + "_set"
set, err := table.GetSet(setName)
if err != nil {
if nftables.IsNotFound(err) {
var keyType nftables.SetDataType
if tableDef.IsIPv4 {
keyType = nftables.TypeIPAddr
} else if tableDef.IsIPv6 {
keyType = nftables.TypeIP6Addr
}
set, err = table.AddSet(setName, &nftables.SetOptions{
KeyType: keyType,
HasTimeout: true,
})
if err != nil {
return errors.New("create set '" + setName + "' failed: " + err.Error())
}
} else {
return errors.New("get set '" + setName + "' failed: " + err.Error())
}
}
if set == nil {
return errors.New("can not create set '" + setName + "'")
}
if tableDef.IsIPv4 {
if setAction == "allow" {
this.allowIPv4Set = set
} else {
this.denyIPv4Set = set
}
} else if tableDef.IsIPv6 {
if setAction == "allow" {
this.allowIPv6Set = set
} else {
this.denyIPv6Set = set
}
}
// rule
var ruleName = []byte(setAction)
rule, err := chain.GetRuleWithUserData(ruleName)
if err != nil {
if nftables.IsNotFound(err) {
if tableDef.IsIPv4 {
if setAction == "allow" {
rule, err = chain.AddAcceptIPv4SetRule(setName, ruleName)
} else {
rule, err = chain.AddDropIPv4SetRule(setName, ruleName)
}
} else if tableDef.IsIPv6 {
if setAction == "allow" {
rule, err = chain.AddAcceptIPv6SetRule(setName, ruleName)
} else {
rule, err = chain.AddDropIPv6SetRule(setName, ruleName)
}
}
if err != nil {
return errors.New("add rule failed: " + err.Error())
}
} else {
return errors.New("get rule failed: " + err.Error())
}
}
if rule == nil {
return errors.New("can not create rule '" + string(ruleName) + "'")
}
}
}
this.isReady = true
nftablesIsReady = true
nftablesInstance = this
// load firewalld
var firewalld = NewFirewalld()
if firewalld.IsReady() {
this.firewalld = firewalld
}
return nil
}
// Name 名称
func (this *NFTablesFirewall) Name() string {
return "nftables"
}
// IsReady 是否已准备被调用
func (this *NFTablesFirewall) IsReady() bool {
return this.isReady
}
// IsMock 是否为模拟
func (this *NFTablesFirewall) IsMock() bool {
return false
}
// AllowPort 允许端口
func (this *NFTablesFirewall) AllowPort(port int, protocol string) error {
if this.firewalld != nil {
return this.firewalld.AllowPort(port, protocol)
}
return nil
}
// RemovePort 删除端口
func (this *NFTablesFirewall) RemovePort(port int, protocol string) error {
if this.firewalld != nil {
return this.firewalld.RemovePort(port, protocol)
}
return nil
}
// AllowSourceIP Allow把IP加入白名单
func (this *NFTablesFirewall) AllowSourceIP(ip string) error {
var data = net.ParseIP(ip)
if data == nil {
return errors.New("invalid ip '" + ip + "'")
}
if strings.Contains(ip, ":") { // ipv6
if this.allowIPv6Set == nil {
return errors.New("ipv6 ip set is nil")
}
return this.allowIPv6Set.AddElement(data.To16(), nil)
}
// ipv4
if this.allowIPv4Set == nil {
return errors.New("ipv4 ip set is nil")
}
return this.allowIPv4Set.AddElement(data.To4(), nil)
}
// RejectSourceIP 拒绝某个源IP连接
// we did not create set for drop ip, so we reuse DropSourceIP() method here
func (this *NFTablesFirewall) RejectSourceIP(ip string, timeoutSeconds int) error {
return this.DropSourceIP(ip, timeoutSeconds)
}
// DropSourceIP 丢弃某个源IP数据
func (this *NFTablesFirewall) DropSourceIP(ip string, timeoutSeconds int) error {
var data = net.ParseIP(ip)
if data == nil {
return errors.New("invalid ip '" + ip + "'")
}
if strings.Contains(ip, ":") { // ipv6
if this.denyIPv6Set == nil {
return errors.New("ipv6 ip set is nil")
}
return this.denyIPv6Set.AddElement(data.To16(), &nftables.ElementOptions{
Timeout: time.Duration(timeoutSeconds) * time.Second,
})
}
// ipv4
if this.denyIPv4Set == nil {
return errors.New("ipv4 ip set is nil")
}
return this.denyIPv4Set.AddElement(data.To4(), &nftables.ElementOptions{
Timeout: time.Duration(timeoutSeconds) * time.Second,
})
}
// RemoveSourceIP 删除某个源IP
func (this *NFTablesFirewall) RemoveSourceIP(ip string) error {
var data = net.ParseIP(ip)
if data == nil {
return errors.New("invalid ip '" + ip + "'")
}
if strings.Contains(ip, ":") { // ipv6
if this.denyIPv6Set != nil {
err := this.denyIPv6Set.DeleteElement(data.To16())
if err != nil {
return err
}
}
if this.allowIPv6Set != nil {
err := this.allowIPv6Set.DeleteElement(data.To16())
if err != nil {
return err
}
}
return nil
}
// ipv4
if this.allowIPv4Set != nil {
err := this.denyIPv4Set.DeleteElement(data.To4())
if err != nil {
return err
}
err = this.allowIPv4Set.DeleteElement(data.To4())
if err != nil {
return err
}
}
return nil
}
// 读取版本号
func (this *NFTablesFirewall) readVersion(nftPath string) string {
var cmd = exec.Command(nftPath, "--version")
var output = &bytes.Buffer{}
cmd.Stdout = output
err := cmd.Run()
if err != nil {
return ""
}
var outputString = output.String()
var versionMatches = regexp.MustCompile(`nftables v([\d.]+)`).FindStringSubmatch(outputString)
if len(versionMatches) <= 1 {
return ""
}
return versionMatches[1]
}

View File

@@ -0,0 +1,61 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build !linux
// +build !linux
package firewalls
import (
"errors"
)
func NewNFTablesFirewall() (*NFTablesFirewall, error) {
return nil, errors.New("not implemented")
}
type NFTablesFirewall struct {
}
// Name 名称
func (this *NFTablesFirewall) Name() string {
return "nftables"
}
// IsReady 是否已准备被调用
func (this *NFTablesFirewall) IsReady() bool {
return false
}
// IsMock 是否为模拟
func (this *NFTablesFirewall) IsMock() bool {
return true
}
// AllowPort 允许端口
func (this *NFTablesFirewall) AllowPort(port int, protocol string) error {
return nil
}
// RemovePort 删除端口
func (this *NFTablesFirewall) RemovePort(port int, protocol string) error {
return nil
}
// AllowSourceIP Allow把IP加入白名单
func (this *NFTablesFirewall) AllowSourceIP(ip string) error {
return nil
}
// RejectSourceIP 拒绝某个源IP连接
func (this *NFTablesFirewall) RejectSourceIP(ip string, timeoutSeconds int) error {
return nil
}
// DropSourceIP 丢弃某个源IP数据
func (this *NFTablesFirewall) DropSourceIP(ip string, timeoutSeconds int) error {
return nil
}
// RemoveSourceIP 删除某个源IP
func (this *NFTablesFirewall) RemoveSourceIP(ip string) error {
return nil
}

View File

@@ -0,0 +1 @@
build_remote.sh

View File

@@ -0,0 +1,370 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import (
"bytes"
"errors"
nft "github.com/google/nftables"
"github.com/google/nftables/expr"
)
const MaxChainNameLength = 31
type RuleOptions struct {
Exprs []expr.Any
UserData []byte
}
// Chain chain object in table
type Chain struct {
conn *Conn
rawTable *nft.Table
rawChain *nft.Chain
}
func NewChain(conn *Conn, rawTable *nft.Table, rawChain *nft.Chain) *Chain {
return &Chain{
conn: conn,
rawTable: rawTable,
rawChain: rawChain,
}
}
func (this *Chain) Raw() *nft.Chain {
return this.rawChain
}
func (this *Chain) Name() string {
return this.rawChain.Name
}
func (this *Chain) AddRule(options *RuleOptions) (*Rule, error) {
var rawRule = this.conn.Raw().AddRule(&nft.Rule{
Table: this.rawTable,
Chain: this.rawChain,
Exprs: options.Exprs,
UserData: options.UserData,
})
err := this.conn.Commit()
if err != nil {
return nil, err
}
return NewRule(rawRule), nil
}
func (this *Chain) AddAcceptIPv4Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) AddAcceptIPv6Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) AddDropIPv4Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Verdict{
Kind: expr.VerdictDrop,
},
},
UserData: userData,
})
}
func (this *Chain) AddDropIPv6Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Verdict{
Kind: expr.VerdictDrop,
},
},
UserData: userData,
})
}
func (this *Chain) AddRejectIPv4Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Reject{},
},
UserData: userData,
})
}
func (this *Chain) AddRejectIPv6Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Reject{},
},
UserData: userData,
})
}
func (this *Chain) AddAcceptIPv4SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) AddAcceptIPv6SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) AddDropIPv4SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Verdict{
Kind: expr.VerdictDrop,
},
},
UserData: userData,
})
}
func (this *Chain) AddDropIPv6SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Verdict{
Kind: expr.VerdictDrop,
},
},
UserData: userData,
})
}
func (this *Chain) AddRejectIPv4SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Reject{},
},
UserData: userData,
})
}
func (this *Chain) AddRejectIPv6SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Reject{},
},
UserData: userData,
})
}
func (this *Chain) AddAcceptInterfaceRule(interfaceName string, userData []byte) (*Rule, error) {
if len(interfaceName) >= 16 {
return nil, errors.New("invalid interface name '" + interfaceName + "'")
}
var ifname = make([]byte, 16)
copy(ifname, interfaceName+"\x00")
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Meta{
Key: expr.MetaKeyIIFNAME,
Register: 1,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ifname,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) GetRuleWithUserData(userData []byte) (*Rule, error) {
rawRules, err := this.conn.Raw().GetRule(this.rawTable, this.rawChain)
if err != nil {
return nil, err
}
for _, rawRule := range rawRules {
if bytes.Compare(rawRule.UserData, userData) == 0 {
return NewRule(rawRule), nil
}
}
return nil, ErrRuleNotFound
}
func (this *Chain) GetRules() ([]*Rule, error) {
rawRules, err := this.conn.Raw().GetRule(this.rawTable, this.rawChain)
if err != nil {
return nil, err
}
var result = []*Rule{}
for _, rawRule := range rawRules {
result = append(result, NewRule(rawRule))
}
return result, nil
}
func (this *Chain) DeleteRule(rule *Rule) error {
err := this.conn.Raw().DelRule(rule.Raw())
if err != nil {
return err
}
return this.conn.Commit()
}
func (this *Chain) Flush() error {
this.conn.Raw().FlushChain(this.rawChain)
return this.conn.Commit()
}

View File

@@ -0,0 +1,13 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import nft "github.com/google/nftables"
type ChainPolicy = nft.ChainPolicy
// Possible ChainPolicy values.
const (
ChainPolicyDrop = nft.ChainPolicyDrop
ChainPolicyAccept = nft.ChainPolicyAccept
)

View File

@@ -0,0 +1,130 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables_test
import (
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"net"
"testing"
)
func getIPv4Chain(t *testing.T) *nftables.Chain {
var conn = nftables.NewConn()
table, err := conn.GetTable("test_ipv4", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {
table, err = conn.AddIPv4Table("test_ipv4")
if err != nil {
t.Fatal(err)
}
} else {
t.Fatal(err)
}
}
chain, err := table.GetChain("test_chain")
if err != nil {
if err == nftables.ErrChainNotFound {
chain, err = table.AddAcceptChain("test_chain")
}
}
if err != nil {
t.Fatal(err)
}
return chain
}
func TestChain_AddAcceptIPRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddAcceptIPv4Rule(net.ParseIP("192.168.2.40").To4(), nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_AddDropIPRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddDropIPv4Rule(net.ParseIP("192.168.2.31").To4(), nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_AddAcceptSetRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddAcceptIPv4SetRule("ipv4_black_set", nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_AddDropSetRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddDropIPv4SetRule("ipv4_black_set", nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_AddRejectSetRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddRejectIPv4SetRule("ipv4_black_set", nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_GetRuleWithUserData(t *testing.T) {
var chain = getIPv4Chain(t)
rule, err := chain.GetRuleWithUserData([]byte("test"))
if err != nil {
if err == nftables.ErrRuleNotFound {
t.Log("rule not found")
return
} else {
t.Fatal(err)
}
}
t.Log("rule:", rule)
}
func TestChain_GetRules(t *testing.T) {
var chain = getIPv4Chain(t)
rules, err := chain.GetRules()
if err != nil {
t.Fatal(err)
}
for _, rule := range rules {
t.Log("handle:", rule.Handle(), "set name:", rule.LookupSetName(),
"verdict:", rule.VerDict(), "user data:", string(rule.UserData()))
}
}
func TestChain_DeleteRule(t *testing.T) {
var chain = getIPv4Chain(t)
rule, err := chain.GetRuleWithUserData([]byte("test"))
if err != nil {
if err == nftables.ErrRuleNotFound {
t.Log("rule not found")
return
}
t.Fatal(err)
}
err = chain.DeleteRule(rule)
if err != nil {
t.Fatal(err)
}
}
func TestChain_Flush(t *testing.T) {
var chain = getIPv4Chain(t)
err := chain.Flush()
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -0,0 +1,84 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import (
"errors"
nft "github.com/google/nftables"
"github.com/iwind/TeaGo/types"
)
const MaxTableNameLength = 27
type Conn struct {
rawConn *nft.Conn
}
func NewConn() *Conn {
return &Conn{
rawConn: &nft.Conn{},
}
}
func (this *Conn) Raw() *nft.Conn {
return this.rawConn
}
func (this *Conn) GetTable(name string, family TableFamily) (*Table, error) {
rawTables, err := this.rawConn.ListTables()
if err != nil {
return nil, err
}
for _, rawTable := range rawTables {
if rawTable.Name == name && rawTable.Family == family {
return NewTable(this, rawTable), nil
}
}
return nil, ErrTableNotFound
}
func (this *Conn) AddTable(name string, family TableFamily) (*Table, error) {
if len(name) > MaxTableNameLength {
return nil, errors.New("table name too long (max " + types.String(MaxTableNameLength) + ")")
}
var rawTable = this.rawConn.AddTable(&nft.Table{
Family: family,
Name: name,
})
err := this.Commit()
if err != nil {
return nil, err
}
return NewTable(this, rawTable), nil
}
func (this *Conn) AddIPv4Table(name string) (*Table, error) {
return this.AddTable(name, TableFamilyIPv4)
}
func (this *Conn) AddIPv6Table(name string) (*Table, error) {
return this.AddTable(name, TableFamilyIPv6)
}
func (this *Conn) DeleteTable(name string, family TableFamily) error {
table, err := this.GetTable(name, family)
if err != nil {
if err == ErrTableNotFound {
return nil
}
return err
}
this.rawConn.DelTable(table.Raw())
return this.Commit()
}
func (this *Conn) Commit() error {
return this.rawConn.Flush()
}

View File

@@ -0,0 +1,78 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables_test
import (
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"os/exec"
"testing"
)
func TestConn_Test(t *testing.T) {
_, err := exec.LookPath("nft")
if err == nil {
t.Log("ok")
return
}
t.Log(err)
}
func TestConn_GetTable_NotFound(t *testing.T) {
var conn = nftables.NewConn()
table, err := conn.GetTable("a", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {
t.Log("table not found")
} else {
t.Fatal(err)
}
} else {
t.Log("table:", table)
}
}
func TestConn_GetTable(t *testing.T) {
var conn = nftables.NewConn()
table, err := conn.GetTable("myFilter", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {
t.Log("table not found")
} else {
t.Fatal(err)
}
} else {
t.Log("table:", table)
}
}
func TestConn_AddTable(t *testing.T) {
var conn = nftables.NewConn()
{
table, err := conn.AddIPv4Table("test_ipv4")
if err != nil {
t.Fatal(err)
}
t.Log(table.Name())
}
{
table, err := conn.AddIPv6Table("test_ipv6")
if err != nil {
t.Fatal(err)
}
t.Log(table.Name())
}
}
func TestConn_DeleteTable(t *testing.T) {
var conn = nftables.NewConn()
err := conn.DeleteTable("test_ipv4", nftables.TableFamilyIPv4)
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -0,0 +1,8 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
type Element struct {
}

View File

@@ -0,0 +1,19 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import "errors"
var ErrTableNotFound = errors.New("table not found")
var ErrChainNotFound = errors.New("chain not found")
var ErrSetNotFound = errors.New("set not found")
var ErrRuleNotFound = errors.New("rule not found")
func IsNotFound(err error) bool {
if err == nil {
return false
}
return err == ErrTableNotFound || err == ErrChainNotFound || err == ErrSetNotFound || err == ErrRuleNotFound
}

View File

@@ -0,0 +1,18 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import (
nft "github.com/google/nftables"
)
type TableFamily = nft.TableFamily
const (
TableFamilyINet TableFamily = nft.TableFamilyINet
TableFamilyIPv4 TableFamily = nft.TableFamilyIPv4
TableFamilyIPv6 TableFamily = nft.TableFamilyIPv6
TableFamilyARP TableFamily = nft.TableFamilyARP
TableFamilyNetdev TableFamily = nft.TableFamilyNetdev
TableFamilyBridge TableFamily = nft.TableFamilyBridge
)

View File

@@ -0,0 +1,51 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import (
nft "github.com/google/nftables"
"github.com/google/nftables/expr"
)
type Rule struct {
rawRule *nft.Rule
}
func NewRule(rawRule *nft.Rule) *Rule {
return &Rule{
rawRule: rawRule,
}
}
func (this *Rule) Raw() *nft.Rule {
return this.rawRule
}
func (this *Rule) LookupSetName() string {
for _, e := range this.rawRule.Exprs {
exp, ok := e.(*expr.Lookup)
if ok {
return exp.SetName
}
}
return ""
}
func (this *Rule) VerDict() expr.VerdictKind {
for _, e := range this.rawRule.Exprs {
exp, ok := e.(*expr.Verdict)
if ok {
return exp.Kind
}
}
return -100
}
func (this *Rule) Handle() uint64 {
return this.rawRule.Handle
}
func (this *Rule) UserData() []byte {
return this.rawRule.UserData
}

View File

@@ -0,0 +1,161 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/utils"
nft "github.com/google/nftables"
"net"
"strings"
"time"
)
const MaxSetNameLength = 15
type SetOptions struct {
Id uint32
HasTimeout bool
Timeout time.Duration
KeyType SetDataType
DataType SetDataType
Constant bool
Interval bool
Anonymous bool
IsMap bool
}
type ElementOptions struct {
Timeout time.Duration
}
type Set struct {
conn *Conn
rawSet *nft.Set
batch *SetBatch
}
func NewSet(conn *Conn, rawSet *nft.Set) *Set {
return &Set{
conn: conn,
rawSet: rawSet,
batch: &SetBatch{
conn: conn,
rawSet: rawSet,
},
}
}
func (this *Set) Raw() *nft.Set {
return this.rawSet
}
func (this *Set) Name() string {
return this.rawSet.Name
}
func (this *Set) AddElement(key []byte, options *ElementOptions) error {
var rawElement = nft.SetElement{
Key: key,
}
if options != nil {
rawElement.Timeout = options.Timeout
}
err := this.conn.Raw().SetAddElements(this.rawSet, []nft.SetElement{
rawElement,
})
if err != nil {
return err
}
err = this.conn.Commit()
if err != nil {
// retry if exists
if strings.Contains(err.Error(), "file exists") {
deleteErr := this.conn.Raw().SetDeleteElements(this.rawSet, []nft.SetElement{
{
Key: key,
},
})
if deleteErr == nil {
err = this.conn.Raw().SetAddElements(this.rawSet, []nft.SetElement{
rawElement,
})
if err == nil {
err = this.conn.Commit()
}
}
}
}
return err
}
func (this *Set) AddIPElement(ip string, options *ElementOptions) error {
var ipObj = net.ParseIP(ip)
if ipObj == nil {
return errors.New("invalid ip '" + ip + "'")
}
if utils.IsIPv4(ip) {
return this.AddElement(ipObj.To4(), options)
} else {
return this.AddElement(ipObj.To16(), options)
}
}
func (this *Set) DeleteElement(key []byte) error {
err := this.conn.Raw().SetDeleteElements(this.rawSet, []nft.SetElement{
{
Key: key,
},
})
if err != nil {
return err
}
err = this.conn.Commit()
if err != nil {
if strings.Contains(err.Error(), "no such file or directory") {
err = nil
}
}
return err
}
func (this *Set) DeleteIPElement(ip string) error {
var ipObj = net.ParseIP(ip)
if ipObj == nil {
return errors.New("invalid ip '" + ip + "'")
}
if utils.IsIPv4(ip) {
return this.DeleteElement(ipObj.To4())
} else {
return this.DeleteElement(ipObj.To16())
}
}
func (this *Set) Batch() *SetBatch {
return this.batch
}
func (this *Set) GetIPElements() ([]string, error) {
elements, err := this.conn.Raw().GetSetElements(this.rawSet)
if err != nil {
return nil, err
}
var result = []string{}
for _, element := range elements {
result = append(result, net.IP(element.Key).String())
}
return result, nil
}
// not work current time
/**func (this *Set) Flush() error {
this.conn.Raw().FlushSet(this.rawSet)
return this.conn.Commit()
}**/

View File

@@ -0,0 +1,36 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import (
nft "github.com/google/nftables"
)
type SetBatch struct {
conn *Conn
rawSet *nft.Set
}
func (this *SetBatch) AddElement(key []byte, options *ElementOptions) error {
var rawElement = nft.SetElement{
Key: key,
}
if options != nil {
rawElement.Timeout = options.Timeout
}
return this.conn.Raw().SetAddElements(this.rawSet, []nft.SetElement{
rawElement,
})
}
func (this *SetBatch) DeleteElement(key []byte) error {
return this.conn.Raw().SetDeleteElements(this.rawSet, []nft.SetElement{
{
Key: key,
},
})
}
func (this *SetBatch) Commit() error {
return this.conn.Commit()
}

View File

@@ -0,0 +1,57 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import nft "github.com/google/nftables"
type SetDataType = nft.SetDatatype
var (
TypeInvalid = nft.TypeInvalid
TypeVerdict = nft.TypeVerdict
TypeNFProto = nft.TypeNFProto
TypeBitmask = nft.TypeBitmask
TypeInteger = nft.TypeInteger
TypeString = nft.TypeString
TypeLLAddr = nft.TypeLLAddr
TypeIPAddr = nft.TypeIPAddr
TypeIP6Addr = nft.TypeIP6Addr
TypeEtherAddr = nft.TypeEtherAddr
TypeEtherType = nft.TypeEtherType
TypeARPOp = nft.TypeARPOp
TypeInetProto = nft.TypeInetProto
TypeInetService = nft.TypeInetService
TypeICMPType = nft.TypeICMPType
TypeTCPFlag = nft.TypeTCPFlag
TypeDCCPPktType = nft.TypeDCCPPktType
TypeMHType = nft.TypeMHType
TypeTime = nft.TypeTime
TypeMark = nft.TypeMark
TypeIFIndex = nft.TypeIFIndex
TypeARPHRD = nft.TypeARPHRD
TypeRealm = nft.TypeRealm
TypeClassID = nft.TypeClassID
TypeUID = nft.TypeUID
TypeGID = nft.TypeGID
TypeCTState = nft.TypeCTState
TypeCTDir = nft.TypeCTDir
TypeCTStatus = nft.TypeCTStatus
TypeICMP6Type = nft.TypeICMP6Type
TypeCTLabel = nft.TypeCTLabel
TypePktType = nft.TypePktType
TypeICMPCode = nft.TypeICMPCode
TypeICMPV6Code = nft.TypeICMPV6Code
TypeICMPXCode = nft.TypeICMPXCode
TypeDevGroup = nft.TypeDevGroup
TypeDSCP = nft.TypeDSCP
TypeECN = nft.TypeECN
TypeFIBAddr = nft.TypeFIBAddr
TypeBoolean = nft.TypeBoolean
TypeCTEventBit = nft.TypeCTEventBit
TypeIFName = nft.TypeIFName
TypeIGMPType = nft.TypeIGMPType
TypeTimeDate = nft.TypeTimeDate
TypeTimeHour = nft.TypeTimeHour
TypeTimeDay = nft.TypeTimeDay
TypeCGroupV2 = nft.TypeCGroupV2
)

View File

@@ -0,0 +1,110 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables_test
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"github.com/iwind/TeaGo/types"
"github.com/mdlayher/netlink"
"net"
"testing"
"time"
)
func getIPv4Set(t *testing.T) *nftables.Set {
var table = getIPv4Table(t)
set, err := table.GetSet("test_ipv4_set")
if err != nil {
if err == nftables.ErrSetNotFound {
set, err = table.AddSet("test_ipv4_set", &nftables.SetOptions{
KeyType: nftables.TypeIPAddr,
HasTimeout: true,
})
if err != nil {
t.Fatal(err)
}
} else {
t.Fatal(err)
}
}
return set
}
func TestSet_AddElement(t *testing.T) {
var set = getIPv4Set(t)
err := set.AddElement(net.ParseIP("192.168.2.31").To4(), &nftables.ElementOptions{Timeout: 86400 * time.Second})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestSet_DeleteElement(t *testing.T) {
var set = getIPv4Set(t)
err := set.DeleteElement(net.ParseIP("192.168.2.31").To4())
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestSet_Batch(t *testing.T) {
var batch = getIPv4Set(t).Batch()
for _, ip := range []string{"192.168.2.30", "192.168.2.31", "192.168.2.32", "192.168.2.33", "192.168.2.34"} {
var ipData = net.ParseIP(ip).To4()
//err := batch.DeleteElement(ipData)
//if err != nil {
// t.Fatal(err)
//}
err := batch.AddElement(ipData, &nftables.ElementOptions{Timeout: 10 * time.Second})
if err != nil {
t.Fatal(err)
}
}
err := batch.Commit()
if err != nil {
t.Logf("%#v", errors.Unwrap(err).(*netlink.OpError))
t.Fatal(err)
}
t.Log("ok")
}
func TestSet_Add_Many(t *testing.T) {
var set = getIPv4Set(t)
for i := 0; i < 255; i++ {
t.Log(i)
for j := 0; j < 255; j++ {
var ip = "192.167." + types.String(i) + "." + types.String(j)
var ipData = net.ParseIP(ip).To4()
err := set.Batch().AddElement(ipData, &nftables.ElementOptions{Timeout: 3600 * time.Second})
if err != nil {
t.Fatal(err)
}
if j%10 == 0 {
err = set.Batch().Commit()
if err != nil {
t.Fatal(err)
}
}
}
err := set.Batch().Commit()
if err != nil {
t.Fatal(err)
}
}
t.Log("ok")
}
/**func TestSet_Flush(t *testing.T) {
var set = getIPv4Set(t)
err := set.Flush()
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}**/

View File

@@ -0,0 +1,157 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import (
"errors"
nft "github.com/google/nftables"
"github.com/iwind/TeaGo/types"
"strings"
)
type Table struct {
conn *Conn
rawTable *nft.Table
}
func NewTable(conn *Conn, rawTable *nft.Table) *Table {
return &Table{
conn: conn,
rawTable: rawTable,
}
}
func (this *Table) Raw() *nft.Table {
return this.rawTable
}
func (this *Table) Name() string {
return this.rawTable.Name
}
func (this *Table) Family() TableFamily {
return this.rawTable.Family
}
func (this *Table) GetChain(name string) (*Chain, error) {
rawChains, err := this.conn.Raw().ListChains()
if err != nil {
return nil, err
}
for _, rawChain := range rawChains {
// must compare table name
if rawChain.Name == name && rawChain.Table.Name == this.rawTable.Name {
return NewChain(this.conn, this.rawTable, rawChain), nil
}
}
return nil, ErrChainNotFound
}
func (this *Table) AddChain(name string, chainPolicy *ChainPolicy) (*Chain, error) {
if len(name) > MaxChainNameLength {
return nil, errors.New("chain name too long (max " + types.String(MaxChainNameLength) + ")")
}
var rawChain = this.conn.Raw().AddChain(&nft.Chain{
Name: name,
Table: this.rawTable,
Hooknum: nft.ChainHookInput,
Priority: nft.ChainPriorityFilter,
Type: nft.ChainTypeFilter,
Policy: chainPolicy,
})
err := this.conn.Commit()
if err != nil {
return nil, err
}
return NewChain(this.conn, this.rawTable, rawChain), nil
}
func (this *Table) AddAcceptChain(name string) (*Chain, error) {
var policy = ChainPolicyAccept
return this.AddChain(name, &policy)
}
func (this *Table) AddDropChain(name string) (*Chain, error) {
var policy = ChainPolicyDrop
return this.AddChain(name, &policy)
}
func (this *Table) DeleteChain(name string) error {
chain, err := this.GetChain(name)
if err != nil {
if err == ErrChainNotFound {
return nil
}
return err
}
this.conn.Raw().DelChain(chain.Raw())
return this.conn.Commit()
}
func (this *Table) GetSet(name string) (*Set, error) {
rawSet, err := this.conn.Raw().GetSetByName(this.rawTable, name)
if err != nil {
if strings.Contains(err.Error(), "no such file or directory") {
return nil, ErrSetNotFound
}
return nil, err
}
return NewSet(this.conn, rawSet), nil
}
func (this *Table) AddSet(name string, options *SetOptions) (*Set, error) {
if len(name) > MaxSetNameLength {
return nil, errors.New("set name too long (max " + types.String(MaxSetNameLength) + ")")
}
if options == nil {
options = &SetOptions{}
}
var rawSet = &nft.Set{
Table: this.rawTable,
ID: options.Id,
Name: name,
Anonymous: options.Anonymous,
Constant: options.Constant,
Interval: options.Interval,
IsMap: options.IsMap,
HasTimeout: options.HasTimeout,
Timeout: options.Timeout,
KeyType: options.KeyType,
DataType: options.DataType,
}
err := this.conn.Raw().AddSet(rawSet, nil)
if err != nil {
return nil, err
}
err = this.conn.Commit()
if err != nil {
return nil, err
}
return NewSet(this.conn, rawSet), nil
}
func (this *Table) DeleteSet(name string) error {
set, err := this.GetSet(name)
if err != nil {
if err == ErrSetNotFound {
return nil
}
return err
}
this.conn.Raw().DelSet(set.Raw())
return this.conn.Commit()
}
func (this *Table) Flush() error {
this.conn.Raw().FlushTable(this.rawTable)
return this.conn.Commit()
}

View File

@@ -0,0 +1,140 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables_test
import (
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"testing"
)
func getIPv4Table(t *testing.T) *nftables.Table {
var conn = nftables.NewConn()
table, err := conn.GetTable("test_ipv4", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {
table, err = conn.AddIPv4Table("test_ipv4")
if err != nil {
t.Fatal(err)
}
} else {
t.Fatal(err)
}
}
return table
}
func TestTable_AddChain(t *testing.T) {
var table = getIPv4Table(t)
{
chain, err := table.AddChain("test_default_chain", nil)
if err != nil {
t.Fatal(err)
}
t.Log("created:", chain.Name())
}
{
chain, err := table.AddAcceptChain("test_accept_chain")
if err != nil {
t.Fatal(err)
}
t.Log("created:", chain.Name())
}
// Do not test drop chain before adding accept rule, you will drop yourself!!!!!!!
/**{
chain, err := table.AddDropChain("test_drop_chain")
if err != nil {
t.Fatal(err)
}
t.Log("created:", chain.Name())
}**/
}
func TestTable_GetChain(t *testing.T) {
var table = getIPv4Table(t)
for _, chainName := range []string{"not_found_chain", "test_default_chain"} {
chain, err := table.GetChain(chainName)
if err != nil {
if err == nftables.ErrChainNotFound {
t.Log(chainName, ":", "not found")
} else {
t.Fatal(err)
}
} else {
t.Log(chainName, ":", chain)
}
}
}
func TestTable_DeleteChain(t *testing.T) {
var table = getIPv4Table(t)
err := table.DeleteChain("test_default_chain")
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestTable_AddSet(t *testing.T) {
var table = getIPv4Table(t)
{
set, err := table.AddSet("ipv4_black_set", &nftables.SetOptions{
HasTimeout: false,
KeyType: nftables.TypeIPAddr,
})
if err != nil {
t.Fatal(err)
}
t.Log(set.Name())
}
{
set, err := table.AddSet("ipv6_black_set", &nftables.SetOptions{
HasTimeout: true,
//Timeout: 3600 * time.Second,
KeyType: nftables.TypeIP6Addr,
})
if err != nil {
t.Fatal(err)
}
t.Log(set.Name())
}
}
func TestTable_GetSet(t *testing.T) {
var table = getIPv4Table(t)
for _, setName := range []string{"not_found_set", "ipv4_black_set"} {
set, err := table.GetSet(setName)
if err != nil {
if err == nftables.ErrSetNotFound {
t.Log(setName, ": not found")
} else {
t.Fatal(err)
}
} else {
t.Log(setName, ":", set)
}
}
}
func TestTable_DeleteSet(t *testing.T) {
var table = getIPv4Table(t)
err := table.DeleteSet("ipv4_black_set")
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestTable_Flush(t *testing.T) {
var table = getIPv4Table(t)
err := table.Flush()
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}

Some files were not shown because too many files have changed in this diff Show More