Compare commits

...

237 Commits

Author SHA1 Message Date
刘祥超
a8c8d80e3b 尝试根据端口号自动纠正源站地址中的scheme 2023-06-18 18:05:28 +08:00
刘祥超
c43b6b37ea 优化代码 2023-06-18 10:01:22 +08:00
刘祥超
ac2d57d2f1 同时设置Websocket允许来源域和防盗链时,以Websocket设置为优先 2023-06-16 09:56:37 +08:00
刘祥超
83ac62cda3 缓存条件增加"强制返回区间内容"选项 2023-06-15 15:14:06 +08:00
刘祥超
c0909a2cd0 部分WAF动作输出内容时增加自定义报头 2023-06-12 18:07:07 +08:00
刘祥超
a73b9f2674 版本号改为1.2.0 2023-06-12 14:43:07 +08:00
刘祥超
3e79b71afc WAF在输出内容时也加入自定义的响应报头 2023-06-11 10:46:20 +08:00
刘祥超
d3caccbb55 上传日志时检查节点ID是否为0 2023-06-10 16:47:27 +08:00
刘祥超
f95bac8d38 在Linux上不通过交叉编译器编译时,也可以支持边缘脚本(在有商业版本源码的情况下) 2023-06-10 15:16:06 +08:00
刘祥超
41d2ab728b 手动发送数据(Send()方法)时也可以使用HTTP Header策略等 2023-06-09 14:49:32 +08:00
刘祥超
b319061e85 优化OSS相关代码 2023-06-08 17:47:04 +08:00
刘祥超
99b8686a49 修复部分测试用例 2023-06-07 21:49:42 +08:00
刘祥超
fe8c5b505a 修复一处编译问题 2023-06-07 20:30:52 +08:00
刘祥超
f88d0982ed 修复User-Agent为空时,使用了默认的Go-http-client/1.1的问题 2023-06-07 20:17:07 +08:00
刘祥超
a9389d53e1 优化代码 2023-06-07 19:30:51 +08:00
刘祥超
fc4b45fec7 HTTP服务反向代理时只把HTTP(S)源站加入到状态管理中 2023-06-07 19:28:16 +08:00
刘祥超
9b22e6cf69 初步实现对象存储源站 2023-06-07 17:27:55 +08:00
刘祥超
7bd7f7da45 允许在集群设置 -- “网站设置” 中设置节点IP访问显示的内容 2023-06-05 19:28:01 +08:00
刘祥超
b68e6517df 网站全局设置增加“强制Ln请求“选项 2023-06-05 17:06:03 +08:00
刘祥超
bbae229d08 优化Ln连接性能 2023-06-05 16:38:29 +08:00
刘祥超
c73a6cbfe8 节点监控数据增加UDP数据报速率 2023-06-04 09:58:43 +08:00
刘祥超
e869e8e4d6 缓存写入Header时忽略Strict-Transport-Security和Alt-Svc 2023-06-02 15:23:54 +08:00
刘祥超
bde4e8507f 优化代码 2023-06-02 14:23:54 +08:00
刘祥超
5ae25cffa0 连接列表增加udp支持 2023-06-02 10:54:17 +08:00
刘祥超
44b721d0d3 优化代码 2023-06-01 19:40:15 +08:00
刘祥超
a2d6b7e0a8 初步实现HTTP3 2023-06-01 17:49:06 +08:00
刘祥超
95d65481e3 优化代码 2023-05-29 20:39:08 +08:00
刘祥超
71522243b5 WAF增加“跳转”动作 2023-05-28 17:11:33 +08:00
刘祥超
953533d9a3 版本号改为1.1.0 2023-05-28 16:07:09 +08:00
刘祥超
31509392c9 Update http_request_host_redirect.go 2023-05-27 21:01:55 +08:00
刘祥超
dc30469b2c WAF区域封禁增加文字提示 2023-05-27 20:32:03 +08:00
刘祥超
fa2903ebb9 解析正则表达式关键词时限制组合的关键词数量不超过32个 2023-05-27 11:40:19 +08:00
刘祥超
c43387bf6a WAF国家/地区封禁、省份封禁增加例外URL、限制URL 2023-05-25 12:02:40 +08:00
刘祥超
47f5cbeac9 网站全局设置中增加“自动匹配证书”选项 2023-05-24 17:20:52 +08:00
刘祥超
2aea68fac4 某个网站找不到证书的情况下不再自动匹配证书 2023-05-24 17:00:27 +08:00
刘祥超
9f7e6559d3 实现集群CC防护策略设置 2023-05-23 19:17:13 +08:00
刘祥超
e352e3125c 实现集群自定义页面 2023-05-22 17:31:26 +08:00
刘祥超
6b5e93f5ad 更新UserAgent分析库 2023-05-20 11:52:27 +08:00
刘祥超
cea5ae9c6c 优化UserAgent内存 2023-05-20 10:38:17 +08:00
刘祥超
80b2bbf1eb 修复UserAgent分析可能产生的死锁 2023-05-19 20:11:53 +08:00
刘祥超
c87f25d7bf HTTP Header中支持设置非标Header 2023-05-19 19:52:04 +08:00
刘祥超
82aff804a6 ${response.header.NAME}变量中的NAME可以是非标准格式 2023-05-19 18:10:20 +08:00
刘祥超
973e67acdb ${header.NAME}变量中的NAME可以时非标准格式 2023-05-19 17:50:39 +08:00
刘祥超
93bd9daf76 HTTP Header - CORS跨域设置增加多个选项 2023-05-19 16:34:41 +08:00
刘祥超
d8c121662c CORS默认允许的请求方法增加PATCH 2023-05-19 15:41:47 +08:00
刘祥超
5f33249c5d 监控数据增加整体流量数据 2023-05-17 10:48:59 +08:00
刘祥超
1beafc9976 防盗链增加”同时检查Origin选项“ 2023-05-02 17:06:24 +08:00
刘祥超
b3857adc0f 优化代码 2023-04-26 08:35:07 +08:00
刘祥超
e75010692c 加快ttlcache GC速度 2023-04-25 19:10:37 +08:00
刘祥超
f2df4a1560 完善测试用例 2023-04-25 17:38:59 +08:00
刘祥超
7a4b68de97 增加ttlcache默认最大容量 2023-04-25 17:24:05 +08:00
刘祥超
6fce430469 更新相关库 2023-04-24 21:07:42 +08:00
刘祥超
8856094dd1 更新go.mod 2023-04-24 21:03:48 +08:00
刘祥超
20bee16d28 版本号修改为1.0.4 2023-04-24 10:17:55 +08:00
刘祥超
b1cd971a21 缓存索引数据库加载失败时自动尝试重建数据库文件 2023-04-21 17:38:31 +08:00
刘祥超
d976a39711 环路(127.0.0.1)请求也统计带宽 2023-04-21 15:08:44 +08:00
刘祥超
accd0236ea 优化统计相关代码 2023-04-19 22:15:57 +08:00
刘祥超
fc401a1426 限制单次处理的服务城市数量 2023-04-19 22:06:39 +08:00
刘祥超
7e8c09a684 优化nftables集合元素过期时间判断 2023-04-19 13:20:31 +08:00
刘祥超
37ddff86f1 优化IP名单同步速度 2023-04-19 12:01:02 +08:00
刘祥超
4dc25fb71e 优化请求统计 2023-04-18 15:07:22 +08:00
刘祥超
42883fbe22 优化可用内存检查 2023-04-11 18:51:56 +08:00
刘祥超
a88d9a07be 版本号改为1.1.0 2023-04-11 18:51:47 +08:00
刘祥超
544f1e482a 版本号修改为1.0.1 2023-04-10 09:18:45 +08:00
刘祥超
f53d4c8951 更好地从访问日志中删除非UTF-8字符内容 2023-04-09 17:50:54 +08:00
刘祥超
70d8aa5b33 版本号改为1.0.0 2023-04-09 17:17:49 +08:00
刘祥超
1aa4be9000 优化代码 2023-04-08 13:49:41 +08:00
刘祥超
a7c7c73f70 优化代码:使用fasttime取代以往的utils.UnixTime 2023-04-08 12:47:04 +08:00
刘祥超
0b441021d8 URL跳转时默认对搜索引擎访问使用301,以提升SEO效果 2023-04-07 19:19:53 +08:00
刘祥超
7db0c8cf62 优化HTTP2、HTTP跳转 2023-04-07 15:09:06 +08:00
刘祥超
6da9cb6dcf 从文件恢复带宽数据时跳过非今天的数据 2023-04-07 11:32:40 +08:00
刘祥超
0af580eb26 优化统计性能 2023-04-07 11:23:37 +08:00
刘祥超
52085bdc1c 增加测试用例 2023-04-07 10:52:09 +08:00
刘祥超
72f1eea721 提供批量更新服务配置API(阶段性提交) 2023-04-06 20:50:34 +08:00
刘祥超
6d52b022b2 访问不存在的域名加入到黑名单时,只对当前节点有效 2023-04-05 19:42:14 +08:00
刘祥超
ea41c9b0b3 健康检查时不记录统计 2023-04-05 16:44:26 +08:00
刘祥超
ed6127c2bb 更好地处理访问日志中的非UTF-8字节 2023-04-05 15:54:07 +08:00
刘祥超
b6d95a84fc 进程退出时停止上传带宽数据 2023-04-05 11:34:15 +08:00
刘祥超
c71e68bdea 优化nftables查找程序 2023-04-05 09:33:03 +08:00
刘祥超
c44583f249 优化IP黑名单检测 2023-04-05 09:25:33 +08:00
刘祥超
c53773c2db 可以只更新UAM策略变化 2023-04-03 16:12:14 +08:00
刘祥超
793994a3fe 在节点启动时自动调整系统内核参数 2023-04-02 21:30:03 +08:00
刘祥超
4c3deb1156 将nftables黑名单扩展到5个 2023-04-02 20:32:36 +08:00
刘祥超
24ca5a5ace 优化nftables可执行文件查找方法 2023-04-02 18:37:24 +08:00
刘祥超
8bbbf57827 重启服务前先将带宽统计数据缓存,以便于在重启后继续使用 2023-04-02 17:24:55 +08:00
刘祥超
888df02d0c 优化IP名单上传程序 2023-04-01 20:51:49 +08:00
刘祥超
8988765cef 修复在高并发下修改服务配置可能导致服务崩溃(panic)的问题 2023-04-01 18:41:50 +08:00
刘祥超
f675b88761 nftables:自动升级以前的drop规则为reject规则 2023-04-01 17:09:53 +08:00
刘祥超
9bd4975478 创建nftables规则时,使用REJECT代替DROP 2023-04-01 15:55:24 +08:00
刘祥超
95abb7bfae 集群服务设置增加“支持低版本HTTP”选项/分片内容提示不支持低版本HTTP协议 2023-04-01 09:57:24 +08:00
刘祥超
d9fa3dcc3b 优化WAF黑名单处理 2023-03-31 21:37:15 +08:00
刘祥超
964524816f 支持商业版本状态查询 2023-03-31 14:06:01 +08:00
刘祥超
d124c9be18 增加注释 2023-03-29 20:09:19 +08:00
刘祥超
1a05402076 增加固定Map定义 2023-03-23 10:52:52 +08:00
刘祥超
c4b1790102 上传流量统计数据时同时上传用户ID 2023-03-22 19:51:36 +08:00
刘祥超
613acbff95 限制单个服务每次上传的域名统计数不超过20个 2023-03-22 19:05:10 +08:00
刘祥超
e6ab98ad11 上传带宽信息时同时缓存流量、统计流量、请求数等参数,为将来的流量和带宽合并做准备 2023-03-22 18:00:35 +08:00
刘祥超
1121869f14 增加网站服务加载和删除调试日志 2023-03-19 11:05:03 +08:00
刘祥超
91efe57e1b 不提示单个端口Reload信息,防止不重要的日志过多 2023-03-19 11:00:27 +08:00
刘祥超
95f2573263 增加RPC消息最大尺寸到512MB 2023-03-18 22:43:48 +08:00
刘祥超
09aa85f51c 节点组合配置时服务间可以共用证书数据 2023-03-18 22:18:48 +08:00
刘祥超
c6279a1076 版本号更改为0.6.5 2023-03-17 15:54:44 +08:00
刘祥超
47ccb64cfb 优化日志提示 2023-03-16 17:24:55 +08:00
刘祥超
5c218567e1 在GET302和CAPTCHA验证中不记录特殊URL的访问日志 2023-03-16 10:38:40 +08:00
刘祥超
c161d84fdf 版本号变更为0.6.4.2 2023-03-16 08:59:46 +08:00
刘祥超
495b553285 修复存储空间统计可能为负值的问题 2023-03-16 08:59:35 +08:00
刘祥超
21b770ba8b 修复运行测试用例时init()无法起作用的Bug 2023-03-13 21:49:41 +08:00
刘祥超
e9f94e0767 改进README.md 2023-03-13 08:56:18 +08:00
刘祥超
644ada1da9 优化代码 2023-03-12 20:32:15 +08:00
刘祥超
0c40250849 优化代码 2023-03-12 16:36:59 +08:00
刘祥超
1d1134a86d 优化代码 2023-03-12 16:18:16 +08:00
刘祥超
28e7664eb7 修复源站重试时可能产生的file is writing错误 2023-03-12 16:09:06 +08:00
刘祥超
50f3ad641c 优化统计相关程序 2023-03-12 12:19:25 +08:00
刘祥超
cc7cf5f8c5 限制统计系统和浏览器的最大长度,减少随机UserAgent攻击影响 2023-03-11 11:58:05 +08:00
刘祥超
339f0f6e94 上传统计数据时增加单次上传数量限制 2023-03-11 11:21:03 +08:00
刘祥超
f558e43342 根据系统内存调整访问日志队列长度 2023-03-10 22:31:40 +08:00
刘祥超
e374e5c90c 优化命令识别 2023-03-10 22:05:40 +08:00
刘祥超
563b775e49 优化命令执行速度 2023-03-10 22:01:39 +08:00
刘祥超
de9e1a4515 执行一般命令时不运行init()中内容 2023-03-10 15:14:14 +08:00
刘祥超
f64b36f17a WAF cc防护增加“检查请求来源指纹”选项 2023-03-10 10:41:16 +08:00
刘祥超
f0e8c82d31 增加CC防护(开源用户需要自己实现) 2023-03-09 15:15:22 +08:00
刘祥超
5770d43230 WAF cc2尝试使用指纹统计方法 2023-03-08 16:59:44 +08:00
刘祥超
d4944c236f 修复几处测试用例 2023-03-08 10:13:41 +08:00
刘祥超
33c761a187 修复本地数据库无法异步提交事务的Bug 2023-03-07 16:48:03 +08:00
刘祥超
d7e6da8d2c 对本地数据库文件进行加锁 2023-03-07 16:22:32 +08:00
刘祥超
44d1a2415c 集群服务设置增加“记录找不到网站日志”选项 2023-03-07 10:30:55 +08:00
刘祥超
c98ff50f06 暂时取消GET302和POST307的迟滞 2023-03-07 08:51:39 +08:00
刘祥超
8835fcb09e GET302/POST307增加访问迟滞 2023-03-06 16:28:54 +08:00
刘祥超
77c56e58c0 GET302/POST307兼容safari浏览器 2023-03-06 16:27:06 +08:00
刘祥超
72c65ca4ee 修复GET302和POST307关闭连接后无法响应的问题 2023-03-06 16:10:58 +08:00
刘祥超
ab019b0bdc 修复在连接读写优化模式下fastcgi无法正常工作的Bug 2023-03-06 10:33:54 +08:00
刘祥超
9709e45ad2 修改软内存限制设置错误 2023-03-05 21:19:10 +08:00
刘祥超
be1f80003c 增加软内存最大值限制 2023-03-05 12:34:46 +08:00
刘祥超
252fcca383 增加测试用例 2023-03-03 14:28:58 +08:00
刘祥超
04ae8fa4a0 系统内存不足时,尝试自动回收内存 2023-03-02 10:54:25 +08:00
刘祥超
c95bd7776a WAF拦截动作可以设置最大封禁时间,从而实现封禁时间随机 2023-03-01 19:00:08 +08:00
刘祥超
8219167d05 WAF支持忽略全局WAF规则 2023-03-01 16:46:43 +08:00
刘祥超
e0a6881343 上传带宽数据时同时上传流量数据 2023-02-27 10:48:16 +08:00
刘祥超
6e985d7f06 修复GET302和POST307无限循环的问题 2023-02-24 17:02:59 +08:00
刘祥超
66719b05dd 修复WAF验证码不能输入超出6位数字的Bug 2023-02-16 14:44:56 +08:00
刘祥超
7197583fea 增加变量${requestPathLowerExtension} 2023-02-10 10:43:30 +08:00
刘祥超
ce29024eef 写入Agent IP时,不提示id重复错误 2023-02-01 10:18:08 +08:00
刘祥超
e1ac67f7fa 版本更改为0.6.4 2023-02-01 10:07:30 +08:00
刘祥超
01812144dd 优化带宽统计 2023-01-12 19:09:57 +08:00
刘祥超
1c34e49629 优化代码 2023-01-12 19:02:38 +08:00
刘祥超
f233fbfb25 版本号修改为0.6.3 2023-01-11 15:44:53 +08:00
刘祥超
5387115e4a 优化代码 2023-01-11 15:24:48 +08:00
刘祥超
d82c03db23 修复在HTTPS下无法连接Websocket的问题 2023-01-10 21:20:27 +08:00
刘祥超
230c5c3766 版本号修改为0.6.2 2023-01-10 21:18:53 +08:00
刘祥超
927425149e 优化代码 2023-01-10 09:47:56 +08:00
刘祥超
5ce1aab92c 修复域名跳转时没有携带参数的Bug 2023-01-09 20:06:09 +08:00
刘祥超
195742bb26 修复读超时时间(ReadDeadline)导致WAFGET302、POST307延时关闭连接的问题 2023-01-09 15:56:59 +08:00
刘祥超
006cc2912d 版本修改为0.6.1 2023-01-09 15:49:16 +08:00
刘祥超
2d4ba90c3b 改进在自动读超时模式下的Websocket连接 2023-01-09 12:36:33 +08:00
刘祥超
a2e6aaaa18 WAF增加“在IP列表内”操作符/优化部分操作符代号 2023-01-08 10:15:46 +08:00
刘祥超
8e68da7725 集群服务设置增加自动读超时选项 2023-01-07 20:04:05 +08:00
刘祥超
7abb84c880 优化网络连接关闭速度 2023-01-07 10:03:32 +08:00
刘祥超
a17878f5b2 WAF增加包含任一字符串、包含所有字符串操作符 2023-01-06 20:07:15 +08:00
刘祥超
8a8881ac47 IP范围支持多行 2023-01-06 19:14:09 +08:00
刘祥超
c567404b7a 优化连接相关代码 2023-01-05 11:13:35 +08:00
刘祥超
b220b0f48e 优化读取HTTP请求Header和握手超时时间 2023-01-05 00:40:49 +08:00
刘祥超
9609c90d75 边缘节点增加数据读超时,以改进客户端上传数据过慢的问题 2023-01-04 20:43:10 +08:00
刘祥超
2c3c32af5b 优化代码 2023-01-02 10:44:10 +08:00
刘祥超
b4a4b2e9b1 集群服务设置中增加性能设置 2023-01-01 19:27:38 +08:00
刘祥超
c42ff1e1e9 实现UA名单功能 2022-12-30 20:49:43 +08:00
刘祥超
9fed1141c2 默认情况下内容压缩不支持Partial Content 2022-12-30 11:44:07 +08:00
刘祥超
e87f031293 增加CORS自适应跨域 2022-12-29 17:16:42 +08:00
刘祥超
c4bac7f43c 优化代码 2022-12-27 18:58:29 +08:00
刘祥超
47818f972e 自动转换访问域名中的大写字母 2022-12-25 15:23:56 +08:00
刘祥超
218a0300c5 修复测试用例 2022-12-23 18:53:49 +08:00
刘祥超
63f6c4177f 修复测试用例 2022-12-23 18:17:32 +08:00
刘祥超
1830c22a31 增加自动Agent识别 2022-12-22 11:38:59 +08:00
刘祥超
18611e8a7c 写数据超时时断开同客户端连接 2022-12-21 16:11:55 +08:00
刘祥超
c45f7adf04 优化连接相关代码 2022-12-21 15:59:07 +08:00
刘祥超
1a200918a8 不支持CONNECT方法 2022-12-19 16:27:58 +08:00
刘祥超
b942bb776e 国家/地区封禁、省份封禁时支持IP变量 2022-12-18 16:04:12 +08:00
刘祥超
5cf84efccd 优化内容为空的缓存 2022-12-14 15:26:18 +08:00
刘祥超
ebb6ebd10c 修复WAF中反斜杠符号(\)有可能解析错误的Bug 2022-12-14 12:27:07 +08:00
刘祥超
42d0d63cf4 优化代码 2022-12-13 18:08:50 +08:00
刘祥超
96f8f7e925 增加edge-node ip.close IP命令 2022-12-12 19:23:58 +08:00
刘祥超
e7e7214d58 调整慢连接超时算法 2022-12-12 10:04:36 +08:00
刘祥超
ade979a725 向客户端写入数据超时时立即关闭连接 2022-12-10 19:51:05 +08:00
刘祥超
60a8de13e7 TCP单次向客户端写入数据时超过30秒即认为超时 2022-12-10 18:22:00 +08:00
刘祥超
9fa24bed0a 修复WAF记录IP动作时无法不超时的Bug 2022-12-06 11:01:34 +08:00
刘祥超
87bc1a7e03 优化OpenFileCache 2022-12-05 11:16:04 +08:00
刘祥超
1a05f56149 优化缓存相关代码 2022-12-05 10:46:44 +08:00
刘祥超
f88db576e1 优化代码 2022-12-05 09:57:01 +08:00
刘祥超
dc3f26ea1a 减少WAF预读尺寸 2022-12-02 21:08:03 +08:00
刘祥超
6fc30144f7 在edge-node conns命令中显示连接时长 2022-12-02 17:03:16 +08:00
刘祥超
25b0b98bd4 增加默认的源站连接数 2022-12-02 10:39:07 +08:00
刘祥超
27b5817d5e 优化请求限制逻辑,连接关闭时自动终止内容发送 2022-11-29 19:14:46 +08:00
刘祥超
dcb61dfd33 版本号更改为0.6.0 2022-11-29 15:42:21 +08:00
刘祥超
bbcfdbbf5e 优化代码 2022-11-29 15:33:12 +08:00
刘祥超
b2a1bef08f 修复服务WAF配置无法更新的Bug 2022-11-28 18:13:08 +08:00
刘祥超
2b18b5c2ca 修改版本号为0.5.9 2022-11-28 18:08:19 +08:00
刘祥超
6ff030dbd8 编译时加入configs/cluster.template.yaml文件 2022-11-27 14:52:48 +08:00
刘祥超
0ddeef6986 支持使用域名中含有通配符清除缓存数据 2022-11-26 11:05:46 +08:00
刘祥超
976bd3600b 优化OpenFileCache功能 2022-11-25 14:52:04 +08:00
刘祥超
a64047a934 优化配置重载程序 2022-11-25 10:50:57 +08:00
刘祥超
e82f207935 统计API调用时低于一半的采样率返回总统计 2022-11-23 20:23:46 +08:00
刘祥超
61b5316a1f 优化代码 2022-11-23 20:13:34 +08:00
刘祥超
82329aa8b0 修复一处编译错误 2022-11-22 18:40:03 +08:00
刘祥超
7dabd9c19c 在监控系统运行时上报API连接状况 2022-11-22 11:23:39 +08:00
刘祥超
9437acd18c 优化代码 2022-11-21 21:08:47 +08:00
刘祥超
9da7a34edf 节点可以单独设置所使用的API节点地址 2022-11-21 19:55:28 +08:00
刘祥超
b6a5491dcc 优化Partial Content兼容性 2022-11-20 18:07:46 +08:00
刘祥超
bcee658567 优化Partial Content配置编码速度 2022-11-19 23:11:05 +08:00
刘祥超
afc8f7b703 优化Partial Content缓存 2022-11-19 21:20:53 +08:00
刘祥超
7a4b89d2fb 缓存Header中忽略Set-Cookie 2022-11-19 17:35:23 +08:00
刘祥超
c6299a2fb0 减少文件缓存写入次数 2022-11-19 17:23:45 +08:00
刘祥超
8b5d74af9b 进一步提升文件缓存写入速度 2022-11-19 15:55:05 +08:00
刘祥超
a194360a56 Update go.mod 2022-11-18 17:39:49 +08:00
刘祥超
b12f7f69ba 优化代码 2022-11-17 20:28:55 +08:00
刘祥超
06ec4d3fba 优化代码 2022-11-17 10:38:20 +08:00
刘祥超
c209ab912f 优化代码 2022-11-17 10:35:43 +08:00
刘祥超
32720d772d 优化代码 2022-11-17 10:32:26 +08:00
刘祥超
a89c02fd10 请求变量增加${cname},WAF checkpoint增加cname和isCNAME 2022-11-16 15:01:10 +08:00
刘祥超
37ef86b92f 接收HTTP请求时去除域名后面的点符号 2022-11-16 11:25:11 +08:00
刘祥超
4c19c37f49 写入缓存时减少对缓存目录的检查频率 2022-11-15 22:25:49 +08:00
刘祥超
1bb818b5b0 边缘节点支持设置多个缓存目录 2022-11-15 20:42:25 +08:00
刘祥超
825e46458f 优化代码 2022-11-15 10:06:57 +08:00
刘祥超
a42737bd28 缩短节点运行日志队列长度 2022-11-14 16:42:50 +08:00
刘祥超
5f76be2cfd 优化代码 2022-11-13 10:32:12 +08:00
刘祥超
dbddf8a91a 优化代码 2022-11-08 21:37:20 +08:00
刘祥超
6c457f41f6 优化代码 2022-11-08 20:58:17 +08:00
刘祥超
e4b2a650f0 优化代码 2022-11-08 20:19:51 +08:00
刘祥超
913ba95801 优化缓存相关代码 2022-11-08 11:03:37 +08:00
刘祥超
a9f8e39703 修复节点设置的“缓存磁盘容量”不起作用的问题 2022-11-07 21:32:20 +08:00
刘祥超
534f013f59 使用版本号来读取节点任务,提升任务同步稳定性 2022-11-06 12:07:26 +08:00
刘祥超
258380f75c 修复无法回报任务执行失败的问题 2022-11-05 14:56:57 +08:00
刘祥超
8c0e51ec46 域名跳转增加是否忽略端口选项 2022-11-05 14:30:29 +08:00
刘祥超
4c37c7ab84 时钟同步增加是否检查chrony选项 2022-11-03 14:59:26 +08:00
刘祥超
f005da1d5f 防盗链提示增加缓存时间,以提升性能 2022-11-02 15:24:30 +08:00
刘祥超
e99acc4694 版本修改为0.5.8 2022-11-02 15:11:55 +08:00
刘祥超
408357dfcf 优化DDoS防护相关错误提示信息 2022-11-01 17:37:40 +08:00
刘祥超
0109a27c06 上传访问日志发生网络错误时不提交 2022-11-01 14:55:06 +08:00
刘祥超
e6e2dccc42 版本号修改为0.5.7 2022-10-31 19:14:03 +08:00
243 changed files with 8098 additions and 2064 deletions

1
.gitignore vendored
View File

@@ -1,2 +1,3 @@
*_plus.go
*_plus_test.go
*-plus.sh

View File

@@ -50,11 +50,12 @@ function build() {
fi
cp "$ROOT"/configs/api.template.yaml "$DIST"/configs
cp "$ROOT"/configs/cluster.template.yaml "$DIST"/configs
cp -R "$ROOT"/www "$DIST"/
cp -R "$ROOT"/pages "$DIST"/
# we support TOA on linux/amd64 only
if [ "$OS" == "linux" -a "$ARCH" == "amd64" ]
if [ "$OS" == "linux" ] && [ "$ARCH" == "amd64" ]
then
cp -R "$ROOT"/edge-toa "$DIST"
fi
@@ -113,7 +114,10 @@ function build() {
if [ ! -z $CC_PATH ]; then
env CC=$MUSL_DIR/$CC_PATH CXX=$MUSL_DIR/$CXX_PATH GOOS="${OS}" GOARCH="${ARCH}" CGO_ENABLED=1 go build -trimpath -tags $BUILD_TAG -o "$DIST"/bin/${NAME} -ldflags "-linkmode external -extldflags -static -s -w" "$ROOT"/../cmd/edge-node/main.go
else
env GOOS="${OS}" GOARCH="${ARCH}" CGO_ENABLED=1 go build -trimpath -tags $TAG -o "$DIST"/bin/${NAME} -ldflags="-s -w" "$ROOT"/../cmd/edge-node/main.go
if [[ `uname` == *"Linux"* ]] && [ "$OS" == "linux" ] && [[ "$ARCH" == "amd64" || "$ARCH" == "arm64" ]] && [ "$TAG" == "plus" ]; then
BUILD_TAG="plus,script"
fi
env GOOS="${OS}" GOARCH="${ARCH}" CGO_ENABLED=1 go build -trimpath -tags $BUILD_TAG -o "$DIST"/bin/${NAME} -ldflags="-s -w" "$ROOT"/../cmd/edge-node/main.go
fi
# delete hidden files

View File

@@ -1 +1,2 @@
* `global.yaml` - 全局配置
* `api.template.yaml` - API相关配置模板
* `cluster.template.yaml` - 通过集群自动接入节点模板

9
build/test.sh Executable file
View File

@@ -0,0 +1,9 @@
#!/usr/bin/env bash
TAG=${1}
if [ -z "$TAG" ]; then
TAG="community"
fi
go test -v ../... -tags=${TAG}

View File

@@ -25,7 +25,7 @@ func main() {
Product(teaconst.ProductName).
Usage(teaconst.ProcessName + " [-v|start|stop|restart|status|quit|test|reload|service|daemon|pprof|accesslog]").
Usage(teaconst.ProcessName + " [trackers|goman|conns|gc]").
Usage(teaconst.ProcessName + " [ip.drop|ip.reject|ip.remove] IP")
Usage(teaconst.ProcessName + " [ip.drop|ip.reject|ip.remove|ip.close] IP")
app.On("test", func() {
err := nodes.NewNode().Test()
@@ -241,6 +241,38 @@ func main() {
}
}
})
app.On("ip.close", func() {
var args = os.Args[2:]
if len(args) == 0 {
fmt.Println("Usage: edge-node ip.close IP")
return
}
var ip = args[0]
if len(net.ParseIP(ip)) == 0 {
fmt.Println("IP '" + ip + "' is invalid")
return
}
fmt.Println("close ip '" + ip)
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{
Code: "closeIP",
Params: map[string]any{
"ip": ip,
},
})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
var errString = maps.NewMap(reply.Params).GetString("error")
if len(errString) > 0 {
fmt.Println("[ERROR]" + errString)
} else {
fmt.Println("ok")
}
}
})
app.On("ip.remove", func() {
var args = os.Args[2:]
if len(args) == 0 {

33
go.mod
View File

@@ -5,6 +5,7 @@ go 1.18
replace (
github.com/TeaOSLab/EdgeCommon => ../EdgeCommon
github.com/fsnotify/fsnotify => github.com/iwind/fsnotify v1.5.2-0.20220817040843-193be2051ff4
github.com/google/nftables => github.com/iwind/nftables v0.0.0-20230419014751-9f023a644ad4
)
require (
@@ -16,49 +17,43 @@ require (
github.com/fsnotify/fsnotify v1.5.1
github.com/go-redis/redis/v8 v8.11.5
github.com/golang/protobuf v1.5.2
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6
github.com/iwind/TeaGo v0.0.0-20220807030847-31de8e1cbe55
github.com/google/nftables v0.1.0
github.com/iwind/TeaGo v0.0.0-20230304012706-c1f4a4e27470
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11
github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4
github.com/iwind/gowebp v0.0.0-20220819053541-c235395277b5
github.com/klauspost/compress v1.15.8
github.com/klauspost/compress v1.16.5
github.com/mattn/go-sqlite3 v1.14.9
github.com/mdlayher/netlink v1.4.2
github.com/mdlayher/netlink v1.7.1
github.com/miekg/dns v1.1.43
github.com/mssola/user_agent v0.5.3
github.com/mssola/useragent v1.0.0
github.com/pires/go-proxyproto v0.6.1
github.com/shirou/gopsutil/v3 v3.22.2
golang.org/x/image v0.0.0-20220722155232-062f8c9fd539
golang.org/x/net v0.0.0-20220225172249-27dd8689420f
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab
golang.org/x/text v0.3.7
golang.org/x/image v0.7.0
golang.org/x/net v0.8.0
golang.org/x/sys v0.6.0
golang.org/x/text v0.9.0
google.golang.org/grpc v1.45.0
gopkg.in/yaml.v3 v3.0.1
rogchap.com/v8go v0.7.0
)
require (
github.com/BurntSushi/toml v0.4.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/chai2010/webp v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-sql-driver/mysql v1.5.0 // indirect
github.com/google/go-cmp v0.5.7 // indirect
github.com/josharian/native v0.0.0-20200817173448-b6b71def0850 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/josharian/native v1.0.0 // indirect
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e // indirect
github.com/kr/text v0.2.0 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/mdlayher/socket v0.0.0-20211102153432-57e3fa563ecb // indirect
github.com/mdlayher/socket v0.4.0 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/tklauser/go-sysconf v0.3.9 // indirect
github.com/tklauser/numcpus v0.3.0 // indirect
github.com/yusufpapurcu/wmi v1.2.2 // indirect
golang.org/x/mod v0.5.1 // indirect
golang.org/x/tools v0.1.8 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
golang.org/x/sync v0.1.0 // indirect
google.golang.org/genproto v0.0.0-20220317150908-0efb43f6373e // indirect
google.golang.org/protobuf v1.27.1 // indirect
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
honnef.co/go/tools v0.2.2 // indirect
)

166
go.sum
View File

@@ -1,8 +1,6 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/toml v0.4.1 h1:GaI7EiDXDRfa8VshkTj7Fym7ha+y8/XxIgD2okUIjLw=
github.com/BurntSushi/toml v0.4.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY=
@@ -18,8 +16,6 @@ github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cb
github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chai2010/webp v1.1.1 h1:jTRmEccAJ4MGrhFOrPMpNGIJ/eybIgwKpcACsrTEapk=
github.com/chai2010/webp v1.1.1/go.mod h1:0XVwvZWdjjdxpUEIf7b9g9VkHFnInUSYujwqTLEuldU=
github.com/cilium/ebpf v0.5.0/go.mod h1:4tRaxcgiL706VnOzHOdBlY8IEAIdxINsQBcU4xJJXRs=
github.com/cilium/ebpf v0.7.0/go.mod h1:/oI2+1shJiTGAMgl6/RgJr36Eo1jzrRcAWbcXO2usCA=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
@@ -28,9 +24,8 @@ github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWH
github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f h1:q/DpyjJjZs94bziQ7YkBmIlpqbVP7yw179rnzoNVX1M=
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f/go.mod h1:QGrK8vMWWHQYQ3QU9bw9Y9OPNfxccGzfb41qjvVeXtY=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
@@ -41,14 +36,11 @@ github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1m
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -70,18 +62,15 @@ github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMyw
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE=
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6 h1:btadZscaRmsi/+fOhkyUguRpSnrf6dykNEWxDeUCj9I=
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6/go.mod h1:0F8on3JWMkm+xahTHItkiu/E1SPqMd0TOxNweQv8ptE=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/iwind/TeaGo v0.0.0-20220807030847-31de8e1cbe55 h1:shQNx0flJFBwKsGE7Hs3bI2bDz+YF0zl/4qE8B2KRiY=
github.com/iwind/TeaGo v0.0.0-20220807030847-31de8e1cbe55/go.mod h1:fi/Pq+/5m2HZoseM+39dMF57ANXRt6w4PkGu3NXPc5s=
github.com/iwind/TeaGo v0.0.0-20230304012706-c1f4a4e27470 h1:TuRxvKRv9PxKVijWOkUnZm5TeanQqWGUJyPx9u6cra4=
github.com/iwind/TeaGo v0.0.0-20230304012706-c1f4a4e27470/go.mod h1:fi/Pq+/5m2HZoseM+39dMF57ANXRt6w4PkGu3NXPc5s=
github.com/iwind/fsnotify v1.5.2-0.20220817040843-193be2051ff4 h1:PKtXlgNHJhdwl5ozio7KRV3n0SckMw+8ZC2NCpRSv8U=
github.com/iwind/fsnotify v1.5.2-0.20220817040843-193be2051ff4/go.mod h1:DmAukmDY25inGlriLn0B2jidmvaDR1REOcPXwvzvDPI=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11 h1:DaQjoWZhLNxjhIXedVg4/vFEtHkZhK4IjIwsWdyzBLg=
@@ -90,22 +79,14 @@ github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4 h1:VWGsCqTzObdlbf7UUE
github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4/go.mod h1:H5Q7SXwbx3a97ecJkaS2sD77gspzE7HFUafBO0peEyA=
github.com/iwind/gowebp v0.0.0-20220819053541-c235395277b5 h1:SgKhrqRhWVpu0ZKxQM8MGjdhy7hlH3Xax8snwsZWSic=
github.com/iwind/gowebp v0.0.0-20220819053541-c235395277b5/go.mod h1:Re7TEhwL+ygnxFg52fC0PWy01ULAIZp2QR0q5WwEOQA=
github.com/josharian/native v0.0.0-20200817173448-b6b71def0850 h1:uhL5Gw7BINiiPAo24A2sxkcDI0Jt/sqp1v5xQCniEFA=
github.com/josharian/native v0.0.0-20200817173448-b6b71def0850/go.mod h1:7X/raswPFr05uY3HiLlYeyQntB6OO7E/d2Cu7qoaN2w=
github.com/jsimonetti/rtnetlink v0.0.0-20190606172950-9527aa82566a/go.mod h1:Oz+70psSo5OFh8DBl0Zv2ACw7Esh6pPUphlvZG9x7uw=
github.com/jsimonetti/rtnetlink v0.0.0-20200117123717-f846d4f6c1f4/go.mod h1:WGuG/smIU4J/54PblvSbh+xvCZmpJnFgr3ds6Z55XMQ=
github.com/jsimonetti/rtnetlink v0.0.0-20201009170750-9c6f07d100c1/go.mod h1:hqoO/u39cqLeBLebZ8fWdE96O7FxrAsRYhnVOdgHxok=
github.com/jsimonetti/rtnetlink v0.0.0-20201216134343-bde56ed16391/go.mod h1:cR77jAZG3Y3bsb8hF6fHJbFoyFukLFOkQ98S0pQz3xw=
github.com/jsimonetti/rtnetlink v0.0.0-20201220180245-69540ac93943/go.mod h1:z4c53zj6Eex712ROyh8WI0ihysb5j2ROyV42iNogmAs=
github.com/jsimonetti/rtnetlink v0.0.0-20210122163228-8d122574c736/go.mod h1:ZXpIyOK59ZnN7J0BV99cZUPmsqDRZ3eq5X+st7u/oSA=
github.com/jsimonetti/rtnetlink v0.0.0-20210212075122-66c871082f2b/go.mod h1:8w9Rh8m+aHZIG69YPGGem1i5VzoyRC8nw2kA8B+ik5U=
github.com/jsimonetti/rtnetlink v0.0.0-20210525051524-4cc836578190/go.mod h1:NmKSdU4VGSiv1bMsdqNALI4RSvvjtz65tTMCnD05qLo=
github.com/jsimonetti/rtnetlink v0.0.0-20211022192332-93da33804786 h1:N527AHMa793TP5z5GNAn/VLPzlc0ewzWdeP/25gDfgQ=
github.com/jsimonetti/rtnetlink v0.0.0-20211022192332-93da33804786/go.mod h1:v4hqbTdfQngbVSZJVWUhGE/lbTFf9jb+ygmNUDQMuOs=
github.com/iwind/nftables v0.0.0-20230419014751-9f023a644ad4 h1:RPAH9Sj9l/20zH5zU5/iJGszfwPq6eLjoiC/n/asulA=
github.com/iwind/nftables v0.0.0-20230419014751-9f023a644ad4/go.mod h1:7OLL+86wZKfBnAJxNxmdcZ0ebbgdp/A28fcagx9oJqA=
github.com/josharian/native v1.0.0 h1:Ts/E8zCSEsG17dUqv7joXJFybuMLjQfWE04tsBODTxk=
github.com/josharian/native v1.0.0/go.mod h1:7X/raswPFr05uY3HiLlYeyQntB6OO7E/d2Cu7qoaN2w=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e h1:LvL4XsI70QxOGHed6yhQtAU34Kx3Qq2wwBzGFKY8zKk=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e/go.mod h1:kLgvv7o6UM+0QSf0QjAse3wReFDsb9qbZJdfexWlrQw=
github.com/klauspost/compress v1.15.8 h1:JahtItbkWjf2jzm/T+qgMxkP9EMHsqEUA6vCMGmXvhA=
github.com/klauspost/compress v1.15.8/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI=
github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
@@ -116,31 +97,14 @@ github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/mattn/go-sqlite3 v1.14.9 h1:10HX2Td0ocZpYEjhilsuo6WWtUqttj2Kb0KtD86/KYA=
github.com/mattn/go-sqlite3 v1.14.9/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mdlayher/ethtool v0.0.0-20210210192532-2b88debcdd43/go.mod h1:+t7E0lkKfbBsebllff1xdTmyJt8lH37niI6kwFk9OTo=
github.com/mdlayher/ethtool v0.0.0-20211028163843-288d040e9d60 h1:tHdB+hQRHU10CfcK0furo6rSNgZ38JT8uPh70c/pFD8=
github.com/mdlayher/ethtool v0.0.0-20211028163843-288d040e9d60/go.mod h1:aYbhishWc4Ai3I2U4Gaa2n3kHWSwzme6EsG/46HRQbE=
github.com/mdlayher/genetlink v1.0.0 h1:OoHN1OdyEIkScEmRgxLEe2M9U8ClMytqA5niynLtfj0=
github.com/mdlayher/genetlink v1.0.0/go.mod h1:0rJ0h4itni50A86M2kHcgS85ttZazNt7a8H2a2cw0Gc=
github.com/mdlayher/netlink v0.0.0-20190409211403-11939a169225/go.mod h1:eQB3mZE4aiYnlUsyGGCOpPETfdQq4Jhsgf1fk3cwQaA=
github.com/mdlayher/netlink v1.0.0/go.mod h1:KxeJAFOFLG6AjpyDkQ/iIhxygIUKD+vcwqcnu43w/+M=
github.com/mdlayher/netlink v1.1.0/go.mod h1:H4WCitaheIsdF9yOYu8CFmCgQthAPIWZmcKp9uZHgmY=
github.com/mdlayher/netlink v1.1.1/go.mod h1:WTYpFb/WTvlRJAyKhZL5/uy69TDDpHHu2VZmb2XgV7o=
github.com/mdlayher/netlink v1.2.0/go.mod h1:kwVW1io0AZy9A1E2YYgaD4Cj+C+GPkU6klXCMzIJ9p8=
github.com/mdlayher/netlink v1.2.1/go.mod h1:bacnNlfhqHqqLo4WsYeXSqfyXkInQ9JneWI68v1KwSU=
github.com/mdlayher/netlink v1.2.2-0.20210123213345-5cc92139ae3e/go.mod h1:bacnNlfhqHqqLo4WsYeXSqfyXkInQ9JneWI68v1KwSU=
github.com/mdlayher/netlink v1.3.0/go.mod h1:xK/BssKuwcRXHrtN04UBkwQ6dY9VviGGuriDdoPSWys=
github.com/mdlayher/netlink v1.4.0/go.mod h1:dRJi5IABcZpBD2A3D0Mv/AiX8I9uDEu5oGkAVrekmf8=
github.com/mdlayher/netlink v1.4.1/go.mod h1:e4/KuJ+s8UhfUpO9z00/fDZZmhSrs+oxyqAS9cNgn6Q=
github.com/mdlayher/netlink v1.4.2 h1:3sbnJWe/LETovA7yRZIX3f9McVOWV3OySH6iIBxiFfI=
github.com/mdlayher/netlink v1.4.2/go.mod h1:13VaingaArGUTUxFLf/iEovKxXji32JAtF858jZYEug=
github.com/mdlayher/socket v0.0.0-20210307095302-262dc9984e00/go.mod h1:GAFlyu4/XV68LkQKYzKhIo/WW7j3Zi0YRAz/BOoanUc=
github.com/mdlayher/socket v0.0.0-20211007213009-516dcbdf0267/go.mod h1:nFZ1EtZYK8Gi/k6QNu7z7CgO20i/4ExeQswwWuPmG/g=
github.com/mdlayher/socket v0.0.0-20211102153432-57e3fa563ecb h1:2dC7L10LmTqlyMVzFJ00qM25lqESg9Z4u3GuEXN5iHY=
github.com/mdlayher/socket v0.0.0-20211102153432-57e3fa563ecb/go.mod h1:nFZ1EtZYK8Gi/k6QNu7z7CgO20i/4ExeQswwWuPmG/g=
github.com/mdlayher/netlink v1.7.1 h1:FdUaT/e33HjEXagwELR8R3/KL1Fq5x3G5jgHLp/BTmg=
github.com/mdlayher/netlink v1.7.1/go.mod h1:nKO5CSjE/DJjVhk/TNp6vCE1ktVxEA8VEh8drhZzxsQ=
github.com/mdlayher/socket v0.4.0 h1:280wsy40IC9M9q1uPGcLBwXpcTQDtoGwVt+BNoITxIw=
github.com/mdlayher/socket v0.4.0/go.mod h1:xxFqz5GRCUN3UEOm9CZqEJsAbe1C8OwSK46NlmWuVoc=
github.com/miekg/dns v1.1.43 h1:JKfpVSCB84vrAmHzyrsxB5NAr5kLoMXZArPSw7Qlgyg=
github.com/miekg/dns v1.1.43/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4=
github.com/mssola/user_agent v0.5.3 h1:lBRPML9mdFuIZgI2cmlQ+atbpJdLdeVl2IDodjBR578=
github.com/mssola/user_agent v0.5.3/go.mod h1:TTPno8LPY3wAIEKRpAtkdMT0f8SE24pLRGPahjCH4uw=
github.com/mssola/useragent v1.0.0 h1:WRlDpXyxHDNfvZaPEut5Biveq86Ze4o4EMffyMxmH5o=
github.com/mssola/useragent v1.0.0/go.mod h1:hz9Cqz4RXusgg1EdI4Al0INR62kP7aPSRNHnpU+b85Y=
github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
github.com/onsi/gomega v1.18.1 h1:M1GfJqGRrBrrGGsbxzV5dqM2U2ApXefZCQpkukxYRLE=
@@ -165,26 +129,21 @@ github.com/tklauser/go-sysconf v0.3.9/go.mod h1:11DU/5sG7UexIrp/O6g35hrWzu0JxlwQ
github.com/tklauser/numcpus v0.3.0 h1:ILuRUQBtssgnxw0XXIjKUC56fgnOrFoQQ/4+DeU2biQ=
github.com/tklauser/numcpus v0.3.0/go.mod h1:yFGUr7TUHQRAhyqBcEg0Ge34zDBAsIvJJcyE6boqnA8=
github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc h1:R83G5ikgLMxrBvLh22JhdfI8K6YXEPHx5P03Uu3DRs4=
github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg=
github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/image v0.0.0-20220722155232-062f8c9fd539 h1:/eM0PCrQI2xd471rI+snWuu251/+/jpBpZqir2mPdnU=
golang.org/x/image v0.0.0-20220722155232-062f8c9fd539/go.mod h1:doUCurBvlfPMKfmIpRIywoHmhN3VyhnoFDbvIEWF4hY=
golang.org/x/image v0.7.0 h1:gzS29xtG1J5ybQlv0PuyfE3nmc6R4qB73m6LUUmvFuw=
golang.org/x/image v0.7.0/go.mod h1:nd/q4ef1AKKYl/4kft7g+6UyGbdiqWqTP1ZAbRoV7Rg=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38=
golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -192,95 +151,61 @@ golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191007182048-72f939374954/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201010224723-4f7140c49acb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201216054612-986b41b23924/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210928044308-7d9f5e0b762b/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211020060615-d418f374d309/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211201190559-0a0e4e1bb54c/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211209124913-491a49abca63/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f h1:oA4XRj0qtSt8Yo1Zms0CUlsT3KG69V2UGQWPBxujDmc=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ=
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190411185658-b44545bcd369/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201009025420-dfb3f7c4e634/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201118182958-a01c418693c7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201218084310-7d0127a74742/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210110051926-789bb1bd4061/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210123111255-9b0068b26619/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210216163648-f7da38b97c65/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210305230114-8fe3ee5dd75b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210525143221-35b2ab0089ea/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210816074244-15123e1e1f71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211205182925-97ca703d548d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220111092808-5a964db01320/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab h1:2QkjZIsXupsJbJIdSjjUOgWK3aEtzyuh2mPt3l/CkeU=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo=
golang.org/x/tools v0.1.8 h1:P1HhGGuLW4aAclzjtmJdf0mJOjVUZUzOTqkAkWL+l6w=
golang.org/x/tools v0.1.8/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@@ -323,8 +248,3 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.2.1/go.mod h1:lPVVZ2BS5TfnjLyizF7o7hv7j9/L+8cZY2hLyjP9cGY=
honnef.co/go/tools v0.2.2 h1:MNh1AVMyVX23VUHE2O27jm6lNj3vjO5DexS4A1xvnzk=
honnef.co/go/tools v0.2.2/go.mod h1:lPVVZ2BS5TfnjLyizF7o7hv7j9/L+8cZY2hLyjP9cGY=
rogchap.com/v8go v0.7.0 h1:kgjbiO4zE5itA962ze6Hqmbs4HgZbGzmueCXsZtremg=
rogchap.com/v8go v0.7.0/go.mod h1:MxgP3pL2MW4dpme/72QRs8sgNMmM0pRc8DPhcuLWPAs=

11
internal/apps/main.go Normal file
View File

@@ -0,0 +1,11 @@
// Copyright 2023 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package apps
import teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
func RunMain(f func()) {
if teaconst.IsMain {
f()
}
}

View File

@@ -3,6 +3,7 @@
package caches
const (
SuffixAll = "@GOEDGE_" // 通用后缀
SuffixWebP = "@GOEDGE_WEBP" // WebP后缀
SuffixCompression = "@GOEDGE_" // 压缩后缀 SuffixCompression + Encoding
SuffixMethod = "@GOEDGE_" // 请求方法后缀 SuffixMethod + RequestMethod

View File

@@ -0,0 +1,11 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package caches
import "github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
type FileDir struct {
Path string
Capacity *shared.SizeCapacity
IsFull bool
}

View File

@@ -1,7 +1,8 @@
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/fasttime"
"strings"
"time"
)
@@ -35,7 +36,7 @@ type Item struct {
}
func (this *Item) IsExpired() bool {
return this.ExpiredAt < utils.UnixTime()
return this.ExpiredAt < fasttime.Now().Unix()
}
func (this *Item) TotalSize() int64 {
@@ -59,3 +60,17 @@ func (this *Item) IncreaseHit(week int32) {
this.Week = week
}
}
func (this *Item) RequestURI() string {
var schemeIndex = strings.Index(this.Key, "://")
if schemeIndex <= 0 {
return ""
}
var firstSlashIndex = strings.Index(this.Key[schemeIndex+3:], "/")
if firstSlashIndex <= 0 {
return ""
}
return this.Key[schemeIndex+3+firstSlashIndex:]
}

View File

@@ -81,3 +81,14 @@ func TestItems_Memory2(t *testing.T) {
t.Log(w, len(i))
}
}
func TestItem_RequestURI(t *testing.T) {
for _, u := range []string{
"https://goedge.cn/hello/world",
"https://goedge.cn:8080/hello/world",
"https://goedge.cn/hello/world?v=1&t=123",
} {
var item = &Item{Key: u}
t.Log(u, "=>", item.RequestURI())
}
}

View File

@@ -7,9 +7,9 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/ttlcache"
"github.com/TeaOSLab/EdgeNode/internal/utils/dbs"
"github.com/TeaOSLab/EdgeNode/internal/utils/fnv"
"github.com/iwind/TeaGo/types"
_ "github.com/mattn/go-sqlite3"
"os"
"sync/atomic"
"time"
@@ -160,6 +160,7 @@ func (this *FileList) CleanPrefix(prefix string) error {
}
defer func() {
// TODO 需要优化
this.memoryCache.Clean()
}()
@@ -172,6 +173,46 @@ func (this *FileList) CleanPrefix(prefix string) error {
return nil
}
// CleanMatchKey 清理通配符匹配的缓存数据,类似于 https://*.example.com/hello
func (this *FileList) CleanMatchKey(key string) error {
if len(key) == 0 {
return nil
}
defer func() {
// TODO 需要优化
this.memoryCache.Clean()
}()
for _, db := range this.dbList {
err := db.CleanMatchKey(key)
if err != nil {
return err
}
}
return nil
}
// CleanMatchPrefix 清理通配符匹配的缓存数据,类似于 https://*.example.com/prefix/
func (this *FileList) CleanMatchPrefix(prefix string) error {
if len(prefix) == 0 {
return nil
}
defer func() {
// TODO 需要优化
this.memoryCache.Clean()
}()
for _, db := range this.dbList {
err := db.CleanMatchPrefix(prefix)
if err != nil {
return err
}
}
return nil
}
func (this *FileList) Remove(hash string) error {
_, err := this.remove(hash)
return err
@@ -409,7 +450,7 @@ func (this *FileList) UpgradeV3(oldDir string, brokenOnError bool) error {
remotelogs.Println("CACHE", "upgrading local database finished")
}()
db, err := sql.Open("sqlite3", "file:"+indexDBPath+"?cache=shared&mode=rwc&_journal_mode=WAL&_sync=OFF")
db, err := dbs.OpenWriter("file:" + indexDBPath + "?cache=shared&mode=rwc&_journal_mode=WAL&_sync=OFF&_locking_mode=EXCLUSIVE")
if err != nil {
return err
}

View File

@@ -3,17 +3,20 @@
package caches
import (
"database/sql"
"errors"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/dbs"
"github.com/TeaOSLab/EdgeNode/internal/utils/fasttime"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/types"
timeutil "github.com/iwind/TeaGo/utils/time"
"net"
"net/url"
"os"
"path/filepath"
"runtime"
"strings"
"time"
@@ -79,14 +82,15 @@ func (this *FileListDB) Open(dbPath string) error {
}
// write db
writeDB, err := sql.Open("sqlite3", "file:"+dbPath+"?cache=private&mode=rwc&_journal_mode=WAL&_sync=OFF&_cache_size="+types.String(cacheSize)+"&_secure_delete=FAST")
// 这里不能加 EXCLUSIVE 锁,不然异步事务可能会失败
writeDB, err := dbs.OpenWriter("file:" + dbPath + "?cache=private&mode=rwc&_journal_mode=WAL&_sync=OFF&_cache_size=" + types.String(cacheSize) + "&_secure_delete=FAST")
if err != nil {
return errors.New("open write database failed: " + err.Error())
}
writeDB.SetMaxOpenConns(1)
this.writeDB = dbs.NewDB(writeDB)
this.writeDB = writeDB
// TODO 耗时过长,暂时不整理数据库
// TODO 需要根据行数来判断是否VACUUM
@@ -106,9 +110,9 @@ func (this *FileListDB) Open(dbPath string) error {
}
}
this.writeBatch = dbs.NewBatch(writeDB, 4)
this.writeBatch = dbs.NewBatch(writeDB.RawDB(), 4)
this.writeBatch.OnFail(func(err error) {
remotelogs.Warn("LIST_FILE_DB", "run batch failed: "+err.Error())
remotelogs.Warn("LIST_FILE_DB", "run batch failed: "+err.Error()+" ("+filepath.Base(this.dbPath)+")")
})
goman.New(func() {
@@ -121,14 +125,14 @@ func (this *FileListDB) Open(dbPath string) error {
}
// read db
readDB, err := sql.Open("sqlite3", "file:"+dbPath+"?cache=private&mode=ro&_journal_mode=WAL&_sync=OFF&_cache_size="+types.String(cacheSize))
readDB, err := dbs.OpenReader("file:" + dbPath + "?cache=private&mode=ro&_journal_mode=WAL&_sync=OFF&_cache_size=" + types.String(cacheSize))
if err != nil {
return errors.New("open read database failed: " + err.Error())
}
readDB.SetMaxOpenConns(runtime.NumCPU())
this.readDB = dbs.NewDB(readDB)
this.readDB = readDB
if teaconst.EnableDBStat {
this.readDB.EnableStat(true)
@@ -177,6 +181,9 @@ func (this *FileListDB) Init() error {
}
this.selectHashListStmt, err = this.readDB.Prepare(`SELECT "id", "hash" FROM "` + this.itemsTableName + `" WHERE id>:id ORDER BY id ASC LIMIT 2000`)
if err != nil {
return err
}
this.deleteByHashSQL = `DELETE FROM "` + this.itemsTableName + `" WHERE "hash"=?`
this.deleteByHashStmt, err = this.writeDB.Prepare(this.deleteByHashSQL)
@@ -219,6 +226,20 @@ func (this *FileListDB) Init() error {
err := this.hashMap.Load(this)
if err != nil {
remotelogs.Error("LIST_FILE_DB", "load hash map failed: "+err.Error()+"(file: "+this.dbPath+")")
// 自动修复错误
// TODO 将来希望能尽可能恢复以往数据库中的内容
if strings.Contains(err.Error(), "database is closed") || strings.Contains(err.Error(), "database disk image is malformed") {
_ = this.Close()
this.deleteDB()
remotelogs.Println("LIST_FILE_DB", "recreating the database (file:"+this.dbPath+") ...")
err = this.Open(this.dbPath)
if err != nil {
remotelogs.Error("LIST_FILE_DB", "recreate the database failed: "+err.Error()+" (file:"+this.dbPath+")")
} else {
_ = this.Init()
}
}
}
}()
@@ -240,7 +261,7 @@ func (this *FileListDB) AddAsync(hash string, item *Item) error {
item.StaleAt = item.ExpiredAt
}
this.writeBatch.Add(this.insertSQL, hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.StaleAt, item.Host, item.ServerId, utils.UnixTime(), timeutil.Format("YW"))
this.writeBatch.Add(this.insertSQL, hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.StaleAt, item.Host, item.ServerId, fasttime.Now().Unix(), timeutil.Format("YW"))
return nil
}
@@ -252,7 +273,7 @@ func (this *FileListDB) AddSync(hash string, item *Item) error {
item.StaleAt = item.ExpiredAt
}
_, err := this.insertStmt.Exec(hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.StaleAt, item.Host, item.ServerId, utils.UnixTime(), timeutil.Format("YW"))
_, err := this.insertStmt.Exec(hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.StaleAt, item.Host, item.ServerId, fasttime.Now().Unix(), timeutil.Format("YW"))
if err != nil {
return this.WrapError(err)
}
@@ -371,8 +392,8 @@ func (this *FileListDB) CleanPrefix(prefix string) error {
return nil
}
var count = int64(10000)
var staleLife = 600 // TODO 需要可以设置
var unixTime = utils.UnixTime() // 只删除当前的,不删除新的
var staleLife = 600 // TODO 需要可以设置
var unixTime = fasttime.Now().Unix() // 只删除当前的,不删除新的
for {
result, err := this.writeDB.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0,staleAt=? WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)=1 LIMIT `+types.String(count)+`)`, unixTime+int64(staleLife), unixTime, prefix)
if err != nil {
@@ -388,6 +409,85 @@ func (this *FileListDB) CleanPrefix(prefix string) error {
}
}
func (this *FileListDB) CleanMatchKey(key string) error {
if !this.isReady {
return nil
}
// 忽略 @GOEDGE_
if strings.Contains(key, SuffixAll) {
return nil
}
u, err := url.Parse(key)
if err != nil {
return nil
}
var host = u.Host
hostPart, _, err := net.SplitHostPort(host)
if err == nil && len(hostPart) > 0 {
host = hostPart
}
if len(host) == 0 {
return nil
}
// 转义
var queryKey = strings.ReplaceAll(key, "%", "\\%")
queryKey = strings.ReplaceAll(queryKey, "_", "\\_")
queryKey = strings.Replace(queryKey, "*", "%", 1)
// TODO 检查大批量数据下的操作性能
var staleLife = 600 // TODO 需要可以设置
var unixTime = fasttime.Now().Unix() // 只删除当前的,不删除新的
_, err = this.writeDB.Exec(`UPDATE "`+this.itemsTableName+`" SET "expiredAt"=0, "staleAt"=? WHERE "host" GLOB ? AND "host" NOT GLOB ? AND "key" LIKE ? ESCAPE '\'`, unixTime+int64(staleLife), host, "*."+host, queryKey)
if err != nil {
return err
}
_, err = this.writeDB.Exec(`UPDATE "`+this.itemsTableName+`" SET "expiredAt"=0, "staleAt"=? WHERE "host" GLOB ? AND "host" NOT GLOB ? AND "key" LIKE ? ESCAPE '\'`, unixTime+int64(staleLife), host, "*."+host, queryKey+SuffixAll+"%")
if err != nil {
return err
}
return nil
}
func (this *FileListDB) CleanMatchPrefix(prefix string) error {
if !this.isReady {
return nil
}
u, err := url.Parse(prefix)
if err != nil {
return nil
}
var host = u.Host
hostPart, _, err := net.SplitHostPort(host)
if err == nil && len(hostPart) > 0 {
host = hostPart
}
if len(host) == 0 {
return nil
}
// 转义
var queryPrefix = strings.ReplaceAll(prefix, "%", "\\%")
queryPrefix = strings.ReplaceAll(queryPrefix, "_", "\\_")
queryPrefix = strings.Replace(queryPrefix, "*", "%", 1)
queryPrefix += "%"
// TODO 检查大批量数据下的操作性能
var staleLife = 600 // TODO 需要可以设置
var unixTime = fasttime.Now().Unix() // 只删除当前的,不删除新的
_, err = this.writeDB.Exec(`UPDATE "`+this.itemsTableName+`" SET "expiredAt"=0, "staleAt"=? WHERE "host" GLOB ? AND "host" NOT GLOB ? AND "key" LIKE ? ESCAPE '\'`, unixTime+int64(staleLife), host, "*."+host, queryPrefix)
return err
}
func (this *FileListDB) CleanAll() error {
if !this.isReady {
return nil
@@ -597,3 +697,10 @@ func (this *FileListDB) shouldRecover() bool {
_ = result.Close()
return shouldRecover
}
// 删除数据库文件
func (this *FileListDB) deleteDB() {
_ = os.Remove(this.dbPath)
_ = os.Remove(this.dbPath + "-shm")
_ = os.Remove(this.dbPath + "-wal")
}

View File

@@ -12,6 +12,11 @@ import (
func TestFileListDB_ListLFUItems(t *testing.T) {
var db = caches.NewFileListDB()
defer func() {
_ = db.Close()
}()
err := db.Open(Tea.Root + "/data/cache-db-large.db")
//err := db.Open(Tea.Root + "/data/cache-index/p1/db-0.db")
if err != nil {
@@ -22,10 +27,6 @@ func TestFileListDB_ListLFUItems(t *testing.T) {
t.Fatal(err)
}
defer func() {
_ = db.Close()
}()
hashList, err := db.ListLFUItems(100)
if err != nil {
t.Fatal(err)
@@ -35,15 +36,72 @@ func TestFileListDB_ListLFUItems(t *testing.T) {
func TestFileListDB_IncreaseHitAsync(t *testing.T) {
var db = caches.NewFileListDB()
defer func() {
_ = db.Close()
}()
err := db.Open(Tea.Root + "/data/cache-db-large.db")
if err != nil {
t.Fatal(err)
}
err = db.Init()
err = db.IncreaseHitAsync("4598e5231ba47d6ec7aa9ea640ff2eaf")
if err != nil {
t.Fatal(err)
}
// wait transaction
time.Sleep(1 * time.Second)
}
func TestFileListDB_CleanMatchKey(t *testing.T) {
var db = caches.NewFileListDB()
defer func() {
_ = db.Close()
}()
err := db.Open(Tea.Root + "/data/cache-db-large.db")
if err != nil {
t.Fatal(err)
}
err = db.Init()
err = db.CleanMatchKey("https://*.goedge.cn/large-text")
if err != nil {
t.Fatal(err)
}
err = db.CleanMatchKey("https://*.goedge.cn:1234/large-text?%2B____")
if err != nil {
t.Fatal(err)
}
}
func TestFileListDB_CleanMatchPrefix(t *testing.T) {
var db = caches.NewFileListDB()
defer func() {
_ = db.Close()
}()
err := db.Open(Tea.Root + "/data/cache-db-large.db")
if err != nil {
t.Fatal(err)
}
err = db.Init()
err = db.CleanMatchPrefix("https://*.goedge.cn/large-text")
if err != nil {
t.Fatal(err)
}
err = db.CleanMatchPrefix("https://*.goedge.cn:1234/large-text?%2B____")
if err != nil {
t.Fatal(err)
}
}

View File

@@ -59,15 +59,16 @@ func TestFileListHashMap_BigInt(t *testing.T) {
func TestFileListHashMap_Load(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1").(*caches.FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
}
var m = caches.NewFileListHashMap()
var before = time.Now()
var db = list.GetDB("abc")

View File

@@ -5,6 +5,7 @@ package caches_test
import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/utils/testutils"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
@@ -17,6 +18,11 @@ import (
func TestFileList_Init(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -29,6 +35,11 @@ func TestFileList_Init(t *testing.T) {
func TestFileList_Add(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1").(*caches.FileList)
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -59,16 +70,21 @@ func TestFileList_Add(t *testing.T) {
}
func TestFileList_Add_Many(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
if !testutils.IsSingleTesting() {
return
}
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
}
var before = time.Now()
for i := 0; i < 10_000_000; i++ {
u := "https://edge.teaos.cn/123456" + strconv.Itoa(i)
@@ -92,15 +108,15 @@ func TestFileList_Add_Many(t *testing.T) {
func TestFileList_Exist(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1").(*caches.FileList)
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
total, _ := list.Count()
t.Log("total:", total)
@@ -130,7 +146,7 @@ func TestFileList_Exist_Many_DB(t *testing.T) {
// 测试在多个数据库下的性能
var listSlice = []caches.ListInterface{}
for i := 1; i <= 10; i++ {
list := caches.NewFileList(Tea.Root + "/data/data" + strconv.Itoa(i))
var list = caches.NewFileList(Tea.Root + "/data/data" + strconv.Itoa(i))
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -138,6 +154,12 @@ func TestFileList_Exist_Many_DB(t *testing.T) {
listSlice = append(listSlice, list)
}
defer func() {
for _, list := range listSlice {
_ = list.Close()
}
}()
var wg = sync.WaitGroup{}
var threads = 8
wg.Add(threads)
@@ -181,15 +203,16 @@ func TestFileList_Exist_Many_DB(t *testing.T) {
func TestFileList_CleanPrefix(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
}
before := time.Now()
err = list.CleanPrefix("123")
if err != nil {
@@ -200,15 +223,15 @@ func TestFileList_CleanPrefix(t *testing.T) {
func TestFileList_Remove(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1").(*caches.FileList)
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
list.OnRemove(func(item *caches.Item) {
t.Logf("remove %#v", item)
})
@@ -224,13 +247,15 @@ func TestFileList_Remove(t *testing.T) {
func TestFileList_Purge(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
var count = 0
_, err = list.Purge(caches.CountFileDB*2, func(hash string) error {
@@ -246,13 +271,15 @@ func TestFileList_Purge(t *testing.T) {
func TestFileList_PurgeLFU(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
err = list.IncreaseHit(stringutil.Md5("123456"))
if err != nil {
@@ -273,15 +300,16 @@ func TestFileList_PurgeLFU(t *testing.T) {
func TestFileList_Stat(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
err := list.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
}
stat, err := list.Stat(nil)
if err != nil {
t.Fatal(err)
@@ -291,6 +319,11 @@ func TestFileList_Stat(t *testing.T) {
func TestFileList_Count(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data")
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -305,7 +338,12 @@ func TestFileList_Count(t *testing.T) {
}
func TestFileList_CleanAll(t *testing.T) {
list := caches.NewFileList(Tea.Root + "/data")
var list = caches.NewFileList(Tea.Root + "/data")
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -320,6 +358,11 @@ func TestFileList_CleanAll(t *testing.T) {
func TestFileList_IncreaseHit(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -333,7 +376,13 @@ func TestFileList_IncreaseHit(t *testing.T) {
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
for i := 0; i < 1000_000; i++ {
var count = 1_000_000
if !testutils.IsSingleTesting() {
count = 10
}
for i := 0; i < count; i++ {
err = list.IncreaseHit(stringutil.Md5("abc" + types.String(i)))
}
if err != nil {
@@ -344,6 +393,11 @@ func TestFileList_IncreaseHit(t *testing.T) {
func TestFileList_UpgradeV3(t *testing.T) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p43").(*caches.FileList)
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
t.Fatal(err)
@@ -363,6 +417,11 @@ func TestFileList_UpgradeV3(t *testing.T) {
func BenchmarkFileList_Exist(b *testing.B) {
var list = caches.NewFileList(Tea.Root + "/data/cache-index/p1")
defer func() {
_ = list.Close()
}()
err := list.Init()
if err != nil {
b.Fatal(err)

View File

@@ -18,6 +18,12 @@ type ListInterface interface {
// CleanPrefix 清除某个前缀的缓存
CleanPrefix(prefix string) error
// CleanMatchKey 清除通配符匹配的Key
CleanMatchKey(key string) error
// CleanMatchPrefix 清除通配符匹配的前缀
CleanMatchPrefix(prefix string) error
// Remove 删除内容
Remove(hash string) error

View File

@@ -1,8 +1,11 @@
package caches
import (
"github.com/TeaOSLab/EdgeCommon/pkg/configutils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/logs"
"net"
"net/url"
"strconv"
"strings"
"sync"
@@ -146,6 +149,82 @@ func (this *MemoryList) CleanPrefix(prefix string) error {
return nil
}
// CleanMatchKey 清理通配符匹配的缓存数据,类似于 https://*.example.com/hello
func (this *MemoryList) CleanMatchKey(key string) error {
if strings.Contains(key, SuffixAll) {
return nil
}
u, err := url.Parse(key)
if err != nil {
return nil
}
var host = u.Host
hostPart, _, err := net.SplitHostPort(host)
if err == nil && len(hostPart) > 0 {
host = hostPart
}
if len(host) == 0 {
return nil
}
var requestURI = u.RequestURI()
this.locker.RLock()
defer this.locker.RUnlock()
// TODO 需要优化性能支持千万级数据低于1s的处理速度
for _, itemMap := range this.itemMaps {
for _, item := range itemMap {
if configutils.MatchDomain(host, item.Host) {
var itemRequestURI = item.RequestURI()
if itemRequestURI == requestURI || strings.HasPrefix(itemRequestURI, requestURI+SuffixAll) {
item.ExpiredAt = 0
}
}
}
}
return nil
}
// CleanMatchPrefix 清理通配符匹配的缓存数据,类似于 https://*.example.com/prefix/
func (this *MemoryList) CleanMatchPrefix(prefix string) error {
u, err := url.Parse(prefix)
if err != nil {
return nil
}
var host = u.Host
hostPart, _, err := net.SplitHostPort(host)
if err == nil && len(hostPart) > 0 {
host = hostPart
}
if len(host) == 0 {
return nil
}
var requestURI = u.RequestURI()
var isRootPath = requestURI == "/"
this.locker.RLock()
defer this.locker.RUnlock()
// TODO 需要优化性能支持千万级数据低于1s的处理速度
for _, itemMap := range this.itemMaps {
for _, item := range itemMap {
if configutils.MatchDomain(host, item.Host) {
var itemRequestURI = item.RequestURI()
if isRootPath || strings.HasPrefix(itemRequestURI, requestURI) {
item.ExpiredAt = 0
}
}
}
}
return nil
}
func (this *MemoryList) Remove(hash string) error {
this.locker.Lock()

View File

@@ -2,6 +2,7 @@ package caches
import (
"fmt"
"github.com/TeaOSLab/EdgeNode/internal/utils/testutils"
"github.com/cespare/xxhash"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/rands"
@@ -107,7 +108,9 @@ func TestMemoryList_Purge_Large_List(t *testing.T) {
})
}
time.Sleep(1 * time.Hour)
if testutils.IsSingleTesting() {
time.Sleep(1 * time.Hour)
}
}
func TestMemoryList_Stat(t *testing.T) {
@@ -255,9 +258,11 @@ func TestMemoryList_GC(t *testing.T) {
//runtime.GC()
t.Log("gc cost:", time.Since(before).Seconds()*1000, "ms")
timeout := time.NewTimer(2 * time.Minute)
<-timeout.C
t.Log("2 minutes passed")
if testutils.IsSingleTesting() {
timeout := time.NewTimer(2 * time.Minute)
<-timeout.C
t.Log("2 minutes passed")
time.Sleep(30 * time.Minute)
time.Sleep(30 * time.Minute)
}
}

View File

@@ -3,6 +3,7 @@ package caches
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/lists"
@@ -14,6 +15,10 @@ import (
var SharedManager = NewManager()
func init() {
if !teaconst.IsMain {
return
}
events.On(events.EventQuit, func() {
remotelogs.Println("CACHE", "quiting cache manager")
SharedManager.UpdatePolicies([]*serverconfigs.HTTPCachePolicy{})
@@ -24,7 +29,8 @@ func init() {
type Manager struct {
// 全局配置
MaxDiskCapacity *shared.SizeCapacity
DiskDir string
MainDiskDir string
SubDiskDirs []*serverconfigs.CacheDir
MaxMemoryCapacity *shared.SizeCapacity
policyMap map[int64]*serverconfigs.HTTPCachePolicy // policyId => []*Policy
@@ -47,12 +53,10 @@ func (this *Manager) UpdatePolicies(newPolicies []*serverconfigs.HTTPCachePolicy
this.locker.Lock()
defer this.locker.Unlock()
newPolicyIds := []int64{}
var newPolicyIds = []int64{}
for _, policy := range newPolicies {
// 使用节点单独的缓存目录
if len(this.DiskDir) > 0 {
policy.UpdateDiskDir(this.DiskDir)
}
policy.UpdateDiskDir(this.MainDiskDir, this.SubDiskDirs)
newPolicyIds = append(newPolicyIds, policy.Id)
}
@@ -173,10 +177,15 @@ func (this *Manager) TotalDiskSize() int64 {
this.locker.RLock()
defer this.locker.RUnlock()
total := int64(0)
var total = int64(0)
for _, storage := range this.storageMap {
total += storage.TotalDiskSize()
}
if total < 0 {
total = 0
}
return total
}

View File

@@ -19,7 +19,7 @@ type OpenFileCache struct {
poolList *linkedlist.List
watcher *fsnotify.Watcher
locker sync.Mutex
locker sync.RWMutex
maxSize int
count int
@@ -54,13 +54,18 @@ func NewOpenFileCache(maxSize int) (*OpenFileCache, error) {
}
func (this *OpenFileCache) Get(filename string) *OpenFile {
this.locker.Lock()
defer this.locker.Unlock()
this.locker.RLock()
pool, ok := this.poolMap[filename]
this.locker.RUnlock()
if ok {
file, consumed := pool.Get()
if consumed {
this.locker.Lock()
this.count--
// pool如果为空也不需要从列表中删除避免put时需要重新创建
this.locker.Unlock()
}
return file
}
@@ -124,6 +129,9 @@ func (this *OpenFileCache) Close(filename string) {
pool, ok := this.poolMap[filename]
if ok {
// 设置关闭状态
pool.SetClosing()
delete(this.poolMap, filename)
this.poolList.Remove(pool.linkItem)
_ = this.watcher.Remove(filename)
@@ -146,6 +154,7 @@ func (this *OpenFileCache) CloseAll() {
this.poolMap = map[string]*OpenFilePool{}
this.poolList.Reset()
_ = this.watcher.Close()
this.count = 0
this.locker.Unlock()
}

View File

@@ -0,0 +1,46 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package caches_test
import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/utils/testutils"
"testing"
"time"
)
func TestNewOpenFileCache_Close(t *testing.T) {
cache, err := caches.NewOpenFileCache(1024)
if err != nil {
t.Fatal(err)
}
cache.Debug()
cache.Put("a.txt", caches.NewOpenFile(nil, nil, nil, 0))
cache.Put("b.txt", caches.NewOpenFile(nil, nil, nil, 0))
cache.Put("b.txt", caches.NewOpenFile(nil, nil, nil, 0))
cache.Put("b.txt", caches.NewOpenFile(nil, nil, nil, 0))
cache.Put("c.txt", caches.NewOpenFile(nil, nil, nil, 0))
cache.Get("b.txt")
cache.Get("d.txt")
cache.Close("a.txt")
if testutils.IsSingleTesting() {
time.Sleep(100 * time.Second)
}
}
func TestNewOpenFileCache_CloseAll(t *testing.T) {
cache, err := caches.NewOpenFileCache(1024)
if err != nil {
t.Fatal(err)
}
cache.Debug()
cache.Put("a.txt", caches.NewOpenFile(nil, nil, nil, 0))
cache.Put("b.txt", caches.NewOpenFile(nil, nil, nil, 0))
cache.Put("c.txt", caches.NewOpenFile(nil, nil, nil, 0))
cache.Get("b.txt")
cache.Get("d.txt")
cache.CloseAll()
time.Sleep(6 * time.Second)
}

View File

@@ -3,7 +3,7 @@
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/fasttime"
"github.com/TeaOSLab/EdgeNode/internal/utils/linkedlist"
)
@@ -12,13 +12,14 @@ type OpenFilePool struct {
linkItem *linkedlist.Item
filename string
version int64
isClosed bool
}
func NewOpenFilePool(filename string) *OpenFilePool {
var pool = &OpenFilePool{
filename: filename,
c: make(chan *OpenFile, 1024),
version: utils.UnixTimeMilli(),
version: fasttime.Now().UnixMilli(),
}
pool.linkItem = linkedlist.NewItem(pool)
return pool
@@ -29,26 +30,43 @@ func (this *OpenFilePool) Filename() string {
}
func (this *OpenFilePool) Get() (*OpenFile, bool) {
// 如果已经关闭,直接返回
if this.isClosed {
return nil, false
}
select {
case file := <-this.c:
err := file.SeekStart()
if err != nil {
_ = file.Close()
return nil, true
}
file.version = this.version
if file != nil {
err := file.SeekStart()
if err != nil {
_ = file.Close()
return nil, true
}
file.version = this.version
return file, true
return file, true
}
return nil, false
default:
return nil, false
}
}
func (this *OpenFilePool) Put(file *OpenFile) bool {
// 如果已关闭,则不接受新的文件
if this.isClosed {
_ = file.Close()
return false
}
// 检查文件版本号
if this.version > 0 && file.version > 0 && file.version != this.version {
_ = file.Close()
return false
}
// 加入Pool
select {
case this.c <- file:
return true
@@ -63,14 +81,18 @@ func (this *OpenFilePool) Len() int {
return len(this.c)
}
func (this *OpenFilePool) SetClosing() {
this.isClosed = true
}
func (this *OpenFilePool) Close() {
Loop:
this.isClosed = true
for {
select {
case file := <-this.c:
_ = file.Close()
default:
break Loop
return
}
}
}

View File

@@ -4,6 +4,8 @@ package caches_test
import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/iwind/TeaGo/rands"
"sync"
"testing"
)
@@ -15,3 +17,30 @@ func TestOpenFilePool_Get(t *testing.T) {
t.Log(pool.Get())
t.Log(pool.Get())
}
func TestOpenFilePool_Close(t *testing.T) {
var pool = caches.NewOpenFilePool("a")
pool.Put(caches.NewOpenFile(nil, nil, nil, 0))
pool.Put(caches.NewOpenFile(nil, nil, nil, 0))
pool.Close()
}
func TestOpenFilePool_Concurrent(t *testing.T) {
var pool = caches.NewOpenFilePool("a")
var concurrent = 1000
var wg = &sync.WaitGroup{}
wg.Add(concurrent)
for i := 0; i < concurrent; i++ {
go func() {
defer wg.Done()
if rands.Int(0, 1) == 1 {
pool.Put(caches.NewOpenFile(nil, nil, nil, 0))
}
if rands.Int(0, 1) == 0 {
pool.Get()
}
}()
}
wg.Wait()
}

View File

@@ -3,38 +3,88 @@
package caches
import (
"bytes"
"encoding/json"
"errors"
"github.com/iwind/TeaGo/types"
"os"
"strconv"
)
// PartialRanges 内容分区范围定义
type PartialRanges struct {
Ranges [][2]int64 `json:"ranges"`
Version int `json:"version"` // 版本号
Ranges [][2]int64 `json:"ranges"` // 范围
BodySize int64 `json:"bodySize"` // 总长度
}
// NewPartialRanges 获取新对象
func NewPartialRanges() *PartialRanges {
return &PartialRanges{Ranges: [][2]int64{}}
func NewPartialRanges(expiresAt int64) *PartialRanges {
return &PartialRanges{
Ranges: [][2]int64{},
Version: 1,
}
}
// NewPartialRangesFromData 从数据中解析范围
func NewPartialRangesFromData(data []byte) (*PartialRanges, error) {
var rs = NewPartialRanges(0)
for {
var index = bytes.IndexRune(data, '\n')
if index < 0 {
break
}
var line = data[:index]
var colonIndex = bytes.IndexRune(line, ':')
if colonIndex > 0 {
switch string(line[:colonIndex]) {
case "v": // 版本号
rs.Version = types.Int(line[colonIndex+1:])
case "b": // 总长度
rs.BodySize = types.Int64(line[colonIndex+1:])
case "r": // 范围信息
var commaIndex = bytes.IndexRune(line, ',')
if commaIndex > 0 {
rs.Ranges = append(rs.Ranges, [2]int64{types.Int64(line[colonIndex+1 : commaIndex]), types.Int64(line[commaIndex+1:])})
}
}
}
data = data[index+1:]
if len(data) == 0 {
break
}
}
return rs, nil
}
// NewPartialRangesFromJSON 从JSON中解析范围
func NewPartialRangesFromJSON(data []byte) (*PartialRanges, error) {
var rs = NewPartialRanges()
var rs = NewPartialRanges(0)
err := json.Unmarshal(data, &rs)
if err != nil {
return nil, err
}
rs.Version = 0
return rs, nil
}
// NewPartialRangesFromFile 从文件中加载范围信息
func NewPartialRangesFromFile(path string) (*PartialRanges, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, err
}
return NewPartialRangesFromJSON(data)
if len(data) == 0 {
return NewPartialRanges(0), nil
}
// 兼容老的JSON格式
if data[0] == '{' {
return NewPartialRangesFromJSON(data)
}
// 新的格式
return NewPartialRangesFromData(data)
}
// Add 添加新范围
@@ -105,29 +155,27 @@ func (this *PartialRanges) Nearest(begin int64, end int64) (r [2]int64, ok bool)
return
}
// AsJSON 转换为JSON
func (this *PartialRanges) AsJSON() ([]byte, error) {
return json.Marshal(this)
// 转换为字符串
func (this *PartialRanges) String() string {
var s = "v:" + strconv.Itoa(this.Version) + "\n" + // version
"b:" + this.formatInt64(this.BodySize) + "\n" // bodySize
for _, r := range this.Ranges {
s += "r:" + this.formatInt64(r[0]) + "," + this.formatInt64(r[1]) + "\n" // range
}
return s
}
// Bytes 将内容转换为字节
func (this *PartialRanges) Bytes() []byte {
return []byte(this.String())
}
// WriteToFile 写入到文件中
func (this *PartialRanges) WriteToFile(path string) error {
data, err := this.AsJSON()
if err != nil {
return errors.New("convert to json failed: " + err.Error())
}
return os.WriteFile(path, data, 0666)
}
// ReadFromFile 从文件中读取
func (this *PartialRanges) ReadFromFile(path string) (*PartialRanges, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, err
}
return NewPartialRangesFromJSON(data)
return os.WriteFile(path, this.Bytes(), 0666)
}
// Max 获取最大位置
func (this *PartialRanges) Max() int64 {
if len(this.Ranges) > 0 {
return this.Ranges[len(this.Ranges)-1][1]
@@ -135,6 +183,11 @@ func (this *PartialRanges) Max() int64 {
return 0
}
// Reset 重置范围信息
func (this *PartialRanges) Reset() {
this.Ranges = [][2]int64{}
}
func (this *PartialRanges) merge(index int) {
// forward
var lastIndex = index
@@ -187,3 +240,7 @@ func (this *PartialRanges) max(n1 int64, n2 int64) int64 {
}
return n2
}
func (this *PartialRanges) formatInt64(i int64) string {
return strconv.FormatInt(i, 10)
}

View File

@@ -3,14 +3,16 @@
package caches_test
import (
"encoding/json"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/iwind/TeaGo/assert"
"github.com/iwind/TeaGo/logs"
"testing"
"time"
)
func TestNewPartialRanges(t *testing.T) {
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
r.Add(1, 100)
r.Add(50, 300)
@@ -28,7 +30,7 @@ func TestNewPartialRanges(t *testing.T) {
func TestNewPartialRanges1(t *testing.T) {
var a = assert.NewAssertion(t)
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
r.Add(1, 100)
r.Add(1, 101)
r.Add(1, 102)
@@ -47,7 +49,7 @@ func TestNewPartialRanges1(t *testing.T) {
func TestNewPartialRanges2(t *testing.T) {
// low -> high
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
r.Add(1, 100)
r.Add(1, 101)
r.Add(1, 102)
@@ -63,7 +65,7 @@ func TestNewPartialRanges2(t *testing.T) {
func TestNewPartialRanges3(t *testing.T) {
// high -> low
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
r.Add(301, 302)
r.Add(303, 304)
r.Add(200, 300)
@@ -75,7 +77,7 @@ func TestNewPartialRanges3(t *testing.T) {
func TestNewPartialRanges4(t *testing.T) {
// nearby
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
r.Add(301, 302)
r.Add(303, 304)
r.Add(305, 306)
@@ -90,7 +92,7 @@ func TestNewPartialRanges4(t *testing.T) {
}
func TestNewPartialRanges5(t *testing.T) {
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
for j := 0; j < 1000; j++ {
r.Add(int64(j), int64(j+100))
}
@@ -100,7 +102,7 @@ func TestNewPartialRanges5(t *testing.T) {
func TestNewPartialRanges_Nearest(t *testing.T) {
{
// nearby
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
r.Add(301, 400)
r.Add(401, 500)
r.Add(501, 600)
@@ -112,7 +114,7 @@ func TestNewPartialRanges_Nearest(t *testing.T) {
{
// nearby
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
r.Add(301, 400)
r.Add(450, 500)
r.Add(550, 600)
@@ -131,45 +133,100 @@ func TestNewPartialRanges_Large_Range(t *testing.T) {
var largeSize int64 = 10000000000000
t.Log(largeSize/1024/1024/1024, "G")
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
r.Add(1, largeSize)
jsonData, err := r.AsJSON()
if err != nil {
t.Fatal(err)
}
t.Log(string(jsonData))
var s = r.String()
t.Log(s)
r2, err := caches.NewPartialRangesFromJSON(jsonData)
r2, err := caches.NewPartialRangesFromData([]byte(s))
if err != nil {
t.Fatal(err)
}
a.IsTrue(largeSize == r2.Ranges[0][1])
logs.PrintAsJSON(r, t)
}
func TestNewPartialRanges_AsJSON(t *testing.T) {
var r = caches.NewPartialRanges()
for j := 0; j < 1000; j++ {
r.Add(int64(j), int64(j+100))
func TestPartialRanges_Encode_JSON(t *testing.T) {
var r = caches.NewPartialRanges(0)
for i := 0; i < 10; i++ {
r.Ranges = append(r.Ranges, [2]int64{int64(i * 100), int64(i*100 + 100)})
}
data, err := r.AsJSON()
var before = time.Now()
data, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
t.Log(string(data))
t.Log(time.Since(before).Seconds()*1000, "ms")
t.Log(len(data))
}
r2, err := caches.NewPartialRangesFromJSON(data)
func TestPartialRanges_Encode_String(t *testing.T) {
var r = caches.NewPartialRanges(0)
r.BodySize = 1024
for i := 0; i < 10; i++ {
r.Ranges = append(r.Ranges, [2]int64{int64(i * 100), int64(i*100 + 100)})
}
var before = time.Now()
var data = r.String()
t.Log(time.Since(before).Seconds()*1000, "ms")
t.Log(len(data))
r2, err := caches.NewPartialRangesFromData([]byte(data))
if err != nil {
t.Fatal(err)
}
t.Log(r2.Ranges)
logs.PrintAsJSON(r2, t)
}
func TestPartialRanges_Version(t *testing.T) {
{
ranges, err := caches.NewPartialRangesFromData([]byte(`e:1668928495
r:0,1048576
r:1140260864,1140295164`))
if err != nil {
t.Fatal(err)
}
t.Log("version:", ranges.Version)
}
{
ranges, err := caches.NewPartialRangesFromData([]byte(`e:1668928495
r:0,1048576
r:1140260864,1140295164
v:0
`))
if err != nil {
t.Fatal(err)
}
t.Log("version:", ranges.Version)
}
{
ranges, err := caches.NewPartialRangesFromJSON([]byte(`{}`))
if err != nil {
t.Fatal(err)
}
t.Log("version:", ranges.Version)
}
}
func BenchmarkNewPartialRanges(b *testing.B) {
for i := 0; i < b.N; i++ {
var r = caches.NewPartialRanges()
var r = caches.NewPartialRanges(0)
for j := 0; j < 1000; j++ {
r.Add(int64(j), int64(j+100))
}
}
}
func BenchmarkPartialRanges_String(b *testing.B) {
var r = caches.NewPartialRanges(0)
r.BodySize = 1024
for i := 0; i < 10; i++ {
r.Ranges = append(r.Ranges, [2]int64{int64(i * 100), int64(i*100 + 100)})
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = r.String()
}
}

View File

@@ -42,7 +42,7 @@ func (this *FileReader) InitAutoDiscard(autoDiscard bool) error {
this.header = this.openFile.header
}
isOk := false
var isOk = false
if autoDiscard {
defer func() {
@@ -67,17 +67,17 @@ func (this *FileReader) InitAutoDiscard(autoDiscard bool) error {
this.expiresAt = int64(binary.BigEndian.Uint32(buf[:SizeExpiresAt]))
status := types.Int(string(buf[OffsetStatus : OffsetStatus+SizeStatus]))
var status = types.Int(string(buf[OffsetStatus : OffsetStatus+SizeStatus]))
if status < 100 || status > 999 {
return errors.New("invalid status")
}
this.status = status
// URL
urlLength := binary.BigEndian.Uint32(buf[OffsetURLLength : OffsetURLLength+SizeURLLength])
var urlLength = binary.BigEndian.Uint32(buf[OffsetURLLength : OffsetURLLength+SizeURLLength])
// header
headerSize := int(binary.BigEndian.Uint32(buf[OffsetHeaderLength : OffsetHeaderLength+SizeHeaderLength]))
var headerSize = int(binary.BigEndian.Uint32(buf[OffsetHeaderLength : OffsetHeaderLength+SizeHeaderLength]))
if headerSize == 0 {
return nil
}
@@ -86,7 +86,7 @@ func (this *FileReader) InitAutoDiscard(autoDiscard bool) error {
// body
this.bodyOffset = this.headerOffset + int64(headerSize)
bodySize := int(binary.BigEndian.Uint64(buf[OffsetBodyLength : OffsetBodyLength+SizeBodyLength]))
var bodySize = int(binary.BigEndian.Uint64(buf[OffsetBodyLength : OffsetBodyLength+SizeBodyLength]))
if bodySize == 0 {
isOk = true
return nil
@@ -158,7 +158,7 @@ func (this *FileReader) ReadHeader(buf []byte, callback ReaderFunc) error {
return nil
}
isOk := false
var isOk = false
defer func() {
if !isOk {
@@ -171,7 +171,7 @@ func (this *FileReader) ReadHeader(buf []byte, callback ReaderFunc) error {
return err
}
headerSize := this.headerSize
var headerSize = this.headerSize
for {
n, err := this.fp.Read(buf)
@@ -215,7 +215,11 @@ func (this *FileReader) ReadHeader(buf []byte, callback ReaderFunc) error {
}
func (this *FileReader) ReadBody(buf []byte, callback ReaderFunc) error {
isOk := false
if this.bodySize == 0 {
return nil
}
var isOk = false
defer func() {
if !isOk {
@@ -257,15 +261,22 @@ func (this *FileReader) ReadBody(buf []byte, callback ReaderFunc) error {
}
func (this *FileReader) Read(buf []byte) (n int, err error) {
if this.bodySize == 0 {
n = 0
err = io.EOF
return
}
n, err = this.fp.Read(buf)
if err != nil && err != io.EOF {
_ = this.discard()
}
return
}
func (this *FileReader) ReadBodyRange(buf []byte, start int64, end int64, callback ReaderFunc) error {
isOk := false
var isOk = false
defer func() {
if !isOk {
@@ -273,7 +284,7 @@ func (this *FileReader) ReadBodyRange(buf []byte, start int64, end int64, callba
}
}()
offset := start
var offset = start
if start < 0 {
offset = this.bodyOffset + this.bodySize + end
end = this.bodyOffset + this.bodySize - 1
@@ -296,7 +307,7 @@ func (this *FileReader) ReadBodyRange(buf []byte, start int64, end int64, callba
for {
n, err := this.fp.Read(buf)
if n > 0 {
n2 := int(end-offset) + 1
var n2 = int(end-offset) + 1
if n2 <= n {
_, e := callback(n2)
if e != nil {
@@ -344,12 +355,12 @@ func (this *FileReader) FP() *os.File {
}
func (this *FileReader) Close() error {
if this.openFileCache != nil {
if this.isClosed {
return nil
}
this.isClosed = true
if this.isClosed {
return nil
}
this.isClosed = true
if this.openFileCache != nil {
if this.openFile != nil {
this.openFileCache.Put(this.fp.Name(), this.openFile)
} else {
@@ -359,6 +370,7 @@ func (this *FileReader) Close() error {
}
return nil
}
return this.fp.Close()
}

View File

@@ -8,21 +8,29 @@ import (
)
func TestFileReader(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
}
_, path := storage.keyPath("my-key")
_, path, _ := storage.keyPath("my-key")
fp, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
t.Log("file '" + path + "' not exists")
return
}
t.Fatal(err)
}
defer func() {
@@ -58,6 +66,10 @@ func TestFileReader_ReadHeader(t *testing.T) {
var path = "/Users/WorkSpace/EdgeProject/EdgeCache/p43/12/6b/126bbed90fc80f2bdfb19558948b0d49.cache"
fp, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
t.Log("'" + path + "' not exists")
return
}
t.Fatal(err)
}
defer func() {
@@ -66,6 +78,11 @@ func TestFileReader_ReadHeader(t *testing.T) {
var reader = NewFileReader(fp)
err = reader.Init()
if err != nil {
if os.IsNotExist(err) {
t.Log("file '" + path + "' not exists")
return
}
t.Fatal(err)
}
var buf = make([]byte, 16*1024)
@@ -79,13 +96,16 @@ func TestFileReader_ReadHeader(t *testing.T) {
}
func TestFileReader_Range(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -105,10 +125,14 @@ func TestFileReader_Range(t *testing.T) {
}
_ = writer.Close()**/
_, path := storage.keyPath("my-number")
_, path, _ := storage.keyPath("my-number")
fp, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
t.Log("'" + path + "' not exists")
return
}
t.Fatal(err)
}
defer func() {

View File

@@ -117,13 +117,10 @@ func (this *PartialFileReader) ContainsRange(r rangeutils.Range) (r2 rangeutils.
r2, ok = this.ranges.Nearest(r.Start(), r.End())
if ok && this.bodySize > 0 {
// 考虑可配置
var span int64 = 512 * 1024
if this.bodySize > 1<<30 {
span = 1 << 20
}
const minSpan = 128 << 10
// 这里限制返回的最小缓存,防止因为返回的内容过小而导致请求过多
if r2.Length() < r.Length() && r2.Length() < span {
if r2.Length() < r.Length() && r2.Length() < minSpan {
ok = false
}
}
@@ -138,6 +135,10 @@ func (this *PartialFileReader) MaxLength() int64 {
return this.ranges.Max() + 1
}
func (this *PartialFileReader) Ranges() *PartialRanges {
return this.ranges
}
func (this *PartialFileReader) discard() error {
_ = os.Remove(this.rangePath)
return this.FileReader.discard()

View File

@@ -49,8 +49,7 @@ const (
SizeBodyLength = 8
OffsetBodyLength = OffsetHeaderLength + SizeHeaderLength
SizeMeta = SizeExpiresAt + SizeStatus + SizeURLLength + SizeHeaderLength + SizeBodyLength
OffsetKey = SizeMeta
SizeMeta = SizeExpiresAt + SizeStatus + SizeURLLength + SizeHeaderLength + SizeBodyLength
)
const (
@@ -93,7 +92,10 @@ type FileStorage struct {
openFileCache *OpenFileCache
diskIsFull bool
mainDir string
mainDiskIsFull bool
subDirs []*FileDir
}
func NewFileStorage(policy *serverconfigs.HTTPCachePolicy) *FileStorage {
@@ -157,6 +159,16 @@ func (this *FileStorage) UpdatePolicy(newPolicy *serverconfigs.HTTPCachePolicy)
return
}
var subDirs = []*FileDir{}
for _, subDir := range newOptions.SubDirs {
subDirs = append(subDirs, &FileDir{
Path: subDir.Path,
Capacity: subDir.Capacity,
IsFull: false,
})
}
this.checkDiskSpace()
err = newOptions.Init()
if err != nil {
remotelogs.Error("CACHE", "update policy '"+types.String(this.policy.Id)+"' failed: init options failed: "+err.Error())
@@ -219,6 +231,19 @@ func (this *FileStorage) Init() error {
this.options.Dir = filepath.Clean(this.options.Dir)
var dir = this.options.Dir
var subDirs = []*FileDir{}
for _, subDir := range this.options.SubDirs {
subDirs = append(subDirs, &FileDir{
Path: subDir.Path,
Capacity: subDir.Capacity,
IsFull: false,
})
}
this.subDirs = subDirs
if len(subDirs) > 0 {
this.checkDiskSpace()
}
if len(dir) == 0 {
return errors.New("[CACHE]cache storage dir can not be empty")
}
@@ -321,7 +346,7 @@ func (this *FileStorage) openReader(key string, allowMemory bool, useStale bool,
}
}
hash, path := this.keyPath(key)
hash, path, _ := this.keyPath(key)
// 检查文件记录是否已过期
if !useStale {
@@ -389,26 +414,21 @@ func (this *FileStorage) openReader(key string, allowMemory bool, useStale bool,
}
// OpenWriter 打开缓存文件等待写入
func (this *FileStorage) OpenWriter(key string, expiresAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error) {
return this.openWriter(key, expiresAt, status, size, maxSize, isPartial, false)
func (this *FileStorage) OpenWriter(key string, expiresAt int64, status int, headerSize int, bodySize int64, maxSize int64, isPartial bool) (Writer, error) {
return this.openWriter(key, expiresAt, status, headerSize, bodySize, maxSize, isPartial, false)
}
// OpenFlushWriter 打开从其他媒介直接刷入的写入器
func (this *FileStorage) OpenFlushWriter(key string, expiresAt int64, status int) (Writer, error) {
return this.openWriter(key, expiresAt, status, -1, -1, false, true)
func (this *FileStorage) OpenFlushWriter(key string, expiresAt int64, status int, headerSize int, bodySize int64) (Writer, error) {
return this.openWriter(key, expiresAt, status, headerSize, bodySize, -1, false, true)
}
func (this *FileStorage) openWriter(key string, expiredAt int64, status int, size int64, maxSize int64, isPartial bool, isFlushing bool) (Writer, error) {
func (this *FileStorage) openWriter(key string, expiredAt int64, status int, headerSize int, bodySize int64, maxSize int64, isPartial bool, isFlushing bool) (Writer, error) {
// 是否正在退出
if teaconst.IsQuiting {
return nil, ErrWritingUnavailable
}
// 当前磁盘可用容量是否严重不足
if this.diskIsFull {
return nil, NewCapacityError("the disk is full")
}
// 是否已忽略
if this.ignoreKeys.Has(key) {
return nil, ErrEntityTooLarge
@@ -421,8 +441,8 @@ func (this *FileStorage) openWriter(key string, expiredAt int64, status int, siz
maxMemorySize = maxSize
}
var memoryStorage = this.memoryStorage
if !isFlushing && !isPartial && memoryStorage != nil && ((size > 0 && size < maxMemorySize) || size < 0) {
writer, err := memoryStorage.OpenWriter(key, expiredAt, status, size, maxMemorySize, false)
if !isFlushing && !isPartial && memoryStorage != nil && ((bodySize > 0 && bodySize < maxMemorySize) || bodySize < 0) {
writer, err := memoryStorage.OpenWriter(key, expiredAt, status, headerSize, bodySize, maxMemorySize, false)
if err == nil {
return writer, nil
}
@@ -475,17 +495,9 @@ func (this *FileStorage) openWriter(key string, expiredAt int64, status int, siz
var hash = stringutil.Md5(key)
// TODO 可以只stat一次
var dir = this.options.Dir + "/p" + strconv.FormatInt(this.policy.Id, 10) + "/" + hash[:2] + "/" + hash[2:4]
_, err = os.Stat(dir)
if err != nil {
if !os.IsNotExist(err) {
return nil, err
}
err = os.MkdirAll(dir, 0777)
if err != nil {
return nil, err
}
dir, diskIsFull := this.subDir(hash)
if diskIsFull {
return nil, NewCapacityError("the disk is full")
}
// 检查缓存是否已经生成
@@ -532,19 +544,38 @@ func (this *FileStorage) openWriter(key string, expiredAt int64, status int, siz
// 从已经存储的内容中读取信息
var isNewCreated = true
var partialBodyOffset int64
var partialRanges *PartialRanges
if isPartial {
readerFp, err := os.OpenFile(tmpPath, os.O_RDONLY, 0444)
if err == nil {
var partialReader = NewPartialFileReader(readerFp)
err = partialReader.Init()
_ = partialReader.Close()
if err == nil && partialReader.bodyOffset > 0 {
isNewCreated = false
partialBodyOffset = partialReader.bodyOffset
} else {
_ = this.removeCacheFile(tmpPath)
// 数据库中是否存在
existsCacheItem, _ := this.list.Exist(hash)
if existsCacheItem {
readerFp, err := os.OpenFile(tmpPath, os.O_RDONLY, 0444)
if err == nil {
var partialReader = NewPartialFileReader(readerFp)
err = partialReader.Init()
_ = partialReader.Close()
if err == nil && partialReader.bodyOffset > 0 {
partialRanges = partialReader.Ranges()
if bodySize > 0 && partialRanges != nil && partialRanges.BodySize > 0 && bodySize != partialRanges.BodySize {
_ = this.removeCacheFile(tmpPath)
} else {
isNewCreated = false
partialBodyOffset = partialReader.bodyOffset
}
} else {
_ = this.removeCacheFile(tmpPath)
}
}
}
if isNewCreated {
err = this.list.Remove(hash)
if err != nil {
return nil, err
}
}
if partialRanges == nil {
partialRanges = NewPartialRanges(expiredAt)
}
}
var flags = os.O_CREATE | os.O_WRONLY
@@ -554,7 +585,16 @@ func (this *FileStorage) openWriter(key string, expiredAt int64, status int, siz
var before = time.Now()
writer, err := os.OpenFile(tmpPath, flags, 0666)
if err != nil {
return nil, err
// TODO 检查在各个系统中的稳定性
if os.IsNotExist(err) {
_ = os.MkdirAll(dir, 0777)
// open file again
writer, err = os.OpenFile(tmpPath, flags, 0666)
}
if err != nil {
return nil, err
}
}
if !isFlushing {
if time.Since(before) >= maxOpenFilesSlowCost {
@@ -586,9 +626,12 @@ func (this *FileStorage) openWriter(key string, expiredAt int64, status int, siz
return nil, ErrFileIsWriting
}
var metaBodySize int64 = -1
var metaHeaderSize = -1
if isNewCreated {
// 写入过期时间
var metaBytes = make([]byte, SizeMeta+len(key))
// 写入meta
// 从v0.5.8开始不再在meta中写入Key
var metaBytes = make([]byte, SizeMeta)
binary.BigEndian.PutUint32(metaBytes[OffsetExpiresAt:], uint32(expiredAt))
// 写入状态码
@@ -597,17 +640,17 @@ func (this *FileStorage) openWriter(key string, expiredAt int64, status int, siz
}
copy(metaBytes[OffsetStatus:], strconv.Itoa(status))
// 写入URL长度
binary.BigEndian.PutUint32(metaBytes[OffsetURLLength:], uint32(len(key)))
// 写入Header Length
binary.BigEndian.PutUint32(metaBytes[OffsetHeaderLength:], uint32(0))
if headerSize > 0 {
binary.BigEndian.PutUint32(metaBytes[OffsetHeaderLength:], uint32(headerSize))
metaHeaderSize = headerSize
}
// 写入Body Length
binary.BigEndian.PutUint64(metaBytes[OffsetBodyLength:], uint64(0))
// 写入URL
copy(metaBytes[OffsetKey:], key)
if bodySize > 0 {
binary.BigEndian.PutUint64(metaBytes[OffsetBodyLength:], uint64(bodySize))
metaBodySize = bodySize
}
_, err = writer.Write(metaBytes)
if err != nil {
@@ -617,12 +660,7 @@ func (this *FileStorage) openWriter(key string, expiredAt int64, status int, siz
isOk = true
if isPartial {
ranges, err := NewPartialRangesFromFile(cachePathName + "@ranges.cache")
if err != nil {
ranges = NewPartialRanges()
}
return NewPartialFileWriter(writer, key, expiredAt, isNewCreated, isPartial, partialBodyOffset, ranges, func() {
return NewPartialFileWriter(writer, key, expiredAt, metaHeaderSize, metaBodySize, isNewCreated, isPartial, partialBodyOffset, partialRanges, func() {
sharedWritingFileKeyLocker.Lock()
delete(sharedWritingFileKeyMap, key)
if len(sharedWritingFileKeyMap) == 0 {
@@ -631,7 +669,7 @@ func (this *FileStorage) openWriter(key string, expiredAt int64, status int, siz
sharedWritingFileKeyLocker.Unlock()
}), nil
} else {
return NewFileWriter(this, writer, key, expiredAt, -1, func() {
return NewFileWriter(this, writer, key, expiredAt, metaHeaderSize, metaBodySize, -1, func() {
sharedWritingFileKeyLocker.Lock()
delete(sharedWritingFileKeyMap, key)
if len(sharedWritingFileKeyMap) == 0 {
@@ -658,7 +696,7 @@ func (this *FileStorage) AddToList(item *Item) {
}
item.MetaSize = SizeMeta + 128
hash := stringutil.Md5(item.Key)
var hash = stringutil.Md5(item.Key)
err := this.list.Add(hash, item)
if err != nil && !strings.Contains(err.Error(), "UNIQUE constraint failed") {
remotelogs.Error("CACHE", "add to list failed: "+err.Error())
@@ -672,15 +710,12 @@ func (this *FileStorage) Delete(key string) error {
return nil
}
this.locker.Lock()
defer this.locker.Unlock()
// 先尝试内存缓存
this.runMemoryStorageSafety(func(memoryStorage *MemoryStorage) {
_ = memoryStorage.Delete(key)
})
hash, path := this.keyPath(key)
hash, path, _ := this.keyPath(key)
err := this.list.Remove(hash)
if err != nil {
return err
@@ -695,9 +730,6 @@ func (this *FileStorage) Delete(key string) error {
// Stat 统计
func (this *FileStorage) Stat() (*Stat, error) {
this.locker.RLock()
defer this.locker.RUnlock()
return this.list.Stat(func(hash string) bool {
return true
})
@@ -720,57 +752,72 @@ func (this *FileStorage) CleanAll() error {
// 删除缓存和目录
// 不能直接删除子目录,比较危险
dir := this.dir()
fp, err := os.Open(dir)
if err != nil {
return err
}
defer func() {
_ = fp.Close()
}()
stat, err := fp.Stat()
if err != nil {
return err
}
if !stat.IsDir() {
return nil
}
// 改成待删除
subDirs, err := fp.Readdir(-1)
if err != nil {
return err
}
for _, info := range subDirs {
subDir := info.Name()
// 检查目录名
ok, err := regexp.MatchString(`^[0-9a-f]{2}$`, subDir)
if err != nil {
return err
}
if !ok {
continue
var rootDirs = []string{this.options.Dir}
var subDirs = this.subDirs // copy slice
if len(subDirs) > 0 {
for _, subDir := range subDirs {
rootDirs = append(rootDirs, subDir.Path)
}
}
// 修改目录名
tmpDir := dir + "/" + subDir + "-deleted"
err = os.Rename(dir+"/"+subDir, tmpDir)
var dirNameReg = regexp.MustCompile(`^[0-9a-f]{2}$`)
for _, rootDir := range rootDirs {
var dir = rootDir + "/p" + types.String(this.policy.Id)
err = func(dir string) error {
fp, err := os.Open(dir)
if err != nil {
return err
}
defer func() {
_ = fp.Close()
}()
stat, err := fp.Stat()
if err != nil {
return err
}
if !stat.IsDir() {
return nil
}
// 改成待删除
subDirs, err := fp.Readdir(-1)
if err != nil {
return err
}
for _, info := range subDirs {
subDir := info.Name()
// 检查目录名
if !dirNameReg.MatchString(subDir) {
continue
}
// 修改目录名
tmpDir := dir + "/" + subDir + "-deleted"
err = os.Rename(dir+"/"+subDir, tmpDir)
if err != nil {
return err
}
}
// 重新遍历待删除
goman.New(func() {
err = this.cleanDeletedDirs(dir)
if err != nil {
remotelogs.Warn("CACHE", "delete '*-deleted' dirs failed: "+err.Error())
}
})
return nil
}(dir)
if err != nil {
return err
}
}
// 重新遍历待删除
goman.New(func() {
err = this.cleanDeletedDirs(dir)
if err != nil {
remotelogs.Warn("CACHE", "delete '*-deleted' dirs failed: "+err.Error())
}
})
return nil
}
@@ -781,9 +828,6 @@ func (this *FileStorage) Purge(keys []string, urlType string) error {
return nil
}
this.locker.Lock()
defer this.locker.Unlock()
// 先尝试内存缓存
this.runMemoryStorageSafety(func(memoryStorage *MemoryStorage) {
_ = memoryStorage.Purge(keys, urlType)
@@ -792,6 +836,19 @@ func (this *FileStorage) Purge(keys []string, urlType string) error {
// 目录
if urlType == "dir" {
for _, key := range keys {
// 检查是否有通配符 http(s)://*.example.com
var schemeIndex = strings.Index(key, "://")
if schemeIndex > 0 {
var keyRight = key[schemeIndex+3:]
if strings.HasPrefix(keyRight, "*.") {
err := this.list.CleanMatchPrefix(key)
if err != nil {
return err
}
continue
}
}
err := this.list.CleanPrefix(key)
if err != nil {
return err
@@ -802,7 +859,21 @@ func (this *FileStorage) Purge(keys []string, urlType string) error {
// URL
for _, key := range keys {
hash, path := this.keyPath(key)
// 检查是否有通配符 http(s)://*.example.com
var schemeIndex = strings.Index(key, "://")
if schemeIndex > 0 {
var keyRight = key[schemeIndex+3:]
if strings.HasPrefix(keyRight, "*.") {
err := this.list.CleanMatchKey(key)
if err != nil {
return err
}
continue
}
}
// 普通的Key
hash, path, _ := this.keyPath(key)
err := this.removeCacheFile(path)
if err != nil && !os.IsNotExist(err) {
return err
@@ -828,7 +899,10 @@ func (this *FileStorage) Stop() {
memoryStorage.Stop()
})
_ = this.list.Reset()
if this.list != nil {
_ = this.list.Reset()
}
if this.purgeTicker != nil {
this.purgeTicker.Stop()
}
@@ -836,7 +910,9 @@ func (this *FileStorage) Stop() {
this.hotTicker.Stop()
}
_ = this.list.Close()
if this.list != nil {
_ = this.list.Close()
}
var openFileCache = this.openFileCache
if openFileCache != nil {
@@ -873,25 +949,22 @@ func (this *FileStorage) CanSendfile() bool {
return this.options.EnableSendfile
}
// 绝对路径
func (this *FileStorage) dir() string {
return this.options.Dir + "/p" + strconv.FormatInt(this.policy.Id, 10) + "/"
}
// 获取Key对应的文件路径
func (this *FileStorage) keyPath(key string) (hash string, path string) {
func (this *FileStorage) keyPath(key string) (hash string, path string, diskIsFull bool) {
hash = stringutil.Md5(key)
dir := this.options.Dir + "/p" + strconv.FormatInt(this.policy.Id, 10) + "/" + hash[:2] + "/" + hash[2:4]
var dir string
dir, diskIsFull = this.subDir(hash)
path = dir + "/" + hash + ".cache"
return
}
// 获取Hash对应的文件路径
func (this *FileStorage) hashPath(hash string) (path string) {
func (this *FileStorage) hashPath(hash string) (path string, diskIsFull bool) {
if len(hash) != 32 {
return ""
return "", false
}
dir := this.options.Dir + "/p" + strconv.FormatInt(this.policy.Id, 10) + "/" + hash[:2] + "/" + hash[2:4]
var dir string
dir, diskIsFull = this.subDir(hash)
path = dir + "/" + hash + ".cache"
return
}
@@ -949,18 +1022,31 @@ func (this *FileStorage) initList() error {
}
// 清理任务
// TODO purge每个分区
func (this *FileStorage) purgeLoop() {
// 检查磁盘剩余空间
this.checkDiskSpace()
// 计算是否应该开启LFU清理
var capacityBytes = this.policy.CapacityBytes()
var capacityBytes = this.diskCapacityBytes()
var startLFU = false
var lfuFreePercent = this.policy.PersistenceLFUFreePercent
if lfuFreePercent <= 0 {
lfuFreePercent = 5
}
if this.diskIsFull {
var hasFullDisk = this.mainDiskIsFull
if !hasFullDisk {
var subDirs = this.subDirs // copy slice
for _, subDir := range subDirs {
if subDir.IsFull {
hasFullDisk = true
break
}
}
}
if hasFullDisk {
startLFU = true
} else {
var usedPercent = float32(this.TotalDiskSize()*100) / float32(capacityBytes)
@@ -993,7 +1079,7 @@ func (this *FileStorage) purgeLoop() {
}
for i := 0; i < times; i++ {
countFound, err := this.list.Purge(purgeCount, func(hash string) error {
path := this.hashPath(hash)
path, _ := this.hashPath(hash)
err := this.removeCacheFile(path)
if err != nil && !os.IsNotExist(err) {
remotelogs.Error("CACHE", "purge '"+path+"' error: "+err.Error())
@@ -1027,7 +1113,7 @@ func (this *FileStorage) purgeLoop() {
remotelogs.Println("CACHE", "LFU purge policy '"+this.policy.Name+"' id: "+types.String(this.policy.Id)+", count: "+types.String(count))
err := this.list.PurgeLFU(count, func(hash string) error {
path := this.hashPath(hash)
path, _ := this.hashPath(hash)
err := this.removeCacheFile(path)
if err != nil && !os.IsNotExist(err) {
remotelogs.Error("CACHE", "purge '"+path+"' error: "+err.Error())
@@ -1108,7 +1194,7 @@ func (this *FileStorage) hotLoop() {
expiresAt = bestExpiresAt
}
writer, err := memoryStorage.openWriter(item.Key, expiresAt, reader.Status(), reader.BodySize(), -1, false)
writer, err := memoryStorage.openWriter(item.Key, expiresAt, reader.Status(), types.Int(reader.HeaderSize()), reader.BodySize(), -1, false)
if err != nil {
if !CanIgnoreErr(err) {
remotelogs.Error("CACHE", "transfer hot item failed: "+err.Error())
@@ -1132,9 +1218,12 @@ func (this *FileStorage) hotLoop() {
}
err = reader.ReadBody(buf, func(n int) (goNext bool, err error) {
_, err = writer.Write(buf[:n])
if err == nil {
goNext = true
goNext = true
if n > 0 {
_, err = writer.Write(buf[:n])
if err != nil {
goNext = false
}
}
return
})
@@ -1147,6 +1236,7 @@ func (this *FileStorage) hotLoop() {
memoryStorage.AddToList(&Item{
Type: writer.ItemType(),
Key: item.Key,
Host: ParseHost(item.Key),
ExpiredAt: expiresAt,
HeaderSize: writer.HeaderSize(),
BodySize: writer.BodySize(),
@@ -1354,7 +1444,55 @@ func (this *FileStorage) checkDiskSpace() {
err := unix.Statfs(this.options.Dir, &stat)
if err == nil {
var availableBytes = stat.Bavail * uint64(stat.Bsize)
this.diskIsFull = availableBytes < MinDiskSpace
this.mainDiskIsFull = availableBytes < MinDiskSpace
}
}
var subDirs = this.subDirs // copy slice
for _, subDir := range subDirs {
var stat unix.Statfs_t
err := unix.Statfs(subDir.Path, &stat)
if err == nil {
var availableBytes = stat.Bavail * uint64(stat.Bsize)
subDir.IsFull = availableBytes < MinDiskSpace
}
}
}
// 获取目录
func (this *FileStorage) subDir(hash string) (dirPath string, dirIsFull bool) {
var suffix = "/p" + types.String(this.policy.Id) + "/" + hash[:2] + "/" + hash[2:4]
if len(hash) < 4 {
return this.options.Dir + suffix, this.mainDiskIsFull
}
var subDirs = this.subDirs // copy slice
var countSubDirs = len(subDirs)
if countSubDirs == 0 {
return this.options.Dir + suffix, this.mainDiskIsFull
}
countSubDirs++ // add main dir
// 最多只支持16个目录
if countSubDirs > 16 {
countSubDirs = 16
}
var dirIndex = this.charCode(hash[0]) % uint8(countSubDirs)
if dirIndex == 0 {
return this.options.Dir + suffix, this.mainDiskIsFull
}
var subDir = subDirs[dirIndex-1]
return subDir.Path + suffix, subDir.IsFull
}
func (this *FileStorage) charCode(r byte) uint8 {
if r >= '0' && r <= '9' {
return r - '0'
}
if r >= 'a' && r <= 'z' {
return r - 'a' + 10
}
return 0
}

View File

@@ -18,7 +18,7 @@ import (
)
func TestFileStorage_Init(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
@@ -26,6 +26,8 @@ func TestFileStorage_Init(t *testing.T) {
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -44,13 +46,16 @@ func TestFileStorage_Init(t *testing.T) {
}
func TestFileStorage_OpenWriter(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -62,7 +67,7 @@ func TestFileStorage_OpenWriter(t *testing.T) {
header := []byte("Header")
body := []byte("This is Body")
writer, err := storage.OpenWriter("my-key", time.Now().Unix()+86400, 200, -1, -1, false)
writer, err := storage.OpenWriter("my-key", time.Now().Unix()+86400, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -95,12 +100,15 @@ func TestFileStorage_OpenWriter_Partial(t *testing.T) {
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
}
writer, err := storage.OpenWriter("my-key", time.Now().Unix()+86400, 200, -1, -1, true)
writer, err := storage.OpenWriter("my-key", time.Now().Unix()+86400, 200, -1, -1, -1, true)
if err != nil {
t.Fatal(err)
}
@@ -123,13 +131,16 @@ func TestFileStorage_OpenWriter_Partial(t *testing.T) {
}
func TestFileStorage_OpenWriter_HTTP(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -139,7 +150,7 @@ func TestFileStorage_OpenWriter_HTTP(t *testing.T) {
t.Log(time.Since(now).Seconds()*1000, "ms")
}()
writer, err := storage.OpenWriter("my-http-response", time.Now().Unix()+86400, 200, -1, -1, false)
writer, err := storage.OpenWriter("my-http-response", time.Now().Unix()+86400, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -188,13 +199,16 @@ func TestFileStorage_OpenWriter_HTTP(t *testing.T) {
}
func TestFileStorage_Concurrent_Open_DifferentFile(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -212,7 +226,7 @@ func TestFileStorage_Concurrent_Open_DifferentFile(t *testing.T) {
go func(i int) {
defer wg.Done()
writer, err := storage.OpenWriter("abc"+strconv.Itoa(i), time.Now().Unix()+3600, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc"+strconv.Itoa(i), time.Now().Unix()+3600, 200, -1, -1, -1, false)
if err != nil {
if err != ErrFileIsWriting {
t.Error(err)
@@ -243,13 +257,16 @@ func TestFileStorage_Concurrent_Open_DifferentFile(t *testing.T) {
}
func TestFileStorage_Concurrent_Open_SameFile(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -267,7 +284,7 @@ func TestFileStorage_Concurrent_Open_SameFile(t *testing.T) {
go func(i int) {
defer wg.Done()
writer, err := storage.OpenWriter("abc"+strconv.Itoa(0), time.Now().Unix()+3600, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc"+strconv.Itoa(0), time.Now().Unix()+3600, 200, -1, -1, -1, false)
if err != nil {
if err != ErrFileIsWriting {
t.Error(err)
@@ -299,13 +316,16 @@ func TestFileStorage_Concurrent_Open_SameFile(t *testing.T) {
}
func TestFileStorage_Read(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -335,13 +355,16 @@ func TestFileStorage_Read(t *testing.T) {
}
func TestFileStorage_Read_HTTP_Response(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -388,13 +411,16 @@ func TestFileStorage_Read_HTTP_Response(t *testing.T) {
}
func TestFileStorage_Read_NotFound(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -421,13 +447,16 @@ func TestFileStorage_Read_NotFound(t *testing.T) {
}
func TestFileStorage_Delete(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -440,13 +469,16 @@ func TestFileStorage_Delete(t *testing.T) {
}
func TestFileStorage_Stat(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -465,13 +497,16 @@ func TestFileStorage_Stat(t *testing.T) {
}
func TestFileStorage_CleanAll(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -496,13 +531,16 @@ func TestFileStorage_CleanAll(t *testing.T) {
}
func TestFileStorage_Stop(t *testing.T) {
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -518,16 +556,22 @@ func TestFileStorage_DecodeFile(t *testing.T) {
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
t.Fatal(err)
}
_, path := storage.keyPath("my-key")
_, path, _ := storage.keyPath("my-key")
t.Log(path)
}
func TestFileStorage_RemoveCacheFile(t *testing.T) {
var storage = NewFileStorage(nil)
defer storage.Stop()
t.Log(storage.removeCacheFile("/Users/WorkSpace/EdgeProject/EdgeCache/p43/15/7e/157eba0dfc6dfb6fbbf20b1f9e584674.cache"))
}
@@ -536,13 +580,16 @@ func BenchmarkFileStorage_Read(b *testing.B) {
_ = utils.SetRLimit(1024 * 1024)
storage := NewFileStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewFileStorage(&serverconfigs.HTTPCachePolicy{
Id: 1,
IsOn: true,
Options: map[string]interface{}{
"dir": Tea.Root + "/caches",
},
})
defer storage.Stop()
err := storage.Init()
if err != nil {
b.Fatal(err)
@@ -569,6 +616,6 @@ func BenchmarkFileStorage_KeyPath(b *testing.B) {
}
for i := 0; i < b.N; i++ {
_, _ = storage.keyPath(strconv.Itoa(i))
_, _, _ = storage.keyPath(strconv.Itoa(i))
}
}

View File

@@ -14,10 +14,10 @@ type StorageInterface interface {
// OpenWriter 打开缓存写入器等待写入
// size 和 maxSize 可能为-1
OpenWriter(key string, expiresAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error)
OpenWriter(key string, expiresAt int64, status int, headerSize int, bodySize int64, maxSize int64, isPartial bool) (Writer, error)
// OpenFlushWriter 打开从其他媒介直接刷入的写入器
OpenFlushWriter(key string, expiresAt int64, status int) (Writer, error)
OpenFlushWriter(key string, expiresAt int64, status int, headerSize int, bodySize int64) (Writer, error)
// Delete 删除某个键值对应的缓存
Delete(key string) error

View File

@@ -1,12 +1,12 @@
package caches
import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/fasttime"
setutils "github.com/TeaOSLab/EdgeNode/internal/utils/sets"
"github.com/TeaOSLab/EdgeNode/internal/utils/sizes"
"github.com/TeaOSLab/EdgeNode/internal/zero"
@@ -17,6 +17,7 @@ import (
"math"
"runtime"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
@@ -32,7 +33,7 @@ type MemoryItem struct {
}
func (this *MemoryItem) IsExpired() bool {
return this.ExpiresAt < utils.UnixTime()
return this.ExpiresAt < fasttime.Now().Unix()
}
type MemoryStorage struct {
@@ -119,7 +120,7 @@ func (this *MemoryStorage) OpenReader(key string, useStale bool, isPartial bool)
return nil, ErrNotFound
}
if useStale || (item.ExpiresAt > utils.UnixTime()) {
if useStale || (item.ExpiresAt > fasttime.Now().Unix()) {
reader := NewMemoryReader(item)
err := reader.Init()
if err != nil {
@@ -149,7 +150,7 @@ func (this *MemoryStorage) OpenReader(key string, useStale bool, isPartial bool)
}
// OpenWriter 打开缓存写入器等待写入
func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error) {
func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int, headerSize int, bodySize int64, maxSize int64, isPartial bool) (Writer, error) {
if this.ignoreKeys.Has(key) {
return nil, ErrEntityTooLarge
}
@@ -158,15 +159,15 @@ func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int, s
if isPartial {
return nil, ErrFileIsWriting
}
return this.openWriter(key, expiredAt, status, size, maxSize, true)
return this.openWriter(key, expiredAt, status, headerSize, bodySize, maxSize, true)
}
// OpenFlushWriter 打开从其他媒介直接刷入的写入器
func (this *MemoryStorage) OpenFlushWriter(key string, expiresAt int64, status int) (Writer, error) {
return this.openWriter(key, expiresAt, status, -1, -1, true)
func (this *MemoryStorage) OpenFlushWriter(key string, expiresAt int64, status int, headerSize int, bodySize int64) (Writer, error) {
return this.openWriter(key, expiresAt, status, headerSize, bodySize, -1, true)
}
func (this *MemoryStorage) openWriter(key string, expiresAt int64, status int, size int64, maxSize int64, isDirty bool) (Writer, error) {
func (this *MemoryStorage) openWriter(key string, expiresAt int64, status int, headerSize int, bodySize int64, maxSize int64, isDirty bool) (Writer, error) {
// 待写入队列是否已满
if isDirty &&
this.parentStorage != nil &&
@@ -207,10 +208,10 @@ func (this *MemoryStorage) openWriter(key string, expiresAt int64, status int, s
return nil, NewCapacityError("write memory cache failed: too many keys in cache storage")
}
capacityBytes := this.memoryCapacityBytes()
if size < 0 {
size = 0
if bodySize < 0 {
bodySize = 0
}
if capacityBytes > 0 && capacityBytes <= this.totalSize+size {
if capacityBytes > 0 && capacityBytes <= this.totalSize+bodySize {
return nil, NewCapacityError("write memory cache failed: over memory size: " + strconv.FormatInt(capacityBytes, 10) + ", current size: " + strconv.FormatInt(this.totalSize, 10) + " bytes")
}
@@ -230,10 +231,10 @@ func (this *MemoryStorage) openWriter(key string, expiresAt int64, status int, s
// Delete 删除某个键值对应的缓存
func (this *MemoryStorage) Delete(key string) error {
hash := this.hash(key)
var hash = this.hash(key)
this.locker.Lock()
delete(this.valuesMap, hash)
_ = this.list.Remove(fmt.Sprintf("%d", hash))
_ = this.list.Remove(types.String(hash))
this.locker.Unlock()
return nil
}
@@ -263,6 +264,19 @@ func (this *MemoryStorage) Purge(keys []string, urlType string) error {
// 目录
if urlType == "dir" {
for _, key := range keys {
// 检查是否有通配符 http(s)://*.example.com
var schemeIndex = strings.Index(key, "://")
if schemeIndex > 0 {
var keyRight = key[schemeIndex+3:]
if strings.HasPrefix(keyRight, "*.") {
err := this.list.CleanMatchPrefix(key)
if err != nil {
return err
}
continue
}
}
err := this.list.CleanPrefix(key)
if err != nil {
return err
@@ -273,6 +287,19 @@ func (this *MemoryStorage) Purge(keys []string, urlType string) error {
// URL
for _, key := range keys {
// 检查是否有通配符 http(s)://*.example.com
var schemeIndex = strings.Index(key, "://")
if schemeIndex > 0 {
var keyRight = key[schemeIndex+3:]
if strings.HasPrefix(keyRight, "*.") {
err := this.list.CleanMatchKey(key)
if err != nil {
return err
}
continue
}
}
err := this.Delete(key)
if err != nil {
return err
@@ -336,7 +363,12 @@ func (this *MemoryStorage) CanUpdatePolicy(newPolicy *serverconfigs.HTTPCachePol
// AddToList 将缓存添加到列表
func (this *MemoryStorage) AddToList(item *Item) {
item.MetaSize = int64(len(item.Key)) + 128 /** 128是我们评估的数据结构的长度 **/
hash := fmt.Sprintf("%d", this.hash(item.Key))
var hash = types.String(this.hash(item.Key))
if len(item.Host) == 0 {
item.Host = ParseHost(item.Key)
}
_ = this.list.Add(hash, item)
}
@@ -433,7 +465,7 @@ func (this *MemoryStorage) startFlush() {
var statCount = 0
var writeDelayMS float64 = 0
for hash := range this.dirtyChan {
for key := range this.dirtyChan {
statCount++
if statCount == 100 {
@@ -455,7 +487,7 @@ func (this *MemoryStorage) startFlush() {
}
}
this.flushItem(hash)
this.flushItem(key)
if writeDelayMS > 0 {
time.Sleep(time.Duration(writeDelayMS) * time.Millisecond)
@@ -477,11 +509,15 @@ func (this *MemoryStorage) flushItem(key string) {
if !ok {
return
}
if !item.IsDone || item.IsExpired() {
if !item.IsDone {
remotelogs.Error("CACHE", "flush items failed: open writer failed: item has not been done")
return
}
if item.IsExpired() {
return
}
writer, err := this.parentStorage.OpenFlushWriter(key, item.ExpiresAt, item.Status)
writer, err := this.parentStorage.OpenFlushWriter(key, item.ExpiresAt, item.Status, len(item.HeaderValue), int64(len(item.BodyValue)))
if err != nil {
if !CanIgnoreErr(err) {
remotelogs.Error("CACHE", "flush items failed: open writer failed: "+err.Error())
@@ -513,6 +549,7 @@ func (this *MemoryStorage) flushItem(key string) {
this.parentStorage.AddToList(&Item{
Type: writer.ItemType(),
Key: key,
Host: ParseHost(key),
ExpiredAt: item.ExpiresAt,
HeaderSize: writer.HeaderSize(),
BodySize: writer.BodySize(),
@@ -542,7 +579,7 @@ func (this *MemoryStorage) memoryCapacityBytes() int64 {
func (this *MemoryStorage) deleteWithoutLocker(key string) error {
hash := this.hash(key)
delete(this.valuesMap, hash)
_ = this.list.Remove(fmt.Sprintf("%d", hash))
_ = this.list.Remove(types.String(hash))
return nil
}

View File

@@ -14,15 +14,22 @@ import (
)
func TestMemoryStorage_OpenWriter(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
var storage = NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
if err != nil {
t.Fatal(err)
}
_, _ = writer.WriteHeader([]byte("Header"))
_, _ = writer.Write([]byte("Hello"))
_, _ = writer.Write([]byte(", World"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
t.Log(storage.valuesMap)
{
@@ -30,6 +37,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
if err != nil {
if err == ErrNotFound {
t.Log("not found: abc")
return
} else {
t.Fatal(err)
}
@@ -63,7 +71,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
}
}
writer, err = storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, false)
writer, err = storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
@@ -102,21 +110,29 @@ func TestMemoryStorage_OpenReaderLock(t *testing.T) {
}
func TestMemoryStorage_Delete(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
var storage = NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
{
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
t.Log(len(storage.valuesMap))
}
{
writer, err := storage.OpenWriter("abc1", time.Now().Unix()+60, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc1", time.Now().Unix()+60, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
t.Log(len(storage.valuesMap))
}
_ = storage.Delete("abc1")
@@ -124,14 +140,18 @@ func TestMemoryStorage_Delete(t *testing.T) {
}
func TestMemoryStorage_Stat(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
var storage = NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
t.Log(len(storage.valuesMap))
storage.AddToList(&Item{
Key: "abc",
@@ -140,11 +160,15 @@ func TestMemoryStorage_Stat(t *testing.T) {
})
}
{
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
t.Log(len(storage.valuesMap))
storage.AddToList(&Item{
Key: "abc1",
@@ -161,14 +185,18 @@ func TestMemoryStorage_Stat(t *testing.T) {
}
func TestMemoryStorage_CleanAll(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
var storage = NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
var expiredAt = time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
storage.AddToList(&Item{
Key: "abc",
BodySize: 5,
@@ -176,11 +204,15 @@ func TestMemoryStorage_CleanAll(t *testing.T) {
})
}
{
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
storage.AddToList(&Item{
Key: "abc1",
BodySize: 5,
@@ -199,11 +231,15 @@ func TestMemoryStorage_Purge(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc", expiredAt, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
storage.AddToList(&Item{
Key: "abc",
BodySize: 5,
@@ -211,11 +247,15 @@ func TestMemoryStorage_Purge(t *testing.T) {
})
}
{
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, false)
writer, err := storage.OpenWriter("abc1", expiredAt, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
storage.AddToList(&Item{
Key: "abc1",
BodySize: 5,
@@ -231,7 +271,7 @@ func TestMemoryStorage_Purge(t *testing.T) {
}
func TestMemoryStorage_Expire(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{
var storage = NewMemoryStorage(&serverconfigs.HTTPCachePolicy{
MemoryAutoPurgeInterval: 5,
}, nil)
err := storage.Init()
@@ -242,11 +282,15 @@ func TestMemoryStorage_Expire(t *testing.T) {
for i := 0; i < 1000; i++ {
expiredAt := time.Now().Unix() + int64(rands.Int(0, 60))
key := "abc" + strconv.Itoa(i)
writer, err := storage.OpenWriter(key, expiredAt, 200, -1, -1, false)
writer, err := storage.OpenWriter(key, expiredAt, 200, -1, -1, -1, false)
if err != nil {
t.Fatal(err)
}
_, _ = writer.Write([]byte("Hello"))
err = writer.Close()
if err != nil {
t.Fatal(err)
}
storage.AddToList(&Item{
Key: key,
BodySize: 5,
@@ -257,7 +301,7 @@ func TestMemoryStorage_Expire(t *testing.T) {
}
func TestMemoryStorage_Locker(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
var storage = NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
err := storage.Init()
if err != nil {
t.Fatal(err)

30
internal/caches/utils.go Normal file
View File

@@ -0,0 +1,30 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package caches
import (
"github.com/TeaOSLab/EdgeCommon/pkg/configutils"
"net"
"strings"
)
func ParseHost(key string) string {
var schemeIndex = strings.Index(key, "://")
if schemeIndex <= 0 {
return ""
}
var firstSlashIndex = strings.Index(key[schemeIndex+3:], "/")
if firstSlashIndex <= 0 {
return ""
}
var host = key[schemeIndex+3 : schemeIndex+3+firstSlashIndex]
hostPart, _, err := net.SplitHostPort(host)
if err == nil && len(hostPart) > 0 {
host = configutils.QuoteIP(hostPart)
}
return host
}

View File

@@ -0,0 +1,51 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package caches_test
import (
"fmt"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/cespare/xxhash"
"github.com/iwind/TeaGo/types"
"strconv"
"testing"
)
func TestParseHost(t *testing.T) {
for _, u := range []string{
"https://goedge.cn/hello/world",
"https://goedge.cn:8080/hello/world",
"https://goedge.cn/hello/world?v=1&t=123",
"https://[::1]:1234/hello/world?v=1&t=123",
"https://[::1]/hello/world?v=1&t=123",
"https://127.0.0.1/hello/world?v=1&t=123",
"https:/hello/world?v=1&t=123",
"123456",
} {
t.Log(u, "=>", caches.ParseHost(u))
}
}
func TestUintString(t *testing.T) {
t.Log(strconv.FormatUint(xxhash.Sum64String("https://goedge.cn/"), 10))
t.Log(strconv.FormatUint(123456789, 10))
t.Log(fmt.Sprintf("%d", 1234567890123))
}
func BenchmarkUint_String(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = strconv.FormatUint(1234567890123, 10)
}
}
func BenchmarkUint_String2(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = types.String(1234567890123)
}
}
func BenchmarkUint_String3(b *testing.B) {
for i := 0; i < b.N; i++ {
_ = fmt.Sprintf("%d", 1234567890123)
}
}

View File

@@ -11,25 +11,32 @@ import (
)
type FileWriter struct {
storage StorageInterface
rawWriter *os.File
key string
headerSize int64
bodySize int64
expiredAt int64
maxSize int64
endFunc func()
once sync.Once
storage StorageInterface
rawWriter *os.File
key string
metaHeaderSize int
headerSize int64
metaBodySize int64 // 写入前的内容长度
bodySize int64
expiredAt int64
maxSize int64
endFunc func()
once sync.Once
}
func NewFileWriter(storage StorageInterface, rawWriter *os.File, key string, expiredAt int64, maxSize int64, endFunc func()) *FileWriter {
func NewFileWriter(storage StorageInterface, rawWriter *os.File, key string, expiredAt int64, metaHeaderSize int, metaBodySize int64, maxSize int64, endFunc func()) *FileWriter {
return &FileWriter{
storage: storage,
key: key,
rawWriter: rawWriter,
expiredAt: expiredAt,
maxSize: maxSize,
endFunc: endFunc,
storage: storage,
key: key,
rawWriter: rawWriter,
expiredAt: expiredAt,
maxSize: maxSize,
endFunc: endFunc,
metaHeaderSize: metaHeaderSize,
metaBodySize: metaBodySize,
}
}
@@ -45,7 +52,10 @@ func (this *FileWriter) WriteHeader(data []byte) (n int, err error) {
// WriteHeaderLength 写入Header长度数据
func (this *FileWriter) WriteHeaderLength(headerLength int) error {
bytes4 := make([]byte, 4)
if this.metaHeaderSize > 0 && this.metaHeaderSize == headerLength {
return nil
}
var bytes4 = make([]byte, 4)
binary.BigEndian.PutUint32(bytes4, uint32(headerLength))
_, err := this.rawWriter.Seek(SizeExpiresAt+SizeStatus+SizeURLLength, io.SeekStart)
if err != nil {
@@ -88,7 +98,10 @@ func (this *FileWriter) WriteAt(offset int64, data []byte) error {
// WriteBodyLength 写入Body长度数据
func (this *FileWriter) WriteBodyLength(bodyLength int64) error {
bytes8 := make([]byte, 8)
if this.metaBodySize >= 0 && bodyLength == this.metaBodySize {
return nil
}
var bytes8 = make([]byte, 8)
binary.BigEndian.PutUint64(bytes8, uint64(bodyLength))
_, err := this.rawWriter.Seek(SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength, io.SeekStart)
if err != nil {
@@ -109,7 +122,7 @@ func (this *FileWriter) Close() error {
this.endFunc()
})
path := this.rawWriter.Name()
var path = this.rawWriter.Name()
err := this.WriteHeaderLength(types.Int(this.headerSize))
if err != nil {

View File

@@ -11,13 +11,18 @@ import (
)
type PartialFileWriter struct {
rawWriter *os.File
key string
headerSize int64
bodySize int64
expiredAt int64
endFunc func()
once sync.Once
rawWriter *os.File
key string
metaHeaderSize int
headerSize int64
metaBodySize int64
bodySize int64
expiredAt int64
endFunc func()
once sync.Once
isNew bool
isPartial bool
@@ -27,17 +32,19 @@ type PartialFileWriter struct {
rangePath string
}
func NewPartialFileWriter(rawWriter *os.File, key string, expiredAt int64, isNew bool, isPartial bool, bodyOffset int64, ranges *PartialRanges, endFunc func()) *PartialFileWriter {
func NewPartialFileWriter(rawWriter *os.File, key string, expiredAt int64, metaHeaderSize int, metaBodySize int64, isNew bool, isPartial bool, bodyOffset int64, ranges *PartialRanges, endFunc func()) *PartialFileWriter {
return &PartialFileWriter{
key: key,
rawWriter: rawWriter,
expiredAt: expiredAt,
endFunc: endFunc,
isNew: isNew,
isPartial: isPartial,
bodyOffset: bodyOffset,
ranges: ranges,
rangePath: partialRangesFilePath(rawWriter.Name()),
key: key,
rawWriter: rawWriter,
expiredAt: expiredAt,
endFunc: endFunc,
isNew: isNew,
isPartial: isPartial,
bodyOffset: bodyOffset,
ranges: ranges,
rangePath: partialRangesFilePath(rawWriter.Name()),
metaHeaderSize: metaHeaderSize,
metaBodySize: metaBodySize,
}
}
@@ -71,7 +78,11 @@ func (this *PartialFileWriter) AppendHeader(data []byte) error {
// WriteHeaderLength 写入Header长度数据
func (this *PartialFileWriter) WriteHeaderLength(headerLength int) error {
bytes4 := make([]byte, 4)
if this.metaHeaderSize > 0 && this.metaHeaderSize == headerLength {
return nil
}
var bytes4 = make([]byte, 4)
binary.BigEndian.PutUint32(bytes4, uint32(headerLength))
_, err := this.rawWriter.Seek(SizeExpiresAt+SizeStatus+SizeURLLength, io.SeekStart)
if err != nil {
@@ -110,8 +121,13 @@ func (this *PartialFileWriter) WriteAt(offset int64, data []byte) error {
}
if this.bodyOffset == 0 {
this.bodyOffset = SizeMeta + int64(len(this.key)) + this.headerSize
var keyLength = 0
if this.ranges.Version == 0 { // 以往的版本包含有Key
keyLength = len(this.key)
}
this.bodyOffset = SizeMeta + int64(keyLength) + this.headerSize
}
_, err := this.rawWriter.WriteAt(data, this.bodyOffset+offset)
if err != nil {
return err
@@ -129,7 +145,10 @@ func (this *PartialFileWriter) SetBodyLength(bodyLength int64) {
// WriteBodyLength 写入Body长度数据
func (this *PartialFileWriter) WriteBodyLength(bodyLength int64) error {
bytes8 := make([]byte, 8)
if this.metaBodySize > 0 && this.metaBodySize == bodyLength {
return nil
}
var bytes8 = make([]byte, 8)
binary.BigEndian.PutUint64(bytes8, uint64(bodyLength))
_, err := this.rawWriter.Seek(SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength, io.SeekStart)
if err != nil {
@@ -150,8 +169,11 @@ func (this *PartialFileWriter) Close() error {
this.endFunc()
})
this.ranges.BodySize = this.bodySize
err := this.ranges.WriteToFile(this.rangePath)
if err != nil {
_ = this.rawWriter.Close()
this.remove()
return err
}

View File

@@ -26,8 +26,8 @@ func TestPartialFileWriter_Write(t *testing.T) {
if err != nil {
t.Fatal(err)
}
var ranges = caches.NewPartialRanges()
var writer = caches.NewPartialFileWriter(fp, "test", time.Now().Unix()+86500, true, true, 0, ranges, func() {
var ranges = caches.NewPartialRanges(0)
var writer = caches.NewPartialFileWriter(fp, "test", time.Now().Unix()+86500, -1, -1, true, true, 0, ranges, func() {
t.Log("end")
})
_, err = writer.WriteHeader([]byte("header"))

View File

@@ -11,7 +11,7 @@ import (
var sharedBrotliReaderPool *ReaderPool
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -11,7 +11,7 @@ import (
var sharedDeflateReaderPool *ReaderPool
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -11,7 +11,7 @@ import (
var sharedGzipReaderPool *ReaderPool
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -11,7 +11,7 @@ import (
var sharedZSTDReaderPool *ReaderPool
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -12,7 +12,7 @@ import (
var sharedBrotliWriterPool *WriterPool
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -12,7 +12,7 @@ import (
var sharedDeflateWriterPool *WriterPool
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -12,7 +12,7 @@ import (
var sharedGzipWriterPool *WriterPool
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -12,7 +12,7 @@ import (
var sharedZSTDWriterPool *WriterPool
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -9,11 +9,15 @@ import (
// APIConfig 节点API配置
type APIConfig struct {
RPC struct {
Endpoints []string `yaml:"endpoints"`
DisableUpdate bool `yaml:"disableUpdate"`
} `yaml:"rpc"`
NodeId string `yaml:"nodeId"`
Secret string `yaml:"secret"`
Endpoints []string `yaml:"endpoints" json:"endpoints"`
DisableUpdate bool `yaml:"disableUpdate" json:"disableUpdate"`
} `yaml:"rpc" json:"rpc"`
NodeId string `yaml:"nodeId" json:"nodeId"`
Secret string `yaml:"secret" json:"secret"`
}
func NewAPIConfig() *APIConfig {
return &APIConfig{}
}
func LoadAPIConfig() (*APIConfig, error) {

View File

@@ -3,9 +3,9 @@ package configs
// ClusterConfig 集群配置
type ClusterConfig struct {
RPC struct {
Endpoints []string `yaml:"endpoints"`
DisableUpdate bool `yaml:"disableUpdate"`
} `yaml:"rpc"`
ClusterId string `yaml:"clusterId"`
Secret string `yaml:"secret"`
Endpoints []string `yaml:"endpoints" json:"endpoints"`
DisableUpdate bool `yaml:"disableUpdate" json:"disableUpdate"`
} `yaml:"rpc" json:"rpc"`
ClusterId string `yaml:"clusterId" json:"clusterId"`
Secret string `yaml:"secret" json:"secret"`
}

7
internal/conns/linger.go Normal file
View File

@@ -0,0 +1,7 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package conns
type LingerConn interface {
SetLinger(sec int) error
}

View File

@@ -3,6 +3,7 @@
package conns
import (
"github.com/iwind/TeaGo/types"
"net"
"sync"
)
@@ -10,14 +11,14 @@ import (
var SharedMap = NewMap()
type Map struct {
m map[string]map[int]net.Conn // ip => { port => Conn }
m map[string]map[string]net.Conn // ip => { network_port => Conn }
locker sync.RWMutex
}
func NewMap() *Map {
return &Map{
m: map[string]map[int]net.Conn{},
m: map[string]map[string]net.Conn{},
}
}
@@ -25,23 +26,19 @@ func (this *Map) Add(conn net.Conn) {
if conn == nil {
return
}
tcpAddr, ok := conn.RemoteAddr().(*net.TCPAddr)
key, ip, ok := this.connAddr(conn)
if !ok {
return
}
var ip = tcpAddr.IP.String()
var port = tcpAddr.Port
this.locker.Lock()
defer this.locker.Unlock()
connMap, ok := this.m[ip]
if !ok {
this.m[ip] = map[int]net.Conn{
port: conn,
}
this.m[ip] = map[string]net.Conn{key: conn}
} else {
connMap[port] = conn
connMap[key] = conn
}
}
@@ -49,14 +46,11 @@ func (this *Map) Remove(conn net.Conn) {
if conn == nil {
return
}
tcpAddr, ok := conn.RemoteAddr().(*net.TCPAddr)
key, ip, ok := this.connAddr(conn)
if !ok {
return
}
var ip = tcpAddr.IP.String()
var port = tcpAddr.Port
this.locker.Lock()
defer this.locker.Unlock()
@@ -64,7 +58,7 @@ func (this *Map) Remove(conn net.Conn) {
if !ok {
return
}
delete(connMap, port)
delete(connMap, key)
if len(connMap) == 0 {
delete(this.m, ip)
@@ -96,6 +90,13 @@ func (this *Map) CloseIPConns(ip string) {
if ok {
for _, conn := range conns {
// 设置Linger
lingerConn, isLingerConn := conn.(LingerConn)
if isLingerConn {
_ = lingerConn.SetLinger(0)
}
// 关闭
_ = conn.Close()
}
@@ -109,9 +110,31 @@ func (this *Map) AllConns() []net.Conn {
var result = []net.Conn{}
for _, m := range this.m {
for _, conn := range m {
result = append(result, conn)
for _, connInfo := range m {
result = append(result, connInfo)
}
}
return result
}
func (this *Map) connAddr(conn net.Conn) (key string, ip string, ok bool) {
if conn == nil {
return
}
var addr = conn.RemoteAddr()
switch realAddr := addr.(type) {
case *net.TCPAddr:
return addr.Network() + types.String(realAddr.Port), realAddr.IP.String(), true
case *net.UDPAddr:
return addr.Network() + types.String(realAddr.Port), realAddr.IP.String(), true
default:
var s = addr.String()
host, port, err := net.SplitHostPort(s)
if err != nil {
return
}
return addr.Network() + port, host, true
}
}

View File

@@ -1,7 +1,7 @@
package teaconst
const (
Version = "0.5.6"
Version = "1.2.0"
ProductName = "Edge Node"
ProcessName = "edge-node"

View File

@@ -5,6 +5,7 @@ package teaconst
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"os"
"strings"
)
var (
@@ -15,7 +16,7 @@ var (
NodeId int64 = 0
NodeIdString = ""
IsDaemon = len(os.Args) > 1 && os.Args[1] == "daemon"
IsMain = checkMain()
GlobalProductName = nodeconfigs.DefaultProductName
@@ -24,3 +25,15 @@ var (
DiskIsFast = false // 是否为高速硬盘
)
// 检查是否为主程序
func checkMain() bool {
if len(os.Args) == 1 ||
(len(os.Args) >= 2 && os.Args[1] == "pprof") {
return true
}
exe, _ := os.Executable()
return strings.HasSuffix(exe, ".test") ||
strings.HasSuffix(exe, ".test.exe") ||
strings.Contains(exe, "___")
}

View File

@@ -9,6 +9,7 @@ import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/ddosconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
@@ -19,14 +20,18 @@ import (
"github.com/iwind/TeaGo/types"
stringutil "github.com/iwind/TeaGo/utils/string"
"net"
"os/exec"
"strings"
"sync"
"time"
)
var SharedDDoSProtectionManager = NewDDoSProtectionManager()
func init() {
if !teaconst.IsMain {
return
}
events.On(events.EventReload, func() {
if nftablesInstance == nil {
return
@@ -56,6 +61,8 @@ func init() {
type DDoSProtectionManager struct {
lastAllowIPList []string
lastConfig []byte
locker sync.Mutex
}
// NewDDoSProtectionManager 获取新对象
@@ -65,12 +72,18 @@ func NewDDoSProtectionManager() *DDoSProtectionManager {
// Apply 应用配置
func (this *DDoSProtectionManager) Apply(config *ddosconfigs.ProtectionConfig) error {
// 加锁防止并发更改
if !this.locker.TryLock() {
return nil
}
defer this.locker.Unlock()
// 同集群节点IP白名单
var allowIPListChanged = false
nodeConfig, _ := nodeconfigs.SharedNodeConfig()
if nodeConfig != nil {
var allowIPList = nodeConfig.AllowedIPs
if !utils.ContainsSameStrings(allowIPList, this.lastAllowIPList) {
if !utils.EqualStrings(allowIPList, this.lastAllowIPList) {
allowIPListChanged = true
this.lastAllowIPList = allowIPList
}
@@ -86,11 +99,14 @@ func (this *DDoSProtectionManager) Apply(config *ddosconfigs.ProtectionConfig) e
}
remotelogs.Println("FIREWALL", "change DDoS protection config")
if len(this.nftExe()) == 0 {
if len(nftables.NftExePath()) == 0 {
return errors.New("can not find nft command")
}
if nftablesInstance == nil {
if config == nil || !config.IsOn() {
return nil
}
return errors.New("nftables instance should not be nil")
}
@@ -149,7 +165,7 @@ func (this *DDoSProtectionManager) Apply(config *ddosconfigs.ProtectionConfig) e
// 添加TCP规则
func (this *DDoSProtectionManager) addTCPRules(tcpConfig *ddosconfigs.TCPConfig) error {
var nftExe = this.nftExe()
var nftExe = nftables.NftExePath()
if len(nftExe) == 0 {
return nil
}
@@ -538,7 +554,7 @@ func (this *DDoSProtectionManager) updateAllowIPList(allIPList []string) error {
_, ok := oldMap[ip]
if !ok {
// 不存在则添加
err = set.AddIPElement(ip, nil)
err = set.AddIPElement(ip, nil, false)
if err != nil {
return errors.New("add ip '" + ip + "' failed: " + err.Error())
}
@@ -549,8 +565,3 @@ func (this *DDoSProtectionManager) updateAllowIPList(allIPList []string) error {
return nil
}
func (this *DDoSProtectionManager) nftExe() string {
path, _ := exec.LookPath("nft")
return path
}

View File

@@ -3,6 +3,7 @@
package firewalls
import (
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"runtime"
@@ -14,6 +15,10 @@ var firewallLocker = &sync.Mutex{}
// 初始化
func init() {
if !teaconst.IsMain {
return
}
events.On(events.EventLoaded, func() {
var firewall = Firewall()
if firewall.Name() != "mock" {

View File

@@ -1,11 +1,11 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package firewalls
import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/configutils"
"github.com/TeaOSLab/EdgeNode/internal/conns"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
@@ -13,9 +13,9 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
executils "github.com/TeaOSLab/EdgeNode/internal/utils/exec"
"github.com/google/nftables/expr"
"github.com/iwind/TeaGo/types"
"net"
"os/exec"
"regexp"
"runtime"
"strings"
@@ -24,7 +24,7 @@ import (
// check nft status, if being enabled we load it automatically
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}
@@ -37,8 +37,8 @@ func init() {
ticker.Stop()
break
}
_, err := exec.LookPath("nft")
if err == nil {
var nftExe = nftables.NftExePath()
if len(nftExe) > 0 {
nftablesFirewall, err := NewNFTablesFirewall()
if err != nil {
continue
@@ -88,11 +88,15 @@ type blockIPItem struct {
}
func NewNFTablesFirewall() (*NFTablesFirewall, error) {
conn, err := nftables.NewConn()
if err != nil {
return nil, err
}
var firewall = &NFTablesFirewall{
conn: nftables.NewConn(),
conn: conn,
dropIPQueue: make(chan *blockIPItem, 4096),
}
err := firewall.init()
err = firewall.init()
if err != nil {
return nil, err
}
@@ -110,8 +114,8 @@ type NFTablesFirewall struct {
allowIPv4Set *nftables.Set
allowIPv6Set *nftables.Set
denyIPv4Set *nftables.Set
denyIPv6Set *nftables.Set
denyIPv4Sets []*nftables.Set
denyIPv6Sets []*nftables.Set
firewalld *Firewalld
@@ -120,9 +124,9 @@ type NFTablesFirewall struct {
func (this *NFTablesFirewall) init() error {
// check nft
nftPath, err := exec.LookPath("nft")
if err != nil {
return errors.New("nft not found")
var nftPath = nftables.NftExePath()
if len(nftPath) == 0 {
return errors.New("'nft' not found")
}
this.version = this.readVersion(nftPath)
@@ -186,7 +190,7 @@ func (this *NFTablesFirewall) init() error {
// allow set
// "allow" should be always first
for _, setAction := range []string{"allow", "deny"} {
for _, setAction := range []string{"allow", "deny", "deny1", "deny2", "deny3", "deny4"} {
var setName = setAction + "_set"
set, err := table.GetSet(setName)
@@ -216,32 +220,42 @@ func (this *NFTablesFirewall) init() error {
if setAction == "allow" {
this.allowIPv4Set = set
} else {
this.denyIPv4Set = set
this.denyIPv4Sets = append(this.denyIPv4Sets, set)
}
} else if tableDef.IsIPv6 {
if setAction == "allow" {
this.allowIPv6Set = set
} else {
this.denyIPv6Set = set
this.denyIPv6Sets = append(this.denyIPv6Sets, set)
}
}
// rule
var ruleName = []byte(setAction)
rule, err := chain.GetRuleWithUserData(ruleName)
// 将以前的drop规则删掉替换成后面的reject
if err == nil && setAction != "allow" && rule != nil && rule.VerDict() == expr.VerdictDrop {
deleteErr := chain.DeleteRule(rule)
if deleteErr == nil {
err = nftables.ErrRuleNotFound
rule = nil
}
}
if err != nil {
if nftables.IsNotFound(err) {
if tableDef.IsIPv4 {
if setAction == "allow" {
rule, err = chain.AddAcceptIPv4SetRule(setName, ruleName)
} else {
rule, err = chain.AddDropIPv4SetRule(setName, ruleName)
rule, err = chain.AddRejectIPv4SetRule(setName, ruleName)
}
} else if tableDef.IsIPv6 {
if setAction == "allow" {
rule, err = chain.AddAcceptIPv6SetRule(setName, ruleName)
} else {
rule, err = chain.AddDropIPv6SetRule(setName, ruleName)
rule, err = chain.AddRejectIPv6SetRule(setName, ruleName)
}
}
if err != nil {
@@ -265,7 +279,7 @@ func (this *NFTablesFirewall) init() error {
for ipItem := range this.dropIPQueue {
switch ipItem.action {
case "drop":
err = this.DropSourceIP(ipItem.ip, ipItem.timeoutSeconds, false)
err := this.DropSourceIP(ipItem.ip, ipItem.timeoutSeconds, false)
if err != nil {
remotelogs.Warn("NFTABLES", "drop ip '"+ipItem.ip+"' failed: "+err.Error())
}
@@ -324,14 +338,14 @@ func (this *NFTablesFirewall) AllowSourceIP(ip string) error {
if this.allowIPv6Set == nil {
return errors.New("ipv6 ip set is nil")
}
return this.allowIPv6Set.AddElement(data.To16(), nil)
return this.allowIPv6Set.AddElement(data.To16(), nil, false)
}
// ipv4
if this.allowIPv4Set == nil {
return errors.New("ipv4 ip set is nil")
}
return this.allowIPv4Set.AddElement(data.To4(), nil)
return this.allowIPv4Set.AddElement(data.To4(), nil, false)
}
// RejectSourceIP 拒绝某个源IP连接
@@ -371,22 +385,23 @@ func (this *NFTablesFirewall) DropSourceIP(ip string, timeoutSeconds int, async
// 再次尝试关闭连接
defer conns.SharedMap.CloseIPConns(ip)
var ipLong = configutils.IPString2Long(ip)
if strings.Contains(ip, ":") { // ipv6
if this.denyIPv6Set == nil {
return errors.New("ipv6 ip set is nil")
if len(this.denyIPv6Sets) == 0 {
return errors.New("ipv6 ip set not found")
}
return this.denyIPv6Set.AddElement(data.To16(), &nftables.ElementOptions{
return this.denyIPv6Sets[ipLong%uint64(len(this.denyIPv6Sets))].AddElement(data.To16(), &nftables.ElementOptions{
Timeout: time.Duration(timeoutSeconds) * time.Second,
})
}, false)
}
// ipv4
if this.denyIPv4Set == nil {
return errors.New("ipv4 ip set is nil")
if len(this.denyIPv4Sets) == 0 {
return errors.New("ipv4 ip set not found")
}
return this.denyIPv4Set.AddElement(data.To4(), &nftables.ElementOptions{
return this.denyIPv4Sets[ipLong%uint64(len(this.denyIPv4Sets))].AddElement(data.To4(), &nftables.ElementOptions{
Timeout: time.Duration(timeoutSeconds) * time.Second,
})
}, false)
}
// RemoveSourceIP 删除某个源IP
@@ -396,9 +411,10 @@ func (this *NFTablesFirewall) RemoveSourceIP(ip string) error {
return errors.New("invalid ip '" + ip + "'")
}
var ipLong = configutils.IPString2Long(ip)
if strings.Contains(ip, ":") { // ipv6
if this.denyIPv6Set != nil {
err := this.denyIPv6Set.DeleteElement(data.To16())
if len(this.denyIPv6Sets) > 0 {
err := this.denyIPv6Sets[ipLong%uint64(len(this.denyIPv6Sets))].DeleteElement(data.To16())
if err != nil {
return err
}
@@ -415,13 +431,14 @@ func (this *NFTablesFirewall) RemoveSourceIP(ip string) error {
}
// ipv4
if this.allowIPv4Set != nil {
err := this.denyIPv4Set.DeleteElement(data.To4())
if len(this.denyIPv4Sets) > 0 {
err := this.denyIPv4Sets[ipLong%uint64(len(this.denyIPv4Sets))].DeleteElement(data.To4())
if err != nil {
return err
}
err = this.allowIPv4Set.DeleteElement(data.To4())
}
if this.allowIPv4Set != nil {
err := this.allowIPv4Set.DeleteElement(data.To4())
if err != nil {
return err
}

View File

@@ -1,6 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables

View File

@@ -1,4 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
package nftables

View File

@@ -1,6 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables_test
@@ -11,7 +10,10 @@ import (
)
func getIPv4Chain(t *testing.T) *nftables.Chain {
var conn = nftables.NewConn()
conn, err := nftables.NewConn()
if err != nil {
t.Fatal(err)
}
table, err := conn.GetTable("test_ipv4", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {

View File

@@ -15,10 +15,14 @@ type Conn struct {
rawConn *nft.Conn
}
func NewConn() *Conn {
return &Conn{
rawConn: &nft.Conn{},
func NewConn() (*Conn, error) {
conn, err := nft.New()
if err != nil {
return nil, err
}
return &Conn{
rawConn: conn,
}, nil
}
func (this *Conn) Raw() *nft.Conn {

View File

@@ -4,7 +4,10 @@
package nftables
import "errors"
import (
"errors"
"strings"
)
var ErrTableNotFound = errors.New("table not found")
var ErrChainNotFound = errors.New("chain not found")
@@ -15,5 +18,5 @@ func IsNotFound(err error) bool {
if err == nil {
return false
}
return err == ErrTableNotFound || err == ErrChainNotFound || err == ErrSetNotFound || err == ErrRuleNotFound
return err == ErrTableNotFound || err == ErrChainNotFound || err == ErrSetNotFound || err == ErrRuleNotFound || strings.Contains(err.Error(), "no such file or directory")
}

View File

@@ -0,0 +1,65 @@
// Copyright 2023 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package nftables
import (
"sync"
"time"
)
type Expiration struct {
m map[string]time.Time // key => expires time
lastGCAt int64
locker sync.RWMutex
}
func NewExpiration() *Expiration {
return &Expiration{
m: map[string]time.Time{},
}
}
func (this *Expiration) AddUnsafe(key []byte, expires time.Time) {
this.m[string(key)] = expires
}
func (this *Expiration) Add(key []byte, expires time.Time) {
this.locker.Lock()
this.m[string(key)] = expires
this.gc()
this.locker.Unlock()
}
func (this *Expiration) Remove(key []byte) {
this.locker.Lock()
delete(this.m, string(key))
this.locker.Unlock()
}
func (this *Expiration) Contains(key []byte) bool {
this.locker.RLock()
expires, ok := this.m[string(key)]
if ok && expires.Year() > 2000 && time.Now().After(expires) {
ok = false
}
this.locker.RUnlock()
return ok
}
func (this *Expiration) gc() {
// we won't gc too frequently
var currentTime = time.Now().Unix()
if this.lastGCAt >= currentTime {
return
}
this.lastGCAt = currentTime
var now = time.Now().Add(-10 * time.Second) // gc elements expired before 10 seconds ago
for key, expires := range this.m {
if expires.Year() > 2000 && now.After(expires) {
delete(this.m, key)
}
}
}

View File

@@ -0,0 +1,59 @@
// Copyright 2023 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package nftables_test
import (
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
"net"
"testing"
"time"
)
func TestExpiration_Add(t *testing.T) {
var expiration = nftables.NewExpiration()
{
expiration.Add([]byte{'a', 'b', 'c'}, time.Now())
t.Log(expiration.Contains([]byte{'a', 'b', 'c'}))
}
{
expiration.Add([]byte{'a', 'b', 'c'}, time.Now().Add(1*time.Second))
t.Log(expiration.Contains([]byte{'a', 'b', 'c'}))
}
{
expiration.Add([]byte{'a', 'b', 'c'}, time.Time{})
t.Log(expiration.Contains([]byte{'a', 'b', 'c'}))
}
{
expiration.Add([]byte{'a', 'b', 'c'}, time.Now().Add(-1*time.Second))
t.Log(expiration.Contains([]byte{'a', 'b', 'c'}))
}
{
expiration.Add([]byte{'a', 'b', 'c'}, time.Now().Add(-10*time.Second))
t.Log(expiration.Contains([]byte{'a', 'b', 'c'}))
}
{
expiration.Add([]byte{'a', 'b', 'c'}, time.Now().Add(1*time.Second))
expiration.Remove([]byte{'a', 'b', 'c'})
t.Log(expiration.Contains([]byte{'a', 'b', 'c'}))
}
{
expiration.Add(net.ParseIP("10.254.0.75").To4(), time.Now())
t.Log(expiration.Contains(net.ParseIP("10.254.0.75").To4()))
}
}
func BenchmarkNewExpiration(b *testing.B) {
var expiration = nftables.NewExpiration()
for i := 0; i < 10_000; i++ {
expiration.Add([]byte(types.String(types.String(rands.Int(0, 255))+"."+types.String(rands.Int(0, 255))+"."+types.String(rands.Int(0, 255))+"."+types.String(rands.Int(0, 255)))), time.Now().Add(3600*time.Second))
}
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
expiration.Add([]byte(types.String(types.String(rands.Int(0, 255))+"."+types.String(rands.Int(0, 255))+"."+types.String(rands.Int(0, 255))+"."+types.String(rands.Int(0, 255)))), time.Now().Add(3600*time.Second))
}
})
}

View File

@@ -1,4 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
package nftables

View File

@@ -1,10 +1,12 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
//go:build linux
package nftables
import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
@@ -17,6 +19,10 @@ import (
)
func init() {
if !teaconst.IsMain {
return
}
events.On(events.EventReload, func() {
// linux only
if runtime.GOOS != "linux" {
@@ -33,8 +39,7 @@ func init() {
}
if os.Getgid() == 0 { // root user only
_, err := exec.LookPath("nft")
if err == nil {
if len(NftExePath()) > 0 {
return
}
goman.New(func() {
@@ -48,6 +53,25 @@ func init() {
})
}
// NftExePath 查找nftables可执行文件路径
func NftExePath() string {
path, _ := exec.LookPath("nft")
if len(path) > 0 {
return path
}
for _, possiblePath := range []string{
"/usr/sbin/nft",
} {
_, err := os.Stat(possiblePath)
if err == nil {
return possiblePath
}
}
return ""
}
type Installer struct {
}
@@ -62,8 +86,7 @@ func (this *Installer) Install() error {
}
// 检查是否已经存在
_, err := exec.LookPath("nft")
if err == nil {
if len(NftExePath()) > 0 {
return nil
}

View File

@@ -1,4 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
package nftables

View File

@@ -1,6 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
@@ -35,17 +34,25 @@ type Set struct {
conn *Conn
rawSet *nft.Set
batch *SetBatch
expiration *Expiration
}
func NewSet(conn *Conn, rawSet *nft.Set) *Set {
return &Set{
conn: conn,
rawSet: rawSet,
var set = &Set{
conn: conn,
rawSet: rawSet,
expiration: nil,
batch: &SetBatch{
conn: conn,
rawSet: rawSet,
},
}
// retrieve set elements to improve "delete" speed
set.initElements()
return set
}
func (this *Set) Raw() *nft.Set {
@@ -56,12 +63,22 @@ func (this *Set) Name() string {
return this.rawSet.Name
}
func (this *Set) AddElement(key []byte, options *ElementOptions) error {
func (this *Set) AddElement(key []byte, options *ElementOptions, overwrite bool) error {
// check if already exists
if this.expiration != nil && !overwrite && this.expiration.Contains(key) {
return nil
}
var expiresTime = time.Time{}
var rawElement = nft.SetElement{
Key: key,
}
if options != nil {
rawElement.Timeout = options.Timeout
if options.Timeout > 0 {
expiresTime = time.UnixMilli(time.Now().UnixMilli() + options.Timeout.Milliseconds())
}
}
err := this.conn.Raw().SetAddElements(this.rawSet, []nft.SetElement{
rawElement,
@@ -71,9 +88,19 @@ func (this *Set) AddElement(key []byte, options *ElementOptions) error {
}
err = this.conn.Commit()
if err != nil {
if err == nil {
if this.expiration != nil {
this.expiration.Add(key, expiresTime)
}
} else {
var isFileExistsErr = strings.Contains(err.Error(), "file exists")
if !overwrite && isFileExistsErr {
// ignore file exists error
return nil
}
// retry if exists
if strings.Contains(err.Error(), "file exists") {
if overwrite && isFileExistsErr {
deleteErr := this.conn.Raw().SetDeleteElements(this.rawSet, []nft.SetElement{
{
Key: key,
@@ -85,6 +112,11 @@ func (this *Set) AddElement(key []byte, options *ElementOptions) error {
})
if err == nil {
err = this.conn.Commit()
if err == nil {
if this.expiration != nil {
this.expiration.Add(key, expiresTime)
}
}
}
}
}
@@ -93,20 +125,25 @@ func (this *Set) AddElement(key []byte, options *ElementOptions) error {
return err
}
func (this *Set) AddIPElement(ip string, options *ElementOptions) error {
func (this *Set) AddIPElement(ip string, options *ElementOptions, overwrite bool) error {
var ipObj = net.ParseIP(ip)
if ipObj == nil {
return errors.New("invalid ip '" + ip + "'")
}
if utils.IsIPv4(ip) {
return this.AddElement(ipObj.To4(), options)
return this.AddElement(ipObj.To4(), options, overwrite)
} else {
return this.AddElement(ipObj.To16(), options)
return this.AddElement(ipObj.To16(), options, overwrite)
}
}
func (this *Set) DeleteElement(key []byte) error {
// if set element does not exist, we return immediately
if this.expiration != nil && !this.expiration.Contains(key) {
return nil
}
err := this.conn.Raw().SetDeleteElements(this.rawSet, []nft.SetElement{
{
Key: key,
@@ -116,9 +153,17 @@ func (this *Set) DeleteElement(key []byte) error {
return err
}
err = this.conn.Commit()
if err != nil {
if err == nil {
if this.expiration != nil {
this.expiration.Remove(key)
}
} else {
if strings.Contains(err.Error(), "no such file or directory") {
err = nil
if this.expiration != nil {
this.expiration.Remove(key)
}
}
}
return err

View File

@@ -1,4 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
package nftables

View File

@@ -0,0 +1,8 @@
// Copyright 2023 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
//go:build linux && !plus
package nftables
func (this *Set) initElements() {
// NOT IMPLEMENTED
}

View File

@@ -34,7 +34,7 @@ func getIPv4Set(t *testing.T) *nftables.Set {
func TestSet_AddElement(t *testing.T) {
var set = getIPv4Set(t)
err := set.AddElement(net.ParseIP("192.168.2.31").To4(), &nftables.ElementOptions{Timeout: 86400 * time.Second})
err := set.AddElement(net.ParseIP("192.168.2.31").To4(), &nftables.ElementOptions{Timeout: 86400 * time.Second}, false)
if err != nil {
t.Fatal(err)
}

View File

@@ -1,6 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables

View File

@@ -1,6 +1,5 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables_test
@@ -10,7 +9,10 @@ import (
)
func getIPv4Table(t *testing.T) *nftables.Table {
var conn = nftables.NewConn()
conn, err := nftables.NewConn()
if err != nil {
t.Fatal(err)
}
table, err := conn.GetTable("test_ipv4", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {

View File

@@ -0,0 +1,30 @@
// Copyright 2023 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package firewalls
import (
"time"
)
// DropTemporaryTo 使用本地防火墙临时拦截IP数据包
func DropTemporaryTo(ip string, expiresAt int64) {
// 如果为0则表示是长期有效
if expiresAt <= 0 {
expiresAt = time.Now().Unix() + 3600
}
var timeout = expiresAt - time.Now().Unix()
if timeout < 1 {
return
}
if timeout > 3600 {
timeout = 3600
}
// 使用本地防火墙延长封禁
var fw = Firewall()
if fw != nil && !fw.IsMock() {
// 这里 int(int64) 转换的前提是限制了 timeout <= 3600否则将有整型溢出的风险
_ = fw.DropSourceIP(ip, int(timeout), true)
}
}

View File

@@ -15,7 +15,7 @@ var instanceId = uint64(0)
// New 新创建goroutine
func New(f func()) {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}
@@ -47,7 +47,7 @@ func New(f func()) {
// NewWithArgs 创建带有参数的goroutine
func NewWithArgs(f func(args ...interface{}), args ...interface{}) {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}

View File

@@ -1,6 +1,8 @@
package iplibrary
import "github.com/TeaOSLab/EdgeNode/internal/utils"
import (
"github.com/TeaOSLab/EdgeNode/internal/utils/fasttime"
)
type IPItemType = string
@@ -45,7 +47,7 @@ func (this *IPItem) containsIPv4(ip uint64) bool {
return false
}
}
if this.ExpiredAt > 0 && this.ExpiredAt < utils.UnixTime() {
if this.ExpiredAt > 0 && this.ExpiredAt < fasttime.Now().Unix() {
return false
}
return true
@@ -56,7 +58,7 @@ func (this *IPItem) containsIPv6(ip uint64) bool {
if this.IPFrom != ip {
return false
}
if this.ExpiredAt > 0 && this.ExpiredAt < utils.UnixTime() {
if this.ExpiredAt > 0 && this.ExpiredAt < fasttime.Now().Unix() {
return false
}
return true
@@ -64,7 +66,7 @@ func (this *IPItem) containsIPv6(ip uint64) bool {
// 检查是否包所有IP
func (this *IPItem) containsAll() bool {
if this.ExpiredAt > 0 && this.ExpiredAt < utils.UnixTime() {
if this.ExpiredAt > 0 && this.ExpiredAt < fasttime.Now().Unix() {
return false
}
return true

View File

@@ -3,6 +3,7 @@ package iplibrary
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/expires"
"github.com/TeaOSLab/EdgeNode/internal/utils/fasttime"
"sort"
"sync"
)
@@ -72,6 +73,25 @@ func (this *IPList) Contains(ip uint64) bool {
return item != nil
}
// ContainsExpires 判断是否包含某个IP
func (this *IPList) ContainsExpires(ip uint64) (expiresAt int64, ok bool) {
this.locker.RLock()
if len(this.allItemsMap) > 0 {
this.locker.RUnlock()
return 0, true
}
var item = this.lookupIP(ip)
this.locker.RUnlock()
if item == nil {
return
}
return item.ExpiredAt, true
}
// ContainsIPStrings 是否包含一组IP中的任意一个并返回匹配的第一个Item
func (this *IPList) ContainsIPStrings(ipStrings []string) (item *IPItem, found bool) {
if len(ipStrings) == 0 {
@@ -110,7 +130,7 @@ func (this *IPList) addItem(item *IPItem, sortable bool) {
return
}
if item.ExpiredAt > 0 && item.ExpiredAt < utils.UnixTime() {
if item.ExpiredAt > 0 && item.ExpiredAt < fasttime.Now().Unix() {
return
}
@@ -155,7 +175,7 @@ func (this *IPList) addItem(item *IPItem, sortable bool) {
this.locker.Unlock()
}
// 对列表进行排序
// 对列表进行排序
func (this *IPList) sortItems() {
sort.Slice(this.sortedItems, func(i, j int) bool {
var item1 = this.sortedItems[i]

View File

@@ -3,28 +3,31 @@
package iplibrary
import (
"database/sql"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils/dbs"
"github.com/iwind/TeaGo/Tea"
_ "github.com/mattn/go-sqlite3"
"os"
"path/filepath"
"time"
)
type IPListDB struct {
db *sql.DB
db *dbs.DB
itemTableName string
itemTableName string
versionTableName string
deleteExpiredItemsStmt *sql.Stmt
deleteItemStmt *sql.Stmt
insertItemStmt *sql.Stmt
selectItemsStmt *sql.Stmt
selectMaxVersionStmt *sql.Stmt
deleteExpiredItemsStmt *dbs.Stmt
deleteItemStmt *dbs.Stmt
insertItemStmt *dbs.Stmt
selectItemsStmt *dbs.Stmt
selectMaxItemVersionStmt *dbs.Stmt
selectVersionStmt *dbs.Stmt
updateVersionStmt *dbs.Stmt
cleanTicker *time.Ticker
@@ -35,9 +38,10 @@ type IPListDB struct {
func NewIPListDB() (*IPListDB, error) {
var db = &IPListDB{
itemTableName: "ipItems",
dir: filepath.Clean(Tea.Root + "/data"),
cleanTicker: time.NewTicker(24 * time.Hour),
itemTableName: "ipItems",
versionTableName: "versions",
dir: filepath.Clean(Tea.Root + "/data"),
cleanTicker: time.NewTicker(24 * time.Hour),
}
err := db.init()
return db, err
@@ -56,7 +60,7 @@ func (this *IPListDB) init() error {
var path = this.dir + "/ip_list.db"
db, err := sql.Open("sqlite3", "file:"+path+"?cache=shared&mode=rwc&_journal_mode=WAL&_sync=OFF")
db, err := dbs.OpenWriter("file:" + path + "?cache=shared&mode=rwc&_journal_mode=WAL&_sync=OFF&_locking_mode=EXCLUSIVE")
if err != nil {
return err
}
@@ -109,6 +113,15 @@ ON "` + this.itemTableName + `" (
return err
}
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.versionTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"version" integer DEFAULT 0
);
`)
if err != nil {
return err
}
// 初始化SQL语句
this.deleteExpiredItemsStmt, err = this.db.Prepare(`DELETE FROM "` + this.itemTableName + `" WHERE "expiredAt">0 AND "expiredAt"<?`)
if err != nil {
@@ -130,7 +143,20 @@ ON "` + this.itemTableName + `" (
return err
}
this.selectMaxVersionStmt, err = this.db.Prepare(`SELECT "version" FROM "` + this.itemTableName + `" ORDER BY "id" DESC LIMIT 1`)
this.selectMaxItemVersionStmt, err = this.db.Prepare(`SELECT "version" FROM "` + this.itemTableName + `" ORDER BY "id" DESC LIMIT 1`)
if err != nil {
return err
}
this.selectVersionStmt, err = this.db.Prepare(`SELECT "version" FROM "` + this.versionTableName + `" LIMIT 1`)
if err != nil {
return err
}
this.updateVersionStmt, err = this.db.Prepare(`REPLACE INTO "` + this.versionTableName + `" ("id", "version") VALUES (1, ?)`)
if err != nil {
return err
}
this.db = db
@@ -173,11 +199,15 @@ func (this *IPListDB) AddItem(item *pb.IPItem) error {
// 如果是删除,则不再创建新记录
if item.IsDeleted {
return nil
return this.UpdateMaxVersion(item.Version)
}
_, err = this.insertItemStmt.Exec(item.ListId, item.ListType, item.IsGlobal, item.Type, item.Id, item.IpFrom, item.IpTo, item.ExpiredAt, item.EventLevel, item.IsDeleted, item.Version, item.NodeId, item.ServerId)
return err
if err != nil {
return err
}
return this.UpdateMaxVersion(item.Version)
}
func (this *IPListDB) ReadItems(offset int64, size int64) (items []*pb.IPItem, err error) {
@@ -211,27 +241,63 @@ func (this *IPListDB) ReadMaxVersion() int64 {
return 0
}
var row = this.selectMaxVersionStmt.QueryRow()
if row == nil {
return 0
// from version table
{
var row = this.selectVersionStmt.QueryRow()
if row == nil {
return 0
}
var version int64
err := row.Scan(&version)
if err == nil {
return version
}
}
var version int64
err := row.Scan(&version)
if err != nil {
return 0
// from items table
{
var row = this.selectMaxItemVersionStmt.QueryRow()
if row == nil {
return 0
}
var version int64
err := row.Scan(&version)
if err != nil {
return 0
}
return version
}
return version
}
// UpdateMaxVersion 修改版本号
func (this *IPListDB) UpdateMaxVersion(version int64) error {
if this.isClosed {
return nil
}
_, err := this.updateVersionStmt.Exec(version)
return err
}
func (this *IPListDB) Close() error {
this.isClosed = true
if this.db != nil {
_ = this.deleteExpiredItemsStmt.Close()
_ = this.deleteItemStmt.Close()
_ = this.insertItemStmt.Close()
_ = this.selectItemsStmt.Close()
_ = this.selectMaxVersionStmt.Close()
for _, stmt := range []*dbs.Stmt{
this.deleteExpiredItemsStmt,
this.deleteItemStmt,
this.insertItemStmt,
this.selectItemsStmt,
this.selectMaxItemVersionStmt, // ipItems table
this.selectVersionStmt, // versions table
this.updateVersionStmt,
} {
if stmt != nil {
_ = stmt.Close()
}
}
return this.db.Close()
}

View File

@@ -79,3 +79,15 @@ func TestIPListDB_ReadMaxVersion(t *testing.T) {
}
t.Log(db.ReadMaxVersion())
}
func TestIPListDB_UpdateMaxVersion(t *testing.T) {
db, err := iplibrary.NewIPListDB()
if err != nil {
t.Fatal(err)
}
err = db.UpdateMaxVersion(1027)
if err != nil {
t.Fatal(err)
}
t.Log(db.ReadMaxVersion())
}

View File

@@ -10,50 +10,54 @@ import (
// AllowIP 检查IP是否被允许访问
// 如果一个IP不在任何名单中则允许访问
func AllowIP(ip string, serverId int64) (canGoNext bool, inAllowList bool) {
func AllowIP(ip string, serverId int64) (canGoNext bool, inAllowList bool, expiresAt int64) {
if !Tea.IsTesting() { // 如果在测试环境,我们不加入一些白名单,以便于可以在本地和局域网正常测试
// 放行lo
if ip == "127.0.0.1" || ip == "::1" {
return true, true
return true, true, 0
}
// check node
nodeConfig, err := nodeconfigs.SharedNodeConfig()
if err == nil && nodeConfig.IPIsAutoAllowed(ip) {
return true, true
return true, true, 0
}
}
var ipLong = utils.IP2Long(ip)
if ipLong == 0 {
return false, false
return false, false, 0
}
// check white lists
if GlobalWhiteIPList.Contains(ipLong) {
return true, true
return true, true, 0
}
if serverId > 0 {
var list = SharedServerListManager.FindWhiteList(serverId, false)
if list != nil && list.Contains(ipLong) {
return true, true
return true, true, 0
}
}
// check black lists
if GlobalBlackIPList.Contains(ipLong) {
return false, false
expiresAt, ok := GlobalBlackIPList.ContainsExpires(ipLong)
if ok {
return false, false, expiresAt
}
if serverId > 0 {
var list = SharedServerListManager.FindBlackList(serverId, false)
if list != nil && list.Contains(ipLong) {
return false, false
if list != nil {
expiresAt, ok = list.ContainsExpires(ipLong)
if ok {
return false, false, expiresAt
}
}
}
return true, false
return true, false, 0
}
// IsInWhiteList 检查IP是否在白名单中
@@ -73,7 +77,7 @@ func AllowIPStrings(ipStrings []string, serverId int64) bool {
return true
}
for _, ip := range ipStrings {
isAllowed, _ := AllowIP(ip, serverId)
isAllowed, _, _ := AllowIP(ip, serverId)
if !isAllowed {
return false
}

View File

@@ -20,7 +20,7 @@ var SharedIPListManager = NewIPListManager()
var IPListUpdateNotify = make(chan bool, 1)
func init() {
if teaconst.IsDaemon {
if !teaconst.IsMain {
return
}
@@ -47,17 +47,20 @@ type IPListManager struct {
db *IPListDB
version int64
pageSize int64
lastVersion int64
fetchPageSize int64
listMap map[int64]*IPList
locker sync.Mutex
isFirstTime bool
}
func NewIPListManager() *IPListManager {
return &IPListManager{
pageSize: 1000,
listMap: map[int64]*IPList{},
fetchPageSize: 5_000,
listMap: map[int64]*IPList{},
isFirstTime: true,
}
}
@@ -117,11 +120,11 @@ func (this *IPListManager) init() {
_ = db.DeleteExpiredItems()
// 本地数据库中最大版本号
this.version = db.ReadMaxVersion()
this.lastVersion = db.ReadMaxVersion()
// 从本地数据库中加载
var offset int64 = 0
var size int64 = 1000
var size int64 = 2_000
for {
items, err := db.ReadItems(offset, size)
var l = len(items)
@@ -148,6 +151,11 @@ func (this *IPListManager) loop() error {
return nil
}
// 第一次同步则打印信息
if this.isFirstTime {
remotelogs.Println("IP_LIST_MANAGER", "initializing ip items ...")
}
for {
hasNext, err := this.fetch()
if err != nil {
@@ -159,6 +167,12 @@ func (this *IPListManager) loop() error {
time.Sleep(1 * time.Second)
}
// 第一次同步则打印信息
if this.isFirstTime {
this.isFirstTime = false
remotelogs.Println("IP_LIST_MANAGER", "finished initializing ip items")
}
return nil
}
@@ -168,8 +182,8 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
return false, err
}
itemsResp, err := rpcClient.IPItemRPC.ListIPItemsAfterVersion(rpcClient.Context(), &pb.ListIPItemsAfterVersionRequest{
Version: this.version,
Size: this.pageSize,
Version: this.lastVersion,
Size: this.fetchPageSize,
})
if err != nil {
if rpc.IsConnError(err) {
@@ -211,6 +225,7 @@ func (this *IPListManager) DeleteExpiredItems() {
}
}
// 处理IP条目
func (this *IPListManager) processItems(items []*pb.IPItem, fromRemote bool) {
var changedLists = map[*IPList]zero.Zero{}
for _, item := range items {
@@ -280,8 +295,8 @@ func (this *IPListManager) processItems(items []*pb.IPItem, fromRemote bool) {
if fromRemote {
var latestVersion = items[len(items)-1].Version
if latestVersion > this.version {
this.version = latestVersion
if latestVersion > this.lastVersion {
this.lastVersion = latestVersion
}
}
}

View File

@@ -30,7 +30,6 @@ func TestIPListManager_check(t *testing.T) {
func TestIPListManager_loop(t *testing.T) {
manager := NewIPListManager()
manager.Start()
manager.pageSize = 10
err := manager.loop()
if err != nil {
t.Fatal(err)

View File

@@ -4,6 +4,7 @@ package metrics
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"strconv"
@@ -13,6 +14,10 @@ import (
var SharedManager = NewManager()
func init() {
if !teaconst.IsMain {
return
}
events.On(events.EventQuit, func() {
SharedManager.Quit()
})

View File

@@ -3,7 +3,6 @@
package metrics
import (
"database/sql"
"encoding/json"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
@@ -17,7 +16,6 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/types"
_ "github.com/mattn/go-sqlite3"
"os"
"strconv"
"sync"
@@ -50,11 +48,11 @@ type Task struct {
cleanVersion int32
insertStatStmt *sql.Stmt
deleteByVersionStmt *sql.Stmt
deleteByExpiresTimeStmt *sql.Stmt
selectTopStmt *sql.Stmt
sumStmt *sql.Stmt
insertStatStmt *dbs.Stmt
deleteByVersionStmt *dbs.Stmt
deleteByExpiresTimeStmt *dbs.Stmt
selectTopStmt *dbs.Stmt
sumStmt *dbs.Stmt
serverIdMap map[int64]zero.Zero // 所有的服务Ids
timeMap map[string]zero.Zero // time => bool
@@ -92,12 +90,12 @@ func (this *Task) Init() error {
var path = dir + "/metric." + types.String(this.item.Id) + ".db"
db, err := sql.Open("sqlite3", "file:"+path+"?cache=shared&mode=rwc&_journal_mode=WAL&_sync=OFF")
db, err := dbs.OpenWriter("file:" + path + "?cache=shared&mode=rwc&_journal_mode=WAL&_sync=OFF&_locking_mode=EXCLUSIVE")
if err != nil {
return err
}
db.SetMaxOpenConns(1)
this.db = dbs.NewDB(db)
this.db = db
// 恢复数据库
var recoverEnv, _ = os.LookupEnv("EdgeRecover")

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/metrics"
"github.com/TeaOSLab/EdgeNode/internal/utils/testutils"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/rands"
"testing"
@@ -79,6 +80,10 @@ func TestTask_Add(t *testing.T) {
}
func TestTask_Add_Many(t *testing.T) {
if !testutils.IsSingleTesting() {
return
}
var task = metrics.NewTask(&serverconfigs.MetricItemConfig{
Id: 1,
IsOn: false,

View File

@@ -5,6 +5,7 @@ package monitor
import (
"encoding/json"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
@@ -16,6 +17,10 @@ import (
var SharedValueQueue = NewValueQueue()
func init() {
if !teaconst.IsMain {
return
}
events.On(events.EventLoaded, func() {
goman.New(func() {
SharedValueQueue.Start()

View File

@@ -178,7 +178,7 @@ func (this *APIStream) handleWriteCache(message *pb.NodeStreamMessage) error {
}
expiredAt := time.Now().Unix() + msg.LifeSeconds
writer, err := storage.OpenWriter(msg.Key, expiredAt, 200, int64(len(msg.Value)), -1, false)
writer, err := storage.OpenWriter(msg.Key, expiredAt, 200, -1, int64(len(msg.Value)), -1, false)
if err != nil {
this.replyFail(message.RequestId, "prepare writing failed: "+err.Error())
return err
@@ -407,7 +407,7 @@ func (this *APIStream) handleCheckLocalFirewall(message *pb.NodeStreamMessage) e
var protectionConfig = sharedNodeConfig.DDoSProtection
err = firewalls.SharedDDoSProtectionManager.Apply(protectionConfig)
if err != nil {
this.replyFail(message.RequestId, dataMessage.Name+"was installed, but apply DDoS protection config failed: "+err.Error())
this.replyFail(message.RequestId, dataMessage.Name+" was installed, but apply DDoS protection config failed: "+err.Error())
} else {
this.replyOk(message.RequestId, string(result.AsJSON()))
}

View File

@@ -3,6 +3,7 @@
package nodes
import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/firewallconfigs"
"github.com/TeaOSLab/EdgeNode/internal/conns"
@@ -11,6 +12,8 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/stats"
"github.com/TeaOSLab/EdgeNode/internal/ttlcache"
"github.com/TeaOSLab/EdgeNode/internal/utils"
connutils "github.com/TeaOSLab/EdgeNode/internal/utils/conns"
"github.com/TeaOSLab/EdgeNode/internal/utils/fasttime"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/types"
@@ -25,29 +28,58 @@ import (
type ClientConn struct {
BaseClientConn
isTLS bool
hasDeadline bool
hasRead bool
createdAt int64
isTLS bool
isHTTP bool
hasRead bool
isLO bool // 是否为环路
isNoStat bool // 是否不统计带宽
isInAllowList bool
hasResetSYNFlood bool
lastReadAt int64
lastWriteAt int64
lastErr error
readDeadlineTime int64
isShortReading bool // reading header or tls handshake
isDebugging bool
autoReadTimeout bool
autoWriteTimeout bool
}
func NewClientConn(rawConn net.Conn, isTLS bool, quickClose bool, isInAllowList bool) net.Conn {
func NewClientConn(rawConn net.Conn, isHTTP bool, isTLS bool, isInAllowList bool) net.Conn {
// 是否为环路
var remoteAddr = rawConn.RemoteAddr().String()
var isLO = strings.HasPrefix(remoteAddr, "127.0.0.1:") || strings.HasPrefix(remoteAddr, "[::1]:")
var conn = &ClientConn{
BaseClientConn: BaseClientConn{rawConn: rawConn},
isTLS: isTLS,
isLO: isLO,
isHTTP: isHTTP,
isLO: strings.HasPrefix(remoteAddr, "127.0.0.1:") || strings.HasPrefix(remoteAddr, "[::1]:"),
isNoStat: connutils.IsNoStatConn(remoteAddr),
isInAllowList: isInAllowList,
createdAt: fasttime.Now().Unix(),
}
if quickClose {
if existsLnNodeIP(conn.RawIP()) {
conn.SetIsPersistent(true)
}
// 超时等设置
var globalServerConfig = sharedNodeConfig.GlobalServerConfig
if globalServerConfig != nil {
var performanceConfig = globalServerConfig.Performance
conn.isDebugging = performanceConfig.Debug
conn.autoReadTimeout = performanceConfig.AutoReadTimeout
conn.autoWriteTimeout = performanceConfig.AutoWriteTimeout
}
if isHTTP {
// TODO 可以在配置中设置此值
_ = conn.SetLinger(nodeconfigs.DefaultTCPLinger)
}
@@ -59,6 +91,18 @@ func NewClientConn(rawConn net.Conn, isTLS bool, quickClose bool, isInAllowList
}
func (this *ClientConn) Read(b []byte) (n int, err error) {
if this.isDebugging {
this.lastReadAt = fasttime.Now().Unix()
defer func() {
if err != nil {
this.lastErr = errors.New("read error: " + err.Error())
} else {
this.lastErr = nil
}
}()
}
// 环路直接读取
if this.isLO {
n, err = this.rawConn.Read(b)
@@ -68,34 +112,29 @@ func (this *ClientConn) Read(b []byte) (n int, err error) {
return
}
// TLS
if this.isTLS {
if !this.hasDeadline {
_ = this.rawConn.SetReadDeadline(time.Now().Add(time.Duration(nodeconfigs.DefaultTLSHandshakeTimeout) * time.Second)) // TODO 握手超时时间可以设置
this.hasDeadline = true
defer func() {
_ = this.rawConn.SetReadDeadline(time.Time{})
}()
}
// 设置读超时时间
if this.isHTTP && !this.isPersistent && !this.isShortReading && this.autoReadTimeout {
this.setHTTPReadTimeout()
}
// 开始读取
n, err = this.rawConn.Read(b)
if n > 0 {
atomic.AddUint64(&teaconst.InTrafficBytes, uint64(n))
if !this.hasRead {
this.hasRead = true
}
this.hasRead = true
}
// 检测是否为握手错误
var isHandshakeError = err != nil && os.IsTimeout(err) && !this.hasRead
if isHandshakeError {
// 检测是否为超时错误
var isTimeout = err != nil && os.IsTimeout(err)
var isHandshakeError = isTimeout && !this.hasRead
if isTimeout {
_ = this.SetLinger(0)
} else {
_ = this.SetLinger(nodeconfigs.DefaultTCPLinger)
}
// 忽略白名单和局域网
if !this.isInAllowList && !utils.IsLocalIP(this.RawIP()) {
if !this.isPersistent && this.isHTTP && !this.isInAllowList && !utils.IsLocalIP(this.RawIP()) {
// SYN Flood检测
if this.serverId == 0 || !this.hasResetSYNFlood {
var synFloodConfig = sharedNodeConfig.SYNFloodConfig()
@@ -114,17 +153,67 @@ func (this *ClientConn) Read(b []byte) (n int, err error) {
}
func (this *ClientConn) Write(b []byte) (n int, err error) {
if len(b) == 0 {
return 0, nil
}
if this.isDebugging {
this.lastWriteAt = fasttime.Now().Unix()
defer func() {
if err != nil {
this.lastErr = errors.New("write error: " + err.Error())
} else {
this.lastErr = nil
}
}()
}
// 设置写超时时间
if !this.isPersistent && this.autoWriteTimeout {
var timeoutSeconds = len(b) / 1024
if timeoutSeconds < 3 {
timeoutSeconds = 3
}
_ = this.rawConn.SetWriteDeadline(time.Now().Add(time.Duration(timeoutSeconds) * time.Second)) // TODO 时间可以设置
}
// 延长读超时时间
if this.isHTTP && !this.isPersistent && this.autoReadTimeout {
this.setHTTPReadTimeout()
}
// 开始写入
var before = time.Now()
n, err = this.rawConn.Write(b)
if n > 0 {
// 统计当前服务带宽
if this.serverId > 0 {
if !this.isLO || Tea.IsTesting() { // 环路不统计带宽,避免缓存预热等行为产生带宽
// TODO 需要加入在serverId绑定之前的带宽
if !this.isNoStat || Tea.IsTesting() { // 环路不统计带宽,避免缓存预热等行为产生带宽
atomic.AddUint64(&teaconst.OutTrafficBytes, uint64(n))
stats.SharedBandwidthStatManager.Add(this.userId, this.serverId, int64(n))
var cost = time.Since(before).Seconds()
if cost > 1 {
stats.SharedBandwidthStatManager.AddBandwidth(this.userId, this.serverId, int64(float64(n)/cost), int64(n))
} else {
stats.SharedBandwidthStatManager.AddBandwidth(this.userId, this.serverId, int64(n), int64(n))
}
}
}
}
// 如果是写入超时,则立即关闭连接
if err != nil && os.IsTimeout(err) {
// TODO 考虑对多次慢连接的IP做出惩罚
conn, ok := this.rawConn.(LingerConn)
if ok {
_ = conn.SetLinger(0)
}
_ = this.Close()
}
return
}
@@ -156,6 +245,26 @@ func (this *ClientConn) SetDeadline(t time.Time) error {
}
func (this *ClientConn) SetReadDeadline(t time.Time) error {
// 如果开启了HTTP自动读超时选项则自动控制超时时间
if this.isHTTP && !this.isPersistent && this.autoReadTimeout {
this.isShortReading = false
var unixTime = t.Unix()
if unixTime < 10 {
return nil
}
if unixTime == this.readDeadlineTime {
return nil
}
this.readDeadlineTime = unixTime
var seconds = -time.Since(t)
if seconds <= 0 || seconds > HTTPIdleTimeout {
return nil
}
if seconds < HTTPIdleTimeout-1*time.Second {
this.isShortReading = true
}
}
return this.rawConn.SetReadDeadline(t)
}
@@ -163,6 +272,22 @@ func (this *ClientConn) SetWriteDeadline(t time.Time) error {
return this.rawConn.SetWriteDeadline(t)
}
func (this *ClientConn) CreatedAt() int64 {
return this.createdAt
}
func (this *ClientConn) LastReadAt() int64 {
return this.lastReadAt
}
func (this *ClientConn) LastWriteAt() int64 {
return this.lastWriteAt
}
func (this *ClientConn) LastErr() error {
return this.lastErr
}
func (this *ClientConn) resetSYNFlood() {
ttlcache.SharedCache.Delete("SYN_FLOOD:" + this.RawIP())
}
@@ -170,7 +295,7 @@ func (this *ClientConn) resetSYNFlood() {
func (this *ClientConn) increaseSYNFlood(synFloodConfig *firewallconfigs.SYNFloodConfig) {
var ip = this.RawIP()
if len(ip) > 0 && !iplibrary.IsInWhiteList(ip) && (!synFloodConfig.IgnoreLocal || !utils.IsLocalIP(ip)) {
var timestamp = utils.NextMinuteUnixTime()
var timestamp = fasttime.Now().UnixNextMinute()
var result = ttlcache.SharedCache.IncreaseInt64("SYN_FLOOD:"+ip, 1, timestamp, true)
var minAttempts = synFloodConfig.MinAttempts
if minAttempts < 5 {
@@ -190,7 +315,12 @@ func (this *ClientConn) increaseSYNFlood(synFloodConfig *firewallconfigs.SYNFloo
_ = this.SetLinger(0)
_ = this.Close()
waf.SharedIPBlackList.RecordIP(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip, time.Now().Unix()+int64(timeout), 0, true, 0, 0, "疑似SYN Flood攻击当前1分钟"+types.String(result)+"次空连接")
waf.SharedIPBlackList.RecordIP(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip, fasttime.Now().Unix()+int64(timeout), 0, true, 0, 0, "疑似SYN Flood攻击当前1分钟"+types.String(result)+"次空连接")
}
}
}
// 设置读超时时间
func (this *ClientConn) setHTTPReadTimeout() {
_ = this.SetReadDeadline(time.Now().Add(HTTPIdleTimeout))
}

View File

@@ -4,6 +4,8 @@ package nodes
import (
"crypto/tls"
"github.com/TeaOSLab/EdgeNode/internal/firewalls"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"net"
)
@@ -16,6 +18,9 @@ type BaseClientConn struct {
remoteAddr string
hasLimit bool
isPersistent bool // 是否为持久化连接
fingerprint []byte
isClosed bool
rawIP string
@@ -45,7 +50,21 @@ func (this *BaseClientConn) Bind(serverId int64, remoteAddr string, maxConnsPerS
}
// SetServerId 设置服务ID
func (this *BaseClientConn) SetServerId(serverId int64) {
func (this *BaseClientConn) SetServerId(serverId int64) (goNext bool) {
goNext = true
// 检查服务相关IP黑名单
var rawIP = this.RawIP()
if serverId > 0 && len(rawIP) > 0 {
// 是否在白名单中
ok, _, expiresAt := iplibrary.AllowIP(rawIP, serverId)
if !ok {
_ = this.rawConn.Close()
firewalls.DropTemporaryTo(rawIP, expiresAt)
return false
}
}
this.serverId = serverId
// 设置包装前连接
@@ -58,6 +77,8 @@ func (this *BaseClientConn) SetServerId(serverId int64) {
case *ClientConn:
conn.SetServerId(serverId)
}
return true
}
// ServerId 读取当前连接绑定的服务ID
@@ -103,8 +124,8 @@ func (this *BaseClientConn) TCPConn() (tcpConn *net.TCPConn, ok bool) {
switch conn := this.rawConn.(type) {
case *tls.Conn:
var internalConn = conn.NetConn()
clientConn, ok := internalConn.(*ClientConn)
if ok {
clientConn, isClientConn := internalConn.(*ClientConn)
if isClientConn {
return clientConn.TCPConn()
}
tcpConn, ok = internalConn.(*net.TCPConn)
@@ -122,3 +143,17 @@ func (this *BaseClientConn) SetLinger(seconds int) error {
}
return nil
}
func (this *BaseClientConn) SetIsPersistent(isPersistent bool) {
this.isPersistent = isPersistent
}
// SetFingerprint 设置指纹信息
func (this *BaseClientConn) SetFingerprint(fingerprint []byte) {
this.fingerprint = fingerprint
}
// Fingerprint 读取指纹信息
func (this *BaseClientConn) Fingerprint() []byte {
return this.fingerprint
}

View File

@@ -16,11 +16,20 @@ type ClientConnInterface interface {
ServerId() int64
// SetServerId 设置服务ID
SetServerId(serverId int64)
SetServerId(serverId int64) (goNext bool)
// SetUserId 设置所属服务的用户ID
SetUserId(userId int64)
// UserId 获取当前连接所属服务的用户ID
UserId() int64
// SetIsPersistent 设置是否为持久化
SetIsPersistent(isPersistent bool)
// SetFingerprint 设置指纹信息
SetFingerprint(fingerprint []byte)
// Fingerprint 读取指纹信息
Fingerprint() []byte
}

View File

@@ -15,6 +15,10 @@ import (
// 发送监控流量
func init() {
if !teaconst.IsMain {
return
}
events.On(events.EventStart, func() {
var ticker = time.NewTicker(1 * time.Minute)
goman.New(func() {

View File

@@ -8,20 +8,19 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"net"
"time"
)
// ClientListener 客户端网络监听
type ClientListener struct {
rawListener net.Listener
isHTTP bool
isTLS bool
quickClose bool
}
func NewClientListener(listener net.Listener, quickClose bool) *ClientListener {
func NewClientListener(listener net.Listener, isHTTP bool) *ClientListener {
return &ClientListener{
rawListener: listener,
quickClose: quickClose,
isHTTP: isHTTP,
}
}
@@ -43,25 +42,17 @@ func (this *ClientListener) Accept() (net.Conn, error) {
ip, _, err := net.SplitHostPort(conn.RemoteAddr().String())
var isInAllowList = false
if err == nil {
canGoNext, inAllowList := iplibrary.AllowIP(ip, 0)
canGoNext, inAllowList, expiresAt := iplibrary.AllowIP(ip, 0)
isInAllowList = inAllowList
if !waf.SharedIPWhiteList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip) {
expiresAt, ok := waf.SharedIPBlackList.ContainsExpires(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip)
if ok {
var timeout = expiresAt - time.Now().Unix()
if timeout > 0 {
if !canGoNext {
firewalls.DropTemporaryTo(ip, expiresAt)
} else {
if !waf.SharedIPWhiteList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip) {
var ok = false
expiresAt, ok = waf.SharedIPBlackList.ContainsExpires(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip)
if ok {
canGoNext = false
if timeout > 3600 {
timeout = 3600
}
// 使用本地防火墙延长封禁
var fw = firewalls.Firewall()
if fw != nil && !fw.IsMock() {
// 这里 int(int64) 转换的前提是限制了 timeout <= 3600否则将有整型溢出的风险
_ = fw.DropSourceIP(ip, int(timeout), true)
}
firewalls.DropTemporaryTo(ip, expiresAt)
}
}
}
@@ -78,7 +69,7 @@ func (this *ClientListener) Accept() (net.Conn, error) {
}
}
return NewClientConn(conn, this.isTLS, this.quickClose, isInAllowList), nil
return NewClientConn(conn, this.isHTTP, this.isTLS, isInAllowList), nil
}
func (this *ClientListener) Close() error {

View File

@@ -55,3 +55,30 @@ func (this *ClientTLSConn) SetReadDeadline(t time.Time) error {
func (this *ClientTLSConn) SetWriteDeadline(t time.Time) error {
return this.rawConn.SetWriteDeadline(t)
}
func (this *ClientTLSConn) SetIsPersistent(isPersistent bool) {
tlsConn, ok := this.rawConn.(*tls.Conn)
if ok {
var rawConn = tlsConn.NetConn()
if rawConn != nil {
clientConn, ok := rawConn.(*ClientConn)
if ok {
clientConn.SetIsPersistent(isPersistent)
}
}
}
}
func (this *ClientTLSConn) Fingerprint() []byte {
tlsConn, ok := this.rawConn.(*tls.Conn)
if ok {
var rawConn = tlsConn.NetConn()
if rawConn != nil {
clientConn, ok := rawConn.(*ClientConn)
if ok {
return clientConn.fingerprint
}
}
}
return nil
}

View File

@@ -11,6 +11,7 @@ import (
"google.golang.org/grpc/status"
"strings"
"time"
"unicode/utf8"
)
var sharedHTTPAccessLogQueue = NewHTTPAccessLogQueue()
@@ -25,9 +26,12 @@ type HTTPAccessLogQueue struct {
// NewHTTPAccessLogQueue 获取新对象
func NewHTTPAccessLogQueue() *HTTPAccessLogQueue {
// 队列中最大的值,超出此数量的访问日志会被丢弃
// TODO 需要可以在界面中设置
maxSize := 20000
queue := &HTTPAccessLogQueue{
var maxSize = 2_000 * (1 + utils.SystemMemoryGB()/2)
if maxSize > 20_000 {
maxSize = 20_000
}
var queue = &HTTPAccessLogQueue{
queue: make(chan *pb.HTTPAccessLog, maxSize),
}
goman.New(func() {
@@ -43,7 +47,11 @@ func (this *HTTPAccessLogQueue) Start() {
for range ticker.C {
err := this.loop()
if err != nil {
remotelogs.Error("ACCESS_LOG_QUEUE", err.Error())
if rpc.IsConnError(err) {
remotelogs.Debug("ACCESS_LOG_QUEUE", err.Error())
} else {
remotelogs.Error("ACCESS_LOG_QUEUE", err.Error())
}
}
}
}
@@ -130,14 +138,23 @@ Loop:
return nil
}
// ToValidUTF8 处理访问日志中的非UTF-8字节
func (this *HTTPAccessLogQueue) ToValidUTF8(accessLog *pb.HTTPAccessLog) {
accessLog.RemoteAddr = utils.ToValidUTF8string(accessLog.RemoteAddr)
accessLog.RemoteUser = utils.ToValidUTF8string(accessLog.RemoteUser)
accessLog.RequestURI = utils.ToValidUTF8string(accessLog.RequestURI)
accessLog.RequestPath = utils.ToValidUTF8string(accessLog.RequestPath)
accessLog.RequestFilename = utils.ToValidUTF8string(accessLog.RequestFilename)
accessLog.RequestBody = bytes.ToValidUTF8(accessLog.RequestBody, []byte{})
accessLog.Host = utils.ToValidUTF8string(accessLog.Host)
accessLog.Hostname = utils.ToValidUTF8string(accessLog.Hostname)
for k, v := range accessLog.SentHeader {
if !utf8.ValidString(k) {
delete(accessLog.SentHeader, k)
continue
}
for _, v := range accessLog.SentHeader {
for index, s := range v.Values {
v.Values[index] = utils.ToValidUTF8string(s)
}
@@ -149,15 +166,27 @@ func (this *HTTPAccessLogQueue) ToValidUTF8(accessLog *pb.HTTPAccessLog) {
accessLog.ContentType = utils.ToValidUTF8string(accessLog.ContentType)
for k, c := range accessLog.Cookie {
if !utf8.ValidString(k) {
delete(accessLog.Cookie, k)
continue
}
accessLog.Cookie[k] = utils.ToValidUTF8string(c)
}
accessLog.Args = utils.ToValidUTF8string(accessLog.Args)
accessLog.QueryString = utils.ToValidUTF8string(accessLog.QueryString)
for _, v := range accessLog.Header {
for k, v := range accessLog.Header {
if !utf8.ValidString(k) {
delete(accessLog.Header, k)
continue
}
for index, s := range v.Values {
v.Values[index] = utils.ToValidUTF8string(s)
}
}
for k, v := range accessLog.Errors {
accessLog.Errors[k] = utils.ToValidUTF8string(v)
}
}

View File

@@ -17,6 +17,7 @@ import (
"strings"
"testing"
"time"
"unicode/utf8"
)
func TestHTTPAccessLogQueue_Push(t *testing.T) {
@@ -135,6 +136,16 @@ func TestHTTPAccessLogQueue_Memory(t *testing.T) {
time.Sleep(5 * time.Second)
}
func TestUTF8_IsValid(t *testing.T) {
t.Log(utf8.ValidString("abc"))
var noneUTF8Bytes = []byte{}
for i := 0; i < 254; i++ {
noneUTF8Bytes = append(noneUTF8Bytes, uint8(i))
}
t.Log(utf8.ValidString(string(noneUTF8Bytes)))
}
func BenchmarkHTTPAccessLogQueue_ToValidUTF8(b *testing.B) {
runtime.GOMAXPROCS(1)

View File

@@ -9,10 +9,12 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
connutils "github.com/TeaOSLab/EdgeNode/internal/utils/conns"
"github.com/iwind/TeaGo/Tea"
"io"
"net"
@@ -23,6 +25,10 @@ import (
)
func init() {
if !teaconst.IsMain {
return
}
events.On(events.EventStart, func() {
goman.New(func() {
SharedHTTPCacheTaskManager.Start()
@@ -56,7 +62,12 @@ func NewHTTPCacheTaskManager() *HTTPCacheTaskManager {
if err != nil {
return nil, err
}
return net.Dial(network, "127.0.0.1:"+port)
conn, err := net.Dial(network, "127.0.0.1:"+port)
if err != nil {
return nil, err
}
return connutils.NewNoStat(conn), nil
},
MaxIdleConns: 128,
MaxIdleConnsPerHost: 32,

View File

@@ -1,7 +1,7 @@
package nodes
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/fasttime"
"net/http"
)
@@ -15,7 +15,7 @@ type HTTPClient struct {
func NewHTTPClient(rawClient *http.Client) *HTTPClient {
return &HTTPClient{
rawClient: rawClient,
accessAt: utils.UnixTime(),
accessAt: fasttime.Now().Unix(),
}
}
@@ -26,7 +26,7 @@ func (this *HTTPClient) RawClient() *http.Client {
// UpdateAccessTime 更新访问时间
func (this *HTTPClient) UpdateAccessTime() {
this.accessAt = utils.UnixTime()
this.accessAt = fasttime.Now().Unix()
}
// AccessTime 获取访问时间

View File

@@ -95,11 +95,11 @@ func (this *HTTPClientPool) Client(req *HTTPRequest,
numberCPU = 8
}
if maxConnections <= 0 {
maxConnections = numberCPU * 32
maxConnections = numberCPU * 64
}
if idleConns <= 0 {
idleConns = numberCPU * 8
idleConns = numberCPU * 16
}
// 可以判断为Ln节点请求

View File

@@ -11,12 +11,12 @@ func TestHTTPClientPool_Client(t *testing.T) {
pool := NewHTTPClientPool()
{
origin := &serverconfigs.OriginConfig{
var origin = &serverconfigs.OriginConfig{
Id: 1,
Version: 2,
Addr: &serverconfigs.NetworkAddressConfig{Host: "127.0.0.1", PortRange: "1234"},
}
err := origin.Init()
err := origin.Init(nil)
if err != nil {
t.Fatal(err)
}
@@ -43,7 +43,7 @@ func TestHTTPClientPool_cleanClients(t *testing.T) {
Version: 2,
Addr: &serverconfigs.NetworkAddressConfig{Host: "127.0.0.1", PortRange: "1234"},
}
err := origin.Init()
err := origin.Init(nil)
if err != nil {
t.Fatal(err)
}
@@ -60,17 +60,19 @@ func TestHTTPClientPool_cleanClients(t *testing.T) {
func BenchmarkHTTPClientPool_Client(b *testing.B) {
runtime.GOMAXPROCS(1)
origin := &serverconfigs.OriginConfig{
var origin = &serverconfigs.OriginConfig{
Id: 1,
Version: 2,
Addr: &serverconfigs.NetworkAddressConfig{Host: "127.0.0.1", PortRange: "1234"},
}
err := origin.Init()
err := origin.Init(nil)
if err != nil {
b.Fatal(err)
}
pool := NewHTTPClientPool()
b.ResetTimer()
var pool = NewHTTPClientPool()
for i := 0; i < b.N; i++ {
_, _ = pool.Client(nil, origin, origin.Addr.PickAddress(), nil, false)
}

View File

@@ -46,6 +46,7 @@ type HTTPRequest struct {
ServerAddr string // 实际启动的服务器监听地址
IsHTTP bool
IsHTTPS bool
IsHTTP3 bool
// 共享参数
nodeConfig *nodeconfigs.NodeConfig
@@ -221,6 +222,18 @@ func (this *HTTPRequest) Do() {
}
}
// CC
if !isHealthCheck {
if this.web.CC != nil {
if this.web.CC.IsOn {
if this.doCC() {
this.doEnd()
return
}
}
}
}
// WAF
if this.web.FirewallRef != nil && this.web.FirewallRef.IsOn {
if this.doWAFRequest() {
@@ -237,6 +250,14 @@ func (this *HTTPRequest) Do() {
}
}
// UA名单
if !this.isSubRequest && this.web.UserAgent != nil && this.web.UserAgent.IsOn {
if this.doCheckUserAgent() {
this.doEnd()
return
}
}
// 访问控制
if !this.isSubRequest && this.web.Auth != nil && this.web.Auth.IsOn {
if this.doAuth() {
@@ -383,7 +404,7 @@ func (this *HTTPRequest) doEnd() {
attackBytes = this.CalculateSize()
}
stats.SharedTrafficStatManager.Add(this.ReqServer.Id, this.ReqHost, this.writer.SentBodyBytes()+this.writer.SentHeaderBytes(), cachedBytes, 1, countCached, countAttacks, attackBytes, this.ReqServer.ShouldCheckTrafficLimit(), this.ReqServer.PlanId())
stats.SharedTrafficStatManager.Add(this.ReqServer.UserId, this.ReqServer.Id, this.ReqHost, this.writer.SentBodyBytes()+this.writer.SentHeaderBytes(), cachedBytes, 1, countCached, countAttacks, attackBytes, this.ReqServer.ShouldCheckTrafficLimit(), this.ReqServer.PlanId())
// 指标
if metrics.SharedManager.HasHTTPMetrics() {
@@ -459,6 +480,17 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
// remote addr
if web.RemoteAddr != nil && (web.RemoteAddr.IsPrior || isTop) && web.RemoteAddr.IsOn {
this.web.RemoteAddr = web.RemoteAddr
// check if from proxy
if len(this.web.RemoteAddr.Value) > 0 && this.web.RemoteAddr.Value != "${rawRemoteAddr}" {
var requestConn = this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn != nil {
requestClientConn, ok := requestConn.(ClientConnInterface)
if ok {
requestClientConn.SetIsPersistent(true)
}
}
}
}
// charset
@@ -526,6 +558,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.Referers = web.Referers
}
// user agent
if web.UserAgent != nil && (web.UserAgent.IsPrior || isTop) {
this.web.UserAgent = web.UserAgent
}
// request limit
if web.RequestLimit != nil && (web.RequestLimit.IsPrior || isTop) {
this.web.RequestLimit = web.RequestLimit
@@ -559,6 +596,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.UAM = web.UAM
}
// CC
if web.CC != nil && (web.CC.IsPrior || isTop) {
this.web.CC = web.CC
}
// 重写规则
if len(web.RewriteRefs) > 0 {
for index, ref := range web.RewriteRefs {
@@ -715,6 +757,8 @@ func (this *HTTPRequest) Format(source string) string {
return this.Path()
case "requestPathExtension":
return filepath.Ext(this.Path())
case "requestPathLowerExtension":
return strings.ToLower(filepath.Ext(this.Path()))
case "requestLength":
return strconv.FormatInt(this.requestLength(), 10)
case "requestTime":
@@ -758,6 +802,8 @@ func (this *HTTPRequest) Format(source string) string {
return strconv.FormatInt(this.requestFromTime.Unix(), 10)
case "host":
return this.ReqHost
case "cname":
return this.ReqServer.CNameDomain
case "referer":
return this.RawReq.Referer()
case "referer.host":
@@ -826,13 +872,29 @@ func (this *HTTPRequest) Format(source string) string {
}
// response.xxx.xxx
dotIndex := strings.Index(suffix, ".")
dotIndex = strings.Index(suffix, ".")
if dotIndex < 0 {
return "${" + varName + "}"
}
switch suffix[:dotIndex] {
case "header":
return this.writer.Header().Get(suffix[dotIndex+1:])
var headers = this.writer.Header()
var headerKey = suffix[dotIndex+1:]
v, found := headers[headerKey]
if found {
if len(v) == 0 {
return ""
}
return v[0]
}
var canonicalHeaderKey = http.CanonicalHeaderKey(headerKey)
if canonicalHeaderKey != headerKey {
v = headers[canonicalHeaderKey]
if len(v) > 0 {
return v[0]
}
}
return ""
}
}
@@ -1131,6 +1193,8 @@ func (this *HTTPRequest) requestRemoteAddr(supportVar bool) string {
// 获取请求的客户端地址列表
func (this *HTTPRequest) requestRemoteAddrs() (result []string) {
result = append(result, this.requestRemoteAddr(true))
// X-Forwarded-For
var forwardedFor = this.RawReq.Header.Get("X-Forwarded-For")
if len(forwardedFor) > 0 {
@@ -1252,6 +1316,12 @@ func (this *HTTPRequest) requestQueryParam(name string) string {
func (this *HTTPRequest) requestHeader(key string) string {
v, found := this.RawReq.Header[key]
if !found {
// 转换为canonical header再尝试
var canonicalHeaderKey = http.CanonicalHeaderKey(key)
if canonicalHeaderKey != key {
return strings.Join(this.RawReq.Header[canonicalHeaderKey], ";")
}
return ""
}
return strings.Join(v, ";")
@@ -1552,7 +1622,7 @@ func (this *HTTPRequest) processRequestHeaders(reqHeader http.Header) {
}
// 是否已删除
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
if this.web.RequestHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
}
@@ -1592,6 +1662,20 @@ func (this *HTTPRequest) processRequestHeaders(reqHeader http.Header) {
}
}
}
// 非标Header
if len(this.web.RequestHeaderPolicy.NonStandardHeaders) > 0 {
for _, name := range this.web.RequestHeaderPolicy.NonStandardHeaders {
var canonicalKey = http.CanonicalHeaderKey(name)
if canonicalKey != name {
v, ok := reqHeader[canonicalKey]
if ok {
delete(reqHeader, canonicalKey)
reqHeader[name] = v
}
}
}
}
}
}
@@ -1626,8 +1710,8 @@ func (this *HTTPRequest) fixRequestHeader(header http.Header) {
}
}
// 处理自定义Response Header
func (this *HTTPRequest) processResponseHeaders(responseHeader http.Header, statusCode int) {
// ProcessResponseHeaders 处理自定义Response Header
func (this *HTTPRequest) ProcessResponseHeaders(responseHeader http.Header, statusCode int) {
// 删除/添加/替换Header
// TODO 实现AddTrailers
if this.web.ResponseHeaderPolicy != nil && this.web.ResponseHeaderPolicy.IsOn {
@@ -1690,6 +1774,60 @@ func (this *HTTPRequest) processResponseHeaders(responseHeader http.Header, stat
responseHeader[header.Name] = []string{headerValue}
}
}
// 非标Header
if len(this.web.ResponseHeaderPolicy.NonStandardHeaders) > 0 {
for _, name := range this.web.ResponseHeaderPolicy.NonStandardHeaders {
var canonicalKey = http.CanonicalHeaderKey(name)
if canonicalKey != name {
v, ok := responseHeader[canonicalKey]
if ok {
delete(responseHeader, canonicalKey)
responseHeader[name] = v
}
}
}
}
// CORS
if this.web.ResponseHeaderPolicy.CORS != nil && this.web.ResponseHeaderPolicy.CORS.IsOn && (!this.web.ResponseHeaderPolicy.CORS.OptionsMethodOnly || this.RawReq.Method == http.MethodOptions) {
var corsConfig = this.web.ResponseHeaderPolicy.CORS
// Allow-Origin
if len(corsConfig.AllowOrigin) == 0 {
var origin = this.RawReq.Header.Get("Origin")
if len(origin) > 0 {
responseHeader.Set("Access-Control-Allow-Origin", origin)
}
} else {
responseHeader.Set("Access-Control-Allow-Origin", corsConfig.AllowOrigin)
}
// Allow-Methods
if len(corsConfig.AllowMethods) == 0 {
responseHeader.Set("Access-Control-Allow-Methods", "PUT, GET, POST, DELETE, HEAD, OPTIONS, PATCH")
} else {
responseHeader.Set("Access-Control-Allow-Methods", strings.Join(corsConfig.AllowMethods, ", "))
}
// Max-Age
if corsConfig.MaxAge > 0 {
responseHeader.Set("Access-Control-Max-Age", types.String(corsConfig.MaxAge))
}
// Expose-Headers
if len(corsConfig.ExposeHeaders) > 0 {
responseHeader.Set("Access-Control-Expose-Headers", strings.Join(corsConfig.ExposeHeaders, ", "))
}
// Request-Method
if len(corsConfig.RequestMethod) > 0 {
responseHeader.Set("Access-Control-Request-Method", strings.ToUpper(corsConfig.RequestMethod))
}
// Allow-Credentials
responseHeader.Set("Access-Control-Allow-Credentials", "true")
}
}
// HSTS
@@ -1702,6 +1840,11 @@ func (this *HTTPRequest) processResponseHeaders(responseHeader http.Header, stat
this.ReqServer.HTTPS.SSLPolicy.HSTS.Match(this.ReqHost) {
responseHeader.Set(this.ReqServer.HTTPS.SSLPolicy.HSTS.HeaderKey(), this.ReqServer.HTTPS.SSLPolicy.HSTS.HeaderValue())
}
// HTTP/3
if this.IsHTTPS && !this.IsHTTP3 && this.ReqServer.SupportsHTTP3() {
this.processHTTP3Headers(responseHeader)
}
}
// 添加错误信息
@@ -1721,10 +1864,10 @@ func (this *HTTPRequest) bytePool(contentLength int64) *utils.BytePool {
return utils.BytePool1k
}
if contentLength < 32768 { // 32K
return utils.BytePool4k
return utils.BytePool16k
}
if contentLength < 131072 { // 128K
return utils.BytePool16k
return utils.BytePool32k
}
return utils.BytePool32k
}
@@ -1771,7 +1914,7 @@ func (this *HTTPRequest) canIgnore(err error) bool {
// 检查连接是否已关闭
func (this *HTTPRequest) isConnClosed() bool {
requestConn := this.RawReq.Context().Value(HTTPConnContextKey)
var requestConn = this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn == nil {
return true
}

Some files were not shown because too many files have changed in this diff Show More