Compare commits

..

208 Commits

Author SHA1 Message Date
刘祥超
2525cdc061 根据Accept-Encoding决定是否解压响应内容 2021-12-29 10:57:15 +08:00
刘祥超
4ffc619aad 将获取系统内存函数放入到utils中 2021-12-29 10:53:59 +08:00
刘祥超
f930705fd7 删除不需要的文件 2021-12-24 15:00:49 +08:00
刘祥超
56961c1476 修复测试用例 2021-12-23 14:36:52 +08:00
刘祥超
28b61d493f 优化代码 2021-12-22 16:43:16 +08:00
刘祥超
5cb5ddf2c1 修复并发下,写缓存文件可能冲突的问题 2021-12-21 08:03:09 +08:00
刘祥超
fd0bc37ec7 修复并发下,写缓存文件可能冲突的问题 2021-12-21 00:27:32 +08:00
刘祥超
73666bea7f 修改版本号为0.4.0 2021-12-20 20:01:07 +08:00
刘祥超
462442e21a 缓存数据库升级时从老的数据库转移数据 2021-12-19 18:55:54 +08:00
刘祥超
90fcddfb9f WAF:优化get302/post307代码 2021-12-19 18:54:43 +08:00
刘祥超
8de791079c 优化代码 2021-12-19 16:54:56 +08:00
刘祥超
13b89d5971 当使用quit退出进程时,同时也禁用缓存策略 2021-12-19 14:15:17 +08:00
刘祥超
8b97638624 优化代码 2021-12-19 11:32:26 +08:00
刘祥超
79ea9e795e edge-node conns命令可以打印当前总连接数 2021-12-18 20:13:41 +08:00
刘祥超
38e06e7b03 TLS连接增加握手超时检查 2021-12-18 19:17:40 +08:00
刘祥超
f25de8d5c9 批量清除缓存时延时删除 2021-12-17 11:54:27 +08:00
刘祥超
af74500810 优化代码 2021-12-17 11:54:06 +08:00
刘祥超
189295ffcf X-Cache增加STALE状态 2021-12-17 11:53:59 +08:00
刘祥超
da09889eca 优化代码 2021-12-16 20:36:42 +08:00
刘祥超
9ffa910044 源站没有地址时也尝试Stale Cache/避免write50x()方法进入死循环 2021-12-16 17:38:07 +08:00
刘祥超
a6d711c2a0 实现stale cache读取 2021-12-16 17:27:21 +08:00
刘祥超
6bedc97c95 优化代码 2021-12-15 20:46:10 +08:00
刘祥超
4bdd1eda76 更新依赖 2021-12-15 16:57:06 +08:00
刘祥超
7fd9766565 修复未完成的代码 2021-12-15 15:11:25 +08:00
刘祥超
72983d8d86 优化代码 2021-12-15 15:09:58 +08:00
刘祥超
ec6494fa9c 优化HTTP参数 2021-12-15 13:48:48 +08:00
刘祥超
a4a6e95099 HTTP Header:实现请求方法、域名、状态码等限制,实现内容替换功能 2021-12-14 21:27:24 +08:00
刘祥超
2e11c99b7a 优化代码 2021-12-14 10:49:40 +08:00
刘祥超
014f433191 优化代码 2021-12-14 10:01:21 +08:00
刘祥超
e6ac085025 优化代码 2021-12-13 14:58:24 +08:00
刘祥超
6f60be6a00 服务最大连接数和单IP最大连接数任其一不为0则生效 2021-12-13 11:24:03 +08:00
刘祥超
dceb082a83 修改网络连接错误日志级别 2021-12-13 08:27:39 +08:00
刘祥超
9084794448 路由规则增加专属域名设置 2021-12-12 16:38:38 +08:00
刘祥超
065de8d208 在URL跳转、重写规则跳转、自动跳转到HTTPS等处增加响应Header 2021-12-12 14:10:42 +08:00
刘祥超
e5f9316e33 实现请求连接数等限制 2021-12-12 11:48:01 +08:00
刘祥超
bb5fa38613 实现全局的TCP最大连接数 2021-12-09 17:34:05 +08:00
刘祥超
ccb97b1c79 实现线程数限制 2021-12-09 12:07:59 +08:00
刘祥超
853e4fd0f0 使用空struct{}代替bool节约内存 2021-12-09 12:07:46 +08:00
刘祥超
d3169eaea5 优化代码 2021-12-08 22:19:15 +08:00
刘祥超
68b93bf6b4 可以在缓存条件里设置Expires Header 2021-12-08 17:41:39 +08:00
刘祥超
1279f0d394 优化系统goroutine使用,减少goroutine数量,增加goman查看goroutine数量指令 2021-12-08 15:17:45 +08:00
刘祥超
24fbd740b5 实现记录请求Body 2021-12-07 15:12:15 +08:00
刘祥超
5772fb2309 缓存支持请求方法设置 2021-12-07 10:43:42 +08:00
刘祥超
bf2b889c16 增加${cache.key}变量 2021-12-07 09:28:15 +08:00
刘祥超
9372bc90dd 增加${isArgs}请求变量 2021-12-06 21:47:57 +08:00
刘祥超
5b46c80431 在开发环境下打印Go语言内部HTTP调试信息 2021-12-06 19:28:26 +08:00
刘祥超
1bdb988425 启动时增加sid设置 2021-12-06 19:28:00 +08:00
刘祥超
a544a77669 自动将API节点的IP加入到白名单,防止误封
但要注意:在单个机器上安装API节点和边缘节点,通过局域网IP访问时就无法测试WAF规则,因为会被自动加入到白名单
2021-12-06 10:11:22 +08:00
刘祥超
c61108faa8 优化代码 2021-12-06 08:56:02 +08:00
刘祥超
30ac3118e2 国家/地区统计时上传流量、攻击量等信息 2021-12-05 18:57:30 +08:00
刘祥超
f0e8dd1baa 缓存增加UPDATING状态 2021-12-05 17:10:06 +08:00
刘祥超
04a327ce9a 修复源站主动关闭连接时无法缓存内容的Bug 2021-12-05 16:55:33 +08:00
刘祥超
2ac26f6aa4 回源主机名为“跟随源站”时,获得的源站主机名去除常规端口80和443 2021-12-05 09:30:45 +08:00
刘祥超
d9aac44ea3 WAF忽略客户端断开连接错误 2021-12-04 19:28:02 +08:00
刘祥超
160a1f1466 优化通过IP查询区域性能 2021-12-03 15:51:28 +08:00
刘祥超
38e2c151ec 降低ttlcache最大内存增量 2021-12-03 10:22:03 +08:00
刘祥超
9d54c17695 支持规则集忽略局域网IP 2021-12-02 16:08:25 +08:00
刘祥超
e31d68c1e1 将RPC连接错误级别从error改为warning 2021-12-02 15:14:47 +08:00
刘祥超
7ae9180bf9 多个提示页面增加请求ID、增加变量支持 2021-12-02 14:46:40 +08:00
刘祥超
424f3ae29d 优化验证码页面 2021-12-02 12:08:59 +08:00
刘祥超
ca0571a21b 增加${requestId}变量 2021-12-02 11:30:47 +08:00
刘祥超
c9bd9fd460 URL跳转时检查前后跳转的URL是否一致,防止无限跳转 2021-12-02 10:35:51 +08:00
刘祥超
8d4ec6822c 缓存支持源站设置的max-age 2021-12-02 10:19:02 +08:00
刘祥超
061253b4c3 缓存在响应中可以添加Age Header 2021-12-02 09:54:48 +08:00
刘祥超
f6dfd6acec 增加${cache.age}变量 2021-12-02 09:34:38 +08:00
刘祥超
a35aa2f520 增加是否记录499选项 2021-12-01 21:13:10 +08:00
刘祥超
ea84c41be3 因WAF规则拦截而关闭连接时,不记录499 2021-12-01 20:55:19 +08:00
刘祥超
0f0776fc1a 修复WAF OnAction在并发时无法准确调用请求动作的Bug 2021-12-01 17:43:08 +08:00
刘祥超
6aacf49764 可以上报服务相关配置错误 2021-12-01 15:52:38 +08:00
刘祥超
18a01b9b43 上传访问日志时如果出现string field contains invalid UTF-8,则重新处理后提交 2021-12-01 14:25:55 +08:00
刘祥超
32a3e08332 优化编译脚本 2021-12-01 14:24:56 +08:00
刘祥超
2397695a2d 优化服务日志 2021-11-30 16:43:58 +08:00
刘祥超
0ceebd9902 端口提示被占用时提示语中加入当前占用端口的进程名 2021-11-30 15:03:46 +08:00
刘祥超
7e62c72b79 修复TOA管理中可能出现的panic错误 2021-11-29 11:15:14 +08:00
刘祥超
b3dedbdc31 健康检查不使用密钥加密 2021-11-29 09:52:31 +08:00
刘祥超
fa967b5450 修改版本号为0.3.7 2021-11-28 14:28:24 +08:00
刘祥超
b619eb4efe 修复WAF中scheme checkpoint值为空的问题 2021-11-26 13:42:04 +08:00
刘祥超
7763f26249 修复WAF的临时白名单被当做黑名单使用的Bug 2021-11-26 10:39:04 +08:00
刘祥超
b7ae10e2d0 修复合并URL中多余分隔符时导致参数发生变化的Bug 2021-11-24 15:01:06 +08:00
刘祥超
e54eddc961 服务增加是否合并URL中的多余分隔符选项 2021-11-24 14:50:07 +08:00
刘祥超
93db9d4926 版本号改为0.3.6 2021-11-24 14:04:01 +08:00
刘祥超
8c1af3e699 修复ipset无法提前删除IP的Bug 2021-11-24 10:21:02 +08:00
刘祥超
53c74553bc 修复ipset无法提前删除IP的Bug 2021-11-24 10:20:06 +08:00
刘祥超
3eb9cade0e 修改版本为0.3.5.2 2021-11-24 10:19:36 +08:00
刘祥超
eeee3da941 暂时不删除多余的*.cache.tmp,以防产生性能问题 2021-11-21 16:10:07 +08:00
刘祥超
6af59e0bd0 优化访问日志上传 2021-11-21 10:56:54 +08:00
刘祥超
012233baf2 修复访问日志requestId可能重复的问题 2021-11-21 10:40:19 +08:00
刘祥超
ac069fd7f3 访问日志简化requestId生成方法 2021-11-21 10:27:31 +08:00
刘祥超
40cb1916c2 优化RPC客户端锁 2021-11-20 19:17:57 +08:00
刘祥超
749e0bd0b3 简化节点API配置模板 2021-11-20 18:58:08 +08:00
刘祥超
ae1a9abf5e 实现修改API节点地址的指令 2021-11-20 18:57:46 +08:00
刘祥超
7fc0394f10 修复RPC客户端管理没有加锁的问题 2021-11-20 18:57:00 +08:00
刘祥超
4a169a2dbd 修复节点拦截上报IP时没有上传服务ID的Bug 2021-11-17 19:51:37 +08:00
刘祥超
44d8afeda8 WAF规则匹配后的IP也会上报/实现IP全局名单/将名单存储到本地数据库,提升读写速度 2021-11-17 16:16:09 +08:00
刘祥超
6a0547abec IP名单中IP创建时保存相关节点、服务、WAF策略信息 2021-11-16 16:11:05 +08:00
刘祥超
6d002e2822 优化运行日志上传功能,最近N条重复的不再上传 2021-11-15 16:59:18 +08:00
刘祥超
04271d77c2 大幅提升域名匹配性能 2021-11-15 16:57:18 +08:00
刘祥超
5a6ead1dd7 优化iptables+firewall-cmd找不到时的提示 2021-11-15 09:45:03 +08:00
刘祥超
9ac7b9b2c0 内存缓存周期单位改成天 2021-11-15 09:15:23 +08:00
刘祥超
7ec916c1fb 增加注释 2021-11-14 20:46:08 +08:00
刘祥超
fadc580dff 修复IPTables+IPSet组合时在IPTables中生成了多个重复记录的Bug;增加IPSet最大值为1000000;IP范围只支持D段 2021-11-14 20:35:47 +08:00
刘祥超
fb9e9fb94b IP名单同步时在本地记录上一次同步的位置,以便于下次启动的时候不再重复执行 2021-11-14 20:34:04 +08:00
刘祥超
9c6e4bb8c1 在开发环境下运行日志显示包名 2021-11-14 20:31:49 +08:00
刘祥超
97b04777bc 实现自动将热点数据加载到内存中 2021-11-14 16:15:07 +08:00
刘祥超
4daeca912a 增加对任务的执行时间追踪工具 2021-11-14 10:55:09 +08:00
刘祥超
7e43324b53 反向代理源站错误时提示完整的URL 2021-11-14 08:58:27 +08:00
刘祥超
b9b8472c3a 缓存策略实现LFU算法/实现内存缓存自动Flush数据到磁盘 2021-11-13 21:30:24 +08:00
刘祥超
6858380bb4 节点配置支持压缩 2021-11-11 14:16:57 +08:00
刘祥超
568ecadfc6 上传流量统计的时候记录所属套餐 2021-11-10 21:52:40 +08:00
刘祥超
8210ece2b7 优化错误提示 2021-11-10 21:51:56 +08:00
刘祥超
f8160e35b9 改进流量限制 2021-11-10 14:39:02 +08:00
刘祥超
9e4a1212d2 接收请求时保留URL路径中多于的斜杠(/) 2021-11-10 10:25:02 +08:00
刘祥超
6f52cffabd 将带宽限制改为流量限制 2021-11-09 17:36:49 +08:00
刘祥超
c546b9fc7d 支持套餐日期设置 2021-11-09 15:36:05 +08:00
刘祥超
f7b961d256 规范命名 2021-11-05 15:37:07 +08:00
刘祥超
71cbb2d695 修改版本为0.3.5 2021-11-05 14:58:32 +08:00
刘祥超
2063015eeb 删除IP名单中某个IP时,也会删除WAF保存在内存中的名单中的IP 2021-11-05 14:58:10 +08:00
刘祥超
87cc43b2e0 修复firewalld无法删除规则的Bug 2021-11-05 14:39:08 +08:00
刘祥超
9812883b61 X-Cache加入跳过缓存的原因 2021-11-05 14:15:21 +08:00
刘祥超
068c20e1b9 特殊页面选择读取URL时,保留当前的状态码 2021-11-05 14:10:43 +08:00
刘祥超
17d883a2de 修改部分错误提示级别 2021-11-04 11:14:27 +08:00
刘祥超
083bbb1460 支持info指令查询PID、版本号等信息 2021-11-04 11:14:02 +08:00
刘祥超
4f7b9f4fc6 修改版本为0.3.4 2021-11-01 10:45:33 +08:00
刘祥超
b679fc3b25 修复连接快速关闭时数据可能无法完整传输的问题 2021-10-30 22:35:05 +08:00
刘祥超
ced82d8a30 修复arm64下无法调用syscall.Dup2的问题 2021-10-30 22:34:40 +08:00
刘祥超
73108faa3e 支持gif转webp 2021-10-29 12:19:26 +08:00
刘祥超
e89460e36f WAF增加显示网页动作 2021-10-25 19:42:12 +08:00
刘祥超
cd19a4a7bc 优化连接关闭速度 2021-10-25 19:00:42 +08:00
刘祥超
4dcd47d8bc 修复freebsd编译时的错误 2021-10-22 15:38:53 +08:00
刘祥超
388e7d2683 实现单个服务的带宽限制(商业版) 2021-10-21 17:09:51 +08:00
刘祥超
36f9effbf2 将IP名单默认范围改为global 2021-10-21 09:31:31 +08:00
刘祥超
8bb47fb915 设置客户端连接linger为0 2021-10-20 22:32:02 +08:00
刘祥超
7127d26fff 修复校验ACME证书时受自动跳转等设置的影响的问题 2021-10-20 17:13:45 +08:00
刘祥超
8fae094161 修改Edge-Purge-Key为X-Edge-Purge-Key 2021-10-19 16:41:12 +08:00
刘祥超
16349041fb 健康检查支持UserAgent和是否基础请求设置 2021-10-19 16:31:27 +08:00
刘祥超
2793e0de89 增加防盗链规则参数 2021-10-19 11:38:46 +08:00
刘祥超
82a7971718 修复IP黑名单为服务时不生效的问题 2021-10-19 09:21:58 +08:00
刘祥超
1ef59ccf65 WAF动作支持有效范围 2021-10-18 20:08:43 +08:00
刘祥超
cc5d2f1862 内容压缩支持对已压缩内容重新压缩 2021-10-18 16:50:06 +08:00
刘祥超
5bff24b88d 增加PURGE某个URL缓存功能 2021-10-17 20:23:10 +08:00
刘祥超
bc2f8b1e4c 修复特殊页面无法缓存的Bug 2021-10-16 12:21:28 +08:00
刘祥超
8935f35b4e 支持任意域名通过CNAME访问服务(开启选项后) 2021-10-16 12:03:05 +08:00
刘祥超
874694f662 增大默认的源站的并发连接数(32 * CPU)和空闲连接数(8 * CPU) 2021-10-15 10:30:07 +08:00
刘祥超
27506ae436 修改panic错误提示 2021-10-14 13:46:55 +08:00
刘祥超
3d9f40331d 优化节点日志记录,可以记录和上报panic错误 2021-10-14 10:28:32 +08:00
刘祥超
538f18afb0 WebP支持源站gzip、deflate、br等压缩后的图片内容 2021-10-13 11:56:40 +08:00
刘祥超
b4d9fc02bd WebP无法解析原图时直接返回原图数据 2021-10-13 11:11:57 +08:00
刘祥超
537138c8a1 优化代码 2021-10-12 20:46:45 +08:00
刘祥超
f0d0a39a03 支持PROXY Protocol/修复UDP源站无法修改的问题 2021-10-12 20:20:06 +08:00
刘祥超
bcce2a2767 可以在集群中指定节点时区 2021-10-12 11:43:17 +08:00
刘祥超
86db3dfc49 WAF动作record_ip返回403/优化关闭连接方法 2021-10-12 09:06:28 +08:00
刘祥超
ac120a728c WebP压缩支持ico 2021-10-11 14:53:23 +08:00
刘祥超
871de0e655 版本改为0.3.3 2021-10-11 14:53:20 +08:00
刘祥超
fa6be81abe 特殊页面可以直接使用HTML 2021-10-10 10:35:05 +08:00
刘祥超
1b2f01f0f4 把tcp/udp的连接数记为访问量,增加tcp域名排名记录(需要SNI连接) 2021-10-08 15:50:43 +08:00
刘祥超
09467a4d08 WAF模式从pass改为bypass 2021-10-07 13:51:26 +08:00
刘祥超
a17bbc3df1 缓存预热判断请求来源的时候增加IPv6回路地址判断 2021-10-06 16:35:39 +08:00
刘祥超
dc495c70b3 服务支持自定义访客IP地址获取方式/对X-Real-IP等Header值进行有效性验证 2021-10-06 11:40:48 +08:00
刘祥超
d0cf145f85 优化WAF动作排序 2021-10-06 09:39:57 +08:00
刘祥超
95f1e61489 WAF动作block和record_ip同时存在时,优先执行record_ip 2021-10-06 08:56:38 +08:00
刘祥超
42a0161312 WAF模式为defend时IP黑名单才生效 2021-10-04 18:22:51 +08:00
刘祥超
729443f0b4 大幅优化IP名单查询速度 2021-10-04 17:42:38 +08:00
刘祥超
3888565c0f 根据系统内存自动调节ttlcache的最大条目 2021-10-04 09:12:17 +08:00
刘祥超
1ca967534a 开启缓存后覆盖源站的ETag和Last-Modified 2021-10-04 08:41:44 +08:00
刘祥超
c01bb57dea 不把499状态码加入状态码统计 2021-10-04 08:41:13 +08:00
刘祥超
adadb52d4e 优化WebP+缓存 2021-10-03 18:00:57 +08:00
刘祥超
38a7cc17da 修复WAF策略模式为空导致动作不起作用的问题 2021-10-03 08:35:28 +08:00
刘祥超
246bb45614 限制WebP转换时消耗的内存总量 2021-10-01 18:59:44 +08:00
刘祥超
b320d2dc58 修复WebP缓存长度可能不正确的问题 2021-10-01 17:20:37 +08:00
刘祥超
96c63300f4 恢复源站默认连接数限制 2021-10-01 16:45:46 +08:00
刘祥超
3eaf090aac 缓存内容也支持压缩 2021-10-01 16:24:29 +08:00
刘祥超
b157448ad2 支持自动转换图像文件为WebP 2021-10-01 16:24:17 +08:00
刘祥超
44998a23fb 反向代理去除默认跟源站的连接数限制 2021-10-01 11:17:43 +08:00
刘祥超
e3e30ffee5 节点启动时如果加载的是本地配置则在网络恢复后重新加载配置 2021-10-01 11:13:36 +08:00
刘祥超
12bddc6e82 WAF策略增加观察模式和通过模式 2021-09-30 11:30:58 +08:00
刘祥超
771d2d8013 支持brotli和deflate压缩 2021-09-29 19:37:07 +08:00
刘祥超
3bf94bc032 优化WAF关闭连接操作 2021-09-29 11:06:00 +08:00
刘祥超
a1aa2b9224 Block动作增加默认时间60秒 2021-09-29 09:19:45 +08:00
刘祥超
8d28ba3426 缓存条件增加最小内容尺寸配置 2021-09-26 15:01:46 +08:00
刘祥超
5a72c10d83 版本改为0.3.2 2021-09-26 10:09:56 +08:00
刘祥超
ec113c59ab 将版本修改为0.3.1 2021-09-23 21:00:06 +08:00
刘祥超
7f9d95ba37 修复当缓存内容为空时无法响应缓存的Bug 2021-09-22 21:13:31 +08:00
刘祥超
558314265a 修改部分命名 2021-09-22 19:40:11 +08:00
刘祥超
c793dd9d8c 特殊页面中的URL抓取的内容也支持请求变量 2021-09-21 10:13:30 +08:00
刘祥超
33e899a008 特殊页面支持请求变量 2021-09-21 09:29:17 +08:00
刘祥超
903f511139 反向代理源站实现使用域名分组 2021-09-20 11:54:37 +08:00
刘祥超
49dafafdc5 修复反向代理Hash调度算法无法生效的Bug 2021-09-20 09:59:24 +08:00
刘祥超
c6b01dc10a 修复反向代理sticky调度算法无法生效的Bug 2021-09-20 09:55:42 +08:00
刘祥超
1119b351aa 内存缓存增加最大数量限制 2021-09-19 16:11:46 +08:00
刘祥超
7b8ef8e85b 配置加载成功后才启动某些任务 2021-09-16 15:58:10 +08:00
刘祥超
dd41d88647 指标数据队列增加最大数量限制,防止过载 2021-09-03 15:38:56 +08:00
刘祥超
699cea4382 请求源站错误时增加503、504错误 2021-09-01 08:48:03 +08:00
刘祥超
dfb0e60acc 缓存文件列表关闭时也清除内存缓存 2021-08-29 08:59:32 +08:00
刘祥超
417e10970a 缓存预热时不重复写入 2021-08-26 15:48:09 +08:00
刘祥超
6437875979 优化ACME读取速度 2021-08-25 17:32:53 +08:00
刘祥超
beb251a3cc 调整ACME证书申请链接的优先级为最高,避免因URL跳转而导致无法申请证书 2021-08-25 16:50:07 +08:00
刘祥超
8069f20c77 修改注释 2021-08-25 16:46:05 +08:00
刘祥超
8161270f47 优化指标统计写入数据逻辑 2021-08-22 10:07:04 +08:00
刘祥超
be7de223af 提升文件缓存读取效率大约5% 2021-08-21 21:44:34 +08:00
刘祥超
7469f528a8 通过内存缓存提升文件缓存效率大约20% 2021-08-21 21:06:48 +08:00
刘祥超
1f88db8829 阶段性提交 2021-08-21 20:45:11 +08:00
刘祥超
844a5fc321 调整版本为0.3.0 2021-08-17 09:44:37 +08:00
刘祥超
06db07ac4b 优化代码 2021-08-07 16:27:42 +08:00
刘祥超
00df3fca84 调整版本为0.2.9 2021-08-07 16:11:48 +08:00
刘祥超
4d679e31bc 改进WAF record_ip动作 2021-08-04 15:35:22 +08:00
刘祥超
06e27bb469 调整版本 2021-08-04 15:34:55 +08:00
刘祥超
684ba7082b 修复统计指标数据上传不完整的问题 2021-08-03 14:02:15 +08:00
刘祥超
8934962de2 调整版本号 2021-08-03 10:40:46 +08:00
225 changed files with 8757 additions and 2003 deletions

2
.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
*_plus.go
*-plus.sh

View File

@@ -5,6 +5,7 @@ function build() {
NAME="edge-node"
VERSION=$(lookup-version $ROOT/../internal/const/const.go)
DIST=$ROOT/"../dist/${NAME}"
MUSL_DIR="/usr/local/opt/musl-cross/bin"
OS=${1}
ARCH=${2}
TAG=${3}
@@ -53,7 +54,6 @@ function build() {
echo "building ..."
MUSL_DIR="/usr/local/opt/musl-cross/bin"
CC_PATH=""
CXX_PATH=""
if [[ `uname -a` == *"Darwin"* && "${OS}" == "linux" ]]; then

View File

@@ -1,4 +1,4 @@
rpc:
endpoints: [ ${endpoints} ]
nodeId: "${nodeId}"
secret: "${nodeSecret}"
endpoints: [ "" ]
nodeId: ""
secret: ""

View File

@@ -9,7 +9,7 @@
<h3>403 Forbidden</h3>
<p>Sorry, your access to the page has been denied. Please try again later.</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>404 Not Found</h3>
<p>Sorry, the page you are looking for is not found. Please try again later.</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>An error occurred.</h3>
<p>Sorry, the page you are looking for is currently unavailable. Please try again later.</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>The website is shutdown.</h3>
<p>Sorry, the page you are looking for is currently unavailable. Please try again later.</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>网站升级中</h3>
<p>为了给您提供更好的服务,我们正在升级网站,请稍后重新访问。</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>网站暂时关闭</h3>
<p>网站已被暂时关闭,请耐心等待我们的重新开通通知。</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -1,6 +1,7 @@
package main
import (
"encoding/json"
"fmt"
"github.com/TeaOSLab/EdgeNode/internal/apps"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
@@ -11,6 +12,7 @@ import (
"net/http"
_ "net/http/pprof"
"os"
"sort"
)
func main() {
@@ -60,6 +62,61 @@ func main() {
node := nodes.NewNode()
node.Start()
})
app.On("trackers", func() {
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{Code: "trackers"})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
labelsMap, ok := reply.Params["labels"]
if ok {
labels, ok := labelsMap.(map[string]interface{})
if ok {
if len(labels) == 0 {
fmt.Println("no labels yet")
} else {
var labelNames = []string{}
for label := range labels {
labelNames = append(labelNames, label)
}
sort.Strings(labelNames)
for _, labelName := range labelNames {
fmt.Println(labelName + ": " + fmt.Sprintf("%.6f", labels[labelName]))
}
}
}
}
}
})
app.On("goman", func() {
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{Code: "goman"})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
instancesJSON, err := json.MarshalIndent(reply.Params, "", " ")
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
fmt.Println(string(instancesJSON))
}
}
})
app.On("conns", func() {
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{Code: "conns"})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
resultJSON, err := json.MarshalIndent(reply.Params, "", " ")
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
fmt.Println(string(resultJSON))
}
}
})
app.Run(func() {
node := nodes.NewNode()
node.Start()

32
go.mod
View File

@@ -5,25 +5,31 @@ go 1.15
replace github.com/TeaOSLab/EdgeCommon => ../EdgeCommon
require (
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect
github.com/TeaOSLab/EdgeCommon v0.0.0-00010101000000-000000000000
github.com/andybalholm/brotli v1.0.3
github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670
github.com/cespare/xxhash v1.1.0
github.com/chai2010/webp v1.1.0 // indirect
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f
github.com/go-ole/go-ole v1.2.4 // indirect
github.com/dop251/goja v0.0.0-20210804101310-32956a348b49
github.com/go-yaml/yaml v2.1.0+incompatible
github.com/golang/protobuf v1.5.2
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible
github.com/mattn/go-sqlite3 v2.0.3+incompatible
github.com/mssola/user_agent v0.5.2
github.com/shirou/gopsutil v3.21.5+incompatible
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e // indirect
github.com/mattn/go-sqlite3 v1.14.9
github.com/miekg/dns v1.1.43
github.com/mssola/user_agent v0.5.3
github.com/pires/go-proxyproto v0.6.1
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/tklauser/go-sysconf v0.3.6 // indirect
golang.org/x/net v0.0.0-20210614182718-04defd469f4e
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22
golang.org/x/text v0.3.6
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced // indirect
google.golang.org/grpc v1.38.0
golang.org/x/image v0.0.0-20211028202545-6944b10bf410
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9
golang.org/x/sys v0.0.0-20211214234402-4825e8c3871d
golang.org/x/text v0.3.7
golang.org/x/tools v0.1.3 // indirect
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa // indirect
google.golang.org/grpc v1.43.0
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
)

89
go.sum
View File

@@ -1,4 +1,5 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
@@ -7,32 +8,56 @@ github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUWq3EgK3CesDbo8upS2Vm9/P3FtgI+Jk=
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg=
github.com/andybalholm/brotli v1.0.3 h1:fpcw+r1N1h0Poc1F/pHbW40cUm/lMEQslZtCkBQ0UnM=
github.com/andybalholm/brotli v1.0.3/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/benbjohnson/clock v1.0.3/go.mod h1:bGMdMPoPVvcYyt1gHDf4J2KE153Yf9BuiUKYMaxlTDM=
github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670 h1:FQPKKjDhzG0T4ew6dm6MGrXb4PRAi8ZmTuYuxcF62BM=
github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670/go.mod h1:iRWAFbKXMMkVQyxZ1PfGlkBr1TjATx1zy2MRprV7A3Q=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chai2010/webp v1.1.0 h1:4Ei0/BRroMF9FaXDG2e4OxwFcuW2vcXd+A6tyqTJUQQ=
github.com/chai2010/webp v1.1.0/go.mod h1:LP12PG5IFmLGHUU26tBiCBKnghxx3toZFwDjOYvd3Ow=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f h1:q/DpyjJjZs94bziQ7YkBmIlpqbVP7yw179rnzoNVX1M=
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f/go.mod h1:QGrK8vMWWHQYQ3QU9bw9Y9OPNfxccGzfb41qjvVeXtY=
github.com/dgryski/go-rendezvous v0.0.0-20200624174652-8d2f3be8b2d9/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/dlclark/regexp2 v1.4.1-0.20201116162257-a2a8dda75c91 h1:Izz0+t1Z5nI16/II7vuEo/nHjodOg0p7+OiDpjX5t1E=
github.com/dlclark/regexp2 v1.4.1-0.20201116162257-a2a8dda75c91/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc=
github.com/dop251/goja v0.0.0-20210804101310-32956a348b49 h1:CtSi0QlA2Hy+nOh8JAZoiEBLW5pliAiKJ3l1Iq1472I=
github.com/dop251/goja v0.0.0-20210804101310-32956a348b49/go.mod h1:R9ET47fwRVRPZnOGvHxxhuZcbrMCuiqOz3Rlrh4KSnk=
github.com/dop251/goja_nodejs v0.0.0-20210225215109-d91c329300e7/go.mod h1:hn7BA7c8pLvoGndExHudxTDKZ84Pyvv+90pbBjbTz0Y=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-ole/go-ole v1.2.4 h1:nNBDSCOigTSiarFpYE9J/KtEA1IOW4CNeqT9TQDqCxI=
github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM=
github.com/go-redis/redis/v8 v8.0.0-beta.7/go.mod h1:FGJAWDWFht1sQ4qxyJHZZbVyvnVcKQN0E3u5/5lRz+g=
github.com/go-sourcemap/sourcemap v2.1.3+incompatible h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU=
github.com/go-sourcemap/sourcemap v2.1.3+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-yaml/yaml v2.1.0+incompatible h1:RYi2hDdss1u4YE7GwixGzWwVo47T8UQwnTLB6vQiq+o=
@@ -49,6 +74,7 @@ github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:W
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
@@ -61,27 +87,48 @@ github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060 h1:qdLtK4PDXxk2vMKkTWl5Fl9xqYuRCukzWAgJbLHdfOo=
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060/go.mod h1:KU4mS7QNiZ7QWEuDBk1zw0/Q2LrAPZv3tycEFBsuUwc=
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24 h1:1cGulkD2SNJJRok5OKwyhP/Ddm+PgSWKOupn0cR36/A=
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24/go.mod h1:KU4mS7QNiZ7QWEuDBk1zw0/Q2LrAPZv3tycEFBsuUwc=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11 h1:DaQjoWZhLNxjhIXedVg4/vFEtHkZhK4IjIwsWdyzBLg=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11/go.mod h1:JtbX20untAjUVjZs1ZBtq80f5rJWvwtQNRL6EnuYRnY=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3 h1:aBSonas7vFcgTj9u96/bWGILGv1ZbUSTLiOzcI1ZT6c=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3/go.mod h1:H5Q7SXwbx3a97ecJkaS2sD77gspzE7HFUafBO0peEyA=
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8 h1:AojsHz9Es9B3He2MQQxeRq3TyD//o9huxUo7r1wh44g=
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8/go.mod h1:QJBY2txYhLMzwLO29iB5ujDJ3s3V7DsZ582nw4Ss+tM=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e h1:LvL4XsI70QxOGHed6yhQtAU34Kx3Qq2wwBzGFKY8zKk=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e/go.mod h1:kLgvv7o6UM+0QSf0QjAse3wReFDsb9qbZJdfexWlrQw=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible h1:1qp9iks+69h7IGLazAplzS9Ca14HAxuD5c0rbFdPGy4=
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible/go.mod h1:+ZBN7PBoh5gG6/y0ZQ85vJDBe21WnfbRrQQwTfliJJI=
github.com/mattn/go-sqlite3 v1.14.9 h1:10HX2Td0ocZpYEjhilsuo6WWtUqttj2Kb0KtD86/KYA=
github.com/mattn/go-sqlite3 v1.14.9/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mattn/go-sqlite3 v2.0.3+incompatible h1:gXHsfypPkaMZrKbD5209QV9jbUTJKjyR5WD3HYQSd+U=
github.com/mattn/go-sqlite3 v2.0.3+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/miekg/dns v1.1.43 h1:JKfpVSCB84vrAmHzyrsxB5NAr5kLoMXZArPSw7Qlgyg=
github.com/miekg/dns v1.1.43/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/mssola/user_agent v0.5.2 h1:CZkTUahjL1+OcZ5zv3kZr8QiJ8jy2H08vZIEkBeRbxo=
github.com/mssola/user_agent v0.5.2/go.mod h1:TTPno8LPY3wAIEKRpAtkdMT0f8SE24pLRGPahjCH4uw=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
github.com/mssola/user_agent v0.5.3 h1:lBRPML9mdFuIZgI2cmlQ+atbpJdLdeVl2IDodjBR578=
github.com/mssola/user_agent v0.5.3/go.mod h1:TTPno8LPY3wAIEKRpAtkdMT0f8SE24pLRGPahjCH4uw=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
@@ -92,11 +139,16 @@ github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1Cpa
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/opentracing/opentracing-go v1.1.1-0.20190913142402-a7454ce5950e/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/pires/go-proxyproto v0.6.1 h1:EBupykFmo22SDjv4fQVQd2J9NOoLPmyZA/15ldOGkPw=
github.com/pires/go-proxyproto v0.6.1/go.mod h1:Odh9VFOZJCf9G8cLW5o435Xf1J95Jw9Gw5rnCjcwzAY=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/shirou/gopsutil v3.21.5+incompatible h1:OloQyEerMi7JUrXiNzy8wQ5XN+baemxSl12QgIzt0jc=
github.com/shirou/gopsutil v3.21.5+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shirou/gopsutil v3.21.11+incompatible h1:+1+c1VGhc88SSonWP6foOcLhvnKlUeu/erjjvaPEYiI=
github.com/shirou/gopsutil v3.21.11+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72 h1:qLC7fQah7D6K1B0ujays3HV9gkFtllcxhzImRR7ArPQ=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -104,19 +156,25 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/tklauser/go-sysconf v0.3.6 h1:oc1sJWvKkmvIxhDHeKWvZS4f6AW+YcoguSfRF2/Hmo4=
github.com/tklauser/go-sysconf v0.3.6/go.mod h1:MkWzOF4RMCshBAMXuhXJs64Rte09mITnppBXY/rYEFI=
github.com/tklauser/numcpus v0.2.2 h1:oyhllyrScuYI6g+h/zUvNXNp1wy7x8qQy3t/piefldA=
github.com/tklauser/numcpus v0.2.2/go.mod h1:x3qojaO3uyYt0i56EW/VUYs7uBvdl2fkfZFu0T9wgjM=
github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
go.opentelemetry.io/otel v0.7.0/go.mod h1:aZMyHG5TqDOXEgH2tyLiXSUKly1jT3yqE9PmrzIeCdo=
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20200513190911-00229845015e/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b h1:+qEpEAPhDZ1o0x3tHzZTQDArnOixOzGD9HUJfcg0mb4=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.0.0-20211028202545-6944b10bf410 h1:hTftEOvwiOq2+O8k2D5/Q7COC7k5Qcrgc2TFURJYnvQ=
golang.org/x/image v0.0.0-20211028202545-6944b10bf410/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
@@ -128,19 +186,27 @@ golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210614182718-04defd469f4e h1:XpT3nA5TvE525Ne3hInMh6+GETgn27Zfm9dxsThnX2Q=
golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9 h1:kmreh1vGI63l2FxOAYS3Yv6ATsi7lSTuwNSVbGfJV9I=
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -155,12 +221,15 @@ golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210316164454-77fc1eacc6aa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22 h1:RqytpXGR1iVNX7psjB3ff8y7sNFinVFvkx1c8SjBkio=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211214234402-4825e8c3871d h1:1oIt9o40TWWI9FUaveVpUvBe13FNqBNVXy3ue2fcfkw=
golang.org/x/sys v0.0.0-20211214234402-4825e8c3871d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
@@ -168,6 +237,8 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
@@ -187,17 +258,25 @@ google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20191009194640-548a555dbc03/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced h1:c5geK1iMU3cDKtFrCVQIcjR3W+JOZMuhIyICMCTbtus=
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced/go.mod h1:SzzZ/N+nwJDaO1kznhnlzqS8ocJICar6hYhVyhi++24=
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa h1:I0YcKz0I7OAhddo7ya8kMnvprhcWM045PmkBdMO9zN0=
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.32.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.38.0 h1:/9BgsAsa5nWe26HqOlvlgJnqBuktYOLCgjCPqsa56W0=
google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
google.golang.org/grpc v1.43.0 h1:Eeu7bZtDZ2DpRCsLhUlcrLnvYaMK1Gz86a+hMVvELmM=
google.golang.org/grpc v1.43.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -210,14 +289,18 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

View File

@@ -11,6 +11,7 @@ import (
"os/exec"
"runtime"
"strconv"
"syscall"
"time"
)
@@ -184,7 +185,14 @@ func (this *AppCmd) runStart() {
return
}
_ = os.Setenv("EdgeBackground", "on")
cmd := exec.Command(os.Args[0])
cmd.SysProcAttr = &syscall.SysProcAttr{
Foreground: false,
Setsid: true,
}
err := cmd.Start()
if err != nil {
fmt.Println(this.product+" start failed:", err.Error())

View File

@@ -6,6 +6,10 @@ import (
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/utils/time"
"log"
"os"
"runtime"
"strconv"
"strings"
)
type LogWriter struct {
@@ -34,7 +38,26 @@ func (this *LogWriter) Init() {
}
func (this *LogWriter) Write(message string) {
log.Println(message)
backgroundEnv, _ := os.LookupEnv("EdgeBackground")
if backgroundEnv != "on" {
// 文件和行号
var file string
var line int
if Tea.IsTesting() {
var callDepth = 3
var ok bool
_, file, line, ok = runtime.Caller(callDepth)
if ok {
file = this.packagePath(file)
}
}
if len(file) > 0 {
log.Println(message + " (" + file + ":" + strconv.Itoa(line) + ")")
} else {
log.Println(message)
}
}
if this.fileAppender != nil {
_, err := this.fileAppender.AppendString(timeutil.Format("Y/m/d H:i:s ") + message + "\n")
@@ -49,3 +72,11 @@ func (this *LogWriter) Close() {
_ = this.fileAppender.Close()
}
}
func (this *LogWriter) packagePath(path string) string {
var pieces = strings.Split(path, "/")
if len(pieces) >= 2 {
return strings.Join(pieces[len(pieces)-2:], "/")
}
return path
}

View File

@@ -0,0 +1,10 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
type HotItem struct {
Key string
ExpiresAt int64
Hits uint32
Status int
}

View File

@@ -2,6 +2,7 @@ package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"time"
)
type ItemType = int
@@ -11,15 +12,26 @@ const (
ItemTypeMemory ItemType = 2
)
// 计算当前周
// 不要用YW因为需要计算两周是否临近
func currentWeek() int32 {
return int32(time.Now().Unix() / 86400)
}
type Item struct {
Type ItemType `json:"type"`
Key string `json:"key"`
ExpiredAt int64 `json:"expiredAt"`
StaleAt int64 `json:"staleAt"`
HeaderSize int64 `json:"headerSize"`
BodySize int64 `json:"bodySize"`
MetaSize int64 `json:"metaSize"`
Host string `json:"host"` // 主机名
ServerId int64 `json:"serverId"` // 服务ID
Week1Hits int64 `json:"week1Hits"`
Week2Hits int64 `json:"week2Hits"`
Week int32 `json:"week"`
}
func (this *Item) IsExpired() bool {
@@ -27,9 +39,23 @@ func (this *Item) IsExpired() bool {
}
func (this *Item) TotalSize() int64 {
return this.Size() + this.MetaSize + int64(len(this.Key)) + 64
return this.Size() + this.MetaSize + int64(len(this.Key)) + int64(len(this.Host))
}
func (this *Item) Size() int64 {
return this.HeaderSize + this.BodySize
}
func (this *Item) IncreaseHit(week int32) {
if this.Week == week {
this.Week2Hits++
} else {
if week-this.Week == 1 {
this.Week1Hits = this.Week2Hits
} else {
this.Week1Hits = 0
}
this.Week2Hits = 1
this.Week = week
}
}

View File

@@ -0,0 +1,83 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
"runtime"
"testing"
"time"
)
func TestItem_IncreaseHit(t *testing.T) {
var week = currentWeek()
var item = &Item{}
//item.Week = 2704
item.Week2Hits = 100
item.IncreaseHit(week)
t.Log("week:", item.Week, "week1:", item.Week1Hits, "week2:", item.Week2Hits)
item.IncreaseHit(week)
t.Log("week:", item.Week, "week1:", item.Week1Hits, "week2:", item.Week2Hits)
}
func TestItems_Memory(t *testing.T) {
var stat = &runtime.MemStats{}
runtime.ReadMemStats(stat)
var memory1 = stat.HeapInuse
var items = []*Item{}
for i := 0; i < 10_000_000; i++ {
items = append(items, &Item{
Key: types.String(i),
})
}
runtime.ReadMemStats(stat)
var memory2 = stat.HeapInuse
t.Log(memory1, memory2, (memory2-memory1)/1024/1024, "M")
var weekItems = make(map[string]*Item, 10_000_000)
for _, item := range items {
weekItems[item.Key] = item
}
runtime.ReadMemStats(stat)
var memory3 = stat.HeapInuse
t.Log(memory2, memory3, (memory3-memory2)/1024/1024, "M")
time.Sleep(1 * time.Second)
t.Log(len(items), len(weekItems))
}
func TestItems_Memory2(t *testing.T) {
var stat = &runtime.MemStats{}
runtime.ReadMemStats(stat)
var memory1 = stat.HeapInuse
var items = map[int32]map[string]zero.Zero{}
for i := 0; i < 10_000_000; i++ {
var week = int32((time.Now().Unix() - int64(86400*rands.Int(0, 300))) / (86400 * 7))
m, ok := items[week]
if !ok {
m = map[string]zero.Zero{}
items[week] = m
}
m[types.String(int64(i)*1_000_000)] = zero.New()
}
runtime.ReadMemStats(stat)
var memory2 = stat.HeapInuse
t.Log(memory1, memory2, (memory2-memory1)/1024/1024, "M")
time.Sleep(1 * time.Second)
for w, i := range items {
t.Log(w, len(i))
}
}

View File

@@ -4,12 +4,16 @@ package caches
import (
"database/sql"
"errors"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/ttlcache"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
timeutil "github.com/iwind/TeaGo/utils/time"
_ "github.com/mattn/go-sqlite3"
"os"
"strconv"
"sync/atomic"
"time"
)
@@ -23,6 +27,7 @@ type FileList struct {
onAdd func(item *Item)
onRemove func(item *Item)
// cacheItems
existsByHashStmt *sql.Stmt // 根据hash检查是否存在
insertStmt *sql.Stmt // 写入数据
selectByHashStmt *sql.Stmt // 使用hash查询数据
@@ -31,14 +36,27 @@ type FileList struct {
purgeStmt *sql.Stmt // 清理
deleteAllStmt *sql.Stmt // 删除所有数据
// hits
insertHitStmt *sql.Stmt // 写入数据
increaseHitStmt *sql.Stmt // 增加点击量
deleteHitByHashStmt *sql.Stmt // 根据hash删除数据
lfuHitsStmt *sql.Stmt // 读取老的数据
oldTables []string
itemsTableName string
hitsTableName string
isClosed bool
isReady bool
memoryCache *ttlcache.Cache
}
func NewFileList(dir string) ListInterface {
return &FileList{dir: dir}
return &FileList{
dir: dir,
memoryCache: ttlcache.NewCache(),
}
}
func (this *FileList) Init() error {
@@ -52,28 +70,24 @@ func (this *FileList) Init() error {
remotelogs.Println("CACHE", "create cache dir '"+this.dir+"'")
}
this.itemsTableName = "cacheItems_v2"
this.itemsTableName = "cacheItems_v3"
this.hitsTableName = "hits"
var dir = this.dir
if dir == "/" {
// 防止sqlite提示authority错误
dir = ""
}
db, err := sql.Open("sqlite3", "file:"+dir+"/index.db?cache=shared&mode=rwc&_journal_mode=WAL")
var dbPath = dir + "/index.db"
remotelogs.Println("CACHE", "loading database '"+dbPath+"'")
db, err := sql.Open("sqlite3", "file:"+dbPath+"?cache=shared&mode=rwc&_journal_mode=WAL")
if err != nil {
return err
return errors.New("open database failed: " + err.Error())
}
db.SetMaxOpenConns(1)
this.db = db
// 清除旧表
this.oldTables = []string{
"cacheItems",
}
err = this.removeOldTables()
if err != nil {
remotelogs.Warn("CACHE", "clean old tables failed: "+err.Error())
}
db.SetMaxOpenConns(1)
this.db = db
// TODO 耗时过长,暂时不整理数据库
/**_, err = db.Exec("VACUUM")
@@ -84,7 +98,18 @@ func (this *FileList) Init() error {
// 创建
err = this.initTables(db, 1)
if err != nil {
return err
return errors.New("init tables failed: " + err.Error())
}
// 清除旧表
// 这个一定要在initTables()之后,因为老的数据需要转移
this.oldTables = []string{
"cacheItems",
"cacheItems_v2",
}
err = this.removeOldTables()
if err != nil {
remotelogs.Warn("CACHE", "clean old tables failed: "+err.Error())
}
// 读取总数量
@@ -100,12 +125,12 @@ func (this *FileList) Init() error {
this.total = total
// 常用语句
this.existsByHashStmt, err = this.db.Prepare(`SELECT "bodySize" FROM "` + this.itemsTableName + `" WHERE "hash"=? AND expiredAt>? LIMIT 1`)
this.existsByHashStmt, err = this.db.Prepare(`SELECT "expiredAt" FROM "` + this.itemsTableName + `" WHERE "hash"=? AND expiredAt>? LIMIT 1`)
if err != nil {
return err
}
this.insertStmt, err = this.db.Prepare(`INSERT INTO "` + this.itemsTableName + `" ("hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "host", "serverId", "createdAt") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`)
this.insertStmt, err = this.db.Prepare(`INSERT INTO "` + this.itemsTableName + `" ("hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "staleAt", "host", "serverId", "createdAt") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
return err
}
@@ -125,7 +150,7 @@ func (this *FileList) Init() error {
return err
}
this.purgeStmt, err = this.db.Prepare(`SELECT "hash" FROM "` + this.itemsTableName + `" WHERE expiredAt<=? LIMIT ?`)
this.purgeStmt, err = this.db.Prepare(`SELECT "hash" FROM "` + this.itemsTableName + `" WHERE staleAt<=? LIMIT ?`)
if err != nil {
return err
}
@@ -135,6 +160,25 @@ func (this *FileList) Init() error {
return err
}
this.insertHitStmt, err = this.db.Prepare(`INSERT INTO "` + this.hitsTableName + `" ("hash", "week2Hits", "week") VALUES (?, 1, ?)`)
this.increaseHitStmt, err = this.db.Prepare(`INSERT INTO "` + this.hitsTableName + `" ("hash", "week2Hits", "week") VALUES (?, 1, ?) ON CONFLICT("hash") DO UPDATE SET "week1Hits"=IIF("week"=?, "week1Hits", "week2Hits"), "week2Hits"=IIF("week"=?, "week2Hits"+1, 1), "week"=?`)
if err != nil {
return err
}
this.deleteHitByHashStmt, err = this.db.Prepare(`DELETE FROM "` + this.hitsTableName + `" WHERE "hash"=?`)
if err != nil {
return err
}
this.lfuHitsStmt, err = this.db.Prepare(`SELECT "hash" FROM "` + this.hitsTableName + `" ORDER BY "week" ASC, "week1Hits"+"week2Hits" ASC LIMIT ?`)
if err != nil {
return err
}
this.isReady = true
return nil
}
@@ -144,11 +188,20 @@ func (this *FileList) Reset() error {
}
func (this *FileList) Add(hash string, item *Item) error {
if this.isClosed {
if !this.isReady {
return nil
}
_, err := this.insertStmt.Exec(hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.Host, item.ServerId, utils.UnixTime())
if item.StaleAt == 0 {
item.StaleAt = item.ExpiredAt
}
_, err := this.insertStmt.Exec(hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.StaleAt, item.Host, item.ServerId, utils.UnixTime())
if err != nil {
return err
}
_, err = this.insertHitStmt.Exec(hash, timeutil.Format("YW"))
if err != nil {
return err
}
@@ -162,10 +215,15 @@ func (this *FileList) Add(hash string, item *Item) error {
}
func (this *FileList) Exist(hash string) (bool, error) {
if this.isClosed {
if !this.isReady {
return false, nil
}
item := this.memoryCache.Read(hash)
if item != nil {
return true, nil
}
rows, err := this.existsByHashStmt.Query(hash, time.Now().Unix())
if err != nil {
return false, err
@@ -174,6 +232,12 @@ func (this *FileList) Exist(hash string) (bool, error) {
_ = rows.Close()
}()
if rows.Next() {
var expiredAt int64
err = rows.Scan(&expiredAt)
if err != nil {
return false, nil
}
this.memoryCache.Write(hash, 1, expiredAt)
return true, nil
}
return false, nil
@@ -181,7 +245,7 @@ func (this *FileList) Exist(hash string) (bool, error) {
// CleanPrefix 清理某个前缀的缓存数据
func (this *FileList) CleanPrefix(prefix string) error {
if this.isClosed {
if !this.isReady {
return nil
}
@@ -189,9 +253,14 @@ func (this *FileList) CleanPrefix(prefix string) error {
return nil
}
defer func() {
this.memoryCache.Clean()
}()
var count = int64(10000)
var staleLife = 600 // TODO 需要可以设置
for {
result, err := this.db.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0 WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)=1 LIMIT `+strconv.FormatInt(count, 10)+`)`, utils.UnixTime(), prefix)
result, err := this.db.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0,staleAt=? WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)=1 LIMIT `+types.String(count)+`)`, utils.UnixTime()+int64(staleLife), utils.UnixTime(), prefix)
if err != nil {
return err
}
@@ -206,10 +275,13 @@ func (this *FileList) CleanPrefix(prefix string) error {
}
func (this *FileList) Remove(hash string) error {
if this.isClosed {
if !this.isReady {
return nil
}
// 从缓存中删除
this.memoryCache.Delete(hash)
row := this.selectByHashStmt.QueryRow(hash)
if row.Err() != nil {
return row.Err()
@@ -229,6 +301,11 @@ func (this *FileList) Remove(hash string) error {
return err
}
_, err = this.deleteHitByHashStmt.Exec(hash)
if err != nil {
return err
}
atomic.AddInt64(&this.total, -1)
if this.onRemove != nil {
@@ -241,9 +318,9 @@ func (this *FileList) Remove(hash string) error {
// Purge 清理过期的缓存
// count 每次遍历的最大数量,控制此数字可以保证每次清理的时候不用花太多时间
// callback 每次发现过期key的调用
func (this *FileList) Purge(count int, callback func(hash string) error) error {
if this.isClosed {
return nil
func (this *FileList) Purge(count int, callback func(hash string) error) (int, error) {
if !this.isReady {
return 0, nil
}
if count <= 0 {
@@ -251,11 +328,56 @@ func (this *FileList) Purge(count int, callback func(hash string) error) error {
}
rows, err := this.purgeStmt.Query(time.Now().Unix(), count)
if err != nil {
return 0, err
}
hashStrings := []string{}
var countFound = 0
for rows.Next() {
var hash string
err = rows.Scan(&hash)
if err != nil {
_ = rows.Close()
return 0, err
}
hashStrings = append(hashStrings, hash)
countFound++
}
_ = rows.Close() // 不能使用defer防止读写冲突
// 不在 rows.Next() 循环中操作是为了避免死锁
for _, hash := range hashStrings {
err = this.Remove(hash)
if err != nil {
return 0, err
}
err = callback(hash)
if err != nil {
return 0, err
}
}
return countFound, nil
}
func (this *FileList) PurgeLFU(count int, callback func(hash string) error) error {
if !this.isReady {
return nil
}
if count <= 0 {
return nil
}
rows, err := this.lfuHitsStmt.Query(count)
if err != nil {
return err
}
hashStrings := []string{}
var countFound = 0
for rows.Next() {
var hash string
err = rows.Scan(&hash)
@@ -264,6 +386,7 @@ func (this *FileList) Purge(count int, callback func(hash string) error) error {
return err
}
hashStrings = append(hashStrings, hash)
countFound++
}
_ = rows.Close() // 不能使用defer防止读写冲突
@@ -279,15 +402,16 @@ func (this *FileList) Purge(count int, callback func(hash string) error) error {
return err
}
}
return nil
}
func (this *FileList) CleanAll() error {
if this.isClosed {
if !this.isReady {
return nil
}
this.memoryCache.Clean()
_, err := this.deleteAllStmt.Exec()
if err != nil {
return err
@@ -297,7 +421,7 @@ func (this *FileList) CleanAll() error {
}
func (this *FileList) Stat(check func(hash string) bool) (*Stat, error) {
if this.isClosed {
if !this.isReady {
return &Stat{}, nil
}
@@ -321,6 +445,13 @@ func (this *FileList) Count() (int64, error) {
return atomic.LoadInt64(&this.total), nil
}
// IncreaseHit 增加点击量
func (this *FileList) IncreaseHit(hash string) error {
var week = timeutil.Format("YW")
_, err := this.increaseHitStmt.Exec(hash, week, week, week, week)
return err
}
// OnAdd 添加事件
func (this *FileList) OnAdd(f func(item *Item)) {
this.onAdd = f
@@ -333,6 +464,9 @@ func (this *FileList) OnRemove(f func(item *Item)) {
func (this *FileList) Close() error {
this.isClosed = true
this.isReady = false
this.memoryCache.Destroy()
if this.db != nil {
_ = this.existsByHashStmt.Close()
@@ -343,6 +477,11 @@ func (this *FileList) Close() error {
_ = this.purgeStmt.Close()
_ = this.deleteAllStmt.Close()
_ = this.insertHitStmt.Close()
_ = this.increaseHitStmt.Close()
_ = this.deleteHitByHashStmt.Close()
_ = this.lfuHitsStmt.Close()
return this.db.Close()
}
return nil
@@ -350,7 +489,17 @@ func (this *FileList) Close() error {
// 初始化
func (this *FileList) initTables(db *sql.DB, times int) error {
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
// 检查是否存在
_, err := db.Exec(`SELECT id FROM "` + this.itemsTableName + `" LIMIT 1`)
var notFound = false
if err != nil {
notFound = true
}
{
// expiredAt - 过期时间,用来判断有无过期
// staleAt - 陈旧最大时间,用来清理缓存
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"key" varchar(1024),
@@ -358,6 +507,7 @@ func (this *FileList) initTables(db *sql.DB, times int) error {
"bodySize" integer DEFAULT 0,
"metaSize" integer DEFAULT 0,
"expiredAt" integer DEFAULT 0,
"staleAt" integer DEFAULT 0,
"createdAt" integer DEFAULT 0,
"host" varchar(128),
"serverId" integer
@@ -373,6 +523,11 @@ ON "` + this.itemsTableName + `" (
"expiredAt" ASC
);
CREATE INDEX IF NOT EXISTS "staleAt"
ON "` + this.itemsTableName + `" (
"staleAt" ASC
);
CREATE UNIQUE INDEX IF NOT EXISTS "hash"
ON "` + this.itemsTableName + `" (
"hash" ASC
@@ -383,17 +538,60 @@ ON "` + this.itemsTableName + `" (
"serverId" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.itemsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.itemsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
}
return err
}
return err
}
}
return err
// 如果数据为空,从老数据中加载数据
if notFound {
// v2 => v3
remotelogs.Println("CACHE", "transferring old data from v2 to v3 ...")
result, err := db.Exec(`INSERT INTO "` + this.itemsTableName + `" ("id", "hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "createdAt", "host", "serverId", "staleAt") SELECT "id", "hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "createdAt", "host", "serverId", "expiredAt"+600 FROM cacheItems_v2`)
if err != nil {
remotelogs.Println("CACHE", "transfer old data from v2 to v3 failed: "+err.Error())
} else {
count, _ := result.RowsAffected()
remotelogs.Println("CACHE", "transfer old data from v2 to v3 finished, "+types.String(count)+" rows transferred")
}
}
{
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.hitsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"week1Hits" integer DEFAULT 0,
"week2Hits" integer DEFAULT 0,
"week" varchar(6)
);
CREATE UNIQUE INDEX IF NOT EXISTS "hits_hash"
ON "` + this.hitsTableName + `" (
"hash" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.hitsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
}
return err
}
return err
}
}
return nil
@@ -416,11 +614,11 @@ func (this *FileList) removeOldTables() error {
}
if lists.ContainsString(this.oldTables, name) {
// 异步执行
go func() {
goman.New(func() {
remotelogs.Println("CACHE", "remove old table '"+name+"' ...")
_, _ = this.db.Exec(`DROP TABLE "` + name + `"`)
remotelogs.Println("CACHE", "remove old table '"+name+"' done")
}()
})
}
}

View File

@@ -3,8 +3,10 @@
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
stringutil "github.com/iwind/TeaGo/utils/string"
"strconv"
"sync"
@@ -50,7 +52,7 @@ func TestFileList_Add_Many(t *testing.T) {
}
before := time.Now()
for i := 0; i < 2000_0000; i++ {
u := "http://edge.teaos.cn/123456" + strconv.Itoa(i)
u := "https://edge.teaos.cn/123456" + strconv.Itoa(i)
_ = list.Add(stringutil.Md5(u), &Item{
Key: u,
ExpiredAt: time.Now().Unix() + 3600,
@@ -126,7 +128,7 @@ func TestFileList_Exist_Many_DB(t *testing.T) {
}()
for i := 0; i < threads; i++ {
go func() {
goman.New(func() {
defer wg.Done()
for {
@@ -142,7 +144,7 @@ func TestFileList_Exist_Many_DB(t *testing.T) {
return
}
}
}()
})
}
wg.Wait()
t.Log("left:", count)
@@ -184,7 +186,7 @@ func TestFileList_Purge(t *testing.T) {
if err != nil {
t.Fatal(err)
}
err = list.Purge(2, func(hash string) error {
_, err = list.Purge(2, func(hash string) error {
t.Log(hash)
return nil
})
@@ -257,6 +259,50 @@ func TestFileList_Conflict(t *testing.T) {
t.Log("after exists")
}
func TestFileList_IIF(t *testing.T) {
list := NewFileList(Tea.Root + "/data").(*FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
rows, err := list.db.Query("SELECT IIF(0, 2, 3)")
if err != nil {
t.Fatal(err)
}
defer func() {
_ = rows.Close()
}()
if rows.Next() {
var result int
err = rows.Scan(&result)
if err != nil {
t.Fatal(err)
}
t.Log("result:", result)
}
}
func TestFileList_IncreaseHit(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
err := list.Init()
if err != nil {
t.Fatal(err)
}
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
for i := 0; i < 1000_000; i++ {
err = list.IncreaseHit(stringutil.Md5("abc" + types.String(i)))
}
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func BenchmarkFileList_Exist(b *testing.B) {
list := NewFileList(Tea.Root + "/data")
err := list.Init()

View File

@@ -22,7 +22,10 @@ type ListInterface interface {
Remove(hash string) error
// Purge 清理过期数据
Purge(count int, callback func(hash string) error) error
Purge(count int, callback func(hash string) error) (int, error)
// PurgeLFU 清理LFU数据
PurgeLFU(count int, callback func(hash string) error) error
// CleanAll 清除所有缓存
CleanAll() error
@@ -41,4 +44,7 @@ type ListInterface interface {
// Close 关闭
Close() error
// IncreaseHit 增加点击量
IncreaseHit(hash string) error
}

View File

@@ -1,16 +1,24 @@
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/logs"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
)
// MemoryList 内存缓存列表管理
type MemoryList struct {
count int64
itemMaps map[string]map[string]*Item // prefix => { hash => item }
weekItemMaps map[int32]map[string]zero.Zero // week => { hash => Zero }
minWeek int32
prefixes []string
locker sync.RWMutex
onAdd func(item *Item)
@@ -21,7 +29,9 @@ type MemoryList struct {
func NewMemoryList() ListInterface {
return &MemoryList{
itemMaps: map[string]map[string]*Item{},
itemMaps: map[string]map[string]*Item{},
weekItemMaps: map[int32]map[string]zero.Zero{},
minWeek: currentWeek(),
}
}
@@ -43,11 +53,19 @@ func (this *MemoryList) Reset() error {
for key := range this.itemMaps {
this.itemMaps[key] = map[string]*Item{}
}
this.weekItemMaps = map[int32]map[string]zero.Zero{}
this.locker.Unlock()
atomic.StoreInt64(&this.count, 0)
return nil
}
func (this *MemoryList) Add(hash string, item *Item) error {
if item.Week == 0 {
item.Week = currentWeek()
}
this.locker.Lock()
prefix := this.prefix(hash)
@@ -60,9 +78,20 @@ func (this *MemoryList) Add(hash string, item *Item) error {
// 先删除,为了可以正确触发统计
oldItem, ok := itemMap[hash]
if ok {
// 从week map中删除
if oldItem.Week > 0 {
wm, ok := this.weekItemMaps[oldItem.Week]
if ok {
delete(wm, hash)
}
}
// 回调
if this.onRemove != nil {
this.onRemove(oldItem)
}
} else {
atomic.AddInt64(&this.count, 1)
}
// 添加
@@ -71,6 +100,15 @@ func (this *MemoryList) Add(hash string, item *Item) error {
}
itemMap[hash] = item
// week map
wm, ok := this.weekItemMaps[item.Week]
if ok {
wm[hash] = zero.New()
} else {
this.weekItemMaps[item.Week] = map[string]zero.Zero{hash: zero.New()}
}
this.locker.Unlock()
return nil
}
@@ -122,7 +160,17 @@ func (this *MemoryList) Remove(hash string) error {
if this.onRemove != nil {
this.onRemove(item)
}
atomic.AddInt64(&this.count, -1)
delete(itemMap, hash)
// week map
if item.Week > 0 {
wm, ok := this.weekItemMaps[item.Week]
if ok {
delete(wm, hash)
}
}
}
this.locker.Unlock()
@@ -132,7 +180,7 @@ func (this *MemoryList) Remove(hash string) error {
// Purge 清理过期的缓存
// count 每次遍历的最大数量,控制此数字可以保证每次清理的时候不用花太多时间
// callback 每次发现过期key的调用
func (this *MemoryList) Purge(count int, callback func(hash string) error) error {
func (this *MemoryList) Purge(count int, callback func(hash string) error) (int, error) {
this.locker.Lock()
deletedHashList := []string{}
@@ -146,8 +194,9 @@ func (this *MemoryList) Purge(count int, callback func(hash string) error) error
itemMap, ok := this.itemMaps[prefix]
if !ok {
this.locker.Unlock()
return nil
return 0, nil
}
var countFound = 0
for hash, item := range itemMap {
if count <= 0 {
break
@@ -157,14 +206,100 @@ func (this *MemoryList) Purge(count int, callback func(hash string) error) error
if this.onRemove != nil {
this.onRemove(item)
}
atomic.AddInt64(&this.count, -1)
delete(itemMap, hash)
deletedHashList = append(deletedHashList, hash)
// week map
if item.Week > 0 {
wm, ok := this.weekItemMaps[item.Week]
if ok {
delete(wm, hash)
}
}
countFound++
}
count--
}
this.locker.Unlock()
// 执行外部操作
for _, hash := range deletedHashList {
if callback != nil {
err := callback(hash)
if err != nil {
return 0, err
}
}
}
return countFound, nil
}
func (this *MemoryList) PurgeLFU(count int, callback func(hash string) error) error {
if count <= 0 {
return nil
}
var week = currentWeek()
if this.minWeek > week {
this.minWeek = week
}
var deletedHashList = []string{}
Loop:
for w := this.minWeek; w <= week; w++ {
this.minWeek = w
this.locker.Lock()
wm, ok := this.weekItemMaps[w]
if ok {
var wc = len(wm)
if wc == 0 {
delete(this.weekItemMaps, w)
} else {
if wc <= count {
delete(this.weekItemMaps, w)
}
// TODO 未来支持按照点击量排序
for hash := range wm {
count--
if count < 0 {
this.locker.Unlock()
break Loop
}
delete(wm, hash)
itemMap, ok := this.itemMaps[this.prefix(hash)]
if !ok {
continue
}
item, ok := itemMap[hash]
if !ok {
continue
}
if this.onRemove != nil {
this.onRemove(item)
}
atomic.AddInt64(&this.count, -1)
delete(itemMap, hash)
deletedHashList = append(deletedHashList, hash)
}
}
} else {
delete(this.weekItemMaps, w)
}
this.locker.Unlock()
}
// 执行外部操作
for _, hash := range deletedHashList {
if callback != nil {
@@ -174,6 +309,7 @@ func (this *MemoryList) Purge(count int, callback func(hash string) error) error
}
}
}
return nil
}
@@ -206,13 +342,8 @@ func (this *MemoryList) Stat(check func(hash string) bool) (*Stat, error) {
// Count 总数量
func (this *MemoryList) Count() (int64, error) {
this.locker.RLock()
var count = 0
for _, itemMap := range this.itemMaps {
count += len(itemMap)
}
this.locker.RUnlock()
return int64(count), nil
var count = atomic.LoadInt64(&this.count)
return count, nil
}
// OnAdd 添加事件
@@ -229,6 +360,41 @@ func (this *MemoryList) Close() error {
return nil
}
// IncreaseHit 增加点击量
func (this *MemoryList) IncreaseHit(hash string) error {
this.locker.Lock()
itemMap, ok := this.itemMaps[this.prefix(hash)]
if !ok {
this.locker.Unlock()
return nil
}
item, ok := itemMap[hash]
if ok {
var week = currentWeek()
// 交换位置
if item.Week > 0 && item.Week != week {
wm, ok := this.weekItemMaps[item.Week]
if ok {
delete(wm, hash)
}
wm, ok = this.weekItemMaps[week]
if ok {
wm[hash] = zero.New()
} else {
this.weekItemMaps[week] = map[string]zero.Zero{hash: zero.New()}
}
}
item.IncreaseHit(week)
}
this.locker.Unlock()
return nil
}
func (this *MemoryList) print(t *testing.T) {
this.locker.Lock()
for _, itemMap := range this.itemMaps {

View File

@@ -31,6 +31,8 @@ func TestMemoryList_Add(t *testing.T) {
})
t.Log(list.prefixes)
logs.PrintAsJSON(list.itemMaps, t)
logs.PrintAsJSON(list.weekItemMaps, t)
t.Log(list.Count())
}
func TestMemoryList_Remove(t *testing.T) {
@@ -48,6 +50,8 @@ func TestMemoryList_Remove(t *testing.T) {
})
_ = list.Remove("b")
list.print(t)
logs.PrintAsJSON(list.weekItemMaps, t)
t.Log(list.Count())
}
func TestMemoryList_Purge(t *testing.T) {
@@ -73,19 +77,22 @@ func TestMemoryList_Purge(t *testing.T) {
ExpiredAt: time.Now().Unix() - 2,
HeaderSize: 1024,
})
_ = list.Purge(100, func(hash string) error {
_, _ = list.Purge(100, func(hash string) error {
t.Log("delete:", hash)
return nil
})
list.print(t)
logs.PrintAsJSON(list.weekItemMaps, t)
for i := 0; i < 1000; i++ {
_ = list.Purge(100, func(hash string) error {
_, _ = list.Purge(100, func(hash string) error {
t.Log("delete:", hash)
return nil
})
t.Log(list.purgeIndex)
}
t.Log(list.Count())
}
func TestMemoryList_Purge_Large_List(t *testing.T) {
@@ -139,7 +146,7 @@ func TestMemoryList_CleanPrefix(t *testing.T) {
_ = list.Init()
before := time.Now()
for i := 0; i < 1_000_000; i++ {
key := "http://www.teaos.cn/hello/" + strconv.Itoa(i/10000) + "/" + strconv.Itoa(i) + ".html"
key := "https://www.teaos.cn/hello/" + strconv.Itoa(i/10000) + "/" + strconv.Itoa(i) + ".html"
_ = list.Add(fmt.Sprintf("%d", xxhash.Sum64String(key)), &Item{
Key: key,
ExpiredAt: time.Now().Unix() + 3600,
@@ -150,7 +157,7 @@ func TestMemoryList_CleanPrefix(t *testing.T) {
t.Log(time.Since(before).Seconds()*1000, "ms")
before = time.Now()
err := list.CleanPrefix("http://www.teaos.cn/hello/10")
err := list.CleanPrefix("https://www.teaos.cn/hello/10")
if err != nil {
t.Fatal(err)
}
@@ -162,11 +169,77 @@ func TestMemoryList_CleanPrefix(t *testing.T) {
t.Log(time.Since(before).Seconds()*1000, "ms")
}
func TestMemoryList_PurgeLFU(t *testing.T) {
var list = NewMemoryList().(*MemoryList)
list.minWeek = 2704
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
t.Log("current week:", currentWeek())
_ = list.Add("1", &Item{})
_ = list.Add("2", &Item{})
_ = list.Add("3", &Item{})
_ = list.Add("4", &Item{})
_ = list.Add("5", &Item{})
_ = list.Add("6", &Item{Week: 2704})
_ = list.Add("7", &Item{Week: 2704})
_ = list.Add("8", &Item{Week: 2705})
err := list.PurgeLFU(2, func(hash string) error {
t.Log("purge lfu:", hash)
return nil
})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
logs.PrintAsJSON(list.weekItemMaps, t)
t.Log(list.Count())
}
func TestMemoryList_IncreaseHit(t *testing.T) {
var list = NewMemoryList().(*MemoryList)
var item = &Item{}
item.Week = 2705
item.Week2Hits = 100
_ = list.Add("a", &Item{})
_ = list.Add("a", item)
t.Log("hits1:", item.Week1Hits, "hits2:", item.Week2Hits, "week:", item.Week)
logs.PrintAsJSON(list.weekItemMaps, t)
_ = list.IncreaseHit("a")
t.Log("hits1:", item.Week1Hits, "hits2:", item.Week2Hits, "week:", item.Week)
logs.PrintAsJSON(list.weekItemMaps, t)
_ = list.IncreaseHit("a")
t.Log("hits1:", item.Week1Hits, "hits2:", item.Week2Hits, "week:", item.Week)
logs.PrintAsJSON(list.weekItemMaps, t)
}
func TestMemoryList_CleanAll(t *testing.T) {
var list = NewMemoryList().(*MemoryList)
var item = &Item{}
item.Week = 2705
item.Week2Hits = 100
_ = list.Add("a", &Item{})
_ = list.CleanAll()
logs.PrintAsJSON(list.itemMaps, t)
logs.PrintAsJSON(list.weekItemMaps, t)
t.Log(list.Count())
}
func TestMemoryList_GC(t *testing.T) {
list := NewMemoryList().(*MemoryList)
_ = list.Init()
for i := 0; i < 1_000_000; i++ {
key := "http://www.teaos.cn/hello" + strconv.Itoa(i/100000) + "/" + strconv.Itoa(i) + ".html"
key := "https://www.teaos.cn/hello" + strconv.Itoa(i/100000) + "/" + strconv.Itoa(i) + ".html"
_ = list.Add(fmt.Sprintf("%d", xxhash.Sum64String(key)), &Item{
Key: key,
ExpiredAt: 0,

View File

@@ -3,8 +3,10 @@ package caches
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/types"
"strconv"
"sync"
@@ -12,6 +14,13 @@ import (
var SharedManager = NewManager()
func init() {
events.On(events.EventQuit, func() {
logs.Println("CACHE", "quiting cache manager")
SharedManager.UpdatePolicies([]*serverconfigs.HTTPCachePolicy{})
})
}
// Manager 缓存策略管理器
type Manager struct {
// 全局配置
@@ -25,10 +34,12 @@ type Manager struct {
// NewManager 获取管理器对象
func NewManager() *Manager {
return &Manager{
var m = &Manager{
policyMap: map[int64]*serverconfigs.HTTPCachePolicy{},
storageMap: map[int64]StorageInterface{},
}
return m
}
// UpdatePolicies 重新设置策略
@@ -135,7 +146,7 @@ func (this *Manager) NewStorageWithPolicy(policy *serverconfigs.HTTPCachePolicy)
case serverconfigs.CachePolicyStorageFile:
return NewFileStorage(policy)
case serverconfigs.CachePolicyStorageMemory:
return NewMemoryStorage(policy)
return NewMemoryStorage(policy, nil)
}
return nil
}

View File

@@ -9,6 +9,9 @@ type Reader interface {
// TypeName 类型名称
TypeName() string
// ExpiresAt 过期时间
ExpiresAt() int64
// Status 状态码
Status() int
@@ -21,6 +24,9 @@ type Reader interface {
// ReadBody 读取Body
ReadBody(buf []byte, callback ReaderFunc) error
// Read 实现io.Reader接口
Read(buf []byte) (int, error)
// ReadBodyRange 读取某个范围内的Body
ReadBodyRange(buf []byte, start int64, end int64, callback ReaderFunc) error

View File

@@ -11,6 +11,7 @@ import (
type FileReader struct {
fp *os.File
expiresAt int64
status int
headerOffset int64
headerSize int
@@ -34,13 +35,7 @@ func (this *FileReader) Init() error {
}
}()
// 读取状态
_, err := this.fp.Seek(SizeExpiresAt, io.SeekStart)
if err != nil {
_ = this.discard()
return err
}
buf := make([]byte, 3)
var buf = make([]byte, SizeMeta)
ok, err := this.readToBuff(this.fp, buf)
if err != nil {
return err
@@ -48,37 +43,20 @@ func (this *FileReader) Init() error {
if !ok {
return ErrNotFound
}
status := types.Int(string(buf))
this.expiresAt = int64(binary.BigEndian.Uint32(buf[:SizeExpiresAt]))
status := types.Int(string(buf[SizeExpiresAt : SizeExpiresAt+SizeStatus]))
if status < 100 || status > 999 {
return errors.New("invalid status")
}
this.status = status
// URL
_, err = this.fp.Seek(SizeExpiresAt+SizeStatus, io.SeekStart)
if err != nil {
return err
}
bytes4 := make([]byte, 4)
ok, err = this.readToBuff(this.fp, bytes4)
if err != nil {
return err
}
if !ok {
return ErrNotFound
}
urlLength := binary.BigEndian.Uint32(bytes4)
urlLength := binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus : SizeExpiresAt+SizeStatus+SizeURLLength])
// header
ok, err = this.readToBuff(this.fp, bytes4)
if err != nil {
return err
}
if !ok {
return ErrNotFound
}
headerSize := int(binary.BigEndian.Uint32(bytes4))
headerSize := int(binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus+SizeURLLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength]))
if headerSize == 0 {
return nil
}
@@ -86,16 +64,9 @@ func (this *FileReader) Init() error {
this.headerOffset = int64(SizeMeta) + int64(urlLength)
// body
bytes8 := make([]byte, 8)
ok, err = this.readToBuff(this.fp, bytes8)
if err != nil {
return err
}
if !ok {
return ErrNotFound
}
bodySize := int(binary.BigEndian.Uint64(bytes8))
bodySize := int(binary.BigEndian.Uint64(buf[SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength+SizeBodyLength]))
if bodySize == 0 {
isOk = true
return nil
}
this.bodySize = int64(bodySize)
@@ -110,6 +81,10 @@ func (this *FileReader) TypeName() string {
return "disk"
}
func (this *FileReader) ExpiresAt() int64 {
return this.expiresAt
}
func (this *FileReader) Status() int {
return this.status
}
@@ -247,6 +222,37 @@ func (this *FileReader) ReadBody(buf []byte, callback ReaderFunc) error {
return nil
}
func (this *FileReader) Read(buf []byte) (n int, err error) {
var isOk = false
defer func() {
if !isOk {
_ = this.discard()
}
}()
// 直接返回从Header中剩余的
if this.bodyBufLen > 0 && len(buf) >= this.bodyBufLen {
copy(buf, this.bodyBuf)
isOk = true
n = this.bodyBufLen
if this.bodySize <= int64(this.bodyBufLen) {
err = io.EOF
return
}
this.bodyBufLen = 0
return
}
n, err = this.fp.Read(buf)
if err == nil || err == io.EOF {
isOk = true
}
return
}
func (this *FileReader) ReadBodyRange(buf []byte, start int64, end int64, callback ReaderFunc) error {
isOk := false

View File

@@ -49,6 +49,9 @@ func TestFileReader(t *testing.T) {
t.Log("body:", string(buf[:n]))
return true, nil
})
if err != nil {
t.Fatal(err)
}
}
func TestFileReader_Range(t *testing.T) {

View File

@@ -2,10 +2,13 @@ package caches
import (
"errors"
"io"
)
type MemoryReader struct {
item *MemoryItem
offset int
}
func NewMemoryReader(item *MemoryItem) *MemoryReader {
@@ -20,6 +23,10 @@ func (this *MemoryReader) TypeName() string {
return "memory"
}
func (this *MemoryReader) ExpiresAt() int64 {
return this.item.ExpiredAt
}
func (this *MemoryReader) Status() int {
return this.item.Status
}
@@ -107,6 +114,33 @@ func (this *MemoryReader) ReadBody(buf []byte, callback ReaderFunc) error {
return nil
}
func (this *MemoryReader) Read(buf []byte) (n int, err error) {
bufLen := len(buf)
if bufLen == 0 {
return 0, errors.New("using empty buffer")
}
bodySize := len(this.item.BodyValue)
left := bodySize - this.offset
if bufLen <= left {
copy(buf, this.item.BodyValue[this.offset:this.offset+bufLen])
n = bufLen
this.offset += bufLen
if this.offset >= bodySize {
err = io.EOF
return
}
return
} else {
copy(buf, this.item.BodyValue[this.offset:])
n = left
err = io.EOF
return
}
}
func (this *MemoryReader) ReadBodyRange(buf []byte, start int64, end int64, callback ReaderFunc) error {
offset := start
bodySize := int64(len(this.item.BodyValue))

View File

@@ -8,16 +8,23 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
stringutil "github.com/iwind/TeaGo/utils/string"
"golang.org/x/text/language"
"golang.org/x/text/message"
"io"
"math"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
@@ -36,6 +43,13 @@ const (
SizeMeta = SizeExpiresAt + SizeStatus + SizeURLLength + SizeHeaderLength + SizeBodyLength
)
const (
HotItemSize = 1024
)
var sharedWritingFileKeyMap = map[string]zero.Zero{} // key => bool
var sharedWritingFileKeyLocker = sync.Mutex{}
// FileStorage 文件缓存
// 文件结构:
// [expires time] | [ status ] | [url length] | [header length] | [body length] | [url] [header data] [body data]
@@ -45,16 +59,21 @@ type FileStorage struct {
memoryStorage *MemoryStorage // 一级缓存
totalSize int64
list ListInterface
writingKeyMap map[string]bool // key => bool
locker sync.RWMutex
ticker *utils.Ticker
list ListInterface
locker sync.RWMutex
purgeTicker *utils.Ticker
hotMap map[string]*HotItem // key => count
hotMapLocker sync.Mutex
lastHotSize int
hotTicker *utils.Ticker
}
func NewFileStorage(policy *serverconfigs.HTTPCachePolicy) *FileStorage {
return &FileStorage{
policy: policy,
writingKeyMap: map[string]bool{},
policy: policy,
hotMap: map[string]*HotItem{},
lastHotSize: -1,
}
}
@@ -165,12 +184,16 @@ func (this *FileStorage) Init() error {
Life: this.policy.Life,
MinLife: this.policy.MinLife,
MaxLife: this.policy.MaxLife,
MemoryAutoPurgeCount: this.policy.MemoryAutoPurgeCount,
MemoryAutoPurgeInterval: this.policy.MemoryAutoPurgeInterval,
MemoryLFUFreePercent: this.policy.MemoryLFUFreePercent,
}
err = memoryPolicy.Init()
if err != nil {
return err
}
memoryStorage := NewMemoryStorage(memoryPolicy)
memoryStorage := NewMemoryStorage(memoryPolicy, this)
err = memoryStorage.Init()
if err != nil {
return err
@@ -182,10 +205,19 @@ func (this *FileStorage) Init() error {
return nil
}
func (this *FileStorage) OpenReader(key string) (Reader, error) {
func (this *FileStorage) OpenReader(key string, useStale bool) (Reader, error) {
return this.openReader(key, true, useStale)
}
func (this *FileStorage) openReader(key string, allowMemory bool, useStale bool) (Reader, error) {
// 使用陈旧缓存的时候,我们认为是短暂的,只需要从文件里检查即可
if useStale {
allowMemory = false
}
// 先尝试内存缓存
if this.memoryStorage != nil {
reader, err := this.memoryStorage.OpenReader(key)
if allowMemory && this.memoryStorage != nil {
reader, err := this.memoryStorage.OpenReader(key, useStale)
if err == nil {
return reader, err
}
@@ -193,6 +225,17 @@ func (this *FileStorage) OpenReader(key string) (Reader, error) {
hash, path := this.keyPath(key)
// 检查文件记录是否已过期
if !useStale {
exists, err := this.list.Exist(hash)
if err != nil {
return nil, err
}
if !exists {
return nil, ErrNotFound
}
}
// TODO 尝试使用mmap加快读取速度
var isOk = false
fp, err := os.OpenFile(path, os.O_RDONLY, 0444)
@@ -209,15 +252,6 @@ func (this *FileStorage) OpenReader(key string) (Reader, error) {
}
}()
// 检查文件记录是否已过期
exists, err := this.list.Exist(hash)
if err != nil {
return nil, err
}
if !exists {
return nil, ErrNotFound
}
reader := NewFileReader(fp)
if err != nil {
return nil, err
@@ -227,6 +261,45 @@ func (this *FileStorage) OpenReader(key string) (Reader, error) {
return nil, err
}
// 增加点击量
// 1/1000采样
if allowMemory {
var rate = this.policy.PersistenceHitSampleRate
if rate <= 0 {
rate = 1000
}
if this.lastHotSize == 0 {
// 自动降低采样率来增加热点数据的缓存几率
rate = rate / 10
}
if rands.Int(0, rate) == 0 {
var hitErr = this.list.IncreaseHit(hash)
if hitErr != nil {
// 此错误可以忽略
remotelogs.Error("CACHE", "increase hit failed: "+hitErr.Error())
}
// 增加到热点
// 这里不收录缓存尺寸过大的文件
if this.memoryStorage != nil && reader.BodySize() > 0 && reader.BodySize() < 128*1024*1024 {
this.hotMapLocker.Lock()
hotItem, ok := this.hotMap[key]
if ok {
hotItem.Hits++
hotItem.ExpiresAt = reader.expiresAt
} else if len(this.hotMap) < HotItemSize { // 控制数量
this.hotMap[key] = &HotItem{
Key: key,
ExpiresAt: reader.ExpiresAt(),
Status: reader.Status(),
Hits: 1,
}
}
this.hotMapLocker.Unlock()
}
}
}
isOk = true
return reader, nil
}
@@ -242,21 +315,20 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int) (Wr
}
// 是否正在写入
var isWriting = false
this.locker.Lock()
_, ok := this.writingKeyMap[key]
this.locker.Unlock()
var isOk = false
sharedWritingFileKeyLocker.Lock()
_, ok := sharedWritingFileKeyMap[key]
if ok {
sharedWritingFileKeyLocker.Unlock()
return nil, ErrFileIsWriting
}
this.locker.Lock()
this.writingKeyMap[key] = true
this.locker.Unlock()
sharedWritingFileKeyMap[key] = zero.New()
sharedWritingFileKeyLocker.Unlock()
defer func() {
if !isWriting {
this.locker.Lock()
delete(this.writingKeyMap, key)
this.locker.Unlock()
if !isOk {
sharedWritingFileKeyLocker.Lock()
delete(sharedWritingFileKeyMap, key)
sharedWritingFileKeyLocker.Unlock()
}
}()
@@ -286,21 +358,27 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int) (Wr
}
}
// 检查缓存是否已经生成
var cachePath = dir + "/" + hash + ".cache"
stat, err := os.Stat(cachePath)
if err == nil && time.Now().Sub(stat.ModTime()) <= 1*time.Second {
// 防止并发连续写入
return nil, ErrFileIsWriting
}
var tmpPath = cachePath + ".tmp"
// 先删除
err = this.list.Remove(hash)
if err != nil {
return nil, err
}
path := dir + "/" + hash + ".cache.tmp"
writer, err := os.OpenFile(path, os.O_CREATE|os.O_SYNC|os.O_WRONLY, 0666)
writer, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_SYNC|os.O_WRONLY, 0666)
if err != nil {
return nil, err
}
isWriting = true
isOk := false
removeOnFailure := true
var removeOnFailure = true
defer func() {
if err != nil {
isOk = false
@@ -310,7 +388,7 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int) (Wr
if !isOk {
_ = writer.Close()
if removeOnFailure {
_ = os.Remove(path)
_ = os.Remove(tmpPath)
}
}
}()
@@ -381,11 +459,10 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int) (Wr
}
isOk = true
return NewFileWriter(writer, key, expiredAt, func() {
this.locker.Lock()
delete(this.writingKeyMap, key)
this.locker.Unlock()
sharedWritingFileKeyLocker.Lock()
delete(sharedWritingFileKeyMap, key)
sharedWritingFileKeyLocker.Unlock()
}), nil
}
@@ -398,7 +475,7 @@ func (this *FileStorage) AddToList(item *Item) {
}
}
item.MetaSize = SizeMeta
item.MetaSize = SizeMeta + 128
hash := stringutil.Md5(item.Key)
err := this.list.Add(hash, item)
if err != nil && !strings.Contains(err.Error(), "UNIQUE constraint failed") {
@@ -499,12 +576,12 @@ func (this *FileStorage) CleanAll() error {
}
// 重新遍历待删除
go func() {
goman.New(func() {
err = this.cleanDeletedDirs(dir)
if err != nil {
remotelogs.Warn("CACHE", "delete '*-deleted' dirs failed: "+err.Error())
}
}()
})
return nil
}
@@ -555,8 +632,11 @@ func (this *FileStorage) Stop() {
}
_ = this.list.Reset()
if this.ticker != nil {
this.ticker.Stop()
if this.purgeTicker != nil {
this.purgeTicker.Stop()
}
if this.hotTicker != nil {
this.hotTicker.Stop()
}
_ = this.list.Close()
@@ -606,32 +686,49 @@ func (this *FileStorage) initList() error {
}
// 使用异步防止阻塞主线程
go func() {
/**goman.New(func() {
dir := this.dir()
// 清除tmp
files, err := filepath.Glob(dir + "/*/*/*.cache.tmp")
if err == nil && len(files) > 0 {
for _, path := range files {
_ = os.Remove(path)
}
}
}()
// TODO 需要一个更加高效的实现
})**/
// 启动定时清理任务
this.ticker = utils.NewTicker(30 * time.Second)
var autoPurgeInterval = this.policy.PersistenceAutoPurgeInterval
if autoPurgeInterval <= 0 {
autoPurgeInterval = 30
if Tea.IsTesting() {
autoPurgeInterval = 10
}
}
this.purgeTicker = utils.NewTicker(time.Duration(autoPurgeInterval) * time.Second)
events.On(events.EventQuit, func() {
remotelogs.Println("CACHE", "quit clean timer")
var ticker = this.ticker
var ticker = this.purgeTicker
if ticker != nil {
ticker.Stop()
}
})
go func() {
for this.ticker.Next() {
this.purgeLoop()
goman.New(func() {
for this.purgeTicker.Next() {
trackers.Run("FILE_CACHE_STORAGE_PURGE_LOOP", func() {
this.purgeLoop()
})
}
}()
})
// 热点处理任务
this.hotTicker = utils.NewTicker(1 * time.Minute)
if Tea.IsTesting() {
this.hotTicker = utils.NewTicker(10 * time.Second)
}
goman.New(func() {
for this.hotTicker.Next() {
trackers.Run("FILE_CACHE_STORAGE_HOT_LOOP", func() {
this.hotLoop()
})
}
})
return nil
}
@@ -711,9 +808,9 @@ func (this *FileStorage) decodeFile(path string) (*Item, error) {
// URL
if urlSize > 0 {
data := utils.BytePool1024.Get()
data := utils.BytePool1k.Get()
result, ok, err := this.readN(fp, data, int(urlSize))
utils.BytePool1024.Put(data)
utils.BytePool1k.Put(data)
if err != nil {
return nil, err
}
@@ -730,16 +827,184 @@ func (this *FileStorage) decodeFile(path string) (*Item, error) {
// 清理任务
func (this *FileStorage) purgeLoop() {
err := this.list.Purge(1000, func(hash string) error {
path := this.hashPath(hash)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
remotelogs.Error("CACHE", "purge '"+path+"' error: "+err.Error())
// 计算是否应该开启LFU清理
var capacityBytes = this.policy.CapacityBytes()
var startLFU = false
var usedPercent = float32(this.TotalDiskSize()*100) / float32(capacityBytes)
var lfuFreePercent = this.policy.PersistenceLFUFreePercent
if lfuFreePercent <= 0 {
lfuFreePercent = 5
}
if capacityBytes > 0 {
if lfuFreePercent < 100 {
if usedPercent >= 100-lfuFreePercent {
startLFU = true
}
}
}
// 清理过期
{
var times = 1
// 空闲时间多清理
if utils.SharedFreeHoursManager.IsFreeHour() {
times = 5
}
// 处于LFU阈值时多清理
if startLFU {
times = 5
}
var purgeCount = this.policy.PersistenceAutoPurgeCount
if purgeCount <= 0 {
purgeCount = 1000
}
for i := 0; i < times; i++ {
countFound, err := this.list.Purge(purgeCount, func(hash string) error {
path := this.hashPath(hash)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
remotelogs.Error("CACHE", "purge '"+path+"' error: "+err.Error())
}
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge file storage failed: "+err.Error())
continue
}
if countFound < purgeCount {
break
}
time.Sleep(1 * time.Second)
}
}
// 磁盘空间不足时,清除老旧的缓存
if startLFU {
var total, _ = this.list.Count()
if total > 0 {
var count = types.Int(math.Ceil(float64(total) * float64(lfuFreePercent*2) / 100))
if count > 0 {
// 限制单次清理的条数,防止占用太多系统资源
if count > 2000 {
count = 2000
}
remotelogs.Println("CACHE", "LFU purge policy '"+this.policy.Name+"' id: "+types.String(this.policy.Id)+", count: "+types.String(count))
err := this.list.PurgeLFU(count, func(hash string) error {
path := this.hashPath(hash)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
remotelogs.Error("CACHE", "purge '"+path+"' error: "+err.Error())
}
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge file storage in LFU failed: "+err.Error())
}
}
}
}
}
// 热点数据任务
func (this *FileStorage) hotLoop() {
var memoryStorage = this.memoryStorage
if memoryStorage == nil {
return
}
this.hotMapLocker.Lock()
if len(this.hotMap) == 0 {
this.hotMapLocker.Unlock()
this.lastHotSize = 0
return
}
this.lastHotSize = len(this.hotMap)
var result = []*HotItem{} // [ {key: ..., hits: ...}, ... ]
for _, v := range this.hotMap {
result = append(result, v)
}
this.hotMap = map[string]*HotItem{}
this.hotMapLocker.Unlock()
// 取Top10写入内存
if len(result) > 0 {
sort.Slice(result, func(i, j int) bool {
return result[i].Hits > result[j].Hits
})
var size = 1
if len(result) < 10 {
size = 1
} else {
size = len(result) / 10
}
var buf = utils.BytePool16k.Get()
defer utils.BytePool16k.Put(buf)
for _, item := range result[:size] {
reader, err := this.openReader(item.Key, false, false)
if err != nil {
continue
}
if reader == nil {
continue
}
if reader.ExpiresAt() <= time.Now().Unix() {
continue
}
writer, err := this.memoryStorage.openWriter(item.Key, item.ExpiresAt, item.Status, false)
if err != nil {
if !CanIgnoreErr(err) {
remotelogs.Error("CACHE", "transfer hot item failed: "+err.Error())
}
_ = reader.Close()
continue
}
if writer == nil {
_ = reader.Close()
continue
}
err = reader.ReadHeader(buf, func(n int) (goNext bool, err error) {
_, err = writer.WriteHeader(buf[:n])
return
})
if err != nil {
_ = reader.Close()
_ = writer.Discard()
continue
}
err = reader.ReadBody(buf, func(n int) (goNext bool, err error) {
_, err = writer.Write(buf[:n])
return
})
if err != nil {
_ = reader.Close()
_ = writer.Discard()
continue
}
this.memoryStorage.AddToList(&Item{
Type: writer.ItemType(),
Key: item.Key,
ExpiredAt: item.ExpiresAt,
HeaderSize: writer.HeaderSize(),
BodySize: writer.BodySize(),
})
_ = reader.Close()
_ = writer.Close()
}
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge file storage failed: "+err.Error())
}
}

View File

@@ -270,7 +270,7 @@ func TestFileStorage_Read(t *testing.T) {
t.Fatal(err)
}
now := time.Now()
reader, err := storage.OpenReader("my-key")
reader, err := storage.OpenReader("my-key", false)
if err != nil {
t.Fatal(err)
}
@@ -306,7 +306,7 @@ func TestFileStorage_Read_HTTP_Response(t *testing.T) {
t.Fatal(err)
}
now := time.Now()
reader, err := storage.OpenReader("my-http-response")
reader, err := storage.OpenReader("my-http-response", false)
if err != nil {
t.Fatal(err)
}
@@ -360,7 +360,7 @@ func TestFileStorage_Read_NotFound(t *testing.T) {
}
now := time.Now()
buf := make([]byte, 6)
reader, err := storage.OpenReader("my-key-10000")
reader, err := storage.OpenReader("my-key-10000", false)
if err != nil {
if err == ErrNotFound {
t.Log("cache not fund")
@@ -506,7 +506,7 @@ func BenchmarkFileStorage_Read(b *testing.B) {
b.Fatal(err)
}
for i := 0; i < b.N; i++ {
reader, err := storage.OpenReader("my-key")
reader, err := storage.OpenReader("my-key", false)
if err != nil {
b.Fatal(err)
}
@@ -517,3 +517,16 @@ func BenchmarkFileStorage_Read(b *testing.B) {
_ = reader.Close()
}
}
func BenchmarkFileStorage_KeyPath(b *testing.B) {
runtime.GOMAXPROCS(1)
var storage = &FileStorage{
cacheConfig: &serverconfigs.HTTPFileCacheStorage{},
policy: &serverconfigs.HTTPCachePolicy{Id: 1},
}
for i := 0; i < b.N; i++ {
_, _ = storage.keyPath(strconv.Itoa(i))
}
}

View File

@@ -10,7 +10,7 @@ type StorageInterface interface {
Init() error
// OpenReader 读取缓存
OpenReader(key string) (Reader, error)
OpenReader(key string, useStale bool) (reader Reader, err error)
// OpenWriter 打开缓存写入器等待写入
OpenWriter(key string, expiredAt int64, status int) (Writer, error)

View File

@@ -3,9 +3,16 @@ package caches
import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/cespare/xxhash"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
"math"
"runtime"
"strconv"
"sync"
"sync/atomic"
@@ -26,23 +33,38 @@ func (this *MemoryItem) IsExpired() bool {
}
type MemoryStorage struct {
policy *serverconfigs.HTTPCachePolicy
list ListInterface
locker *sync.RWMutex
valuesMap map[uint64]*MemoryItem
ticker *utils.Ticker
purgeDuration time.Duration
parentStorage StorageInterface
policy *serverconfigs.HTTPCachePolicy
list ListInterface
locker *sync.RWMutex
valuesMap map[uint64]*MemoryItem // hash => item
dirtyChan chan string // hash chan
purgeTicker *utils.Ticker
totalSize int64
writingKeyMap map[string]bool // key => bool
writingKeyMap map[string]zero.Zero // key => bool
}
func NewMemoryStorage(policy *serverconfigs.HTTPCachePolicy) *MemoryStorage {
func NewMemoryStorage(policy *serverconfigs.HTTPCachePolicy, parentStorage StorageInterface) *MemoryStorage {
var dirtyChan chan string
if parentStorage != nil {
var queueSize = policy.MemoryAutoFlushQueueSize
if queueSize <= 0 {
queueSize = 2048
}
dirtyChan = make(chan string, queueSize)
}
return &MemoryStorage{
parentStorage: parentStorage,
policy: policy,
list: NewMemoryList(),
locker: &sync.RWMutex{},
valuesMap: map[uint64]*MemoryItem{},
writingKeyMap: map[string]bool{},
dirtyChan: dirtyChan,
writingKeyMap: map[string]zero.Zero{},
}
}
@@ -57,23 +79,33 @@ func (this *MemoryStorage) Init() error {
atomic.AddInt64(&this.totalSize, -item.TotalSize())
})
if this.purgeDuration <= 0 {
this.purgeDuration = 10 * time.Second
var autoPurgeInterval = this.policy.MemoryAutoPurgeInterval
if autoPurgeInterval <= 0 {
autoPurgeInterval = 5
}
// 启动定时清理任务
this.ticker = utils.NewTicker(this.purgeDuration)
go func() {
for this.ticker.Next() {
this.purgeTicker = utils.NewTicker(time.Duration(autoPurgeInterval) * time.Second)
goman.New(func() {
for this.purgeTicker.Next() {
var tr = trackers.Begin("MEMORY_CACHE_STORAGE_PURGE_LOOP")
this.purgeLoop()
tr.End()
}
}()
})
// 启动定时Flush memory to disk任务
goman.New(func() {
for hash := range this.dirtyChan {
this.flushItem(hash)
}
})
return nil
}
// OpenReader 读取缓存
func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
func (this *MemoryStorage) OpenReader(key string, useStale bool) (Reader, error) {
hash := this.hash(key)
this.locker.RLock()
@@ -83,7 +115,7 @@ func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
return nil, ErrNotFound
}
if item.ExpiredAt > utils.UnixTime() {
if useStale || (item.ExpiredAt > utils.UnixTime()) {
reader := NewMemoryReader(item)
err := reader.Init()
if err != nil {
@@ -91,6 +123,18 @@ func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
return nil, err
}
this.locker.RUnlock()
// 增加点击量
// 1/1000采样
// TODO 考虑是否在缓存策略里设置
if rands.Int(0, 1000) == 0 {
var hitErr = this.list.IncreaseHit(types.String(hash))
if hitErr != nil {
// 此错误可以忽略
remotelogs.Error("CACHE", "increase hit failed: "+hitErr.Error())
}
}
return reader, nil
}
this.locker.RUnlock()
@@ -102,6 +146,10 @@ func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
// OpenWriter 打开缓存写入器等待写入
func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int) (Writer, error) {
return this.openWriter(key, expiredAt, status, true)
}
func (this *MemoryStorage) openWriter(key string, expiredAt int64, status int, isDirty bool) (Writer, error) {
this.locker.Lock()
defer this.locker.Unlock()
@@ -111,7 +159,7 @@ func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int) (
if ok {
return nil, ErrFileIsWriting
}
this.writingKeyMap[key] = true
this.writingKeyMap[key] = zero.New()
defer func() {
if !isWriting {
delete(this.writingKeyMap, key)
@@ -145,7 +193,7 @@ func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int) (
}
isWriting = true
return NewMemoryWriter(this.valuesMap, key, expiredAt, status, this.locker, func() {
return NewMemoryWriter(this, key, expiredAt, status, isDirty, func() {
this.locker.Lock()
delete(this.writingKeyMap, key)
this.locker.Unlock()
@@ -208,16 +256,23 @@ func (this *MemoryStorage) Stop() {
this.locker.Lock()
this.valuesMap = map[uint64]*MemoryItem{}
this.writingKeyMap = map[string]bool{}
this.writingKeyMap = map[string]zero.Zero{}
_ = this.list.Reset()
if this.ticker != nil {
this.ticker.Stop()
if this.purgeTicker != nil {
this.purgeTicker.Stop()
}
if this.parentStorage != nil && this.dirtyChan != nil {
close(this.dirtyChan)
}
_ = this.list.Close()
this.locker.Unlock()
// 回收内存
runtime.GC()
remotelogs.Println("CACHE", "close memory storage '"+strconv.FormatInt(this.policy.Id, 10)+"'")
}
@@ -228,7 +283,7 @@ func (this *MemoryStorage) Policy() *serverconfigs.HTTPCachePolicy {
// AddToList 将缓存添加到列表
func (this *MemoryStorage) AddToList(item *Item) {
item.MetaSize = int64(len(item.Key)) + 32 /** 32是我们评估的数据结构的长度 **/
item.MetaSize = int64(len(item.Key)) + 128 /** 128是我们评估的数据结构的长度 **/
hash := fmt.Sprintf("%d", this.hash(item.Key))
_ = this.list.Add(hash, item)
}
@@ -250,7 +305,28 @@ func (this *MemoryStorage) hash(key string) uint64 {
// 清理任务
func (this *MemoryStorage) purgeLoop() {
_ = this.list.Purge(2048, func(hash string) error {
// 计算是否应该开启LFU清理
var capacityBytes = this.policy.CapacityBytes()
var startLFU = false
var usedPercent = float32(this.TotalMemorySize()*100) / float32(capacityBytes)
var lfuFreePercent = this.policy.MemoryLFUFreePercent
if lfuFreePercent <= 0 {
lfuFreePercent = 5
}
if capacityBytes > 0 {
if lfuFreePercent < 100 {
if usedPercent >= 100-lfuFreePercent {
startLFU = true
}
}
}
// 清理过期
var purgeCount = this.policy.MemoryAutoPurgeCount
if purgeCount <= 0 {
purgeCount = 2000
}
_, _ = this.list.Purge(purgeCount, func(hash string) error {
uintHash, err := strconv.ParseUint(hash, 10, 64)
if err == nil {
this.locker.Lock()
@@ -259,6 +335,92 @@ func (this *MemoryStorage) purgeLoop() {
}
return nil
})
// LFU
if startLFU {
var total, _ = this.list.Count()
if total > 0 {
var count = types.Int(math.Ceil(float64(total) * float64(lfuFreePercent*2) / 100))
if count > 0 {
// 限制单次清理的条数,防止占用太多系统资源
if count > 2000 {
count = 2000
}
// 这里不提示LFU因为此事件将会非常频繁
err := this.list.PurgeLFU(count, func(hash string) error {
uintHash, err := strconv.ParseUint(hash, 10, 64)
if err == nil {
this.locker.Lock()
delete(this.valuesMap, uintHash)
this.locker.Unlock()
}
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge memory storage in LFU failed: "+err.Error())
}
}
}
}
}
// Flush任务
func (this *MemoryStorage) flushItem(key string) {
if this.parentStorage == nil {
return
}
var hash = this.hash(key)
this.locker.RLock()
item, ok := this.valuesMap[hash]
this.locker.RUnlock()
if !ok {
return
}
if !item.IsDone || item.IsExpired() {
return
}
writer, err := this.parentStorage.OpenWriter(key, item.ExpiredAt, item.Status)
if err != nil {
if !CanIgnoreErr(err) {
remotelogs.Error("CACHE", "flush items failed: open writer failed: "+err.Error())
}
return
}
_, err = writer.WriteHeader(item.HeaderValue)
if err != nil {
_ = writer.Discard()
remotelogs.Error("CACHE", "flush items failed: write header failed: "+err.Error())
return
}
_, err = writer.Write(item.BodyValue)
if err != nil {
_ = writer.Discard()
remotelogs.Error("CACHE", "flush items failed: writer body failed: "+err.Error())
return
}
err = writer.Close()
if err != nil {
_ = writer.Discard()
remotelogs.Error("CACHE", "flush items failed: close writer failed: "+err.Error())
}
this.parentStorage.AddToList(&Item{
Type: writer.ItemType(),
Key: key,
ExpiredAt: item.ExpiredAt,
HeaderSize: writer.HeaderSize(),
BodySize: writer.BodySize(),
})
return
}
func (this *MemoryStorage) memoryCapacityBytes() int64 {

View File

@@ -1,16 +1,19 @@
package caches
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/rands"
"runtime"
"runtime/debug"
"strconv"
"testing"
"time"
)
func TestMemoryStorage_OpenWriter(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200)
if err != nil {
@@ -22,7 +25,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
t.Log(storage.valuesMap)
{
reader, err := storage.OpenReader("abc")
reader, err := storage.OpenReader("abc", false)
if err != nil {
if err == ErrNotFound {
t.Log("not found: abc")
@@ -49,7 +52,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
}
{
_, err := storage.OpenReader("abc 2")
_, err := storage.OpenReader("abc 2", false)
if err != nil {
if err == ErrNotFound {
t.Log("not found: abc2")
@@ -65,7 +68,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
}
_, _ = writer.Write([]byte("Hello123"))
{
reader, err := storage.OpenReader("abc")
reader, err := storage.OpenReader("abc", false)
if err != nil {
if err == ErrNotFound {
t.Log("not found: abc")
@@ -85,7 +88,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
}
func TestMemoryStorage_OpenReaderLock(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
_ = storage.Init()
var h = storage.hash("test")
@@ -94,11 +97,11 @@ func TestMemoryStorage_OpenReaderLock(t *testing.T) {
IsDone: true,
},
}
_, _ = storage.OpenReader("test")
_, _ = storage.OpenReader("test", false)
}
func TestMemoryStorage_Delete(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
{
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200)
if err != nil {
@@ -120,7 +123,7 @@ func TestMemoryStorage_Delete(t *testing.T) {
}
func TestMemoryStorage_Stat(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200)
@@ -157,7 +160,7 @@ func TestMemoryStorage_Stat(t *testing.T) {
}
func TestMemoryStorage_CleanAll(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200)
@@ -192,7 +195,7 @@ func TestMemoryStorage_CleanAll(t *testing.T) {
}
func TestMemoryStorage_Purge(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200)
@@ -227,8 +230,9 @@ func TestMemoryStorage_Purge(t *testing.T) {
}
func TestMemoryStorage_Expire(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage.purgeDuration = 5 * time.Second
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{
MemoryAutoPurgeInterval: 5,
}, nil)
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -252,7 +256,7 @@ func TestMemoryStorage_Expire(t *testing.T) {
}
func TestMemoryStorage_Locker(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -265,3 +269,38 @@ func TestMemoryStorage_Locker(t *testing.T) {
}
t.Log("ok")
}
func TestMemoryStorage_Stop(t *testing.T) {
var stat1 = &runtime.MemStats{}
runtime.ReadMemStats(stat1)
var m = map[uint64]*MemoryItem{}
for i := 0; i < 1_000_000; i++ {
m[uint64(i)] = &MemoryItem{
HeaderValue: []byte("Hello, World"),
BodyValue: bytes.Repeat([]byte("Hello"), 1024),
}
}
m = map[uint64]*MemoryItem{}
var before = time.Now()
//runtime.GC()
debug.FreeOSMemory()
/**go func() {
time.Sleep(10 * time.Second)
runtime.GC()
}()**/
t.Log(time.Since(before).Seconds()*1000, "ms")
var stat2 = &runtime.MemStats{}
runtime.ReadMemStats(stat2)
if stat2.HeapInuse > stat1.HeapInuse {
t.Log(stat2.HeapInuse, stat1.HeapInuse, (stat2.HeapInuse-stat1.HeapInuse)/1024/1024, "MB")
} else {
t.Log("0 MB")
}
t.Log(len(m))
}

View File

@@ -0,0 +1,76 @@
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/compressions"
)
type compressionWriter struct {
rawWriter Writer
writer compressions.Writer
key string
expiredAt int64
}
func NewCompressionWriter(gw Writer, cpWriter compressions.Writer, key string, expiredAt int64) Writer {
return &compressionWriter{
rawWriter: gw,
writer: cpWriter,
key: key,
expiredAt: expiredAt,
}
}
func (this *compressionWriter) WriteHeader(data []byte) (n int, err error) {
return this.writer.Write(data)
}
// WriteHeaderLength 写入Header长度数据
func (this *compressionWriter) WriteHeaderLength(headerLength int) error {
return nil
}
// WriteBodyLength 写入Body长度数据
func (this *compressionWriter) WriteBodyLength(bodyLength int64) error {
return nil
}
func (this *compressionWriter) Write(data []byte) (n int, err error) {
return this.writer.Write(data)
}
func (this *compressionWriter) Close() error {
err := this.writer.Close()
if err != nil {
return err
}
return this.rawWriter.Close()
}
func (this *compressionWriter) Discard() error {
err := this.writer.Close()
if err != nil {
return err
}
return this.rawWriter.Discard()
}
func (this *compressionWriter) Key() string {
return this.key
}
func (this *compressionWriter) ExpiredAt() int64 {
return this.expiredAt
}
func (this *compressionWriter) HeaderSize() int64 {
return this.rawWriter.HeaderSize()
}
func (this *compressionWriter) BodySize() int64 {
return this.rawWriter.BodySize()
}
// ItemType 内容类型
func (this *compressionWriter) ItemType() ItemType {
return this.rawWriter.ItemType()
}

View File

@@ -6,6 +6,7 @@ import (
"io"
"os"
"strings"
"sync"
)
type FileWriter struct {
@@ -15,6 +16,7 @@ type FileWriter struct {
bodySize int64
expiredAt int64
endFunc func()
once sync.Once
}
func NewFileWriter(rawWriter *os.File, key string, expiredAt int64, endFunc func()) *FileWriter {
@@ -82,18 +84,25 @@ func (this *FileWriter) WriteBodyLength(bodyLength int64) error {
// Close 关闭
func (this *FileWriter) Close() error {
defer this.endFunc()
defer this.once.Do(func() {
this.endFunc()
})
path := this.rawWriter.Name()
err := this.WriteHeaderLength(types.Int(this.headerSize))
if err != nil {
_ = this.rawWriter.Close()
_ = os.Remove(path)
return err
}
err = this.WriteBodyLength(this.bodySize)
if err != nil {
_ = this.rawWriter.Close()
_ = os.Remove(path)
return err
}
path := this.rawWriter.Name()
err = this.rawWriter.Close()
if err != nil {
_ = os.Remove(path)
@@ -109,7 +118,9 @@ func (this *FileWriter) Close() error {
// Discard 丢弃
func (this *FileWriter) Discard() error {
defer this.endFunc()
defer this.once.Do(func() {
this.endFunc()
})
_ = this.rawWriter.Close()

View File

@@ -1,76 +0,0 @@
package caches
import (
"compress/gzip"
)
type gzipWriter struct {
rawWriter Writer
writer *gzip.Writer
key string
expiredAt int64
}
func NewGzipWriter(gw Writer, key string, expiredAt int64) Writer {
return &gzipWriter{
rawWriter: gw,
writer: gzip.NewWriter(gw),
key: key,
expiredAt: expiredAt,
}
}
func (this *gzipWriter) WriteHeader(data []byte) (n int, err error) {
return this.writer.Write(data)
}
// 写入Header长度数据
func (this *gzipWriter) WriteHeaderLength(headerLength int) error {
return nil
}
// 写入Body长度数据
func (this *gzipWriter) WriteBodyLength(bodyLength int64) error {
return nil
}
func (this *gzipWriter) Write(data []byte) (n int, err error) {
return this.writer.Write(data)
}
func (this *gzipWriter) Close() error {
err := this.writer.Close()
if err != nil {
return err
}
return this.rawWriter.Close()
}
func (this *gzipWriter) Discard() error {
err := this.writer.Close()
if err != nil {
return err
}
return this.rawWriter.Discard()
}
func (this *gzipWriter) Key() string {
return this.key
}
func (this *gzipWriter) ExpiredAt() int64 {
return this.expiredAt
}
func (this *gzipWriter) HeaderSize() int64 {
return this.rawWriter.HeaderSize()
}
func (this *gzipWriter) BodySize() int64 {
return this.rawWriter.BodySize()
}
// 内容类型
func (this *gzipWriter) ItemType() ItemType {
return this.rawWriter.ItemType()
}

View File

@@ -7,31 +7,33 @@ import (
)
type MemoryWriter struct {
storage *MemoryStorage
key string
expiredAt int64
m map[uint64]*MemoryItem
locker *sync.RWMutex
headerSize int64
bodySize int64
status int
isDirty bool
hash uint64
item *MemoryItem
endFunc func()
once sync.Once
}
func NewMemoryWriter(m map[uint64]*MemoryItem, key string, expiredAt int64, status int, locker *sync.RWMutex, endFunc func()) *MemoryWriter {
func NewMemoryWriter(memoryStorage *MemoryStorage, key string, expiredAt int64, status int, isDirty bool, endFunc func()) *MemoryWriter {
w := &MemoryWriter{
m: m,
storage: memoryStorage,
key: key,
expiredAt: expiredAt,
locker: locker,
item: &MemoryItem{
ExpiredAt: expiredAt,
ModifiedAt: time.Now().Unix(),
Status: status,
},
status: status,
isDirty: isDirty,
endFunc: endFunc,
}
w.hash = w.calculateHash(key)
@@ -66,16 +68,27 @@ func (this *MemoryWriter) BodySize() int64 {
// Close 关闭
func (this *MemoryWriter) Close() error {
// 需要在Locker之外
defer this.endFunc()
defer this.once.Do(func() {
this.endFunc()
})
if this.item == nil {
return nil
}
this.locker.Lock()
this.storage.locker.Lock()
this.item.IsDone = true
this.m[this.hash] = this.item
this.locker.Unlock()
this.storage.valuesMap[this.hash] = this.item
if this.isDirty {
if this.storage.parentStorage != nil {
select {
case this.storage.dirtyChan <- this.key:
default:
}
}
}
this.storage.locker.Unlock()
return nil
}
@@ -83,11 +96,13 @@ func (this *MemoryWriter) Close() error {
// Discard 丢弃
func (this *MemoryWriter) Discard() error {
// 需要在Locker之外
defer this.endFunc()
defer this.once.Do(func() {
this.endFunc()
})
this.locker.Lock()
delete(this.m, this.hash)
this.locker.Unlock()
this.storage.locker.Lock()
delete(this.storage.valuesMap, this.hash)
this.storage.locker.Unlock()
return nil
}

View File

@@ -0,0 +1,8 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
type Reader interface {
Read(p []byte) (n int, err error)
Close() error
}

View File

@@ -0,0 +1,29 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"github.com/andybalholm/brotli"
"io"
"strings"
)
type BrotliReader struct {
reader *brotli.Reader
}
func NewBrotliReader(reader io.Reader) (Reader, error) {
return &BrotliReader{reader: brotli.NewReader(reader)}, nil
}
func (this *BrotliReader) Read(p []byte) (n int, err error) {
n, err = this.reader.Read(p)
if err != nil && strings.Contains(err.Error(), "excessive") {
err = io.EOF
}
return
}
func (this *BrotliReader) Close() error {
return nil
}

View File

@@ -0,0 +1,24 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"compress/flate"
"io"
)
type DeflateReader struct {
reader io.ReadCloser
}
func NewDeflateReader(reader io.Reader) (Reader, error) {
return &DeflateReader{reader: flate.NewReader(reader)}, nil
}
func (this *DeflateReader) Read(p []byte) (n int, err error) {
return this.reader.Read(p)
}
func (this *DeflateReader) Close() error {
return this.reader.Close()
}

View File

@@ -0,0 +1,30 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"compress/gzip"
"io"
)
type GzipReader struct {
reader *gzip.Reader
}
func NewGzipReader(reader io.Reader) (Reader, error) {
r, err := gzip.NewReader(reader)
if err != nil {
return nil, err
}
return &GzipReader{
reader: r,
}, nil
}
func (this *GzipReader) Read(p []byte) (n int, err error) {
return this.reader.Read(p)
}
func (this *GzipReader) Close() error {
return this.reader.Close()
}

View File

@@ -0,0 +1,46 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"io"
)
type ContentEncoding = string
const (
ContentEncodingBr ContentEncoding = "br"
ContentEncodingGzip ContentEncoding = "gzip"
ContentEncodingDeflate ContentEncoding = "deflate"
)
var ErrNotSupportedContentEncoding = errors.New("not supported content encoding")
// NewReader 获取Reader
func NewReader(reader io.Reader, contentEncoding ContentEncoding) (Reader, error) {
switch contentEncoding {
case ContentEncodingBr:
return NewBrotliReader(reader)
case ContentEncodingGzip:
return NewGzipReader(reader)
case ContentEncodingDeflate:
return NewDeflateReader(reader)
}
return nil, ErrNotSupportedContentEncoding
}
// NewWriter 获取Writer
// TODO 考虑重用Writer
func NewWriter(writer io.Writer, compressType serverconfigs.HTTPCompressionType, level int) (Writer, error) {
switch compressType {
case serverconfigs.HTTPCompressionTypeGzip:
return NewGzipWriter(writer, level)
case serverconfigs.HTTPCompressionTypeDeflate:
return NewDeflateWriter(writer, level)
case serverconfigs.HTTPCompressionTypeBrotli:
return NewBrotliWriter(writer, level)
}
return nil, errors.New("invalid compression type '" + compressType + "'")
}

View File

@@ -0,0 +1,10 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
type Writer interface {
Write(p []byte) (int, error)
Flush() error
Close() error
Level() int
}

View File

@@ -0,0 +1,41 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"github.com/andybalholm/brotli"
"io"
)
type BrotliWriter struct {
writer *brotli.Writer
level int
}
func NewBrotliWriter(writer io.Writer, level int) (Writer, error) {
if level <= 0 {
level = brotli.BestSpeed
} else if level > brotli.BestCompression {
level = brotli.BestCompression
}
return &BrotliWriter{
writer: brotli.NewWriterLevel(writer, level),
level: level,
}, nil
}
func (this *BrotliWriter) Write(p []byte) (int, error) {
return this.writer.Write(p)
}
func (this *BrotliWriter) Flush() error {
return this.writer.Flush()
}
func (this *BrotliWriter) Close() error {
return this.writer.Close()
}
func (this *BrotliWriter) Level() int {
return this.level
}

View File

@@ -0,0 +1,47 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"compress/flate"
"io"
)
type DeflateWriter struct {
writer *flate.Writer
level int
}
func NewDeflateWriter(writer io.Writer, level int) (Writer, error) {
if level <= 0 {
level = flate.BestSpeed
} else if level > flate.BestCompression {
level = flate.BestCompression
}
flateWriter, err := flate.NewWriter(writer, level)
if err != nil {
return nil, err
}
return &DeflateWriter{
writer: flateWriter,
level: level,
}, nil
}
func (this *DeflateWriter) Write(p []byte) (int, error) {
return this.writer.Write(p)
}
func (this *DeflateWriter) Flush() error {
return this.writer.Flush()
}
func (this *DeflateWriter) Close() error {
return this.writer.Close()
}
func (this *DeflateWriter) Level() int {
return this.level
}

View File

@@ -0,0 +1,51 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"bytes"
"io"
)
type EncodingWriter struct {
contentEncoding ContentEncoding
writer Writer
buf *bytes.Buffer
}
func NewEncodingWriter(contentEncoding ContentEncoding, writer Writer) (Writer, error) {
return &EncodingWriter{
contentEncoding: contentEncoding,
writer: writer,
buf: &bytes.Buffer{},
}, nil
}
func (this *EncodingWriter) Write(p []byte) (int, error) {
return this.buf.Write(p)
}
func (this *EncodingWriter) Flush() error {
return this.writer.Flush()
}
func (this *EncodingWriter) Close() error {
reader, err := NewReader(this.buf, this.contentEncoding)
if err != nil {
_ = this.writer.Close()
return err
}
_, err = io.Copy(this.writer, reader)
if err != nil {
_ = reader.Close()
_ = this.writer.Close()
return err
}
_ = reader.Close()
return this.writer.Close()
}
func (this *EncodingWriter) Level() int {
return this.writer.Level()
}

View File

@@ -0,0 +1,47 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"testing"
)
func TestNewEncodingWriter(t *testing.T) {
var buf = &bytes.Buffer{}
subWriter, err := NewWriter(buf, serverconfigs.HTTPCompressionTypeGzip, 5)
if err != nil {
t.Fatal(err)
}
writer, err := NewEncodingWriter(ContentEncodingGzip, subWriter)
if err != nil {
t.Fatal(err)
}
gzipBuf := &bytes.Buffer{}
gzipWriter, err := NewGzipWriter(gzipBuf, 5)
if err != nil {
t.Fatal(err)
}
_, err = gzipWriter.Write([]byte("Hello"))
if err != nil {
t.Fatal(err)
}
_, err = gzipWriter.Write([]byte("World"))
if err != nil {
t.Fatal(err)
}
_ = gzipWriter.Close()
_, err = writer.Write(gzipBuf.Bytes())
if err != nil {
t.Fatal(err)
}
_ = writer.Close()
t.Log(buf.String())
}

View File

@@ -0,0 +1,47 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"compress/gzip"
"io"
)
type GzipWriter struct {
writer *gzip.Writer
level int
}
func NewGzipWriter(writer io.Writer, level int) (Writer, error) {
if level <= 0 {
level = gzip.BestSpeed
} else if level > gzip.BestCompression {
level = gzip.BestCompression
}
gzipWriter, err := gzip.NewWriterLevel(writer, level)
if err != nil {
return nil, err
}
return &GzipWriter{
writer: gzipWriter,
level: level,
}, nil
}
func (this *GzipWriter) Write(p []byte) (int, error) {
return this.writer.Write(p)
}
func (this *GzipWriter) Flush() error {
return this.writer.Flush()
}
func (this *GzipWriter) Close() error {
return this.writer.Close()
}
func (this *GzipWriter) Level() int {
return this.level
}

View File

@@ -1,7 +1,7 @@
package teaconst
const (
Version = "0.2.6"
Version = "0.4.0"
ProductName = "Edge Node"
ProcessName = "edge-node"

13
internal/const/vars.go Normal file
View File

@@ -0,0 +1,13 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package teaconst
var (
// 流量统计
InTrafficBytes = uint64(0)
OutTrafficBytes = uint64(0)
NodeId int64 = 0
NodeIdString = ""
)

View File

@@ -0,0 +1,12 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package goman
import "time"
type Instance struct {
Id uint64
CreatedTime time.Time
File string
Line int
}

81
internal/goman/lib.go Normal file
View File

@@ -0,0 +1,81 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package goman
import (
"runtime"
"sync"
"time"
)
var locker = &sync.Mutex{}
var instanceMap = map[uint64]*Instance{} // id => *Instance
var instanceId = uint64(0)
// New 新创建goroutine
func New(f func()) {
_, file, line, _ := runtime.Caller(1)
go func() {
locker.Lock()
instanceId++
var instance = &Instance{
Id: instanceId,
CreatedTime: time.Now(),
}
instance.File = file
instance.Line = line
instanceMap[instanceId] = instance
locker.Unlock()
// run function
f()
locker.Lock()
delete(instanceMap, instanceId)
locker.Unlock()
}()
}
// NewWithArgs 创建带有参数的goroutine
func NewWithArgs(f func(args ...interface{}), args ...interface{}) {
_, file, line, _ := runtime.Caller(1)
go func() {
locker.Lock()
instanceId++
var instance = &Instance{
Id: instanceId,
CreatedTime: time.Now(),
}
instance.File = file
instance.Line = line
instanceMap[instanceId] = instance
locker.Unlock()
// run function
f(args...)
locker.Lock()
delete(instanceMap, instanceId)
locker.Unlock()
}()
}
// List 列出所有正在运行goroutine
func List() []*Instance {
locker.Lock()
defer locker.Unlock()
var result = []*Instance{}
for _, instance := range instanceMap {
result = append(result, instance)
}
return result
}

View File

@@ -0,0 +1,28 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package goman
import (
"testing"
"time"
)
func TestNew(t *testing.T) {
New(func() {
t.Log("Hello")
t.Log(List())
})
time.Sleep(1 * time.Second)
t.Log(List())
time.Sleep(1 * time.Second)
}
func TestNewWithArgs(t *testing.T) {
NewWithArgs(func(args ...interface{}) {
t.Log(args[0], args[1])
}, 1, 2)
time.Sleep(1 * time.Second)
}

View File

@@ -0,0 +1,5 @@
# IPList
List Check Order:
~~~
Global List --> Node List--> Server List --> WAF List --> Bind List
~~~

View File

@@ -13,7 +13,7 @@ func (this *BaseAction) Close() error {
return nil
}
// 处理HTTP请求
// DoHTTP 处理HTTP请求
func (this *BaseAction) DoHTTP(req *http.Request, resp http.ResponseWriter) (goNext bool, err error) {
return true, nil
}

View File

@@ -1,6 +1,6 @@
package iplibrary
// 是否是致命错误
// FataError 是否是致命错误
type FataError struct {
err string
}

View File

@@ -11,7 +11,7 @@ import (
"time"
)
// Firewalld动作管理
// FirewalldAction Firewalld动作管理
// 常用命令:
// - 查询列表: firewall-cmd --list-all
// - 添加IPfirewall-cmd --add-rich-rule="rule family='ipv4' source address='192.168.2.32' reject" --timeout=30s
@@ -20,6 +20,8 @@ type FirewalldAction struct {
BaseAction
config *firewallconfigs.FirewallActionFirewalldConfig
firewalldNotFound bool
}
func NewFirewalldAction() *FirewalldAction {
@@ -82,6 +84,10 @@ func (this *FirewalldAction) runActionSingleIP(action string, listType IPListTyp
if len(path) == 0 {
path, err = exec.LookPath("firewall-cmd")
if err != nil {
if this.firewalldNotFound {
return nil
}
this.firewalldNotFound = true
return err
}
}
@@ -126,10 +132,12 @@ func (this *FirewalldAction) runActionSingleIP(action string, listType IPListTyp
}
args := []string{opt}
if item.ExpiredAt > timestamp {
args = append(args, "--timeout="+fmt.Sprintf("%d", item.ExpiredAt-timestamp)+"s")
} else {
// TODO 思考是否需要permanent不然--reload之后会丢失
if action == "addItem" {
if item.ExpiredAt > timestamp {
args = append(args, "--timeout="+fmt.Sprintf("%d", item.ExpiredAt-timestamp)+"s")
} else {
// TODO 思考是否需要permanent不然--reload之后会丢失
}
}
if runtime.GOOS == "darwin" {

View File

@@ -6,19 +6,19 @@ import (
"net/http"
)
// HTML动作
// HTMLAction HTML动作
type HTMLAction struct {
BaseAction
config *firewallconfigs.FirewallActionHTMLConfig
}
// 获取新对象
// NewHTMLAction 获取新对象
func NewHTMLAction() *HTMLAction {
return &HTMLAction{}
}
// 初始化
// Init 初始化
func (this *HTMLAction) Init(config *firewallconfigs.FirewallActionConfig) error {
this.config = &firewallconfigs.FirewallActionHTMLConfig{}
err := this.convertParams(config.Params, this.config)
@@ -28,22 +28,22 @@ func (this *HTMLAction) Init(config *firewallconfigs.FirewallActionConfig) error
return nil
}
// 添加
// AddItem 添加
func (this *HTMLAction) AddItem(listType IPListType, item *pb.IPItem) error {
return nil
}
// 删除
// DeleteItem 删除
func (this *HTMLAction) DeleteItem(listType IPListType, item *pb.IPItem) error {
return nil
}
// 关闭
// Close 关闭
func (this *HTMLAction) Close() error {
return nil
}
// 处理HTTP请求
// DoHTTP 处理HTTP请求
func (this *HTMLAction) DoHTTP(req *http.Request, resp http.ResponseWriter) (goNext bool, err error) {
if this.config == nil {
goNext = true

View File

@@ -7,18 +7,18 @@ import (
)
type ActionInterface interface {
// 初始化
// Init 初始化
Init(config *firewallconfigs.FirewallActionConfig) error
// 添加
// AddItem 添加
AddItem(listType IPListType, item *pb.IPItem) error
// 删除
// DeleteItem 删除
DeleteItem(listType IPListType, item *pb.IPItem) error
// 关闭
// Close 关闭
Close() error
// 处理HTTP请求
// DoHTTP 处理HTTP请求
DoHTTP(req *http.Request, resp http.ResponseWriter) (goNext bool, err error)
}

View File

@@ -5,13 +5,15 @@ import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/firewallconfigs"
"github.com/iwind/TeaGo/types"
"os/exec"
"runtime"
"strconv"
"strings"
"time"
)
// IPSet动作
// IPSetAction IPSet动作
// 相关命令:
// - 利用Firewalld管理set
// - 添加firewall-cmd --permanent --new-ipset=edge_ip_list --type=hash:ip --option="timeout=0"
@@ -23,14 +25,21 @@ import (
// - 添加Itemipset add edge_ip_list 192.168.2.32 timeout 30
// - 删除Item: ipset del edge_ip_list 192.168.2.32
// - 创建setipset create edge_ip_list hash:ip timeout 0
// - 查看统计ipset -t list edge_black_list
// - 删除setipset destroy edge_black_list
type IPSetAction struct {
BaseAction
config *firewallconfigs.FirewallActionIPSetConfig
config *firewallconfigs.FirewallActionIPSetConfig
errorBuf *bytes.Buffer
ipsetNotfound bool
}
func NewIPSetAction() *IPSetAction {
return &IPSetAction{}
return &IPSetAction{
errorBuf: &bytes.Buffer{},
}
}
func (this *IPSetAction) Init(config *firewallconfigs.FirewallActionConfig) error {
@@ -54,7 +63,7 @@ func (this *IPSetAction) Init(config *firewallconfigs.FirewallActionConfig) erro
return err
}
{
cmd := exec.Command(path, "create", this.config.WhiteName, "hash:ip", "timeout", "0")
cmd := exec.Command(path, "create", this.config.WhiteName, "hash:ip", "timeout", "0", "maxelem", "1000000")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
err := cmd.Run()
@@ -68,7 +77,7 @@ func (this *IPSetAction) Init(config *firewallconfigs.FirewallActionConfig) erro
}
}
{
cmd := exec.Command(path, "create", this.config.BlackName, "hash:ip", "timeout", "0")
cmd := exec.Command(path, "create", this.config.BlackName, "hash:ip", "timeout", "0", "maxelem", "1000000")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
err := cmd.Run()
@@ -163,24 +172,39 @@ func (this *IPSetAction) Init(config *firewallconfigs.FirewallActionConfig) erro
}
{
cmd := exec.Command(path, "-A", "INPUT", "-m", "set", "--match-set", this.config.WhiteName, "src", "-j", "ACCEPT")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
// 检查规则是否存在
var cmd = exec.Command(path, "-C", "INPUT", "-m", "set", "--match-set", this.config.WhiteName, "src", "-j", "ACCEPT")
err := cmd.Run()
if err != nil {
output := stderr.Bytes()
return errors.New("iptables add rule: " + err.Error() + ", output: " + string(output))
var exists = err == nil
// 添加规则
if !exists {
var cmd = exec.Command(path, "-A", "INPUT", "-m", "set", "--match-set", this.config.WhiteName, "src", "-j", "ACCEPT")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
output := stderr.Bytes()
return errors.New("iptables add rule: " + err.Error() + ", output: " + string(output))
}
}
}
{
cmd := exec.Command(path, "-A", "INPUT", "-m", "set", "--match-set", this.config.BlackName, "src", "-j", "REJECT")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
// 检查规则是否存在
var cmd = exec.Command(path, "-C", "INPUT", "-m", "set", "--match-set", this.config.BlackName, "src", "-j", "REJECT")
err := cmd.Run()
if err != nil {
output := stderr.Bytes()
return errors.New("iptables add rule: " + err.Error() + ", output: " + string(output))
var exists = err == nil
if !exists {
var cmd = exec.Command(path, "-A", "INPUT", "-m", "set", "--match-set", this.config.BlackName, "src", "-j", "REJECT")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
output := stderr.Bytes()
return errors.New("iptables add rule: " + err.Error() + ", output: " + string(output))
}
}
}
}
@@ -212,6 +236,16 @@ func (this *IPSetAction) runAction(action string, listType IPListType, item *pb.
return nil
}
for _, cidr := range cidrList {
index := strings.Index(cidr, "/")
if index <= 0 {
continue
}
// 只支持/24以下的
if types.Int(cidr[index+1:]) < 24 {
continue
}
item.IpFrom = cidr
item.IpTo = ""
err := this.runActionSingleIP(action, listType, item)
@@ -246,6 +280,11 @@ func (this *IPSetAction) runActionSingleIP(action string, listType IPListType, i
if len(path) == 0 {
path, err = exec.LookPath("ipset")
if err != nil {
// 找不到ipset命令错误只提示一次
if this.ipsetNotfound {
return nil
}
this.ipsetNotfound = true
return err
}
}
@@ -258,19 +297,30 @@ func (this *IPSetAction) runActionSingleIP(action string, listType IPListType, i
case "deleteItem":
args = append(args, "del")
}
args = append(args, listName, item.IpFrom)
timestamp := time.Now().Unix()
if item.ExpiredAt > timestamp {
args = append(args, "timeout", strconv.FormatInt(item.ExpiredAt-timestamp, 10))
}
//logs.Println(args)
args = append(args, listName, item.IpFrom)
if action == "addItem" {
timestamp := time.Now().Unix()
if item.ExpiredAt > timestamp {
args = append(args, "timeout", strconv.FormatInt(item.ExpiredAt-timestamp, 10))
}
}
if runtime.GOOS == "darwin" {
// MAC OS直接返回
return nil
}
this.errorBuf.Reset()
cmd := exec.Command(path, args...)
return cmd.Run()
cmd.Stderr = this.errorBuf
err = cmd.Run()
if err != nil {
var errString = this.errorBuf.String()
if action == "deleteItem" && strings.Contains(errString, "not added") {
return nil
}
return errors.New(strings.TrimSpace(errString))
}
return nil
}

View File

@@ -9,15 +9,18 @@ import (
"runtime"
)
// IPTables动作
// IPTablesAction IPTables动作
// 相关命令:
// iptables -A INPUT -s "192.168.2.32" -j ACCEPT
// iptables -A INPUT -s "192.168.2.32" -j REJECT
// iptables -D ...
// iptables -D INPUT ...
// iptables -F INPUT
type IPTablesAction struct {
BaseAction
config *firewallconfigs.FirewallActionIPTablesConfig
iptablesNotFound bool
}
func NewIPTablesAction() *IPTablesAction {
@@ -76,6 +79,10 @@ func (this *IPTablesAction) runActionSingleIP(action string, listType IPListType
if len(path) == 0 {
path, err = exec.LookPath("iptables")
if err != nil {
if this.iptablesNotFound {
return nil
}
this.iptablesNotFound = true
return err
}
}

View File

@@ -14,7 +14,7 @@ import (
var SharedActionManager = NewActionManager()
// 动作管理器定义
// ActionManager 动作管理器定义
type ActionManager struct {
locker sync.Mutex
@@ -23,7 +23,7 @@ type ActionManager struct {
instanceMap map[int64]ActionInterface // id => instance
}
// 获取动作管理对象
// NewActionManager 获取动作管理对象
func NewActionManager() *ActionManager {
return &ActionManager{
configMap: map[int64]*firewallconfigs.FirewallActionConfig{},
@@ -31,7 +31,7 @@ func NewActionManager() *ActionManager {
}
}
// 更新配置
// UpdateActions 更新配置
func (this *ActionManager) UpdateActions(actions []*firewallconfigs.FirewallActionConfig) {
this.locker.Lock()
defer this.locker.Unlock()
@@ -108,14 +108,14 @@ func (this *ActionManager) UpdateActions(actions []*firewallconfigs.FirewallActi
}
}
// 查找事件对应的动作
// FindEventActions 查找事件对应的动作
func (this *ActionManager) FindEventActions(eventLevel string) []ActionInterface {
this.locker.Lock()
defer this.locker.Unlock()
return this.eventMap[eventLevel]
}
// 执行添加IP动作
// AddItem 执行添加IP动作
func (this *ActionManager) AddItem(listType IPListType, item *pb.IPItem) {
instances, ok := this.eventMap[item.EventLevel]
if ok {
@@ -128,7 +128,7 @@ func (this *ActionManager) AddItem(listType IPListType, item *pb.IPItem) {
}
}
// 执行删除IP动作
// DeleteItem 执行删除IP动作
func (this *ActionManager) DeleteItem(listType IPListType, item *pb.IPItem) {
instances, ok := this.eventMap[item.EventLevel]
if ok {

View File

@@ -10,7 +10,7 @@ import (
"path/filepath"
)
// 脚本命令动作
// ScriptAction 脚本命令动作
type ScriptAction struct {
BaseAction

View File

@@ -0,0 +1,130 @@
// 源码改自https://github.com/lionsoul2014/ip2region/blob/master/binding/golang/ip2region/ip2Region.go
package iplibrary
import (
"errors"
"io/ioutil"
"strconv"
"strings"
)
const (
IndexBlockLength = 12
)
var err error
type IP2Region struct {
headerSip []int64
headerPtr []int64
headerLen int64
// super block index info
firstIndexPtr int64
lastIndexPtr int64
totalBlocks int64
dbData []byte
}
type IpInfo struct {
CityId int64
Country string
Region string
Province string
City string
ISP string
}
func (ip IpInfo) String() string {
return strconv.FormatInt(ip.CityId, 10) + "|" + ip.Country + "|" + ip.Region + "|" + ip.Province + "|" + ip.City + "|" + ip.ISP
}
func getIpInfo(cityId int64, line []byte) *IpInfo {
lineSlice := strings.Split(string(line), "|")
ipInfo := &IpInfo{}
length := len(lineSlice)
ipInfo.CityId = cityId
if length < 5 {
for i := 0; i <= 5-length; i++ {
lineSlice = append(lineSlice, "")
}
}
ipInfo.Country = lineSlice[0]
ipInfo.Region = lineSlice[1]
ipInfo.Province = lineSlice[2]
ipInfo.City = lineSlice[3]
ipInfo.ISP = lineSlice[4]
return ipInfo
}
func NewIP2Region(path string) (*IP2Region, error) {
var region = &IP2Region{}
region.dbData, err = ioutil.ReadFile(path)
if err != nil {
return nil, err
}
region.firstIndexPtr = region.ipLongAtOffset(0)
region.lastIndexPtr = region.ipLongAtOffset(4)
region.totalBlocks = (region.lastIndexPtr-region.firstIndexPtr)/IndexBlockLength + 1
return region, nil
}
func (this *IP2Region) MemorySearch(ipStr string) (ipInfo *IpInfo, err error) {
ip, err := ip2long(ipStr)
if err != nil {
return nil, err
}
h := this.totalBlocks
var dataPtr, l int64
for l <= h {
m := (l + h) >> 1
p := this.firstIndexPtr + m*IndexBlockLength
sip := this.ipLongAtOffset(p)
if ip < sip {
h = m - 1
} else {
eip := this.ipLongAtOffset(p + 4)
if ip > eip {
l = m + 1
} else {
dataPtr = this.ipLongAtOffset(p + 8)
break
}
}
}
if dataPtr == 0 {
return nil, errors.New("not found")
}
dataLen := (dataPtr >> 24) & 0xFF
dataPtr = dataPtr & 0x00FFFFFF
return getIpInfo(this.ipLongAtOffset(dataPtr), this.dbData[(dataPtr)+4:dataPtr+dataLen]), nil
}
func (this *IP2Region) ipLongAtOffset(offset int64) int64 {
return int64(this.dbData[offset]) |
int64(this.dbData[offset+1])<<8 |
int64(this.dbData[offset+2])<<16 |
int64(this.dbData[offset+3])<<24
}
func ip2long(IpStr string) (int64, error) {
bits := strings.Split(IpStr, ".")
if len(bits) != 4 {
return 0, errors.New("ip format error")
}
var sum int64
for i, n := range bits {
bit, _ := strconv.ParseInt(n, 10, 64)
sum += bit << uint(24-8*i)
}
return sum, nil
}

View File

@@ -10,7 +10,7 @@ const (
IPItemTypeAll IPItemType = "all" // 所有IP
)
// IP条目
// IPItem IP条目
type IPItem struct {
Type string `json:"type"`
Id int64 `json:"id"`
@@ -20,7 +20,7 @@ type IPItem struct {
EventLevel string `json:"eventLevel"`
}
// 检查是否包含某个IP
// Contains 检查是否包含某个IP
func (this *IPItem) Contains(ip uint64) bool {
switch this.Type {
case IPItemTypeIPv4:
@@ -28,7 +28,7 @@ func (this *IPItem) Contains(ip uint64) bool {
case IPItemTypeIPv6:
return this.containsIPv6(ip)
case IPItemTypeAll:
return this.containsAll(ip)
return this.containsAll()
default:
return this.containsIPv4(ip)
}
@@ -63,7 +63,7 @@ func (this *IPItem) containsIPv6(ip uint64) bool {
}
// 检查是否包所有IP
func (this *IPItem) containsAll(ip uint64) bool {
func (this *IPItem) containsAll() bool {
if this.ExpiredAt > 0 && this.ExpiredAt < utils.UnixTime() {
return false
}

View File

@@ -3,6 +3,7 @@ package iplibrary
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/assert"
"runtime"
"testing"
"time"
)
@@ -72,3 +73,36 @@ func TestIPItem_Contains(t *testing.T) {
a.IsTrue(item.Contains(utils.IP2Long("192.168.1.1")))
}
}
func TestIPItem_Memory(t *testing.T) {
var list = NewIPList()
for i := 0; i < 2_000_000; i ++ {
list.Add(&IPItem{
Type: "ip",
Id: int64(i),
IPFrom: utils.IP2Long("192.168.1.1"),
IPTo: 0,
ExpiredAt: time.Now().Unix(),
EventLevel: "",
})
}
t.Log("waiting")
time.Sleep(10 * time.Second)
}
func BenchmarkIPItem_Contains(b *testing.B) {
runtime.GOMAXPROCS(1)
item := &IPItem{
IPFrom: utils.IP2Long("192.168.1.1"),
IPTo: utils.IP2Long("192.168.1.101"),
ExpiredAt: 0,
}
ip := utils.IP2Long("192.168.1.1")
for i := 0; i < b.N; i++ {
for j := 0; j < 10_000; j++ {
item.Contains(ip)
}
}
}

View File

@@ -3,45 +3,128 @@ package iplibrary
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/expires"
"sort"
"sync"
)
// IPList IP名单
type IPList struct {
itemsMap map[int64]*IPItem // id => item
ipMap map[uint64][]int64 // ip => itemIds
expireList *expires.List
var GlobalBlackIPList = NewIPList()
var GlobalWhiteIPList = NewIPList()
isAll bool
// IPList IP名单
// TODO IP名单可以分片关闭这样让每一片的数据量减少查询更快
type IPList struct {
itemsMap map[int64]*IPItem // id => item
sortedItems []*IPItem
allItemsMap map[int64]*IPItem // id => item
expireList *expires.List
locker sync.RWMutex
}
func NewIPList() *IPList {
list := &IPList{
itemsMap: map[int64]*IPItem{},
ipMap: map[uint64][]int64{},
itemsMap: map[int64]*IPItem{},
allItemsMap: map[int64]*IPItem{},
}
expireList := expires.NewList()
go func() {
expireList.StartGC(func(itemId int64) {
list.Delete(itemId)
})
}()
expireList.OnGC(func(itemId int64) {
list.Delete(itemId)
})
list.expireList = expireList
return list
}
func (this *IPList) Add(item *IPItem) {
this.addItem(item, true)
}
// AddDelay 延迟添加需要手工调用Sort()函数
func (this *IPList) AddDelay(item *IPItem) {
this.addItem(item, false)
}
func (this *IPList) Sort() {
this.locker.Lock()
this.sortItems()
this.locker.Unlock()
}
func (this *IPList) Delete(itemId int64) {
this.locker.Lock()
this.deleteItem(itemId)
this.locker.Unlock()
}
// Contains 判断是否包含某个IP
func (this *IPList) Contains(ip uint64) bool {
this.locker.RLock()
if len(this.allItemsMap) > 0 {
this.locker.RUnlock()
return true
}
var item = this.lookupIP(ip)
this.locker.RUnlock()
return item != nil
}
// ContainsIPStrings 是否包含一组IP中的任意一个并返回匹配的第一个Item
func (this *IPList) ContainsIPStrings(ipStrings []string) (item *IPItem, found bool) {
if len(ipStrings) == 0 {
return
}
this.locker.RLock()
if len(this.allItemsMap) > 0 {
for _, allItem := range this.allItemsMap {
item = allItem
break
}
if item != nil {
this.locker.RUnlock()
found = true
return
}
}
for _, ipString := range ipStrings {
if len(ipString) == 0 {
continue
}
item = this.lookupIP(utils.IP2Long(ipString))
if item != nil {
this.locker.RUnlock()
found = true
return
}
}
this.locker.RUnlock()
return
}
func (this *IPList) addItem(item *IPItem, sortable bool) {
if item == nil {
return
}
if item.ExpiredAt > 0 && item.ExpiredAt < utils.UnixTime() {
return
}
if item.IPFrom == 0 && item.IPTo == 0 {
if item.Type != "all" {
if item.Type != IPItemTypeAll {
return
}
} else if item.IPTo > 0 {
if item.IPFrom > item.IPTo {
item.IPFrom, item.IPTo = item.IPTo, item.IPFrom
} else if item.IPFrom == 0 {
item.IPFrom = item.IPTo
item.IPTo = 0
}
}
this.locker.Lock()
@@ -56,157 +139,86 @@ func (this *IPList) Add(item *IPItem) {
// 展开
if item.IPFrom > 0 {
if item.IPTo == 0 {
this.addIP(item.IPFrom, item.Id)
} else {
if item.IPFrom > item.IPTo {
item.IPTo, item.IPFrom = item.IPFrom, item.IPTo
}
for i := item.IPFrom; i <= item.IPTo; i++ {
// 最多不能超过65535防止整个系统内存爆掉
if i >= item.IPFrom+65535 {
break
}
this.addIP(i, item.Id)
}
}
} else if item.IPTo > 0 {
this.addIP(item.IPTo, item.Id)
this.sortedItems = append(this.sortedItems, item)
} else {
this.addIP(0, item.Id)
// 更新isAll
this.isAll = true
this.allItemsMap[item.Id] = item
}
if item.ExpiredAt > 0 {
this.expireList.Add(item.Id, item.ExpiredAt)
}
if sortable {
this.sortItems()
}
this.locker.Unlock()
}
func (this *IPList) Delete(itemId int64) {
this.locker.Lock()
defer this.locker.Unlock()
this.deleteItem(itemId)
// 更新isAll
this.isAll = len(this.ipMap[0]) > 0
// 对列表进行排序
func (this *IPList) sortItems() {
sort.Slice(this.sortedItems, func(i, j int) bool {
var item1 = this.sortedItems[i]
var item2 = this.sortedItems[j]
if item1.IPFrom == item2.IPFrom {
return item1.IPTo < item2.IPTo
}
return item1.IPFrom < item2.IPFrom
})
}
// Contains 判断是否包含某个IP
func (this *IPList) Contains(ip uint64) bool {
this.locker.RLock()
if this.isAll {
this.locker.RUnlock()
return true
}
_, ok := this.ipMap[ip]
this.locker.RUnlock()
return ok
}
// ContainsIPStrings 是否包含一组IP
func (this *IPList) ContainsIPStrings(ipStrings []string) (found bool, item *IPItem) {
if len(ipStrings) == 0 {
return
}
this.locker.RLock()
if this.isAll {
itemIds := this.ipMap[0]
if len(itemIds) > 0 {
itemId := itemIds[0]
item = this.itemsMap[itemId]
}
this.locker.RUnlock()
found = true
return
}
for _, ipString := range ipStrings {
if len(ipString) == 0 {
continue
}
itemIds, ok := this.ipMap[utils.IP2Long(ipString)]
if ok {
if len(itemIds) > 0 {
itemId := itemIds[0]
item = this.itemsMap[itemId]
// 不加锁的情况下查找Item
func (this *IPList) lookupIP(ip uint64) *IPItem {
var count = len(this.sortedItems)
var resultIndex = -1
sort.Search(count, func(i int) bool {
var item = this.sortedItems[i]
if item.IPFrom < ip {
if item.IPTo >= ip {
resultIndex = i
}
this.locker.RUnlock()
found = true
return
return false
} else if item.IPFrom == ip {
resultIndex = i
return false
}
return true
})
if resultIndex < 0 || resultIndex >= count {
return nil
}
this.locker.RUnlock()
return
return this.sortedItems[resultIndex]
}
// 在不加锁的情况下删除某个Item
// 将会被别的方法引用,切记不能加锁
func (this *IPList) deleteItem(itemId int64) {
item, ok := this.itemsMap[itemId]
_, ok := this.itemsMap[itemId]
if !ok {
return
}
delete(this.itemsMap, itemId)
// 展开
if item.IPFrom > 0 {
if item.IPTo == 0 {
this.deleteIP(item.IPFrom, item.Id)
} else {
if item.IPFrom > item.IPTo {
item.IPTo, item.IPFrom = item.IPFrom, item.IPTo
}
for i := item.IPFrom; i <= item.IPTo; i++ {
// 最多不能超过65535防止整个系统内存爆掉
if i >= item.IPFrom+65535 {
break
}
this.deleteIP(i, item.Id)
}
}
} else if item.IPTo > 0 {
this.deleteIP(item.IPTo, item.Id)
} else {
this.deleteIP(0, item.Id)
}
}
// 添加单个IP
func (this *IPList) addIP(ip uint64, itemId int64) {
itemIds, ok := this.ipMap[ip]
// 是否为All Item
_, ok = this.allItemsMap[itemId]
if ok {
itemIds = append(itemIds, itemId)
} else {
itemIds = []int64{itemId}
}
this.ipMap[ip] = itemIds
}
// 删除单个IP
func (this *IPList) deleteIP(ip uint64, itemId int64) {
itemIds, ok := this.ipMap[ip]
if !ok {
delete(this.allItemsMap, itemId)
return
}
newItemIds := []int64{}
for _, oldItemId := range itemIds {
if oldItemId == itemId {
continue
// 删除排序中的Item
var index = -1
for itemIndex, item := range this.sortedItems {
if item.Id == itemId {
index = itemIndex
break
}
newItemIds = append(newItemIds, oldItemId)
}
if len(newItemIds) > 0 {
this.ipMap[ip] = newItemIds
} else {
delete(this.ipMap, ip)
if index >= 0 {
copy(this.sortedItems[index:], this.sortedItems[index+1:])
this.sortedItems = this.sortedItems[:len(this.sortedItems)-1]
}
}

View File

@@ -0,0 +1,145 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import (
"database/sql"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/Tea"
_ "github.com/mattn/go-sqlite3"
"os"
"path/filepath"
)
type IPListDB struct {
db *sql.DB
itemTableName string
deleteItemStmt *sql.Stmt
insertItemStmt *sql.Stmt
selectItemsStmt *sql.Stmt
dir string
}
func NewIPListDB() (*IPListDB, error) {
var db = &IPListDB{
itemTableName: "ipItems",
dir: filepath.Clean(Tea.Root + "/data"),
}
err := db.init()
return db, err
}
func (this *IPListDB) init() error {
// 检查目录是否存在
_, err := os.Stat(this.dir)
if err != nil {
err = os.MkdirAll(this.dir, 0777)
if err != nil {
return err
}
remotelogs.Println("CACHE", "create cache dir '"+this.dir+"'")
}
db, err := sql.Open("sqlite3", "file:"+this.dir+"/ip_list.db?cache=shared&mode=rwc&_journal_mode=WAL")
if err != nil {
return err
}
db.SetMaxOpenConns(1)
this.db = db
// 初始化数据库
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"listId" integer DEFAULT 0,
"listType" varchar(32),
"isGlobal" integer(1) DEFAULT 0,
"type" varchar(16),
"itemId" integer DEFAULT 0,
"ipFrom" varchar(64) DEFAULT 0,
"ipTo" varchar(64) DEFAULT 0,
"expiredAt" integer DEFAULT 0,
"eventLevel" varchar(32),
"isDeleted" integer(1) DEFAULT 0,
"version" integer DEFAULT 0,
"nodeId" integer DEFAULT 0,
"serverId" integer DEFAULT 0
);
CREATE INDEX IF NOT EXISTS "ip_list_itemId"
ON "` + this.itemTableName + `" (
"itemId" ASC
);
CREATE INDEX IF NOT EXISTS "ip_list_expiredAt"
ON "` + this.itemTableName + `" (
"expiredAt" ASC
);
`)
if err != nil {
return err
}
// 初始化SQL语句
this.deleteItemStmt, err = this.db.Prepare(`DELETE FROM "` + this.itemTableName + `" WHERE "itemId"=?`)
if err != nil {
return err
}
this.insertItemStmt, err = this.db.Prepare(`INSERT INTO "` + this.itemTableName + `" ("listId", "listType", "isGlobal", "type", "itemId", "ipFrom", "ipTo", "expiredAt", "eventLevel", "isDeleted", "version", "nodeId", "serverId") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
return err
}
this.selectItemsStmt, err = this.db.Prepare(`SELECT "listId", "listType", "isGlobal", "type", "itemId", "ipFrom", "ipTo", "expiredAt", "eventLevel", "isDeleted", "version", "nodeId", "serverId" FROM "` + this.itemTableName + `" ORDER BY "version" ASC, "itemId" ASC LIMIT ?, ?`)
if err != nil {
return err
}
this.db = db
return nil
}
func (this *IPListDB) AddItem(item *pb.IPItem) error {
_, err := this.deleteItemStmt.Exec(item.Id)
if err != nil {
return err
}
_, err = this.insertItemStmt.Exec(item.ListId, item.ListType, item.IsGlobal, item.Type, item.Id, item.IpFrom, item.IpTo, item.ExpiredAt, item.EventLevel, item.IsDeleted, item.Version, item.NodeId, item.ServerId)
return err
}
func (this *IPListDB) ReadItems(offset int64, size int64) (items []*pb.IPItem, err error) {
rows, err := this.selectItemsStmt.Query(offset, size)
if err != nil {
return nil, err
}
defer func() {
_ = rows.Close()
}()
for rows.Next() {
// "listId", "listType", "isGlobal", "type", "itemId", "ipFrom", "ipTo", "expiredAt", "eventLevel", "isDeleted", "version", "nodeId", "serverId"
var pbItem = &pb.IPItem{}
err = rows.Scan(&pbItem.ListId, &pbItem.ListType, &pbItem.IsGlobal, &pbItem.Type, &pbItem.Id, &pbItem.IpFrom, &pbItem.IpTo, &pbItem.ExpiredAt, &pbItem.EventLevel, &pbItem.IsDeleted, &pbItem.Version, &pbItem.NodeId, &pbItem.ServerId)
if err != nil {
return nil, err
}
items = append(items, pbItem)
}
return
}
func (this *IPListDB) Close() error {
if this.db != nil {
_ = this.deleteItemStmt.Close()
_ = this.insertItemStmt.Close()
_ = this.selectItemsStmt.Close()
return this.db.Close()
}
return nil
}

View File

@@ -0,0 +1,60 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/logs"
"testing"
"time"
)
func TestIPListDB_AddItem(t *testing.T) {
db, err := NewIPListDB()
if err != nil {
t.Fatal(err)
}
err = db.AddItem(&pb.IPItem{
Id: 1,
IpFrom: "192.168.1.101",
IpTo: "",
Version: 1024,
ExpiredAt: time.Now().Unix(),
Reason: "",
ListId: 2,
IsDeleted: true,
Type: "ipv4",
EventLevel: "error",
ListType: "black",
IsGlobal: true,
CreatedAt: 0,
NodeId: 11,
ServerId: 22,
SourceNodeId: 0,
SourceServerId: 0,
SourceHTTPFirewallPolicyId: 0,
SourceHTTPFirewallRuleGroupId: 0,
SourceHTTPFirewallRuleSetId: 0,
SourceServer: nil,
SourceHTTPFirewallPolicy: nil,
SourceHTTPFirewallRuleGroup: nil,
SourceHTTPFirewallRuleSet: nil,
})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestIPListDB_ReadItems(t *testing.T) {
db, err := NewIPListDB()
if err != nil {
t.Fatal(err)
}
items, err := db.ReadItems(0, 2)
if err != nil {
t.Fatal(err)
}
logs.PrintAsJSON(items, t)
}

View File

@@ -4,8 +4,11 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/assert"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/rands"
"runtime"
"runtime/debug"
"strconv"
"sync"
"testing"
"time"
)
@@ -16,7 +19,7 @@ func TestIPList_Add_Empty(t *testing.T) {
Id: 1,
})
logs.PrintAsJSON(ipList.itemsMap, t)
logs.PrintAsJSON(ipList.ipMap, t)
logs.PrintAsJSON(ipList.allItemsMap, t)
}
func TestIPList_Add_One(t *testing.T) {
@@ -31,15 +34,30 @@ func TestIPList_Add_One(t *testing.T) {
})
ipList.Add(&IPItem{
Id: 3,
IPFrom: utils.IP2Long("2001:db8:0:1::101"),
IPFrom: utils.IP2Long("192.168.0.2"),
})
ipList.Add(&IPItem{
Id: 4,
IPFrom: utils.IP2Long("192.168.0.2"),
IPTo: utils.IP2Long("192.168.0.1"),
})
ipList.Add(&IPItem{
Id: 5,
IPFrom: utils.IP2Long("2001:db8:0:1::101"),
})
ipList.Add(&IPItem{
Id: 6,
IPFrom: 0,
Type: "all",
})
t.Log("===items===")
logs.PrintAsJSON(ipList.itemsMap, t)
logs.PrintAsJSON(ipList.ipMap, t) // ip => items
t.Log("===sorted items===")
logs.PrintAsJSON(ipList.sortedItems, t)
t.Log("===all items===")
logs.PrintAsJSON(ipList.allItemsMap, t) // ip => items
}
func TestIPList_Update(t *testing.T) {
@@ -50,14 +68,31 @@ func TestIPList_Update(t *testing.T) {
})
/**ipList.Add(&IPItem{
Id: 2,
IPFrom: IP2Long("192.168.1.1"),
IPFrom: utils.IP2Long("192.168.1.1"),
})**/
ipList.Add(&IPItem{
Id: 1,
IPTo: utils.IP2Long("192.168.1.2"),
})
logs.PrintAsJSON(ipList.itemsMap, t)
logs.PrintAsJSON(ipList.ipMap, t)
logs.PrintAsJSON(ipList.sortedItems, t)
}
func TestIPList_Update_AllItems(t *testing.T) {
ipList := NewIPList()
ipList.Add(&IPItem{
Id: 1,
Type: IPItemTypeAll,
IPFrom: 0,
})
ipList.Add(&IPItem{
Id: 1,
IPTo: 0,
})
t.Log("===items map===")
logs.PrintAsJSON(ipList.itemsMap, t)
t.Log("===all items map===")
logs.PrintAsJSON(ipList.allItemsMap, t)
}
func TestIPList_Add_Range(t *testing.T) {
@@ -71,9 +106,9 @@ func TestIPList_Add_Range(t *testing.T) {
Id: 2,
IPTo: utils.IP2Long("192.168.1.2"),
})
t.Log(len(ipList.ipMap), "ips")
t.Log(len(ipList.itemsMap), "ips")
logs.PrintAsJSON(ipList.itemsMap, t)
logs.PrintAsJSON(ipList.ipMap, t)
logs.PrintAsJSON(ipList.allItemsMap, t)
}
func TestIPList_Add_Overflow(t *testing.T) {
@@ -85,8 +120,8 @@ func TestIPList_Add_Overflow(t *testing.T) {
IPFrom: utils.IP2Long("192.168.1.1"),
IPTo: utils.IP2Long("192.169.255.1"),
})
t.Log(len(ipList.ipMap), "ips")
a.IsTrue(len(ipList.ipMap) <= 65535)
t.Log(len(ipList.itemsMap), "ips")
a.IsTrue(len(ipList.itemsMap) <= 65535)
}
func TestNewIPList_Memory(t *testing.T) {
@@ -104,20 +139,50 @@ func TestNewIPList_Memory(t *testing.T) {
}
func TestIPList_Contains(t *testing.T) {
var a = assert.NewAssertion(t)
list := NewIPList()
for i := 0; i < 255; i++ {
list.Add(&IPItem{
list.AddDelay(&IPItem{
Id: int64(i),
IPFrom: utils.IP2Long(strconv.Itoa(i) + ".168.0.1"),
IPTo: utils.IP2Long(strconv.Itoa(i) + ".168.255.1"),
ExpiredAt: 0,
})
}
t.Log(len(list.ipMap), "ip")
for i := 0; i < 255; i++ {
list.AddDelay(&IPItem{
Id: int64(1000 + i),
IPFrom: utils.IP2Long("192.167.2." + strconv.Itoa(i)),
})
}
list.Sort()
t.Log(len(list.itemsMap), "ip")
before := time.Now()
t.Log(list.Contains(utils.IP2Long("192.168.1.100")))
t.Log(list.Contains(utils.IP2Long("192.168.2.100")))
a.IsTrue(list.Contains(utils.IP2Long("192.168.1.100")))
a.IsTrue(list.Contains(utils.IP2Long("192.168.2.100")))
a.IsFalse(list.Contains(utils.IP2Long("192.169.3.100")))
a.IsFalse(list.Contains(utils.IP2Long("192.167.3.100")))
a.IsTrue(list.Contains(utils.IP2Long("192.167.2.100")))
t.Log(time.Since(before).Seconds()*1000, "ms")
}
func TestIPList_Contains_Many(t *testing.T) {
list := NewIPList()
for i := 0; i < 1_000_000; i++ {
list.AddDelay(&IPItem{
Id: int64(i),
IPFrom: utils.IP2Long(strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255))),
IPTo: utils.IP2Long(strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255))),
ExpiredAt: 0,
})
}
list.Sort()
t.Log(len(list.itemsMap), "ip")
before := time.Now()
_ = list.Contains(utils.IP2Long("192.168.1.100"))
t.Log(time.Since(before).Seconds()*1000, "ms")
}
@@ -146,6 +211,32 @@ func TestIPList_ContainsAll(t *testing.T) {
}
func TestIPList_ContainsIPStrings(t *testing.T) {
var a = assert.NewAssertion(t)
list := NewIPList()
for i := 0; i < 255; i++ {
list.Add(&IPItem{
Id: int64(i),
IPFrom: utils.IP2Long(strconv.Itoa(i) + ".168.0.1"),
IPTo: utils.IP2Long(strconv.Itoa(i) + ".168.255.1"),
ExpiredAt: 0,
})
}
t.Log(len(list.itemsMap), "ip")
{
item, ok := list.ContainsIPStrings([]string{"192.168.1.100"})
t.Log("item:", item)
a.IsTrue(ok)
}
{
item, ok := list.ContainsIPStrings([]string{"192.167.1.100"})
t.Log("item:", item)
a.IsFalse(ok)
}
}
func TestIPList_Delete(t *testing.T) {
list := NewIPList()
list.Add(&IPItem{
@@ -160,13 +251,13 @@ func TestIPList_Delete(t *testing.T) {
})
t.Log("===BEFORE===")
logs.PrintAsJSON(list.itemsMap, t)
logs.PrintAsJSON(list.ipMap, t)
logs.PrintAsJSON(list.allItemsMap, t)
list.Delete(1)
t.Log("===AFTER===")
logs.PrintAsJSON(list.itemsMap, t)
logs.PrintAsJSON(list.ipMap, t)
logs.PrintAsJSON(list.allItemsMap, t)
}
func TestGC(t *testing.T) {
@@ -184,27 +275,43 @@ func TestGC(t *testing.T) {
ExpiredAt: 0,
})
logs.PrintAsJSON(list.itemsMap, t)
logs.PrintAsJSON(list.ipMap, t)
logs.PrintAsJSON(list.allItemsMap, t)
time.Sleep(2 * time.Second)
t.Log("===AFTER GC===")
logs.PrintAsJSON(list.itemsMap, t)
logs.PrintAsJSON(list.ipMap, t)
logs.PrintAsJSON(list.sortedItems, t)
}
func TestTooManyLists(t *testing.T) {
debug.SetMaxThreads(20)
var lists = []*IPList{}
var locker = &sync.Mutex{}
for i := 0; i < 1000; i++ {
locker.Lock()
lists = append(lists, NewIPList())
locker.Unlock()
}
time.Sleep(1 * time.Second)
t.Log(runtime.NumGoroutine())
t.Log(len(lists), "lists")
}
func BenchmarkIPList_Contains(b *testing.B) {
runtime.GOMAXPROCS(1)
list := NewIPList()
for i := 192; i < 194; i++ {
for i := 1; i < 194; i++ {
list.Add(&IPItem{
Id: int64(1),
IPFrom: utils.IP2Long(strconv.Itoa(i) + ".1.0.1"),
IPTo: utils.IP2Long(strconv.Itoa(i) + ".2.0.1"),
Id: int64(i),
IPFrom: utils.IP2Long(strconv.Itoa(i%255) + "." + strconv.Itoa(i%255) + ".0.1"),
IPTo: utils.IP2Long(strconv.Itoa(i%255) + "." + strconv.Itoa(i%255) + ".0.1"),
ExpiredAt: time.Now().Unix() + 60,
})
}
b.Log(len(list.ipMap), "ip")
b.Log(len(list.itemsMap), "ip")
for i := 0; i < b.N; i++ {
_ = list.Contains(utils.IP2Long("192.168.1.100"))
}

View File

@@ -4,17 +4,16 @@ import (
"fmt"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/lionsoul2014/ip2region/binding/golang/ip2region"
"net"
"strings"
)
type IP2RegionLibrary struct {
db *ip2region.Ip2Region
db *IP2Region
}
func (this *IP2RegionLibrary) Load(dbPath string) error {
db, err := ip2region.New(dbPath)
db, err := NewIP2Region(dbPath)
if err != nil {
return err
}
@@ -49,6 +48,10 @@ func (this *IP2RegionLibrary) Lookup(ip string) (*Result, error) {
return nil, err
}
if info == nil {
return nil, nil
}
if info.Country == "0" {
info.Country = ""
}
@@ -76,7 +79,5 @@ func (this *IP2RegionLibrary) Lookup(ip string) (*Result, error) {
}
func (this *IP2RegionLibrary) Close() {
if this.db != nil {
this.db.Close()
}
}

View File

@@ -3,25 +3,83 @@ package iplibrary
import (
"github.com/iwind/TeaGo/Tea"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/rands"
"runtime"
"strconv"
"sync"
"testing"
"time"
)
func TestIP2RegionLibrary_Lookup_MemoryUsage(t *testing.T) {
var mem = &runtime.MemStats{}
runtime.ReadMemStats(mem)
library := &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
t.Fatal(err)
}
var mem2 = &runtime.MemStats{}
runtime.ReadMemStats(mem2)
t.Log((mem2.HeapInuse-mem.HeapInuse)/1024/1024, "MB")
}
func TestIP2RegionLibrary_Lookup_Single(t *testing.T) {
library := &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
t.Fatal(err)
}
for _, ip := range []string{"8.8.9.9"} {
result, err := library.Lookup(ip)
if err != nil {
t.Fatal(err)
}
t.Log("IP:", ip, "result:", result)
}
}
func TestIP2RegionLibrary_Lookup(t *testing.T) {
library := &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
t.Fatal(err)
}
result, err := library.Lookup("114.240.223.47")
for _, ip := range []string{"", "a", "1.1.1", "192.168.1.100", "114.240.223.47", "8.8.9.9", "::1"} {
result, err := library.Lookup(ip)
if err != nil {
t.Fatal(err)
}
t.Log("IP:", ip, "result:", result)
}
}
func TestIP2RegionLibrary_Lookup_Concurrent(t *testing.T) {
library := &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
t.Fatal(err)
}
logs.PrintAsJSON(result, t)
var count = 4000
var wg = sync.WaitGroup{}
wg.Add(count)
for i := 0; i < count; i++ {
go func() {
defer wg.Done()
for i := 0; i < 100; i++ {
_, _ = library.Lookup(strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)))
}
}()
}
wg.Done()
t.Log("ok")
}
func TestIP2RegionLibrary_Memory(t *testing.T) {
@@ -43,13 +101,13 @@ func TestIP2RegionLibrary_Memory(t *testing.T) {
func BenchmarkIP2RegionLibrary_Lookup(b *testing.B) {
runtime.GOMAXPROCS(1)
library := &IP2RegionLibrary{}
var library = &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
b.Fatal(err)
}
for i := 0; i < b.N; i++ {
_, _ = library.Lookup(strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)))
_, _ = library.Lookup("8.8.8.8")
}
}

View File

@@ -0,0 +1,55 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
)
// AllowIP 检查IP是否被允许访问
func AllowIP(ip string, serverId int64) bool {
var ipLong = utils.IP2Long(ip)
if ipLong == 0 {
return false
}
// check white lists
if GlobalWhiteIPList.Contains(ipLong) {
return true
}
if serverId > 0 {
var list = SharedServerListManager.FindWhiteList(serverId, false)
if list != nil && list.Contains(ipLong) {
return true
}
}
// check black lists
if GlobalBlackIPList.Contains(ipLong) {
return false
}
if serverId > 0 {
var list = SharedServerListManager.FindBlackList(serverId, false)
if list != nil && list.Contains(ipLong) {
return false
}
}
return true
}
// AllowIPStrings 检查一组IP是否被允许访问
func AllowIPStrings(ipStrings []string, serverId int64) bool {
if len(ipStrings) == 0 {
return true
}
for _, ip := range ipStrings {
isAllowed := AllowIP(ip, serverId)
if !isAllowed {
return false
}
}
return true
}

View File

@@ -0,0 +1,20 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import (
"testing"
"time"
)
func TestIPIsAllowed(t *testing.T) {
manager := NewIPListManager()
manager.init()
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
t.Log(AllowIP("127.0.0.1", 0))
t.Log(AllowIP("127.0.0.1", 23))
}

View File

@@ -5,9 +5,9 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/files"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/types"
"regexp"
"strings"
@@ -21,7 +21,7 @@ func init() {
// 初始化
library, err := SharedManager.Load()
if err != nil {
logs.Println("[IP_LIBRARY]" + err.Error())
remotelogs.ErrorObject("IP_LIBRARY", err)
return
}
SharedLibrary = library

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
@@ -20,12 +21,14 @@ import (
var SharedCountryManager = NewCountryManager()
func init() {
events.On(events.EventStart, func() {
go SharedCountryManager.Start()
events.On(events.EventLoaded, func() {
goman.New(func() {
SharedCountryManager.Start()
})
})
}
// 国家信息管理
// CountryManager 国家/地区信息管理
type CountryManager struct {
cacheFile string
@@ -46,13 +49,13 @@ func (this *CountryManager) Start() {
// 从缓存中读取
err := this.load()
if err != nil {
remotelogs.Error("COUNTRY_MANAGER", err.Error())
remotelogs.ErrorObject("COUNTRY_MANAGER", err)
}
// 第一次更新
err = this.loop()
if err != nil {
remotelogs.Error("COUNTRY_MANAGER", err.Error())
remotelogs.ErrorObject("COUNTRY_MANAGER", err)
}
// 定时更新
@@ -60,10 +63,10 @@ func (this *CountryManager) Start() {
events.On(events.EventQuit, func() {
ticker.Stop()
})
for range ticker.C {
for ticker.Next() {
err := this.loop()
if err != nil {
remotelogs.Error("COUNTRY_MANAGER", err.Error())
remotelogs.ErrorObject("COUNTRY_MANAGER", err)
}
}
}

View File

@@ -3,9 +3,12 @@ package iplibrary
import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/Tea"
"sync"
"time"
@@ -15,16 +18,16 @@ var SharedIPListManager = NewIPListManager()
var IPListUpdateNotify = make(chan bool, 1)
func init() {
events.On(events.EventStart, func() {
go SharedIPListManager.Start()
events.On(events.EventLoaded, func() {
goman.New(func() {
SharedIPListManager.Start()
})
})
}
// IP名单管理
// IPListManager IP名单管理
type IPListManager struct {
// 缓存文件
// 每行一个数据id|from|to|expiredAt
cacheFile string
db *IPListDB
version int64
pageSize int64
@@ -35,22 +38,24 @@ type IPListManager struct {
func NewIPListManager() *IPListManager {
return &IPListManager{
cacheFile: Tea.Root + "/configs/ip_list.cache",
pageSize: 1000,
listMap: map[int64]*IPList{},
pageSize: 500,
listMap: map[int64]*IPList{},
}
}
func (this *IPListManager) Start() {
// TODO 从缓存当中读取数据
this.init()
// 第一次读取
err := this.loop()
if err != nil {
remotelogs.Error("IP_LIST_MANAGER", err.Error())
remotelogs.ErrorObject("IP_LIST_MANAGER", err)
}
ticker := time.NewTicker(60 * time.Second)
if Tea.IsTesting() {
ticker = time.NewTicker(10 * time.Second)
}
events.On(events.EventQuit, func() {
ticker.Stop()
})
@@ -64,7 +69,7 @@ func (this *IPListManager) Start() {
if err != nil {
countErrors++
remotelogs.Error("IP_LIST_MANAGER", err.Error())
remotelogs.ErrorObject("IP_LIST_MANAGER", err)
// 连续错误小于3次的我们立即重试
if countErrors <= 3 {
@@ -79,6 +84,31 @@ func (this *IPListManager) Start() {
}
}
func (this *IPListManager) init() {
// 从数据库中当中读取数据
db, err := NewIPListDB()
if err != nil {
remotelogs.Error("IP_LIST_MANAGER", "create ip list local database failed: "+err.Error())
} else {
this.db = db
var offset int64 = 0
var size int64 = 1000
for {
items, err := db.ReadItems(offset, size)
if err != nil {
remotelogs.Error("IP_LIST_MANAGER", "read ip list from local database failed: "+err.Error())
} else {
if len(items) == 0 {
break
}
this.processItems(items, false)
}
offset += int64(len(items))
}
}
}
func (this *IPListManager) loop() error {
for {
hasNext, err := this.fetch()
@@ -88,10 +118,9 @@ func (this *IPListManager) loop() error {
if !hasNext {
break
}
time.Sleep(1 * time.Second)
}
// TODO 写入到缓存当中
return nil
}
@@ -111,37 +140,18 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
if len(items) == 0 {
return false, nil
}
this.locker.Lock()
for _, item := range items {
list, ok := this.listMap[item.ListId]
if !ok {
list = NewIPList()
this.listMap[item.ListId] = list
// 保存到本地数据库
if this.db != nil {
for _, item := range items {
err = this.db.AddItem(item)
if err != nil {
remotelogs.Error("IP_LIST_MANAGER", "insert item to local database failed: "+err.Error())
}
}
if item.IsDeleted {
list.Delete(item.Id)
// 操作事件
SharedActionManager.DeleteItem(item.ListType, item)
continue
}
list.Add(&IPItem{
Id: item.Id,
Type: item.Type,
IPFrom: utils.IP2Long(item.IpFrom),
IPTo: utils.IP2Long(item.IpTo),
ExpiredAt: item.ExpiredAt,
EventLevel: item.EventLevel,
})
// 事件操作
SharedActionManager.DeleteItem(item.ListType, item)
SharedActionManager.AddItem(item.ListType, item)
}
this.locker.Unlock()
this.version = items[len(items)-1].Version
this.processItems(items, true)
return true, nil
}
@@ -152,3 +162,71 @@ func (this *IPListManager) FindList(listId int64) *IPList {
this.locker.Unlock()
return list
}
func (this *IPListManager) processItems(items []*pb.IPItem, shouldExecute bool) {
this.locker.Lock()
var changedLists = map[*IPList]zero.Zero{}
for _, item := range items {
var list *IPList
// TODO 实现节点专有List
if item.ServerId > 0 { // 服务专有List
switch item.ListType {
case "black":
list = SharedServerListManager.FindBlackList(item.ServerId, true)
case "white":
list = SharedServerListManager.FindWhiteList(item.ServerId, true)
}
} else if item.IsGlobal { // 全局List
switch item.ListType {
case "black":
list = GlobalBlackIPList
case "white":
list = GlobalWhiteIPList
}
} else { // 其他List
list = this.listMap[item.ListId]
}
if list == nil {
list = NewIPList()
this.listMap[item.ListId] = list
}
changedLists[list] = zero.New()
if item.IsDeleted {
list.Delete(item.Id)
// 从WAF名单中删除
waf.SharedIPBlackList.RemoveIP(item.IpFrom, item.ServerId)
// 操作事件
if shouldExecute {
SharedActionManager.DeleteItem(item.ListType, item)
}
continue
}
list.AddDelay(&IPItem{
Id: item.Id,
Type: item.Type,
IPFrom: utils.IP2Long(item.IpFrom),
IPTo: utils.IP2Long(item.IpTo),
ExpiredAt: item.ExpiredAt,
EventLevel: item.EventLevel,
})
// 事件操作
if shouldExecute {
SharedActionManager.DeleteItem(item.ListType, item)
SharedActionManager.AddItem(item.ListType, item)
}
}
for changedList := range changedLists {
changedList.Sort()
}
this.locker.Unlock()
this.version = items[len(items)-1].Version
}

View File

@@ -1,10 +1,36 @@
package iplibrary
import "testing"
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/logs"
"testing"
"time"
)
func TestIPListManager_init(t *testing.T) {
manager := NewIPListManager()
manager.init()
t.Log(manager.listMap)
t.Log(SharedServerListManager.blackMap)
logs.PrintAsJSON(GlobalBlackIPList.sortedItems, t)
}
func TestIPListManager_check(t *testing.T) {
manager := NewIPListManager()
manager.init()
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
t.Log(SharedServerListManager.FindBlackList(23, true).Contains(utils.IP2Long("127.0.0.2")))
t.Log(GlobalBlackIPList.Contains(utils.IP2Long("127.0.0.6")))
}
func TestIPListManager_loop(t *testing.T) {
manager := NewIPListManager()
manager.pageSize = 2
manager.Start()
manager.pageSize = 10
err := manager.loop()
if err != nil {
t.Fatal(err)

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
@@ -24,12 +25,14 @@ const (
var SharedProvinceManager = NewProvinceManager()
func init() {
events.On(events.EventStart, func() {
go SharedProvinceManager.Start()
events.On(events.EventLoaded, func() {
goman.New(func() {
SharedProvinceManager.Start()
})
})
}
// 国家信息管理
// ProvinceManager 中国省份信息管理
type ProvinceManager struct {
cacheFile string
@@ -50,13 +53,13 @@ func (this *ProvinceManager) Start() {
// 从缓存中读取
err := this.load()
if err != nil {
remotelogs.Error("PROVINCE_MANAGER", err.Error())
remotelogs.ErrorObject("PROVINCE_MANAGER", err)
}
// 第一次更新
err = this.loop()
if err != nil {
remotelogs.Error("PROVINCE_MANAGER", err.Error())
remotelogs.ErrorObject("PROVINCE_MANAGER", err)
}
// 定时更新
@@ -64,10 +67,10 @@ func (this *ProvinceManager) Start() {
events.On(events.EventQuit, func() {
ticker.Stop()
})
for range ticker.C {
for ticker.Next() {
err := this.loop()
if err != nil {
remotelogs.Error("PROVINCE_MANAGER", err.Error())
remotelogs.ErrorObject("PROVINCE_MANAGER", err)
}
}
}

View File

@@ -0,0 +1,61 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import "sync"
var SharedServerListManager = NewServerListManager()
// ServerListManager 服务相关名单
type ServerListManager struct {
whiteMap map[int64]*IPList // serverId => *List
blackMap map[int64]*IPList // serverId => *List
locker sync.RWMutex
}
func NewServerListManager() *ServerListManager {
return &ServerListManager{
whiteMap: map[int64]*IPList{},
blackMap: map[int64]*IPList{},
}
}
func (this *ServerListManager) FindWhiteList(serverId int64, autoCreate bool) *IPList {
this.locker.RLock()
list, ok := this.whiteMap[serverId]
this.locker.RUnlock()
if ok {
return list
}
if autoCreate {
list = NewIPList()
this.locker.Lock()
this.whiteMap[serverId] = list
this.locker.Unlock()
return list
}
return nil
}
func (this *ServerListManager) FindBlackList(serverId int64, autoCreate bool) *IPList {
this.locker.RLock()
list, ok := this.blackMap[serverId]
this.locker.RUnlock()
if ok {
return list
}
if autoCreate {
list = NewIPList()
this.locker.Lock()
this.blackMap[serverId] = list
this.locker.Unlock()
return list
}
return nil
}

View File

@@ -7,9 +7,10 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/logs"
"os"
"time"
)
@@ -21,27 +22,27 @@ func init() {
})
}
// IP库更新程序
// Updater IP库更新程序
type Updater struct {
}
// 获取新对象
// NewUpdater 获取新对象
func NewUpdater() *Updater {
return &Updater{}
}
// 开始更新
// Start 开始更新
func (this *Updater) Start() {
// 这里不需要太频繁检查更新因为通常不需要更新IP库
ticker := time.NewTicker(1 * time.Hour)
go func() {
goman.New(func() {
for range ticker.C {
err := this.loop()
if err != nil {
logs.Println("[IP_LIBRARY]" + err.Error())
remotelogs.ErrorObject("IP_LIBRARY", err)
}
}
}()
})
}
// 单次任务

1
internal/js/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
*

View File

@@ -3,6 +3,7 @@
package metrics
import (
"encoding/json"
"github.com/cespare/xxhash"
"strconv"
)
@@ -13,10 +14,9 @@ type Stat struct {
Hash string
Value int64
Time string
keysData []byte
}
func (this *Stat) Sum(version int32, itemId int64) {
this.Hash = strconv.FormatUint(xxhash.Sum64String(strconv.FormatInt(this.ServerId, 10)+"@"+string(this.keysData)+"@"+this.Time+"@"+strconv.Itoa(int(version))+"@"+strconv.FormatInt(itemId, 10)), 10)
func SumStat(serverId int64, keys []string, time string, version int32, itemId int64) string {
keysData, _ := json.Marshal(keys)
return strconv.FormatUint(xxhash.Sum64String(strconv.FormatInt(serverId, 10)+"@"+string(keysData)+"@"+time+"@"+strconv.Itoa(int(version))+"@"+strconv.FormatInt(itemId, 10)), 10)
}

View File

@@ -7,9 +7,12 @@ import (
"encoding/json"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/Tea"
_ "github.com/mattn/go-sqlite3"
"os"
@@ -19,6 +22,8 @@ import (
"time"
)
const MaxQueueSize = 10240
// Task 单个指标任务
// 数据库存储:
// data/
@@ -35,7 +40,6 @@ type Task struct {
db *sql.DB
statTableName string
statsChan chan *Stat
isStopped bool
cleanTicker *utils.Ticker
@@ -49,18 +53,22 @@ type Task struct {
selectTopStmt *sql.Stmt
sumStmt *sql.Stmt
serverIdMap map[int64]bool // 所有的服务Ids
timeMap map[string]bool // time => bool
serverIdMap map[int64]zero.Zero // 所有的服务Ids
timeMap map[string]zero.Zero // time => bool
serverIdMapLocker sync.Mutex
statsMap map[string]*Stat
statsLocker sync.Mutex
statsTicker *utils.Ticker
}
// NewTask 获取新任务
func NewTask(item *serverconfigs.MetricItemConfig) *Task {
return &Task{
item: item,
statsChan: make(chan *Stat, 40960),
serverIdMap: map[int64]bool{},
timeMap: map[string]bool{},
serverIdMap: map[int64]zero.Zero{},
timeMap: map[string]zero.Zero{},
statsMap: map[string]*Stat{},
}
}
@@ -131,7 +139,7 @@ ON "` + this.statTableName + `" (
}
// select topN stmt
this.selectTopStmt, err = db.Prepare(`SELECT "id", "hash", "keys", "value", "isUploaded" FROM "` + this.statTableName + `" WHERE "serverId"=? AND "version"=? AND time=? ORDER BY "value" DESC LIMIT 100`)
this.selectTopStmt, err = db.Prepare(`SELECT "id", "hash", "keys", "value", "isUploaded" FROM "` + this.statTableName + `" WHERE "serverId"=? AND "version"=? AND time=? ORDER BY "value" DESC LIMIT 20`)
if err != nil {
return err
}
@@ -156,39 +164,52 @@ ON "` + this.statTableName + `" (
// Start 启动任务
func (this *Task) Start() error {
// 读取数据
go func() {
for stat := range this.statsChan {
if stat == nil {
return
}
err := this.InsertStat(stat)
if err != nil {
remotelogs.Error("METRIC", "insert stat failed: "+err.Error())
this.statsTicker = utils.NewTicker(1 * time.Minute)
goman.New(func() {
for this.statsTicker.Next() {
var tr = trackers.Begin("[METRIC]DUMP_STATS_TO_LOCAL_DATABASE")
this.statsLocker.Lock()
var statsMap = this.statsMap
this.statsMap = map[string]*Stat{}
this.statsLocker.Unlock()
for _, stat := range statsMap {
err := this.InsertStat(stat)
if err != nil {
remotelogs.Error("METRIC", "insert stat failed: "+err.Error())
}
}
tr.End()
}
}()
})
// 清理
this.cleanTicker = utils.NewTicker(24 * time.Hour)
go func() {
goman.New(func() {
for this.cleanTicker.Next() {
var tr = trackers.Begin("[METRIC]CLEAN_EXPIRED")
err := this.CleanExpired()
tr.End()
if err != nil {
remotelogs.Error("METRIC", "clean expired stats failed: "+err.Error())
}
}
}()
})
// 上传
this.uploadTicker = utils.NewTicker(this.item.UploadDuration())
go func() {
goman.New(func() {
for this.uploadTicker.Next() {
var tr = trackers.Begin("[METRIC]UPLOAD_STATS")
err := this.Upload(1 * time.Second)
tr.End()
if err != nil {
remotelogs.Error("METRIC", "upload stats failed: "+err.Error())
}
}
}()
})
return nil
}
@@ -202,6 +223,12 @@ func (this *Task) Add(obj MetricInterface) {
var keys = []string{}
for _, key := range this.item.Keys {
k := obj.MetricKey(key)
// 忽略499状态
if key == "${status}" && k == "499" {
return
}
keys = append(keys, k)
}
@@ -210,18 +237,25 @@ func (this *Task) Add(obj MetricInterface) {
return
}
var stat = &Stat{
ServerId: obj.MetricServerId(),
Keys: keys,
Value: v,
Time: this.item.CurrentTime(),
}
select {
case this.statsChan <- stat:
default:
// 丢弃
var hash = SumStat(obj.MetricServerId(), keys, this.item.CurrentTime(), this.item.Version, this.item.Id)
this.statsLocker.Lock()
oldStat, ok := this.statsMap[hash]
if ok {
oldStat.Value += v
oldStat.Hash = hash
} else {
// 防止过载
if len(this.statsMap) < MaxQueueSize {
this.statsMap[hash] = &Stat{
ServerId: obj.MetricServerId(),
Keys: keys,
Value: v,
Time: this.item.CurrentTime(),
Hash: hash,
}
}
}
this.statsLocker.Unlock()
}
// Stop 停止任务
@@ -234,6 +268,9 @@ func (this *Task) Stop() error {
if this.uploadTicker != nil {
this.uploadTicker.Stop()
}
if this.statsTicker != nil {
this.statsTicker.Stop()
}
_ = this.insertStatStmt.Close()
_ = this.deleteByVersionStmt.Close()
@@ -241,14 +278,6 @@ func (this *Task) Stop() error {
_ = this.selectTopStmt.Close()
_ = this.sumStmt.Close()
if this.statsChan != nil {
go func() {
// 延时关闭,防止关闭时写入
time.Sleep(5 * time.Second)
close(this.statsChan)
}()
}
if this.db != nil {
_ = this.db.Close()
}
@@ -266,18 +295,16 @@ func (this *Task) InsertStat(stat *Stat) error {
}
this.serverIdMapLocker.Lock()
this.serverIdMap[stat.ServerId] = true
this.timeMap[stat.Time] = true
this.serverIdMap[stat.ServerId] = zero.New()
this.timeMap[stat.Time] = zero.New()
this.serverIdMapLocker.Unlock()
keyData, err := json.Marshal(stat.Keys)
if err != nil {
return err
}
stat.keysData = keyData
stat.Sum(this.item.Version, this.item.Id)
_, err = this.insertStatStmt.Exec(stat.ServerId, stat.Hash, stat.keysData, stat.Value, stat.Time, this.item.Version, stat.Value)
_, err = this.insertStatStmt.Exec(stat.ServerId, stat.Hash, keyData, stat.Value, stat.Time, this.item.Version, stat.Value)
if err != nil {
return err
}
@@ -321,14 +348,14 @@ func (this *Task) Upload(pauseDuration time.Duration) error {
for serverId := range this.serverIdMap {
serverIds = append(serverIds, serverId)
}
this.serverIdMap = map[int64]bool{} // 清空数据
this.serverIdMap = map[int64]zero.Zero{} // 清空数据
// 时间
var times = []string{}
for t := range this.timeMap {
times = append(times, t)
}
this.timeMap = map[string]bool{} // 清空数据
this.timeMap = map[string]zero.Zero{} // 清空数据
this.serverIdMapLocker.Unlock()
@@ -354,8 +381,7 @@ func (this *Task) Upload(pauseDuration time.Duration) error {
var pbStats []*pb.UploadingMetricStat
for rows.Next() {
var pbStat = &pb.UploadingMetricStat{
}
var pbStat = &pb.UploadingMetricStat{}
// "id", "hash", "keys", "value", "isUploaded"
var isUploaded int
var keysData []byte
@@ -363,9 +389,11 @@ func (this *Task) Upload(pauseDuration time.Duration) error {
if err != nil {
return nil, err
}
if isUploaded == 1 {
// TODO 先不判断是否已经上传需要改造API进行配合
/**if isUploaded == 1 {
continue
}
}**/
if len(keysData) > 0 {
err = json.Unmarshal(keysData, &pbStat.Keys)
if err != nil {
@@ -444,7 +472,7 @@ func (this *Task) loadServerIdMap() error {
return err
}
this.serverIdMapLocker.Lock()
this.serverIdMap[serverId] = true
this.serverIdMap[serverId] = zero.New()
this.serverIdMapLocker.Unlock()
}
}
@@ -465,7 +493,7 @@ func (this *Task) loadServerIdMap() error {
return err
}
this.serverIdMapLocker.Lock()
this.timeMap[timeString] = true
this.timeMap[timeString] = zero.New()
this.serverIdMapLocker.Unlock()
}
}

View File

@@ -6,6 +6,7 @@ import (
"encoding/json"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/iwind/TeaGo/maps"
@@ -15,8 +16,10 @@ import (
var SharedValueQueue = NewValueQueue()
func init() {
events.On(events.EventStart, func() {
go SharedValueQueue.Start()
events.On(events.EventLoaded, func() {
goman.New(func() {
SharedValueQueue.Start()
})
})
}
@@ -36,7 +39,7 @@ func (this *ValueQueue) Start() {
// 这里单次循环就行因为Loop里已经使用了Range通道
err := this.Loop()
if err != nil {
remotelogs.Error("MONITOR_QUEUE", err.Error())
remotelogs.ErrorObject("MONITOR_QUEUE", err)
}
}
@@ -72,7 +75,11 @@ func (this *ValueQueue) Loop() error {
CreatedAt: value.CreatedAt,
})
if err != nil {
remotelogs.Error("MONITOR", err.Error())
if rpc.IsConnError(err) {
remotelogs.Warn("MONITOR", err.Error())
} else {
remotelogs.Error("MONITOR", err.Error())
}
continue
}
}

View File

@@ -0,0 +1,28 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package monitor
import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/logs"
"google.golang.org/grpc/status"
"testing"
)
func TestValueQueue_RPC(t *testing.T) {
rpcClient, err := rpc.SharedRPC()
if err != nil {
t.Fatal(err)
}
_, err = rpcClient.NodeValueRPC().CreateNodeValue(rpcClient.Context(), &pb.CreateNodeValueRequest{})
if err != nil {
statusErr, ok:= status.FromError(err)
if ok {
logs.Println(statusErr.Code())
}
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -2,21 +2,26 @@ package nodes
import (
"context"
"crypto/tls"
"encoding/json"
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/messageconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/configs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/Tea"
"io"
"net"
"net/http"
"net/url"
"os/exec"
"strconv"
"strings"
@@ -26,6 +31,9 @@ import (
type APIStream struct {
stream pb.NodeService_NodeStreamClient
isQuiting bool
cancelFunc context.CancelFunc
}
func NewAPIStream() *APIStream {
@@ -33,17 +41,19 @@ func NewAPIStream() *APIStream {
}
func (this *APIStream) Start() {
isQuiting := false
events.On(events.EventQuit, func() {
isQuiting = true
this.isQuiting = true
if this.cancelFunc != nil {
this.cancelFunc()
}
})
for {
if isQuiting {
if this.isQuiting {
return
}
err := this.loop()
if err != nil {
remotelogs.Error("API_STREAM", err.Error())
remotelogs.Warn("API_STREAM", err.Error())
time.Sleep(10 * time.Second)
continue
}
@@ -56,19 +66,17 @@ func (this *APIStream) loop() error {
if err != nil {
return errors.Wrap(err)
}
isQuiting := false
ctx, cancelFunc := context.WithCancel(rpcClient.Context())
nodeStream, err := rpcClient.NodeRPC().NodeStream(ctx)
events.On(events.EventQuit, func() {
isQuiting = true
remotelogs.Println("API_STREAM", "quiting")
if nodeStream != nil {
cancelFunc()
}
})
ctx, cancelFunc := context.WithCancel(rpcClient.Context())
this.cancelFunc = cancelFunc
defer func() {
cancelFunc()
}()
nodeStream, err := rpcClient.NodeRPC().NodeStream(ctx)
if err != nil {
if isQuiting {
if this.isQuiting {
return nil
}
return errors.Wrap(err)
@@ -76,14 +84,14 @@ func (this *APIStream) loop() error {
this.stream = nodeStream
for {
if isQuiting {
logs.Println("API_STREAM", "quit")
if this.isQuiting {
remotelogs.Println("API_STREAM", "quit")
break
}
message, err := nodeStream.Recv()
if err != nil {
if isQuiting {
if this.isQuiting {
remotelogs.Println("API_STREAM", "quit")
return nil
}
@@ -110,6 +118,8 @@ func (this *APIStream) loop() error {
err = this.handleNewNodeTask(message)
case messageconfigs.MessageCodeCheckSystemdService: // 检查Systemd服务
err = this.handleCheckSystemdService(message)
case messageconfigs.MessageCodeChangeAPINode: // 修改API节点地址
err = this.handleChangeAPINode(message)
default:
err = this.handleUnknownMessage(message)
}
@@ -133,16 +143,22 @@ func (this *APIStream) handleConnectedAPINode(message *pb.NodeStreamMessage) err
return errors.Wrap(err)
}
rpcClient, err := rpc.SharedRPC()
_, err = rpc.SharedRPC()
if err != nil {
return errors.Wrap(err)
}
_, err = rpcClient.NodeRPC().UpdateNodeConnectedAPINodes(rpcClient.Context(), &pb.UpdateNodeConnectedAPINodesRequest{ApiNodeIds: []int64{msg.APINodeId}})
if err != nil {
return errors.Wrap(err)
}
remotelogs.Println("API_STREAM", "connected to api node '"+strconv.FormatInt(msg.APINodeId, 10)+"'")
// 重新读取配置
if nodeConfigUpdatedAt == 0 {
select {
case nodeConfigChangedNotify <- true:
default:
}
}
return nil
}
@@ -223,7 +239,7 @@ func (this *APIStream) handleReadCache(message *pb.NodeStreamMessage) error {
}()
}
reader, err := storage.OpenReader(msg.Key)
reader, err := storage.OpenReader(msg.Key, false)
if err != nil {
if err == caches.ErrNotFound {
this.replyFail(message.RequestId, "key not found")
@@ -330,6 +346,15 @@ func (this *APIStream) handlePurgeCache(message *pb.NodeStreamMessage) error {
}()
}
// WEBP缓存
if msg.Type == "file" {
var keys = msg.Keys
for _, key := range keys {
keys = append(keys, key+webpSuffix)
}
msg.Keys = keys
}
err = storage.Purge(msg.Keys, msg.Type)
if err != nil {
this.replyFail(message.RequestId, "purge keys failed: "+err.Error())
@@ -367,7 +392,28 @@ func (this *APIStream) handlePreheatCache(message *pb.NodeStreamMessage) error {
wg := sync.WaitGroup{}
wg.Add(len(msg.Keys))
client := http.Client{} // TODO 可以设置请求超时事件
client := &http.Client{
Timeout: 30 * time.Second, // TODO 可以设置请求超时时间
Transport: &http.Transport{
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
_, port, err := net.SplitHostPort(addr)
if err != nil {
return nil, err
}
return net.Dial(network, "127.0.0.1:"+port)
},
MaxIdleConns: 4096,
MaxIdleConnsPerHost: 32,
MaxConnsPerHost: 32,
IdleConnTimeout: 2 * time.Minute,
ExpectContinueTimeout: 1 * time.Second,
TLSHandshakeTimeout: 0,
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
},
},
}
defer client.CloseIdleConnections()
errorMessages := []string{}
locker := sync.Mutex{}
for _, key := range msg.Keys {
@@ -381,7 +427,9 @@ func (this *APIStream) handlePreheatCache(message *pb.NodeStreamMessage) error {
locker.Unlock()
return
}
// TODO 可以在管理界面自定义Header
req.Header.Set("X-Cache-Action", "preheat")
req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36")
req.Header.Set("Accept-Encoding", "gzip, deflate, br") // TODO 这里需要记录下缓存是否为gzip的
resp, err := client.Do(req)
@@ -527,6 +575,65 @@ func (this *APIStream) handleCheckSystemdService(message *pb.NodeStreamMessage)
return nil
}
// 修改API地址
func (this *APIStream) handleChangeAPINode(message *pb.NodeStreamMessage) error {
config, err := configs.LoadAPIConfig()
if err != nil {
this.replyFail(message.RequestId, "read config error: "+err.Error())
return nil
}
var messageData = &messageconfigs.ChangeAPINodeMessage{}
err = json.Unmarshal(message.DataJSON, messageData)
if err != nil {
this.replyFail(message.RequestId, "unmarshal message failed: "+err.Error())
return nil
}
_, err = url.Parse(messageData.Addr)
if err != nil {
this.replyFail(message.RequestId, "invalid new api node address: '"+messageData.Addr+"'")
return nil
}
config.RPC.Endpoints = []string{messageData.Addr}
// 保存到文件
err = config.WriteFile(Tea.ConfigFile("api.yaml"))
if err != nil {
this.replyFail(message.RequestId, "save config file failed: "+err.Error())
return nil
}
this.replyOk(message.RequestId, "")
goman.New(func() {
// 延后生效防止变更前的API无法读取到状态
time.Sleep(1 * time.Second)
rpcClient, err := rpc.SharedRPC()
if err != nil {
remotelogs.Error("API_STREAM", "change rpc endpoint to '"+
messageData.Addr+"' failed: "+err.Error())
return
}
rpcClient.Close()
err = rpcClient.UpdateConfig(config)
if err != nil {
remotelogs.Error("API_STREAM", "change rpc endpoint to '"+
messageData.Addr+"' failed: "+err.Error())
return
}
remotelogs.Println("API_STREAM", "change rpc endpoint to '"+
messageData.Addr+"' successfully")
})
return nil
}
// 处理未知消息
func (this *APIStream) handleUnknownMessage(message *pb.NodeStreamMessage) error {
this.replyFail(message.RequestId, "unknown message code '"+message.Code+"'")

View File

@@ -0,0 +1,101 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/ratelimit"
"net"
"sync"
"sync/atomic"
"time"
)
// ClientConn 客户端连接
type ClientConn struct {
once sync.Once
globalLimiter *ratelimit.Counter
isTLS bool
hasRead bool
BaseClientConn
}
func NewClientConn(conn net.Conn, isTLS bool, quickClose bool, globalLimiter *ratelimit.Counter) net.Conn {
if quickClose {
// TCP
tcpConn, ok := conn.(*net.TCPConn)
if ok {
// TODO 可以在配置中设置此值
_ = tcpConn.SetLinger(nodeconfigs.DefaultTCPLinger)
}
}
return &ClientConn{BaseClientConn: BaseClientConn{rawConn: conn}, isTLS: isTLS, globalLimiter: globalLimiter}
}
func (this *ClientConn) Read(b []byte) (n int, err error) {
if this.isTLS {
if !this.hasRead {
_ = this.rawConn.SetReadDeadline(time.Now().Add(time.Duration(nodeconfigs.DefaultTLSHandshakeTimeout) * time.Second)) // TODO 握手超时时间可以设置
this.hasRead = true
defer func() {
_ = this.rawConn.SetReadDeadline(time.Time{})
}()
}
}
n, err = this.rawConn.Read(b)
if n > 0 {
atomic.AddUint64(&teaconst.InTrafficBytes, uint64(n))
}
return
}
func (this *ClientConn) Write(b []byte) (n int, err error) {
n, err = this.rawConn.Write(b)
if n > 0 {
atomic.AddUint64(&teaconst.OutTrafficBytes, uint64(n))
}
return
}
func (this *ClientConn) Close() error {
this.isClosed = true
err := this.rawConn.Close()
// 全局并发数限制
this.once.Do(func() {
if this.globalLimiter != nil {
this.globalLimiter.Release()
}
})
// 单个服务并发数限制
sharedClientConnLimiter.Remove(this.rawConn.RemoteAddr().String())
return err
}
func (this *ClientConn) LocalAddr() net.Addr {
return this.rawConn.LocalAddr()
}
func (this *ClientConn) RemoteAddr() net.Addr {
return this.rawConn.RemoteAddr()
}
func (this *ClientConn) SetDeadline(t time.Time) error {
return this.rawConn.SetDeadline(t)
}
func (this *ClientConn) SetReadDeadline(t time.Time) error {
return this.rawConn.SetReadDeadline(t)
}
func (this *ClientConn) SetWriteDeadline(t time.Time) error {
return this.rawConn.SetWriteDeadline(t)
}

View File

@@ -0,0 +1,38 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import "net"
type BaseClientConn struct {
rawConn net.Conn
isBound bool
serverId int64
remoteAddr string
isClosed bool
}
func (this *BaseClientConn) IsClosed() bool {
return this.isClosed
}
// IsBound 是否已绑定服务
func (this *BaseClientConn) IsBound() bool {
return this.isBound
}
// Bind 绑定服务
func (this *BaseClientConn) Bind(serverId int64, remoteAddr string, maxConnsPerServer int, maxConnsPerIP int) bool {
if this.isBound {
return true
}
this.isBound = true
this.serverId = serverId
this.remoteAddr = remoteAddr
// 检查是否可以连接
return sharedClientConnLimiter.Add(this.rawConn.RemoteAddr().String(), serverId, remoteAddr, maxConnsPerServer, maxConnsPerIP)
}

View File

@@ -0,0 +1,14 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
type ClientConnInterface interface {
// IsClosed 是否已关闭
IsClosed() bool
// IsBound 是否已绑定服务
IsBound() bool
// Bind 绑定服务
Bind(serverId int64, remoteAddr string, maxConnsPerServer int, maxConnsPerIP int) bool
}

View File

@@ -0,0 +1,130 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/TeaOSLab/EdgeNode/internal/zero"
"sync"
)
var sharedClientConnLimiter = NewClientConnLimiter()
// ClientConnRemoteAddr 客户端地址定义
type ClientConnRemoteAddr struct {
remoteAddr string
serverId int64
}
// ClientConnLimiter 客户端连接数限制
type ClientConnLimiter struct {
remoteAddrMap map[string]*ClientConnRemoteAddr // raw remote addr => remoteAddr
ipConns map[string]map[string]zero.Zero // remoteAddr => { raw remote addr => Zero }
serverConns map[int64]map[string]zero.Zero // serverId => { remoteAddr => Zero }
locker sync.Mutex
}
func NewClientConnLimiter() *ClientConnLimiter {
return &ClientConnLimiter{
remoteAddrMap: map[string]*ClientConnRemoteAddr{},
ipConns: map[string]map[string]zero.Zero{},
serverConns: map[int64]map[string]zero.Zero{},
}
}
// Add 添加新连接
// 返回值为true的时候表示允许添加否则表示不允许添加
func (this *ClientConnLimiter) Add(rawRemoteAddr string, serverId int64, remoteAddr string, maxConnsPerServer int, maxConnsPerIP int) bool {
if (maxConnsPerServer <= 0 && maxConnsPerIP <= 0) || len(remoteAddr) == 0 || serverId <= 0 {
return true
}
this.locker.Lock()
defer this.locker.Unlock()
// 检查服务连接数
var serverMap = this.serverConns[serverId]
if maxConnsPerServer > 0 {
if serverMap == nil {
serverMap = map[string]zero.Zero{}
this.serverConns[serverId] = serverMap
}
if maxConnsPerServer <= len(serverMap) {
return false
}
}
// 检查IP连接数
var ipMap = this.ipConns[remoteAddr]
if maxConnsPerIP > 0 {
if ipMap == nil {
ipMap = map[string]zero.Zero{}
this.ipConns[remoteAddr] = ipMap
}
if maxConnsPerIP > 0 && maxConnsPerIP <= len(ipMap) {
return false
}
}
this.remoteAddrMap[rawRemoteAddr] = &ClientConnRemoteAddr{
remoteAddr: remoteAddr,
serverId: serverId,
}
if maxConnsPerServer > 0 {
serverMap[rawRemoteAddr] = zero.New()
}
if maxConnsPerIP > 0 {
ipMap[rawRemoteAddr] = zero.New()
}
return true
}
// Remove 删除连接
func (this *ClientConnLimiter) Remove(rawRemoteAddr string) {
this.locker.Lock()
defer this.locker.Unlock()
addr, ok := this.remoteAddrMap[rawRemoteAddr]
if !ok {
return
}
delete(this.remoteAddrMap, rawRemoteAddr)
delete(this.ipConns[addr.remoteAddr], rawRemoteAddr)
delete(this.serverConns[addr.serverId], rawRemoteAddr)
if len(this.ipConns[addr.remoteAddr]) == 0 {
delete(this.ipConns, addr.remoteAddr)
}
if len(this.serverConns[addr.serverId]) == 0 {
delete(this.serverConns, addr.serverId)
}
}
// Conns 获取连接信息
// 用于调试
func (this *ClientConnLimiter) Conns() (ipConns map[string][]string, serverConns map[int64][]string) {
this.locker.Lock()
defer this.locker.Unlock()
ipConns = map[string][]string{} // ip => [addr1, addr2, ...]
serverConns = map[int64][]string{} // serverId => [addr1, addr2, ...]
for ip, m := range this.ipConns {
for addr := range m {
ipConns[ip] = append(ipConns[ip], addr)
}
}
for serverId, m := range this.serverConns {
for addr := range m {
serverConns[serverId] = append(serverConns[serverId], addr)
}
}
return
}

View File

@@ -0,0 +1,38 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/iwind/TeaGo/logs"
"testing"
)
func TestClientConnLimiter_Add(t *testing.T) {
var limiter = NewClientConnLimiter()
{
b := limiter.Add("127.0.0.1:1234", 1, "192.168.1.100", 10, 5)
t.Log(b)
}
{
b := limiter.Add("127.0.0.1:1235", 1, "192.168.1.100", 10, 5)
t.Log(b)
}
{
b := limiter.Add("127.0.0.1:1236", 1, "192.168.1.100", 10, 5)
t.Log(b)
}
{
b := limiter.Add("127.0.0.1:1237", 1, "192.168.1.101", 10, 5)
t.Log(b)
}
{
b := limiter.Add("127.0.0.1:1238", 1, "192.168.1.100", 5, 5)
t.Log(b)
}
limiter.Remove("127.0.0.1:1238")
limiter.Remove("127.0.0.1:1239")
limiter.Remove("127.0.0.1:1237")
logs.PrintAsJSON(limiter.remoteAddrMap, t)
logs.PrintAsJSON(limiter.ipConns, t)
logs.PrintAsJSON(limiter.serverConns, t)
}

View File

@@ -0,0 +1,40 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/monitor"
"github.com/iwind/TeaGo/maps"
"sync/atomic"
"time"
)
// 发送监控流量
func init() {
events.On(events.EventStart, func() {
ticker := time.NewTicker(1 * time.Minute)
goman.New(func() {
for range ticker.C {
// 加入到数据队列中
if teaconst.InTrafficBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficIn, maps.Map{
"total": teaconst.InTrafficBytes,
})
}
if teaconst.OutTrafficBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficOut, maps.Map{
"total": teaconst.OutTrafficBytes,
})
}
// 重置数据
atomic.StoreUint64(&teaconst.InTrafficBytes, 0)
atomic.StoreUint64(&teaconst.OutTrafficBytes, 0)
}
})
})
}

View File

@@ -0,0 +1,20 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"net"
)
// 判断客户端连接是否已关闭
func isClientConnClosed(conn net.Conn) bool {
if conn == nil {
return true
}
clientConn, ok := conn.(ClientConnInterface)
if ok {
return clientConn.IsClosed()
}
return true
}

View File

@@ -0,0 +1,74 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/firewallconfigs"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"github.com/TeaOSLab/EdgeNode/internal/ratelimit"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"net"
)
var sharedConnectionsLimiter = ratelimit.NewCounter(nodeconfigs.DefaultTCPMaxConnections)
// ClientListener 客户端网络监听
type ClientListener struct {
rawListener net.Listener
isTLS bool
quickClose bool
}
func NewClientListener(listener net.Listener, quickClose bool) *ClientListener {
return &ClientListener{
rawListener: listener,
quickClose: quickClose,
}
}
func (this *ClientListener) SetIsTLS(isTLS bool) {
this.isTLS = isTLS
}
func (this *ClientListener) IsTLS() bool {
return this.isTLS
}
func (this *ClientListener) Accept() (net.Conn, error) {
// 限制并发连接数
var limiter = sharedConnectionsLimiter
limiter.Ack()
conn, err := this.rawListener.Accept()
if err != nil {
limiter.Release()
return nil, err
}
// 是否在WAF名单中
ip, _, err := net.SplitHostPort(conn.RemoteAddr().String())
if err == nil {
if !iplibrary.AllowIP(ip, 0) || (!waf.SharedIPWhiteList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip) &&
waf.SharedIPBlackList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip)) {
tcpConn, ok := conn.(*net.TCPConn)
if ok {
_ = tcpConn.SetLinger(0)
}
_ = conn.Close()
limiter.Release()
return this.Accept()
}
}
return NewClientConn(conn, this.isTLS, this.quickClose, limiter), nil
}
func (this *ClientListener) Close() error {
return this.rawListener.Close()
}
func (this *ClientListener) Addr() net.Addr {
return this.rawListener.Addr()
}

View File

@@ -0,0 +1,57 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"crypto/tls"
"net"
"time"
)
// ClientTLSConn TLS连接封装
type ClientTLSConn struct {
BaseClientConn
}
func NewClientTLSConn(conn *tls.Conn) net.Conn {
return &ClientTLSConn{BaseClientConn{rawConn: conn}}
}
func (this *ClientTLSConn) Read(b []byte) (n int, err error) {
n, err = this.rawConn.Read(b)
return
}
func (this *ClientTLSConn) Write(b []byte) (n int, err error) {
n, err = this.rawConn.Write(b)
return
}
func (this *ClientTLSConn) Close() error {
this.isClosed = true
// 单个服务并发数限制
sharedClientConnLimiter.Remove(this.rawConn.RemoteAddr().String())
return this.rawConn.Close()
}
func (this *ClientTLSConn) LocalAddr() net.Addr {
return this.rawConn.LocalAddr()
}
func (this *ClientTLSConn) RemoteAddr() net.Addr {
return this.rawConn.RemoteAddr()
}
func (this *ClientTLSConn) SetDeadline(t time.Time) error {
return this.rawConn.SetDeadline(t)
}
func (this *ClientTLSConn) SetReadDeadline(t time.Time) error {
return this.rawConn.SetReadDeadline(t)
}
func (this *ClientTLSConn) SetWriteDeadline(t time.Time) error {
return this.rawConn.SetWriteDeadline(t)
}

View File

@@ -1,33 +1,40 @@
package nodes
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"strings"
"time"
)
var sharedHTTPAccessLogQueue = NewHTTPAccessLogQueue()
// HTTP访问日志队列
// HTTPAccessLogQueue HTTP访问日志队列
type HTTPAccessLogQueue struct {
queue chan *pb.HTTPAccessLog
rpcClient *rpc.RPCClient
}
// 获取新对象
// NewHTTPAccessLogQueue 获取新对象
func NewHTTPAccessLogQueue() *HTTPAccessLogQueue {
// 队列中最大的值,超出此数量的访问日志会被
// 队列中最大的值,超出此数量的访问日志会被
// TODO 需要可以在界面中设置
maxSize := 10000
maxSize := 20000
queue := &HTTPAccessLogQueue{
queue: make(chan *pb.HTTPAccessLog, maxSize),
}
go queue.Start()
goman.New(func() {
queue.Start()
})
return queue
}
// 开始处理访问日志
// Start 开始处理访问日志
func (this *HTTPAccessLogQueue) Start() {
ticker := time.NewTicker(1 * time.Second)
for range ticker.C {
@@ -38,7 +45,7 @@ func (this *HTTPAccessLogQueue) Start() {
}
}
// 加入新访问日志
// Push 加入新访问日志
func (this *HTTPAccessLogQueue) Push(accessLog *pb.HTTPAccessLog) {
select {
case this.queue <- accessLog:
@@ -49,8 +56,9 @@ func (this *HTTPAccessLogQueue) Push(accessLog *pb.HTTPAccessLog) {
// 上传访问日志
func (this *HTTPAccessLogQueue) loop() error {
accessLogs := []*pb.HTTPAccessLog{}
count := 0
var accessLogs = []*pb.HTTPAccessLog{}
var count = 0
Loop:
for {
select {
@@ -59,7 +67,7 @@ Loop:
count++
// 每次只提交 N 条访问日志,防止网络拥堵
if count > 1000 {
if count > 2000 {
break Loop
}
default:
@@ -72,15 +80,65 @@ Loop:
}
// 发送到API
client, err := rpc.SharedRPC()
if err != nil {
return err
if this.rpcClient == nil {
client, err := rpc.SharedRPC()
if err != nil {
return err
}
this.rpcClient = client
}
_, err = client.HTTPAccessLogRPC().CreateHTTPAccessLogs(client.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: accessLogs})
_, err := this.rpcClient.HTTPAccessLogRPC().CreateHTTPAccessLogs(this.rpcClient.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: accessLogs})
if err != nil {
// 是否包含了invalid UTF-8
if strings.Contains(err.Error(), "string field contains invalid UTF-8") {
for _, accessLog := range accessLogs {
this.toValidUTF8(accessLog)
}
// 重新提交
_, err = this.rpcClient.HTTPAccessLogRPC().CreateHTTPAccessLogs(this.rpcClient.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: accessLogs})
return err
}
return err
}
return nil
}
func (this *HTTPAccessLogQueue) toValidUTF8(accessLog *pb.HTTPAccessLog) {
accessLog.RemoteUser = this.toValidUTF8string(accessLog.RemoteUser)
accessLog.RequestURI = this.toValidUTF8string(accessLog.RequestURI)
accessLog.RequestPath = this.toValidUTF8string(accessLog.RequestPath)
accessLog.RequestFilename = this.toValidUTF8string(accessLog.RequestFilename)
accessLog.RequestBody = bytes.ToValidUTF8(accessLog.RequestBody, []byte{})
for _, v := range accessLog.SentHeader {
for index, s := range v.Values {
v.Values[index] = this.toValidUTF8string(s)
}
}
accessLog.Referer = this.toValidUTF8string(accessLog.Referer)
accessLog.UserAgent = this.toValidUTF8string(accessLog.UserAgent)
accessLog.Request = this.toValidUTF8string(accessLog.Request)
accessLog.ContentType = this.toValidUTF8string(accessLog.ContentType)
for k, c := range accessLog.Cookie {
accessLog.Cookie[k] = this.toValidUTF8string(c)
}
accessLog.Args = this.toValidUTF8string(accessLog.Args)
accessLog.QueryString = this.toValidUTF8string(accessLog.QueryString)
for _, v := range accessLog.Header {
for index, s := range v.Values {
v.Values[index] = this.toValidUTF8string(s)
}
}
}
func (this *HTTPAccessLogQueue) toValidUTF8string(v string) string {
return strings.ToValidUTF8(v, "")
}

View File

@@ -0,0 +1,133 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
_ "github.com/iwind/TeaGo/bootstrap"
"google.golang.org/grpc/status"
"reflect"
"runtime"
"strconv"
"strings"
"testing"
"time"
)
func TestHTTPAccessLogQueue_Push(t *testing.T) {
// 发送到API
client, err := rpc.SharedRPC()
if err != nil {
t.Fatal(err)
}
var requestId = 1_000_000
var utf8Bytes = []byte{}
for i := 0; i < 254; i++ {
utf8Bytes = append(utf8Bytes, uint8(i))
}
//bytes = []byte("真不错")
var accessLog = &pb.HTTPAccessLog{
ServerId: 23,
RequestId: strconv.FormatInt(time.Now().Unix(), 10) + strconv.Itoa(requestId) + strconv.FormatInt(1, 10),
NodeId: 48,
Host: "www.hello.com",
RequestURI: string(utf8Bytes),
RequestPath: string(utf8Bytes),
Timestamp: time.Now().Unix(),
Cookie: map[string]string{"test": string(utf8Bytes)},
Header: map[string]*pb.Strings{
"test": {Values: []string{string(utf8Bytes)}},
},
}
new(HTTPAccessLogQueue).toValidUTF8(accessLog)
// logs.PrintAsJSON(accessLog)
//t.Log(strings.ToValidUTF8(string(utf8Bytes), ""))
_, err = client.HTTPAccessLogRPC().CreateHTTPAccessLogs(client.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: []*pb.HTTPAccessLog{
accessLog,
}})
if err != nil {
// 这里只是为了重现错误
t.Logf("%#v, %s", err, err.Error())
statusErr, ok := status.FromError(err)
if ok {
t.Logf("%#v", statusErr)
}
return
}
t.Log("ok")
}
func TestHTTPAccessLogQueue_Push2(t *testing.T) {
var utf8Bytes = []byte{}
for i := 0; i < 254; i++ {
utf8Bytes = append(utf8Bytes, uint8(i))
}
var accessLog = &pb.HTTPAccessLog{
ServerId: 23,
RequestId: strconv.FormatInt(time.Now().Unix(), 10) + strconv.Itoa(1) + strconv.FormatInt(1, 10),
NodeId: 48,
Host: "www.hello.com",
RequestURI: string(utf8Bytes),
RequestPath: string(utf8Bytes),
Timestamp: time.Now().Unix(),
}
var v = reflect.Indirect(reflect.ValueOf(accessLog))
var countFields = v.NumField()
for i := 0; i < countFields; i++ {
var field = v.Field(i)
if field.Kind() == reflect.String {
field.SetString(strings.ToValidUTF8(field.String(), ""))
}
}
client, err := rpc.SharedRPC()
if err != nil {
t.Fatal(err)
}
_, err = client.HTTPAccessLogRPC().CreateHTTPAccessLogs(client.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: []*pb.HTTPAccessLog{
accessLog,
}})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func BenchmarkHTTPAccessLogQueue_ToValidUTF8(b *testing.B) {
runtime.GOMAXPROCS(1)
var utf8Bytes = []byte{}
for i := 0; i < 254; i++ {
utf8Bytes = append(utf8Bytes, uint8(i))
}
for i := 0; i < b.N; i++ {
_ = bytes.ToValidUTF8(utf8Bytes, nil)
}
}
func BenchmarkHTTPAccessLogQueue_ToValidUTF8String(b *testing.B) {
runtime.GOMAXPROCS(1)
var utf8Bytes = []byte{}
for i := 0; i < 254; i++ {
utf8Bytes = append(utf8Bytes, uint8(i))
}
var s = string(utf8Bytes)
for i := 0; i < b.N; i++ {
_ = strings.ToValidUTF8(s, "")
}
}

View File

@@ -5,11 +5,14 @@ import (
"crypto/tls"
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/pires/go-proxyproto"
"net"
"net/http"
"runtime"
"strconv"
"strings"
"sync"
"time"
)
@@ -31,13 +34,15 @@ func NewHTTPClientPool() *HTTPClientPool {
clientsMap: map[string]*HTTPClient{},
}
go pool.cleanClients()
goman.New(func() {
pool.cleanClients()
})
return pool
}
// Client 根据地址获取客户端
func (this *HTTPClientPool) Client(req *http.Request, origin *serverconfigs.OriginConfig, originAddr string) (rawClient *http.Client, err error) {
func (this *HTTPClientPool) Client(req *HTTPRequest, origin *serverconfigs.OriginConfig, originAddr string, proxyProtocol *serverconfigs.ProxyProtocolConfig) (rawClient *http.Client, err error) {
if origin.Addr == nil {
return nil, errors.New("origin addr should not be empty (originId:" + strconv.FormatInt(origin.Id, 10) + ")")
}
@@ -73,11 +78,11 @@ func (this *HTTPClientPool) Client(req *http.Request, origin *serverconfigs.Orig
numberCPU = 8
}
if maxConnections <= 0 {
maxConnections = numberCPU * 8
maxConnections = numberCPU * 32
}
if idleConns <= 0 {
idleConns = numberCPU * 4
idleConns = numberCPU * 8
}
//logs.Println("[ORIGIN]max connections:", maxConnections)
@@ -105,7 +110,7 @@ func (this *HTTPClientPool) Client(req *http.Request, origin *serverconfigs.Orig
for i := 1; i <= retries; i++ {
port := int(toaConfig.RandLocalPort())
// TODO 思考是否支持X-Real-IP/X-Forwarded-IP
err := sharedTOAManager.SendMsg("add:" + strconv.Itoa(port) + ":" + req.RemoteAddr)
err := sharedTOAManager.SendMsg("add:" + strconv.Itoa(port) + ":" + req.requestRemoteAddr(true))
if err != nil {
remotelogs.Error("TOA", "add failed: "+err.Error())
} else {
@@ -126,17 +131,50 @@ func (this *HTTPClientPool) Client(req *http.Request, origin *serverconfigs.Orig
}
// 普通的连接
return (&net.Dialer{
conn, err := (&net.Dialer{
Timeout: connectionTimeout,
KeepAlive: 1 * time.Minute,
}).DialContext(ctx, network, originAddr)
if err != nil {
return nil, err
}
if proxyProtocol != nil && proxyProtocol.IsOn && (proxyProtocol.Version == serverconfigs.ProxyProtocolVersion1 || proxyProtocol.Version == serverconfigs.ProxyProtocolVersion2) {
var remoteAddr = req.requestRemoteAddr(true)
var transportProtocol = proxyproto.TCPv4
if strings.Contains(remoteAddr, ":") {
transportProtocol = proxyproto.TCPv6
}
var destAddr = conn.RemoteAddr()
var reqConn = req.RawReq.Context().Value(HTTPConnContextKey)
if reqConn != nil {
destAddr = reqConn.(net.Conn).LocalAddr()
}
header := proxyproto.Header{
Version: byte(proxyProtocol.Version),
Command: proxyproto.PROXY,
TransportProtocol: transportProtocol,
SourceAddr: &net.TCPAddr{
IP: net.ParseIP(remoteAddr),
Port: req.requestRemotePort(),
},
DestinationAddr: destAddr,
}
_, err = header.WriteTo(conn)
if err != nil {
_ = conn.Close()
return nil, err
}
}
return conn, nil
},
MaxIdleConns: 0,
MaxIdleConnsPerHost: idleConns,
MaxConnsPerHost: maxConnections,
IdleConnTimeout: idleTimeout,
ExpectContinueTimeout: 1 * time.Second,
TLSHandshakeTimeout: 0, // 不限
TLSHandshakeTimeout: 10 * time.Second,
TLSClientConfig: tlsConfig,
Proxy: nil,
}

View File

@@ -21,14 +21,14 @@ func TestHTTPClientPool_Client(t *testing.T) {
t.Fatal(err)
}
{
client, err := pool.Client(nil, origin, origin.Addr.PickAddress())
client, err := pool.Client(nil, origin, origin.Addr.PickAddress(), nil)
if err != nil {
t.Fatal(err)
}
t.Log("client:", client)
}
for i := 0; i < 10; i++ {
client, err := pool.Client(nil, origin, origin.Addr.PickAddress())
client, err := pool.Client(nil, origin, origin.Addr.PickAddress(), nil)
if err != nil {
t.Fatal(err)
}
@@ -53,7 +53,7 @@ func TestHTTPClientPool_cleanClients(t *testing.T) {
for i := 0; i < 10; i++ {
t.Log("get", i)
_, _ = pool.Client(nil, origin, origin.Addr.PickAddress())
_, _ = pool.Client(nil, origin, origin.Addr.PickAddress(), nil)
time.Sleep(1 * time.Second)
}
}
@@ -73,6 +73,6 @@ func BenchmarkHTTPClientPool_Client(b *testing.B) {
pool := NewHTTPClientPool()
for i := 0; i < b.N; i++ {
_, _ = pool.Client(nil, origin, origin.Addr.PickAddress())
_, _ = pool.Client(nil, origin, origin.Addr.PickAddress(), nil)
}
}

View File

@@ -1,6 +1,7 @@
package nodes
import (
"bytes"
"context"
"errors"
"fmt"
@@ -10,8 +11,10 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/metrics"
"github.com/TeaOSLab/EdgeNode/internal/stats"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
"io"
"io/ioutil"
"net"
"net/http"
"net/url"
@@ -25,17 +28,13 @@ import (
// 环境变量
var HOSTNAME, _ = os.Hostname()
// byte pool
var bytePool256b = utils.NewBytePool(20480, 256)
var bytePool1k = utils.NewBytePool(20480, 1024)
var bytePool32k = utils.NewBytePool(20480, 32*1024)
var bytePool128k = utils.NewBytePool(20480, 128*1024)
// errors
var errWritingToClient = errors.New("writing to client error")
// HTTPRequest HTTP请求
type HTTPRequest struct {
requestId string
// 外部参数
RawReq *http.Request
RawWriter http.ResponseWriter
@@ -64,11 +63,14 @@ type HTTPRequest struct {
rewriteRule *serverconfigs.HTTPRewriteRule // 匹配到的重写规则
rewriteReplace string // 重写规则的目标
rewriteIsExternalURL bool // 重写目标是否为外部URL
cacheRef *serverconfigs.HTTPCacheRef // 缓存设置
cacheKey string // 缓存使用的Key
isCached bool // 是否已经被缓存
isAttack bool // 是否是攻击请求
bodyData []byte // 读取的Body内容
cacheRef *serverconfigs.HTTPCacheRef // 缓存设置
cacheKey string // 缓存使用的Key
isCached bool // 是否已经被缓存
cacheCanTryStale bool // 是否可以尝试使用Stale缓存
isAttack bool // 是否是攻击请求
requestBodyData []byte // 读取的Body内容
// WAF相关
firewallPolicyId int64
@@ -86,25 +88,36 @@ type HTTPRequest struct {
// 初始化
func (this *HTTPRequest) init() {
this.writer = NewHTTPWriter(this, this.RawWriter)
this.web = &serverconfigs.HTTPWebConfig{IsOn: true}
this.web = &serverconfigs.HTTPWebConfig{
IsOn: true,
}
// this.uri = this.RawReq.URL.RequestURI()
// 之所以不使用RequestURI()是不想让URL中的Path被Encode
var urlPath = this.RawReq.URL.Path
if this.Server.Web != nil && this.Server.Web.MergeSlashes {
urlPath = utils.CleanPath(urlPath)
this.web.MergeSlashes = true
}
if len(this.RawReq.URL.RawQuery) > 0 {
this.uri = this.RawReq.URL.Path + "?" + this.RawReq.URL.RawQuery
this.uri = urlPath + "?" + this.RawReq.URL.RawQuery
} else {
this.uri = this.RawReq.URL.Path
this.uri = urlPath
}
this.rawURI = this.uri
this.varMapping = map[string]string{
// 缓存相关初始化
"cache.status": "BYPASS",
"cache.age": "0",
"cache.key": "",
"cache.policy.name": "",
"cache.policy.id": "0",
"cache.policy.type": "",
}
this.logAttrs = map[string]string{}
this.requestFromTime = time.Now()
this.requestId = httpRequestNextId()
}
// Do 执行请求
@@ -121,7 +134,32 @@ func (this *HTTPRequest) Do() {
// Web配置
err := this.configureWeb(this.Server.Web, true, 0)
if err != nil {
this.write500(err)
this.write50x(err, http.StatusInternalServerError, false)
this.doEnd()
return
}
// 特殊URL处理
if len(this.rawURI) > 1 && this.rawURI[1] == '.' {
// ACME
// TODO 需要配置是否启用ACME检测
if strings.HasPrefix(this.rawURI, "/.well-known/acme-challenge/") {
this.doACME()
this.doEnd()
return
}
}
// 套餐
if this.Server.UserPlan != nil && !this.Server.UserPlan.IsAvailable() {
this.doPlanExpires()
this.doEnd()
return
}
// 流量限制
if this.Server.TrafficLimit != nil && this.Server.TrafficLimit.IsOn && !this.Server.TrafficLimit.IsEmpty() && this.Server.TrafficLimitStatus != nil && this.Server.TrafficLimitStatus.IsValid() {
this.doTrafficLimit()
this.doEnd()
return
}
@@ -150,9 +188,9 @@ func (this *HTTPRequest) Do() {
}
}
// Gzip
if this.web.GzipRef != nil && this.web.GzipRef.IsOn && this.web.Gzip != nil && this.web.Gzip.IsOn && this.web.Gzip.Level > 0 {
this.writer.Gzip(this.web.Gzip)
// Compression
if this.web.Compression != nil && this.web.Compression.IsOn && this.web.Compression.Level > 0 {
this.writer.SetCompression(this.web.Compression)
}
// 开始调用
@@ -167,9 +205,34 @@ func (this *HTTPRequest) Do() {
// 开始调用
func (this *HTTPRequest) doBegin() {
// 统计
if this.web.StatRef != nil && this.web.StatRef.IsOn {
this.doStat()
// 处理request limit
if this.web.RequestLimit != nil &&
this.web.RequestLimit.IsOn {
if this.doRequestLimit() {
return
}
}
// 处理requestBody
if this.RawReq.ContentLength > 0 &&
this.web.AccessLogRef != nil &&
this.web.AccessLogRef.IsOn &&
this.web.AccessLogRef.ContainsField(serverconfigs.HTTPAccessLogFieldRequestBody) {
var err error
this.requestBodyData, err = ioutil.ReadAll(io.LimitReader(this.RawReq.Body, AccessLogMaxRequestBodySize))
if err != nil {
this.write50x(err, http.StatusBadGateway, false)
return
}
this.RawReq.Body = ioutil.NopCloser(io.MultiReader(bytes.NewBuffer(this.requestBodyData), this.RawReq.Body))
}
// 处理健康检查
var healthCheckKey = this.RawReq.Header.Get(serverconfigs.HealthCheckHeaderName)
if len(healthCheckKey) > 0 {
if this.doHealthCheck(healthCheckKey) {
return
}
}
// 跳转
@@ -179,16 +242,6 @@ func (this *HTTPRequest) doBegin() {
}
}
// 特殊URL处理
if len(this.rawURI) > 1 && this.rawURI[1] == '.' {
// ACME
// TODO 需要配置是否启用ACME检测
if strings.HasPrefix(this.rawURI, "/.well-known/acme-challenge/") {
this.doACME()
return
}
}
// 临时关闭页面
if this.web.Shutdown != nil && this.web.Shutdown.IsOn {
this.doShutdown()
@@ -197,7 +250,7 @@ func (this *HTTPRequest) doBegin() {
// 缓存
if this.web.Cache != nil && this.web.Cache.IsOn {
if this.doCacheRead() {
if this.doCacheRead(false) {
return
}
}
@@ -246,14 +299,15 @@ func (this *HTTPRequest) doEnd() {
// 流量统计
// TODO 增加是否开启开关
// TODO 增加Header统计考虑从Conn中读取
if this.Server != nil {
if this.isCached {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, this.writer.sentBodyBytes, 1, 1, 0, 0)
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, this.writer.sentBodyBytes, 1, 1, 0, 0, this.Server.ShouldCheckTrafficLimit(), this.Server.PlanId())
} else {
if this.isAttack {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 1, this.writer.sentBodyBytes)
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 1, this.writer.sentBodyBytes, this.Server.ShouldCheckTrafficLimit(), this.Server.PlanId())
} else {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 0, 0)
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 0, 0, this.Server.ShouldCheckTrafficLimit(), this.Server.PlanId())
}
}
}
@@ -262,6 +316,12 @@ func (this *HTTPRequest) doEnd() {
if metrics.SharedManager.HasHTTPMetrics() {
this.doMetricsResponse()
}
// 统计
if this.web.StatRef != nil && this.web.StatRef.IsOn {
// 放到最后执行
this.doStat()
}
}
// RawURI 原始的请求URI
@@ -322,6 +382,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.Root = web.Root
}
// remote addr
if web.RemoteAddr != nil && (web.RemoteAddr.IsPrior || isTop) && web.RemoteAddr.IsOn {
this.web.RemoteAddr = web.RemoteAddr
}
// charset
if web.Charset != nil && (web.Charset.IsPrior || isTop) {
this.web.Charset = web.Charset
@@ -333,10 +398,14 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.Websocket = web.Websocket
}
// gzip
if web.GzipRef != nil && (web.GzipRef.IsPrior || isTop) {
this.web.GzipRef = web.GzipRef
this.web.Gzip = web.Gzip
// compression
if web.Compression != nil && (web.Compression.IsPrior || isTop) {
this.web.Compression = web.Compression
}
// webp
if web.WebP != nil && (web.WebP.IsPrior || isTop) {
this.web.WebP = web.WebP
}
// cache
@@ -378,6 +447,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.Auth = web.Auth
}
// request limit
if web.RequestLimit != nil && (web.RequestLimit.IsPrior || isTop) {
this.web.RequestLimit = web.RequestLimit
}
// 重写规则
if len(web.RewriteRefs) > 0 {
for index, ref := range web.RewriteRefs {
@@ -446,6 +520,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
continue
}
if varMapping, isMatched := location.Match(rawPath, this.Format); isMatched {
// 检查专属域名
if len(location.Domains) > 0 && !configutils.MatchDomains(location.Domains, this.Host) {
continue
}
if len(varMapping) > 0 {
this.addVarMapping(varMapping)
}
@@ -501,7 +580,9 @@ func (this *HTTPRequest) Format(source string) string {
case "edgeVersion":
return teaconst.Version
case "remoteAddr":
return this.requestRemoteAddr()
return this.requestRemoteAddr(true)
case "remoteAddrValue":
return this.requestRemoteAddr(false)
case "rawRemoteAddr":
addr := this.RawReq.RemoteAddr
host, _, err := net.SplitHostPort(addr)
@@ -513,6 +594,8 @@ func (this *HTTPRequest) Format(source string) string {
return strconv.Itoa(this.requestRemotePort())
case "remoteUser":
return this.requestRemoteUser()
case "requestId":
return this.requestId
case "requestURI", "requestUri":
return this.rawURI
case "requestURL":
@@ -584,6 +667,11 @@ func (this *HTTPRequest) Format(source string) string {
return this.requestString()
case "cookies":
return this.requestCookiesString()
case "isArgs":
if strings.Contains(this.uri, "?") {
return "?"
}
return ""
case "args", "queryString":
return this.requestQueryString()
case "headers":
@@ -757,22 +845,36 @@ func (this *HTTPRequest) addVarMapping(varMapping map[string]string) {
}
// 获取请求的客户端地址
func (this *HTTPRequest) requestRemoteAddr() string {
func (this *HTTPRequest) requestRemoteAddr(supportVar bool) string {
if supportVar &&
this.web.RemoteAddr != nil &&
this.web.RemoteAddr.IsOn &&
!this.web.RemoteAddr.IsEmpty() {
var remoteAddr = this.Format(this.web.RemoteAddr.Value)
if net.ParseIP(remoteAddr) != nil {
return remoteAddr
}
}
// X-Forwarded-For
forwardedFor := this.RawReq.Header.Get("X-Forwarded-For")
if len(forwardedFor) > 0 {
commaIndex := strings.Index(forwardedFor, ",")
if commaIndex > 0 {
return forwardedFor[:commaIndex]
forwardedFor = forwardedFor[:commaIndex]
}
if net.ParseIP(forwardedFor) != nil {
return forwardedFor
}
return forwardedFor
}
// Real-IP
{
realIP, ok := this.RawReq.Header["X-Real-IP"]
if ok && len(realIP) > 0 {
return realIP[0]
if net.ParseIP(realIP[0]) != nil {
return realIP[0]
}
}
}
@@ -780,7 +882,9 @@ func (this *HTTPRequest) requestRemoteAddr() string {
{
realIP, ok := this.RawReq.Header["X-Real-Ip"]
if ok && len(realIP) > 0 {
return realIP[0]
if net.ParseIP(realIP[0]) != nil {
return realIP[0]
}
}
}
@@ -864,7 +968,7 @@ func (this *HTTPRequest) requestRemotePort() int {
return 0
}
// 情趣的URI中的参数部分
// 获取的URI中的参数部分
func (this *HTTPRequest) requestQueryString() string {
uri, err := url.ParseRequestURI(this.uri)
if err != nil {
@@ -1038,44 +1142,53 @@ func (this *HTTPRequest) processRequestHeaders(reqHeader http.Header) {
}
}
// Add
for _, header := range this.web.RequestHeaderPolicy.AddHeaders {
if !header.IsOn {
continue
}
oldValues, _ := this.RawReq.Header[header.Name]
newHeaderValue := header.Value // 因为我们不能修改header所以在这里使用新变量
if header.HasVariables() {
newHeaderValue = this.Format(header.Value)
}
oldValues = append(oldValues, newHeaderValue)
reqHeader[header.Name] = oldValues
// 支持修改Host
if header.Name == "Host" && len(header.Value) > 0 {
this.RawReq.Host = newHeaderValue
}
}
// Set
for _, header := range this.web.RequestHeaderPolicy.SetHeaders {
if !header.IsOn {
continue
}
newHeaderValue := header.Value // 因为我们不能修改header所以在这里使用新变量
if header.HasVariables() {
newHeaderValue = this.Format(header.Value)
// 是否已删除
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
}
// 请求方法
if len(header.Methods) > 0 && !lists.ContainsString(header.Methods, this.RawReq.Method) {
continue
}
// 域名
if len(header.Domains) > 0 && !configutils.MatchDomains(header.Domains, this.Host) {
continue
}
var headerValue = header.Value
if header.ShouldReplace {
if len(headerValue) == 0 {
headerValue = reqHeader.Get(header.Name) // 原有值
} else if header.HasVariables() {
headerValue = this.Format(header.Value)
}
for _, v := range header.ReplaceValues {
headerValue = v.Replace(headerValue)
}
} else if header.HasVariables() {
headerValue = this.Format(header.Value)
}
reqHeader[header.Name] = []string{newHeaderValue}
// 支持修改Host
if header.Name == "Host" && len(header.Value) > 0 {
this.RawReq.Host = newHeaderValue
this.RawReq.Host = headerValue
} else {
if header.ShouldAppend {
reqHeader[header.Name] = append(reqHeader[header.Name], headerValue)
} else {
reqHeader[header.Name] = []string{headerValue}
}
}
}
// Replace
// TODO 需要实现
}
}
@@ -1100,7 +1213,6 @@ func (this *HTTPRequest) processResponseHeaders(statusCode int) {
// 删除/添加/替换Header
// TODO 实现AddTrailers
// TODO 实现ReplaceHeaders
if this.web.ResponseHeaderPolicy != nil && this.web.ResponseHeaderPolicy.IsOn {
// 删除某些Header
for name := range responseHeader {
@@ -1109,44 +1221,58 @@ func (this *HTTPRequest) processResponseHeaders(statusCode int) {
}
}
// Add
for _, header := range this.web.ResponseHeaderPolicy.AddHeaders {
if !header.IsOn {
continue
}
if header.Match(statusCode) {
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
}
oldValues, _ := responseHeader[header.Name]
if header.HasVariables() {
oldValues = append(oldValues, this.Format(header.Value))
} else {
oldValues = append(oldValues, header.Value)
}
responseHeader[header.Name] = oldValues
}
}
// Set
for _, header := range this.web.ResponseHeaderPolicy.SetHeaders {
if !header.IsOn {
continue
}
if header.Match(statusCode) {
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
// 是否已删除
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
}
// 状态码
if header.Status != nil && !header.Status.Match(statusCode) {
continue
}
// 请求方法
if len(header.Methods) > 0 && !lists.ContainsString(header.Methods, this.RawReq.Method) {
continue
}
// 域名
if len(header.Domains) > 0 && !configutils.MatchDomains(header.Domains, this.Host) {
continue
}
// 是否为跳转
if header.DisableRedirect && httpStatusIsRedirect(statusCode) {
continue
}
var headerValue = header.Value
if header.ShouldReplace {
if len(headerValue) == 0 {
headerValue = responseHeader.Get(header.Name) // 原有值
} else if header.HasVariables() {
headerValue = this.Format(header.Value)
}
if header.HasVariables() {
responseHeader[header.Name] = []string{this.Format(header.Value)}
} else {
responseHeader[header.Name] = []string{header.Value}
for _, v := range header.ReplaceValues {
headerValue = v.Replace(headerValue)
}
} else if header.HasVariables() {
headerValue = this.Format(header.Value)
}
if header.ShouldAppend {
responseHeader[header.Name] = append(responseHeader[header.Name], headerValue)
} else {
responseHeader[header.Name] = []string{headerValue}
}
}
// Replace
// TODO
}
// HSTS
@@ -1171,19 +1297,16 @@ func (this *HTTPRequest) addError(err error) {
// 计算合适的buffer size
func (this *HTTPRequest) bytePool(contentLength int64) *utils.BytePool {
if contentLength <= 0 {
return bytePool1k
}
if contentLength < 1024 { // 1K
return bytePool256b
if contentLength < 8192 { // 8K
return utils.BytePool1k
}
if contentLength < 32768 { // 32K
return bytePool1k
return utils.BytePool4k
}
if contentLength < 1048576 { // 1M
return bytePool32k
if contentLength < 131072 { // 128K
return utils.BytePool16k
}
return bytePool128k
return utils.BytePool32k
}
// 检查是否可以忽略错误
@@ -1220,3 +1343,34 @@ func (this *HTTPRequest) canIgnore(err error) bool {
return false
}
// 关闭当前连接
func (this *HTTPRequest) closeConn() {
requestConn := this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn == nil {
return
}
conn, ok := requestConn.(net.Conn)
if ok {
_ = conn.Close()
return
}
return
}
// 检查连接是否已关闭
func (this *HTTPRequest) isConnClosed() bool {
requestConn := this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn == nil {
return true
}
conn, ok := requestConn.(net.Conn)
if ok {
return isClientConnClosed(conn)
}
return true
}

Some files were not shown because too many files have changed in this diff Show More