Compare commits

...

103 Commits

Author SHA1 Message Date
刘祥超
77bb1cf14e 版本更改为0.4.8.1 2022-06-20 09:34:06 +08:00
刘祥超
274284dbe1 静态文件分发也支持压缩、WebP转换 2022-06-19 11:39:21 +08:00
刘祥超
cd6d7221e8 不限制206 Partial Content两次写入文件的时间差 2022-06-18 20:05:09 +08:00
刘祥超
b1d0c8852e 如果缓存条件支持206 Partial Content,则第一次加载时自动尝试从分片缓存中读取内容 2022-06-18 19:31:10 +08:00
刘祥超
4c4033bb56 TCP负载均衡实现流量限制,达到限制后,关闭连接 2022-06-17 21:49:15 +08:00
刘祥超
6d642b75f6 延长预热超时时间 2022-06-16 20:32:11 +08:00
刘祥超
eb47e3a08c 删除缓存的时同时删除相关的缓存(压缩格式、WebP格式、http和https互换URL) 2022-06-15 12:54:56 +08:00
刘祥超
d82d16e28d 修复内容为空时无法缓存的Bug 2022-06-09 20:26:36 +08:00
刘祥超
b2fc785543 WAF规则中增加${requestURL}参数 2022-06-09 19:44:11 +08:00
刘祥超
189e3342ce 将缓存maxOpenFiles最小值从2改为4 2022-06-09 19:12:29 +08:00
刘祥超
885defbf31 优化nftables相关代码 2022-06-09 19:12:10 +08:00
刘祥超
74f1bf330d DNS解析库默认使用Go原生库 2022-06-07 11:49:38 +08:00
刘祥超
ad843d9d10 修复源站从http跳转到https导致无限循环的问题 2022-06-07 11:25:09 +08:00
刘祥超
13e718742d 节点状态上报时增加时间戳字段 2022-06-07 11:23:40 +08:00
刘祥超
771eff8fb1 增加刷新、预热缓存任务管理 2022-06-05 17:15:02 +08:00
刘祥超
20d7e0b1bf ACME申请证书时如果找不到Token,则直接跳过执行后面请求 2022-06-02 15:34:14 +08:00
刘祥超
e6c7bbec06 修复一个源站主备切换不灵敏的问题/WebSocket也支持源站主备自动切换 2022-05-23 20:01:26 +08:00
刘祥超
be61ef89fe fix typo 2022-05-23 16:15:02 +08:00
刘祥超
3d7d8f1e63 优化代码 2022-05-23 11:34:58 +08:00
刘祥超
a4fb465a19 在严格匹配域名模式下仍然可以通过节点IP进行健康检查 2022-05-23 11:17:53 +08:00
刘祥超
96c725c13d 增加LICENSE和README 2022-05-22 11:36:43 +08:00
刘祥超
7635def2fa 修正自动使用本地防火墙延长封禁时间逻辑 2022-05-21 22:15:11 +08:00
刘祥超
b704a73338 修复一个日志typo 2022-05-21 22:04:23 +08:00
刘祥超
123b5f5969 自动将同集群节点IP加入白名单/尝试使用本地防火墙提升黑名单连接封锁效率 2022-05-21 21:32:10 +08:00
刘祥超
eea2037444 优化验证码失败次数统计 2022-05-21 20:02:35 +08:00
刘祥超
4e6d2fa5ea WAF策略中增加验证码相关定制设置 2022-05-21 11:17:53 +08:00
刘祥超
14bb131e8d WAF CAPTCHA:刷新验证码页面也算入校验失败次数 2022-05-20 11:56:06 +08:00
刘祥超
31814bb54c 忽略301, 302, 303, 307, 308响应中没有Location的错误提示 2022-05-19 20:16:40 +08:00
刘祥超
49b8fd6e97 健康检查增加是否记录访问日志选项 2022-05-19 17:13:20 +08:00
刘祥超
a9d31a2e35 增加edge-node accesslog命令,用来在本地查看访问日志 2022-05-18 23:14:57 +08:00
刘祥超
298cef7f05 缩短指标统计队列长度 2022-05-18 21:41:34 +08:00
刘祥超
9bdd9a433c 实现基础的DDoS防护 2022-05-18 21:03:51 +08:00
刘祥超
45620dcdb7 计算CC的时候不再跨时间范围累积 2022-05-12 21:48:33 +08:00
刘祥超
84a5d38b0b 在启动时检查节点时间戳是否和API节点一致,如果不一致则上报 2022-05-12 21:07:45 +08:00
刘祥超
e812b3fcf6 X-Forwarded-For中包含当前客户端的IP 2022-05-08 16:56:10 +08:00
刘祥超
1bd16fa1d3 往硬盘刷数据时不统计maxOpenFiles 2022-05-07 22:02:41 +08:00
刘祥超
f3ea4957be fix typo 2022-05-05 11:01:03 +08:00
刘祥超
04da107c94 路由规则可以单独设置UAM(仅企业版可用) 2022-05-04 20:32:25 +08:00
刘祥超
e77de69a15 节点增加DNS解析库类型设置 2022-05-04 16:40:25 +08:00
刘祥超
e88eda56f5 自动替换Location中的地址时检查域名是否为当前域名 2022-05-04 10:48:14 +08:00
刘祥超
d6ceccc52e 增加基准测试 2022-05-01 10:40:19 +08:00
刘祥超
cd948ac68c 实现新的gzip库提升gzip性能 2022-04-30 22:22:30 +08:00
刘祥超
9eac8afa3d 停止节点时systemctl命令不阻塞当前进程 2022-04-26 12:22:00 +08:00
刘祥超
fb3610966a 白名单中的IP不受请求限制的影响 2022-04-25 11:11:25 +08:00
刘祥超
a6673449db 修改版本为0.4.8 2022-04-25 11:11:03 +08:00
刘祥超
1d33284b2e 节点状态中增加本地防火墙名称 2022-04-22 14:59:00 +08:00
刘祥超
09ec4507be 优化WAF日志逻辑 2022-04-21 19:44:19 +08:00
刘祥超
a15f0e3051 默认记录WAF的条件从检测到攻击改为所有匹配WAF规则集的请求 2022-04-21 19:02:17 +08:00
刘祥超
67b982c67a 节点状态记录是否检查到本地防火墙 2022-04-21 18:14:53 +08:00
刘祥超
5e1f8f305c 调整默认缓存容量 2022-04-21 09:40:20 +08:00
刘祥超
adfdd5f1b6 强制记录攻击日志 2022-04-21 09:40:05 +08:00
刘祥超
553deda20b 优化代码 2022-04-20 20:05:16 +08:00
刘祥超
7225cef18e 修正文件缓存“慢”打开文件耗时阈值 2022-04-20 18:41:53 +08:00
刘祥超
0e5aef923d 文件缓存增加自动限速/提升本地缓存数据库写入和查询速度 2022-04-20 18:23:26 +08:00
刘祥超
5c54a47587 优化节点停止逻辑 2022-04-20 10:32:40 +08:00
刘祥超
6c294d0282 更新相关库 2022-04-20 10:22:25 +08:00
刘祥超
d6eec0fa52 去除错误提示中的程序文件名中的Workspace目录 2022-04-18 16:31:48 +08:00
刘祥超
9c79152efe 修复UDP服务端口变化时导致的死循环 2022-04-18 15:39:42 +08:00
刘祥超
8765f3a0c6 修复服务配置变化可能导致的死锁 2022-04-18 15:39:02 +08:00
刘祥超
8272fe7fa5 优化缓存相关代码 2022-04-15 14:23:06 +08:00
刘祥超
1636ef8891 优化缓存相关代码 2022-04-14 10:25:34 +08:00
刘祥超
ed0c562b2e 优化缓存相关代码 2022-04-14 09:36:02 +08:00
刘祥超
08a307d8d8 优化代码 2022-04-13 19:24:23 +08:00
刘祥超
5e2bf493b2 优化代码 2022-04-12 21:43:19 +08:00
刘祥超
065dda1dbf 优化基准测试 2022-04-10 21:21:45 +08:00
刘祥超
715e79c3e1 优化代码 2022-04-10 18:46:44 +08:00
刘祥超
2490d6f9d8 优化本地日志 2022-04-10 15:54:30 +08:00
刘祥超
8c4e7129f3 如果Header中Location字段含有跟源站一样的地址,则自动修改为当前域名 2022-04-09 20:37:05 +08:00
刘祥超
3f621c5af0 优化ttlcache 2022-04-09 18:49:52 +08:00
刘祥超
dba9c2c47d 优化ttlcache 2022-04-09 18:44:51 +08:00
刘祥超
ded2f98cce 优化ttlcache 2022-04-09 18:28:22 +08:00
刘祥超
02469f2886 增加基准测试 2022-04-09 10:02:09 +08:00
刘祥超
2121ebe2e0 优化代码 2022-04-08 16:11:05 +08:00
刘祥超
51b9ce6f48 修改个别错误级别 2022-04-08 15:25:53 +08:00
刘祥超
bed19bf844 优化代码/CPU监控信息增加CPU逻辑核数 2022-04-07 09:45:55 +08:00
刘祥超
20a77021f6 将RPC Canceled错误级别调整为警告 2022-04-07 09:45:27 +08:00
刘祥超
c5d061dbe2 优化代码 2022-04-05 15:10:32 +08:00
刘祥超
bd9c8b3d0e 保存L2节点数据时同时记录缓存时间 2022-04-05 11:00:55 +08:00
刘祥超
221d7e6434 缓存文件实现Sendfile 2022-04-04 19:45:57 +08:00
刘祥超
a59007a249 优化代码 2022-04-04 18:25:54 +08:00
刘祥超
6998d3468b 优化代码/商业版支持L2节点 2022-04-04 12:06:53 +08:00
刘祥超
d7ec36b12c 修改一处日志级别 2022-04-02 15:34:00 +08:00
刘祥超
79f4db5a6c response body buffer默认改为16k 2022-04-02 10:41:02 +08:00
刘祥超
048c6f213b 集群可以设置WebP策略 2022-04-01 16:18:15 +08:00
刘祥超
59faf95885 限制WebP可转换的最大长度为128M(非ChunkEncoding下) 2022-03-31 16:30:15 +08:00
刘祥超
7b0b9ac3b4 只有满足缓存条件的图片内容才会被转换 2022-03-31 16:22:23 +08:00
刘祥超
f7ed942779 修复临时关闭页面内容切换到HTML无法使用的问题 2022-03-31 15:17:30 +08:00
刘祥超
6e6be5d8d1 优化OpenFileCache fsnotify事件 2022-03-31 13:30:52 +08:00
刘祥超
221d00b450 修复OpenFileCache可能无法更新的Bug 2022-03-31 11:47:31 +08:00
刘祥超
9689baccf7 修复编译386时可能出现的错误 2022-03-30 22:53:06 +08:00
刘祥超
21de83d31e 反向代理错误增加URL显示 2022-03-30 17:31:47 +08:00
刘祥超
f925d86107 商业版增加UAM功能 2022-03-29 21:24:57 +08:00
刘祥超
db6e20ba37 支持路由定义请求脚本 2022-03-26 22:03:40 +08:00
刘祥超
33a8bae7c5 服务相关流量统计增加Header 2022-03-26 12:29:34 +08:00
刘祥超
b243d6c92e 攻击流量统计使用上行流量 2022-03-26 12:16:25 +08:00
刘祥超
050dd02fa5 优化编译脚本 2022-03-26 11:47:25 +08:00
刘祥超
e5b44ce178 优化代码 2022-03-25 14:11:34 +08:00
刘祥超
f6a983e683 HttpWriter暴露两个方法/默认Buffer为4K 2022-03-24 21:42:03 +08:00
刘祥超
99227bb4f2 IP找不到不再提示错误 2022-03-24 21:41:32 +08:00
刘祥超
d1ea67581e 版本号改为0.4.7 2022-03-23 14:44:43 +08:00
刘祥超
eb6cc9d781 Age改为在缓存中的已存活时间 2022-03-20 21:17:54 +08:00
刘祥超
3c75d860e4 优化代码 2022-03-20 21:15:25 +08:00
刘祥超
ad59c28200 优化代码 2022-03-20 20:58:34 +08:00
152 changed files with 5639 additions and 1211 deletions

29
LICENSE Normal file
View File

@@ -0,0 +1,29 @@
BSD 3-Clause License
Copyright (c) 2020, LiuXiangChao
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

1
README.md Normal file
View File

@@ -0,0 +1 @@
GoEdge边缘节点源码

View File

@@ -42,6 +42,11 @@ function build() {
mkdir $DIST/configs
mkdir $DIST/logs
mkdir $DIST/data
if [ "$TAG" = "plus" ]; then
mkdir $DIST/scripts
mkdir $DIST/scripts/js
fi
fi
cp $ROOT/configs/api.template.yaml $DIST/configs

View File

@@ -22,7 +22,7 @@ func main() {
app := apps.NewAppCmd().
Version(teaconst.Version).
Product(teaconst.ProductName).
Usage(teaconst.ProcessName + " [-v|start|stop|restart|status|quit|test|reload|service|daemon|pprof]").
Usage(teaconst.ProcessName + " [-v|start|stop|restart|status|quit|test|reload|service|daemon|pprof|accesslog]").
Usage(teaconst.ProcessName + " [trackers|goman|conns|gc]").
Usage(teaconst.ProcessName + " [ip.drop|ip.reject|ip.remove] IP")
@@ -258,6 +258,53 @@ func main() {
}
}
})
app.On("accesslog", func() {
// local sock
var tmpDir = os.TempDir()
var sockFile = tmpDir + "/" + teaconst.AccessLogSockName
_, err := os.Stat(sockFile)
if err != nil {
if !os.IsNotExist(err) {
fmt.Println("[ERROR]" + err.Error())
return
}
}
var processSock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := processSock.Send(&gosock.Command{
Code: "accesslog",
})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
return
}
if reply.Code == "error" {
var errString = maps.NewMap(reply.Params).GetString("error")
if len(errString) > 0 {
fmt.Println("[ERROR]" + errString)
return
}
}
conn, err := net.Dial("unix", sockFile)
if err != nil {
fmt.Println("[ERROR]start reading access log failed: " + err.Error())
return
}
defer func() {
_ = conn.Close()
}()
var buf = make([]byte, 1024)
for {
n, err := conn.Read(buf)
if n > 0 {
fmt.Print(string(buf[:n]))
}
if err != nil {
break
}
}
})
app.Run(func() {
node := nodes.NewNode()
node.Start()

7
go.mod
View File

@@ -14,12 +14,15 @@ require (
github.com/chai2010/webp v1.1.0 // indirect
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f
github.com/fsnotify/fsnotify v1.5.1
github.com/go-redis/redis/v8 v8.11.5
github.com/golang/protobuf v1.5.2
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6
github.com/iwind/TeaGo v0.0.0-20220304043459-0dd944a5b475
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3
github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e // indirect
github.com/klauspost/compress v1.15.2 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mattn/go-sqlite3 v1.14.9
github.com/miekg/dns v1.1.43
@@ -28,7 +31,7 @@ require (
github.com/shirou/gopsutil/v3 v3.22.2
golang.org/x/image v0.0.0-20211028202545-6944b10bf410
golang.org/x/net v0.0.0-20220225172249-27dd8689420f
golang.org/x/sys v0.0.0-20220319134239-a9b59b0215f8
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad
golang.org/x/text v0.3.7
google.golang.org/grpc v1.45.0
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect

145
go.sum
View File

@@ -2,6 +2,8 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMT
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/toml v0.4.1 h1:GaI7EiDXDRfa8VshkTj7Fym7ha+y8/XxIgD2okUIjLw=
github.com/BurntSushi/toml v0.4.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/DataDog/sketches-go v0.0.0-20190923095040-43f19ad77ff7/go.mod h1:Q5DbzQ+3AkgGwymQO7aZFNP7ns2lZKGtvRBzRXfdi60=
github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE=
@@ -15,10 +17,17 @@ github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670/go.mod h1:iRWAF
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chai2010/webp v1.1.0 h1:4Ei0/BRroMF9FaXDG2e4OxwFcuW2vcXd+A6tyqTJUQQ=
github.com/chai2010/webp v1.1.0/go.mod h1:LP12PG5IFmLGHUU26tBiCBKnghxx3toZFwDjOYvd3Ow=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/cilium/ebpf v0.5.0/go.mod h1:4tRaxcgiL706VnOzHOdBlY8IEAIdxINsQBcU4xJJXRs=
github.com/cilium/ebpf v0.7.0 h1:1k/q3ATgxSXRdrmPfH8d7YK0GfqVsEKZAX9dQZvs56k=
github.com/cilium/ebpf v0.7.0/go.mod h1:/oI2+1shJiTGAMgl6/RgJr36Eo1jzrRcAWbcXO2usCA=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
@@ -33,12 +42,16 @@ github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f h1:q/DpyjJjZs94bziQ7YkBmIlpqbVP7yw179rnzoNVX1M=
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f/go.mod h1:QGrK8vMWWHQYQ3QU9bw9Y9OPNfxccGzfb41qjvVeXtY=
github.com/dgryski/go-rendezvous v0.0.0-20200624174652-8d2f3be8b2d9/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/frankban/quicktest v1.11.3 h1:8sXhOn0uLys67V8EsXLc6eszDs8VXWxL3iRvebPhedY=
github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsnotify/fsnotify v1.5.1 h1:mZcQUHVQUQWoPXXtuf9yuEXKudkV2sx1E06UadKWpgI=
@@ -48,8 +61,11 @@ github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-redis/redis/v8 v8.0.0-beta.7/go.mod h1:FGJAWDWFht1sQ4qxyJHZZbVyvnVcKQN0E3u5/5lRz+g=
github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -71,25 +87,47 @@ github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMyw
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6 h1:btadZscaRmsi/+fOhkyUguRpSnrf6dykNEWxDeUCj9I=
github.com/google/nftables v0.0.0-20220407195405-950e408d48c6/go.mod h1:0F8on3JWMkm+xahTHItkiu/E1SPqMd0TOxNweQv8ptE=
github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/iwind/TeaGo v0.0.0-20220304043459-0dd944a5b475 h1:EseyfFaQOjWanGiby9KMw7PjDBMg/95tLDgIw/ns0Cw=
github.com/iwind/TeaGo v0.0.0-20220304043459-0dd944a5b475/go.mod h1:HRHK0zoC/og3c9/hKosD9yYVMTnnzm3PgXUdhRYHaLc=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11 h1:DaQjoWZhLNxjhIXedVg4/vFEtHkZhK4IjIwsWdyzBLg=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11/go.mod h1:JtbX20untAjUVjZs1ZBtq80f5rJWvwtQNRL6EnuYRnY=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3 h1:aBSonas7vFcgTj9u96/bWGILGv1ZbUSTLiOzcI1ZT6c=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3/go.mod h1:H5Q7SXwbx3a97ecJkaS2sD77gspzE7HFUafBO0peEyA=
github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4 h1:VWGsCqTzObdlbf7UUE3oceIpcEKi4C/YBUszQXk118A=
github.com/iwind/gosock v0.0.0-20211103081026-ee4652210ca4/go.mod h1:H5Q7SXwbx3a97ecJkaS2sD77gspzE7HFUafBO0peEyA=
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8 h1:AojsHz9Es9B3He2MQQxeRq3TyD//o9huxUo7r1wh44g=
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8/go.mod h1:QJBY2txYhLMzwLO29iB5ujDJ3s3V7DsZ582nw4Ss+tM=
github.com/josharian/native v0.0.0-20200817173448-b6b71def0850 h1:uhL5Gw7BINiiPAo24A2sxkcDI0Jt/sqp1v5xQCniEFA=
github.com/josharian/native v0.0.0-20200817173448-b6b71def0850/go.mod h1:7X/raswPFr05uY3HiLlYeyQntB6OO7E/d2Cu7qoaN2w=
github.com/jsimonetti/rtnetlink v0.0.0-20190606172950-9527aa82566a/go.mod h1:Oz+70psSo5OFh8DBl0Zv2ACw7Esh6pPUphlvZG9x7uw=
github.com/jsimonetti/rtnetlink v0.0.0-20200117123717-f846d4f6c1f4/go.mod h1:WGuG/smIU4J/54PblvSbh+xvCZmpJnFgr3ds6Z55XMQ=
github.com/jsimonetti/rtnetlink v0.0.0-20201009170750-9c6f07d100c1/go.mod h1:hqoO/u39cqLeBLebZ8fWdE96O7FxrAsRYhnVOdgHxok=
github.com/jsimonetti/rtnetlink v0.0.0-20201216134343-bde56ed16391/go.mod h1:cR77jAZG3Y3bsb8hF6fHJbFoyFukLFOkQ98S0pQz3xw=
github.com/jsimonetti/rtnetlink v0.0.0-20201220180245-69540ac93943/go.mod h1:z4c53zj6Eex712ROyh8WI0ihysb5j2ROyV42iNogmAs=
github.com/jsimonetti/rtnetlink v0.0.0-20210122163228-8d122574c736/go.mod h1:ZXpIyOK59ZnN7J0BV99cZUPmsqDRZ3eq5X+st7u/oSA=
github.com/jsimonetti/rtnetlink v0.0.0-20210212075122-66c871082f2b/go.mod h1:8w9Rh8m+aHZIG69YPGGem1i5VzoyRC8nw2kA8B+ik5U=
github.com/jsimonetti/rtnetlink v0.0.0-20210525051524-4cc836578190/go.mod h1:NmKSdU4VGSiv1bMsdqNALI4RSvvjtz65tTMCnD05qLo=
github.com/jsimonetti/rtnetlink v0.0.0-20211022192332-93da33804786 h1:N527AHMa793TP5z5GNAn/VLPzlc0ewzWdeP/25gDfgQ=
github.com/jsimonetti/rtnetlink v0.0.0-20211022192332-93da33804786/go.mod h1:v4hqbTdfQngbVSZJVWUhGE/lbTFf9jb+ygmNUDQMuOs=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e h1:LvL4XsI70QxOGHed6yhQtAU34Kx3Qq2wwBzGFKY8zKk=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e/go.mod h1:kLgvv7o6UM+0QSf0QjAse3wReFDsb9qbZJdfexWlrQw=
github.com/klauspost/compress v1.15.2 h1:3WH+AG7s2+T8o3nrM/8u2rdqUEcQhmga7smjrT41nAw=
github.com/klauspost/compress v1.15.2/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@@ -101,6 +139,27 @@ github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/mattn/go-sqlite3 v1.14.9 h1:10HX2Td0ocZpYEjhilsuo6WWtUqttj2Kb0KtD86/KYA=
github.com/mattn/go-sqlite3 v1.14.9/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mdlayher/ethtool v0.0.0-20210210192532-2b88debcdd43/go.mod h1:+t7E0lkKfbBsebllff1xdTmyJt8lH37niI6kwFk9OTo=
github.com/mdlayher/ethtool v0.0.0-20211028163843-288d040e9d60 h1:tHdB+hQRHU10CfcK0furo6rSNgZ38JT8uPh70c/pFD8=
github.com/mdlayher/ethtool v0.0.0-20211028163843-288d040e9d60/go.mod h1:aYbhishWc4Ai3I2U4Gaa2n3kHWSwzme6EsG/46HRQbE=
github.com/mdlayher/genetlink v1.0.0 h1:OoHN1OdyEIkScEmRgxLEe2M9U8ClMytqA5niynLtfj0=
github.com/mdlayher/genetlink v1.0.0/go.mod h1:0rJ0h4itni50A86M2kHcgS85ttZazNt7a8H2a2cw0Gc=
github.com/mdlayher/netlink v0.0.0-20190409211403-11939a169225/go.mod h1:eQB3mZE4aiYnlUsyGGCOpPETfdQq4Jhsgf1fk3cwQaA=
github.com/mdlayher/netlink v1.0.0/go.mod h1:KxeJAFOFLG6AjpyDkQ/iIhxygIUKD+vcwqcnu43w/+M=
github.com/mdlayher/netlink v1.1.0/go.mod h1:H4WCitaheIsdF9yOYu8CFmCgQthAPIWZmcKp9uZHgmY=
github.com/mdlayher/netlink v1.1.1/go.mod h1:WTYpFb/WTvlRJAyKhZL5/uy69TDDpHHu2VZmb2XgV7o=
github.com/mdlayher/netlink v1.2.0/go.mod h1:kwVW1io0AZy9A1E2YYgaD4Cj+C+GPkU6klXCMzIJ9p8=
github.com/mdlayher/netlink v1.2.1/go.mod h1:bacnNlfhqHqqLo4WsYeXSqfyXkInQ9JneWI68v1KwSU=
github.com/mdlayher/netlink v1.2.2-0.20210123213345-5cc92139ae3e/go.mod h1:bacnNlfhqHqqLo4WsYeXSqfyXkInQ9JneWI68v1KwSU=
github.com/mdlayher/netlink v1.3.0/go.mod h1:xK/BssKuwcRXHrtN04UBkwQ6dY9VviGGuriDdoPSWys=
github.com/mdlayher/netlink v1.4.0/go.mod h1:dRJi5IABcZpBD2A3D0Mv/AiX8I9uDEu5oGkAVrekmf8=
github.com/mdlayher/netlink v1.4.1/go.mod h1:e4/KuJ+s8UhfUpO9z00/fDZZmhSrs+oxyqAS9cNgn6Q=
github.com/mdlayher/netlink v1.4.2 h1:3sbnJWe/LETovA7yRZIX3f9McVOWV3OySH6iIBxiFfI=
github.com/mdlayher/netlink v1.4.2/go.mod h1:13VaingaArGUTUxFLf/iEovKxXji32JAtF858jZYEug=
github.com/mdlayher/socket v0.0.0-20210307095302-262dc9984e00/go.mod h1:GAFlyu4/XV68LkQKYzKhIo/WW7j3Zi0YRAz/BOoanUc=
github.com/mdlayher/socket v0.0.0-20211007213009-516dcbdf0267/go.mod h1:nFZ1EtZYK8Gi/k6QNu7z7CgO20i/4ExeQswwWuPmG/g=
github.com/mdlayher/socket v0.0.0-20211102153432-57e3fa563ecb h1:2dC7L10LmTqlyMVzFJ00qM25lqESg9Z4u3GuEXN5iHY=
github.com/mdlayher/socket v0.0.0-20211102153432-57e3fa563ecb/go.mod h1:nFZ1EtZYK8Gi/k6QNu7z7CgO20i/4ExeQswwWuPmG/g=
github.com/miekg/dns v1.1.43 h1:JKfpVSCB84vrAmHzyrsxB5NAr5kLoMXZArPSw7Qlgyg=
github.com/miekg/dns v1.1.43/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -109,13 +168,23 @@ github.com/mssola/user_agent v0.5.3 h1:lBRPML9mdFuIZgI2cmlQ+atbpJdLdeVl2IDodjBR5
github.com/mssola/user_agent v0.5.3/go.mod h1:TTPno8LPY3wAIEKRpAtkdMT0f8SE24pLRGPahjCH4uw=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
github.com/onsi/ginkgo/v2 v2.0.0 h1:CcuG/HvWNkkaqCUpJifQY8z7qEMBJya6aLPx6ftGyjQ=
github.com/onsi/ginkgo/v2 v2.0.0/go.mod h1:vw5CSIxN1JObi/U8gcbwft7ZxR2dgaR70JSE3/PpL4c=
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/onsi/gomega v1.17.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY=
github.com/onsi/gomega v1.18.1 h1:M1GfJqGRrBrrGGsbxzV5dqM2U2ApXefZCQpkukxYRLE=
github.com/onsi/gomega v1.18.1/go.mod h1:0q+aL8jAiMXy9hbwj2mr5GziHiwhAIQpFmmtT5hitRs=
github.com/opentracing/opentracing-go v1.1.1-0.20190913142402-a7454ce5950e/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/pires/go-proxyproto v0.6.1 h1:EBupykFmo22SDjv4fQVQd2J9NOoLPmyZA/15ldOGkPw=
github.com/pires/go-proxyproto v0.6.1/go.mod h1:Odh9VFOZJCf9G8cLW5o435Xf1J95Jw9Gw5rnCjcwzAY=
@@ -139,6 +208,11 @@ github.com/tklauser/go-sysconf v0.3.9 h1:JeUVdAOWhhxVcU6Eqr/ATFHgXk/mmiItdKeJPev
github.com/tklauser/go-sysconf v0.3.9/go.mod h1:11DU/5sG7UexIrp/O6g35hrWzu0JxlwQ3LSFUzyeuhs=
github.com/tklauser/numcpus v0.3.0 h1:ILuRUQBtssgnxw0XXIjKUC56fgnOrFoQQ/4+DeU2biQ=
github.com/tklauser/numcpus v0.3.0/go.mod h1:yFGUr7TUHQRAhyqBcEg0Ge34zDBAsIvJJcyE6boqnA8=
github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc h1:R83G5ikgLMxrBvLh22JhdfI8K6YXEPHx5P03Uu3DRs4=
github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg=
github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.opentelemetry.io/otel v0.7.0/go.mod h1:aZMyHG5TqDOXEgH2tyLiXSUKly1jT3yqE9PmrzIeCdo=
@@ -159,6 +233,10 @@ golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHl
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38=
golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -167,11 +245,28 @@ golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191007182048-72f939374954/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201010224723-4f7140c49acb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201216054612-986b41b23924/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210928044308-7d9f5e0b762b/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211020060615-d418f374d309/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211201190559-0a0e4e1bb54c/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211209124913-491a49abca63/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f h1:oA4XRj0qtSt8Yo1Zms0CUlsT3KG69V2UGQWPBxujDmc=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
@@ -180,35 +275,61 @@ golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190411185658-b44545bcd369/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201009025420-dfb3f7c4e634/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201118182958-a01c418693c7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201218084310-7d0127a74742/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210110051926-789bb1bd4061/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210123111255-9b0068b26619/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210216163648-f7da38b97c65/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210305230114-8fe3ee5dd75b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210525143221-35b2ab0089ea/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210816074244-15123e1e1f71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211205182925-97ca703d548d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220111092808-5a964db01320/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9 h1:nhht2DYV/Sn3qOayu8lM+cU1ii9sTLUeBQwQQfUHtrs=
golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220319134239-a9b59b0215f8 h1:OH54vjqzRWmbJ62fjuhxy7AxFFgoHN0/DPc/UrL8cAs=
golang.org/x/sys v0.0.0-20220319134239-a9b59b0215f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad h1:ntjMns5wyP/fN65tdBD4g8J5w8n015+iIIs9rtjXkY0=
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -223,7 +344,14 @@ golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGm
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo=
golang.org/x/tools v0.1.8 h1:P1HhGGuLW4aAclzjtmJdf0mJOjVUZUzOTqkAkWL+l6w=
golang.org/x/tools v0.1.8/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
@@ -235,8 +363,6 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98
google.golang.org/genproto v0.0.0-20191009194640-548a555dbc03/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20220303160752-862486edd9cc h1:fb/ViRpv3ln/LvbqZtTpoOd1YQDNH12gaGZreoSFovE=
google.golang.org/genproto v0.0.0-20220303160752-862486edd9cc/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI=
google.golang.org/genproto v0.0.0-20220317150908-0efb43f6373e h1:fNKDNuUyC4WH+inqDMpfXDdfvwfYILbsX+oskGZ8hxg=
google.golang.org/genproto v0.0.0-20220317150908-0efb43f6373e/go.mod h1:hAL49I2IFola2sVEjAn7MEwsja0xp51I0tlGAf9hz4E=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
@@ -246,8 +372,6 @@ google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.44.0 h1:weqSxi/TMs1SqFRMHCtBgXRs8k3X39QIDEZ0pRcttUg=
google.golang.org/grpc v1.44.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU=
google.golang.org/grpc v1.45.0 h1:NEpgUqV3Z+ZjkqMsxMg11IaDrXY4RY6CQukSGK0uI1M=
google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
@@ -269,6 +393,7 @@ gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -276,8 +401,14 @@ gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.2.1/go.mod h1:lPVVZ2BS5TfnjLyizF7o7hv7j9/L+8cZY2hLyjP9cGY=
honnef.co/go/tools v0.2.2 h1:MNh1AVMyVX23VUHE2O27jm6lNj3vjO5DexS4A1xvnzk=
honnef.co/go/tools v0.2.2/go.mod h1:lPVVZ2BS5TfnjLyizF7o7hv7j9/L+8cZY2hLyjP9cGY=
rogchap.com/v8go v0.7.0/go.mod h1:MxgP3pL2MW4dpme/72QRs8sgNMmM0pRc8DPhcuLWPAs=

View File

@@ -213,7 +213,19 @@ func (this *AppCmd) runStop() {
return
}
_, _ = this.sock.Send(&gosock.Command{Code: "stop"})
// 从systemd中停止
if runtime.GOOS == "linux" {
systemctl, _ := exec.LookPath("systemctl")
if len(systemctl) > 0 {
go func() {
// 有可能会长时间执行,这里不阻塞进程
_ = exec.Command(systemctl, "stop", teaconst.SystemdServiceName).Run()
}()
}
}
// 如果仍在运行,则发送停止指令
_, _ = this.sock.SendTimeout(&gosock.Command{Code: "stop"}, 1*time.Second)
fmt.Println(this.product+" stopped ok, pid:", types.String(pid))
}

View File

@@ -1,10 +1,12 @@
package apps
import (
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/sizes"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/files"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/utils/time"
timeutil "github.com/iwind/TeaGo/utils/time"
"log"
"os"
"runtime"
@@ -13,27 +15,55 @@ import (
)
type LogWriter struct {
fileAppender *files.Appender
fp *os.File
c chan string
}
func (this *LogWriter) Init() {
// 创建目录
dir := files.NewFile(Tea.LogDir())
var dir = files.NewFile(Tea.LogDir())
if !dir.Exists() {
err := dir.Mkdir()
if err != nil {
log.Println("[error]" + err.Error())
log.Println("[LOG]create log dir failed: " + err.Error())
}
}
logFile := files.NewFile(Tea.LogFile("run.log"))
// 打开要写入的日志文件
appender, err := logFile.Appender()
var logPath = Tea.LogFile("run.log")
fp, err := os.OpenFile(logPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
logs.Error(err)
log.Println("[LOG]open log file failed: " + err.Error())
} else {
this.fileAppender = appender
this.fp = fp
}
this.c = make(chan string, 1024)
// 异步写入文件
var maxFileSize = 2 * sizes.G // 文件最大尺寸,超出此尺寸则清空
if fp != nil {
goman.New(func() {
var totalSize int64 = 0
stat, err := fp.Stat()
if err == nil {
totalSize = stat.Size()
}
for message := range this.c {
totalSize += int64(len(message))
_, err := fp.WriteString(timeutil.Format("Y/m/d H:i:s ") + message + "\n")
if err != nil {
log.Println("[LOG]write log failed: " + err.Error())
} else {
// 如果太大则Truncate
if totalSize > maxFileSize {
_ = fp.Truncate(0)
totalSize = 0
}
}
}
})
}
}
@@ -48,7 +78,7 @@ func (this *LogWriter) Write(message string) {
var ok bool
_, file, line, ok = runtime.Caller(callDepth)
if ok {
file = this.packagePath(file)
file = utils.RemoveWorkspace(this.packagePath(file))
}
}
@@ -59,18 +89,15 @@ func (this *LogWriter) Write(message string) {
}
}
if this.fileAppender != nil {
_, err := this.fileAppender.AppendString(timeutil.Format("Y/m/d H:i:s ") + message + "\n")
if err != nil {
log.Println("[error]" + err.Error())
}
}
this.c <- message
}
func (this *LogWriter) Close() {
if this.fileAppender != nil {
_ = this.fileAppender.Close()
if this.fp != nil {
_ = this.fp.Close()
}
close(this.c)
}
func (this *LogWriter) packagePath(path string) string {

View File

@@ -12,6 +12,7 @@ var (
ErrEntityTooLarge = errors.New("entity too large")
ErrWritingUnavailable = errors.New("writing unavailable")
ErrWritingQueueFull = errors.New("writing queue full")
ErrTooManyOpenFiles = errors.New("too many open files")
)
// CapacityError 容量错误
@@ -36,7 +37,8 @@ func CanIgnoreErr(err error) bool {
if err == ErrFileIsWriting ||
err == ErrEntityTooLarge ||
err == ErrWritingUnavailable ||
err == ErrWritingQueueFull {
err == ErrWritingQueueFull ||
err == ErrTooManyOpenFiles {
return true
}
_, ok := err.(*CapacityError)

View File

@@ -58,6 +58,7 @@ func (this *FileListDB) Open(dbPath string) error {
writeDB.SetMaxOpenConns(1)
// TODO 耗时过长,暂时不整理数据库
// TODO 需要根据行数来判断是否VACUUM
/**_, err = db.Exec("VACUUM")
if err != nil {
return err
@@ -109,7 +110,7 @@ func (this *FileListDB) Init() error {
this.total = total
// 常用语句
this.existsByHashStmt, err = this.readDB.Prepare(`SELECT "expiredAt" FROM "` + this.itemsTableName + `" WHERE "hash"=? AND expiredAt>? LIMIT 1`)
this.existsByHashStmt, err = this.readDB.Prepare(`SELECT "expiredAt" FROM "` + this.itemsTableName + `" INDEXED BY "hash" WHERE "hash"=? AND expiredAt>? LIMIT 1`)
if err != nil {
return err
}
@@ -354,7 +355,7 @@ func (this *FileListDB) Close() error {
func (this *FileListDB) initTables(times int) error {
{
// expiredAt - 过期时间,用来判断有无过期
// staleAt - 陈旧最大时间,用来清理缓存
// staleAt - 过时缓存最大时间,用来清理缓存
_, err := this.writeDB.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
@@ -370,15 +371,9 @@ func (this *FileListDB) initTables(times int) error {
"serverId" integer
);
CREATE INDEX IF NOT EXISTS "createdAt"
ON "` + this.itemsTableName + `" (
"createdAt" ASC
);
CREATE INDEX IF NOT EXISTS "expiredAt"
ON "` + this.itemsTableName + `" (
"expiredAt" ASC
);
DROP INDEX IF EXISTS "createdAt";
DROP INDEX IF EXISTS "expiredAt";
DROP INDEX IF EXISTS "serverId";
CREATE INDEX IF NOT EXISTS "staleAt"
ON "` + this.itemsTableName + `" (
@@ -389,11 +384,6 @@ CREATE UNIQUE INDEX IF NOT EXISTS "hash"
ON "` + this.itemsTableName + `" (
"hash" ASC
);
CREATE INDEX IF NOT EXISTS "serverId"
ON "` + this.itemsTableName + `" (
"serverId" ASC
);
`)
if err != nil {

View File

@@ -61,7 +61,7 @@ func (this *Manager) UpdatePolicies(newPolicies []*serverconfigs.HTTPCachePolicy
// 停止旧有的
for _, oldPolicy := range this.policyMap {
if !lists.ContainsInt64(newPolicyIds, oldPolicy.Id) {
remotelogs.Error("CACHE", "remove policy "+strconv.FormatInt(oldPolicy.Id, 10))
remotelogs.Println("CACHE", "remove policy "+strconv.FormatInt(oldPolicy.Id, 10))
delete(this.policyMap, oldPolicy.Id)
storage, ok := this.storageMap[oldPolicy.Id]
if ok {
@@ -214,3 +214,15 @@ func (this *Manager) FindAllCachePaths() []string {
}
return result
}
// FindAllStorages 读取所有缓存存储
func (this *Manager) FindAllStorages() []StorageInterface {
this.locker.Lock()
defer this.locker.Unlock()
var storages = []StorageInterface{}
for _, storage := range this.storageMap {
storages = append(storages, storage)
}
return storages
}

View File

@@ -0,0 +1,98 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package caches
import (
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"sync/atomic"
"time"
)
const (
minOpenFilesValue int32 = 4
maxOpenFilesValue int32 = 65535
modeSlow int32 = 1
modeFast int32 = 2
)
// MaxOpenFiles max open files manager
type MaxOpenFiles struct {
step int32
maxOpenFiles int32
ptr *int32
ticker *time.Ticker
mode int32
lastOpens int32
currentOpens int32
}
func NewMaxOpenFiles(step int32) *MaxOpenFiles {
if step <= 0 {
step = 2
}
var f = &MaxOpenFiles{
step: step,
maxOpenFiles: minOpenFilesValue,
}
if teaconst.DiskIsFast {
f.maxOpenFiles = 32
}
f.ptr = &f.maxOpenFiles
f.ticker = time.NewTicker(1 * time.Second)
f.init()
return f
}
func (this *MaxOpenFiles) init() {
goman.New(func() {
for range this.ticker.C {
var mod = atomic.LoadInt32(&this.mode)
switch mod {
case modeSlow:
// we decrease more quickly, with more steps
if atomic.AddInt32(this.ptr, -this.step*2) <= 0 {
atomic.StoreInt32(this.ptr, minOpenFilesValue)
}
case modeFast:
// we increase only when file opens increases
var currentOpens = atomic.LoadInt32(&this.currentOpens)
if currentOpens > this.lastOpens {
if atomic.AddInt32(this.ptr, this.step) >= maxOpenFilesValue {
atomic.StoreInt32(this.ptr, maxOpenFilesValue)
}
}
this.lastOpens = currentOpens
atomic.StoreInt32(&this.currentOpens, 0)
}
// reset mode
atomic.StoreInt32(&this.mode, 0)
}
})
}
func (this *MaxOpenFiles) Fast() {
if atomic.LoadInt32(&this.mode) == 0 {
this.mode = modeFast
}
atomic.AddInt32(&this.currentOpens, 1)
}
func (this *MaxOpenFiles) Slow() {
atomic.StoreInt32(&this.mode, modeSlow)
}
func (this *MaxOpenFiles) Max() int32 {
if atomic.LoadInt32(&this.mode) == modeSlow {
return 0
}
var v = atomic.LoadInt32(this.ptr)
if v <= minOpenFilesValue {
return minOpenFilesValue
}
return v
}

View File

@@ -0,0 +1,28 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package caches_test
import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
"testing"
"time"
)
func TestNewMaxOpenFiles(t *testing.T) {
var maxOpenFiles = caches.NewMaxOpenFiles(2)
maxOpenFiles.Fast()
t.Log(maxOpenFiles.Max())
maxOpenFiles.Fast()
time.Sleep(1 * time.Second)
t.Log(maxOpenFiles.Max())
maxOpenFiles.Slow()
t.Log(maxOpenFiles.Max())
maxOpenFiles.Slow()
t.Log(maxOpenFiles.Max())
maxOpenFiles.Slow()
t.Log(maxOpenFiles.Max())
}

View File

@@ -14,11 +14,12 @@ type OpenFile struct {
version int64
}
func NewOpenFile(fp *os.File, meta []byte, header []byte) *OpenFile {
func NewOpenFile(fp *os.File, meta []byte, header []byte, version int64) *OpenFile {
return &OpenFile{
fp: fp,
meta: meta,
header: header,
fp: fp,
meta: meta,
header: header,
version: version,
}
}

View File

@@ -9,6 +9,7 @@ import (
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/types"
"path/filepath"
"runtime"
"sync"
"time"
)
@@ -43,7 +44,7 @@ func NewOpenFileCache(maxSize int) (*OpenFileCache, error) {
goman.New(func() {
for event := range watcher.Events {
if event.Op&fsnotify.Chmod != fsnotify.Chmod {
if runtime.GOOS == "linux" || event.Op&fsnotify.Chmod != fsnotify.Chmod {
cache.Close(event.Name)
}
}
@@ -77,6 +78,7 @@ func (this *OpenFileCache) Put(filename string, file *OpenFile) {
} else {
_ = this.watcher.Add(filename)
pool = NewOpenFilePool(filename)
pool.version = file.version
this.poolMap[filename] = pool
success = pool.Put(file)
}

View File

@@ -45,7 +45,7 @@ func (this *OpenFilePool) Get() (*OpenFile, bool) {
}
func (this *OpenFilePool) Put(file *OpenFile) bool {
if file.version > 0 && file.version != this.version {
if this.version > 0 && file.version > 0 && file.version != this.version {
_ = file.Close()
return false
}

View File

@@ -11,7 +11,7 @@ func TestOpenFilePool_Get(t *testing.T) {
var pool = caches.NewOpenFilePool("a")
t.Log(pool.Filename())
t.Log(pool.Get())
t.Log(pool.Put(caches.NewOpenFile(nil, nil, []byte{})))
t.Log(pool.Put(caches.NewOpenFile(nil, nil, nil, 0)))
t.Log(pool.Get())
t.Log(pool.Get())
}

View File

@@ -67,17 +67,17 @@ func (this *FileReader) InitAutoDiscard(autoDiscard bool) error {
this.expiresAt = int64(binary.BigEndian.Uint32(buf[:SizeExpiresAt]))
status := types.Int(string(buf[SizeExpiresAt : SizeExpiresAt+SizeStatus]))
status := types.Int(string(buf[OffsetStatus : OffsetStatus+SizeStatus]))
if status < 100 || status > 999 {
return errors.New("invalid status")
}
this.status = status
// URL
urlLength := binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus : SizeExpiresAt+SizeStatus+SizeURLLength])
urlLength := binary.BigEndian.Uint32(buf[OffsetURLLength : OffsetURLLength+SizeURLLength])
// header
headerSize := int(binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus+SizeURLLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength]))
headerSize := int(binary.BigEndian.Uint32(buf[OffsetHeaderLength : OffsetHeaderLength+SizeHeaderLength]))
if headerSize == 0 {
return nil
}
@@ -86,7 +86,7 @@ func (this *FileReader) InitAutoDiscard(autoDiscard bool) error {
// body
this.bodyOffset = this.headerOffset + int64(headerSize)
bodySize := int(binary.BigEndian.Uint64(buf[SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength+SizeBodyLength]))
bodySize := int(binary.BigEndian.Uint64(buf[OffsetBodyLength : OffsetBodyLength+SizeBodyLength]))
if bodySize == 0 {
isOk = true
return nil
@@ -338,6 +338,11 @@ func (this *FileReader) ContainsRange(r rangeutils.Range) (r2 rangeutils.Range,
return r, true
}
// FP 原始的文件句柄
func (this *FileReader) FP() *os.File {
return this.fp
}
func (this *FileReader) Close() error {
if this.openFileCache != nil {
if this.isClosed {
@@ -348,7 +353,9 @@ func (this *FileReader) Close() error {
if this.openFile != nil {
this.openFileCache.Put(this.fp.Name(), this.openFile)
} else {
this.openFileCache.Put(this.fp.Name(), NewOpenFile(this.fp, this.meta, this.header))
var cacheMeta = make([]byte, len(this.meta))
copy(cacheMeta, this.meta)
this.openFileCache.Put(this.fp.Name(), NewOpenFile(this.fp, cacheMeta, this.header, this.LastModified()))
}
return nil
}
@@ -367,5 +374,12 @@ func (this *FileReader) readToBuff(fp *os.File, buf []byte) (ok bool, err error)
func (this *FileReader) discard() error {
_ = this.fp.Close()
this.isClosed = true
// close open file cache
if this.openFileCache != nil {
this.openFileCache.Close(this.fp.Name())
}
// remove file
return os.Remove(this.fp.Name())
}

View File

@@ -25,7 +25,7 @@ func (this *MemoryReader) TypeName() string {
}
func (this *MemoryReader) ExpiresAt() int64 {
return this.item.ExpiredAt
return this.item.ExpiresAt
}
func (this *MemoryReader) Status() int {

View File

@@ -4,7 +4,7 @@ import "testing"
func TestMemoryReader_Header(t *testing.T) {
item := &MemoryItem{
ExpiredAt: 0,
ExpiresAt: 0,
HeaderValue: []byte("0123456789"),
BodyValue: nil,
Status: 2000,
@@ -22,7 +22,7 @@ func TestMemoryReader_Header(t *testing.T) {
func TestMemoryReader_Body(t *testing.T) {
item := &MemoryItem{
ExpiredAt: 0,
ExpiresAt: 0,
HeaderValue: nil,
BodyValue: []byte("0123456789"),
Status: 2000,
@@ -40,7 +40,7 @@ func TestMemoryReader_Body(t *testing.T) {
func TestMemoryReader_Body_Range(t *testing.T) {
item := &MemoryItem{
ExpiredAt: 0,
ExpiresAt: 0,
HeaderValue: nil,
BodyValue: []byte("0123456789"),
Status: 2000,

View File

@@ -37,24 +37,36 @@ import (
)
const (
SizeExpiresAt = 4
SizeStatus = 3
SizeURLLength = 4
SizeHeaderLength = 4
SizeBodyLength = 8
SizeExpiresAt = 4
OffsetExpiresAt = 0
SizeStatus = 3
OffsetStatus = SizeExpiresAt
SizeURLLength = 4
OffsetURLLength = OffsetStatus + SizeStatus
SizeHeaderLength = 4
OffsetHeaderLength = OffsetURLLength + SizeURLLength
SizeBodyLength = 8
OffsetBodyLength = OffsetHeaderLength + SizeHeaderLength
SizeMeta = SizeExpiresAt + SizeStatus + SizeURLLength + SizeHeaderLength + SizeBodyLength
SizeMeta = SizeExpiresAt + SizeStatus + SizeURLLength + SizeHeaderLength + SizeBodyLength
OffsetKey = SizeMeta
)
const (
HotItemSize = 1024 // 热点数据数量
HotItemLifeSeconds int64 = 3600 // 热点数据生命周期
FileToMemoryMaxSize int64 = 32 * sizes.M // 可以从文件写入到内存的最大文件尺寸
FileStorageMaxIgnoreKeys = 32768 // 最大可忽略的键值数(尺寸过大的键值)
HotItemSize = 1024 // 热点数据数量
HotItemLifeSeconds int64 = 3600 // 热点数据生命周期
FileToMemoryMaxSize = 32 * sizes.M // 可以从文件写入到内存的最大文件尺寸
FileTmpSuffix = ".tmp"
)
var sharedWritingFileKeyMap = map[string]zero.Zero{} // key => bool
var sharedWritingFileKeyLocker = sync.Mutex{}
var maxOpenFiles = NewMaxOpenFiles(2)
const maxOpenFilesSlowCost = 500 * time.Microsecond // 0.5ms
// FileStorage 文件缓存
// 文件结构:
// [expires time] | [ status ] | [url length] | [header length] | [body length] | [url] [header data] [body data]
@@ -83,7 +95,7 @@ func NewFileStorage(policy *serverconfigs.HTTPCachePolicy) *FileStorage {
policy: policy,
hotMap: map[string]*HotItem{},
lastHotSize: -1,
ignoreKeys: setutils.NewFixedSet(32768),
ignoreKeys: setutils.NewFixedSet(FileStorageMaxIgnoreKeys),
}
}
@@ -244,14 +256,14 @@ func (this *FileStorage) Init() error {
var count = stat.Count
var size = stat.Size
cost := time.Since(before).Seconds() * 1000
var cost = time.Since(before).Seconds() * 1000
sizeMB := strconv.FormatInt(size, 10) + " Bytes"
if size > 1024*1024*1024 {
sizeMB = fmt.Sprintf("%.3f G", float64(size)/1024/1024/1024)
} else if size > 1024*1024 {
sizeMB = fmt.Sprintf("%.3f M", float64(size)/1024/1024)
} else if size > 1024 {
sizeMB = fmt.Sprintf("%.3f K", float64(size)/1024)
if size > 1*sizes.G {
sizeMB = fmt.Sprintf("%.3f G", float64(size)/float64(sizes.G))
} else if size > 1*sizes.M {
sizeMB = fmt.Sprintf("%.3f M", float64(size)/float64(sizes.M))
} else if size > 1*sizes.K {
sizeMB = fmt.Sprintf("%.3f K", float64(size)/float64(sizes.K))
}
remotelogs.Println("CACHE", "init policy "+strconv.FormatInt(this.policy.Id, 10)+" from '"+this.options.Dir+"', cost: "+fmt.Sprintf("%.2f", cost)+" ms, count: "+message.NewPrinter(language.English).Sprintf("%d", count)+", size: "+sizeMB)
}()
@@ -368,7 +380,16 @@ func (this *FileStorage) openReader(key string, allowMemory bool, useStale bool,
}
// OpenWriter 打开缓存文件等待写入
func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error) {
func (this *FileStorage) OpenWriter(key string, expiresAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error) {
return this.openWriter(key, expiresAt, status, size, maxSize, isPartial, false)
}
// OpenFlushWriter 打开从其他媒介直接刷入的写入器
func (this *FileStorage) OpenFlushWriter(key string, expiresAt int64, status int) (Writer, error) {
return this.openWriter(key, expiresAt, status, -1, -1, false, true)
}
func (this *FileStorage) openWriter(key string, expiredAt int64, status int, size int64, maxSize int64, isPartial bool, isFlushing bool) (Writer, error) {
// 是否正在退出
if teaconst.IsQuiting {
return nil, ErrWritingUnavailable
@@ -386,7 +407,7 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int, siz
maxMemorySize = maxSize
}
var memoryStorage = this.memoryStorage
if !isPartial && memoryStorage != nil && ((size > 0 && size < maxMemorySize) || size < 0) {
if !isFlushing && !isPartial && memoryStorage != nil && ((size > 0 && size < maxMemorySize) || size < 0) {
writer, err := memoryStorage.OpenWriter(key, expiredAt, status, size, maxMemorySize, false)
if err == nil {
return writer, nil
@@ -406,6 +427,12 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int, siz
sharedWritingFileKeyLocker.Unlock()
return nil, ErrFileIsWriting
}
if !isFlushing && len(sharedWritingFileKeyMap) >= int(maxOpenFiles.Max()) {
sharedWritingFileKeyLocker.Unlock()
return nil, ErrTooManyOpenFiles
}
sharedWritingFileKeyMap[key] = zero.New()
sharedWritingFileKeyLocker.Unlock()
defer func() {
@@ -430,6 +457,8 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int, siz
}
var hash = stringutil.Md5(key)
// TODO 可以只stat一次
var dir = this.options.Dir + "/p" + strconv.FormatInt(this.policy.Id, 10) + "/" + hash[:2] + "/" + hash[2:4]
_, err = os.Stat(dir)
if err != nil {
@@ -445,12 +474,32 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int, siz
// 检查缓存是否已经生成
var cachePathName = dir + "/" + hash
var cachePath = cachePathName + ".cache"
// 关闭OpenFileCache
var openFileCache = this.openFileCache
if openFileCache != nil {
openFileCache.Close(cachePath)
}
// 查询当前已有缓存文件
stat, err := os.Stat(cachePath)
if err == nil && time.Now().Sub(stat.ModTime()) <= 1*time.Second {
// 检查两次写入缓存的时间是否过于相近,分片内容不受此限制
if err == nil && !isPartial && time.Now().Sub(stat.ModTime()) <= 1*time.Second {
// 防止并发连续写入
return nil, ErrFileIsWriting
}
var tmpPath = cachePath + ".tmp"
// 构造文件名
var tmpPath = cachePath
var existsFile = false
if stat != nil {
existsFile = true
// 如果已经存在,则增加一个.tmp后缀防止读写冲突
tmpPath += FileTmpSuffix
}
if isPartial {
tmpPath = cachePathName + ".cache"
}
@@ -481,10 +530,22 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int, siz
}
}
writer, err := os.OpenFile(tmpPath, os.O_CREATE|os.O_WRONLY, 0666)
var flags = os.O_CREATE | os.O_WRONLY
if isNewCreated && existsFile {
flags |= os.O_TRUNC
}
var before = time.Now()
writer, err := os.OpenFile(tmpPath, flags, 0666)
if err != nil {
return nil, err
}
if !isFlushing {
if time.Since(before) >= maxOpenFilesSlowCost {
maxOpenFiles.Slow()
} else {
maxOpenFiles.Fast()
}
}
var removeOnFailure = true
defer func() {
@@ -509,60 +570,29 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int, siz
}
if isNewCreated {
err = writer.Truncate(0)
if err != nil {
return nil, err
}
// 写入过期时间
bytes4 := make([]byte, 4)
{
binary.BigEndian.PutUint32(bytes4, uint32(expiredAt))
_, err = writer.Write(bytes4)
if err != nil {
return nil, err
}
}
var metaBytes = make([]byte, SizeMeta+len(key))
binary.BigEndian.PutUint32(metaBytes[OffsetExpiresAt:], uint32(expiredAt))
// 写入状态码
if status > 999 || status < 100 {
status = 200
}
_, err = writer.WriteString(strconv.Itoa(status))
if err != nil {
return nil, err
}
copy(metaBytes[OffsetStatus:], strconv.Itoa(status))
// 写入URL长度
{
binary.BigEndian.PutUint32(bytes4, uint32(len(key)))
_, err = writer.Write(bytes4)
if err != nil {
return nil, err
}
}
binary.BigEndian.PutUint32(metaBytes[OffsetURLLength:], uint32(len(key)))
// 写入Header Length
{
binary.BigEndian.PutUint32(bytes4, uint32(0))
_, err = writer.Write(bytes4)
if err != nil {
return nil, err
}
}
binary.BigEndian.PutUint32(metaBytes[OffsetHeaderLength:], uint32(0))
// 写入Body Length
{
b := make([]byte, SizeBodyLength)
binary.BigEndian.PutUint64(b, uint64(0))
_, err = writer.Write(b)
if err != nil {
return nil, err
}
}
binary.BigEndian.PutUint64(metaBytes[OffsetBodyLength:], uint64(0))
// 写入URL
_, err = writer.WriteString(key)
copy(metaBytes[OffsetKey:], key)
_, err = writer.Write(metaBytes)
if err != nil {
return nil, err
}
@@ -744,9 +774,10 @@ func (this *FileStorage) Purge(keys []string, urlType string) error {
return err
}
}
return nil
}
// 文件
// URL
for _, key := range keys {
hash, path := this.keyPath(key)
err := this.removeCacheFile(path)
@@ -784,8 +815,9 @@ func (this *FileStorage) Stop() {
_ = this.list.Close()
if this.openFileCache != nil {
this.openFileCache.CloseAll()
var openFileCache = this.openFileCache
if openFileCache != nil {
openFileCache.CloseAll()
}
this.ignoreKeys.Reset()
@@ -810,6 +842,14 @@ func (this *FileStorage) IgnoreKey(key string) {
this.ignoreKeys.Push(key)
}
// CanSendfile 是否支持Sendfile
func (this *FileStorage) CanSendfile() bool {
if this.options == nil {
return false
}
return this.options.EnableSendfile
}
// 绝对路径
func (this *FileStorage) dir() string {
return this.options.Dir + "/p" + strconv.FormatInt(this.policy.Id, 10) + "/"
@@ -997,7 +1037,7 @@ func (this *FileStorage) hotLoop() {
this.hotMap = map[string]*HotItem{}
this.hotMapLocker.Unlock()
// 取Top10写入内存
// 取Top10%写入内存
if len(result) > 0 {
sort.Slice(result, func(i, j int) bool {
return result[i].Hits > result[j].Hits
@@ -1151,7 +1191,7 @@ func (this *FileStorage) increaseHit(key string, hash string, reader Reader) {
// 增加到热点
// 这里不收录缓存尺寸过大的文件
if memoryStorage != nil && reader.BodySize() > 0 && reader.BodySize() < 128*1024*1024 {
if memoryStorage != nil && reader.BodySize() > 0 && reader.BodySize() < 128*sizes.M {
this.hotMapLocker.Lock()
hotItem, ok := this.hotMap[key]
@@ -1177,12 +1217,21 @@ func (this *FileStorage) increaseHit(key string, hash string, reader Reader) {
// 删除缓存文件
func (this *FileStorage) removeCacheFile(path string) error {
var openFileCache = this.openFileCache
if openFileCache != nil {
openFileCache.Close(path)
}
var err = os.Remove(path)
if err == nil || os.IsNotExist(err) {
err = nil
// 删除Partial相关
_ = os.Remove(partialRangesFilePath(path))
var partialPath = partialRangesFilePath(path)
if openFileCache != nil {
openFileCache.Close(partialPath)
}
_ = os.Remove(partialPath)
}
return err
}

View File

@@ -14,7 +14,10 @@ type StorageInterface interface {
// OpenWriter 打开缓存写入器等待写入
// size 和 maxSize 可能为-1
OpenWriter(key string, expiredAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error)
OpenWriter(key string, expiresAt int64, status int, size int64, maxSize int64, isPartial bool) (Writer, error)
// OpenFlushWriter 打开从其他媒介直接刷入的写入器
OpenFlushWriter(key string, expiresAt int64, status int) (Writer, error)
// Delete 删除某个键值对应的缓存
Delete(key string) error
@@ -32,6 +35,7 @@ type StorageInterface interface {
CleanAll() error
// Purge 批量删除缓存
// urlType 值为file|dir
Purge(keys []string, urlType string) error
// Stop 停止缓存策略
@@ -51,4 +55,7 @@ type StorageInterface interface {
// IgnoreKey 忽略某个Key即不缓存某个Key
IgnoreKey(key string)
// CanSendfile 是否支持Sendfile
CanSendfile() bool
}

View File

@@ -23,7 +23,7 @@ import (
)
type MemoryItem struct {
ExpiredAt int64
ExpiresAt int64
HeaderValue []byte
BodyValue []byte
Status int
@@ -32,7 +32,7 @@ type MemoryItem struct {
}
func (this *MemoryItem) IsExpired() bool {
return this.ExpiredAt < utils.UnixTime()
return this.ExpiresAt < utils.UnixTime()
}
type MemoryStorage struct {
@@ -118,7 +118,7 @@ func (this *MemoryStorage) OpenReader(key string, useStale bool, isPartial bool)
return nil, ErrNotFound
}
if useStale || (item.ExpiredAt > utils.UnixTime()) {
if useStale || (item.ExpiresAt > utils.UnixTime()) {
reader := NewMemoryReader(item)
err := reader.Init()
if err != nil {
@@ -160,7 +160,12 @@ func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int, s
return this.openWriter(key, expiredAt, status, size, maxSize, true)
}
func (this *MemoryStorage) openWriter(key string, expiredAt int64, status int, size int64, maxSize int64, isDirty bool) (Writer, error) {
// OpenFlushWriter 打开从其他媒介直接刷入的写入器
func (this *MemoryStorage) OpenFlushWriter(key string, expiresAt int64, status int) (Writer, error) {
return this.openWriter(key, expiresAt, status, -1, -1, true)
}
func (this *MemoryStorage) openWriter(key string, expiresAt int64, status int, size int64, maxSize int64, isDirty bool) (Writer, error) {
// 待写入队列是否已满
if isDirty &&
this.parentStorage != nil &&
@@ -215,7 +220,7 @@ func (this *MemoryStorage) openWriter(key string, expiredAt int64, status int, s
}
isWriting = true
return NewMemoryWriter(this, key, expiredAt, status, isDirty, maxSize, func() {
return NewMemoryWriter(this, key, expiresAt, status, isDirty, maxSize, func() {
this.locker.Lock()
delete(this.writingKeyMap, key)
this.locker.Unlock()
@@ -262,8 +267,10 @@ func (this *MemoryStorage) Purge(keys []string, urlType string) error {
return err
}
}
return nil
}
// URL
for _, key := range keys {
err := this.Delete(key)
if err != nil {
@@ -347,6 +354,11 @@ func (this *MemoryStorage) IgnoreKey(key string) {
this.ignoreKeys.Push(key)
}
// CanSendfile 是否支持Sendfile
func (this *MemoryStorage) CanSendfile() bool {
return false
}
// 计算Key Hash
func (this *MemoryStorage) hash(key string) uint64 {
return xxhash.Sum64String(key)
@@ -466,7 +478,7 @@ func (this *MemoryStorage) flushItem(key string) {
return
}
writer, err := this.parentStorage.OpenWriter(key, item.ExpiredAt, item.Status, -1, -1, false)
writer, err := this.parentStorage.OpenFlushWriter(key, item.ExpiresAt, item.Status)
if err != nil {
if !CanIgnoreErr(err) {
remotelogs.Error("CACHE", "flush items failed: open writer failed: "+err.Error())
@@ -498,7 +510,7 @@ func (this *MemoryStorage) flushItem(key string) {
this.parentStorage.AddToList(&Item{
Type: writer.ItemType(),
Key: key,
ExpiredAt: item.ExpiredAt,
ExpiredAt: item.ExpiresAt,
HeaderSize: writer.HeaderSize(),
BodySize: writer.BodySize(),
})

View File

@@ -127,8 +127,8 @@ func (this *FileWriter) Close() error {
err = this.rawWriter.Close()
if err != nil {
_ = os.Remove(path)
} else {
err = os.Rename(path, strings.Replace(path, ".tmp", "", 1))
} else if strings.HasSuffix(path, FileTmpSuffix) {
err = os.Rename(path, strings.Replace(path, FileTmpSuffix, "", 1))
if err != nil {
_ = os.Remove(path)
}

View File

@@ -30,7 +30,7 @@ func NewMemoryWriter(memoryStorage *MemoryStorage, key string, expiredAt int64,
key: key,
expiredAt: expiredAt,
item: &MemoryItem{
ExpiredAt: expiredAt,
ExpiresAt: expiredAt,
ModifiedAt: time.Now().Unix(),
Status: status,
},

View File

@@ -3,7 +3,7 @@
package compressions
import (
"compress/gzip"
"github.com/klauspost/compress/gzip"
"io"
)

View File

@@ -3,7 +3,7 @@
package compressions
import (
"compress/gzip"
"github.com/klauspost/compress/gzip"
"io"
)

View File

@@ -34,3 +34,31 @@ func BenchmarkGzipWriter_Write(b *testing.B) {
_ = writer.Close()
}
}
func BenchmarkGzipWriter_Write_Parallel(b *testing.B) {
var data = []byte(strings.Repeat("A", 1024))
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
var buf = &bytes.Buffer{}
writer, err := compressions.NewGzipWriter(buf, 5)
if err != nil {
b.Fatal(err)
}
for j := 0; j < 100; j++ {
_, err = writer.Write(data)
if err != nil {
b.Fatal(err)
}
/**err = writer.Flush()
if err != nil {
b.Fatal(err)
}**/
}
_ = writer.Close()
}
})
}

View File

@@ -1,6 +1,6 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build community
// +build community
//go:build !plus
// +build !plus
package teaconst

View File

@@ -1,7 +1,7 @@
package teaconst
const (
Version = "0.4.5"
Version = "0.4.8.1"
ProductName = "Edge Node"
ProcessName = "edge-node"
@@ -13,4 +13,6 @@ const (
// SystemdServiceName systemd
SystemdServiceName = "edge-node"
AccessLogSockName = "edge-node.accesslog.sock"
)

View File

@@ -17,4 +17,6 @@ var (
IsQuiting = false // 是否正在退出
EnableDBStat = false // 是否开启本地数据库统计
DiskIsFast = false // 是否为高速硬盘
)

View File

@@ -1,12 +1,12 @@
package encrypt
type MethodInterface interface {
// 初始化
// Init 初始化
Init(key []byte, iv []byte) error
// 加密
// Encrypt 加密
Encrypt(src []byte) (dst []byte, err error)
// 解密
// Decrypt 解密
Decrypt(dst []byte) (src []byte, err error)
}

View File

@@ -2,6 +2,7 @@ package errors
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"path/filepath"
"runtime"
"strconv"
@@ -15,7 +16,7 @@ type errorObj struct {
}
func (this *errorObj) Error() string {
s := this.err.Error() + "\n " + this.file
s := this.err.Error() + "\n " + utils.RemoveWorkspace(this.file)
if len(this.funcName) > 0 {
s += ":" + this.funcName + "()"
}
@@ -23,7 +24,7 @@ func (this *errorObj) Error() string {
return s
}
// 新错误
// New 新错误
func New(errText string) error {
ptr, file, line, ok := runtime.Caller(1)
funcName := ""
@@ -39,7 +40,7 @@ func New(errText string) error {
}
}
// 包装已有错误
// Wrap 包装已有错误
func Wrap(err error) error {
if err == nil {
return nil

View File

@@ -3,9 +3,10 @@ package events
type Event = string
const (
EventStart Event = "start" // start loading
EventLoaded Event = "loaded" // first load
EventQuit Event = "quit" // quit node gracefully
EventReload Event = "reload" // reload config
EventTerminated Event = "terminated" // process terminated
EventStart Event = "start" // start loading
EventLoaded Event = "loaded" // first load
EventQuit Event = "quit" // quit node gracefully
EventReload Event = "reload" // reload config
EventTerminated Event = "terminated" // process terminated
EventNFTablesReady Event = "nftablesReady" // nftables ready
)

View File

@@ -0,0 +1,502 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package firewalls
import (
"bytes"
"encoding/json"
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/ddosconfigs"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
stringutil "github.com/iwind/TeaGo/utils/string"
"net"
"os/exec"
"strings"
)
var SharedDDoSProtectionManager = NewDDoSProtectionManager()
func init() {
events.On(events.EventReload, func() {
if nftablesInstance == nil {
return
}
nodeConfig, _ := nodeconfigs.SharedNodeConfig()
if nodeConfig != nil {
err := SharedDDoSProtectionManager.Apply(nodeConfig.DDOSProtection)
if err != nil {
remotelogs.Error("FIREWALL", "apply DDoS protection failed: "+err.Error())
}
}
})
events.On(events.EventNFTablesReady, func() {
nodeConfig, _ := nodeconfigs.SharedNodeConfig()
if nodeConfig != nil {
err := SharedDDoSProtectionManager.Apply(nodeConfig.DDOSProtection)
if err != nil {
remotelogs.Error("FIREWALL", "apply DDoS protection failed: "+err.Error())
}
}
})
}
// DDoSProtectionManager DDoS防护
type DDoSProtectionManager struct {
nftPath string
lastAllowIPList []string
lastConfig []byte
}
// NewDDoSProtectionManager 获取新对象
func NewDDoSProtectionManager() *DDoSProtectionManager {
nftPath, _ := exec.LookPath("nft")
return &DDoSProtectionManager{
nftPath: nftPath,
}
}
// Apply 应用配置
func (this *DDoSProtectionManager) Apply(config *ddosconfigs.ProtectionConfig) error {
// 同集群节点IP白名单
var allowIPListChanged = false
nodeConfig, _ := nodeconfigs.SharedNodeConfig()
if nodeConfig != nil {
var allowIPList = nodeConfig.AllowedIPs
if !utils.ContainsSameStrings(allowIPList, this.lastAllowIPList) {
allowIPListChanged = true
this.lastAllowIPList = allowIPList
}
}
// 对比配置
configJSON, err := json.Marshal(config)
if err != nil {
return errors.New("encode config to json failed: " + err.Error())
}
if !allowIPListChanged && bytes.Equal(this.lastConfig, configJSON) {
return nil
}
remotelogs.Println("FIREWALL", "change DDoS protection config")
if len(this.nftPath) == 0 {
return errors.New("can not find nft command")
}
if nftablesInstance == nil {
return errors.New("nftables instance should not be nil")
}
if config == nil {
// TCP
err := this.removeTCPRules()
if err != nil {
return err
}
// TODO other protocols
return nil
}
// TCP
if config.TCP == nil {
err := this.removeTCPRules()
if err != nil {
return err
}
} else {
// allow ip list
var allowIPList = []string{}
for _, ipConfig := range config.TCP.AllowIPList {
allowIPList = append(allowIPList, ipConfig.IP)
}
for _, ip := range this.lastAllowIPList {
if !lists.ContainsString(allowIPList, ip) {
allowIPList = append(allowIPList, ip)
}
}
err = this.updateAllowIPList(allowIPList)
if err != nil {
return err
}
// tcp
if config.TCP.IsOn {
err := this.addTCPRules(config.TCP)
if err != nil {
return err
}
} else {
err := this.removeTCPRules()
if err != nil {
return err
}
}
}
this.lastConfig = configJSON
return nil
}
// 添加TCP规则
func (this *DDoSProtectionManager) addTCPRules(tcpConfig *ddosconfigs.TCPConfig) error {
// 检查nft版本不能小于0.9
if len(nftablesInstance.version) > 0 && stringutil.VersionCompare("0.9", nftablesInstance.version) > 0 {
return nil
}
var ports = []int32{}
for _, portConfig := range tcpConfig.Ports {
if !lists.ContainsInt32(ports, portConfig.Port) {
ports = append(ports, portConfig.Port)
}
}
if len(ports) == 0 {
ports = []int32{80, 443}
}
for _, filter := range nftablesFilters {
chain, oldRules, err := this.getRules(filter)
if err != nil {
return errors.New("get old rules failed: " + err.Error())
}
var protocol = filter.protocol()
// max connections
var maxConnections = tcpConfig.MaxConnections
if maxConnections <= 0 {
maxConnections = nodeconfigs.DefaultTCPMaxConnections
if maxConnections <= 0 {
maxConnections = 100000
}
}
// max connections per ip
var maxConnectionsPerIP = tcpConfig.MaxConnectionsPerIP
if maxConnectionsPerIP <= 0 {
maxConnectionsPerIP = nodeconfigs.DefaultTCPMaxConnectionsPerIP
if maxConnectionsPerIP <= 0 {
maxConnectionsPerIP = 100000
}
}
// new connections rate
var newConnectionsRate = tcpConfig.NewConnectionsRate
if newConnectionsRate <= 0 {
newConnectionsRate = nodeconfigs.DefaultTCPNewConnectionsRate
if newConnectionsRate <= 0 {
newConnectionsRate = 100000
}
}
// 检查是否有变化
var hasChanges = false
for _, port := range ports {
if !this.existsRule(oldRules, []string{"tcp", types.String(port), "maxConnections", types.String(maxConnections)}) {
hasChanges = true
break
}
if !this.existsRule(oldRules, []string{"tcp", types.String(port), "maxConnectionsPerIP", types.String(maxConnectionsPerIP)}) {
hasChanges = true
break
}
if !this.existsRule(oldRules, []string{"tcp", types.String(port), "newConnectionsRate", types.String(newConnectionsRate)}) {
hasChanges = true
break
}
}
if !hasChanges {
// 检查是否有多余的端口
var oldPorts = this.getTCPPorts(oldRules)
if !this.eqPorts(ports, oldPorts) {
hasChanges = true
}
}
if !hasChanges {
return nil
}
// 先清空所有相关规则
err = this.removeOldTCPRules(chain, oldRules)
if err != nil {
return errors.New("delete old rules failed: " + err.Error())
}
// 添加新规则
for _, port := range ports {
if maxConnections > 0 {
var cmd = exec.Command(this.nftPath, "add", "rule", protocol, filter.Name, nftablesChainName, "tcp", "dport", types.String(port), "ct", "count", "over", types.String(maxConnections), "counter", "drop", "comment", this.encodeUserData([]string{"tcp", types.String(port), "maxConnections", types.String(maxConnections)}))
var stderr = &bytes.Buffer{}
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
return errors.New("add nftables rule '" + cmd.String() + "' failed: " + err.Error() + " (" + stderr.String() + ")")
}
}
if maxConnectionsPerIP > 0 {
var cmd = exec.Command(this.nftPath, "add", "rule", protocol, filter.Name, nftablesChainName, "tcp", "dport", types.String(port), "meter", "meter-"+protocol+"-"+types.String(port)+"-max-connections", "{ "+protocol+" saddr ct count over "+types.String(maxConnectionsPerIP)+" }", "counter", "drop", "comment", this.encodeUserData([]string{"tcp", types.String(port), "maxConnectionsPerIP", types.String(maxConnectionsPerIP)}))
var stderr = &bytes.Buffer{}
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
return errors.New("add nftables rule '" + cmd.String() + "' failed: " + err.Error() + " (" + stderr.String() + ")")
}
}
if newConnectionsRate > 0 {
// TODO 思考是否有惩罚机制
var cmd = exec.Command(this.nftPath, "add", "rule", protocol, filter.Name, nftablesChainName, "tcp", "dport", types.String(port), "ct", "state", "new", "meter", "meter-"+protocol+"-"+types.String(port)+"-new-connections-rate", "{ "+protocol+" saddr limit rate over "+types.String(newConnectionsRate)+"/minute burst "+types.String(newConnectionsRate+3)+" packets }" /**"add", "@deny_set", "{"+protocol+" saddr}",**/, "counter", "drop", "comment", this.encodeUserData([]string{"tcp", types.String(port), "newConnectionsRate", types.String(newConnectionsRate)}))
var stderr = &bytes.Buffer{}
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
return errors.New("add nftables rule '" + cmd.String() + "' failed: " + err.Error() + " (" + stderr.String() + ")")
}
}
}
}
return nil
}
// 删除TCP规则
func (this *DDoSProtectionManager) removeTCPRules() error {
for _, filter := range nftablesFilters {
chain, rules, err := this.getRules(filter)
// TCP
err = this.removeOldTCPRules(chain, rules)
if err != nil {
return err
}
}
return nil
}
// 组合user data
// 数据中不能包含字母、数字、下划线以外的数据
func (this *DDoSProtectionManager) encodeUserData(attrs []string) string {
if attrs == nil {
return ""
}
return "ZZ" + strings.Join(attrs, "_") + "ZZ"
}
// 解码user data
func (this *DDoSProtectionManager) decodeUserData(data []byte) []string {
if len(data) == 0 {
return nil
}
var dataCopy = make([]byte, len(data))
copy(dataCopy, data)
var separatorLen = 2
var index1 = bytes.Index(dataCopy, []byte{'Z', 'Z'})
if index1 < 0 {
return nil
}
dataCopy = dataCopy[index1+separatorLen:]
var index2 = bytes.LastIndex(dataCopy, []byte{'Z', 'Z'})
if index2 < 0 {
return nil
}
var s = string(dataCopy[:index2])
var pieces = strings.Split(s, "_")
for index, piece := range pieces {
pieces[index] = strings.TrimSpace(piece)
}
return pieces
}
// 清除规则
func (this *DDoSProtectionManager) removeOldTCPRules(chain *nftables.Chain, rules []*nftables.Rule) error {
for _, rule := range rules {
var pieces = this.decodeUserData(rule.UserData())
if len(pieces) != 4 {
continue
}
if pieces[0] != "tcp" {
continue
}
switch pieces[2] {
case "maxConnections", "maxConnectionsPerIP", "newConnectionsRate":
err := chain.DeleteRule(rule)
if err != nil {
return err
}
}
}
return nil
}
// 根据参数检查规则是否存在
func (this *DDoSProtectionManager) existsRule(rules []*nftables.Rule, attrs []string) (exists bool) {
if len(attrs) == 0 {
return false
}
for _, oldRule := range rules {
var pieces = this.decodeUserData(oldRule.UserData())
if len(attrs) != len(pieces) {
continue
}
var isSame = true
for index, piece := range pieces {
if strings.TrimSpace(piece) != attrs[index] {
isSame = false
break
}
}
if isSame {
return true
}
}
return false
}
// 获取规则中的端口号
func (this *DDoSProtectionManager) getTCPPorts(rules []*nftables.Rule) []int32 {
var ports = []int32{}
for _, rule := range rules {
var pieces = this.decodeUserData(rule.UserData())
if len(pieces) != 4 {
continue
}
if pieces[0] != "tcp" {
continue
}
var port = types.Int32(pieces[1])
if port > 0 && !lists.ContainsInt32(ports, port) {
ports = append(ports, port)
}
}
return ports
}
// 检查端口是否一样
func (this *DDoSProtectionManager) eqPorts(ports1 []int32, ports2 []int32) bool {
if len(ports1) != len(ports2) {
return false
}
var portMap = map[int32]bool{}
for _, port := range ports2 {
portMap[port] = true
}
for _, port := range ports1 {
_, ok := portMap[port]
if !ok {
return false
}
}
return true
}
// 查找Table
func (this *DDoSProtectionManager) getTable(filter *nftablesTableDefinition) (*nftables.Table, error) {
var family nftables.TableFamily
if filter.IsIPv4 {
family = nftables.TableFamilyIPv4
} else if filter.IsIPv6 {
family = nftables.TableFamilyIPv6
} else {
return nil, errors.New("table '" + filter.Name + "' should be IPv4 or IPv6")
}
return nftablesInstance.conn.GetTable(filter.Name, family)
}
// 查找所有规则
func (this *DDoSProtectionManager) getRules(filter *nftablesTableDefinition) (*nftables.Chain, []*nftables.Rule, error) {
table, err := this.getTable(filter)
if err != nil {
return nil, nil, errors.New("get table failed: " + err.Error())
}
chain, err := table.GetChain(nftablesChainName)
if err != nil {
return nil, nil, errors.New("get chain failed: " + err.Error())
}
rules, err := chain.GetRules()
return chain, rules, err
}
// 更新白名单
func (this *DDoSProtectionManager) updateAllowIPList(allIPList []string) error {
if nftablesInstance == nil {
return nil
}
var allMap = map[string]zero.Zero{}
for _, ip := range allIPList {
allMap[ip] = zero.New()
}
for _, set := range []*nftables.Set{nftablesInstance.allowIPv4Set, nftablesInstance.allowIPv6Set} {
var isIPv4 = set == nftablesInstance.allowIPv4Set
var isIPv6 = !isIPv4
// 现有的
oldList, err := set.GetIPElements()
if err != nil {
return err
}
var oldMap = map[string]zero.Zero{} // ip=> zero
for _, ip := range oldList {
oldMap[ip] = zero.New()
if (utils.IsIPv4(ip) && isIPv4) || (utils.IsIPv6(ip) && isIPv6) {
_, ok := allMap[ip]
if !ok {
// 不存在则删除
err = set.DeleteIPElement(ip)
if err != nil {
return errors.New("delete ip element '" + ip + "' failed: " + err.Error())
}
}
}
}
// 新增的
for _, ip := range allIPList {
var ipObj = net.ParseIP(ip)
if ipObj == nil {
continue
}
if (utils.IsIPv4(ip) && isIPv4) || (utils.IsIPv6(ip) && isIPv6) {
_, ok := oldMap[ip]
if !ok {
// 不存在则添加
err = set.AddIPElement(ip, nil)
if err != nil {
return errors.New("add ip '" + ip + "' failed: " + err.Error())
}
}
}
}
}
return nil
}

View File

@@ -0,0 +1,23 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build !linux
// +build !linux
package firewalls
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/ddosconfigs"
)
var SharedDDoSProtectionManager = NewDDoSProtectionManager()
type DDoSProtectionManager struct {
nftPath string
}
func NewDDoSProtectionManager() *DDoSProtectionManager {
return &DDoSProtectionManager{}
}
func (this *DDoSProtectionManager) Apply(config *ddosconfigs.ProtectionConfig) error {
return nil
}

View File

@@ -5,17 +5,18 @@ package firewalls
import (
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"runtime"
"sync"
)
var currentFirewall FirewallInterface
var firewallLocker = &sync.Mutex{}
// 初始化
func init() {
events.On(events.EventLoaded, func() {
var firewall = Firewall()
if firewall.Name() == "mock" {
remotelogs.Warn("FIREWALL", "'firewalld' on this system should be enabled to block attackers more effectively")
} else {
if firewall.Name() != "mock" {
remotelogs.Println("FIREWALL", "found local firewall '"+firewall.Name()+"'")
}
})
@@ -23,12 +24,30 @@ func init() {
// Firewall 查找当前系统中最适合的防火墙
func Firewall() FirewallInterface {
firewallLocker.Lock()
defer firewallLocker.Unlock()
if currentFirewall != nil {
return currentFirewall
}
// nftables
if runtime.GOOS == "linux" {
nftables, err := NewNFTablesFirewall()
if err != nil {
remotelogs.Warn("FIREWALL", "'nftables' should be installed on the system to enhance security (init failed: "+err.Error()+")")
} else {
if nftables.IsReady() {
currentFirewall = nftables
events.Notify(events.EventNFTablesReady)
return nftables
} else {
remotelogs.Warn("FIREWALL", "'nftables' should be enabled on the system to enhance security")
}
}
}
// firewalld
{
if runtime.GOOS == "linux" {
var firewalld = NewFirewalld()
if firewalld.IsReady() {
currentFirewall = firewalld

View File

@@ -23,10 +23,13 @@ func NewFirewalld() *Firewalld {
path, err := exec.LookPath("firewall-cmd")
if err == nil && len(path) > 0 {
var cmd = exec.Command(path, "-V")
var cmd = exec.Command(path, "--state")
err := cmd.Run()
if err == nil {
firewalld.exe = path
// TODO check firewalld status with 'firewall-cmd --state' (running or not running),
// but we should recover the state when firewalld state changes, maybe check it every minutes
firewalld.isReady = true
firewalld.init()
}
@@ -58,6 +61,11 @@ func (this *Firewalld) IsReady() bool {
return this.isReady
}
// IsMock 是否为模拟
func (this *Firewalld) IsMock() bool {
return false
}
func (this *Firewalld) AllowPort(port int, protocol string) error {
if !this.isReady {
return nil

View File

@@ -10,6 +10,9 @@ type FirewallInterface interface {
// IsReady 是否已准备被调用
IsReady() bool
// IsMock 是否为模拟
IsMock() bool
// AllowPort 允许端口
AllowPort(port int, protocol string) error

View File

@@ -20,6 +20,11 @@ func (this *MockFirewall) IsReady() bool {
return true
}
// IsMock 是否为模拟
func (this *MockFirewall) IsMock() bool {
return true
}
// AllowPort 允许端口
func (this *MockFirewall) AllowPort(port int, protocol string) error {
_ = port

View File

@@ -0,0 +1,395 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package firewalls
import (
"bytes"
"errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/types"
"net"
"os/exec"
"regexp"
"runtime"
"strings"
"time"
)
// check nft status, if being enabled we load it automatically
func init() {
if runtime.GOOS == "linux" {
var ticker = time.NewTicker(3 * time.Minute)
go func() {
for range ticker.C {
// if already ready, we break
if nftablesIsReady {
ticker.Stop()
break
}
_, err := exec.LookPath("nft")
if err == nil {
nftablesFirewall, err := NewNFTablesFirewall()
if err != nil {
continue
}
currentFirewall = nftablesFirewall
remotelogs.Println("FIREWALL", "nftables is ready")
// fire event
if nftablesFirewall.IsReady() {
events.Notify(events.EventNFTablesReady)
}
ticker.Stop()
break
}
}
}()
}
}
var nftablesInstance *NFTablesFirewall
var nftablesIsReady = false
var nftablesFilters = []*nftablesTableDefinition{
// we shorten the name for table name length restriction
{Name: "edge_dft_v4", IsIPv4: true},
{Name: "edge_dft_v6", IsIPv6: true},
}
var nftablesChainName = "input"
type nftablesTableDefinition struct {
Name string
IsIPv4 bool
IsIPv6 bool
}
func (this *nftablesTableDefinition) protocol() string {
if this.IsIPv6 {
return "ip6"
}
return "ip"
}
func NewNFTablesFirewall() (*NFTablesFirewall, error) {
var firewall = &NFTablesFirewall{
conn: nftables.NewConn(),
}
err := firewall.init()
if err != nil {
return nil, err
}
return firewall, nil
}
type NFTablesFirewall struct {
conn *nftables.Conn
isReady bool
version string
allowIPv4Set *nftables.Set
allowIPv6Set *nftables.Set
denyIPv4Set *nftables.Set
denyIPv6Set *nftables.Set
firewalld *Firewalld
}
func (this *NFTablesFirewall) init() error {
// check nft
nftPath, err := exec.LookPath("nft")
if err != nil {
return errors.New("nft not found")
}
this.version = this.readVersion(nftPath)
// table
for _, tableDef := range nftablesFilters {
var family nftables.TableFamily
if tableDef.IsIPv4 {
family = nftables.TableFamilyIPv4
} else if tableDef.IsIPv6 {
family = nftables.TableFamilyIPv6
} else {
return errors.New("invalid table family: " + types.String(tableDef))
}
table, err := this.conn.GetTable(tableDef.Name, family)
if err != nil {
if nftables.IsNotFound(err) {
if tableDef.IsIPv4 {
table, err = this.conn.AddIPv4Table(tableDef.Name)
} else if tableDef.IsIPv6 {
table, err = this.conn.AddIPv6Table(tableDef.Name)
}
if err != nil {
return errors.New("create table '" + tableDef.Name + "' failed: " + err.Error())
}
} else {
return errors.New("get table '" + tableDef.Name + "' failed: " + err.Error())
}
}
if table == nil {
return errors.New("can not create table '" + tableDef.Name + "'")
}
// chain
var chainName = nftablesChainName
chain, err := table.GetChain(chainName)
if err != nil {
if nftables.IsNotFound(err) {
chain, err = table.AddAcceptChain(chainName)
if err != nil {
return errors.New("create chain '" + chainName + "' failed: " + err.Error())
}
} else {
return errors.New("get chain '" + chainName + "' failed: " + err.Error())
}
}
if chain == nil {
return errors.New("can not create chain '" + chainName + "'")
}
// allow lo
var loRuleName = []byte("lo")
_, err = chain.GetRuleWithUserData(loRuleName)
if err != nil {
if nftables.IsNotFound(err) {
_, err = chain.AddAcceptInterfaceRule("lo", loRuleName)
}
if err != nil {
return errors.New("add 'lo' rule failed: " + err.Error())
}
}
// allow set
// "allow" should be always first
for _, setAction := range []string{"allow", "deny"} {
var setName = setAction + "_set"
set, err := table.GetSet(setName)
if err != nil {
if nftables.IsNotFound(err) {
var keyType nftables.SetDataType
if tableDef.IsIPv4 {
keyType = nftables.TypeIPAddr
} else if tableDef.IsIPv6 {
keyType = nftables.TypeIP6Addr
}
set, err = table.AddSet(setName, &nftables.SetOptions{
KeyType: keyType,
HasTimeout: true,
})
if err != nil {
return errors.New("create set '" + setName + "' failed: " + err.Error())
}
} else {
return errors.New("get set '" + setName + "' failed: " + err.Error())
}
}
if set == nil {
return errors.New("can not create set '" + setName + "'")
}
if tableDef.IsIPv4 {
if setAction == "allow" {
this.allowIPv4Set = set
} else {
this.denyIPv4Set = set
}
} else if tableDef.IsIPv6 {
if setAction == "allow" {
this.allowIPv6Set = set
} else {
this.denyIPv6Set = set
}
}
// rule
var ruleName = []byte(setAction)
rule, err := chain.GetRuleWithUserData(ruleName)
if err != nil {
if nftables.IsNotFound(err) {
if tableDef.IsIPv4 {
if setAction == "allow" {
rule, err = chain.AddAcceptIPv4SetRule(setName, ruleName)
} else {
rule, err = chain.AddDropIPv4SetRule(setName, ruleName)
}
} else if tableDef.IsIPv6 {
if setAction == "allow" {
rule, err = chain.AddAcceptIPv6SetRule(setName, ruleName)
} else {
rule, err = chain.AddDropIPv6SetRule(setName, ruleName)
}
}
if err != nil {
return errors.New("add rule failed: " + err.Error())
}
} else {
return errors.New("get rule failed: " + err.Error())
}
}
if rule == nil {
return errors.New("can not create rule '" + string(ruleName) + "'")
}
}
}
this.isReady = true
nftablesIsReady = true
nftablesInstance = this
// load firewalld
var firewalld = NewFirewalld()
if firewalld.IsReady() {
this.firewalld = firewalld
}
return nil
}
// Name 名称
func (this *NFTablesFirewall) Name() string {
return "nftables"
}
// IsReady 是否已准备被调用
func (this *NFTablesFirewall) IsReady() bool {
return this.isReady
}
// IsMock 是否为模拟
func (this *NFTablesFirewall) IsMock() bool {
return false
}
// AllowPort 允许端口
func (this *NFTablesFirewall) AllowPort(port int, protocol string) error {
if this.firewalld != nil {
return this.firewalld.AllowPort(port, protocol)
}
return nil
}
// RemovePort 删除端口
func (this *NFTablesFirewall) RemovePort(port int, protocol string) error {
if this.firewalld != nil {
return this.firewalld.RemovePort(port, protocol)
}
return nil
}
// AllowSourceIP Allow把IP加入白名单
func (this *NFTablesFirewall) AllowSourceIP(ip string) error {
var data = net.ParseIP(ip)
if data == nil {
return errors.New("invalid ip '" + ip + "'")
}
if strings.Contains(ip, ":") { // ipv6
if this.allowIPv6Set == nil {
return errors.New("ipv6 ip set is nil")
}
return this.allowIPv6Set.AddElement(data.To16(), nil)
}
// ipv4
if this.allowIPv4Set == nil {
return errors.New("ipv4 ip set is nil")
}
return this.allowIPv4Set.AddElement(data.To4(), nil)
}
// RejectSourceIP 拒绝某个源IP连接
// we did not create set for drop ip, so we reuse DropSourceIP() method here
func (this *NFTablesFirewall) RejectSourceIP(ip string, timeoutSeconds int) error {
return this.DropSourceIP(ip, timeoutSeconds)
}
// DropSourceIP 丢弃某个源IP数据
func (this *NFTablesFirewall) DropSourceIP(ip string, timeoutSeconds int) error {
var data = net.ParseIP(ip)
if data == nil {
return errors.New("invalid ip '" + ip + "'")
}
if strings.Contains(ip, ":") { // ipv6
if this.denyIPv6Set == nil {
return errors.New("ipv6 ip set is nil")
}
return this.denyIPv6Set.AddElement(data.To16(), &nftables.ElementOptions{
Timeout: time.Duration(timeoutSeconds) * time.Second,
})
}
// ipv4
if this.denyIPv4Set == nil {
return errors.New("ipv4 ip set is nil")
}
return this.denyIPv4Set.AddElement(data.To4(), &nftables.ElementOptions{
Timeout: time.Duration(timeoutSeconds) * time.Second,
})
}
// RemoveSourceIP 删除某个源IP
func (this *NFTablesFirewall) RemoveSourceIP(ip string) error {
var data = net.ParseIP(ip)
if data == nil {
return errors.New("invalid ip '" + ip + "'")
}
if strings.Contains(ip, ":") { // ipv6
if this.denyIPv6Set != nil {
err := this.denyIPv6Set.DeleteElement(data.To16())
if err != nil {
return err
}
}
if this.allowIPv6Set != nil {
err := this.allowIPv6Set.DeleteElement(data.To16())
if err != nil {
return err
}
}
return nil
}
// ipv4
if this.allowIPv4Set != nil {
err := this.denyIPv4Set.DeleteElement(data.To4())
if err != nil {
return err
}
err = this.allowIPv4Set.DeleteElement(data.To4())
if err != nil {
return err
}
}
return nil
}
// 读取版本号
func (this *NFTablesFirewall) readVersion(nftPath string) string {
var cmd = exec.Command(nftPath, "--version")
var output = &bytes.Buffer{}
cmd.Stdout = output
err := cmd.Run()
if err != nil {
return ""
}
var outputString = output.String()
var versionMatches = regexp.MustCompile(`nftables v([\d.]+)`).FindStringSubmatch(outputString)
if len(versionMatches) <= 1 {
return ""
}
return versionMatches[1]
}

View File

@@ -0,0 +1,61 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build !linux
// +build !linux
package firewalls
import (
"errors"
)
func NewNFTablesFirewall() (*NFTablesFirewall, error) {
return nil, errors.New("not implemented")
}
type NFTablesFirewall struct {
}
// Name 名称
func (this *NFTablesFirewall) Name() string {
return "nftables"
}
// IsReady 是否已准备被调用
func (this *NFTablesFirewall) IsReady() bool {
return false
}
// IsMock 是否为模拟
func (this *NFTablesFirewall) IsMock() bool {
return true
}
// AllowPort 允许端口
func (this *NFTablesFirewall) AllowPort(port int, protocol string) error {
return nil
}
// RemovePort 删除端口
func (this *NFTablesFirewall) RemovePort(port int, protocol string) error {
return nil
}
// AllowSourceIP Allow把IP加入白名单
func (this *NFTablesFirewall) AllowSourceIP(ip string) error {
return nil
}
// RejectSourceIP 拒绝某个源IP连接
func (this *NFTablesFirewall) RejectSourceIP(ip string, timeoutSeconds int) error {
return nil
}
// DropSourceIP 丢弃某个源IP数据
func (this *NFTablesFirewall) DropSourceIP(ip string, timeoutSeconds int) error {
return nil
}
// RemoveSourceIP 删除某个源IP
func (this *NFTablesFirewall) RemoveSourceIP(ip string) error {
return nil
}

View File

@@ -0,0 +1 @@
build_remote.sh

View File

@@ -0,0 +1,370 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import (
"bytes"
"errors"
nft "github.com/google/nftables"
"github.com/google/nftables/expr"
)
const MaxChainNameLength = 31
type RuleOptions struct {
Exprs []expr.Any
UserData []byte
}
// Chain chain object in table
type Chain struct {
conn *Conn
rawTable *nft.Table
rawChain *nft.Chain
}
func NewChain(conn *Conn, rawTable *nft.Table, rawChain *nft.Chain) *Chain {
return &Chain{
conn: conn,
rawTable: rawTable,
rawChain: rawChain,
}
}
func (this *Chain) Raw() *nft.Chain {
return this.rawChain
}
func (this *Chain) Name() string {
return this.rawChain.Name
}
func (this *Chain) AddRule(options *RuleOptions) (*Rule, error) {
var rawRule = this.conn.Raw().AddRule(&nft.Rule{
Table: this.rawTable,
Chain: this.rawChain,
Exprs: options.Exprs,
UserData: options.UserData,
})
err := this.conn.Commit()
if err != nil {
return nil, err
}
return NewRule(rawRule), nil
}
func (this *Chain) AddAcceptIPv4Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) AddAcceptIPv6Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) AddDropIPv4Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Verdict{
Kind: expr.VerdictDrop,
},
},
UserData: userData,
})
}
func (this *Chain) AddDropIPv6Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Verdict{
Kind: expr.VerdictDrop,
},
},
UserData: userData,
})
}
func (this *Chain) AddRejectIPv4Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Reject{},
},
UserData: userData,
})
}
func (this *Chain) AddRejectIPv6Rule(ip []byte, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ip,
},
&expr.Reject{},
},
UserData: userData,
})
}
func (this *Chain) AddAcceptIPv4SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) AddAcceptIPv6SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) AddDropIPv4SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Verdict{
Kind: expr.VerdictDrop,
},
},
UserData: userData,
})
}
func (this *Chain) AddDropIPv6SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Verdict{
Kind: expr.VerdictDrop,
},
},
UserData: userData,
})
}
func (this *Chain) AddRejectIPv4SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 12,
Len: 4,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Reject{},
},
UserData: userData,
})
}
func (this *Chain) AddRejectIPv6SetRule(setName string, userData []byte) (*Rule, error) {
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Payload{
DestRegister: 1,
Base: expr.PayloadBaseNetworkHeader,
Offset: 8,
Len: 16,
},
&expr.Lookup{
SourceRegister: 1,
SetName: setName,
},
&expr.Reject{},
},
UserData: userData,
})
}
func (this *Chain) AddAcceptInterfaceRule(interfaceName string, userData []byte) (*Rule, error) {
if len(interfaceName) >= 16 {
return nil, errors.New("invalid interface name '" + interfaceName + "'")
}
var ifname = make([]byte, 16)
copy(ifname, interfaceName+"\x00")
return this.AddRule(&RuleOptions{
Exprs: []expr.Any{
&expr.Meta{
Key: expr.MetaKeyIIFNAME,
Register: 1,
},
&expr.Cmp{
Op: expr.CmpOpEq,
Register: 1,
Data: ifname,
},
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
UserData: userData,
})
}
func (this *Chain) GetRuleWithUserData(userData []byte) (*Rule, error) {
rawRules, err := this.conn.Raw().GetRule(this.rawTable, this.rawChain)
if err != nil {
return nil, err
}
for _, rawRule := range rawRules {
if bytes.Compare(rawRule.UserData, userData) == 0 {
return NewRule(rawRule), nil
}
}
return nil, ErrRuleNotFound
}
func (this *Chain) GetRules() ([]*Rule, error) {
rawRules, err := this.conn.Raw().GetRule(this.rawTable, this.rawChain)
if err != nil {
return nil, err
}
var result = []*Rule{}
for _, rawRule := range rawRules {
result = append(result, NewRule(rawRule))
}
return result, nil
}
func (this *Chain) DeleteRule(rule *Rule) error {
err := this.conn.Raw().DelRule(rule.Raw())
if err != nil {
return err
}
return this.conn.Commit()
}
func (this *Chain) Flush() error {
this.conn.Raw().FlushChain(this.rawChain)
return this.conn.Commit()
}

View File

@@ -0,0 +1,13 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import nft "github.com/google/nftables"
type ChainPolicy = nft.ChainPolicy
// Possible ChainPolicy values.
const (
ChainPolicyDrop = nft.ChainPolicyDrop
ChainPolicyAccept = nft.ChainPolicyAccept
)

View File

@@ -0,0 +1,130 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables_test
import (
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"net"
"testing"
)
func getIPv4Chain(t *testing.T) *nftables.Chain {
var conn = nftables.NewConn()
table, err := conn.GetTable("test_ipv4", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {
table, err = conn.AddIPv4Table("test_ipv4")
if err != nil {
t.Fatal(err)
}
} else {
t.Fatal(err)
}
}
chain, err := table.GetChain("test_chain")
if err != nil {
if err == nftables.ErrChainNotFound {
chain, err = table.AddAcceptChain("test_chain")
}
}
if err != nil {
t.Fatal(err)
}
return chain
}
func TestChain_AddAcceptIPRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddAcceptIPv4Rule(net.ParseIP("192.168.2.40").To4(), nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_AddDropIPRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddDropIPv4Rule(net.ParseIP("192.168.2.31").To4(), nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_AddAcceptSetRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddAcceptIPv4SetRule("ipv4_black_set", nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_AddDropSetRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddDropIPv4SetRule("ipv4_black_set", nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_AddRejectSetRule(t *testing.T) {
var chain = getIPv4Chain(t)
_, err := chain.AddRejectIPv4SetRule("ipv4_black_set", nil)
if err != nil {
t.Fatal(err)
}
}
func TestChain_GetRuleWithUserData(t *testing.T) {
var chain = getIPv4Chain(t)
rule, err := chain.GetRuleWithUserData([]byte("test"))
if err != nil {
if err == nftables.ErrRuleNotFound {
t.Log("rule not found")
return
} else {
t.Fatal(err)
}
}
t.Log("rule:", rule)
}
func TestChain_GetRules(t *testing.T) {
var chain = getIPv4Chain(t)
rules, err := chain.GetRules()
if err != nil {
t.Fatal(err)
}
for _, rule := range rules {
t.Log("handle:", rule.Handle(), "set name:", rule.LookupSetName(),
"verdict:", rule.VerDict(), "user data:", string(rule.UserData()))
}
}
func TestChain_DeleteRule(t *testing.T) {
var chain = getIPv4Chain(t)
rule, err := chain.GetRuleWithUserData([]byte("test"))
if err != nil {
if err == nftables.ErrRuleNotFound {
t.Log("rule not found")
return
}
t.Fatal(err)
}
err = chain.DeleteRule(rule)
if err != nil {
t.Fatal(err)
}
}
func TestChain_Flush(t *testing.T) {
var chain = getIPv4Chain(t)
err := chain.Flush()
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -0,0 +1,84 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import (
"errors"
nft "github.com/google/nftables"
"github.com/iwind/TeaGo/types"
)
const MaxTableNameLength = 27
type Conn struct {
rawConn *nft.Conn
}
func NewConn() *Conn {
return &Conn{
rawConn: &nft.Conn{},
}
}
func (this *Conn) Raw() *nft.Conn {
return this.rawConn
}
func (this *Conn) GetTable(name string, family TableFamily) (*Table, error) {
rawTables, err := this.rawConn.ListTables()
if err != nil {
return nil, err
}
for _, rawTable := range rawTables {
if rawTable.Name == name && rawTable.Family == family {
return NewTable(this, rawTable), nil
}
}
return nil, ErrTableNotFound
}
func (this *Conn) AddTable(name string, family TableFamily) (*Table, error) {
if len(name) > MaxTableNameLength {
return nil, errors.New("table name too long (max " + types.String(MaxTableNameLength) + ")")
}
var rawTable = this.rawConn.AddTable(&nft.Table{
Family: family,
Name: name,
})
err := this.Commit()
if err != nil {
return nil, err
}
return NewTable(this, rawTable), nil
}
func (this *Conn) AddIPv4Table(name string) (*Table, error) {
return this.AddTable(name, TableFamilyIPv4)
}
func (this *Conn) AddIPv6Table(name string) (*Table, error) {
return this.AddTable(name, TableFamilyIPv6)
}
func (this *Conn) DeleteTable(name string, family TableFamily) error {
table, err := this.GetTable(name, family)
if err != nil {
if err == ErrTableNotFound {
return nil
}
return err
}
this.rawConn.DelTable(table.Raw())
return this.Commit()
}
func (this *Conn) Commit() error {
return this.rawConn.Flush()
}

View File

@@ -0,0 +1,78 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables_test
import (
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"os/exec"
"testing"
)
func TestConn_Test(t *testing.T) {
_, err := exec.LookPath("nft")
if err == nil {
t.Log("ok")
return
}
t.Log(err)
}
func TestConn_GetTable_NotFound(t *testing.T) {
var conn = nftables.NewConn()
table, err := conn.GetTable("a", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {
t.Log("table not found")
} else {
t.Fatal(err)
}
} else {
t.Log("table:", table)
}
}
func TestConn_GetTable(t *testing.T) {
var conn = nftables.NewConn()
table, err := conn.GetTable("myFilter", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {
t.Log("table not found")
} else {
t.Fatal(err)
}
} else {
t.Log("table:", table)
}
}
func TestConn_AddTable(t *testing.T) {
var conn = nftables.NewConn()
{
table, err := conn.AddIPv4Table("test_ipv4")
if err != nil {
t.Fatal(err)
}
t.Log(table.Name())
}
{
table, err := conn.AddIPv6Table("test_ipv6")
if err != nil {
t.Fatal(err)
}
t.Log(table.Name())
}
}
func TestConn_DeleteTable(t *testing.T) {
var conn = nftables.NewConn()
err := conn.DeleteTable("test_ipv4", nftables.TableFamilyIPv4)
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -0,0 +1,8 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
type Element struct {
}

View File

@@ -0,0 +1,19 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import "errors"
var ErrTableNotFound = errors.New("table not found")
var ErrChainNotFound = errors.New("chain not found")
var ErrSetNotFound = errors.New("set not found")
var ErrRuleNotFound = errors.New("rule not found")
func IsNotFound(err error) bool {
if err == nil {
return false
}
return err == ErrTableNotFound || err == ErrChainNotFound || err == ErrSetNotFound || err == ErrRuleNotFound
}

View File

@@ -0,0 +1,18 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import (
nft "github.com/google/nftables"
)
type TableFamily = nft.TableFamily
const (
TableFamilyINet TableFamily = nft.TableFamilyINet
TableFamilyIPv4 TableFamily = nft.TableFamilyIPv4
TableFamilyIPv6 TableFamily = nft.TableFamilyIPv6
TableFamilyARP TableFamily = nft.TableFamilyARP
TableFamilyNetdev TableFamily = nft.TableFamilyNetdev
TableFamilyBridge TableFamily = nft.TableFamilyBridge
)

View File

@@ -0,0 +1,51 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import (
nft "github.com/google/nftables"
"github.com/google/nftables/expr"
)
type Rule struct {
rawRule *nft.Rule
}
func NewRule(rawRule *nft.Rule) *Rule {
return &Rule{
rawRule: rawRule,
}
}
func (this *Rule) Raw() *nft.Rule {
return this.rawRule
}
func (this *Rule) LookupSetName() string {
for _, e := range this.rawRule.Exprs {
exp, ok := e.(*expr.Lookup)
if ok {
return exp.SetName
}
}
return ""
}
func (this *Rule) VerDict() expr.VerdictKind {
for _, e := range this.rawRule.Exprs {
exp, ok := e.(*expr.Verdict)
if ok {
return exp.Kind
}
}
return -100
}
func (this *Rule) Handle() uint64 {
return this.rawRule.Handle
}
func (this *Rule) UserData() []byte {
return this.rawRule.UserData
}

View File

@@ -0,0 +1,161 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/utils"
nft "github.com/google/nftables"
"net"
"strings"
"time"
)
const MaxSetNameLength = 15
type SetOptions struct {
Id uint32
HasTimeout bool
Timeout time.Duration
KeyType SetDataType
DataType SetDataType
Constant bool
Interval bool
Anonymous bool
IsMap bool
}
type ElementOptions struct {
Timeout time.Duration
}
type Set struct {
conn *Conn
rawSet *nft.Set
batch *SetBatch
}
func NewSet(conn *Conn, rawSet *nft.Set) *Set {
return &Set{
conn: conn,
rawSet: rawSet,
batch: &SetBatch{
conn: conn,
rawSet: rawSet,
},
}
}
func (this *Set) Raw() *nft.Set {
return this.rawSet
}
func (this *Set) Name() string {
return this.rawSet.Name
}
func (this *Set) AddElement(key []byte, options *ElementOptions) error {
var rawElement = nft.SetElement{
Key: key,
}
if options != nil {
rawElement.Timeout = options.Timeout
}
err := this.conn.Raw().SetAddElements(this.rawSet, []nft.SetElement{
rawElement,
})
if err != nil {
return err
}
err = this.conn.Commit()
if err != nil {
// retry if exists
if strings.Contains(err.Error(), "file exists") {
deleteErr := this.conn.Raw().SetDeleteElements(this.rawSet, []nft.SetElement{
{
Key: key,
},
})
if deleteErr == nil {
err = this.conn.Raw().SetAddElements(this.rawSet, []nft.SetElement{
rawElement,
})
if err == nil {
err = this.conn.Commit()
}
}
}
}
return err
}
func (this *Set) AddIPElement(ip string, options *ElementOptions) error {
var ipObj = net.ParseIP(ip)
if ipObj == nil {
return errors.New("invalid ip '" + ip + "'")
}
if utils.IsIPv4(ip) {
return this.AddElement(ipObj.To4(), options)
} else {
return this.AddElement(ipObj.To16(), options)
}
}
func (this *Set) DeleteElement(key []byte) error {
err := this.conn.Raw().SetDeleteElements(this.rawSet, []nft.SetElement{
{
Key: key,
},
})
if err != nil {
return err
}
err = this.conn.Commit()
if err != nil {
if strings.Contains(err.Error(), "no such file or directory") {
err = nil
}
}
return err
}
func (this *Set) DeleteIPElement(ip string) error {
var ipObj = net.ParseIP(ip)
if ipObj == nil {
return errors.New("invalid ip '" + ip + "'")
}
if utils.IsIPv4(ip) {
return this.DeleteElement(ipObj.To4())
} else {
return this.DeleteElement(ipObj.To16())
}
}
func (this *Set) Batch() *SetBatch {
return this.batch
}
func (this *Set) GetIPElements() ([]string, error) {
elements, err := this.conn.Raw().GetSetElements(this.rawSet)
if err != nil {
return nil, err
}
var result = []string{}
for _, element := range elements {
result = append(result, net.IP(element.Key).String())
}
return result, nil
}
// not work current time
/**func (this *Set) Flush() error {
this.conn.Raw().FlushSet(this.rawSet)
return this.conn.Commit()
}**/

View File

@@ -0,0 +1,36 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import (
nft "github.com/google/nftables"
)
type SetBatch struct {
conn *Conn
rawSet *nft.Set
}
func (this *SetBatch) AddElement(key []byte, options *ElementOptions) error {
var rawElement = nft.SetElement{
Key: key,
}
if options != nil {
rawElement.Timeout = options.Timeout
}
return this.conn.Raw().SetAddElements(this.rawSet, []nft.SetElement{
rawElement,
})
}
func (this *SetBatch) DeleteElement(key []byte) error {
return this.conn.Raw().SetDeleteElements(this.rawSet, []nft.SetElement{
{
Key: key,
},
})
}
func (this *SetBatch) Commit() error {
return this.conn.Commit()
}

View File

@@ -0,0 +1,57 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables
import nft "github.com/google/nftables"
type SetDataType = nft.SetDatatype
var (
TypeInvalid = nft.TypeInvalid
TypeVerdict = nft.TypeVerdict
TypeNFProto = nft.TypeNFProto
TypeBitmask = nft.TypeBitmask
TypeInteger = nft.TypeInteger
TypeString = nft.TypeString
TypeLLAddr = nft.TypeLLAddr
TypeIPAddr = nft.TypeIPAddr
TypeIP6Addr = nft.TypeIP6Addr
TypeEtherAddr = nft.TypeEtherAddr
TypeEtherType = nft.TypeEtherType
TypeARPOp = nft.TypeARPOp
TypeInetProto = nft.TypeInetProto
TypeInetService = nft.TypeInetService
TypeICMPType = nft.TypeICMPType
TypeTCPFlag = nft.TypeTCPFlag
TypeDCCPPktType = nft.TypeDCCPPktType
TypeMHType = nft.TypeMHType
TypeTime = nft.TypeTime
TypeMark = nft.TypeMark
TypeIFIndex = nft.TypeIFIndex
TypeARPHRD = nft.TypeARPHRD
TypeRealm = nft.TypeRealm
TypeClassID = nft.TypeClassID
TypeUID = nft.TypeUID
TypeGID = nft.TypeGID
TypeCTState = nft.TypeCTState
TypeCTDir = nft.TypeCTDir
TypeCTStatus = nft.TypeCTStatus
TypeICMP6Type = nft.TypeICMP6Type
TypeCTLabel = nft.TypeCTLabel
TypePktType = nft.TypePktType
TypeICMPCode = nft.TypeICMPCode
TypeICMPV6Code = nft.TypeICMPV6Code
TypeICMPXCode = nft.TypeICMPXCode
TypeDevGroup = nft.TypeDevGroup
TypeDSCP = nft.TypeDSCP
TypeECN = nft.TypeECN
TypeFIBAddr = nft.TypeFIBAddr
TypeBoolean = nft.TypeBoolean
TypeCTEventBit = nft.TypeCTEventBit
TypeIFName = nft.TypeIFName
TypeIGMPType = nft.TypeIGMPType
TypeTimeDate = nft.TypeTimeDate
TypeTimeHour = nft.TypeTimeHour
TypeTimeDay = nft.TypeTimeDay
TypeCGroupV2 = nft.TypeCGroupV2
)

View File

@@ -0,0 +1,110 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nftables_test
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"github.com/iwind/TeaGo/types"
"github.com/mdlayher/netlink"
"net"
"testing"
"time"
)
func getIPv4Set(t *testing.T) *nftables.Set {
var table = getIPv4Table(t)
set, err := table.GetSet("test_ipv4_set")
if err != nil {
if err == nftables.ErrSetNotFound {
set, err = table.AddSet("test_ipv4_set", &nftables.SetOptions{
KeyType: nftables.TypeIPAddr,
HasTimeout: true,
})
if err != nil {
t.Fatal(err)
}
} else {
t.Fatal(err)
}
}
return set
}
func TestSet_AddElement(t *testing.T) {
var set = getIPv4Set(t)
err := set.AddElement(net.ParseIP("192.168.2.31").To4(), &nftables.ElementOptions{Timeout: 86400 * time.Second})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestSet_DeleteElement(t *testing.T) {
var set = getIPv4Set(t)
err := set.DeleteElement(net.ParseIP("192.168.2.31").To4())
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestSet_Batch(t *testing.T) {
var batch = getIPv4Set(t).Batch()
for _, ip := range []string{"192.168.2.30", "192.168.2.31", "192.168.2.32", "192.168.2.33", "192.168.2.34"} {
var ipData = net.ParseIP(ip).To4()
//err := batch.DeleteElement(ipData)
//if err != nil {
// t.Fatal(err)
//}
err := batch.AddElement(ipData, &nftables.ElementOptions{Timeout: 10 * time.Second})
if err != nil {
t.Fatal(err)
}
}
err := batch.Commit()
if err != nil {
t.Logf("%#v", errors.Unwrap(err).(*netlink.OpError))
t.Fatal(err)
}
t.Log("ok")
}
func TestSet_Add_Many(t *testing.T) {
var set = getIPv4Set(t)
for i := 0; i < 255; i++ {
t.Log(i)
for j := 0; j < 255; j++ {
var ip = "192.167." + types.String(i) + "." + types.String(j)
var ipData = net.ParseIP(ip).To4()
err := set.Batch().AddElement(ipData, &nftables.ElementOptions{Timeout: 3600 * time.Second})
if err != nil {
t.Fatal(err)
}
if j%10 == 0 {
err = set.Batch().Commit()
if err != nil {
t.Fatal(err)
}
}
}
err := set.Batch().Commit()
if err != nil {
t.Fatal(err)
}
}
t.Log("ok")
}
/**func TestSet_Flush(t *testing.T) {
var set = getIPv4Set(t)
err := set.Flush()
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}**/

View File

@@ -0,0 +1,157 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables
import (
"errors"
nft "github.com/google/nftables"
"github.com/iwind/TeaGo/types"
"strings"
)
type Table struct {
conn *Conn
rawTable *nft.Table
}
func NewTable(conn *Conn, rawTable *nft.Table) *Table {
return &Table{
conn: conn,
rawTable: rawTable,
}
}
func (this *Table) Raw() *nft.Table {
return this.rawTable
}
func (this *Table) Name() string {
return this.rawTable.Name
}
func (this *Table) Family() TableFamily {
return this.rawTable.Family
}
func (this *Table) GetChain(name string) (*Chain, error) {
rawChains, err := this.conn.Raw().ListChains()
if err != nil {
return nil, err
}
for _, rawChain := range rawChains {
// must compare table name
if rawChain.Name == name && rawChain.Table.Name == this.rawTable.Name {
return NewChain(this.conn, this.rawTable, rawChain), nil
}
}
return nil, ErrChainNotFound
}
func (this *Table) AddChain(name string, chainPolicy *ChainPolicy) (*Chain, error) {
if len(name) > MaxChainNameLength {
return nil, errors.New("chain name too long (max " + types.String(MaxChainNameLength) + ")")
}
var rawChain = this.conn.Raw().AddChain(&nft.Chain{
Name: name,
Table: this.rawTable,
Hooknum: nft.ChainHookInput,
Priority: nft.ChainPriorityFilter,
Type: nft.ChainTypeFilter,
Policy: chainPolicy,
})
err := this.conn.Commit()
if err != nil {
return nil, err
}
return NewChain(this.conn, this.rawTable, rawChain), nil
}
func (this *Table) AddAcceptChain(name string) (*Chain, error) {
var policy = ChainPolicyAccept
return this.AddChain(name, &policy)
}
func (this *Table) AddDropChain(name string) (*Chain, error) {
var policy = ChainPolicyDrop
return this.AddChain(name, &policy)
}
func (this *Table) DeleteChain(name string) error {
chain, err := this.GetChain(name)
if err != nil {
if err == ErrChainNotFound {
return nil
}
return err
}
this.conn.Raw().DelChain(chain.Raw())
return this.conn.Commit()
}
func (this *Table) GetSet(name string) (*Set, error) {
rawSet, err := this.conn.Raw().GetSetByName(this.rawTable, name)
if err != nil {
if strings.Contains(err.Error(), "no such file or directory") {
return nil, ErrSetNotFound
}
return nil, err
}
return NewSet(this.conn, rawSet), nil
}
func (this *Table) AddSet(name string, options *SetOptions) (*Set, error) {
if len(name) > MaxSetNameLength {
return nil, errors.New("set name too long (max " + types.String(MaxSetNameLength) + ")")
}
if options == nil {
options = &SetOptions{}
}
var rawSet = &nft.Set{
Table: this.rawTable,
ID: options.Id,
Name: name,
Anonymous: options.Anonymous,
Constant: options.Constant,
Interval: options.Interval,
IsMap: options.IsMap,
HasTimeout: options.HasTimeout,
Timeout: options.Timeout,
KeyType: options.KeyType,
DataType: options.DataType,
}
err := this.conn.Raw().AddSet(rawSet, nil)
if err != nil {
return nil, err
}
err = this.conn.Commit()
if err != nil {
return nil, err
}
return NewSet(this.conn, rawSet), nil
}
func (this *Table) DeleteSet(name string) error {
set, err := this.GetSet(name)
if err != nil {
if err == ErrSetNotFound {
return nil
}
return err
}
this.conn.Raw().DelSet(set.Raw())
return this.conn.Commit()
}
func (this *Table) Flush() error {
this.conn.Raw().FlushTable(this.rawTable)
return this.conn.Commit()
}

View File

@@ -0,0 +1,140 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build linux
// +build linux
package nftables_test
import (
"github.com/TeaOSLab/EdgeNode/internal/firewalls/nftables"
"testing"
)
func getIPv4Table(t *testing.T) *nftables.Table {
var conn = nftables.NewConn()
table, err := conn.GetTable("test_ipv4", nftables.TableFamilyIPv4)
if err != nil {
if err == nftables.ErrTableNotFound {
table, err = conn.AddIPv4Table("test_ipv4")
if err != nil {
t.Fatal(err)
}
} else {
t.Fatal(err)
}
}
return table
}
func TestTable_AddChain(t *testing.T) {
var table = getIPv4Table(t)
{
chain, err := table.AddChain("test_default_chain", nil)
if err != nil {
t.Fatal(err)
}
t.Log("created:", chain.Name())
}
{
chain, err := table.AddAcceptChain("test_accept_chain")
if err != nil {
t.Fatal(err)
}
t.Log("created:", chain.Name())
}
// Do not test drop chain before adding accept rule, you will drop yourself!!!!!!!
/**{
chain, err := table.AddDropChain("test_drop_chain")
if err != nil {
t.Fatal(err)
}
t.Log("created:", chain.Name())
}**/
}
func TestTable_GetChain(t *testing.T) {
var table = getIPv4Table(t)
for _, chainName := range []string{"not_found_chain", "test_default_chain"} {
chain, err := table.GetChain(chainName)
if err != nil {
if err == nftables.ErrChainNotFound {
t.Log(chainName, ":", "not found")
} else {
t.Fatal(err)
}
} else {
t.Log(chainName, ":", chain)
}
}
}
func TestTable_DeleteChain(t *testing.T) {
var table = getIPv4Table(t)
err := table.DeleteChain("test_default_chain")
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestTable_AddSet(t *testing.T) {
var table = getIPv4Table(t)
{
set, err := table.AddSet("ipv4_black_set", &nftables.SetOptions{
HasTimeout: false,
KeyType: nftables.TypeIPAddr,
})
if err != nil {
t.Fatal(err)
}
t.Log(set.Name())
}
{
set, err := table.AddSet("ipv6_black_set", &nftables.SetOptions{
HasTimeout: true,
//Timeout: 3600 * time.Second,
KeyType: nftables.TypeIP6Addr,
})
if err != nil {
t.Fatal(err)
}
t.Log(set.Name())
}
}
func TestTable_GetSet(t *testing.T) {
var table = getIPv4Table(t)
for _, setName := range []string{"not_found_set", "ipv4_black_set"} {
set, err := table.GetSet(setName)
if err != nil {
if err == nftables.ErrSetNotFound {
t.Log(setName, ": not found")
} else {
t.Fatal(err)
}
} else {
t.Log(setName, ":", set)
}
}
}
func TestTable_DeleteSet(t *testing.T) {
var table = getIPv4Table(t)
err := table.DeleteSet("ipv4_black_set")
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestTable_Flush(t *testing.T) {
var table = getIPv4Table(t)
err := table.Flush()
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -99,7 +99,7 @@ func (this *IP2Region) MemorySearch(ipStr string) (ipInfo *IpInfo, err error) {
}
}
if dataPtr == 0 {
return nil, errors.New("not found")
return nil, nil
}
dataLen := (dataPtr >> 24) & 0xFF

View File

@@ -13,7 +13,7 @@ const (
// IPItem IP条目
type IPItem struct {
Type string `json:"type"`
Id int64 `json:"id"`
Id uint64 `json:"id"`
IPFrom uint64 `json:"ipFrom"`
IPTo uint64 `json:"ipTo"`
ExpiredAt int64 `json:"expiredAt"`

View File

@@ -13,9 +13,9 @@ var GlobalWhiteIPList = NewIPList()
// IPList IP名单
// TODO IP名单可以分片关闭这样让每一片的数据量减少查询更快
type IPList struct {
itemsMap map[int64]*IPItem // id => item
itemsMap map[uint64]*IPItem // id => item
sortedItems []*IPItem
allItemsMap map[int64]*IPItem // id => item
allItemsMap map[uint64]*IPItem // id => item
expireList *expires.List
@@ -24,12 +24,12 @@ type IPList struct {
func NewIPList() *IPList {
list := &IPList{
itemsMap: map[int64]*IPItem{},
allItemsMap: map[int64]*IPItem{},
itemsMap: map[uint64]*IPItem{},
allItemsMap: map[uint64]*IPItem{},
}
expireList := expires.NewList()
expireList.OnGC(func(itemId int64) {
expireList.OnGC(func(itemId uint64) {
list.Delete(itemId)
})
list.expireList = expireList
@@ -51,7 +51,7 @@ func (this *IPList) Sort() {
this.locker.Unlock()
}
func (this *IPList) Delete(itemId int64) {
func (this *IPList) Delete(itemId uint64) {
this.locker.Lock()
this.deleteItem(itemId)
this.locker.Unlock()
@@ -198,7 +198,7 @@ func (this *IPList) lookupIP(ip uint64) *IPItem {
// 在不加锁的情况下删除某个Item
// 将会被别的方法引用,切记不能加锁
func (this *IPList) deleteItem(itemId int64) {
func (this *IPList) deleteItem(itemId uint64) {
_, ok := this.itemsMap[itemId]
if !ok {
return

View File

@@ -48,7 +48,7 @@ func (this *IPListDB) init() error {
if err != nil {
return err
}
remotelogs.Println("CACHE", "create cache dir '"+this.dir+"'")
remotelogs.Println("IP_LIST_DB", "create data dir '"+this.dir+"'")
}
db, err := sql.Open("sqlite3", "file:"+this.dir+"/ip_list.db?cache=shared&mode=rwc&_journal_mode=WAL&_sync=OFF")

View File

@@ -3,17 +3,29 @@
package iplibrary
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
)
// AllowIP 检查IP是否被允许访问
// 如果一个IP不在任何名单中则允许访问
func AllowIP(ip string, serverId int64) (canGoNext bool, inAllowList bool) {
// 放行lo
if ip == "127.0.0.1" {
return true, true
}
var ipLong = utils.IP2Long(ip)
if ipLong == 0 {
return false, false
}
// check node
nodeConfig, err := nodeconfigs.SharedNodeConfig()
if err == nil && nodeConfig.IPIsAutoAllowed(ip) {
return true, true
}
// check white lists
if GlobalWhiteIPList.Contains(ipLong) {
return true, true

View File

@@ -149,6 +149,10 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
Size: this.pageSize,
})
if err != nil {
if rpc.IsConnError(err) {
remotelogs.Warn("IP_LIST_MANAGER", "rpc connection error: "+err.Error())
return false, nil
}
return false, err
}
items := itemsResp.IpItems
@@ -209,7 +213,7 @@ func (this *IPListManager) processItems(items []*pb.IPItem, fromRemote bool) {
changedLists[list] = zero.New()
if item.IsDeleted {
list.Delete(item.Id)
list.Delete(uint64(item.Id))
// 从WAF名单中删除
waf.SharedIPBlackList.RemoveIP(item.IpFrom, item.ServerId, fromRemote)
@@ -223,7 +227,7 @@ func (this *IPListManager) processItems(items []*pb.IPItem, fromRemote bool) {
}
list.AddDelay(&IPItem{
Id: item.Id,
Id: uint64(item.Id),
Type: item.Type,
IPFrom: utils.IP2Long(item.IpFrom),
IPTo: utils.IP2Long(item.IpTo),

View File

@@ -3,9 +3,9 @@
package metrics
import (
"encoding/json"
"github.com/cespare/xxhash"
"github.com/TeaOSLab/EdgeNode/internal/utils/fnv"
"strconv"
"strings"
)
type Stat struct {
@@ -17,6 +17,6 @@ type Stat struct {
}
func SumStat(serverId int64, keys []string, time string, version int32, itemId int64) string {
keysData, _ := json.Marshal(keys)
return strconv.FormatUint(xxhash.Sum64String(strconv.FormatInt(serverId, 10)+"@"+string(keysData)+"@"+time+"@"+strconv.Itoa(int(version))+"@"+strconv.FormatInt(itemId, 10)), 10)
keysData := strings.Join(keys, "$EDGE$")
return strconv.FormatUint(fnv.HashString(strconv.FormatInt(serverId, 10)+"@"+keysData+"@"+time+"@"+strconv.Itoa(int(version))+"@"+strconv.FormatInt(itemId, 10)), 10)
}

View File

@@ -23,7 +23,7 @@ import (
"time"
)
const MaxQueueSize = 10240
const MaxQueueSize = 2048
// Task 单个指标任务
// 数据库存储:
@@ -58,7 +58,7 @@ type Task struct {
timeMap map[string]zero.Zero // time => bool
serverIdMapLocker sync.Mutex
statsMap map[string]*Stat
statsMap map[string]*Stat // 待写入队列hash => *Stat
statsLocker sync.Mutex
statsTicker *utils.Ticker
}

View File

@@ -1,32 +1,30 @@
package nodes
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/messageconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"github.com/TeaOSLab/EdgeNode/internal/configs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/firewalls"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/Tea"
"io"
"net"
"net/http"
"github.com/iwind/TeaGo/maps"
"net/url"
"os/exec"
"regexp"
"runtime"
"strconv"
"strings"
"sync"
"time"
)
@@ -111,14 +109,12 @@ func (this *APIStream) loop() error {
err = this.handleStatCache(message)
case messageconfigs.MessageCodeCleanCache: // 清理缓存
err = this.handleCleanCache(message)
case messageconfigs.MessageCodePurgeCache: // 删除缓存
err = this.handlePurgeCache(message)
case messageconfigs.MessageCodePreheatCache: // 预热缓存
err = this.handlePreheatCache(message)
case messageconfigs.MessageCodeNewNodeTask: // 有新的任务
err = this.handleNewNodeTask(message)
case messageconfigs.MessageCodeCheckSystemdService: // 检查Systemd服务
err = this.handleCheckSystemdService(message)
case messageconfigs.MessageCodeCheckLocalFirewall: // 检查本地防火墙
err = this.handleCheckLocalFirewall(message)
case messageconfigs.MessageCodeChangeAPINode: // 修改API节点地址
err = this.handleChangeAPINode(message)
default:
@@ -328,224 +324,6 @@ func (this *APIStream) handleCleanCache(message *pb.NodeStreamMessage) error {
return nil
}
// 删除缓存
func (this *APIStream) handlePurgeCache(message *pb.NodeStreamMessage) error {
msg := &messageconfigs.PurgeCacheMessage{}
err := json.Unmarshal(message.DataJSON, msg)
if err != nil {
this.replyFail(message.RequestId, "decode message data failed: "+err.Error())
return err
}
storage, shouldStop, err := this.cacheStorage(message, msg.CachePolicyJSON)
if err != nil {
return err
}
if shouldStop {
defer func() {
storage.Stop()
}()
}
// WEBP缓存
if msg.Type == "file" {
var keys = msg.Keys
for _, key := range keys {
keys = append(keys,
key+caches.SuffixMethod+"HEAD",
key+caches.SuffixWebP,
key+caches.SuffixPartial,
)
// TODO 根据实际缓存的内容进行组合
for _, encoding := range compressions.AllEncodings() {
keys = append(keys, key+caches.SuffixCompression+encoding)
keys = append(keys, key+caches.SuffixWebP+caches.SuffixCompression+encoding)
}
}
msg.Keys = keys
}
err = storage.Purge(msg.Keys, msg.Type)
if err != nil {
this.replyFail(message.RequestId, "purge keys failed: "+err.Error())
return err
}
this.replyOk(message.RequestId, "ok")
return nil
}
// 预热缓存
func (this *APIStream) handlePreheatCache(message *pb.NodeStreamMessage) error {
msg := &messageconfigs.PreheatCacheMessage{}
err := json.Unmarshal(message.DataJSON, msg)
if err != nil {
this.replyFail(message.RequestId, "decode message data failed: "+err.Error())
return err
}
storage, shouldStop, err := this.cacheStorage(message, msg.CachePolicyJSON)
if err != nil {
return err
}
if shouldStop {
defer func() {
storage.Stop()
}()
}
if len(msg.Keys) == 0 {
this.replyOk(message.RequestId, "ok")
return nil
}
wg := sync.WaitGroup{}
wg.Add(len(msg.Keys))
client := &http.Client{
Timeout: 30 * time.Second, // TODO 可以设置请求超时时间
Transport: &http.Transport{
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
_, port, err := net.SplitHostPort(addr)
if err != nil {
return nil, err
}
return net.Dial(network, "127.0.0.1:"+port)
},
MaxIdleConns: 4096,
MaxIdleConnsPerHost: 32,
MaxConnsPerHost: 32,
IdleConnTimeout: 2 * time.Minute,
ExpectContinueTimeout: 1 * time.Second,
TLSHandshakeTimeout: 0,
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
},
},
}
defer client.CloseIdleConnections()
errorMessages := []string{}
locker := sync.Mutex{}
for _, key := range msg.Keys {
go func(key string) {
defer wg.Done()
req, err := http.NewRequest(http.MethodGet, key, nil)
if err != nil {
locker.Lock()
errorMessages = append(errorMessages, "invalid url: "+key+": "+err.Error())
locker.Unlock()
return
}
// TODO 可以在管理界面自定义Header
req.Header.Set("X-Cache-Action", "preheat")
req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36")
req.Header.Set("Accept-Encoding", "gzip, deflate, br") // TODO 这里需要记录下缓存是否为gzip的
resp, err := client.Do(req)
if err != nil {
locker.Lock()
errorMessages = append(errorMessages, "request failed: "+key+": "+err.Error())
locker.Unlock()
return
}
if resp.StatusCode != 200 {
locker.Lock()
errorMessages = append(errorMessages, "request failed: "+key+": status code '"+strconv.Itoa(resp.StatusCode)+"'")
locker.Unlock()
return
}
defer func() {
_ = resp.Body.Close()
}()
// 检查最大内容长度
// TODO 需要解决Chunked Transfer Encoding的长度判断问题
maxSize := storage.Policy().MaxSizeBytes()
if maxSize > 0 && resp.ContentLength > maxSize {
locker.Lock()
errorMessages = append(errorMessages, "request failed: the content is too larger than policy setting")
locker.Unlock()
return
}
expiredAt := time.Now().Unix() + 8600
writer, err := storage.OpenWriter(key, expiredAt, 200, resp.ContentLength, -1, false) // TODO 可以设置缓存过期时间
if err != nil {
locker.Lock()
errorMessages = append(errorMessages, "open cache writer failed: "+key+": "+err.Error())
locker.Unlock()
return
}
buf := make([]byte, 16*1024)
isClosed := false
// 写入Header
for k, v := range resp.Header {
for _, v1 := range v {
_, err = writer.WriteHeader([]byte(k + ":" + v1 + "\n"))
if err != nil {
locker.Lock()
errorMessages = append(errorMessages, "write failed: "+key+": "+err.Error())
locker.Unlock()
return
}
}
}
// 写入Body
for {
n, err := resp.Body.Read(buf)
if n > 0 {
_, writerErr := writer.Write(buf[:n])
if writerErr != nil {
locker.Lock()
errorMessages = append(errorMessages, "write failed: "+key+": "+writerErr.Error())
locker.Unlock()
break
}
}
if err != nil {
if err == io.EOF {
err = writer.Close()
if err == nil {
storage.AddToList(&caches.Item{
Type: writer.ItemType(),
Key: key,
ExpiredAt: expiredAt,
})
}
isClosed = true
} else {
locker.Lock()
errorMessages = append(errorMessages, "read url failed: "+key+": "+err.Error())
locker.Unlock()
}
break
}
}
if !isClosed {
_ = writer.Close()
}
}(key)
}
wg.Wait()
if len(errorMessages) > 0 {
this.replyFail(message.RequestId, strings.Join(errorMessages, ", "))
return nil
}
this.replyOk(message.RequestId, "ok")
return nil
}
// 处理配置变化
func (this *APIStream) handleNewNodeTask(message *pb.NodeStreamMessage) error {
select {
@@ -569,7 +347,7 @@ func (this *APIStream) handleCheckSystemdService(message *pb.NodeStreamMessage)
return nil
}
cmd := utils.NewCommandExecutor()
var cmd = utils.NewCommandExecutor()
shortName := teaconst.SystemdServiceName
cmd.Add(systemctl, "is-enabled", shortName)
output, err := cmd.Run()
@@ -585,6 +363,63 @@ func (this *APIStream) handleCheckSystemdService(message *pb.NodeStreamMessage)
return nil
}
// 检查本地防火墙
func (this *APIStream) handleCheckLocalFirewall(message *pb.NodeStreamMessage) error {
var dataMessage = &messageconfigs.CheckLocalFirewallMessage{}
err := json.Unmarshal(message.DataJSON, dataMessage)
if err != nil {
this.replyFail(message.RequestId, "decode message data failed: "+err.Error())
return nil
}
// nft
if dataMessage.Name == "nftables" {
if runtime.GOOS != "linux" {
this.replyFail(message.RequestId, "not Linux system")
return nil
}
nft, err := exec.LookPath("nft")
if err != nil {
this.replyFail(message.RequestId, "'nft' not found: "+err.Error())
return nil
}
var cmd = exec.Command(nft, "--version")
var output = &bytes.Buffer{}
cmd.Stdout = output
err = cmd.Run()
if err != nil {
this.replyFail(message.RequestId, "get version failed: "+err.Error())
return nil
}
var outputString = output.String()
var versionMatches = regexp.MustCompile(`nftables v([\d.]+)`).FindStringSubmatch(outputString)
if len(versionMatches) <= 1 {
this.replyFail(message.RequestId, "can not get nft version")
return nil
}
var version = versionMatches[1]
var result = maps.Map{
"version": version,
}
var protectionConfig = sharedNodeConfig.DDOSProtection
err = firewalls.SharedDDoSProtectionManager.Apply(protectionConfig)
if err != nil {
this.replyFail(message.RequestId, dataMessage.Name+"was installed, but apply DDoS protection config failed: "+err.Error())
} else {
this.replyOk(message.RequestId, string(result.AsJSON()))
}
} else {
this.replyFail(message.RequestId, "invalid firewall name '"+dataMessage.Name+"'")
}
return nil
}
// 修改API地址
func (this *APIStream) handleChangeAPINode(message *pb.NodeStreamMessage) error {
config, err := configs.LoadAPIConfig()
@@ -660,6 +495,11 @@ func (this *APIStream) replyOk(requestId int64, message string) {
_ = this.stream.Send(&pb.NodeStreamMessage{RequestId: requestId, IsOk: true, Message: message})
}
// 回复成功并包含数据
func (this *APIStream) replyOkData(requestId int64, message string, dataJSON []byte) {
_ = this.stream.Send(&pb.NodeStreamMessage{RequestId: requestId, IsOk: true, Message: message, DataJSON: dataJSON})
}
// 获取缓存存取对象
func (this *APIStream) cacheStorage(message *pb.NodeStreamMessage, cachePolicyJSON []byte) (storage caches.StorageInterface, shouldStop bool, err error) {
cachePolicy := &serverconfigs.HTTPCachePolicy{}

View File

@@ -7,7 +7,6 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/firewallconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"github.com/TeaOSLab/EdgeNode/internal/ratelimit"
"github.com/TeaOSLab/EdgeNode/internal/ttlcache"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/waf"
@@ -21,8 +20,7 @@ import (
// ClientConn 客户端连接
type ClientConn struct {
once sync.Once
globalLimiter *ratelimit.Counter
once sync.Once
isTLS bool
hasDeadline bool
@@ -33,7 +31,7 @@ type ClientConn struct {
BaseClientConn
}
func NewClientConn(conn net.Conn, isTLS bool, quickClose bool, globalLimiter *ratelimit.Counter) net.Conn {
func NewClientConn(conn net.Conn, isTLS bool, quickClose bool) net.Conn {
if quickClose {
// TCP
tcpConn, ok := conn.(*net.TCPConn)
@@ -43,7 +41,7 @@ func NewClientConn(conn net.Conn, isTLS bool, quickClose bool, globalLimiter *ra
}
}
return &ClientConn{BaseClientConn: BaseClientConn{rawConn: conn}, isTLS: isTLS, globalLimiter: globalLimiter}
return &ClientConn{BaseClientConn: BaseClientConn{rawConn: conn}, isTLS: isTLS}
}
func (this *ClientConn) Read(b []byte) (n int, err error) {
@@ -96,13 +94,6 @@ func (this *ClientConn) Close() error {
err := this.rawConn.Close()
// 全局并发数限制
this.once.Do(func() {
if this.globalLimiter != nil {
this.globalLimiter.Release()
}
})
// 单个服务并发数限制
sharedClientConnLimiter.Remove(this.rawConn.RemoteAddr().String())
@@ -137,7 +128,7 @@ func (this *ClientConn) increaseSYNFlood(synFloodConfig *firewallconfigs.SYNFloo
var ip = this.RawIP()
if len(ip) > 0 && !iplibrary.IsInWhiteList(ip) && (!synFloodConfig.IgnoreLocal || !utils.IsLocalIP(ip)) {
var timestamp = utils.NextMinuteUnixTime()
var result = ttlcache.SharedCache.IncreaseInt64("SYN_FLOOD:"+ip, 1, timestamp)
var result = ttlcache.SharedCache.IncreaseInt64("SYN_FLOOD:"+ip, 1, timestamp, true)
var minAttempts = synFloodConfig.MinAttempts
if minAttempts < 5 {
minAttempts = 5

View File

@@ -20,20 +20,21 @@ func init() {
goman.New(func() {
for range ticker.C {
// 加入到数据队列中
if teaconst.InTrafficBytes > 0 {
var inBytes = atomic.LoadUint64(&teaconst.InTrafficBytes)
atomic.StoreUint64(&teaconst.InTrafficBytes, 0) // 重置数据
if inBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficIn, maps.Map{
"total": teaconst.InTrafficBytes,
})
}
if teaconst.OutTrafficBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficOut, maps.Map{
"total": teaconst.OutTrafficBytes,
"total": inBytes,
})
}
// 重置数据
atomic.StoreUint64(&teaconst.InTrafficBytes, 0)
atomic.StoreUint64(&teaconst.OutTrafficBytes, 0)
var outBytes = atomic.LoadUint64(&teaconst.OutTrafficBytes)
atomic.StoreUint64(&teaconst.OutTrafficBytes, 0) // 重置数据
if outBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficOut, maps.Map{
"total": outBytes,
})
}
}
})
})

View File

@@ -3,16 +3,13 @@
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/firewallconfigs"
"github.com/TeaOSLab/EdgeNode/internal/firewalls"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"github.com/TeaOSLab/EdgeNode/internal/ratelimit"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"net"
)
var sharedConnectionsLimiter = ratelimit.NewCounter(nodeconfigs.DefaultTCPMaxConnections)
// ClientListener 客户端网络监听
type ClientListener struct {
rawListener net.Listener
@@ -36,13 +33,8 @@ func (this *ClientListener) IsTLS() bool {
}
func (this *ClientListener) Accept() (net.Conn, error) {
// 限制并发连接数
var limiter = sharedConnectionsLimiter
limiter.Ack()
conn, err := this.rawListener.Accept()
if err != nil {
limiter.Release()
return nil, err
}
@@ -50,22 +42,30 @@ func (this *ClientListener) Accept() (net.Conn, error) {
ip, _, err := net.SplitHostPort(conn.RemoteAddr().String())
if err == nil {
canGoNext, _ := iplibrary.AllowIP(ip, 0)
var beingDenied = !waf.SharedIPWhiteList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip) &&
waf.SharedIPBlackList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip)
if !canGoNext ||
(!waf.SharedIPWhiteList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip) &&
waf.SharedIPBlackList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip)) {
if !canGoNext || beingDenied {
tcpConn, ok := conn.(*net.TCPConn)
if ok {
_ = tcpConn.SetLinger(0)
}
_ = conn.Close()
limiter.Release()
// 使用本地防火墙延长封禁
if beingDenied {
var fw = firewalls.Firewall()
if fw != nil && !fw.IsMock() {
_ = fw.DropSourceIP(ip, 60)
}
}
return this.Accept()
}
}
return NewClientConn(conn, this.isTLS, this.quickClose, limiter), nil
return NewClientConn(conn, this.isTLS, this.quickClose), nil
}
func (this *ClientListener) Close() error {

View File

@@ -0,0 +1,7 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package nodes
type LingerConn interface {
SetLinger(sec int) error
}

View File

@@ -80,6 +80,13 @@ Loop:
return nil
}
// 发送到本地
if sharedHTTPAccessLogViewer.HasConns() {
for _, accessLog := range accessLogs {
sharedHTTPAccessLogViewer.Send(accessLog)
}
}
// 发送到API
if this.rpcClient == nil {
client, err := rpc.SharedRPC()

View File

@@ -0,0 +1,116 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package nodes
import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/types"
"net"
"os"
"sync"
"sync/atomic"
)
var sharedHTTPAccessLogViewer = NewHTTPAccessLogViewer()
// HTTPAccessLogViewer 本地访问日志浏览器
type HTTPAccessLogViewer struct {
sockFile string
listener net.Listener
connMap map[int64]net.Conn // connId => net.Conn
connId int64
locker sync.Mutex
}
// NewHTTPAccessLogViewer 获取新对象
func NewHTTPAccessLogViewer() *HTTPAccessLogViewer {
return &HTTPAccessLogViewer{
sockFile: os.TempDir() + "/" + teaconst.AccessLogSockName,
connMap: map[int64]net.Conn{},
}
}
// Start 启动
func (this *HTTPAccessLogViewer) Start() error {
this.locker.Lock()
defer this.locker.Unlock()
if this.listener == nil {
// remove if exists
_ = os.Remove(this.sockFile)
// start listening
listener, err := net.Listen("unix", this.sockFile)
if err != nil {
return err
}
this.listener = listener
go func() {
for {
conn, err := this.listener.Accept()
if err != nil {
remotelogs.Error("ACCESS_LOG", "start local reading failed: "+err.Error())
break
}
this.locker.Lock()
var connId = this.nextConnId()
this.connMap[connId] = conn
go func() {
this.startReading(conn, connId)
}()
this.locker.Unlock()
}
}()
}
return nil
}
// HasConns 检查是否有连接
func (this *HTTPAccessLogViewer) HasConns() bool {
this.locker.Lock()
defer this.locker.Unlock()
return len(this.connMap) > 0
}
// Send 发送日志
func (this *HTTPAccessLogViewer) Send(accessLog *pb.HTTPAccessLog) {
var conns = []net.Conn{}
this.locker.Lock()
for _, conn := range this.connMap {
conns = append(conns, conn)
}
this.locker.Unlock()
if len(conns) == 0 {
return
}
for _, conn := range conns {
// ignore error
_, _ = conn.Write([]byte(accessLog.RemoteAddr + " [" + accessLog.TimeLocal + "] \"" + accessLog.RequestMethod + " " + accessLog.Scheme + "://" + accessLog.Host + accessLog.RequestURI + " " + accessLog.Proto + "\" " + types.String(accessLog.Status) + " - " + fmt.Sprintf("%.2fms", accessLog.RequestTime*1000) + "\n"))
}
}
func (this *HTTPAccessLogViewer) nextConnId() int64 {
return atomic.AddInt64(&this.connId, 1)
}
func (this *HTTPAccessLogViewer) startReading(conn net.Conn, connId int64) {
var buf = make([]byte, 1024)
for {
_, err := conn.Read(buf)
if err != nil {
this.locker.Lock()
delete(this.connMap, connId)
this.locker.Unlock()
break
}
}
}

View File

@@ -0,0 +1,247 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package nodes
import (
"context"
"crypto/tls"
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/iwind/TeaGo/Tea"
"io"
"io/ioutil"
"net"
"net/http"
"regexp"
"strings"
"time"
)
func init() {
events.On(events.EventStart, func() {
goman.New(func() {
SharedHTTPCacheTaskManager.Start()
})
})
}
var SharedHTTPCacheTaskManager = NewHTTPCacheTaskManager()
// HTTPCacheTaskManager 缓存任务管理
type HTTPCacheTaskManager struct {
ticker *time.Ticker
httpClient *http.Client
protocolReg *regexp.Regexp
taskQueue chan *pb.PurgeServerCacheRequest
}
func NewHTTPCacheTaskManager() *HTTPCacheTaskManager {
var duration = 30 * time.Second
if Tea.IsTesting() {
duration = 10 * time.Second
}
return &HTTPCacheTaskManager{
ticker: time.NewTicker(duration),
httpClient: &http.Client{
Timeout: 10 * time.Minute, // TODO 可以设置请求超时时间
Transport: &http.Transport{
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
_, port, err := net.SplitHostPort(addr)
if err != nil {
return nil, err
}
return net.Dial(network, "127.0.0.1:"+port)
},
MaxIdleConns: 128,
MaxIdleConnsPerHost: 32,
MaxConnsPerHost: 32,
IdleConnTimeout: 2 * time.Minute,
ExpectContinueTimeout: 1 * time.Second,
TLSHandshakeTimeout: 0,
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
},
},
},
protocolReg: regexp.MustCompile(`^(?i)(http|https)://`),
taskQueue: make(chan *pb.PurgeServerCacheRequest, 1024),
}
}
func (this *HTTPCacheTaskManager) Start() {
// task queue
goman.New(func() {
rpcClient, _ := rpc.SharedRPC()
if rpcClient != nil {
for taskReq := range this.taskQueue {
_, err := rpcClient.ServerRPC().PurgeServerCache(rpcClient.Context(), taskReq)
if err != nil {
remotelogs.Error("HTTP_CACHE_TASK_MANAGER", "create purge task failed: "+err.Error())
}
}
}
})
// Loop
for range this.ticker.C {
err := this.Loop()
if err != nil {
remotelogs.Error("HTTP_CACHE_TASK_MANAGER", "execute task failed: "+err.Error())
}
}
}
func (this *HTTPCacheTaskManager) Loop() error {
rpcClient, err := rpc.SharedRPC()
if err != nil {
return err
}
resp, err := rpcClient.HTTPCacheTaskKeyRPC().FindDoingHTTPCacheTaskKeys(rpcClient.Context(), &pb.FindDoingHTTPCacheTaskKeysRequest{})
if err != nil {
return err
}
var keys = resp.HttpCacheTaskKeys
if len(keys) == 0 {
return nil
}
var pbResults = []*pb.UpdateHTTPCacheTaskKeysStatusRequest_KeyResult{}
for _, key := range keys {
err = this.processKey(key)
var pbResult = &pb.UpdateHTTPCacheTaskKeysStatusRequest_KeyResult{
Id: key.Id,
NodeClusterId: key.NodeClusterId,
Error: "",
}
if err != nil {
pbResult.Error = err.Error()
}
pbResults = append(pbResults, pbResult)
}
_, err = rpcClient.HTTPCacheTaskKeyRPC().UpdateHTTPCacheTaskKeysStatus(rpcClient.Context(), &pb.UpdateHTTPCacheTaskKeysStatusRequest{KeyResults: pbResults})
if err != nil {
return err
}
return nil
}
func (this *HTTPCacheTaskManager) PushTaskKeys(keys []string) {
select {
case this.taskQueue <- &pb.PurgeServerCacheRequest{
Keys: keys,
Prefixes: nil,
}:
default:
}
}
func (this *HTTPCacheTaskManager) processKey(key *pb.HTTPCacheTaskKey) error {
switch key.Type {
case "purge":
var storages = caches.SharedManager.FindAllStorages()
for _, storage := range storages {
switch key.KeyType {
case "key":
var cacheKeys = []string{key.Key}
if strings.HasPrefix(key.Key, "http://") {
cacheKeys = append(cacheKeys, strings.Replace(key.Key, "http://", "https://", 1))
} else if strings.HasPrefix(key.Key, "https://") {
cacheKeys = append(cacheKeys, strings.Replace(key.Key, "https://", "http://", 1))
}
// TODO 提升效率
for _, cacheKey := range cacheKeys {
var subKeys = []string{
cacheKey,
cacheKey + caches.SuffixMethod + "HEAD",
cacheKey + caches.SuffixWebP,
cacheKey + caches.SuffixPartial,
}
// TODO 根据实际缓存的内容进行组合
for _, encoding := range compressions.AllEncodings() {
subKeys = append(subKeys, cacheKey+caches.SuffixCompression+encoding)
subKeys = append(subKeys, cacheKey+caches.SuffixWebP+caches.SuffixCompression+encoding)
}
err := storage.Purge(subKeys, "file")
if err != nil {
return err
}
}
case "prefix":
var prefixes = []string{key.Key}
if strings.HasPrefix(key.Key, "http://") {
prefixes = append(prefixes, strings.Replace(key.Key, "http://", "https://", 1))
} else if strings.HasPrefix(key.Key, "https://") {
prefixes = append(prefixes, strings.Replace(key.Key, "https://", "http://", 1))
}
err := storage.Purge(prefixes, "dir")
if err != nil {
return err
}
}
}
case "fetch":
err := this.fetchKey(key)
if err != nil {
return err
}
default:
return errors.New("invalid operation type '" + key.Type + "'")
}
return nil
}
// TODO 增加失败重试
// TODO 使用并发操作
func (this *HTTPCacheTaskManager) fetchKey(key *pb.HTTPCacheTaskKey) error {
var fullKey = key.Key
if !this.protocolReg.MatchString(fullKey) {
fullKey = "https://" + fullKey
}
req, err := http.NewRequest(http.MethodGet, fullKey, nil)
if err != nil {
return errors.New("invalid url: " + fullKey + ": " + err.Error())
}
// TODO 可以在管理界面自定义Header
req.Header.Set("X-Edge-Cache-Action", "fetch")
req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36") // TODO 可以定义
req.Header.Set("Accept-Encoding", "gzip, deflate, br")
resp, err := this.httpClient.Do(req)
if err != nil {
return errors.New("request failed: " + fullKey + ": " + err.Error())
}
defer func() {
_ = resp.Body.Close()
}()
// 读取内容,以便于生成缓存
_, _ = io.Copy(ioutil.Discard, resp.Body)
// 处理502
if resp.StatusCode == http.StatusBadGateway {
return errors.New("read origin site timeout")
}
return nil
}

View File

@@ -0,0 +1,25 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package nodes_test
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/nodes"
"testing"
)
func TestHTTPCacheTaskManager_Loop(t *testing.T) {
// initialize cache policies
config, err := nodeconfigs.SharedNodeConfig()
if err != nil {
t.Fatal(err)
}
caches.SharedManager.UpdatePolicies(config.HTTPCachePolicies)
var manager = nodes.NewHTTPCacheTaskManager()
err = manager.Loop()
if err != nil {
t.Fatal(err)
}
}

View File

@@ -29,7 +29,7 @@ type HTTPClientPool struct {
// NewHTTPClientPool 获取新对象
func NewHTTPClientPool() *HTTPClientPool {
pool := &HTTPClientPool{
var pool = &HTTPClientPool{
clientExpiredDuration: 3600 * time.Second,
clientsMap: map[string]*HTTPClient{},
}
@@ -42,12 +42,16 @@ func NewHTTPClientPool() *HTTPClientPool {
}
// Client 根据地址获取客户端
func (this *HTTPClientPool) Client(req *HTTPRequest, origin *serverconfigs.OriginConfig, originAddr string, proxyProtocol *serverconfigs.ProxyProtocolConfig, followRedirects bool) (rawClient *http.Client, err error) {
func (this *HTTPClientPool) Client(req *HTTPRequest,
origin *serverconfigs.OriginConfig,
originAddr string,
proxyProtocol *serverconfigs.ProxyProtocolConfig,
followRedirects bool) (rawClient *http.Client, err error) {
if origin.Addr == nil {
return nil, errors.New("origin addr should not be empty (originId:" + strconv.FormatInt(origin.Id, 10) + ")")
}
key := origin.UniqueKey() + "@" + originAddr
var key = origin.UniqueKey() + "@" + originAddr
this.locker.Lock()
defer this.locker.Unlock()
@@ -58,11 +62,11 @@ func (this *HTTPClientPool) Client(req *HTTPRequest, origin *serverconfigs.Origi
return client.RawClient(), nil
}
maxConnections := origin.MaxConns
connectionTimeout := origin.ConnTimeoutDuration()
readTimeout := origin.ReadTimeoutDuration()
idleTimeout := origin.IdleTimeoutDuration()
idleConns := origin.MaxIdleConns
var maxConnections = origin.MaxConns
var connectionTimeout = origin.ConnTimeoutDuration()
var readTimeout = origin.ReadTimeoutDuration()
var idleTimeout = origin.IdleTimeoutDuration()
var idleConns = origin.MaxIdleConns
// 超时时间
if connectionTimeout <= 0 {
@@ -73,7 +77,7 @@ func (this *HTTPClientPool) Client(req *HTTPRequest, origin *serverconfigs.Origi
idleTimeout = 2 * time.Minute
}
numberCPU := runtime.NumCPU()
var numberCPU = runtime.NumCPU()
if numberCPU < 8 {
numberCPU = 8
}
@@ -100,49 +104,52 @@ func (this *HTTPClientPool) Client(req *HTTPRequest, origin *serverconfigs.Origi
}
}
var transport = &http.Transport{
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
// 支持TOA的连接
conn, err := this.handleTOA(req, ctx, network, originAddr, connectionTimeout)
if conn != nil || err != nil {
return conn, err
}
var transport = &HTTPClientTransport{
Transport: &http.Transport{
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
// 支持TOA的连接
conn, err := this.handleTOA(req, ctx, network, originAddr, connectionTimeout)
if conn != nil || err != nil {
return conn, err
}
// 普通的连接
conn, err = (&net.Dialer{
Timeout: connectionTimeout,
KeepAlive: 1 * time.Minute,
}).DialContext(ctx, network, originAddr)
if err != nil {
return nil, err
}
// 普通的连接
conn, err = (&net.Dialer{
Timeout: connectionTimeout,
KeepAlive: 1 * time.Minute,
}).DialContext(ctx, network, originAddr)
if err != nil {
return nil, err
}
// 处理PROXY protocol
err = this.handlePROXYProtocol(conn, req, proxyProtocol)
if err != nil {
return nil, err
}
// 处理PROXY protocol
err = this.handlePROXYProtocol(conn, req, proxyProtocol)
if err != nil {
return nil, err
}
return conn, nil
return conn, nil
},
MaxIdleConns: 0,
MaxIdleConnsPerHost: idleConns,
MaxConnsPerHost: maxConnections,
IdleConnTimeout: idleTimeout,
ExpectContinueTimeout: 1 * time.Second,
TLSHandshakeTimeout: 3 * time.Second,
TLSClientConfig: tlsConfig,
Proxy: nil,
},
MaxIdleConns: 0,
MaxIdleConnsPerHost: idleConns,
MaxConnsPerHost: maxConnections,
IdleConnTimeout: idleTimeout,
ExpectContinueTimeout: 1 * time.Second,
TLSHandshakeTimeout: 3 * time.Second,
TLSClientConfig: tlsConfig,
Proxy: nil,
}
rawClient = &http.Client{
Timeout: readTimeout,
Transport: transport,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(targetReq *http.Request, via []*http.Request) error {
// 是否跟随
if followRedirects {
var schemeIsSame = true
for _, r := range via {
if r.URL.Scheme != req.URL.Scheme {
if r.URL.Scheme != targetReq.URL.Scheme {
schemeIsSame = false
break
}
@@ -163,7 +170,7 @@ func (this *HTTPClientPool) Client(req *HTTPRequest, origin *serverconfigs.Origi
// 清理不使用的Client
func (this *HTTPClientPool) cleanClients() {
ticker := time.NewTicker(this.clientExpiredDuration)
var ticker = time.NewTicker(this.clientExpiredDuration)
for range ticker.C {
currentAt := time.Now().Unix()
@@ -181,11 +188,11 @@ func (this *HTTPClientPool) cleanClients() {
// 支持TOA
func (this *HTTPClientPool) handleTOA(req *HTTPRequest, ctx context.Context, network string, originAddr string, connectionTimeout time.Duration) (net.Conn, error) {
// TODO 每个服务读取自身所属集群的TOA设置
toaConfig := sharedTOAManager.Config()
var toaConfig = sharedTOAManager.Config()
if toaConfig != nil && toaConfig.IsOn {
retries := 3
var retries = 3
for i := 1; i <= retries; i++ {
port := int(toaConfig.RandLocalPort())
var port = int(toaConfig.RandLocalPort())
// TODO 思考是否支持X-Real-IP/X-Forwarded-IP
err := sharedTOAManager.SendMsg("add:" + strconv.Itoa(port) + ":" + req.requestRemoteAddr(true))
if err != nil {
@@ -223,7 +230,7 @@ func (this *HTTPClientPool) handlePROXYProtocol(conn net.Conn, req *HTTPRequest,
if reqConn != nil {
destAddr = reqConn.(net.Conn).LocalAddr()
}
header := proxyproto.Header{
var header = proxyproto.Header{
Version: byte(proxyProtocol.Version),
Command: proxyproto.PROXY,
TransportProtocol: transportProtocol,

View File

@@ -0,0 +1,26 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved. Official site: https://goedge.cn .
package nodes
import (
"net/http"
)
const emptyHTTPLocation = "/$EmptyHTTPLocation$"
type HTTPClientTransport struct {
*http.Transport
}
func (this *HTTPClientTransport) RoundTrip(req *http.Request) (*http.Response, error) {
resp, err := this.Transport.RoundTrip(req)
if err != nil {
return resp, err
}
// 检查在跳转相关状态中Location是否存在
if httpStatusIsRedirect(resp.StatusCode) && len(resp.Header.Get("Location")) == 0 {
resp.Header.Set("Location", emptyHTTPLocation)
}
return resp, nil
}

View File

@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/configutils"
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
@@ -47,6 +48,13 @@ type HTTPRequest struct {
IsHTTP bool
IsHTTPS bool
// 共享参数
nodeConfig *nodeconfigs.NodeConfig
// ln request
isLnRequest bool
lnRemoteAddr string
// 内部参数
isSubRequest bool
writer *HTTPWriter
@@ -85,7 +93,8 @@ type HTTPRequest struct {
logAttrs map[string]string
disableLog bool // 请求中关闭Log
disableLog bool // 是否在当前请求中关闭Log
forceLog bool // 是否强制记录日志
// script相关操作
isDone bool
@@ -145,6 +154,9 @@ func (this *HTTPRequest) Do() {
return
}
// 是否为低级别节点
this.isLnRequest = this.checkLnRequest()
// 回调事件
this.onInit()
if this.writer.isFinished {
@@ -152,58 +164,61 @@ func (this *HTTPRequest) Do() {
return
}
// 特殊URL处理
if len(this.rawURI) > 1 && this.rawURI[1] == '.' {
// ACME
// TODO 需要配置是否启用ACME检测
if strings.HasPrefix(this.rawURI, "/.well-known/acme-challenge/") {
this.doACME()
if !this.isLnRequest {
// 特殊URL处理
if len(this.rawURI) > 1 && this.rawURI[1] == '.' {
// ACME
// TODO 需要配置是否启用ACME检测
if strings.HasPrefix(this.rawURI, "/.well-known/acme-challenge/") {
if this.doACME() {
this.doEnd()
return
}
}
}
// 套餐
if this.ReqServer.UserPlan != nil && !this.ReqServer.UserPlan.IsAvailable() {
this.doPlanExpires()
this.doEnd()
return
}
}
// 套餐
if this.ReqServer.UserPlan != nil && !this.ReqServer.UserPlan.IsAvailable() {
this.doPlanExpires()
this.doEnd()
return
}
// 流量限制
if this.ReqServer.TrafficLimit != nil && this.ReqServer.TrafficLimit.IsOn && !this.ReqServer.TrafficLimit.IsEmpty() && this.ReqServer.TrafficLimitStatus != nil && this.ReqServer.TrafficLimitStatus.IsValid() {
this.doTrafficLimit()
this.doEnd()
return
}
// WAF
if this.web.FirewallRef != nil && this.web.FirewallRef.IsOn {
if this.doWAFRequest() {
// 流量限制
if this.ReqServer.TrafficLimit != nil && this.ReqServer.TrafficLimit.IsOn && !this.ReqServer.TrafficLimit.IsEmpty() && this.ReqServer.TrafficLimitStatus != nil && this.ReqServer.TrafficLimitStatus.IsValid() {
this.doTrafficLimit()
this.doEnd()
return
}
}
// 访问控制
if !this.isSubRequest && this.web.Auth != nil && this.web.Auth.IsOn {
if this.doAuth() {
this.doEnd()
return
// WAF
if this.web.FirewallRef != nil && this.web.FirewallRef.IsOn {
if this.doWAFRequest() {
this.doEnd()
return
}
}
}
// 自动跳转到HTTPS
if this.IsHTTP && this.web.RedirectToHttps != nil && this.web.RedirectToHttps.IsOn {
if this.doRedirectToHTTPS(this.web.RedirectToHttps) {
this.doEnd()
return
// 访问控制
if !this.isSubRequest && this.web.Auth != nil && this.web.Auth.IsOn {
if this.doAuth() {
this.doEnd()
return
}
}
}
// Compression
if this.web.Compression != nil && this.web.Compression.IsOn && this.web.Compression.Level > 0 {
this.writer.SetCompression(this.web.Compression)
// 自动跳转到HTTPS
if this.IsHTTP && this.web.RedirectToHttps != nil && this.web.RedirectToHttps.IsOn {
if this.doRedirectToHTTPS(this.web.RedirectToHttps) {
this.doEnd()
return
}
}
// Compression
if this.web.Compression != nil && this.web.Compression.IsOn && this.web.Compression.Level > 0 {
this.writer.SetCompression(this.web.Compression)
}
}
// 开始调用
@@ -218,49 +233,69 @@ func (this *HTTPRequest) Do() {
// 开始调用
func (this *HTTPRequest) doBegin() {
// 处理request limit
if this.web.RequestLimit != nil &&
this.web.RequestLimit.IsOn {
if this.doRequestLimit() {
if !this.isLnRequest {
// 处理request limit
if this.web.RequestLimit != nil &&
this.web.RequestLimit.IsOn {
if this.doRequestLimit() {
return
}
}
// 处理requestBody
if this.RawReq.ContentLength > 0 &&
this.web.AccessLogRef != nil &&
this.web.AccessLogRef.IsOn &&
this.web.AccessLogRef.ContainsField(serverconfigs.HTTPAccessLogFieldRequestBody) {
var err error
this.requestBodyData, err = ioutil.ReadAll(io.LimitReader(this.RawReq.Body, AccessLogMaxRequestBodySize))
if err != nil {
this.write50x(err, http.StatusBadGateway, false)
return
}
this.RawReq.Body = ioutil.NopCloser(io.MultiReader(bytes.NewBuffer(this.requestBodyData), this.RawReq.Body))
}
// 处理健康检查
var isHealthCheck = false
var healthCheckKey = this.RawReq.Header.Get(serverconfigs.HealthCheckHeaderName)
if len(healthCheckKey) > 0 {
if this.doHealthCheck(healthCheckKey, &isHealthCheck) {
return
}
}
// UAM
if !isHealthCheck {
if this.web.UAM != nil {
if this.web.UAM.IsOn {
if this.doUAM() {
this.doEnd()
return
}
}
} else if this.ReqServer.UAM != nil && this.ReqServer.UAM.IsOn {
if this.doUAM() {
this.doEnd()
return
}
}
}
// 跳转
if len(this.web.HostRedirects) > 0 {
if this.doHostRedirect() {
return
}
}
// 临时关闭页面
if this.web.Shutdown != nil && this.web.Shutdown.IsOn {
this.doShutdown()
return
}
}
// 处理requestBody
if this.RawReq.ContentLength > 0 &&
this.web.AccessLogRef != nil &&
this.web.AccessLogRef.IsOn &&
this.web.AccessLogRef.ContainsField(serverconfigs.HTTPAccessLogFieldRequestBody) {
var err error
this.requestBodyData, err = ioutil.ReadAll(io.LimitReader(this.RawReq.Body, AccessLogMaxRequestBodySize))
if err != nil {
this.write50x(err, http.StatusBadGateway, false)
return
}
this.RawReq.Body = ioutil.NopCloser(io.MultiReader(bytes.NewBuffer(this.requestBodyData), this.RawReq.Body))
}
// 处理健康检查
var healthCheckKey = this.RawReq.Header.Get(serverconfigs.HealthCheckHeaderName)
if len(healthCheckKey) > 0 {
if this.doHealthCheck(healthCheckKey) {
return
}
}
// 跳转
if len(this.web.HostRedirects) > 0 {
if this.doHostRedirect() {
return
}
}
// 临时关闭页面
if this.web.Shutdown != nil && this.web.Shutdown.IsOn {
this.doShutdown()
return
}
// 缓存
if this.web.Cache != nil && this.web.Cache.IsOn {
if this.doCacheRead(false) {
@@ -268,30 +303,32 @@ func (this *HTTPRequest) doBegin() {
}
}
// 重写规则
if this.rewriteRule != nil {
if this.doRewrite() {
return
}
}
// Fastcgi
if this.web.FastcgiRef != nil && this.web.FastcgiRef.IsOn && len(this.web.FastcgiList) > 0 {
if this.doFastcgi() {
return
}
}
// root
if this.web.Root != nil && this.web.Root.IsOn {
// 如果处理成功,则终止请求的处理
if this.doRoot() {
return
if !this.isLnRequest {
// 重写规则
if this.rewriteRule != nil {
if this.doRewrite() {
return
}
}
// 如果明确设置了终止,则也会自动终止
if this.web.Root.IsBreak {
return
// Fastcgi
if this.web.FastcgiRef != nil && this.web.FastcgiRef.IsOn && len(this.web.FastcgiList) > 0 {
if this.doFastcgi() {
return
}
}
// root
if this.web.Root != nil && this.web.Root.IsOn {
// 如果处理成功,则终止请求的处理
if this.doRoot() {
return
}
// 如果明确设置了终止,则也会自动终止
if this.web.Root.IsBreak {
return
}
}
}
@@ -312,17 +349,23 @@ func (this *HTTPRequest) doEnd() {
// 流量统计
// TODO 增加是否开启开关
// TODO 增加Header统计考虑从Conn中读取
if this.ReqServer != nil {
var countCached int64 = 0
var cachedBytes int64 = 0
var countAttacks int64 = 0
var attackBytes int64 = 0
if this.isCached {
stats.SharedTrafficStatManager.Add(this.ReqServer.Id, this.ReqHost, this.writer.SentBodyBytes(), this.writer.SentBodyBytes(), 1, 1, 0, 0, this.ReqServer.ShouldCheckTrafficLimit(), this.ReqServer.PlanId())
} else {
if this.isAttack {
stats.SharedTrafficStatManager.Add(this.ReqServer.Id, this.ReqHost, this.writer.SentBodyBytes(), 0, 1, 0, 1, this.writer.SentBodyBytes(), this.ReqServer.ShouldCheckTrafficLimit(), this.ReqServer.PlanId())
} else {
stats.SharedTrafficStatManager.Add(this.ReqServer.Id, this.ReqHost, this.writer.SentBodyBytes(), 0, 1, 0, 0, 0, this.ReqServer.ShouldCheckTrafficLimit(), this.ReqServer.PlanId())
}
countCached = 1
cachedBytes = this.writer.SentBodyBytes() + this.writer.SentHeaderBytes()
}
if this.isAttack {
countAttacks = 1
attackBytes = this.CalculateSize()
}
stats.SharedTrafficStatManager.Add(this.ReqServer.Id, this.ReqHost, this.writer.SentBodyBytes()+this.writer.SentHeaderBytes(), cachedBytes, 1, countCached, countAttacks, attackBytes, this.ReqServer.ShouldCheckTrafficLimit(), this.ReqServer.PlanId())
}
// 指标
@@ -468,7 +511,10 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
// request scripts
if web.RequestScripts != nil {
if this.web.RequestScripts == nil {
this.web.RequestScripts = web.RequestScripts
this.web.RequestScripts = &serverconfigs.HTTPRequestScriptsConfig{
InitGroup: web.RequestScripts.InitGroup,
RequestGroup: web.RequestScripts.RequestGroup,
} // 不要直接赋值,需要复制,防止在运行时被修改
} else {
if web.RequestScripts.InitGroup != nil && (web.RequestScripts.InitGroup.IsPrior || isTop) {
if this.web.RequestScripts == nil {
@@ -485,6 +531,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
}
}
// UAM
if web.UAM != nil && (web.UAM.IsPrior || isTop) {
this.web.UAM = web.UAM
}
// 重写规则
if len(web.RewriteRefs) > 0 {
for index, ref := range web.RewriteRefs {
@@ -791,9 +842,9 @@ func (this *HTTPRequest) Format(source string) string {
if prefix == "node" {
switch suffix {
case "id":
return strconv.FormatInt(sharedNodeConfig.Id, 10)
return strconv.FormatInt(this.nodeConfig.Id, 10)
case "name":
return sharedNodeConfig.Name
return this.nodeConfig.Name
case "role":
return teaconst.Role
}
@@ -952,13 +1003,13 @@ func (this *HTTPRequest) Format(source string) string {
if prefix == "product" {
switch suffix {
case "name":
if sharedNodeConfig.ProductConfig != nil && len(sharedNodeConfig.ProductConfig.Name) > 0 {
return sharedNodeConfig.ProductConfig.Name
if this.nodeConfig.ProductConfig != nil && len(this.nodeConfig.ProductConfig.Name) > 0 {
return this.nodeConfig.ProductConfig.Name
}
return teaconst.GlobalProductName
case "version":
if sharedNodeConfig.ProductConfig != nil && len(sharedNodeConfig.ProductConfig.Version) > 0 {
return sharedNodeConfig.ProductConfig.Version
if this.nodeConfig.ProductConfig != nil && len(this.nodeConfig.ProductConfig.Version) > 0 {
return this.nodeConfig.ProductConfig.Version
}
return teaconst.Version
}
@@ -977,6 +1028,10 @@ func (this *HTTPRequest) addVarMapping(varMapping map[string]string) {
// 获取请求的客户端地址
func (this *HTTPRequest) requestRemoteAddr(supportVar bool) string {
if len(this.lnRemoteAddr) > 0 {
return this.lnRemoteAddr
}
if supportVar && len(this.remoteAddr) > 0 {
return this.remoteAddr
}
@@ -993,7 +1048,7 @@ func (this *HTTPRequest) requestRemoteAddr(supportVar bool) string {
}
// X-Forwarded-For
forwardedFor := this.RawReq.Header.Get("X-Forwarded-For")
var forwardedFor = this.RawReq.Header.Get("X-Forwarded-For")
if len(forwardedFor) > 0 {
commaIndex := strings.Index(forwardedFor, ",")
if commaIndex > 0 {
@@ -1278,6 +1333,28 @@ func (this *HTTPRequest) ContentLength() int64 {
return this.RawReq.ContentLength
}
// CalculateSize 计算当前请求的尺寸(预估)
func (this *HTTPRequest) CalculateSize() (size int64) {
// Get /xxx HTTP/1.1
size += int64(len(this.RawReq.Method)) + 1
size += int64(len(this.RawReq.URL.String())) + 1
size += int64(len(this.RawReq.Proto)) + 1
for k, v := range this.RawReq.Header {
for _, v1 := range v {
size += int64(len(k) + 2 /** : **/ + len(v1) + 1)
}
}
size += 1 /** \r\n **/
if this.RawReq.ContentLength > 0 {
size += this.RawReq.ContentLength
} else if len(this.requestBodyData) > 0 {
size += int64(len(this.requestBodyData))
}
return size
}
// Method 请求方法
func (this *HTTPRequest) Method() string {
return this.RawReq.Method
@@ -1385,7 +1462,12 @@ func (this *HTTPRequest) setForwardHeaders(header http.Header) {
header["X-Forwarded-For"] = []string{strings.Join(forwardedFor, ", ") + ", " + remoteAddr}
}
} else {
header["X-Forwarded-For"] = []string{remoteAddr}
var clientRemoteAddr = this.requestRemoteAddr(true)
if len(clientRemoteAddr) > 0 && clientRemoteAddr != remoteAddr {
header["X-Forwarded-For"] = []string{clientRemoteAddr + ", " + remoteAddr}
} else {
header["X-Forwarded-For"] = []string{remoteAddr}
}
}
}
@@ -1496,7 +1578,7 @@ func (this *HTTPRequest) fixRequestHeader(header http.Header) {
// 处理自定义Response Header
func (this *HTTPRequest) processResponseHeaders(statusCode int) {
responseHeader := this.writer.Header()
var responseHeader = this.writer.Header()
// 删除/添加/替换Header
// TODO 实现AddTrailers
@@ -1584,6 +1666,9 @@ func (this *HTTPRequest) addError(err error) {
// 计算合适的buffer size
func (this *HTTPRequest) bytePool(contentLength int64) *utils.BytePool {
if contentLength < 0 {
return utils.BytePool16k
}
if contentLength < 8192 { // 8K
return utils.BytePool1k
}

View File

@@ -4,34 +4,36 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"net/http"
"path/filepath"
)
func (this *HTTPRequest) doACME() {
func (this *HTTPRequest) doACME() (shouldStop bool) {
// TODO 对请求进行校验,防止恶意攻击
token := filepath.Base(this.RawReq.URL.Path)
var token = filepath.Base(this.RawReq.URL.Path)
if token == "acme-challenge" || len(token) <= 32 {
this.writer.WriteHeader(http.StatusNotFound)
return
return false
}
rpcClient, err := rpc.SharedRPC()
if err != nil {
remotelogs.Error("RPC", "[ACME]rpc failed: "+err.Error())
return
return false
}
keyResp, err := rpcClient.ACMEAuthenticationRPC().FindACMEAuthenticationKeyWithToken(rpcClient.Context(), &pb.FindACMEAuthenticationKeyWithTokenRequest{Token: token})
if err != nil {
remotelogs.Error("RPC", "[ACME]read key for token failed: "+err.Error())
return
return false
}
if len(keyResp.Key) == 0 {
this.writer.WriteHeader(http.StatusNotFound)
} else {
this.writer.Header().Set("Content-Type", "text/plain")
_, _ = this.writer.WriteString(keyResp.Key)
return false
}
this.tags = append(this.tags, "ACME")
this.writer.Header().Set("Content-Type", "text/plain")
_, _ = this.writer.WriteString(keyResp.Key)
return true
}

View File

@@ -3,12 +3,9 @@ package nodes
import (
"bytes"
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
rangeutils "github.com/TeaOSLab/EdgeNode/internal/utils/ranges"
"github.com/iwind/TeaGo/types"
@@ -33,11 +30,6 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
return
}
// 判断是否在预热
if (strings.HasPrefix(this.RawReq.RemoteAddr, "127.") || strings.HasPrefix(this.RawReq.RemoteAddr, "[::1]")) && this.RawReq.Header.Get("X-Cache-Action") == "preheat" {
return
}
// 添加 X-Cache Header
var addStatusHeader = this.web.Cache.AddStatusHeader
if addStatusHeader {
@@ -89,6 +81,12 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
return
}
// 是否正在Purge
var isPurging = this.web.Cache.PurgeIsOn && strings.ToUpper(this.RawReq.Method) == "PURGE" && this.RawReq.Header.Get("X-Edge-Purge-Key") == this.web.Cache.PurgeKey
if isPurging {
this.RawReq.Method = http.MethodGet
}
// 校验请求
if !this.cacheRef.MatchRequest(this.RawReq) {
this.cacheRef = nil
@@ -136,8 +134,13 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
}
this.writer.cacheStorage = storage
// 如果正在预热,则不读取缓存,等待下一个步骤重新生成
if (strings.HasPrefix(this.RawReq.RemoteAddr, "127.") || strings.HasPrefix(this.RawReq.RemoteAddr, "[::1]")) && this.RawReq.Header.Get("X-Edge-Cache-Action") == "fetch" {
return
}
// 判断是否在Purge
if this.web.Cache.PurgeIsOn && strings.ToUpper(this.RawReq.Method) == "PURGE" && this.RawReq.Header.Get("X-Edge-Purge-Key") == this.web.Cache.PurgeKey {
if isPurging {
this.varMapping["cache.status"] = "PURGE"
var subKeys = []string{
@@ -159,22 +162,7 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
}
// 通过API节点清除别节点上的的Key
// TODO 改为队列不需要每个请求都使用goroutine
goman.New(func() {
rpcClient, err := rpc.SharedRPC()
if err == nil {
for _, rpcServerService := range rpcClient.ServerRPCList() {
_, err = rpcServerService.PurgeServerCache(rpcClient.Context(), &pb.PurgeServerCacheRequest{
Domains: []string{this.ReqHost},
Keys: []string{key},
Prefixes: nil,
})
if err != nil {
remotelogs.Error("HTTP_REQUEST_CACHE", "purge failed: "+err.Error())
}
}
}
})
SharedHTTPCacheTaskManager.PushTaskKeys([]string{key})
return true
}
@@ -203,20 +191,35 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
webPIsEnabled = true
}
// 检查压缩缓存
// 检查WebP压缩缓存
if webPIsEnabled && !isPartialRequest && !isHeadMethod && reader == nil {
if this.web.Compression != nil && this.web.Compression.IsOn {
_, encoding, ok := this.web.Compression.MatchAcceptEncoding(this.RawReq.Header.Get("Accept-Encoding"))
if ok {
reader, _ = storage.OpenReader(key+caches.SuffixWebP+caches.SuffixCompression+encoding, useStale, false)
if reader != nil {
tags = append(tags, "webp", encoding)
}
}
}
}
// 检查WebP
if webPIsEnabled && !isPartialRequest &&
!isHeadMethod &&
reader == nil {
reader, _ = storage.OpenReader(key+caches.SuffixWebP, useStale, false)
if reader != nil {
this.writer.cacheReaderSuffix = caches.SuffixWebP
tags = append(tags, "webp")
}
}
// 检查普通压缩缓存
if !isPartialRequest && !isHeadMethod && reader == nil {
if this.web.Compression != nil && this.web.Compression.IsOn {
_, encoding, ok := this.web.Compression.MatchAcceptEncoding(this.RawReq.Header.Get("Accept-Encoding"))
if ok {
// 检查支持WebP的压缩缓存
if webPIsEnabled {
reader, _ = storage.OpenReader(key+caches.SuffixWebP+caches.SuffixCompression+encoding, useStale, false)
if reader != nil {
tags = append(tags, "webp", encoding)
}
}
// 检查普通压缩缓存
if reader == nil {
reader, _ = storage.OpenReader(key+caches.SuffixCompression+encoding, useStale, false)
if reader != nil {
@@ -227,24 +230,17 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
}
}
// 检查WebP
if !isPartialRequest &&
!isHeadMethod &&
reader == nil &&
webPIsEnabled {
reader, _ = storage.OpenReader(key+caches.SuffixWebP, useStale, false)
if reader != nil {
this.writer.cacheReaderSuffix = caches.SuffixWebP
tags = append(tags, "webp")
}
}
// 检查正常的文件
var isPartialCache = false
var partialRanges []rangeutils.Range
if reader == nil {
reader, err = storage.OpenReader(key, useStale, false)
if err != nil && this.cacheRef.AllowPartialContent {
// 尝试读取分片的缓存内容
if len(rangeHeader) == 0 {
// 默认读取开头
rangeHeader = "bytes=0-"
}
pReader, ranges := this.tryPartialReader(storage, key, useStale, rangeHeader)
if pReader != nil {
isPartialCache = true
@@ -304,6 +300,7 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
// 读取Header
var headerBuf = []byte{}
this.writer.SetSentHeaderBytes(reader.HeaderSize())
err = reader.ReadHeader(buf, func(n int) (goNext bool, err error) {
headerBuf = append(headerBuf, buf[:n]...)
for {
@@ -331,7 +328,7 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
}
// 设置cache.age变量
var age = strconv.FormatInt(reader.ExpiresAt()-utils.UnixTime(), 10)
var age = strconv.FormatInt(utils.UnixTime()-reader.LastModified(), 10)
this.varMapping["cache.age"] = age
if addStatusHeader {
@@ -375,7 +372,7 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
}
// 支持 If-None-Match
if !isPartialCache && len(eTag) > 0 && this.requestHeader("If-None-Match") == eTag {
if !this.isLnRequest && !isPartialCache && len(eTag) > 0 && this.requestHeader("If-None-Match") == eTag {
// 自定义Header
this.processResponseHeaders(http.StatusNotModified)
this.writer.WriteHeader(http.StatusNotModified)
@@ -386,7 +383,7 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
}
// 支持 If-Modified-Since
if !isPartialCache && len(modifiedTime) > 0 && this.requestHeader("If-Modified-Since") == modifiedTime {
if !this.isLnRequest && !isPartialCache && len(modifiedTime) > 0 && this.requestHeader("If-Modified-Since") == modifiedTime {
// 自定义Header
this.processResponseHeaders(http.StatusNotModified)
this.writer.WriteHeader(http.StatusNotModified)
@@ -399,6 +396,11 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
this.processResponseHeaders(reader.Status())
this.addExpiresHeader(reader.ExpiresAt())
// 返回上级节点过期时间
if this.isLnRequest {
respHeader.Set(LNExpiresHeader, types.String(reader.ExpiresAt()))
}
// 输出Body
if this.RawReq.Method == http.MethodHead {
this.writer.WriteHeader(reader.Status())
@@ -529,11 +531,22 @@ func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
return true
}
} else { // 没有Range
var resp = &http.Response{Body: reader}
var resp = &http.Response{
Body: reader,
ContentLength: reader.BodySize(),
}
this.writer.Prepare(resp, fileSize, reader.Status(), false)
this.writer.WriteHeader(reader.Status())
_, err = io.CopyBuffer(this.writer, resp.Body, buf)
if storage.CanSendfile() {
if fp, canSendFile := this.writer.canSendfile(); canSendFile {
this.writer.sentBodyBytes, err = io.CopyBuffer(this.writer.rawWriter, fp, buf)
} else {
_, err = io.CopyBuffer(this.writer, resp.Body, buf)
}
} else {
_, err = io.CopyBuffer(this.writer, resp.Body, buf)
}
if err == io.EOF {
err = nil
}

View File

@@ -9,7 +9,7 @@ import (
)
// 健康检查
func (this *HTTPRequest) doHealthCheck(key string) (stop bool) {
func (this *HTTPRequest) doHealthCheck(key string, isHealthCheck *bool) (stop bool) {
this.tags = append(this.tags, "healthCheck")
this.RawReq.Header.Del(serverconfigs.HealthCheckHeaderName)
@@ -19,9 +19,15 @@ func (this *HTTPRequest) doHealthCheck(key string) (stop bool) {
remotelogs.Error("HTTP_REQUEST_HEALTH_CHECK", "decode key failed: "+err.Error())
return
}
*isHealthCheck = true
if !data.GetBool("accessLogIsOn") {
this.disableLog = true
}
if data.GetBool("onlyBasicRequest") {
return true
}
return
}

View File

@@ -2,9 +2,18 @@
package nodes
import "net/http"
import (
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"net/http"
)
func (this *HTTPRequest) doRequestLimit() (shouldStop bool) {
// 是否在全局名单中
_, isInAllowedList := iplibrary.AllowIP(this.RemoteAddr(), this.ReqServer.Id)
if isInAllowedList {
return false
}
// 检查请求Body尺寸
// TODO 处理分片提交的内容
if this.web.RequestLimit.MaxBodyBytes() > 0 &&

View File

@@ -0,0 +1,21 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build !plus
// +build !plus
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
)
const (
LNExpiresHeader = "X-Edge-Ln-Expires"
)
func (this *HTTPRequest) checkLnRequest() bool {
return false
}
func (this *HTTPRequest) getLnOrigin() *serverconfigs.OriginConfig {
return nil
}

View File

@@ -14,85 +14,88 @@ const (
// 日志
func (this *HTTPRequest) log() {
if this.disableLog {
return
var ref *serverconfigs.HTTPAccessLogRef
if !this.forceLog {
if this.disableLog {
return
}
// 计算请求时间
this.requestCost = time.Since(this.requestFromTime).Seconds()
ref = this.web.AccessLogRef
if ref == nil {
ref = serverconfigs.DefaultHTTPAccessLogRef
}
if !ref.IsOn {
return
}
if !ref.Match(this.writer.StatusCode()) {
return
}
if ref.FirewallOnly && this.firewallPolicyId == 0 {
return
}
// 是否记录499
if !ref.EnableClientClosed && this.writer.StatusCode() == 499 {
return
}
}
// 计算请求时间
this.requestCost = time.Since(this.requestFromTime).Seconds()
ref := this.web.AccessLogRef
if ref == nil {
ref = serverconfigs.DefaultHTTPAccessLogRef
}
if !ref.IsOn {
return
}
if !ref.Match(this.writer.StatusCode()) {
return
}
if ref.FirewallOnly && this.firewallPolicyId == 0 {
return
}
// 是否记录499
if !ref.EnableClientClosed && this.writer.StatusCode() == 499 {
return
}
addr := this.RawReq.RemoteAddr
index := strings.LastIndex(addr, ":")
var addr = this.RawReq.RemoteAddr
var index = strings.LastIndex(addr, ":")
if index > 0 {
addr = addr[:index]
}
// 请求Cookie
cookies := map[string]string{}
if ref.ContainsField(serverconfigs.HTTPAccessLogFieldCookie) {
var cookies = map[string]string{}
if ref == nil || ref.ContainsField(serverconfigs.HTTPAccessLogFieldCookie) {
for _, cookie := range this.RawReq.Cookies() {
cookies[cookie.Name] = cookie.Value
}
}
// 请求Header
pbReqHeader := map[string]*pb.Strings{}
if ref.ContainsField(serverconfigs.HTTPAccessLogFieldHeader) {
var pbReqHeader = map[string]*pb.Strings{}
if ref == nil || ref.ContainsField(serverconfigs.HTTPAccessLogFieldHeader) {
for k, v := range this.RawReq.Header {
pbReqHeader[k] = &pb.Strings{Values: v}
}
}
// 响应Header
pbResHeader := map[string]*pb.Strings{}
if ref.ContainsField(serverconfigs.HTTPAccessLogFieldSentHeader) {
var pbResHeader = map[string]*pb.Strings{}
if ref == nil || ref.ContainsField(serverconfigs.HTTPAccessLogFieldSentHeader) {
for k, v := range this.writer.Header() {
pbResHeader[k] = &pb.Strings{Values: v}
}
}
// 参数列表
queryString := ""
if ref.ContainsField(serverconfigs.HTTPAccessLogFieldArg) {
var queryString = ""
if ref == nil || ref.ContainsField(serverconfigs.HTTPAccessLogFieldArg) {
queryString = this.requestQueryString()
}
// 浏览器
userAgent := ""
if ref.ContainsField(serverconfigs.HTTPAccessLogFieldUserAgent) || ref.ContainsField(serverconfigs.HTTPAccessLogFieldExtend) {
var userAgent = ""
if ref == nil || ref.ContainsField(serverconfigs.HTTPAccessLogFieldUserAgent) || ref.ContainsField(serverconfigs.HTTPAccessLogFieldExtend) {
userAgent = this.RawReq.UserAgent()
}
// 请求来源
referer := ""
if ref.ContainsField(serverconfigs.HTTPAccessLogFieldReferer) {
var referer = ""
if ref == nil || ref.ContainsField(serverconfigs.HTTPAccessLogFieldReferer) {
referer = this.RawReq.Referer()
}
accessLog := &pb.HTTPAccessLog{
var accessLog = &pb.HTTPAccessLog{
RequestId: this.requestId,
NodeId: sharedNodeConfig.Id,
NodeId: this.nodeConfig.Id,
ServerId: this.ReqServer.Id,
RemoteAddr: this.requestRemoteAddr(true),
RawRemoteAddr: addr,
@@ -146,7 +149,8 @@ func (this *HTTPRequest) log() {
}
// 请求Body
if ref.ContainsField(serverconfigs.HTTPAccessLogFieldRequestBody) {
// TODO 考虑在被攻击时记录攻击的requestBody如果requestBody匹配规则的话但要考虑请求尺寸、数据库容量避免因为日志而导致服务不稳定
if ref != nil && ref.ContainsField(serverconfigs.HTTPAccessLogFieldRequestBody) {
accessLog.RequestBody = this.requestBodyData
if len(accessLog.RequestBody) > AccessLogMaxRequestBodySize {
@@ -154,7 +158,7 @@ func (this *HTTPRequest) log() {
}
}
// TODO 记录匹配的 locationId和rewriteId
// TODO 记录匹配的 locationId和rewriteId,非必要需求
sharedHTTPAccessLogQueue.Push(accessLog)
}

View File

@@ -36,22 +36,40 @@ func (this *HTTPRequest) doReverseProxy() {
requestCall.Request = this.RawReq
requestCall.Formatter = this.Format
requestCall.Domain = this.ReqHost
var origin = this.reverseProxy.NextOrigin(requestCall)
requestCall.CallResponseCallbacks(this.writer)
var origin *serverconfigs.OriginConfig
// 二级节点
if this.cacheRef != nil {
origin = this.getLnOrigin()
if origin != nil {
// 强制变更原来访问的域名
requestHost = this.ReqHost
}
}
// 自定义源站
if origin == nil {
err := errors.New(this.URL() + ": no available origin sites for reverse proxy")
remotelogs.ServerError(this.ReqServer.Id, "HTTP_REQUEST_REVERSE_PROXY", err.Error(), "", nil)
this.write50x(err, http.StatusBadGateway, true)
return
origin = this.reverseProxy.NextOrigin(requestCall)
requestCall.CallResponseCallbacks(this.writer)
if origin == nil {
err := errors.New(this.URL() + ": no available origin sites for reverse proxy")
remotelogs.ServerError(this.ReqServer.Id, "HTTP_REQUEST_REVERSE_PROXY", err.Error(), "", nil)
this.write50x(err, http.StatusBadGateway, true)
return
}
if len(origin.StripPrefix) > 0 {
stripPrefix = origin.StripPrefix
}
if len(origin.RequestURI) > 0 {
requestURI = origin.RequestURI
requestURIHasVariables = origin.RequestURIHasVariables()
}
}
this.origin = origin // 设置全局变量是为了日志等处理
if len(origin.StripPrefix) > 0 {
stripPrefix = origin.StripPrefix
}
if len(origin.RequestURI) > 0 {
requestURI = origin.RequestURI
requestURIHasVariables = origin.RequestURIHasVariables()
}
if len(origin.RequestHost) > 0 {
requestHost = origin.RequestHost
requestHostHasVariables = origin.RequestHostHasVariables()
@@ -161,7 +179,7 @@ func (this *HTTPRequest) doReverseProxy() {
// 获取请求客户端
client, err := SharedHTTPClientPool.Client(this, origin, originAddr, this.reverseProxy.ProxyProtocol, this.reverseProxy.FollowRedirects)
if err != nil {
remotelogs.Error("HTTP_REQUEST_REVERSE_PROXY", err.Error())
remotelogs.Error("HTTP_REQUEST_REVERSE_PROXY", this.URL()+": "+err.Error())
this.write50x(err, http.StatusBadGateway, true)
return
}
@@ -195,7 +213,7 @@ func (this *HTTPRequest) doReverseProxy() {
this.write50x(err, http.StatusBadGateway, true)
}
if httpErr.Err != io.EOF {
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.RawReq.URL.String()+"': "+err.Error())
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.URL()+": "+err.Error())
}
} else {
// 是否为客户端方面的错误
@@ -234,7 +252,7 @@ func (this *HTTPRequest) doReverseProxy() {
if this.doWAFResponse(resp) {
err = resp.Body.Close()
if err != nil {
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", err.Error())
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.URL()+": "+err.Error())
}
return
}
@@ -244,7 +262,7 @@ func (this *HTTPRequest) doReverseProxy() {
if len(this.web.Pages) > 0 && this.doPage(resp.StatusCode) {
err = resp.Body.Close()
if err != nil {
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", err.Error())
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.URL()+": "+err.Error())
}
return
}
@@ -261,6 +279,38 @@ func (this *HTTPRequest) doReverseProxy() {
}
}
// 替换Location中的源站地址
var locationHeader = resp.Header.Get("Location")
if len(locationHeader) > 0 {
// 空Location处理
if locationHeader == emptyHTTPLocation {
resp.Header.Del("Location")
} else {
// 自动修正Location中的源站地址
locationURL, err := url.Parse(locationHeader)
if err == nil && locationURL.Host != this.ReqHost && (locationURL.Host == originAddr || strings.HasPrefix(originAddr, locationURL.Host+":")) {
locationURL.Host = this.ReqHost
var oldScheme = locationURL.Scheme
// 尝试和当前Scheme一致
if this.IsHTTP {
locationURL.Scheme = "http"
} else if this.IsHTTPS {
locationURL.Scheme = "https"
}
// 如果和当前URL一样则可能是http -> https防止无限循环
if locationURL.String() == this.URL() {
locationURL.Scheme = oldScheme
resp.Header.Set("Location", locationURL.String())
} else {
resp.Header.Set("Location", locationURL.String())
}
}
}
}
// 响应Header
this.writer.AddHeaders(resp.Header)
this.processResponseHeaders(resp.StatusCode)
@@ -278,7 +328,12 @@ func (this *HTTPRequest) doReverseProxy() {
// 是否有内容
if resp.ContentLength == 0 && len(resp.TransferEncoding) == 0 {
// 即使内容为0也需要读取一次以便于触发相关事件
var buf = utils.BytePool4k.Get()
_, _ = io.CopyBuffer(this.writer, resp.Body, buf)
utils.BytePool4k.Put(buf)
_ = resp.Body.Close()
this.writer.SetOk()
return
}
@@ -309,13 +364,13 @@ func (this *HTTPRequest) doReverseProxy() {
var closeErr = resp.Body.Close()
if closeErr != nil {
if !this.canIgnore(closeErr) {
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", closeErr.Error())
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.URL()+": "+closeErr.Error())
}
}
if err != nil && err != io.EOF {
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", err.Error())
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.URL()+": "+err.Error())
this.addError(err)
}
}

View File

@@ -51,7 +51,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
return true
}
rootDir := this.web.Root.Dir
var rootDir = this.web.Root.Dir
if this.web.Root.HasVariables() {
rootDir = this.Format(rootDir)
}
@@ -59,9 +59,9 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
rootDir = Tea.Root + Tea.DS + rootDir
}
requestPath := this.uri
var requestPath = this.uri
questionMarkIndex := strings.Index(this.uri, "?")
var questionMarkIndex = strings.Index(this.uri, "?")
if questionMarkIndex > -1 {
requestPath = this.uri[:questionMarkIndex]
}
@@ -75,7 +75,9 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
if err == nil {
requestPath = p
} else {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
}
}
@@ -92,8 +94,8 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
}
}
filename := strings.Replace(requestPath, "/", Tea.DS, -1)
filePath := ""
var filename = strings.Replace(requestPath, "/", Tea.DS, -1)
var filePath = ""
if len(filename) > 0 && filename[0:1] == Tea.DS {
filePath = rootDir + filename
} else {
@@ -113,7 +115,9 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
return
} else {
this.write50x(err, http.StatusInternalServerError, true)
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
}
@@ -142,7 +146,9 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
return
} else {
this.write50x(err, http.StatusInternalServerError, true)
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
}
@@ -152,24 +158,24 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
}
// 响应header
respHeader := this.writer.Header()
var respHeader = this.writer.Header()
// mime type
contentType := ""
var contentType = ""
if this.web.ResponseHeaderPolicy == nil || !this.web.ResponseHeaderPolicy.IsOn || !this.web.ResponseHeaderPolicy.ContainsHeader("CONTENT-TYPE") {
ext := filepath.Ext(filePath)
var ext = filepath.Ext(filePath)
if len(ext) > 0 {
mimeType := mime.TypeByExtension(ext)
if len(mimeType) > 0 {
semicolonIndex := strings.Index(mimeType, ";")
mimeTypeKey := mimeType
var semicolonIndex = strings.Index(mimeType, ";")
var mimeTypeKey = mimeType
if semicolonIndex > 0 {
mimeTypeKey = mimeType[:semicolonIndex]
}
if _, found := textMimeMap[mimeTypeKey]; found {
if this.web.Charset != nil && this.web.Charset.IsOn && len(this.web.Charset.Charset) > 0 {
charset := this.web.Charset.Charset
var charset = this.web.Charset.Charset
if this.web.Charset.IsUpper {
charset = strings.ToUpper(charset)
}
@@ -197,7 +203,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
}
// 支持 ETag
eTag := "\"e" + fmt.Sprintf("%0x", xxhash.Sum64String(filename+strconv.FormatInt(stat.ModTime().UnixNano(), 10)+strconv.FormatInt(stat.Size(), 10))) + "\""
var eTag = "\"e" + fmt.Sprintf("%0x", xxhash.Sum64String(filename+strconv.FormatInt(stat.ModTime().UnixNano(), 10)+strconv.FormatInt(stat.Size(), 10))) + "\""
if len(respHeader.Get("ETag")) == 0 {
respHeader.Set("ETag", eTag)
}
@@ -227,7 +233,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
// 支持Range
respHeader.Set("Accept-Ranges", "bytes")
ifRangeHeaders, ok := this.RawReq.Header["If-Range"]
supportRange := true
var supportRange = true
if ok {
supportRange = false
for _, v := range ifRangeHeaders {
@@ -244,7 +250,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
// 支持Range
var ranges = []rangeutils.Range{}
if supportRange {
contentRange := this.RawReq.Header.Get("Range")
var contentRange = this.RawReq.Header.Get("Range")
if len(contentRange) > 0 {
if fileSize == 0 {
this.processResponseHeaders(http.StatusRequestedRangeNotSatisfiable)
@@ -277,7 +283,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
respHeader.Set("Content-Length", strconv.FormatInt(fileSize, 10))
}
reader, err := os.OpenFile(filePath, os.O_RDONLY, 0444)
fileReader, err := os.OpenFile(filePath, os.O_RDONLY, 0444)
if err != nil {
this.write50x(err, http.StatusInternalServerError, true)
return true
@@ -291,12 +297,16 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
this.cacheRef = nil // 不支持缓存
}
this.writer.Prepare(nil, fileSize, http.StatusOK, true)
var resp = &http.Response{
ContentLength: fileSize,
Body: fileReader,
StatusCode: http.StatusOK,
}
this.writer.Prepare(resp, fileSize, http.StatusOK, true)
pool := this.bytePool(fileSize)
buf := pool.Get()
var pool = this.bytePool(fileSize)
var buf = pool.Get()
defer func() {
_ = reader.Close()
pool.Put(buf)
}()
@@ -304,12 +314,14 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
respHeader.Set("Content-Range", ranges[0].ComposeContentRangeHeader(types.String(fileSize)))
this.writer.WriteHeader(http.StatusPartialContent)
ok, err := httpRequestReadRange(reader, buf, ranges[0].Start(), ranges[0].End(), func(buf []byte, n int) error {
ok, err := httpRequestReadRange(resp.Body, buf, ranges[0].Start(), ranges[0].End(), func(buf []byte, n int) error {
_, err := this.writer.Write(buf[:n])
return err
})
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
if !ok {
@@ -318,7 +330,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
return true
}
} else if len(ranges) > 1 {
boundary := httpRequestGenBoundary()
var boundary = httpRequestGenBoundary()
respHeader.Set("Content-Type", "multipart/byteranges; boundary="+boundary)
this.writer.WriteHeader(http.StatusPartialContent)
@@ -330,30 +342,38 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
_, err = this.writer.WriteString("\r\n--" + boundary + "\r\n")
}
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
_, err = this.writer.WriteString("Content-Range: " + r.ComposeContentRangeHeader(types.String(fileSize)) + "\r\n")
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
if len(contentType) > 0 {
_, err = this.writer.WriteString("Content-Type: " + contentType + "\r\n\r\n")
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
}
ok, err := httpRequestReadRange(reader, buf, r.Start(), r.End(), func(buf []byte, n int) error {
ok, err := httpRequestReadRange(resp.Body, buf, r.Start(), r.End(), func(buf []byte, n int) error {
_, err := this.writer.Write(buf[:n])
return err
})
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
if !ok {
@@ -365,14 +385,17 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
_, err = this.writer.WriteString("\r\n--" + boundary + "--\r\n")
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
} else {
_, err = io.CopyBuffer(this.writer, reader, buf)
_, err = io.CopyBuffer(this.writer, resp.Body, buf)
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
return true
}
}
@@ -400,7 +423,9 @@ func (this *HTTPRequest) findIndexFile(dir string) (filename string, stat os.Fil
if strings.Contains(index, "*") {
indexFiles, err := filepath.Glob(dir + Tea.DS + index)
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
logs.Error(err)
}
this.addError(err)
continue
}

View File

@@ -17,12 +17,13 @@ func (this *HTTPRequest) doShutdown() {
return
}
if urlPrefixRegexp.MatchString(shutdown.URL) { // URL
this.doURL(http.MethodGet, shutdown.URL, "", shutdown.Status, true)
return
}
if len(shutdown.BodyType) == 0 || shutdown.BodyType == shared.BodyTypeURL {
// URL
if urlPrefixRegexp.MatchString(shutdown.URL) {
this.doURL(http.MethodGet, shutdown.URL, "", shutdown.Status, true)
return
}
// URL为空则显示文本
if len(shutdown.URL) == 0 {
// 自定义响应Headers

View File

@@ -8,7 +8,7 @@ import (
)
func TestHTTPRequest_RedirectToHTTPS(t *testing.T) {
a := assert.NewAssertion(t)
var a = assert.NewAssertion(t)
{
req := &HTTPRequest{
ReqServer: &serverconfigs.ServerConfig{
@@ -47,4 +47,5 @@ func TestHTTPRequest_Memory(t *testing.T) {
var stat2 = &runtime.MemStats{}
runtime.ReadMemStats(stat2)
t.Log((stat2.HeapInuse-stat1.HeapInuse)/1024/1024, "MB,")
t.Log(len(requests))
}

View File

@@ -0,0 +1,10 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build !plus
// +build !plus
package nodes
// UAM
func (this *HTTPRequest) doUAM() (block bool) {
return false
}

View File

@@ -24,7 +24,7 @@ func (this *HTTPRequest) doWAFRequest() (blocked bool) {
var remoteAddr = this.requestRemoteAddr(true)
// 检查是否为白名单直连
if !Tea.IsTesting() && sharedNodeConfig.IPIsAutoAllowed(remoteAddr) {
if !Tea.IsTesting() && this.nodeConfig.IPIsAutoAllowed(remoteAddr) {
return
}
@@ -53,9 +53,11 @@ func (this *HTTPRequest) doWAFRequest() (blocked bool) {
return true
}
var forceLog = this.ReqServer.HTTPFirewallPolicy != nil && this.ReqServer.HTTPFirewallPolicy.IsOn && this.ReqServer.HTTPFirewallPolicy.Log != nil && this.ReqServer.HTTPFirewallPolicy.Log.IsOn
// 当前服务的独立设置
if this.web.FirewallPolicy != nil && this.web.FirewallPolicy.IsOn {
blocked, breakChecking := this.checkWAFRequest(this.web.FirewallPolicy)
blocked, breakChecking := this.checkWAFRequest(this.web.FirewallPolicy, forceLog)
if blocked {
return true
}
@@ -66,7 +68,7 @@ func (this *HTTPRequest) doWAFRequest() (blocked bool) {
// 公用的防火墙设置
if this.ReqServer.HTTPFirewallPolicy != nil && this.ReqServer.HTTPFirewallPolicy.IsOn {
blocked, breakChecking := this.checkWAFRequest(this.ReqServer.HTTPFirewallPolicy)
blocked, breakChecking := this.checkWAFRequest(this.ReqServer.HTTPFirewallPolicy, forceLog)
if blocked {
return true
}
@@ -78,7 +80,7 @@ func (this *HTTPRequest) doWAFRequest() (blocked bool) {
return
}
func (this *HTTPRequest) checkWAFRequest(firewallPolicy *firewallconfigs.HTTPFirewallPolicy) (blocked bool, breakChecking bool) {
func (this *HTTPRequest) checkWAFRequest(firewallPolicy *firewallconfigs.HTTPFirewallPolicy, forceLog bool) (blocked bool, breakChecking bool) {
// 检查配置是否为空
if firewallPolicy == nil || !firewallPolicy.IsOn || firewallPolicy.Inbound == nil || !firewallPolicy.Inbound.IsOn || firewallPolicy.Mode == firewallconfigs.FirewallModeBypass {
return
@@ -192,7 +194,7 @@ func (this *HTTPRequest) checkWAFRequest(firewallPolicy *firewallconfigs.HTTPFir
}
// 规则测试
w := sharedWAFManager.FindWAF(firewallPolicy.Id)
w := waf.SharedWAFManager.FindWAF(firewallPolicy.Id)
if w == nil {
return
}
@@ -206,6 +208,10 @@ func (this *HTTPRequest) checkWAFRequest(firewallPolicy *firewallconfigs.HTTPFir
}
if ruleSet != nil {
if forceLog {
this.forceLog = true
}
if ruleSet.HasSpecialActions() {
this.firewallPolicyId = firewallPolicy.Id
this.firewallRuleGroupId = types.Int64(ruleGroup.Id)
@@ -232,8 +238,9 @@ func (this *HTTPRequest) doWAFResponse(resp *http.Response) (blocked bool) {
}
// 当前服务的独立设置
var forceLog = this.ReqServer.HTTPFirewallPolicy != nil && this.ReqServer.HTTPFirewallPolicy.IsOn && this.ReqServer.HTTPFirewallPolicy.Log != nil && this.ReqServer.HTTPFirewallPolicy.Log.IsOn
if this.web.FirewallPolicy != nil && this.web.FirewallPolicy.IsOn {
blocked := this.checkWAFResponse(this.web.FirewallPolicy, resp)
blocked := this.checkWAFResponse(this.web.FirewallPolicy, resp, forceLog)
if blocked {
return true
}
@@ -241,7 +248,7 @@ func (this *HTTPRequest) doWAFResponse(resp *http.Response) (blocked bool) {
// 公用的防火墙设置
if this.ReqServer.HTTPFirewallPolicy != nil && this.ReqServer.HTTPFirewallPolicy.IsOn {
blocked := this.checkWAFResponse(this.ReqServer.HTTPFirewallPolicy, resp)
blocked := this.checkWAFResponse(this.ReqServer.HTTPFirewallPolicy, resp, forceLog)
if blocked {
return true
}
@@ -249,12 +256,12 @@ func (this *HTTPRequest) doWAFResponse(resp *http.Response) (blocked bool) {
return
}
func (this *HTTPRequest) checkWAFResponse(firewallPolicy *firewallconfigs.HTTPFirewallPolicy, resp *http.Response) (blocked bool) {
func (this *HTTPRequest) checkWAFResponse(firewallPolicy *firewallconfigs.HTTPFirewallPolicy, resp *http.Response, forceLog bool) (blocked bool) {
if firewallPolicy == nil || !firewallPolicy.IsOn || !firewallPolicy.Outbound.IsOn || firewallPolicy.Mode == firewallconfigs.FirewallModeBypass {
return
}
w := sharedWAFManager.FindWAF(firewallPolicy.Id)
w := waf.SharedWAFManager.FindWAF(firewallPolicy.Id)
if w == nil {
return
}
@@ -268,6 +275,10 @@ func (this *HTTPRequest) checkWAFResponse(firewallPolicy *firewallconfigs.HTTPFi
}
if ruleSet != nil {
if forceLog {
this.forceLog = true
}
if ruleSet.HasSpecialActions() {
this.firewallPolicyId = firewallPolicy.Id
this.firewallRuleGroupId = types.Int64(ruleGroup.Id)

View File

@@ -2,7 +2,6 @@ package nodes
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
"net/http"
@@ -20,7 +19,7 @@ func (this *HTTPRequest) doWebsocket() {
// TODO 实现handshakeTimeout
// 校验来源
requestOrigin := this.RawReq.Header.Get("Origin")
var requestOrigin = this.RawReq.Header.Get("Origin")
if len(requestOrigin) > 0 {
u, err := url.Parse(requestOrigin)
if err == nil {
@@ -34,7 +33,7 @@ func (this *HTTPRequest) doWebsocket() {
// 设置指定的来源域
if !this.web.Websocket.RequestSameOrigin && len(this.web.Websocket.RequestOrigin) > 0 {
newRequestOrigin := this.web.Websocket.RequestOrigin
var newRequestOrigin = this.web.Websocket.RequestOrigin
if this.web.Websocket.RequestOriginHasVariables() {
newRequestOrigin = this.Format(newRequestOrigin)
}
@@ -45,8 +44,21 @@ func (this *HTTPRequest) doWebsocket() {
originConn, err := OriginConnect(this.origin, this.RawReq.RemoteAddr)
if err != nil {
this.write50x(err, http.StatusBadGateway, false)
// 增加失败次数
SharedOriginStateManager.Fail(this.origin, this.reverseProxy, func() {
this.reverseProxy.ResetScheduling()
})
return
}
if !this.origin.IsOk {
SharedOriginStateManager.Success(this.origin, func() {
this.reverseProxy.ResetScheduling()
})
}
defer func() {
_ = originConn.Close()
}()
@@ -66,7 +78,7 @@ func (this *HTTPRequest) doWebsocket() {
_ = clientConn.Close()
}()
goman.New(func() {
go func() {
var buf = utils.BytePool4k.Get()
defer utils.BytePool4k.Put(buf)
for {
@@ -84,6 +96,6 @@ func (this *HTTPRequest) doWebsocket() {
}
_ = clientConn.Close()
_ = originConn.Close()
})
}()
_, _ = io.Copy(originConn, clientConn)
}

View File

@@ -6,6 +6,7 @@ import (
"bufio"
"bytes"
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
@@ -58,8 +59,9 @@ type HTTPWriter struct {
size int64
statusCode int
sentBodyBytes int64
statusCode int
sentBodyBytes int64
sentHeaderBytes int64
isOk bool // 是否完全成功
isFinished bool // 是否已完成
@@ -281,12 +283,28 @@ func (this *HTTPWriter) PrepareCache(resp *http.Response, size int64) {
}
}
var expiredAt = utils.UnixTime() + life
var expiresAt = utils.UnixTime() + life
if this.req.isLnRequest {
// 返回上级节点过期时间
this.SetHeader(LNExpiresHeader, []string{types.String(expiresAt)})
} else {
var expiresHeader = this.Header().Get(LNExpiresHeader)
if len(expiresHeader) > 0 {
this.Header().Del(LNExpiresHeader)
var expiresHeaderInt64 = types.Int64(expiresHeader)
if expiresHeaderInt64 > 0 {
expiresAt = expiresHeaderInt64
}
}
}
var cacheKey = this.req.cacheKey
if this.isPartial {
cacheKey += caches.SuffixPartial
}
cacheWriter, err := storage.OpenWriter(cacheKey, expiredAt, this.StatusCode(), size, cacheRef.MaxSizeBytes(), this.isPartial)
cacheWriter, err := storage.OpenWriter(cacheKey, expiresAt, this.StatusCode(), size, cacheRef.MaxSizeBytes(), this.isPartial)
if err != nil {
if err == caches.ErrEntityTooLarge && addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, entity too large")
@@ -294,6 +312,9 @@ func (this *HTTPWriter) PrepareCache(resp *http.Response, size int64) {
if !caches.CanIgnoreErr(err) {
remotelogs.Error("HTTP_WRITER", "write cache failed: "+err.Error())
this.Header().Set("X-Cache", "BYPASS, write cache failed")
} else {
this.Header().Set("X-Cache", "BYPASS, "+err.Error())
}
return
}
@@ -428,7 +449,9 @@ func (this *HTTPWriter) PrepareCache(resp *http.Response, size int64) {
this.rawReader = cacheReader
cacheReader.OnFail(func(err error) {
_ = this.cacheWriter.Discard()
if this.cacheWriter != nil {
_ = this.cacheWriter.Discard()
}
this.cacheWriter = nil
})
cacheReader.OnEOF(func() {
@@ -442,6 +465,30 @@ func (this *HTTPWriter) PrepareWebP(resp *http.Response, size int64) {
return
}
// 集群配置
var policy = this.req.nodeConfig.FindWebPImagePolicyWithClusterId(this.req.ReqServer.ClusterId)
if policy == nil {
policy = nodeconfigs.DefaultWebPImagePolicy
}
if !policy.IsOn {
return
}
// 只有在开启了缓存之后,才会转换,防止占用的系统资源过高
if policy.RequireCache && this.req.cacheRef == nil {
return
}
// 限制最小和最大尺寸
// TODO 需要将reader修改为LimitReader
if resp.ContentLength == 0 {
return
}
if resp.ContentLength > 0 && (resp.ContentLength < policy.MinLengthBytes() || (policy.MaxLengthBytes() > 0 && resp.ContentLength > policy.MaxLengthBytes())) {
return
}
var contentType = this.GetHeader("Content-Type")
if this.req.web != nil &&
@@ -692,6 +739,23 @@ func (this *HTTPWriter) SentBodyBytes() int64 {
return this.sentBodyBytes
}
// SentHeaderBytes 计算发送的Header字节数
func (this *HTTPWriter) SentHeaderBytes() int64 {
if this.sentHeaderBytes > 0 {
return this.sentHeaderBytes
}
for k, v := range this.Header() {
for _, v1 := range v {
this.sentHeaderBytes += int64(len(k) + 2 + len(v1) + 1)
}
}
return this.sentHeaderBytes
}
func (this *HTTPWriter) SetSentHeaderBytes(sentHeaderBytes int64) {
this.sentHeaderBytes = sentHeaderBytes
}
// WriteHeader 写入状态码
func (this *HTTPWriter) WriteHeader(statusCode int) {
if this.rawWriter != nil {
@@ -740,6 +804,27 @@ func (this *HTTPWriter) SendFile(status int, path string) (int64, error) {
return written, nil
}
// SendResp 发送响应对象
func (this *HTTPWriter) SendResp(resp *http.Response) (int64, error) {
this.isFinished = true
for k, v := range resp.Header {
this.SetHeader(k, v)
}
this.WriteHeader(resp.StatusCode)
var bufPool = this.req.bytePool(resp.ContentLength)
var buf = bufPool.Get()
defer bufPool.Put(buf)
return io.CopyBuffer(this, resp.Body, buf)
}
// Redirect 跳转
func (this *HTTPWriter) Redirect(status int, url string) {
http.Redirect(this.rawWriter, this.req.RawReq, url, status)
this.isFinished = true
}
// StatusCode 读取状态码
func (this *HTTPWriter) StatusCode() int {
if this.statusCode == 0 {
@@ -754,7 +839,7 @@ func (this *HTTPWriter) HeaderData() []byte {
return nil
}
resp := &http.Response{}
var resp = &http.Response{}
resp.Header = this.Header()
if this.statusCode == 0 {
this.statusCode = http.StatusOK
@@ -777,6 +862,70 @@ func (this *HTTPWriter) SetOk() {
// Close 关闭
func (this *HTTPWriter) Close() {
this.finishWebP()
this.finishRequest()
this.finishCache()
this.finishCompression()
// 统计
if this.sentBodyBytes == 0 {
this.sentBodyBytes = this.counterWriter.TotalBytes()
}
}
// Hijack Hijack
func (this *HTTPWriter) Hijack() (conn net.Conn, buf *bufio.ReadWriter, err error) {
hijack, ok := this.rawWriter.(http.Hijacker)
if ok {
return hijack.Hijack()
}
return
}
// Flush Flush
func (this *HTTPWriter) Flush() {
flusher, ok := this.rawWriter.(http.Flusher)
if ok {
flusher.Flush()
}
}
// DelayRead 是否延迟读取Reader
func (this *HTTPWriter) DelayRead() bool {
return this.delayRead
}
// 计算stale时长
func (this *HTTPWriter) calculateStaleLife() int {
var staleLife = 600 // TODO 可以在缓存策略里设置此时间
var staleConfig = this.req.web.Cache.Stale
if staleConfig != nil && staleConfig.IsOn {
// 从Header中读取stale-if-error
var isDefinedInHeader = false
if staleConfig.SupportStaleIfErrorHeader {
var cacheControl = this.GetHeader("Cache-Control")
var pieces = strings.Split(cacheControl, ",")
for _, piece := range pieces {
var eqIndex = strings.Index(piece, "=")
if eqIndex > 0 && strings.TrimSpace(piece[:eqIndex]) == "stale-if-error" {
// 这里预示着如果stale-if-error=0可以关闭stale功能
staleLife = types.Int(strings.TrimSpace(piece[eqIndex+1:]))
isDefinedInHeader = true
break
}
}
}
// 自定义
if !isDefinedInHeader && staleConfig.Life != nil {
staleLife = types.Int(staleConfig.Life.Duration().Seconds())
}
}
return staleLife
}
// 结束WebP
func (this *HTTPWriter) finishWebP() {
// 处理WebP
if this.webpIsEncoding {
var webpCacheWriter caches.Writer
@@ -837,6 +986,7 @@ func (this *HTTPWriter) Close() {
}
if err != nil {
// 发生了错误终止处理
return
}
@@ -866,7 +1016,7 @@ func (this *HTTPWriter) Close() {
//webpConfig.SetLossless(1)
webpConfig.SetQuality(f)
timeline := 0
var timeline = 0
for i, img := range gifImage.Image {
err = anim.AddFrame(img, timeline, webpConfig)
@@ -906,15 +1056,10 @@ func (this *HTTPWriter) Close() {
}
}
}
}
if this.writer != nil {
_ = this.writer.Close()
}
if this.rawReader != nil {
_ = this.rawReader.Close()
}
// 结束缓存相关处理
func (this *HTTPWriter) finishCache() {
// 缓存
if this.cacheWriter != nil {
if this.isOk && this.cacheIsFinished {
@@ -972,7 +1117,10 @@ func (this *HTTPWriter) Close() {
}
}
}
}
// 结束压缩相关处理
func (this *HTTPWriter) finishCompression() {
if this.compressionCacheWriter != nil {
if this.isOk {
err := this.compressionCacheWriter.Close()
@@ -993,57 +1141,15 @@ func (this *HTTPWriter) Close() {
_ = this.compressionCacheWriter.Discard()
}
}
this.sentBodyBytes = this.counterWriter.TotalBytes()
}
// Hijack Hijack
func (this *HTTPWriter) Hijack() (conn net.Conn, buf *bufio.ReadWriter, err error) {
hijack, ok := this.rawWriter.(http.Hijacker)
if ok {
return hijack.Hijack()
// 最终关闭
func (this *HTTPWriter) finishRequest() {
if this.writer != nil {
_ = this.writer.Close()
}
return
}
// Flush Flush
func (this *HTTPWriter) Flush() {
flusher, ok := this.rawWriter.(http.Flusher)
if ok {
flusher.Flush()
if this.rawReader != nil {
_ = this.rawReader.Close()
}
}
// DelayRead 是否延迟读取Reader
func (this *HTTPWriter) DelayRead() bool {
return this.delayRead
}
// 计算stale时长
func (this *HTTPWriter) calculateStaleLife() int {
var staleLife = 600 // TODO 可以在缓存策略里设置此时间
var staleConfig = this.req.web.Cache.Stale
if staleConfig != nil && staleConfig.IsOn {
// 从Header中读取stale-if-error
var isDefinedInHeader = false
if staleConfig.SupportStaleIfErrorHeader {
var cacheControl = this.GetHeader("Cache-Control")
var pieces = strings.Split(cacheControl, ",")
for _, piece := range pieces {
var eqIndex = strings.Index(piece, "=")
if eqIndex > 0 && strings.TrimSpace(piece[:eqIndex]) == "stale-if-error" {
// 这里预示着如果stale-if-error=0可以关闭stale功能
staleLife = types.Int(strings.TrimSpace(piece[eqIndex+1:]))
isDefinedInHeader = true
break
}
}
}
// 自定义
if !isDefinedInHeader && staleConfig.Life != nil {
staleLife = types.Int(staleConfig.Life.Duration().Seconds())
}
}
return staleLife
}

View File

@@ -0,0 +1,13 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build !plus
// +build !plus
package nodes
import (
"os"
)
func (this *HTTPWriter) canSendfile() (*os.File, bool) {
return nil, false
}

View File

@@ -3,6 +3,7 @@ package nodes
import (
"context"
"crypto/tls"
"github.com/TeaOSLab/EdgeCommon/pkg/nodeutils"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/zero"
@@ -134,7 +135,7 @@ func (this *HTTPListener) Reload(group *serverconfigs.ServerAddressGroup) {
// ServerHTTP 处理HTTP请求
func (this *HTTPListener) ServeHTTP(rawWriter http.ResponseWriter, rawReq *http.Request) {
// 域名
reqHost := rawReq.Host
var reqHost = rawReq.Host
// TLS域名
if this.isIP(reqHost) {
@@ -170,34 +171,8 @@ func (this *HTTPListener) ServeHTTP(rawWriter http.ResponseWriter, rawReq *http.
server, serverName := this.findNamedServer(domain)
if server == nil {
server = this.findServerWithCNAME(domain)
if server == nil {
// 严格匹配域名模式下,我们拒绝用户访问
if sharedNodeConfig.GlobalConfig != nil && sharedNodeConfig.GlobalConfig.HTTPAll.MatchDomainStrictly {
httpAllConfig := sharedNodeConfig.GlobalConfig.HTTPAll
mismatchAction := httpAllConfig.DomainMismatchAction
if mismatchAction != nil && mismatchAction.Code == "page" {
if mismatchAction.Options != nil {
rawWriter.Header().Set("Content-Type", "text/html; charset=utf-8")
rawWriter.WriteHeader(mismatchAction.Options.GetInt("statusCode"))
_, _ = rawWriter.Write([]byte(mismatchAction.Options.GetString("contentHTML")))
} else {
http.Error(rawWriter, "404 page not found: '"+rawReq.URL.String()+"'", http.StatusNotFound)
}
return
} else {
hijacker, ok := rawWriter.(http.Hijacker)
if ok {
conn, _, _ := hijacker.Hijack()
if conn != nil {
_ = conn.Close()
return
}
}
}
}
http.Error(rawWriter, "404 page not found: '"+rawReq.URL.String()+"'", http.StatusNotFound)
this.handleMismatch(rawReq, rawWriter)
return
} else {
serverName = domain
@@ -205,7 +180,7 @@ func (this *HTTPListener) ServeHTTP(rawWriter http.ResponseWriter, rawReq *http.
}
// 包装新请求对象
req := &HTTPRequest{
var req = &HTTPRequest{
RawReq: rawReq,
RawWriter: rawWriter,
ReqServer: server,
@@ -214,10 +189,54 @@ func (this *HTTPListener) ServeHTTP(rawWriter http.ResponseWriter, rawReq *http.
ServerAddr: this.addr,
IsHTTP: this.isHTTP,
IsHTTPS: this.isHTTPS,
nodeConfig: sharedNodeConfig,
}
req.Do()
}
// 处理域名不匹配的情况
func (this *HTTPListener) handleMismatch(rawReq *http.Request, rawWriter http.ResponseWriter) {
// TODO 需要记录访问记录和防止CC
// 是否为健康检查
var healthCheckKey = rawReq.Header.Get(serverconfigs.HealthCheckHeaderName)
if len(healthCheckKey) > 0 {
_, err := nodeutils.Base64DecodeMap(healthCheckKey)
if err == nil {
rawWriter.WriteHeader(http.StatusOK)
return
}
}
// 严格匹配域名模式下,我们拒绝用户访问
if sharedNodeConfig.GlobalConfig != nil && sharedNodeConfig.GlobalConfig.HTTPAll.MatchDomainStrictly {
var httpAllConfig = sharedNodeConfig.GlobalConfig.HTTPAll
var mismatchAction = httpAllConfig.DomainMismatchAction
if mismatchAction != nil && mismatchAction.Code == "page" {
if mismatchAction.Options != nil {
rawWriter.Header().Set("Content-Type", "text/html; charset=utf-8")
rawWriter.WriteHeader(mismatchAction.Options.GetInt("statusCode"))
_, _ = rawWriter.Write([]byte(mismatchAction.Options.GetString("contentHTML")))
} else {
http.Error(rawWriter, "404 page not found: '"+rawReq.URL.String()+"'", http.StatusNotFound)
}
return
} else {
hijacker, ok := rawWriter.(http.Hijacker)
if ok {
conn, _, _ := hijacker.Hijack()
if conn != nil {
_ = conn.Close()
return
}
}
}
}
http.Error(rawWriter, "404 page not found: '"+rawReq.URL.String()+"'", http.StatusNotFound)
}
func (this *HTTPListener) isIP(host string) bool {
// IPv6
if strings.Index(host, "[") > -1 {

View File

@@ -257,6 +257,12 @@ func (this *ListenerManager) addToFirewalld(groupAddrs []string) {
return
}
// 检查状态
err = exec.Command(firewallCmd, "--state").Run()
if err != nil {
return
}
remotelogs.Println("FIREWALLD", "open ports automatically")
for _, port := range ports {
{

View File

@@ -21,7 +21,7 @@ type TCPListener struct {
}
func (this *TCPListener) Serve() error {
listener := this.Listener
var listener = this.Listener
if this.Group.IsTLS() {
listener = tls.NewListener(listener, this.buildTLSConfig())
}
@@ -52,14 +52,29 @@ func (this *TCPListener) Reload(group *serverconfigs.ServerAddressGroup) {
}
func (this *TCPListener) handleConn(conn net.Conn) error {
firstServer := this.Group.FirstServer()
if firstServer == nil {
var server = this.Group.FirstServer()
if server == nil {
return errors.New("no server available")
}
if firstServer.ReverseProxy == nil {
if server.ReverseProxy == nil {
return errors.New("no ReverseProxy configured for the server")
}
// 是否已达到流量限制
if this.reachedTrafficLimit() {
// 关闭连接
tcpConn, ok := conn.(LingerConn)
if ok {
_ = tcpConn.SetLinger(0)
}
_ = conn.Close()
// TODO 使用系统防火墙drop当前端口的数据包一段时间1分钟
// 不能使用阻止IP的方法因为边缘节点只上有可能还有别的代理服务
return nil
}
// 记录域名排行
tlsConn, ok := conn.(*tls.Conn)
var recordStat = false
@@ -67,17 +82,17 @@ func (this *TCPListener) handleConn(conn net.Conn) error {
var serverName = tlsConn.ConnectionState().ServerName
if len(serverName) > 0 {
// 统计
stats.SharedTrafficStatManager.Add(firstServer.Id, serverName, 0, 0, 1, 0, 0, 0, firstServer.ShouldCheckTrafficLimit(), firstServer.PlanId())
stats.SharedTrafficStatManager.Add(server.Id, serverName, 0, 0, 1, 0, 0, 0, server.ShouldCheckTrafficLimit(), server.PlanId())
recordStat = true
}
}
// 统计
if !recordStat {
stats.SharedTrafficStatManager.Add(firstServer.Id, "", 0, 0, 1, 0, 0, 0, firstServer.ShouldCheckTrafficLimit(), firstServer.PlanId())
stats.SharedTrafficStatManager.Add(server.Id, "", 0, 0, 1, 0, 0, 0, server.ShouldCheckTrafficLimit(), server.PlanId())
}
originConn, err := this.connectOrigin(firstServer.Id, firstServer.ReverseProxy, conn.RemoteAddr().String())
originConn, err := this.connectOrigin(server.Id, server.ReverseProxy, conn.RemoteAddr().String())
if err != nil {
return err
}
@@ -88,17 +103,17 @@ func (this *TCPListener) handleConn(conn net.Conn) error {
}
// PROXY Protocol
if firstServer.ReverseProxy != nil &&
firstServer.ReverseProxy.ProxyProtocol != nil &&
firstServer.ReverseProxy.ProxyProtocol.IsOn &&
(firstServer.ReverseProxy.ProxyProtocol.Version == serverconfigs.ProxyProtocolVersion1 || firstServer.ReverseProxy.ProxyProtocol.Version == serverconfigs.ProxyProtocolVersion2) {
if server.ReverseProxy != nil &&
server.ReverseProxy.ProxyProtocol != nil &&
server.ReverseProxy.ProxyProtocol.IsOn &&
(server.ReverseProxy.ProxyProtocol.Version == serverconfigs.ProxyProtocolVersion1 || server.ReverseProxy.ProxyProtocol.Version == serverconfigs.ProxyProtocolVersion2) {
var remoteAddr = conn.RemoteAddr()
var transportProtocol = proxyproto.TCPv4
if strings.Contains(remoteAddr.String(), "[") {
transportProtocol = proxyproto.TCPv6
}
header := proxyproto.Header{
Version: byte(firstServer.ReverseProxy.ProxyProtocol.Version),
var header = proxyproto.Header{
Version: byte(server.ReverseProxy.ProxyProtocol.Version),
Command: proxyproto.PROXY,
TransportProtocol: transportProtocol,
SourceAddr: remoteAddr,
@@ -113,7 +128,7 @@ func (this *TCPListener) handleConn(conn net.Conn) error {
// 从源站读取
goman.New(func() {
originBuffer := utils.BytePool16k.Get()
var originBuffer = utils.BytePool16k.Get()
defer func() {
utils.BytePool16k.Put(originBuffer)
}()
@@ -127,8 +142,8 @@ func (this *TCPListener) handleConn(conn net.Conn) error {
}
// 记录流量
if firstServer != nil {
stats.SharedTrafficStatManager.Add(firstServer.Id, "", int64(n), 0, 0, 0, 0, 0, firstServer.ShouldCheckTrafficLimit(), firstServer.PlanId())
if server != nil {
stats.SharedTrafficStatManager.Add(server.Id, "", int64(n), 0, 0, 0, 0, 0, server.ShouldCheckTrafficLimit(), server.PlanId())
}
}
if err != nil {
@@ -139,11 +154,17 @@ func (this *TCPListener) handleConn(conn net.Conn) error {
})
// 从客户端读取
clientBuffer := utils.BytePool16k.Get()
var clientBuffer = utils.BytePool16k.Get()
defer func() {
utils.BytePool16k.Put(clientBuffer)
}()
for {
// 是否已达到流量限制
if this.reachedTrafficLimit() {
closer()
return nil
}
n, err := conn.Read(clientBuffer)
if n > 0 {
_, err = originConn.Write(clientBuffer[:n])
@@ -188,3 +209,12 @@ func (this *TCPListener) connectOrigin(serverId int64, reverseProxy *serverconfi
err = errors.New("no origin can be used")
return
}
// 检查是否已经达到流量限制
func (this *TCPListener) reachedTrafficLimit() bool {
var server = this.Group.FirstServer()
if server == nil {
return true
}
return server.TrafficLimitStatus != nil && server.TrafficLimitStatus.IsValid()
}

View File

@@ -24,6 +24,8 @@ type UDPListener struct {
connTicker *utils.Ticker
reverseProxy *serverconfigs.ReverseProxyConfig
isClosed bool
}
func (this *UDPListener) Serve() error {
@@ -46,7 +48,18 @@ func (this *UDPListener) Serve() error {
var buffer = make([]byte, 4*1024)
for {
n, addr, _ := this.Listener.ReadFrom(buffer)
if this.isClosed {
return nil
}
n, addr, err := this.Listener.ReadFrom(buffer)
if err != nil {
if this.isClosed {
return nil
}
return err
}
if n > 0 {
this.connLocker.Lock()
conn, ok := this.connMap[addr.String()]
@@ -76,6 +89,8 @@ func (this *UDPListener) Serve() error {
}
func (this *UDPListener) Close() error {
this.isClosed = true
if this.connTicker != nil {
this.connTicker.Stop()
}

View File

@@ -7,6 +7,7 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/ddosconfigs"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/configs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
@@ -15,12 +16,12 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"github.com/TeaOSLab/EdgeNode/internal/metrics"
"github.com/TeaOSLab/EdgeNode/internal/ratelimit"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/stats"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"github.com/andybalholm/brotli"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/lists"
@@ -111,7 +112,11 @@ func (this *Node) Start() {
return
}
// 检查硬盘类型
this.checkDisk()
// 读取API配置
remotelogs.Println("NODE", "init config ...")
err = this.syncConfig(0)
if err != nil {
_, err := nodeconfigs.SharedNodeConfig()
@@ -333,6 +338,79 @@ func (this *Node) loop() error {
goman.New(func() {
sharedUpgradeManager.Start()
})
case "scriptsChanged":
err = this.reloadCommonScripts()
if err != nil {
return errors.New("reload common scripts failed: " + err.Error())
}
// 修改为已同步
_, err = rpcClient.NodeTaskRPC().ReportNodeTaskDone(nodeCtx, &pb.ReportNodeTaskDoneRequest{
NodeTaskId: task.Id,
IsOk: true,
Error: "",
})
if err != nil {
return err
}
case "nodeLevelChanged":
levelInfoResp, err := rpcClient.NodeRPC().FindNodeLevelInfo(nodeCtx, &pb.FindNodeLevelInfoRequest{})
if err != nil {
return err
}
sharedNodeConfig.Level = levelInfoResp.Level
var parentNodes = map[int64][]*nodeconfigs.ParentNodeConfig{}
if len(levelInfoResp.ParentNodesMapJSON) > 0 {
err = json.Unmarshal(levelInfoResp.ParentNodesMapJSON, &parentNodes)
if err != nil {
return errors.New("decode level info failed: " + err.Error())
}
}
sharedNodeConfig.ParentNodes = parentNodes
// 修改为已同步
_, err = rpcClient.NodeTaskRPC().ReportNodeTaskDone(nodeCtx, &pb.ReportNodeTaskDoneRequest{
NodeTaskId: task.Id,
IsOk: true,
Error: "",
})
if err != nil {
return err
}
case "ddosProtectionChanged":
resp, err := rpcClient.NodeRPC().FindNodeDDoSProtection(nodeCtx, &pb.FindNodeDDoSProtectionRequest{})
if err != nil {
return err
}
if len(resp.DdosProtectionJSON) == 0 {
if sharedNodeConfig != nil {
sharedNodeConfig.DDOSProtection = nil
}
} else {
var ddosProtectionConfig = &ddosconfigs.ProtectionConfig{}
err = json.Unmarshal(resp.DdosProtectionJSON, ddosProtectionConfig)
if err != nil {
return errors.New("decode DDoS protection config failed: " + err.Error())
}
err = firewalls.SharedDDoSProtectionManager.Apply(ddosProtectionConfig)
if err != nil {
// 不阻塞
remotelogs.Error("NODE", "apply DDoS protection failed: "+err.Error())
}
}
// 修改为已同步
_, err = rpcClient.NodeTaskRPC().ReportNodeTaskDone(nodeCtx, &pb.ReportNodeTaskDoneRequest{
NodeTaskId: task.Id,
IsOk: true,
Error: "",
})
if err != nil {
return err
}
}
}
@@ -352,7 +430,7 @@ func (this *Node) syncConfig(taskVersion int64) error {
clusterErr := this.checkClusterConfig()
if clusterErr != nil {
if os.IsNotExist(clusterErr) {
return err
return errors.New("can not find config file 'configs/api.yaml'")
}
return errors.New("check cluster config failed: " + clusterErr.Error())
}
@@ -382,7 +460,7 @@ func (this *Node) syncConfig(taskVersion int64) error {
return nil
}
configJSON := configResp.NodeJSON
var configJSON = configResp.NodeJSON
if configResp.IsCompressed {
var reader = brotli.NewReader(bytes.NewReader(configJSON))
var configBuf = &bytes.Buffer{}
@@ -401,7 +479,7 @@ func (this *Node) syncConfig(taskVersion int64) error {
nodeConfigUpdatedAt = time.Now().Unix()
nodeConfig := &nodeconfigs.NodeConfig{}
var nodeConfig = &nodeconfigs.NodeConfig{}
err = json.Unmarshal(configJSON, nodeConfig)
if err != nil {
return errors.New("decode config failed: " + err.Error())
@@ -409,6 +487,15 @@ func (this *Node) syncConfig(taskVersion int64) error {
teaconst.NodeId = nodeConfig.Id
teaconst.NodeIdString = types.String(teaconst.NodeId)
// 检查时间是否一致
// 这个需要在 teaconst.NodeId 设置之后因为上报到API节点的时候需要节点ID
if configResp.Timestamp > 0 {
var timestampDelta = configResp.Timestamp - time.Now().Unix()
if timestampDelta > 60 || timestampDelta < -60 {
remotelogs.Error("NODE", "node timestamp ('"+types.String(time.Now().Unix())+"') is not same as api node ('"+types.String(configResp.Timestamp)+"'), please sync the time")
}
}
// 写入到文件中
err = nodeConfig.Save()
if err != nil {
@@ -493,42 +580,7 @@ func (this *Node) startSyncTimer() {
continue
}
case <-serverChangeTicker.C: // 服务变化
this.locker.Lock()
if len(this.updatingServerMap) > 0 {
var updatingServerMap = this.updatingServerMap
this.updatingServerMap = map[int64]*serverconfigs.ServerConfig{}
newNodeConfig, err := nodeconfigs.CloneNodeConfig(sharedNodeConfig)
if err != nil {
remotelogs.Error("NODE", "apply server config error: "+err.Error())
continue
}
for serverId, serverConfig := range updatingServerMap {
if serverConfig != nil {
newNodeConfig.AddServer(serverConfig)
} else {
newNodeConfig.RemoveServer(serverId)
}
}
err, serverErrors := newNodeConfig.Init()
if err != nil {
remotelogs.Error("NODE", "apply server config error: "+err.Error())
continue
}
if len(serverErrors) > 0 {
for _, serverErr := range serverErrors {
remotelogs.ServerError(serverErr.Id, "NODE", serverErr.Message, nodeconfigs.NodeLogTypeServerConfigInitFailed, maps.Map{})
}
}
this.onReload(newNodeConfig)
err = sharedListenerManager.Start(newNodeConfig)
if err != nil {
remotelogs.Error("NODE", "apply server config error: "+err.Error())
}
}
this.locker.Unlock()
this.reloadServer()
case <-nodeTaskNotify: // 有新的更新任务
err := this.loop()
if err != nil {
@@ -712,7 +764,6 @@ func (this *Node) listenSock() error {
"ipConns": ipConns,
"serverConns": serverConns,
"total": sharedListenerManager.TotalActiveConnections(),
"limiter": sharedConnectionsLimiter.Len(),
},
})
case "dropIP":
@@ -771,6 +822,18 @@ func (this *Node) listenSock() error {
} else {
_ = cmd.ReplyOk()
}
case "accesslog":
err := sharedHTTPAccessLogViewer.Start()
if err != nil {
_ = cmd.Reply(&gosock.Command{
Code: "error",
Params: map[string]interface{}{
"message": "start failed: " + err.Error(),
},
})
} else {
_ = cmd.ReplyOk()
}
}
})
@@ -804,7 +867,7 @@ func (this *Node) onReload(config *nodeconfigs.NodeConfig) {
}
// WAF策略
sharedWAFManager.UpdatePolicies(config.FindAllFirewallPolicies())
waf.SharedWAFManager.UpdatePolicies(config.FindAllFirewallPolicies())
iplibrary.SharedActionManager.UpdateActions(config.FirewallActions)
// 统计指标
@@ -836,17 +899,6 @@ func (this *Node) onReload(config *nodeconfigs.NodeConfig) {
this.maxThreads = config.MaxThreads
}
// max tcp connections
if config.TCPMaxConnections <= 0 {
config.TCPMaxConnections = nodeconfigs.DefaultTCPMaxConnections
}
if config.TCPMaxConnections != sharedConnectionsLimiter.Count() {
remotelogs.Println("NODE", "[TCP]changed tcp max connections to '"+types.String(config.TCPMaxConnections)+"'")
sharedConnectionsLimiter.Close()
sharedConnectionsLimiter = ratelimit.NewCounter(config.TCPMaxConnections)
}
// timezone
var timeZone = config.TimeZone
if len(timeZone) == 0 {
@@ -869,4 +921,86 @@ func (this *Node) onReload(config *nodeconfigs.NodeConfig) {
if config.ProductConfig != nil {
teaconst.GlobalProductName = config.ProductConfig.Name
}
// DNS resolver
if config.DNSResolver != nil {
var err error
switch config.DNSResolver.Type {
case nodeconfigs.DNSResolverTypeGoNative:
err = os.Setenv("GODEBUG", "netdns=go")
case nodeconfigs.DNSResolverTypeCGO:
err = os.Setenv("GODEBUG", "netdns=cgo")
default:
// 默认使用go原生
err = os.Setenv("GODEBUG", "netdns=go")
}
if err != nil {
remotelogs.Error("NODE", "[DNS_RESOLVER]set env failed: "+err.Error())
}
} else {
// 默认使用go原生
err := os.Setenv("GODEBUG", "netdns=go")
if err != nil {
remotelogs.Error("NODE", "[DNS_RESOLVER]set env failed: "+err.Error())
}
}
}
// reload server config
func (this *Node) reloadServer() {
this.locker.Lock()
defer this.locker.Unlock()
if len(this.updatingServerMap) > 0 {
var updatingServerMap = this.updatingServerMap
this.updatingServerMap = map[int64]*serverconfigs.ServerConfig{}
newNodeConfig, err := nodeconfigs.CloneNodeConfig(sharedNodeConfig)
if err != nil {
remotelogs.Error("NODE", "apply server config error: "+err.Error())
return
}
for serverId, serverConfig := range updatingServerMap {
if serverConfig != nil {
newNodeConfig.AddServer(serverConfig)
} else {
newNodeConfig.RemoveServer(serverId)
}
}
err, serverErrors := newNodeConfig.Init()
if err != nil {
remotelogs.Error("NODE", "apply server config error: "+err.Error())
return
}
if len(serverErrors) > 0 {
for _, serverErr := range serverErrors {
remotelogs.ServerError(serverErr.Id, "NODE", serverErr.Message, nodeconfigs.NodeLogTypeServerConfigInitFailed, maps.Map{})
}
}
this.onReload(newNodeConfig)
err = sharedListenerManager.Start(newNodeConfig)
if err != nil {
remotelogs.Error("NODE", "apply server config error: "+err.Error())
}
}
}
func (this *Node) checkDisk() {
if runtime.GOOS == "linux" {
for _, path := range []string{
"/sys/block/vda/queue/rotational",
"/sys/block/sda/queue/rotational",
} {
data, err := ioutil.ReadFile(path)
if err != nil {
continue
}
if string(data) == "0" {
teaconst.DiskIsFast = true
}
break
}
}
}

View File

@@ -0,0 +1,9 @@
// Copyright 2022 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
//go:build !script
// +build !script
package nodes
func (this *Node) reloadCommonScripts() error {
return nil
}

View File

@@ -7,6 +7,7 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/caches"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/firewalls"
"github.com/TeaOSLab/EdgeNode/internal/monitor"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
@@ -62,7 +63,7 @@ func (this *NodeStatusExecutor) update() {
var tr = trackers.Begin("UPLOAD_NODE_STATUS")
defer tr.End()
status := &nodeconfigs.NodeStatus{}
var status = &nodeconfigs.NodeStatus{}
status.BuildVersion = teaconst.Version
status.BuildVersionCode = utils.VersionToLong(teaconst.Version)
status.OS = runtime.GOOS
@@ -74,6 +75,10 @@ func (this *NodeStatusExecutor) update() {
status.CacheTotalMemorySize = caches.SharedManager.TotalMemorySize()
status.TrafficInBytes = teaconst.InTrafficBytes
status.TrafficOutBytes = teaconst.OutTrafficBytes
var localFirewall = firewalls.Firewall()
if localFirewall != nil && !localFirewall.IsMock() {
status.LocalFirewallName = localFirewall.Name()
}
// 记录监控数据
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemConnections, maps.Map{
@@ -104,6 +109,7 @@ func (this *NodeStatusExecutor) update() {
cacheSpaceTR.End()
status.UpdatedAt = time.Now().Unix()
status.Timestamp = status.UpdatedAt
// 发送数据
jsonData, err := json.Marshal(status)
@@ -148,6 +154,7 @@ func (this *NodeStatusExecutor) updateCPU(status *nodeconfigs.NodeStatus) {
// 记录监控数据
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemCPU, maps.Map{
"usage": status.CPUUsage,
"cores": runtime.NumCPU(),
})
if this.cpuLogicalCount == 0 && this.cpuPhysicalCount == 0 {

Some files were not shown because too many files have changed in this diff Show More