Compare commits

...

115 Commits

Author SHA1 Message Date
刘祥超
462442e21a 缓存数据库升级时从老的数据库转移数据 2021-12-19 18:55:54 +08:00
刘祥超
90fcddfb9f WAF:优化get302/post307代码 2021-12-19 18:54:43 +08:00
刘祥超
8de791079c 优化代码 2021-12-19 16:54:56 +08:00
刘祥超
13b89d5971 当使用quit退出进程时,同时也禁用缓存策略 2021-12-19 14:15:17 +08:00
刘祥超
8b97638624 优化代码 2021-12-19 11:32:26 +08:00
刘祥超
79ea9e795e edge-node conns命令可以打印当前总连接数 2021-12-18 20:13:41 +08:00
刘祥超
38e06e7b03 TLS连接增加握手超时检查 2021-12-18 19:17:40 +08:00
刘祥超
f25de8d5c9 批量清除缓存时延时删除 2021-12-17 11:54:27 +08:00
刘祥超
af74500810 优化代码 2021-12-17 11:54:06 +08:00
刘祥超
189295ffcf X-Cache增加STALE状态 2021-12-17 11:53:59 +08:00
刘祥超
da09889eca 优化代码 2021-12-16 20:36:42 +08:00
刘祥超
9ffa910044 源站没有地址时也尝试Stale Cache/避免write50x()方法进入死循环 2021-12-16 17:38:07 +08:00
刘祥超
a6d711c2a0 实现stale cache读取 2021-12-16 17:27:21 +08:00
刘祥超
6bedc97c95 优化代码 2021-12-15 20:46:10 +08:00
刘祥超
4bdd1eda76 更新依赖 2021-12-15 16:57:06 +08:00
刘祥超
7fd9766565 修复未完成的代码 2021-12-15 15:11:25 +08:00
刘祥超
72983d8d86 优化代码 2021-12-15 15:09:58 +08:00
刘祥超
ec6494fa9c 优化HTTP参数 2021-12-15 13:48:48 +08:00
刘祥超
a4a6e95099 HTTP Header:实现请求方法、域名、状态码等限制,实现内容替换功能 2021-12-14 21:27:24 +08:00
刘祥超
2e11c99b7a 优化代码 2021-12-14 10:49:40 +08:00
刘祥超
014f433191 优化代码 2021-12-14 10:01:21 +08:00
刘祥超
e6ac085025 优化代码 2021-12-13 14:58:24 +08:00
刘祥超
6f60be6a00 服务最大连接数和单IP最大连接数任其一不为0则生效 2021-12-13 11:24:03 +08:00
刘祥超
dceb082a83 修改网络连接错误日志级别 2021-12-13 08:27:39 +08:00
刘祥超
9084794448 路由规则增加专属域名设置 2021-12-12 16:38:38 +08:00
刘祥超
065de8d208 在URL跳转、重写规则跳转、自动跳转到HTTPS等处增加响应Header 2021-12-12 14:10:42 +08:00
刘祥超
e5f9316e33 实现请求连接数等限制 2021-12-12 11:48:01 +08:00
刘祥超
bb5fa38613 实现全局的TCP最大连接数 2021-12-09 17:34:05 +08:00
刘祥超
ccb97b1c79 实现线程数限制 2021-12-09 12:07:59 +08:00
刘祥超
853e4fd0f0 使用空struct{}代替bool节约内存 2021-12-09 12:07:46 +08:00
刘祥超
d3169eaea5 优化代码 2021-12-08 22:19:15 +08:00
刘祥超
68b93bf6b4 可以在缓存条件里设置Expires Header 2021-12-08 17:41:39 +08:00
刘祥超
1279f0d394 优化系统goroutine使用,减少goroutine数量,增加goman查看goroutine数量指令 2021-12-08 15:17:45 +08:00
刘祥超
24fbd740b5 实现记录请求Body 2021-12-07 15:12:15 +08:00
刘祥超
5772fb2309 缓存支持请求方法设置 2021-12-07 10:43:42 +08:00
刘祥超
bf2b889c16 增加${cache.key}变量 2021-12-07 09:28:15 +08:00
刘祥超
9372bc90dd 增加${isArgs}请求变量 2021-12-06 21:47:57 +08:00
刘祥超
5b46c80431 在开发环境下打印Go语言内部HTTP调试信息 2021-12-06 19:28:26 +08:00
刘祥超
1bdb988425 启动时增加sid设置 2021-12-06 19:28:00 +08:00
刘祥超
a544a77669 自动将API节点的IP加入到白名单,防止误封
但要注意:在单个机器上安装API节点和边缘节点,通过局域网IP访问时就无法测试WAF规则,因为会被自动加入到白名单
2021-12-06 10:11:22 +08:00
刘祥超
c61108faa8 优化代码 2021-12-06 08:56:02 +08:00
刘祥超
30ac3118e2 国家/地区统计时上传流量、攻击量等信息 2021-12-05 18:57:30 +08:00
刘祥超
f0e8dd1baa 缓存增加UPDATING状态 2021-12-05 17:10:06 +08:00
刘祥超
04a327ce9a 修复源站主动关闭连接时无法缓存内容的Bug 2021-12-05 16:55:33 +08:00
刘祥超
2ac26f6aa4 回源主机名为“跟随源站”时,获得的源站主机名去除常规端口80和443 2021-12-05 09:30:45 +08:00
刘祥超
d9aac44ea3 WAF忽略客户端断开连接错误 2021-12-04 19:28:02 +08:00
刘祥超
160a1f1466 优化通过IP查询区域性能 2021-12-03 15:51:28 +08:00
刘祥超
38e2c151ec 降低ttlcache最大内存增量 2021-12-03 10:22:03 +08:00
刘祥超
9d54c17695 支持规则集忽略局域网IP 2021-12-02 16:08:25 +08:00
刘祥超
e31d68c1e1 将RPC连接错误级别从error改为warning 2021-12-02 15:14:47 +08:00
刘祥超
7ae9180bf9 多个提示页面增加请求ID、增加变量支持 2021-12-02 14:46:40 +08:00
刘祥超
424f3ae29d 优化验证码页面 2021-12-02 12:08:59 +08:00
刘祥超
ca0571a21b 增加${requestId}变量 2021-12-02 11:30:47 +08:00
刘祥超
c9bd9fd460 URL跳转时检查前后跳转的URL是否一致,防止无限跳转 2021-12-02 10:35:51 +08:00
刘祥超
8d4ec6822c 缓存支持源站设置的max-age 2021-12-02 10:19:02 +08:00
刘祥超
061253b4c3 缓存在响应中可以添加Age Header 2021-12-02 09:54:48 +08:00
刘祥超
f6dfd6acec 增加${cache.age}变量 2021-12-02 09:34:38 +08:00
刘祥超
a35aa2f520 增加是否记录499选项 2021-12-01 21:13:10 +08:00
刘祥超
ea84c41be3 因WAF规则拦截而关闭连接时,不记录499 2021-12-01 20:55:19 +08:00
刘祥超
0f0776fc1a 修复WAF OnAction在并发时无法准确调用请求动作的Bug 2021-12-01 17:43:08 +08:00
刘祥超
6aacf49764 可以上报服务相关配置错误 2021-12-01 15:52:38 +08:00
刘祥超
18a01b9b43 上传访问日志时如果出现string field contains invalid UTF-8,则重新处理后提交 2021-12-01 14:25:55 +08:00
刘祥超
32a3e08332 优化编译脚本 2021-12-01 14:24:56 +08:00
刘祥超
2397695a2d 优化服务日志 2021-11-30 16:43:58 +08:00
刘祥超
0ceebd9902 端口提示被占用时提示语中加入当前占用端口的进程名 2021-11-30 15:03:46 +08:00
刘祥超
7e62c72b79 修复TOA管理中可能出现的panic错误 2021-11-29 11:15:14 +08:00
刘祥超
b3dedbdc31 健康检查不使用密钥加密 2021-11-29 09:52:31 +08:00
刘祥超
fa967b5450 修改版本号为0.3.7 2021-11-28 14:28:24 +08:00
刘祥超
b619eb4efe 修复WAF中scheme checkpoint值为空的问题 2021-11-26 13:42:04 +08:00
刘祥超
7763f26249 修复WAF的临时白名单被当做黑名单使用的Bug 2021-11-26 10:39:04 +08:00
刘祥超
b7ae10e2d0 修复合并URL中多余分隔符时导致参数发生变化的Bug 2021-11-24 15:01:06 +08:00
刘祥超
e54eddc961 服务增加是否合并URL中的多余分隔符选项 2021-11-24 14:50:07 +08:00
刘祥超
93db9d4926 版本号改为0.3.6 2021-11-24 14:04:01 +08:00
刘祥超
8c1af3e699 修复ipset无法提前删除IP的Bug 2021-11-24 10:21:02 +08:00
刘祥超
53c74553bc 修复ipset无法提前删除IP的Bug 2021-11-24 10:20:06 +08:00
刘祥超
3eb9cade0e 修改版本为0.3.5.2 2021-11-24 10:19:36 +08:00
刘祥超
eeee3da941 暂时不删除多余的*.cache.tmp,以防产生性能问题 2021-11-21 16:10:07 +08:00
刘祥超
6af59e0bd0 优化访问日志上传 2021-11-21 10:56:54 +08:00
刘祥超
012233baf2 修复访问日志requestId可能重复的问题 2021-11-21 10:40:19 +08:00
刘祥超
ac069fd7f3 访问日志简化requestId生成方法 2021-11-21 10:27:31 +08:00
刘祥超
40cb1916c2 优化RPC客户端锁 2021-11-20 19:17:57 +08:00
刘祥超
749e0bd0b3 简化节点API配置模板 2021-11-20 18:58:08 +08:00
刘祥超
ae1a9abf5e 实现修改API节点地址的指令 2021-11-20 18:57:46 +08:00
刘祥超
7fc0394f10 修复RPC客户端管理没有加锁的问题 2021-11-20 18:57:00 +08:00
刘祥超
4a169a2dbd 修复节点拦截上报IP时没有上传服务ID的Bug 2021-11-17 19:51:37 +08:00
刘祥超
44d8afeda8 WAF规则匹配后的IP也会上报/实现IP全局名单/将名单存储到本地数据库,提升读写速度 2021-11-17 16:16:09 +08:00
刘祥超
6a0547abec IP名单中IP创建时保存相关节点、服务、WAF策略信息 2021-11-16 16:11:05 +08:00
刘祥超
6d002e2822 优化运行日志上传功能,最近N条重复的不再上传 2021-11-15 16:59:18 +08:00
刘祥超
04271d77c2 大幅提升域名匹配性能 2021-11-15 16:57:18 +08:00
刘祥超
5a6ead1dd7 优化iptables+firewall-cmd找不到时的提示 2021-11-15 09:45:03 +08:00
刘祥超
9ac7b9b2c0 内存缓存周期单位改成天 2021-11-15 09:15:23 +08:00
刘祥超
7ec916c1fb 增加注释 2021-11-14 20:46:08 +08:00
刘祥超
fadc580dff 修复IPTables+IPSet组合时在IPTables中生成了多个重复记录的Bug;增加IPSet最大值为1000000;IP范围只支持D段 2021-11-14 20:35:47 +08:00
刘祥超
fb9e9fb94b IP名单同步时在本地记录上一次同步的位置,以便于下次启动的时候不再重复执行 2021-11-14 20:34:04 +08:00
刘祥超
9c6e4bb8c1 在开发环境下运行日志显示包名 2021-11-14 20:31:49 +08:00
刘祥超
97b04777bc 实现自动将热点数据加载到内存中 2021-11-14 16:15:07 +08:00
刘祥超
4daeca912a 增加对任务的执行时间追踪工具 2021-11-14 10:55:09 +08:00
刘祥超
7e43324b53 反向代理源站错误时提示完整的URL 2021-11-14 08:58:27 +08:00
刘祥超
b9b8472c3a 缓存策略实现LFU算法/实现内存缓存自动Flush数据到磁盘 2021-11-13 21:30:24 +08:00
刘祥超
6858380bb4 节点配置支持压缩 2021-11-11 14:16:57 +08:00
刘祥超
568ecadfc6 上传流量统计的时候记录所属套餐 2021-11-10 21:52:40 +08:00
刘祥超
8210ece2b7 优化错误提示 2021-11-10 21:51:56 +08:00
刘祥超
f8160e35b9 改进流量限制 2021-11-10 14:39:02 +08:00
刘祥超
9e4a1212d2 接收请求时保留URL路径中多于的斜杠(/) 2021-11-10 10:25:02 +08:00
刘祥超
6f52cffabd 将带宽限制改为流量限制 2021-11-09 17:36:49 +08:00
刘祥超
c546b9fc7d 支持套餐日期设置 2021-11-09 15:36:05 +08:00
刘祥超
f7b961d256 规范命名 2021-11-05 15:37:07 +08:00
刘祥超
71cbb2d695 修改版本为0.3.5 2021-11-05 14:58:32 +08:00
刘祥超
2063015eeb 删除IP名单中某个IP时,也会删除WAF保存在内存中的名单中的IP 2021-11-05 14:58:10 +08:00
刘祥超
87cc43b2e0 修复firewalld无法删除规则的Bug 2021-11-05 14:39:08 +08:00
刘祥超
9812883b61 X-Cache加入跳过缓存的原因 2021-11-05 14:15:21 +08:00
刘祥超
068c20e1b9 特殊页面选择读取URL时,保留当前的状态码 2021-11-05 14:10:43 +08:00
刘祥超
17d883a2de 修改部分错误提示级别 2021-11-04 11:14:27 +08:00
刘祥超
083bbb1460 支持info指令查询PID、版本号等信息 2021-11-04 11:14:02 +08:00
刘祥超
4f7b9f4fc6 修改版本为0.3.4 2021-11-01 10:45:33 +08:00
172 changed files with 5861 additions and 1222 deletions

View File

@@ -5,6 +5,7 @@ function build() {
NAME="edge-node"
VERSION=$(lookup-version $ROOT/../internal/const/const.go)
DIST=$ROOT/"../dist/${NAME}"
MUSL_DIR="/usr/local/opt/musl-cross/bin"
OS=${1}
ARCH=${2}
TAG=${3}
@@ -53,7 +54,6 @@ function build() {
echo "building ..."
MUSL_DIR="/usr/local/opt/musl-cross/bin"
CC_PATH=""
CXX_PATH=""
if [[ `uname -a` == *"Darwin"* && "${OS}" == "linux" ]]; then

View File

@@ -1,4 +1,4 @@
rpc:
endpoints: [ ${endpoints} ]
nodeId: "${nodeId}"
secret: "${nodeSecret}"
endpoints: [ "" ]
nodeId: ""
secret: ""

View File

@@ -1,6 +1,7 @@
package main
import (
"encoding/json"
"fmt"
"github.com/TeaOSLab/EdgeNode/internal/apps"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
@@ -11,6 +12,7 @@ import (
"net/http"
_ "net/http/pprof"
"os"
"sort"
)
func main() {
@@ -60,6 +62,61 @@ func main() {
node := nodes.NewNode()
node.Start()
})
app.On("trackers", func() {
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{Code: "trackers"})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
labelsMap, ok := reply.Params["labels"]
if ok {
labels, ok := labelsMap.(map[string]interface{})
if ok {
if len(labels) == 0 {
fmt.Println("no labels yet")
} else {
var labelNames = []string{}
for label := range labels {
labelNames = append(labelNames, label)
}
sort.Strings(labelNames)
for _, labelName := range labelNames {
fmt.Println(labelName + ": " + fmt.Sprintf("%.6f", labels[labelName]))
}
}
}
}
}
})
app.On("goman", func() {
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{Code: "goman"})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
instancesJSON, err := json.MarshalIndent(reply.Params, "", " ")
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
fmt.Println(string(instancesJSON))
}
}
})
app.On("conns", func() {
var sock = gosock.NewTmpSock(teaconst.ProcessName)
reply, err := sock.Send(&gosock.Command{Code: "conns"})
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
resultJSON, err := json.MarshalIndent(reply.Params, "", " ")
if err != nil {
fmt.Println("[ERROR]" + err.Error())
} else {
fmt.Println(string(resultJSON))
}
}
})
app.Run(func() {
node := nodes.NewNode()
node.Start()

25
go.mod
View File

@@ -5,7 +5,6 @@ go 1.15
replace github.com/TeaOSLab/EdgeCommon => ../EdgeCommon
require (
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect
github.com/TeaOSLab/EdgeCommon v0.0.0-00010101000000-000000000000
github.com/andybalholm/brotli v1.0.3
github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670
@@ -13,26 +12,24 @@ require (
github.com/chai2010/webp v1.1.0 // indirect
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f
github.com/dop251/goja v0.0.0-20210804101310-32956a348b49
github.com/go-ole/go-ole v1.2.4 // indirect
github.com/go-yaml/yaml v2.1.0+incompatible
github.com/golang/protobuf v1.5.2
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e // indirect
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible
github.com/mattn/go-sqlite3 v2.0.3+incompatible
github.com/mattn/go-sqlite3 v1.14.9
github.com/miekg/dns v1.1.43
github.com/mssola/user_agent v0.5.2
github.com/mssola/user_agent v0.5.3
github.com/pires/go-proxyproto v0.6.1
github.com/shirou/gopsutil v3.21.5+incompatible
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/tklauser/go-sysconf v0.3.6 // indirect
golang.org/x/image v0.0.0-20190802002840-cff245a6509b
golang.org/x/net v0.0.0-20210614182718-04defd469f4e
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22
golang.org/x/text v0.3.6
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced // indirect
google.golang.org/grpc v1.38.0
golang.org/x/image v0.0.0-20211028202545-6944b10bf410
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9
golang.org/x/sys v0.0.0-20211214234402-4825e8c3871d
golang.org/x/text v0.3.7
golang.org/x/tools v0.1.3 // indirect
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa // indirect
google.golang.org/grpc v1.43.0
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
)

54
go.sum
View File

@@ -1,4 +1,5 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
@@ -9,6 +10,7 @@ github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUW
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg=
github.com/andybalholm/brotli v1.0.3 h1:fpcw+r1N1h0Poc1F/pHbW40cUm/lMEQslZtCkBQ0UnM=
github.com/andybalholm/brotli v1.0.3/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/benbjohnson/clock v1.0.3/go.mod h1:bGMdMPoPVvcYyt1gHDf4J2KE153Yf9BuiUKYMaxlTDM=
github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670 h1:FQPKKjDhzG0T4ew6dm6MGrXb4PRAi8ZmTuYuxcF62BM=
github.com/biessek/golang-ico v0.0.0-20180326222316-d348d9ea4670/go.mod h1:iRWAFbKXMMkVQyxZ1PfGlkBr1TjATx1zy2MRprV7A3Q=
@@ -22,6 +24,11 @@ github.com/chai2010/webp v1.1.0/go.mod h1:LP12PG5IFmLGHUU26tBiCBKnghxx3toZFwDjOY
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XPnfI9Z40ddWsdw2W/uZgQLFXToKeRcDiI=
github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
@@ -37,10 +44,14 @@ github.com/dop251/goja_nodejs v0.0.0-20210225215109-d91c329300e7/go.mod h1:hn7BA
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-ole/go-ole v1.2.4 h1:nNBDSCOigTSiarFpYE9J/KtEA1IOW4CNeqT9TQDqCxI=
github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM=
@@ -63,6 +74,7 @@ github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:W
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
@@ -75,9 +87,12 @@ github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060 h1:qdLtK4PDXxk2vMKkTWl5Fl9xqYuRCukzWAgJbLHdfOo=
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060/go.mod h1:KU4mS7QNiZ7QWEuDBk1zw0/Q2LrAPZv3tycEFBsuUwc=
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24 h1:1cGulkD2SNJJRok5OKwyhP/Ddm+PgSWKOupn0cR36/A=
github.com/iwind/TeaGo v0.0.0-20211026123858-7de7a21cad24/go.mod h1:KU4mS7QNiZ7QWEuDBk1zw0/Q2LrAPZv3tycEFBsuUwc=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11 h1:DaQjoWZhLNxjhIXedVg4/vFEtHkZhK4IjIwsWdyzBLg=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11/go.mod h1:JtbX20untAjUVjZs1ZBtq80f5rJWvwtQNRL6EnuYRnY=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3 h1:aBSonas7vFcgTj9u96/bWGILGv1ZbUSTLiOzcI1ZT6c=
@@ -85,6 +100,8 @@ github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3/go.mod h1:H5Q7SXwbx3a
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8 h1:AojsHz9Es9B3He2MQQxeRq3TyD//o9huxUo7r1wh44g=
github.com/iwind/gowebp v0.0.0-20211029040624-7331ecc78ed8/go.mod h1:QJBY2txYhLMzwLO29iB5ujDJ3s3V7DsZ582nw4Ss+tM=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e h1:LvL4XsI70QxOGHed6yhQtAU34Kx3Qq2wwBzGFKY8zKk=
github.com/jsummers/gobmp v0.0.0-20151104160322-e2ba15ffa76e/go.mod h1:kLgvv7o6UM+0QSf0QjAse3wReFDsb9qbZJdfexWlrQw=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
@@ -96,14 +113,22 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible h1:1qp9iks+69h7IGLazAplzS9Ca14HAxuD5c0rbFdPGy4=
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible/go.mod h1:+ZBN7PBoh5gG6/y0ZQ85vJDBe21WnfbRrQQwTfliJJI=
github.com/mattn/go-sqlite3 v1.14.9 h1:10HX2Td0ocZpYEjhilsuo6WWtUqttj2Kb0KtD86/KYA=
github.com/mattn/go-sqlite3 v1.14.9/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mattn/go-sqlite3 v2.0.3+incompatible h1:gXHsfypPkaMZrKbD5209QV9jbUTJKjyR5WD3HYQSd+U=
github.com/mattn/go-sqlite3 v2.0.3+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/miekg/dns v1.1.43 h1:JKfpVSCB84vrAmHzyrsxB5NAr5kLoMXZArPSw7Qlgyg=
github.com/miekg/dns v1.1.43/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/mssola/user_agent v0.5.2 h1:CZkTUahjL1+OcZ5zv3kZr8QiJ8jy2H08vZIEkBeRbxo=
github.com/mssola/user_agent v0.5.2/go.mod h1:TTPno8LPY3wAIEKRpAtkdMT0f8SE24pLRGPahjCH4uw=
github.com/mssola/user_agent v0.5.3 h1:lBRPML9mdFuIZgI2cmlQ+atbpJdLdeVl2IDodjBR578=
github.com/mssola/user_agent v0.5.3/go.mod h1:TTPno8LPY3wAIEKRpAtkdMT0f8SE24pLRGPahjCH4uw=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
@@ -119,8 +144,11 @@ github.com/pires/go-proxyproto v0.6.1/go.mod h1:Odh9VFOZJCf9G8cLW5o435Xf1J95Jw9G
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/shirou/gopsutil v3.21.5+incompatible h1:OloQyEerMi7JUrXiNzy8wQ5XN+baemxSl12QgIzt0jc=
github.com/shirou/gopsutil v3.21.5+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shirou/gopsutil v3.21.11+incompatible h1:+1+c1VGhc88SSonWP6foOcLhvnKlUeu/erjjvaPEYiI=
github.com/shirou/gopsutil v3.21.11+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72 h1:qLC7fQah7D6K1B0ujays3HV9gkFtllcxhzImRR7ArPQ=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -128,20 +156,25 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/tklauser/go-sysconf v0.3.6 h1:oc1sJWvKkmvIxhDHeKWvZS4f6AW+YcoguSfRF2/Hmo4=
github.com/tklauser/go-sysconf v0.3.6/go.mod h1:MkWzOF4RMCshBAMXuhXJs64Rte09mITnppBXY/rYEFI=
github.com/tklauser/numcpus v0.2.2 h1:oyhllyrScuYI6g+h/zUvNXNp1wy7x8qQy3t/piefldA=
github.com/tklauser/numcpus v0.2.2/go.mod h1:x3qojaO3uyYt0i56EW/VUYs7uBvdl2fkfZFu0T9wgjM=
github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
go.opentelemetry.io/otel v0.7.0/go.mod h1:aZMyHG5TqDOXEgH2tyLiXSUKly1jT3yqE9PmrzIeCdo=
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20200513190911-00229845015e/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b h1:+qEpEAPhDZ1o0x3tHzZTQDArnOixOzGD9HUJfcg0mb4=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.0.0-20211028202545-6944b10bf410 h1:hTftEOvwiOq2+O8k2D5/Q7COC7k5Qcrgc2TFURJYnvQ=
golang.org/x/image v0.0.0-20211028202545-6944b10bf410/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
@@ -153,19 +186,25 @@ golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210614182718-04defd469f4e h1:XpT3nA5TvE525Ne3hInMh6+GETgn27Zfm9dxsThnX2Q=
golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9 h1:kmreh1vGI63l2FxOAYS3Yv6ATsi7lSTuwNSVbGfJV9I=
golang.org/x/net v0.0.0-20211215060638-4ddde0e984e9/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -189,6 +228,8 @@ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22 h1:RqytpXGR1iVNX7psjB3ff8y7sNFinVFvkx1c8SjBkio=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211214234402-4825e8c3871d h1:1oIt9o40TWWI9FUaveVpUvBe13FNqBNVXy3ue2fcfkw=
golang.org/x/sys v0.0.0-20211214234402-4825e8c3871d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
@@ -196,6 +237,8 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
@@ -215,17 +258,25 @@ google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20191009194640-548a555dbc03/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced h1:c5geK1iMU3cDKtFrCVQIcjR3W+JOZMuhIyICMCTbtus=
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced/go.mod h1:SzzZ/N+nwJDaO1kznhnlzqS8ocJICar6hYhVyhi++24=
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa h1:I0YcKz0I7OAhddo7ya8kMnvprhcWM045PmkBdMO9zN0=
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.32.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0=
google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/grpc v1.38.0 h1:/9BgsAsa5nWe26HqOlvlgJnqBuktYOLCgjCPqsa56W0=
google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
google.golang.org/grpc v1.43.0 h1:Eeu7bZtDZ2DpRCsLhUlcrLnvYaMK1Gz86a+hMVvELmM=
google.golang.org/grpc v1.43.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -238,6 +289,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -247,6 +300,7 @@ gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMy
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.3/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

View File

@@ -11,6 +11,7 @@ import (
"os/exec"
"runtime"
"strconv"
"syscall"
"time"
)
@@ -187,6 +188,11 @@ func (this *AppCmd) runStart() {
_ = os.Setenv("EdgeBackground", "on")
cmd := exec.Command(os.Args[0])
cmd.SysProcAttr = &syscall.SysProcAttr{
Foreground: false,
Setsid: true,
}
err := cmd.Start()
if err != nil {
fmt.Println(this.product+" start failed:", err.Error())

View File

@@ -7,9 +7,9 @@ import (
"github.com/iwind/TeaGo/utils/time"
"log"
"os"
"path/filepath"
"runtime"
"strconv"
"strings"
)
type LogWriter struct {
@@ -38,18 +38,20 @@ func (this *LogWriter) Init() {
}
func (this *LogWriter) Write(message string) {
// 文件和行号
var callDepth = 2
var file string
var line int
var ok bool
_, file, line, ok = runtime.Caller(callDepth)
if ok {
file = filepath.Base(file)
}
backgroundEnv, _ := os.LookupEnv("EdgeBackground")
if backgroundEnv != "on" {
// 文件和行号
var file string
var line int
if Tea.IsTesting() {
var callDepth = 3
var ok bool
_, file, line, ok = runtime.Caller(callDepth)
if ok {
file = this.packagePath(file)
}
}
if len(file) > 0 {
log.Println(message + " (" + file + ":" + strconv.Itoa(line) + ")")
} else {
@@ -70,3 +72,11 @@ func (this *LogWriter) Close() {
_ = this.fileAppender.Close()
}
}
func (this *LogWriter) packagePath(path string) string {
var pieces = strings.Split(path, "/")
if len(pieces) >= 2 {
return strings.Join(pieces[len(pieces)-2:], "/")
}
return path
}

View File

@@ -0,0 +1,10 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
type HotItem struct {
Key string
ExpiresAt int64
Hits uint32
Status int
}

View File

@@ -2,6 +2,7 @@ package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"time"
)
type ItemType = int
@@ -11,15 +12,26 @@ const (
ItemTypeMemory ItemType = 2
)
// 计算当前周
// 不要用YW因为需要计算两周是否临近
func currentWeek() int32 {
return int32(time.Now().Unix() / 86400)
}
type Item struct {
Type ItemType `json:"type"`
Key string `json:"key"`
ExpiredAt int64 `json:"expiredAt"`
StaleAt int64 `json:"staleAt"`
HeaderSize int64 `json:"headerSize"`
BodySize int64 `json:"bodySize"`
MetaSize int64 `json:"metaSize"`
Host string `json:"host"` // 主机名
ServerId int64 `json:"serverId"` // 服务ID
Week1Hits int64 `json:"week1Hits"`
Week2Hits int64 `json:"week2Hits"`
Week int32 `json:"week"`
}
func (this *Item) IsExpired() bool {
@@ -27,9 +39,23 @@ func (this *Item) IsExpired() bool {
}
func (this *Item) TotalSize() int64 {
return this.Size() + this.MetaSize + int64(len(this.Key)) + int64(len(this.Host)) + 64
return this.Size() + this.MetaSize + int64(len(this.Key)) + int64(len(this.Host))
}
func (this *Item) Size() int64 {
return this.HeaderSize + this.BodySize
}
func (this *Item) IncreaseHit(week int32) {
if this.Week == week {
this.Week2Hits++
} else {
if week-this.Week == 1 {
this.Week1Hits = this.Week2Hits
} else {
this.Week1Hits = 0
}
this.Week2Hits = 1
this.Week = week
}
}

View File

@@ -0,0 +1,83 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
"runtime"
"testing"
"time"
)
func TestItem_IncreaseHit(t *testing.T) {
var week = currentWeek()
var item = &Item{}
//item.Week = 2704
item.Week2Hits = 100
item.IncreaseHit(week)
t.Log("week:", item.Week, "week1:", item.Week1Hits, "week2:", item.Week2Hits)
item.IncreaseHit(week)
t.Log("week:", item.Week, "week1:", item.Week1Hits, "week2:", item.Week2Hits)
}
func TestItems_Memory(t *testing.T) {
var stat = &runtime.MemStats{}
runtime.ReadMemStats(stat)
var memory1 = stat.HeapInuse
var items = []*Item{}
for i := 0; i < 10_000_000; i++ {
items = append(items, &Item{
Key: types.String(i),
})
}
runtime.ReadMemStats(stat)
var memory2 = stat.HeapInuse
t.Log(memory1, memory2, (memory2-memory1)/1024/1024, "M")
var weekItems = make(map[string]*Item, 10_000_000)
for _, item := range items {
weekItems[item.Key] = item
}
runtime.ReadMemStats(stat)
var memory3 = stat.HeapInuse
t.Log(memory2, memory3, (memory3-memory2)/1024/1024, "M")
time.Sleep(1 * time.Second)
t.Log(len(items), len(weekItems))
}
func TestItems_Memory2(t *testing.T) {
var stat = &runtime.MemStats{}
runtime.ReadMemStats(stat)
var memory1 = stat.HeapInuse
var items = map[int32]map[string]zero.Zero{}
for i := 0; i < 10_000_000; i++ {
var week = int32((time.Now().Unix() - int64(86400*rands.Int(0, 300))) / (86400 * 7))
m, ok := items[week]
if !ok {
m = map[string]zero.Zero{}
items[week] = m
}
m[types.String(int64(i)*1_000_000)] = zero.New()
}
runtime.ReadMemStats(stat)
var memory2 = stat.HeapInuse
t.Log(memory1, memory2, (memory2-memory1)/1024/1024, "M")
time.Sleep(1 * time.Second)
for w, i := range items {
t.Log(w, len(i))
}
}

View File

@@ -4,13 +4,16 @@ package caches
import (
"database/sql"
"errors"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/ttlcache"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
timeutil "github.com/iwind/TeaGo/utils/time"
_ "github.com/mattn/go-sqlite3"
"os"
"strconv"
"sync/atomic"
"time"
)
@@ -24,6 +27,7 @@ type FileList struct {
onAdd func(item *Item)
onRemove func(item *Item)
// cacheItems
existsByHashStmt *sql.Stmt // 根据hash检查是否存在
insertStmt *sql.Stmt // 写入数据
selectByHashStmt *sql.Stmt // 使用hash查询数据
@@ -32,10 +36,18 @@ type FileList struct {
purgeStmt *sql.Stmt // 清理
deleteAllStmt *sql.Stmt // 删除所有数据
// hits
insertHitStmt *sql.Stmt // 写入数据
increaseHitStmt *sql.Stmt // 增加点击量
deleteHitByHashStmt *sql.Stmt // 根据hash删除数据
lfuHitsStmt *sql.Stmt // 读取老的数据
oldTables []string
itemsTableName string
hitsTableName string
isClosed bool
isReady bool
memoryCache *ttlcache.Cache
}
@@ -58,28 +70,24 @@ func (this *FileList) Init() error {
remotelogs.Println("CACHE", "create cache dir '"+this.dir+"'")
}
this.itemsTableName = "cacheItems_v2"
this.itemsTableName = "cacheItems_v3"
this.hitsTableName = "hits"
var dir = this.dir
if dir == "/" {
// 防止sqlite提示authority错误
dir = ""
}
db, err := sql.Open("sqlite3", "file:"+dir+"/index.db?cache=shared&mode=rwc&_journal_mode=WAL")
var dbPath = dir + "/index.db"
remotelogs.Println("CACHE", "loading database '"+dbPath+"'")
db, err := sql.Open("sqlite3", "file:"+dbPath+"?cache=shared&mode=rwc&_journal_mode=WAL")
if err != nil {
return err
return errors.New("open database failed: " + err.Error())
}
db.SetMaxOpenConns(1)
this.db = db
// 清除旧表
this.oldTables = []string{
"cacheItems",
}
err = this.removeOldTables()
if err != nil {
remotelogs.Warn("CACHE", "clean old tables failed: "+err.Error())
}
db.SetMaxOpenConns(1)
this.db = db
// TODO 耗时过长,暂时不整理数据库
/**_, err = db.Exec("VACUUM")
@@ -90,7 +98,18 @@ func (this *FileList) Init() error {
// 创建
err = this.initTables(db, 1)
if err != nil {
return err
return errors.New("init tables failed: " + err.Error())
}
// 清除旧表
// 这个一定要在initTables()之后,因为老的数据需要转移
this.oldTables = []string{
"cacheItems",
"cacheItems_v2",
}
err = this.removeOldTables()
if err != nil {
remotelogs.Warn("CACHE", "clean old tables failed: "+err.Error())
}
// 读取总数量
@@ -111,7 +130,7 @@ func (this *FileList) Init() error {
return err
}
this.insertStmt, err = this.db.Prepare(`INSERT INTO "` + this.itemsTableName + `" ("hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "host", "serverId", "createdAt") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`)
this.insertStmt, err = this.db.Prepare(`INSERT INTO "` + this.itemsTableName + `" ("hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "staleAt", "host", "serverId", "createdAt") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
return err
}
@@ -131,7 +150,7 @@ func (this *FileList) Init() error {
return err
}
this.purgeStmt, err = this.db.Prepare(`SELECT "hash" FROM "` + this.itemsTableName + `" WHERE expiredAt<=? LIMIT ?`)
this.purgeStmt, err = this.db.Prepare(`SELECT "hash" FROM "` + this.itemsTableName + `" WHERE staleAt<=? LIMIT ?`)
if err != nil {
return err
}
@@ -141,6 +160,25 @@ func (this *FileList) Init() error {
return err
}
this.insertHitStmt, err = this.db.Prepare(`INSERT INTO "` + this.hitsTableName + `" ("hash", "week2Hits", "week") VALUES (?, 1, ?)`)
this.increaseHitStmt, err = this.db.Prepare(`INSERT INTO "` + this.hitsTableName + `" ("hash", "week2Hits", "week") VALUES (?, 1, ?) ON CONFLICT("hash") DO UPDATE SET "week1Hits"=IIF("week"=?, "week1Hits", "week2Hits"), "week2Hits"=IIF("week"=?, "week2Hits"+1, 1), "week"=?`)
if err != nil {
return err
}
this.deleteHitByHashStmt, err = this.db.Prepare(`DELETE FROM "` + this.hitsTableName + `" WHERE "hash"=?`)
if err != nil {
return err
}
this.lfuHitsStmt, err = this.db.Prepare(`SELECT "hash" FROM "` + this.hitsTableName + `" ORDER BY "week" ASC, "week1Hits"+"week2Hits" ASC LIMIT ?`)
if err != nil {
return err
}
this.isReady = true
return nil
}
@@ -150,11 +188,20 @@ func (this *FileList) Reset() error {
}
func (this *FileList) Add(hash string, item *Item) error {
if this.isClosed {
if !this.isReady {
return nil
}
_, err := this.insertStmt.Exec(hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.Host, item.ServerId, utils.UnixTime())
if item.StaleAt == 0 {
item.StaleAt = item.ExpiredAt
}
_, err := this.insertStmt.Exec(hash, item.Key, item.HeaderSize, item.BodySize, item.MetaSize, item.ExpiredAt, item.StaleAt, item.Host, item.ServerId, utils.UnixTime())
if err != nil {
return err
}
_, err = this.insertHitStmt.Exec(hash, timeutil.Format("YW"))
if err != nil {
return err
}
@@ -168,7 +215,7 @@ func (this *FileList) Add(hash string, item *Item) error {
}
func (this *FileList) Exist(hash string) (bool, error) {
if this.isClosed {
if !this.isReady {
return false, nil
}
@@ -198,7 +245,7 @@ func (this *FileList) Exist(hash string) (bool, error) {
// CleanPrefix 清理某个前缀的缓存数据
func (this *FileList) CleanPrefix(prefix string) error {
if this.isClosed {
if !this.isReady {
return nil
}
@@ -211,8 +258,9 @@ func (this *FileList) CleanPrefix(prefix string) error {
}()
var count = int64(10000)
var staleLife = 600 // TODO 需要可以设置
for {
result, err := this.db.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0 WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)=1 LIMIT `+strconv.FormatInt(count, 10)+`)`, utils.UnixTime(), prefix)
result, err := this.db.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0,staleAt=? WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)=1 LIMIT `+types.String(count)+`)`, utils.UnixTime()+int64(staleLife), utils.UnixTime(), prefix)
if err != nil {
return err
}
@@ -227,7 +275,7 @@ func (this *FileList) CleanPrefix(prefix string) error {
}
func (this *FileList) Remove(hash string) error {
if this.isClosed {
if !this.isReady {
return nil
}
@@ -253,6 +301,11 @@ func (this *FileList) Remove(hash string) error {
return err
}
_, err = this.deleteHitByHashStmt.Exec(hash)
if err != nil {
return err
}
atomic.AddInt64(&this.total, -1)
if this.onRemove != nil {
@@ -265,9 +318,9 @@ func (this *FileList) Remove(hash string) error {
// Purge 清理过期的缓存
// count 每次遍历的最大数量,控制此数字可以保证每次清理的时候不用花太多时间
// callback 每次发现过期key的调用
func (this *FileList) Purge(count int, callback func(hash string) error) error {
if this.isClosed {
return nil
func (this *FileList) Purge(count int, callback func(hash string) error) (int, error) {
if !this.isReady {
return 0, nil
}
if count <= 0 {
@@ -275,11 +328,56 @@ func (this *FileList) Purge(count int, callback func(hash string) error) error {
}
rows, err := this.purgeStmt.Query(time.Now().Unix(), count)
if err != nil {
return 0, err
}
hashStrings := []string{}
var countFound = 0
for rows.Next() {
var hash string
err = rows.Scan(&hash)
if err != nil {
_ = rows.Close()
return 0, err
}
hashStrings = append(hashStrings, hash)
countFound++
}
_ = rows.Close() // 不能使用defer防止读写冲突
// 不在 rows.Next() 循环中操作是为了避免死锁
for _, hash := range hashStrings {
err = this.Remove(hash)
if err != nil {
return 0, err
}
err = callback(hash)
if err != nil {
return 0, err
}
}
return countFound, nil
}
func (this *FileList) PurgeLFU(count int, callback func(hash string) error) error {
if !this.isReady {
return nil
}
if count <= 0 {
return nil
}
rows, err := this.lfuHitsStmt.Query(count)
if err != nil {
return err
}
hashStrings := []string{}
var countFound = 0
for rows.Next() {
var hash string
err = rows.Scan(&hash)
@@ -288,6 +386,7 @@ func (this *FileList) Purge(count int, callback func(hash string) error) error {
return err
}
hashStrings = append(hashStrings, hash)
countFound++
}
_ = rows.Close() // 不能使用defer防止读写冲突
@@ -303,12 +402,11 @@ func (this *FileList) Purge(count int, callback func(hash string) error) error {
return err
}
}
return nil
}
func (this *FileList) CleanAll() error {
if this.isClosed {
if !this.isReady {
return nil
}
@@ -323,7 +421,7 @@ func (this *FileList) CleanAll() error {
}
func (this *FileList) Stat(check func(hash string) bool) (*Stat, error) {
if this.isClosed {
if !this.isReady {
return &Stat{}, nil
}
@@ -347,6 +445,13 @@ func (this *FileList) Count() (int64, error) {
return atomic.LoadInt64(&this.total), nil
}
// IncreaseHit 增加点击量
func (this *FileList) IncreaseHit(hash string) error {
var week = timeutil.Format("YW")
_, err := this.increaseHitStmt.Exec(hash, week, week, week, week)
return err
}
// OnAdd 添加事件
func (this *FileList) OnAdd(f func(item *Item)) {
this.onAdd = f
@@ -359,6 +464,7 @@ func (this *FileList) OnRemove(f func(item *Item)) {
func (this *FileList) Close() error {
this.isClosed = true
this.isReady = false
this.memoryCache.Destroy()
@@ -371,6 +477,11 @@ func (this *FileList) Close() error {
_ = this.purgeStmt.Close()
_ = this.deleteAllStmt.Close()
_ = this.insertHitStmt.Close()
_ = this.increaseHitStmt.Close()
_ = this.deleteHitByHashStmt.Close()
_ = this.lfuHitsStmt.Close()
return this.db.Close()
}
return nil
@@ -378,7 +489,17 @@ func (this *FileList) Close() error {
// 初始化
func (this *FileList) initTables(db *sql.DB, times int) error {
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
// 检查是否存在
_, err := db.Exec(`SELECT id FROM "` + this.itemsTableName + `" LIMIT 1`)
var notFound = false
if err != nil {
notFound = true
}
{
// expiredAt - 过期时间,用来判断有无过期
// staleAt - 陈旧最大时间,用来清理缓存
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"key" varchar(1024),
@@ -386,6 +507,7 @@ func (this *FileList) initTables(db *sql.DB, times int) error {
"bodySize" integer DEFAULT 0,
"metaSize" integer DEFAULT 0,
"expiredAt" integer DEFAULT 0,
"staleAt" integer DEFAULT 0,
"createdAt" integer DEFAULT 0,
"host" varchar(128),
"serverId" integer
@@ -401,6 +523,11 @@ ON "` + this.itemsTableName + `" (
"expiredAt" ASC
);
CREATE INDEX IF NOT EXISTS "staleAt"
ON "` + this.itemsTableName + `" (
"staleAt" ASC
);
CREATE UNIQUE INDEX IF NOT EXISTS "hash"
ON "` + this.itemsTableName + `" (
"hash" ASC
@@ -411,17 +538,60 @@ ON "` + this.itemsTableName + `" (
"serverId" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.itemsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.itemsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
}
return err
}
return err
}
}
return err
// 如果数据为空,从老数据中加载数据
if notFound {
// v2 => v3
remotelogs.Println("CACHE", "transferring old data from v2 to v3 ...")
result, err := db.Exec(`INSERT INTO "` + this.itemsTableName + `" ("id", "hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "createdAt", "host", "serverId", "staleAt") SELECT "id", "hash", "key", "headerSize", "bodySize", "metaSize", "expiredAt", "createdAt", "host", "serverId", "expiredAt"+600 FROM cacheItems_v2`)
if err != nil {
remotelogs.Println("CACHE", "transfer old data from v2 to v3 failed: "+err.Error())
} else {
count, _ := result.RowsAffected()
remotelogs.Println("CACHE", "transfer old data from v2 to v3 finished, "+types.String(count)+" rows transferred")
}
}
{
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.hitsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"week1Hits" integer DEFAULT 0,
"week2Hits" integer DEFAULT 0,
"week" varchar(6)
);
CREATE UNIQUE INDEX IF NOT EXISTS "hits_hash"
ON "` + this.hitsTableName + `" (
"hash" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.hitsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
}
return err
}
return err
}
}
return nil
@@ -444,11 +614,11 @@ func (this *FileList) removeOldTables() error {
}
if lists.ContainsString(this.oldTables, name) {
// 异步执行
go func() {
goman.New(func() {
remotelogs.Println("CACHE", "remove old table '"+name+"' ...")
_, _ = this.db.Exec(`DROP TABLE "` + name + `"`)
remotelogs.Println("CACHE", "remove old table '"+name+"' done")
}()
})
}
}

View File

@@ -3,8 +3,10 @@
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
stringutil "github.com/iwind/TeaGo/utils/string"
"strconv"
"sync"
@@ -126,7 +128,7 @@ func TestFileList_Exist_Many_DB(t *testing.T) {
}()
for i := 0; i < threads; i++ {
go func() {
goman.New(func() {
defer wg.Done()
for {
@@ -142,7 +144,7 @@ func TestFileList_Exist_Many_DB(t *testing.T) {
return
}
}
}()
})
}
wg.Wait()
t.Log("left:", count)
@@ -184,7 +186,7 @@ func TestFileList_Purge(t *testing.T) {
if err != nil {
t.Fatal(err)
}
err = list.Purge(2, func(hash string) error {
_, err = list.Purge(2, func(hash string) error {
t.Log(hash)
return nil
})
@@ -257,6 +259,50 @@ func TestFileList_Conflict(t *testing.T) {
t.Log("after exists")
}
func TestFileList_IIF(t *testing.T) {
list := NewFileList(Tea.Root + "/data").(*FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
rows, err := list.db.Query("SELECT IIF(0, 2, 3)")
if err != nil {
t.Fatal(err)
}
defer func() {
_ = rows.Close()
}()
if rows.Next() {
var result int
err = rows.Scan(&result)
if err != nil {
t.Fatal(err)
}
t.Log("result:", result)
}
}
func TestFileList_IncreaseHit(t *testing.T) {
list := NewFileList(Tea.Root + "/data")
err := list.Init()
if err != nil {
t.Fatal(err)
}
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
for i := 0; i < 1000_000; i++ {
err = list.IncreaseHit(stringutil.Md5("abc" + types.String(i)))
}
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func BenchmarkFileList_Exist(b *testing.B) {
list := NewFileList(Tea.Root + "/data")
err := list.Init()

View File

@@ -22,7 +22,10 @@ type ListInterface interface {
Remove(hash string) error
// Purge 清理过期数据
Purge(count int, callback func(hash string) error) error
Purge(count int, callback func(hash string) error) (int, error)
// PurgeLFU 清理LFU数据
PurgeLFU(count int, callback func(hash string) error) error
// CleanAll 清除所有缓存
CleanAll() error
@@ -41,4 +44,7 @@ type ListInterface interface {
// Close 关闭
Close() error
// IncreaseHit 增加点击量
IncreaseHit(hash string) error
}

View File

@@ -1,16 +1,24 @@
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/logs"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
)
// MemoryList 内存缓存列表管理
type MemoryList struct {
count int64
itemMaps map[string]map[string]*Item // prefix => { hash => item }
weekItemMaps map[int32]map[string]zero.Zero // week => { hash => Zero }
minWeek int32
prefixes []string
locker sync.RWMutex
onAdd func(item *Item)
@@ -21,7 +29,9 @@ type MemoryList struct {
func NewMemoryList() ListInterface {
return &MemoryList{
itemMaps: map[string]map[string]*Item{},
itemMaps: map[string]map[string]*Item{},
weekItemMaps: map[int32]map[string]zero.Zero{},
minWeek: currentWeek(),
}
}
@@ -43,11 +53,19 @@ func (this *MemoryList) Reset() error {
for key := range this.itemMaps {
this.itemMaps[key] = map[string]*Item{}
}
this.weekItemMaps = map[int32]map[string]zero.Zero{}
this.locker.Unlock()
atomic.StoreInt64(&this.count, 0)
return nil
}
func (this *MemoryList) Add(hash string, item *Item) error {
if item.Week == 0 {
item.Week = currentWeek()
}
this.locker.Lock()
prefix := this.prefix(hash)
@@ -60,9 +78,20 @@ func (this *MemoryList) Add(hash string, item *Item) error {
// 先删除,为了可以正确触发统计
oldItem, ok := itemMap[hash]
if ok {
// 从week map中删除
if oldItem.Week > 0 {
wm, ok := this.weekItemMaps[oldItem.Week]
if ok {
delete(wm, hash)
}
}
// 回调
if this.onRemove != nil {
this.onRemove(oldItem)
}
} else {
atomic.AddInt64(&this.count, 1)
}
// 添加
@@ -71,6 +100,15 @@ func (this *MemoryList) Add(hash string, item *Item) error {
}
itemMap[hash] = item
// week map
wm, ok := this.weekItemMaps[item.Week]
if ok {
wm[hash] = zero.New()
} else {
this.weekItemMaps[item.Week] = map[string]zero.Zero{hash: zero.New()}
}
this.locker.Unlock()
return nil
}
@@ -122,7 +160,17 @@ func (this *MemoryList) Remove(hash string) error {
if this.onRemove != nil {
this.onRemove(item)
}
atomic.AddInt64(&this.count, -1)
delete(itemMap, hash)
// week map
if item.Week > 0 {
wm, ok := this.weekItemMaps[item.Week]
if ok {
delete(wm, hash)
}
}
}
this.locker.Unlock()
@@ -132,7 +180,7 @@ func (this *MemoryList) Remove(hash string) error {
// Purge 清理过期的缓存
// count 每次遍历的最大数量,控制此数字可以保证每次清理的时候不用花太多时间
// callback 每次发现过期key的调用
func (this *MemoryList) Purge(count int, callback func(hash string) error) error {
func (this *MemoryList) Purge(count int, callback func(hash string) error) (int, error) {
this.locker.Lock()
deletedHashList := []string{}
@@ -146,8 +194,9 @@ func (this *MemoryList) Purge(count int, callback func(hash string) error) error
itemMap, ok := this.itemMaps[prefix]
if !ok {
this.locker.Unlock()
return nil
return 0, nil
}
var countFound = 0
for hash, item := range itemMap {
if count <= 0 {
break
@@ -157,14 +206,100 @@ func (this *MemoryList) Purge(count int, callback func(hash string) error) error
if this.onRemove != nil {
this.onRemove(item)
}
atomic.AddInt64(&this.count, -1)
delete(itemMap, hash)
deletedHashList = append(deletedHashList, hash)
// week map
if item.Week > 0 {
wm, ok := this.weekItemMaps[item.Week]
if ok {
delete(wm, hash)
}
}
countFound++
}
count--
}
this.locker.Unlock()
// 执行外部操作
for _, hash := range deletedHashList {
if callback != nil {
err := callback(hash)
if err != nil {
return 0, err
}
}
}
return countFound, nil
}
func (this *MemoryList) PurgeLFU(count int, callback func(hash string) error) error {
if count <= 0 {
return nil
}
var week = currentWeek()
if this.minWeek > week {
this.minWeek = week
}
var deletedHashList = []string{}
Loop:
for w := this.minWeek; w <= week; w++ {
this.minWeek = w
this.locker.Lock()
wm, ok := this.weekItemMaps[w]
if ok {
var wc = len(wm)
if wc == 0 {
delete(this.weekItemMaps, w)
} else {
if wc <= count {
delete(this.weekItemMaps, w)
}
// TODO 未来支持按照点击量排序
for hash := range wm {
count--
if count < 0 {
this.locker.Unlock()
break Loop
}
delete(wm, hash)
itemMap, ok := this.itemMaps[this.prefix(hash)]
if !ok {
continue
}
item, ok := itemMap[hash]
if !ok {
continue
}
if this.onRemove != nil {
this.onRemove(item)
}
atomic.AddInt64(&this.count, -1)
delete(itemMap, hash)
deletedHashList = append(deletedHashList, hash)
}
}
} else {
delete(this.weekItemMaps, w)
}
this.locker.Unlock()
}
// 执行外部操作
for _, hash := range deletedHashList {
if callback != nil {
@@ -174,6 +309,7 @@ func (this *MemoryList) Purge(count int, callback func(hash string) error) error
}
}
}
return nil
}
@@ -206,13 +342,8 @@ func (this *MemoryList) Stat(check func(hash string) bool) (*Stat, error) {
// Count 总数量
func (this *MemoryList) Count() (int64, error) {
this.locker.RLock()
var count = 0
for _, itemMap := range this.itemMaps {
count += len(itemMap)
}
this.locker.RUnlock()
return int64(count), nil
var count = atomic.LoadInt64(&this.count)
return count, nil
}
// OnAdd 添加事件
@@ -229,6 +360,41 @@ func (this *MemoryList) Close() error {
return nil
}
// IncreaseHit 增加点击量
func (this *MemoryList) IncreaseHit(hash string) error {
this.locker.Lock()
itemMap, ok := this.itemMaps[this.prefix(hash)]
if !ok {
this.locker.Unlock()
return nil
}
item, ok := itemMap[hash]
if ok {
var week = currentWeek()
// 交换位置
if item.Week > 0 && item.Week != week {
wm, ok := this.weekItemMaps[item.Week]
if ok {
delete(wm, hash)
}
wm, ok = this.weekItemMaps[week]
if ok {
wm[hash] = zero.New()
} else {
this.weekItemMaps[week] = map[string]zero.Zero{hash: zero.New()}
}
}
item.IncreaseHit(week)
}
this.locker.Unlock()
return nil
}
func (this *MemoryList) print(t *testing.T) {
this.locker.Lock()
for _, itemMap := range this.itemMaps {

View File

@@ -31,6 +31,8 @@ func TestMemoryList_Add(t *testing.T) {
})
t.Log(list.prefixes)
logs.PrintAsJSON(list.itemMaps, t)
logs.PrintAsJSON(list.weekItemMaps, t)
t.Log(list.Count())
}
func TestMemoryList_Remove(t *testing.T) {
@@ -48,6 +50,8 @@ func TestMemoryList_Remove(t *testing.T) {
})
_ = list.Remove("b")
list.print(t)
logs.PrintAsJSON(list.weekItemMaps, t)
t.Log(list.Count())
}
func TestMemoryList_Purge(t *testing.T) {
@@ -73,19 +77,22 @@ func TestMemoryList_Purge(t *testing.T) {
ExpiredAt: time.Now().Unix() - 2,
HeaderSize: 1024,
})
_ = list.Purge(100, func(hash string) error {
_, _ = list.Purge(100, func(hash string) error {
t.Log("delete:", hash)
return nil
})
list.print(t)
logs.PrintAsJSON(list.weekItemMaps, t)
for i := 0; i < 1000; i++ {
_ = list.Purge(100, func(hash string) error {
_, _ = list.Purge(100, func(hash string) error {
t.Log("delete:", hash)
return nil
})
t.Log(list.purgeIndex)
}
t.Log(list.Count())
}
func TestMemoryList_Purge_Large_List(t *testing.T) {
@@ -139,7 +146,7 @@ func TestMemoryList_CleanPrefix(t *testing.T) {
_ = list.Init()
before := time.Now()
for i := 0; i < 1_000_000; i++ {
key := "http://www.teaos.cn/hello/" + strconv.Itoa(i/10000) + "/" + strconv.Itoa(i) + ".html"
key := "https://www.teaos.cn/hello/" + strconv.Itoa(i/10000) + "/" + strconv.Itoa(i) + ".html"
_ = list.Add(fmt.Sprintf("%d", xxhash.Sum64String(key)), &Item{
Key: key,
ExpiredAt: time.Now().Unix() + 3600,
@@ -150,7 +157,7 @@ func TestMemoryList_CleanPrefix(t *testing.T) {
t.Log(time.Since(before).Seconds()*1000, "ms")
before = time.Now()
err := list.CleanPrefix("http://www.teaos.cn/hello/10")
err := list.CleanPrefix("https://www.teaos.cn/hello/10")
if err != nil {
t.Fatal(err)
}
@@ -162,11 +169,77 @@ func TestMemoryList_CleanPrefix(t *testing.T) {
t.Log(time.Since(before).Seconds()*1000, "ms")
}
func TestMemoryList_PurgeLFU(t *testing.T) {
var list = NewMemoryList().(*MemoryList)
list.minWeek = 2704
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
t.Log("current week:", currentWeek())
_ = list.Add("1", &Item{})
_ = list.Add("2", &Item{})
_ = list.Add("3", &Item{})
_ = list.Add("4", &Item{})
_ = list.Add("5", &Item{})
_ = list.Add("6", &Item{Week: 2704})
_ = list.Add("7", &Item{Week: 2704})
_ = list.Add("8", &Item{Week: 2705})
err := list.PurgeLFU(2, func(hash string) error {
t.Log("purge lfu:", hash)
return nil
})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
logs.PrintAsJSON(list.weekItemMaps, t)
t.Log(list.Count())
}
func TestMemoryList_IncreaseHit(t *testing.T) {
var list = NewMemoryList().(*MemoryList)
var item = &Item{}
item.Week = 2705
item.Week2Hits = 100
_ = list.Add("a", &Item{})
_ = list.Add("a", item)
t.Log("hits1:", item.Week1Hits, "hits2:", item.Week2Hits, "week:", item.Week)
logs.PrintAsJSON(list.weekItemMaps, t)
_ = list.IncreaseHit("a")
t.Log("hits1:", item.Week1Hits, "hits2:", item.Week2Hits, "week:", item.Week)
logs.PrintAsJSON(list.weekItemMaps, t)
_ = list.IncreaseHit("a")
t.Log("hits1:", item.Week1Hits, "hits2:", item.Week2Hits, "week:", item.Week)
logs.PrintAsJSON(list.weekItemMaps, t)
}
func TestMemoryList_CleanAll(t *testing.T) {
var list = NewMemoryList().(*MemoryList)
var item = &Item{}
item.Week = 2705
item.Week2Hits = 100
_ = list.Add("a", &Item{})
_ = list.CleanAll()
logs.PrintAsJSON(list.itemMaps, t)
logs.PrintAsJSON(list.weekItemMaps, t)
t.Log(list.Count())
}
func TestMemoryList_GC(t *testing.T) {
list := NewMemoryList().(*MemoryList)
_ = list.Init()
for i := 0; i < 1_000_000; i++ {
key := "http://www.teaos.cn/hello" + strconv.Itoa(i/100000) + "/" + strconv.Itoa(i) + ".html"
key := "https://www.teaos.cn/hello" + strconv.Itoa(i/100000) + "/" + strconv.Itoa(i) + ".html"
_ = list.Add(fmt.Sprintf("%d", xxhash.Sum64String(key)), &Item{
Key: key,
ExpiredAt: 0,

View File

@@ -3,8 +3,10 @@ package caches
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/types"
"strconv"
"sync"
@@ -12,6 +14,13 @@ import (
var SharedManager = NewManager()
func init() {
events.On(events.EventQuit, func() {
logs.Println("CACHE", "quiting cache manager")
SharedManager.UpdatePolicies([]*serverconfigs.HTTPCachePolicy{})
})
}
// Manager 缓存策略管理器
type Manager struct {
// 全局配置
@@ -25,10 +34,12 @@ type Manager struct {
// NewManager 获取管理器对象
func NewManager() *Manager {
return &Manager{
var m = &Manager{
policyMap: map[int64]*serverconfigs.HTTPCachePolicy{},
storageMap: map[int64]StorageInterface{},
}
return m
}
// UpdatePolicies 重新设置策略
@@ -135,7 +146,7 @@ func (this *Manager) NewStorageWithPolicy(policy *serverconfigs.HTTPCachePolicy)
case serverconfigs.CachePolicyStorageFile:
return NewFileStorage(policy)
case serverconfigs.CachePolicyStorageMemory:
return NewMemoryStorage(policy)
return NewMemoryStorage(policy, nil)
}
return nil
}

View File

@@ -9,6 +9,9 @@ type Reader interface {
// TypeName 类型名称
TypeName() string
// ExpiresAt 过期时间
ExpiresAt() int64
// Status 状态码
Status() int

View File

@@ -11,6 +11,7 @@ import (
type FileReader struct {
fp *os.File
expiresAt int64
status int
headerOffset int64
headerSize int
@@ -43,6 +44,8 @@ func (this *FileReader) Init() error {
return ErrNotFound
}
this.expiresAt = int64(binary.BigEndian.Uint32(buf[:SizeExpiresAt]))
status := types.Int(string(buf[SizeExpiresAt : SizeExpiresAt+SizeStatus]))
if status < 100 || status > 999 {
return errors.New("invalid status")
@@ -78,6 +81,10 @@ func (this *FileReader) TypeName() string {
return "disk"
}
func (this *FileReader) ExpiresAt() int64 {
return this.expiresAt
}
func (this *FileReader) Status() int {
return this.status
}

View File

@@ -20,6 +20,10 @@ func (this *MemoryReader) TypeName() string {
return "memory"
}
func (this *MemoryReader) ExpiresAt() int64 {
return this.item.ExpiredAt
}
func (this *MemoryReader) Status() int {
return this.item.Status
}

View File

@@ -8,16 +8,23 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
stringutil "github.com/iwind/TeaGo/utils/string"
"golang.org/x/text/language"
"golang.org/x/text/message"
"io"
"math"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
@@ -36,6 +43,10 @@ const (
SizeMeta = SizeExpiresAt + SizeStatus + SizeURLLength + SizeHeaderLength + SizeBodyLength
)
const (
HotItemSize = 1024
)
// FileStorage 文件缓存
// 文件结构:
// [expires time] | [ status ] | [url length] | [header length] | [body length] | [url] [header data] [body data]
@@ -46,15 +57,22 @@ type FileStorage struct {
totalSize int64
list ListInterface
writingKeyMap map[string]bool // key => bool
writingKeyMap map[string]zero.Zero // key => bool
locker sync.RWMutex
ticker *utils.Ticker
purgeTicker *utils.Ticker
hotMap map[string]*HotItem // key => count
hotMapLocker sync.Mutex
lastHotSize int
hotTicker *utils.Ticker
}
func NewFileStorage(policy *serverconfigs.HTTPCachePolicy) *FileStorage {
return &FileStorage{
policy: policy,
writingKeyMap: map[string]bool{},
writingKeyMap: map[string]zero.Zero{},
hotMap: map[string]*HotItem{},
lastHotSize: -1,
}
}
@@ -165,12 +183,16 @@ func (this *FileStorage) Init() error {
Life: this.policy.Life,
MinLife: this.policy.MinLife,
MaxLife: this.policy.MaxLife,
MemoryAutoPurgeCount: this.policy.MemoryAutoPurgeCount,
MemoryAutoPurgeInterval: this.policy.MemoryAutoPurgeInterval,
MemoryLFUFreePercent: this.policy.MemoryLFUFreePercent,
}
err = memoryPolicy.Init()
if err != nil {
return err
}
memoryStorage := NewMemoryStorage(memoryPolicy)
memoryStorage := NewMemoryStorage(memoryPolicy, this)
err = memoryStorage.Init()
if err != nil {
return err
@@ -182,10 +204,19 @@ func (this *FileStorage) Init() error {
return nil
}
func (this *FileStorage) OpenReader(key string) (Reader, error) {
func (this *FileStorage) OpenReader(key string, useStale bool) (Reader, error) {
return this.openReader(key, true, useStale)
}
func (this *FileStorage) openReader(key string, allowMemory bool, useStale bool) (Reader, error) {
// 使用陈旧缓存的时候,我们认为是短暂的,只需要从文件里检查即可
if useStale {
allowMemory = false
}
// 先尝试内存缓存
if this.memoryStorage != nil {
reader, err := this.memoryStorage.OpenReader(key)
if allowMemory && this.memoryStorage != nil {
reader, err := this.memoryStorage.OpenReader(key, useStale)
if err == nil {
return reader, err
}
@@ -193,6 +224,17 @@ func (this *FileStorage) OpenReader(key string) (Reader, error) {
hash, path := this.keyPath(key)
// 检查文件记录是否已过期
if !useStale {
exists, err := this.list.Exist(hash)
if err != nil {
return nil, err
}
if !exists {
return nil, ErrNotFound
}
}
// TODO 尝试使用mmap加快读取速度
var isOk = false
fp, err := os.OpenFile(path, os.O_RDONLY, 0444)
@@ -209,15 +251,6 @@ func (this *FileStorage) OpenReader(key string) (Reader, error) {
}
}()
// 检查文件记录是否已过期
exists, err := this.list.Exist(hash)
if err != nil {
return nil, err
}
if !exists {
return nil, ErrNotFound
}
reader := NewFileReader(fp)
if err != nil {
return nil, err
@@ -227,6 +260,45 @@ func (this *FileStorage) OpenReader(key string) (Reader, error) {
return nil, err
}
// 增加点击量
// 1/1000采样
if allowMemory {
var rate = this.policy.PersistenceHitSampleRate
if rate <= 0 {
rate = 1000
}
if this.lastHotSize == 0 {
// 自动降低采样率来增加热点数据的缓存几率
rate = rate / 10
}
if rands.Int(0, rate) == 0 {
var hitErr = this.list.IncreaseHit(hash)
if hitErr != nil {
// 此错误可以忽略
remotelogs.Error("CACHE", "increase hit failed: "+hitErr.Error())
}
// 增加到热点
// 这里不收录缓存尺寸过大的文件
if this.memoryStorage != nil && reader.BodySize() > 0 && reader.BodySize() < 128*1024*1024 {
this.hotMapLocker.Lock()
hotItem, ok := this.hotMap[key]
if ok {
hotItem.Hits++
hotItem.ExpiresAt = reader.expiresAt
} else if len(this.hotMap) < HotItemSize { // 控制数量
this.hotMap[key] = &HotItem{
Key: key,
ExpiresAt: reader.ExpiresAt(),
Status: reader.Status(),
Hits: 1,
}
}
this.hotMapLocker.Unlock()
}
}
}
isOk = true
return reader, nil
}
@@ -250,7 +322,7 @@ func (this *FileStorage) OpenWriter(key string, expiredAt int64, status int) (Wr
return nil, ErrFileIsWriting
}
this.locker.Lock()
this.writingKeyMap[key] = true
this.writingKeyMap[key] = zero.New()
this.locker.Unlock()
defer func() {
if !isWriting {
@@ -398,7 +470,7 @@ func (this *FileStorage) AddToList(item *Item) {
}
}
item.MetaSize = SizeMeta
item.MetaSize = SizeMeta + 128
hash := stringutil.Md5(item.Key)
err := this.list.Add(hash, item)
if err != nil && !strings.Contains(err.Error(), "UNIQUE constraint failed") {
@@ -499,12 +571,12 @@ func (this *FileStorage) CleanAll() error {
}
// 重新遍历待删除
go func() {
goman.New(func() {
err = this.cleanDeletedDirs(dir)
if err != nil {
remotelogs.Warn("CACHE", "delete '*-deleted' dirs failed: "+err.Error())
}
}()
})
return nil
}
@@ -555,8 +627,11 @@ func (this *FileStorage) Stop() {
}
_ = this.list.Reset()
if this.ticker != nil {
this.ticker.Stop()
if this.purgeTicker != nil {
this.purgeTicker.Stop()
}
if this.hotTicker != nil {
this.hotTicker.Stop()
}
_ = this.list.Close()
@@ -606,32 +681,49 @@ func (this *FileStorage) initList() error {
}
// 使用异步防止阻塞主线程
go func() {
/**goman.New(func() {
dir := this.dir()
// 清除tmp
files, err := filepath.Glob(dir + "/*/*/*.cache.tmp")
if err == nil && len(files) > 0 {
for _, path := range files {
_ = os.Remove(path)
}
}
}()
// TODO 需要一个更加高效的实现
})**/
// 启动定时清理任务
this.ticker = utils.NewTicker(30 * time.Second)
var autoPurgeInterval = this.policy.PersistenceAutoPurgeInterval
if autoPurgeInterval <= 0 {
autoPurgeInterval = 30
if Tea.IsTesting() {
autoPurgeInterval = 10
}
}
this.purgeTicker = utils.NewTicker(time.Duration(autoPurgeInterval) * time.Second)
events.On(events.EventQuit, func() {
remotelogs.Println("CACHE", "quit clean timer")
var ticker = this.ticker
var ticker = this.purgeTicker
if ticker != nil {
ticker.Stop()
}
})
go func() {
for this.ticker.Next() {
this.purgeLoop()
goman.New(func() {
for this.purgeTicker.Next() {
trackers.Run("FILE_CACHE_STORAGE_PURGE_LOOP", func() {
this.purgeLoop()
})
}
}()
})
// 热点处理任务
this.hotTicker = utils.NewTicker(1 * time.Minute)
if Tea.IsTesting() {
this.hotTicker = utils.NewTicker(10 * time.Second)
}
goman.New(func() {
for this.hotTicker.Next() {
trackers.Run("FILE_CACHE_STORAGE_HOT_LOOP", func() {
this.hotLoop()
})
}
})
return nil
}
@@ -711,9 +803,9 @@ func (this *FileStorage) decodeFile(path string) (*Item, error) {
// URL
if urlSize > 0 {
data := utils.BytePool1024.Get()
data := utils.BytePool1k.Get()
result, ok, err := this.readN(fp, data, int(urlSize))
utils.BytePool1024.Put(data)
utils.BytePool1k.Put(data)
if err != nil {
return nil, err
}
@@ -730,16 +822,184 @@ func (this *FileStorage) decodeFile(path string) (*Item, error) {
// 清理任务
func (this *FileStorage) purgeLoop() {
err := this.list.Purge(1000, func(hash string) error {
path := this.hashPath(hash)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
remotelogs.Error("CACHE", "purge '"+path+"' error: "+err.Error())
// 计算是否应该开启LFU清理
var capacityBytes = this.policy.CapacityBytes()
var startLFU = false
var usedPercent = float32(this.TotalDiskSize()*100) / float32(capacityBytes)
var lfuFreePercent = this.policy.PersistenceLFUFreePercent
if lfuFreePercent <= 0 {
lfuFreePercent = 5
}
if capacityBytes > 0 {
if lfuFreePercent < 100 {
if usedPercent >= 100-lfuFreePercent {
startLFU = true
}
}
}
// 清理过期
{
var times = 1
// 空闲时间多清理
if utils.SharedFreeHoursManager.IsFreeHour() {
times = 5
}
// 处于LFU阈值时多清理
if startLFU {
times = 5
}
var purgeCount = this.policy.PersistenceAutoPurgeCount
if purgeCount <= 0 {
purgeCount = 1000
}
for i := 0; i < times; i++ {
countFound, err := this.list.Purge(purgeCount, func(hash string) error {
path := this.hashPath(hash)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
remotelogs.Error("CACHE", "purge '"+path+"' error: "+err.Error())
}
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge file storage failed: "+err.Error())
continue
}
if countFound < purgeCount {
break
}
time.Sleep(1 * time.Second)
}
}
// 磁盘空间不足时,清除老旧的缓存
if startLFU {
var total, _ = this.list.Count()
if total > 0 {
var count = types.Int(math.Ceil(float64(total) * float64(lfuFreePercent*2) / 100))
if count > 0 {
// 限制单次清理的条数,防止占用太多系统资源
if count > 2000 {
count = 2000
}
remotelogs.Println("CACHE", "LFU purge policy '"+this.policy.Name+"' id: "+types.String(this.policy.Id)+", count: "+types.String(count))
err := this.list.PurgeLFU(count, func(hash string) error {
path := this.hashPath(hash)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
remotelogs.Error("CACHE", "purge '"+path+"' error: "+err.Error())
}
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge file storage in LFU failed: "+err.Error())
}
}
}
}
}
// 热点数据任务
func (this *FileStorage) hotLoop() {
var memoryStorage = this.memoryStorage
if memoryStorage == nil {
return
}
this.hotMapLocker.Lock()
if len(this.hotMap) == 0 {
this.hotMapLocker.Unlock()
this.lastHotSize = 0
return
}
this.lastHotSize = len(this.hotMap)
var result = []*HotItem{} // [ {key: ..., hits: ...}, ... ]
for _, v := range this.hotMap {
result = append(result, v)
}
this.hotMap = map[string]*HotItem{}
this.hotMapLocker.Unlock()
// 取Top10写入内存
if len(result) > 0 {
sort.Slice(result, func(i, j int) bool {
return result[i].Hits > result[j].Hits
})
var size = 1
if len(result) < 10 {
size = 1
} else {
size = len(result) / 10
}
var buf = utils.BytePool16k.Get()
defer utils.BytePool16k.Put(buf)
for _, item := range result[:size] {
reader, err := this.openReader(item.Key, false, false)
if err != nil {
continue
}
if reader == nil {
continue
}
if reader.ExpiresAt() <= time.Now().Unix() {
continue
}
writer, err := this.memoryStorage.openWriter(item.Key, item.ExpiresAt, item.Status, false)
if err != nil {
if !CanIgnoreErr(err) {
remotelogs.Error("CACHE", "transfer hot item failed: "+err.Error())
}
_ = reader.Close()
continue
}
if writer == nil {
_ = reader.Close()
continue
}
err = reader.ReadHeader(buf, func(n int) (goNext bool, err error) {
_, err = writer.WriteHeader(buf[:n])
return
})
if err != nil {
_ = reader.Close()
_ = writer.Discard()
continue
}
err = reader.ReadBody(buf, func(n int) (goNext bool, err error) {
_, err = writer.Write(buf[:n])
return
})
if err != nil {
_ = reader.Close()
_ = writer.Discard()
continue
}
this.memoryStorage.AddToList(&Item{
Type: writer.ItemType(),
Key: item.Key,
ExpiredAt: item.ExpiresAt,
HeaderSize: writer.HeaderSize(),
BodySize: writer.BodySize(),
})
_ = reader.Close()
_ = writer.Close()
}
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge file storage failed: "+err.Error())
}
}

View File

@@ -517,3 +517,16 @@ func BenchmarkFileStorage_Read(b *testing.B) {
_ = reader.Close()
}
}
func BenchmarkFileStorage_KeyPath(b *testing.B) {
runtime.GOMAXPROCS(1)
var storage = &FileStorage{
cacheConfig: &serverconfigs.HTTPFileCacheStorage{},
policy: &serverconfigs.HTTPCachePolicy{Id: 1},
}
for i := 0; i < b.N; i++ {
_, _ = storage.keyPath(strconv.Itoa(i))
}
}

View File

@@ -10,7 +10,7 @@ type StorageInterface interface {
Init() error
// OpenReader 读取缓存
OpenReader(key string) (Reader, error)
OpenReader(key string, useStale bool) (reader Reader, err error)
// OpenWriter 打开缓存写入器等待写入
OpenWriter(key string, expiredAt int64, status int) (Writer, error)

View File

@@ -3,9 +3,16 @@ package caches
import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/cespare/xxhash"
"github.com/iwind/TeaGo/rands"
"github.com/iwind/TeaGo/types"
"math"
"runtime"
"strconv"
"sync"
"sync/atomic"
@@ -26,23 +33,38 @@ func (this *MemoryItem) IsExpired() bool {
}
type MemoryStorage struct {
policy *serverconfigs.HTTPCachePolicy
list ListInterface
locker *sync.RWMutex
valuesMap map[uint64]*MemoryItem
ticker *utils.Ticker
purgeDuration time.Duration
parentStorage StorageInterface
policy *serverconfigs.HTTPCachePolicy
list ListInterface
locker *sync.RWMutex
valuesMap map[uint64]*MemoryItem // hash => item
dirtyChan chan string // hash chan
purgeTicker *utils.Ticker
totalSize int64
writingKeyMap map[string]bool // key => bool
writingKeyMap map[string]zero.Zero // key => bool
}
func NewMemoryStorage(policy *serverconfigs.HTTPCachePolicy) *MemoryStorage {
func NewMemoryStorage(policy *serverconfigs.HTTPCachePolicy, parentStorage StorageInterface) *MemoryStorage {
var dirtyChan chan string
if parentStorage != nil {
var queueSize = policy.MemoryAutoFlushQueueSize
if queueSize <= 0 {
queueSize = 2048
}
dirtyChan = make(chan string, queueSize)
}
return &MemoryStorage{
parentStorage: parentStorage,
policy: policy,
list: NewMemoryList(),
locker: &sync.RWMutex{},
valuesMap: map[uint64]*MemoryItem{},
writingKeyMap: map[string]bool{},
dirtyChan: dirtyChan,
writingKeyMap: map[string]zero.Zero{},
}
}
@@ -57,23 +79,33 @@ func (this *MemoryStorage) Init() error {
atomic.AddInt64(&this.totalSize, -item.TotalSize())
})
if this.purgeDuration <= 0 {
this.purgeDuration = 10 * time.Second
var autoPurgeInterval = this.policy.MemoryAutoPurgeInterval
if autoPurgeInterval <= 0 {
autoPurgeInterval = 5
}
// 启动定时清理任务
this.ticker = utils.NewTicker(this.purgeDuration)
go func() {
for this.ticker.Next() {
this.purgeTicker = utils.NewTicker(time.Duration(autoPurgeInterval) * time.Second)
goman.New(func() {
for this.purgeTicker.Next() {
var tr = trackers.Begin("MEMORY_CACHE_STORAGE_PURGE_LOOP")
this.purgeLoop()
tr.End()
}
}()
})
// 启动定时Flush memory to disk任务
goman.New(func() {
for hash := range this.dirtyChan {
this.flushItem(hash)
}
})
return nil
}
// OpenReader 读取缓存
func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
func (this *MemoryStorage) OpenReader(key string, useStale bool) (Reader, error) {
hash := this.hash(key)
this.locker.RLock()
@@ -83,7 +115,7 @@ func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
return nil, ErrNotFound
}
if item.ExpiredAt > utils.UnixTime() {
if useStale || (item.ExpiredAt > utils.UnixTime()) {
reader := NewMemoryReader(item)
err := reader.Init()
if err != nil {
@@ -91,6 +123,18 @@ func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
return nil, err
}
this.locker.RUnlock()
// 增加点击量
// 1/1000采样
// TODO 考虑是否在缓存策略里设置
if rands.Int(0, 1000) == 0 {
var hitErr = this.list.IncreaseHit(types.String(hash))
if hitErr != nil {
// 此错误可以忽略
remotelogs.Error("CACHE", "increase hit failed: "+hitErr.Error())
}
}
return reader, nil
}
this.locker.RUnlock()
@@ -102,6 +146,10 @@ func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
// OpenWriter 打开缓存写入器等待写入
func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int) (Writer, error) {
return this.openWriter(key, expiredAt, status, true)
}
func (this *MemoryStorage) openWriter(key string, expiredAt int64, status int, isDirty bool) (Writer, error) {
this.locker.Lock()
defer this.locker.Unlock()
@@ -111,7 +159,7 @@ func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int) (
if ok {
return nil, ErrFileIsWriting
}
this.writingKeyMap[key] = true
this.writingKeyMap[key] = zero.New()
defer func() {
if !isWriting {
delete(this.writingKeyMap, key)
@@ -145,7 +193,7 @@ func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int) (
}
isWriting = true
return NewMemoryWriter(this.valuesMap, key, expiredAt, status, this.locker, func() {
return NewMemoryWriter(this, key, expiredAt, status, isDirty, func() {
this.locker.Lock()
delete(this.writingKeyMap, key)
this.locker.Unlock()
@@ -208,16 +256,23 @@ func (this *MemoryStorage) Stop() {
this.locker.Lock()
this.valuesMap = map[uint64]*MemoryItem{}
this.writingKeyMap = map[string]bool{}
this.writingKeyMap = map[string]zero.Zero{}
_ = this.list.Reset()
if this.ticker != nil {
this.ticker.Stop()
if this.purgeTicker != nil {
this.purgeTicker.Stop()
}
if this.parentStorage != nil && this.dirtyChan != nil {
close(this.dirtyChan)
}
_ = this.list.Close()
this.locker.Unlock()
// 回收内存
runtime.GC()
remotelogs.Println("CACHE", "close memory storage '"+strconv.FormatInt(this.policy.Id, 10)+"'")
}
@@ -228,7 +283,7 @@ func (this *MemoryStorage) Policy() *serverconfigs.HTTPCachePolicy {
// AddToList 将缓存添加到列表
func (this *MemoryStorage) AddToList(item *Item) {
item.MetaSize = int64(len(item.Key)) + 32 /** 32是我们评估的数据结构的长度 **/
item.MetaSize = int64(len(item.Key)) + 128 /** 128是我们评估的数据结构的长度 **/
hash := fmt.Sprintf("%d", this.hash(item.Key))
_ = this.list.Add(hash, item)
}
@@ -250,7 +305,28 @@ func (this *MemoryStorage) hash(key string) uint64 {
// 清理任务
func (this *MemoryStorage) purgeLoop() {
_ = this.list.Purge(2048, func(hash string) error {
// 计算是否应该开启LFU清理
var capacityBytes = this.policy.CapacityBytes()
var startLFU = false
var usedPercent = float32(this.TotalMemorySize()*100) / float32(capacityBytes)
var lfuFreePercent = this.policy.MemoryLFUFreePercent
if lfuFreePercent <= 0 {
lfuFreePercent = 5
}
if capacityBytes > 0 {
if lfuFreePercent < 100 {
if usedPercent >= 100-lfuFreePercent {
startLFU = true
}
}
}
// 清理过期
var purgeCount = this.policy.MemoryAutoPurgeCount
if purgeCount <= 0 {
purgeCount = 2000
}
_, _ = this.list.Purge(purgeCount, func(hash string) error {
uintHash, err := strconv.ParseUint(hash, 10, 64)
if err == nil {
this.locker.Lock()
@@ -259,6 +335,92 @@ func (this *MemoryStorage) purgeLoop() {
}
return nil
})
// LFU
if startLFU {
var total, _ = this.list.Count()
if total > 0 {
var count = types.Int(math.Ceil(float64(total) * float64(lfuFreePercent*2) / 100))
if count > 0 {
// 限制单次清理的条数,防止占用太多系统资源
if count > 2000 {
count = 2000
}
// 这里不提示LFU因为此事件将会非常频繁
err := this.list.PurgeLFU(count, func(hash string) error {
uintHash, err := strconv.ParseUint(hash, 10, 64)
if err == nil {
this.locker.Lock()
delete(this.valuesMap, uintHash)
this.locker.Unlock()
}
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge memory storage in LFU failed: "+err.Error())
}
}
}
}
}
// Flush任务
func (this *MemoryStorage) flushItem(key string) {
if this.parentStorage == nil {
return
}
var hash = this.hash(key)
this.locker.RLock()
item, ok := this.valuesMap[hash]
this.locker.RUnlock()
if !ok {
return
}
if !item.IsDone || item.IsExpired() {
return
}
writer, err := this.parentStorage.OpenWriter(key, item.ExpiredAt, item.Status)
if err != nil {
if !CanIgnoreErr(err) {
remotelogs.Error("CACHE", "flush items failed: open writer failed: "+err.Error())
}
return
}
_, err = writer.WriteHeader(item.HeaderValue)
if err != nil {
_ = writer.Discard()
remotelogs.Error("CACHE", "flush items failed: write header failed: "+err.Error())
return
}
_, err = writer.Write(item.BodyValue)
if err != nil {
_ = writer.Discard()
remotelogs.Error("CACHE", "flush items failed: writer body failed: "+err.Error())
return
}
err = writer.Close()
if err != nil {
_ = writer.Discard()
remotelogs.Error("CACHE", "flush items failed: close writer failed: "+err.Error())
}
this.parentStorage.AddToList(&Item{
Type: writer.ItemType(),
Key: key,
ExpiredAt: item.ExpiredAt,
HeaderSize: writer.HeaderSize(),
BodySize: writer.BodySize(),
})
return
}
func (this *MemoryStorage) memoryCapacityBytes() int64 {

View File

@@ -13,7 +13,7 @@ import (
)
func TestMemoryStorage_OpenWriter(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200)
if err != nil {
@@ -88,7 +88,7 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
}
func TestMemoryStorage_OpenReaderLock(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
_ = storage.Init()
var h = storage.hash("test")
@@ -101,7 +101,7 @@ func TestMemoryStorage_OpenReaderLock(t *testing.T) {
}
func TestMemoryStorage_Delete(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
{
writer, err := storage.OpenWriter("abc", time.Now().Unix()+60, 200)
if err != nil {
@@ -123,7 +123,7 @@ func TestMemoryStorage_Delete(t *testing.T) {
}
func TestMemoryStorage_Stat(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200)
@@ -160,7 +160,7 @@ func TestMemoryStorage_Stat(t *testing.T) {
}
func TestMemoryStorage_CleanAll(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200)
@@ -195,7 +195,7 @@ func TestMemoryStorage_CleanAll(t *testing.T) {
}
func TestMemoryStorage_Purge(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
expiredAt := time.Now().Unix() + 60
{
writer, err := storage.OpenWriter("abc", expiredAt, 200)
@@ -230,8 +230,9 @@ func TestMemoryStorage_Purge(t *testing.T) {
}
func TestMemoryStorage_Expire(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage.purgeDuration = 5 * time.Second
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{
MemoryAutoPurgeInterval: 5,
}, nil)
err := storage.Init()
if err != nil {
t.Fatal(err)
@@ -255,7 +256,7 @@ func TestMemoryStorage_Expire(t *testing.T) {
}
func TestMemoryStorage_Locker(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{}, nil)
err := storage.Init()
if err != nil {
t.Fatal(err)

View File

@@ -2,36 +2,36 @@ package caches
import (
"github.com/cespare/xxhash"
"sync"
"time"
)
type MemoryWriter struct {
storage *MemoryStorage
key string
expiredAt int64
m map[uint64]*MemoryItem
locker *sync.RWMutex
headerSize int64
bodySize int64
status int
isDirty bool
hash uint64
item *MemoryItem
endFunc func()
}
func NewMemoryWriter(m map[uint64]*MemoryItem, key string, expiredAt int64, status int, locker *sync.RWMutex, endFunc func()) *MemoryWriter {
func NewMemoryWriter(memoryStorage *MemoryStorage, key string, expiredAt int64, status int, isDirty bool, endFunc func()) *MemoryWriter {
w := &MemoryWriter{
m: m,
storage: memoryStorage,
key: key,
expiredAt: expiredAt,
locker: locker,
item: &MemoryItem{
ExpiredAt: expiredAt,
ModifiedAt: time.Now().Unix(),
Status: status,
},
status: status,
isDirty: isDirty,
endFunc: endFunc,
}
w.hash = w.calculateHash(key)
@@ -72,10 +72,19 @@ func (this *MemoryWriter) Close() error {
return nil
}
this.locker.Lock()
this.storage.locker.Lock()
this.item.IsDone = true
this.m[this.hash] = this.item
this.locker.Unlock()
this.storage.valuesMap[this.hash] = this.item
if this.isDirty {
if this.storage.parentStorage != nil {
select {
case this.storage.dirtyChan <- this.key:
default:
}
}
}
this.storage.locker.Unlock()
return nil
}
@@ -85,9 +94,9 @@ func (this *MemoryWriter) Discard() error {
// 需要在Locker之外
defer this.endFunc()
this.locker.Lock()
delete(this.m, this.hash)
this.locker.Unlock()
this.storage.locker.Lock()
delete(this.storage.valuesMap, this.hash)
this.storage.locker.Unlock()
return nil
}

View File

@@ -1,7 +1,7 @@
package teaconst
const (
Version = "0.3.3"
Version = "0.3.7"
ProductName = "Edge Node"
ProcessName = "edge-node"

13
internal/const/vars.go Normal file
View File

@@ -0,0 +1,13 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package teaconst
var (
// 流量统计
InTrafficBytes = uint64(0)
OutTrafficBytes = uint64(0)
NodeId int64 = 0
NodeIdString = ""
)

View File

@@ -0,0 +1,12 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package goman
import "time"
type Instance struct {
Id uint64
CreatedTime time.Time
File string
Line int
}

81
internal/goman/lib.go Normal file
View File

@@ -0,0 +1,81 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package goman
import (
"runtime"
"sync"
"time"
)
var locker = &sync.Mutex{}
var instanceMap = map[uint64]*Instance{} // id => *Instance
var instanceId = uint64(0)
// New 新创建goroutine
func New(f func()) {
_, file, line, _ := runtime.Caller(1)
go func() {
locker.Lock()
instanceId++
var instance = &Instance{
Id: instanceId,
CreatedTime: time.Now(),
}
instance.File = file
instance.Line = line
instanceMap[instanceId] = instance
locker.Unlock()
// run function
f()
locker.Lock()
delete(instanceMap, instanceId)
locker.Unlock()
}()
}
// NewWithArgs 创建带有参数的goroutine
func NewWithArgs(f func(args ...interface{}), args ...interface{}) {
_, file, line, _ := runtime.Caller(1)
go func() {
locker.Lock()
instanceId++
var instance = &Instance{
Id: instanceId,
CreatedTime: time.Now(),
}
instance.File = file
instance.Line = line
instanceMap[instanceId] = instance
locker.Unlock()
// run function
f(args...)
locker.Lock()
delete(instanceMap, instanceId)
locker.Unlock()
}()
}
// List 列出所有正在运行goroutine
func List() []*Instance {
locker.Lock()
defer locker.Unlock()
var result = []*Instance{}
for _, instance := range instanceMap {
result = append(result, instance)
}
return result
}

View File

@@ -0,0 +1,28 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package goman
import (
"testing"
"time"
)
func TestNew(t *testing.T) {
New(func() {
t.Log("Hello")
t.Log(List())
})
time.Sleep(1 * time.Second)
t.Log(List())
time.Sleep(1 * time.Second)
}
func TestNewWithArgs(t *testing.T) {
NewWithArgs(func(args ...interface{}) {
t.Log(args[0], args[1])
}, 1, 2)
time.Sleep(1 * time.Second)
}

View File

@@ -0,0 +1,5 @@
# IPList
List Check Order:
~~~
Global List --> Node List--> Server List --> WAF List --> Bind List
~~~

View File

@@ -13,7 +13,7 @@ func (this *BaseAction) Close() error {
return nil
}
// 处理HTTP请求
// DoHTTP 处理HTTP请求
func (this *BaseAction) DoHTTP(req *http.Request, resp http.ResponseWriter) (goNext bool, err error) {
return true, nil
}

View File

@@ -1,6 +1,6 @@
package iplibrary
// 是否是致命错误
// FataError 是否是致命错误
type FataError struct {
err string
}

View File

@@ -11,7 +11,7 @@ import (
"time"
)
// Firewalld动作管理
// FirewalldAction Firewalld动作管理
// 常用命令:
// - 查询列表: firewall-cmd --list-all
// - 添加IPfirewall-cmd --add-rich-rule="rule family='ipv4' source address='192.168.2.32' reject" --timeout=30s
@@ -20,6 +20,8 @@ type FirewalldAction struct {
BaseAction
config *firewallconfigs.FirewallActionFirewalldConfig
firewalldNotFound bool
}
func NewFirewalldAction() *FirewalldAction {
@@ -82,6 +84,10 @@ func (this *FirewalldAction) runActionSingleIP(action string, listType IPListTyp
if len(path) == 0 {
path, err = exec.LookPath("firewall-cmd")
if err != nil {
if this.firewalldNotFound {
return nil
}
this.firewalldNotFound = true
return err
}
}
@@ -126,10 +132,12 @@ func (this *FirewalldAction) runActionSingleIP(action string, listType IPListTyp
}
args := []string{opt}
if item.ExpiredAt > timestamp {
args = append(args, "--timeout="+fmt.Sprintf("%d", item.ExpiredAt-timestamp)+"s")
} else {
// TODO 思考是否需要permanent不然--reload之后会丢失
if action == "addItem" {
if item.ExpiredAt > timestamp {
args = append(args, "--timeout="+fmt.Sprintf("%d", item.ExpiredAt-timestamp)+"s")
} else {
// TODO 思考是否需要permanent不然--reload之后会丢失
}
}
if runtime.GOOS == "darwin" {

View File

@@ -6,19 +6,19 @@ import (
"net/http"
)
// HTML动作
// HTMLAction HTML动作
type HTMLAction struct {
BaseAction
config *firewallconfigs.FirewallActionHTMLConfig
}
// 获取新对象
// NewHTMLAction 获取新对象
func NewHTMLAction() *HTMLAction {
return &HTMLAction{}
}
// 初始化
// Init 初始化
func (this *HTMLAction) Init(config *firewallconfigs.FirewallActionConfig) error {
this.config = &firewallconfigs.FirewallActionHTMLConfig{}
err := this.convertParams(config.Params, this.config)
@@ -28,22 +28,22 @@ func (this *HTMLAction) Init(config *firewallconfigs.FirewallActionConfig) error
return nil
}
// 添加
// AddItem 添加
func (this *HTMLAction) AddItem(listType IPListType, item *pb.IPItem) error {
return nil
}
// 删除
// DeleteItem 删除
func (this *HTMLAction) DeleteItem(listType IPListType, item *pb.IPItem) error {
return nil
}
// 关闭
// Close 关闭
func (this *HTMLAction) Close() error {
return nil
}
// 处理HTTP请求
// DoHTTP 处理HTTP请求
func (this *HTMLAction) DoHTTP(req *http.Request, resp http.ResponseWriter) (goNext bool, err error) {
if this.config == nil {
goNext = true

View File

@@ -7,18 +7,18 @@ import (
)
type ActionInterface interface {
// 初始化
// Init 初始化
Init(config *firewallconfigs.FirewallActionConfig) error
// 添加
// AddItem 添加
AddItem(listType IPListType, item *pb.IPItem) error
// 删除
// DeleteItem 删除
DeleteItem(listType IPListType, item *pb.IPItem) error
// 关闭
// Close 关闭
Close() error
// 处理HTTP请求
// DoHTTP 处理HTTP请求
DoHTTP(req *http.Request, resp http.ResponseWriter) (goNext bool, err error)
}

View File

@@ -5,13 +5,15 @@ import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/firewallconfigs"
"github.com/iwind/TeaGo/types"
"os/exec"
"runtime"
"strconv"
"strings"
"time"
)
// IPSet动作
// IPSetAction IPSet动作
// 相关命令:
// - 利用Firewalld管理set
// - 添加firewall-cmd --permanent --new-ipset=edge_ip_list --type=hash:ip --option="timeout=0"
@@ -23,14 +25,21 @@ import (
// - 添加Itemipset add edge_ip_list 192.168.2.32 timeout 30
// - 删除Item: ipset del edge_ip_list 192.168.2.32
// - 创建setipset create edge_ip_list hash:ip timeout 0
// - 查看统计ipset -t list edge_black_list
// - 删除setipset destroy edge_black_list
type IPSetAction struct {
BaseAction
config *firewallconfigs.FirewallActionIPSetConfig
config *firewallconfigs.FirewallActionIPSetConfig
errorBuf *bytes.Buffer
ipsetNotfound bool
}
func NewIPSetAction() *IPSetAction {
return &IPSetAction{}
return &IPSetAction{
errorBuf: &bytes.Buffer{},
}
}
func (this *IPSetAction) Init(config *firewallconfigs.FirewallActionConfig) error {
@@ -54,7 +63,7 @@ func (this *IPSetAction) Init(config *firewallconfigs.FirewallActionConfig) erro
return err
}
{
cmd := exec.Command(path, "create", this.config.WhiteName, "hash:ip", "timeout", "0")
cmd := exec.Command(path, "create", this.config.WhiteName, "hash:ip", "timeout", "0", "maxelem", "1000000")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
err := cmd.Run()
@@ -68,7 +77,7 @@ func (this *IPSetAction) Init(config *firewallconfigs.FirewallActionConfig) erro
}
}
{
cmd := exec.Command(path, "create", this.config.BlackName, "hash:ip", "timeout", "0")
cmd := exec.Command(path, "create", this.config.BlackName, "hash:ip", "timeout", "0", "maxelem", "1000000")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
err := cmd.Run()
@@ -163,24 +172,39 @@ func (this *IPSetAction) Init(config *firewallconfigs.FirewallActionConfig) erro
}
{
cmd := exec.Command(path, "-A", "INPUT", "-m", "set", "--match-set", this.config.WhiteName, "src", "-j", "ACCEPT")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
// 检查规则是否存在
var cmd = exec.Command(path, "-C", "INPUT", "-m", "set", "--match-set", this.config.WhiteName, "src", "-j", "ACCEPT")
err := cmd.Run()
if err != nil {
output := stderr.Bytes()
return errors.New("iptables add rule: " + err.Error() + ", output: " + string(output))
var exists = err == nil
// 添加规则
if !exists {
var cmd = exec.Command(path, "-A", "INPUT", "-m", "set", "--match-set", this.config.WhiteName, "src", "-j", "ACCEPT")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
output := stderr.Bytes()
return errors.New("iptables add rule: " + err.Error() + ", output: " + string(output))
}
}
}
{
cmd := exec.Command(path, "-A", "INPUT", "-m", "set", "--match-set", this.config.BlackName, "src", "-j", "REJECT")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
// 检查规则是否存在
var cmd = exec.Command(path, "-C", "INPUT", "-m", "set", "--match-set", this.config.BlackName, "src", "-j", "REJECT")
err := cmd.Run()
if err != nil {
output := stderr.Bytes()
return errors.New("iptables add rule: " + err.Error() + ", output: " + string(output))
var exists = err == nil
if !exists {
var cmd = exec.Command(path, "-A", "INPUT", "-m", "set", "--match-set", this.config.BlackName, "src", "-j", "REJECT")
stderr := bytes.NewBuffer([]byte{})
cmd.Stderr = stderr
err := cmd.Run()
if err != nil {
output := stderr.Bytes()
return errors.New("iptables add rule: " + err.Error() + ", output: " + string(output))
}
}
}
}
@@ -212,6 +236,16 @@ func (this *IPSetAction) runAction(action string, listType IPListType, item *pb.
return nil
}
for _, cidr := range cidrList {
index := strings.Index(cidr, "/")
if index <= 0 {
continue
}
// 只支持/24以下的
if types.Int(cidr[index+1:]) < 24 {
continue
}
item.IpFrom = cidr
item.IpTo = ""
err := this.runActionSingleIP(action, listType, item)
@@ -246,6 +280,11 @@ func (this *IPSetAction) runActionSingleIP(action string, listType IPListType, i
if len(path) == 0 {
path, err = exec.LookPath("ipset")
if err != nil {
// 找不到ipset命令错误只提示一次
if this.ipsetNotfound {
return nil
}
this.ipsetNotfound = true
return err
}
}
@@ -258,19 +297,30 @@ func (this *IPSetAction) runActionSingleIP(action string, listType IPListType, i
case "deleteItem":
args = append(args, "del")
}
args = append(args, listName, item.IpFrom)
timestamp := time.Now().Unix()
if item.ExpiredAt > timestamp {
args = append(args, "timeout", strconv.FormatInt(item.ExpiredAt-timestamp, 10))
}
//logs.Println(args)
args = append(args, listName, item.IpFrom)
if action == "addItem" {
timestamp := time.Now().Unix()
if item.ExpiredAt > timestamp {
args = append(args, "timeout", strconv.FormatInt(item.ExpiredAt-timestamp, 10))
}
}
if runtime.GOOS == "darwin" {
// MAC OS直接返回
return nil
}
this.errorBuf.Reset()
cmd := exec.Command(path, args...)
return cmd.Run()
cmd.Stderr = this.errorBuf
err = cmd.Run()
if err != nil {
var errString = this.errorBuf.String()
if action == "deleteItem" && strings.Contains(errString, "not added") {
return nil
}
return errors.New(strings.TrimSpace(errString))
}
return nil
}

View File

@@ -9,15 +9,18 @@ import (
"runtime"
)
// IPTables动作
// IPTablesAction IPTables动作
// 相关命令:
// iptables -A INPUT -s "192.168.2.32" -j ACCEPT
// iptables -A INPUT -s "192.168.2.32" -j REJECT
// iptables -D ...
// iptables -D INPUT ...
// iptables -F INPUT
type IPTablesAction struct {
BaseAction
config *firewallconfigs.FirewallActionIPTablesConfig
iptablesNotFound bool
}
func NewIPTablesAction() *IPTablesAction {
@@ -76,6 +79,10 @@ func (this *IPTablesAction) runActionSingleIP(action string, listType IPListType
if len(path) == 0 {
path, err = exec.LookPath("iptables")
if err != nil {
if this.iptablesNotFound {
return nil
}
this.iptablesNotFound = true
return err
}
}

View File

@@ -14,7 +14,7 @@ import (
var SharedActionManager = NewActionManager()
// 动作管理器定义
// ActionManager 动作管理器定义
type ActionManager struct {
locker sync.Mutex
@@ -23,7 +23,7 @@ type ActionManager struct {
instanceMap map[int64]ActionInterface // id => instance
}
// 获取动作管理对象
// NewActionManager 获取动作管理对象
func NewActionManager() *ActionManager {
return &ActionManager{
configMap: map[int64]*firewallconfigs.FirewallActionConfig{},
@@ -31,7 +31,7 @@ func NewActionManager() *ActionManager {
}
}
// 更新配置
// UpdateActions 更新配置
func (this *ActionManager) UpdateActions(actions []*firewallconfigs.FirewallActionConfig) {
this.locker.Lock()
defer this.locker.Unlock()
@@ -108,14 +108,14 @@ func (this *ActionManager) UpdateActions(actions []*firewallconfigs.FirewallActi
}
}
// 查找事件对应的动作
// FindEventActions 查找事件对应的动作
func (this *ActionManager) FindEventActions(eventLevel string) []ActionInterface {
this.locker.Lock()
defer this.locker.Unlock()
return this.eventMap[eventLevel]
}
// 执行添加IP动作
// AddItem 执行添加IP动作
func (this *ActionManager) AddItem(listType IPListType, item *pb.IPItem) {
instances, ok := this.eventMap[item.EventLevel]
if ok {
@@ -128,7 +128,7 @@ func (this *ActionManager) AddItem(listType IPListType, item *pb.IPItem) {
}
}
// 执行删除IP动作
// DeleteItem 执行删除IP动作
func (this *ActionManager) DeleteItem(listType IPListType, item *pb.IPItem) {
instances, ok := this.eventMap[item.EventLevel]
if ok {

View File

@@ -10,7 +10,7 @@ import (
"path/filepath"
)
// 脚本命令动作
// ScriptAction 脚本命令动作
type ScriptAction struct {
BaseAction

View File

@@ -0,0 +1,130 @@
// 源码改自https://github.com/lionsoul2014/ip2region/blob/master/binding/golang/ip2region/ip2Region.go
package iplibrary
import (
"errors"
"io/ioutil"
"strconv"
"strings"
)
const (
IndexBlockLength = 12
)
var err error
type IP2Region struct {
headerSip []int64
headerPtr []int64
headerLen int64
// super block index info
firstIndexPtr int64
lastIndexPtr int64
totalBlocks int64
dbData []byte
}
type IpInfo struct {
CityId int64
Country string
Region string
Province string
City string
ISP string
}
func (ip IpInfo) String() string {
return strconv.FormatInt(ip.CityId, 10) + "|" + ip.Country + "|" + ip.Region + "|" + ip.Province + "|" + ip.City + "|" + ip.ISP
}
func getIpInfo(cityId int64, line []byte) *IpInfo {
lineSlice := strings.Split(string(line), "|")
ipInfo := &IpInfo{}
length := len(lineSlice)
ipInfo.CityId = cityId
if length < 5 {
for i := 0; i <= 5-length; i++ {
lineSlice = append(lineSlice, "")
}
}
ipInfo.Country = lineSlice[0]
ipInfo.Region = lineSlice[1]
ipInfo.Province = lineSlice[2]
ipInfo.City = lineSlice[3]
ipInfo.ISP = lineSlice[4]
return ipInfo
}
func NewIP2Region(path string) (*IP2Region, error) {
var region = &IP2Region{}
region.dbData, err = ioutil.ReadFile(path)
if err != nil {
return nil, err
}
region.firstIndexPtr = region.ipLongAtOffset(0)
region.lastIndexPtr = region.ipLongAtOffset(4)
region.totalBlocks = (region.lastIndexPtr-region.firstIndexPtr)/IndexBlockLength + 1
return region, nil
}
func (this *IP2Region) MemorySearch(ipStr string) (ipInfo *IpInfo, err error) {
ip, err := ip2long(ipStr)
if err != nil {
return nil, err
}
h := this.totalBlocks
var dataPtr, l int64
for l <= h {
m := (l + h) >> 1
p := this.firstIndexPtr + m*IndexBlockLength
sip := this.ipLongAtOffset(p)
if ip < sip {
h = m - 1
} else {
eip := this.ipLongAtOffset(p + 4)
if ip > eip {
l = m + 1
} else {
dataPtr = this.ipLongAtOffset(p + 8)
break
}
}
}
if dataPtr == 0 {
return nil, errors.New("not found")
}
dataLen := (dataPtr >> 24) & 0xFF
dataPtr = dataPtr & 0x00FFFFFF
return getIpInfo(this.ipLongAtOffset(dataPtr), this.dbData[(dataPtr)+4:dataPtr+dataLen]), nil
}
func (this *IP2Region) ipLongAtOffset(offset int64) int64 {
return int64(this.dbData[offset]) |
int64(this.dbData[offset+1])<<8 |
int64(this.dbData[offset+2])<<16 |
int64(this.dbData[offset+3])<<24
}
func ip2long(IpStr string) (int64, error) {
bits := strings.Split(IpStr, ".")
if len(bits) != 4 {
return 0, errors.New("ip format error")
}
var sum int64
for i, n := range bits {
bit, _ := strconv.ParseInt(n, 10, 64)
sum += bit << uint(24-8*i)
}
return sum, nil
}

View File

@@ -28,7 +28,7 @@ func (this *IPItem) Contains(ip uint64) bool {
case IPItemTypeIPv6:
return this.containsIPv6(ip)
case IPItemTypeAll:
return this.containsAll(ip)
return this.containsAll()
default:
return this.containsIPv4(ip)
}
@@ -63,7 +63,7 @@ func (this *IPItem) containsIPv6(ip uint64) bool {
}
// 检查是否包所有IP
func (this *IPItem) containsAll(ip uint64) bool {
func (this *IPItem) containsAll() bool {
if this.ExpiredAt > 0 && this.ExpiredAt < utils.UnixTime() {
return false
}

View File

@@ -7,6 +7,9 @@ import (
"sync"
)
var GlobalBlackIPList = NewIPList()
var GlobalWhiteIPList = NewIPList()
// IPList IP名单
// TODO IP名单可以分片关闭这样让每一片的数据量减少查询更快
type IPList struct {
@@ -26,11 +29,9 @@ func NewIPList() *IPList {
}
expireList := expires.NewList()
go func() {
expireList.StartGC(func(itemId int64) {
list.Delete(itemId)
})
}()
expireList.OnGC(func(itemId int64) {
list.Delete(itemId)
})
list.expireList = expireList
return list
}

View File

@@ -0,0 +1,145 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import (
"database/sql"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/Tea"
_ "github.com/mattn/go-sqlite3"
"os"
"path/filepath"
)
type IPListDB struct {
db *sql.DB
itemTableName string
deleteItemStmt *sql.Stmt
insertItemStmt *sql.Stmt
selectItemsStmt *sql.Stmt
dir string
}
func NewIPListDB() (*IPListDB, error) {
var db = &IPListDB{
itemTableName: "ipItems",
dir: filepath.Clean(Tea.Root + "/data"),
}
err := db.init()
return db, err
}
func (this *IPListDB) init() error {
// 检查目录是否存在
_, err := os.Stat(this.dir)
if err != nil {
err = os.MkdirAll(this.dir, 0777)
if err != nil {
return err
}
remotelogs.Println("CACHE", "create cache dir '"+this.dir+"'")
}
db, err := sql.Open("sqlite3", "file:"+this.dir+"/ip_list.db?cache=shared&mode=rwc&_journal_mode=WAL")
if err != nil {
return err
}
db.SetMaxOpenConns(1)
this.db = db
// 初始化数据库
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"listId" integer DEFAULT 0,
"listType" varchar(32),
"isGlobal" integer(1) DEFAULT 0,
"type" varchar(16),
"itemId" integer DEFAULT 0,
"ipFrom" varchar(64) DEFAULT 0,
"ipTo" varchar(64) DEFAULT 0,
"expiredAt" integer DEFAULT 0,
"eventLevel" varchar(32),
"isDeleted" integer(1) DEFAULT 0,
"version" integer DEFAULT 0,
"nodeId" integer DEFAULT 0,
"serverId" integer DEFAULT 0
);
CREATE INDEX IF NOT EXISTS "ip_list_itemId"
ON "` + this.itemTableName + `" (
"itemId" ASC
);
CREATE INDEX IF NOT EXISTS "ip_list_expiredAt"
ON "` + this.itemTableName + `" (
"expiredAt" ASC
);
`)
if err != nil {
return err
}
// 初始化SQL语句
this.deleteItemStmt, err = this.db.Prepare(`DELETE FROM "` + this.itemTableName + `" WHERE "itemId"=?`)
if err != nil {
return err
}
this.insertItemStmt, err = this.db.Prepare(`INSERT INTO "` + this.itemTableName + `" ("listId", "listType", "isGlobal", "type", "itemId", "ipFrom", "ipTo", "expiredAt", "eventLevel", "isDeleted", "version", "nodeId", "serverId") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
return err
}
this.selectItemsStmt, err = this.db.Prepare(`SELECT "listId", "listType", "isGlobal", "type", "itemId", "ipFrom", "ipTo", "expiredAt", "eventLevel", "isDeleted", "version", "nodeId", "serverId" FROM "` + this.itemTableName + `" ORDER BY "version" ASC, "itemId" ASC LIMIT ?, ?`)
if err != nil {
return err
}
this.db = db
return nil
}
func (this *IPListDB) AddItem(item *pb.IPItem) error {
_, err := this.deleteItemStmt.Exec(item.Id)
if err != nil {
return err
}
_, err = this.insertItemStmt.Exec(item.ListId, item.ListType, item.IsGlobal, item.Type, item.Id, item.IpFrom, item.IpTo, item.ExpiredAt, item.EventLevel, item.IsDeleted, item.Version, item.NodeId, item.ServerId)
return err
}
func (this *IPListDB) ReadItems(offset int64, size int64) (items []*pb.IPItem, err error) {
rows, err := this.selectItemsStmt.Query(offset, size)
if err != nil {
return nil, err
}
defer func() {
_ = rows.Close()
}()
for rows.Next() {
// "listId", "listType", "isGlobal", "type", "itemId", "ipFrom", "ipTo", "expiredAt", "eventLevel", "isDeleted", "version", "nodeId", "serverId"
var pbItem = &pb.IPItem{}
err = rows.Scan(&pbItem.ListId, &pbItem.ListType, &pbItem.IsGlobal, &pbItem.Type, &pbItem.Id, &pbItem.IpFrom, &pbItem.IpTo, &pbItem.ExpiredAt, &pbItem.EventLevel, &pbItem.IsDeleted, &pbItem.Version, &pbItem.NodeId, &pbItem.ServerId)
if err != nil {
return nil, err
}
items = append(items, pbItem)
}
return
}
func (this *IPListDB) Close() error {
if this.db != nil {
_ = this.deleteItemStmt.Close()
_ = this.insertItemStmt.Close()
_ = this.selectItemsStmt.Close()
return this.db.Close()
}
return nil
}

View File

@@ -0,0 +1,60 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/logs"
"testing"
"time"
)
func TestIPListDB_AddItem(t *testing.T) {
db, err := NewIPListDB()
if err != nil {
t.Fatal(err)
}
err = db.AddItem(&pb.IPItem{
Id: 1,
IpFrom: "192.168.1.101",
IpTo: "",
Version: 1024,
ExpiredAt: time.Now().Unix(),
Reason: "",
ListId: 2,
IsDeleted: true,
Type: "ipv4",
EventLevel: "error",
ListType: "black",
IsGlobal: true,
CreatedAt: 0,
NodeId: 11,
ServerId: 22,
SourceNodeId: 0,
SourceServerId: 0,
SourceHTTPFirewallPolicyId: 0,
SourceHTTPFirewallRuleGroupId: 0,
SourceHTTPFirewallRuleSetId: 0,
SourceServer: nil,
SourceHTTPFirewallPolicy: nil,
SourceHTTPFirewallRuleGroup: nil,
SourceHTTPFirewallRuleSet: nil,
})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestIPListDB_ReadItems(t *testing.T) {
db, err := NewIPListDB()
if err != nil {
t.Fatal(err)
}
items, err := db.ReadItems(0, 2)
if err != nil {
t.Fatal(err)
}
logs.PrintAsJSON(items, t)
}

View File

@@ -6,7 +6,9 @@ import (
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/rands"
"runtime"
"runtime/debug"
"strconv"
"sync"
"testing"
"time"
)
@@ -281,6 +283,22 @@ func TestGC(t *testing.T) {
logs.PrintAsJSON(list.sortedItems, t)
}
func TestTooManyLists(t *testing.T) {
debug.SetMaxThreads(20)
var lists = []*IPList{}
var locker = &sync.Mutex{}
for i := 0; i < 1000; i++ {
locker.Lock()
lists = append(lists, NewIPList())
locker.Unlock()
}
time.Sleep(1 * time.Second)
t.Log(runtime.NumGoroutine())
t.Log(len(lists), "lists")
}
func BenchmarkIPList_Contains(b *testing.B) {
runtime.GOMAXPROCS(1)

View File

@@ -4,17 +4,16 @@ import (
"fmt"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/lionsoul2014/ip2region/binding/golang/ip2region"
"net"
"strings"
)
type IP2RegionLibrary struct {
db *ip2region.Ip2Region
db *IP2Region
}
func (this *IP2RegionLibrary) Load(dbPath string) error {
db, err := ip2region.New(dbPath)
db, err := NewIP2Region(dbPath)
if err != nil {
return err
}
@@ -49,6 +48,10 @@ func (this *IP2RegionLibrary) Lookup(ip string) (*Result, error) {
return nil, err
}
if info == nil {
return nil, nil
}
if info.Country == "0" {
info.Country = ""
}
@@ -76,7 +79,5 @@ func (this *IP2RegionLibrary) Lookup(ip string) (*Result, error) {
}
func (this *IP2RegionLibrary) Close() {
if this.db != nil {
this.db.Close()
}
}

View File

@@ -3,25 +3,83 @@ package iplibrary
import (
"github.com/iwind/TeaGo/Tea"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/rands"
"runtime"
"strconv"
"sync"
"testing"
"time"
)
func TestIP2RegionLibrary_Lookup_MemoryUsage(t *testing.T) {
var mem = &runtime.MemStats{}
runtime.ReadMemStats(mem)
library := &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
t.Fatal(err)
}
var mem2 = &runtime.MemStats{}
runtime.ReadMemStats(mem2)
t.Log((mem2.HeapInuse-mem.HeapInuse)/1024/1024, "MB")
}
func TestIP2RegionLibrary_Lookup_Single(t *testing.T) {
library := &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
t.Fatal(err)
}
for _, ip := range []string{"8.8.9.9"} {
result, err := library.Lookup(ip)
if err != nil {
t.Fatal(err)
}
t.Log("IP:", ip, "result:", result)
}
}
func TestIP2RegionLibrary_Lookup(t *testing.T) {
library := &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
t.Fatal(err)
}
result, err := library.Lookup("114.240.223.47")
for _, ip := range []string{"", "a", "1.1.1", "192.168.1.100", "114.240.223.47", "8.8.9.9", "::1"} {
result, err := library.Lookup(ip)
if err != nil {
t.Fatal(err)
}
t.Log("IP:", ip, "result:", result)
}
}
func TestIP2RegionLibrary_Lookup_Concurrent(t *testing.T) {
library := &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
t.Fatal(err)
}
logs.PrintAsJSON(result, t)
var count = 4000
var wg = sync.WaitGroup{}
wg.Add(count)
for i := 0; i < count; i++ {
go func() {
defer wg.Done()
for i := 0; i < 100; i++ {
_, _ = library.Lookup(strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)))
}
}()
}
wg.Done()
t.Log("ok")
}
func TestIP2RegionLibrary_Memory(t *testing.T) {
@@ -43,13 +101,13 @@ func TestIP2RegionLibrary_Memory(t *testing.T) {
func BenchmarkIP2RegionLibrary_Lookup(b *testing.B) {
runtime.GOMAXPROCS(1)
library := &IP2RegionLibrary{}
var library = &IP2RegionLibrary{}
err := library.Load(Tea.Root + "/resources/ipdata/ip2region/ip2region.db")
if err != nil {
b.Fatal(err)
}
for i := 0; i < b.N; i++ {
_, _ = library.Lookup(strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)) + "." + strconv.Itoa(rands.Int(0, 254)))
_, _ = library.Lookup("8.8.8.8")
}
}

View File

@@ -0,0 +1,55 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
)
// AllowIP 检查IP是否被允许访问
func AllowIP(ip string, serverId int64) bool {
var ipLong = utils.IP2Long(ip)
if ipLong == 0 {
return false
}
// check white lists
if GlobalWhiteIPList.Contains(ipLong) {
return true
}
if serverId > 0 {
var list = SharedServerListManager.FindWhiteList(serverId, false)
if list != nil && list.Contains(ipLong) {
return true
}
}
// check black lists
if GlobalBlackIPList.Contains(ipLong) {
return false
}
if serverId > 0 {
var list = SharedServerListManager.FindBlackList(serverId, false)
if list != nil && list.Contains(ipLong) {
return false
}
}
return true
}
// AllowIPStrings 检查一组IP是否被允许访问
func AllowIPStrings(ipStrings []string, serverId int64) bool {
if len(ipStrings) == 0 {
return true
}
for _, ip := range ipStrings {
isAllowed := AllowIP(ip, serverId)
if !isAllowed {
return false
}
}
return true
}

View File

@@ -0,0 +1,20 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import (
"testing"
"time"
)
func TestIPIsAllowed(t *testing.T) {
manager := NewIPListManager()
manager.init()
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
t.Log(AllowIP("127.0.0.1", 0))
t.Log(AllowIP("127.0.0.1", 23))
}

View File

@@ -21,7 +21,7 @@ func init() {
// 初始化
library, err := SharedManager.Load()
if err != nil {
remotelogs.Error("IP_LIBRARY", err.Error())
remotelogs.ErrorObject("IP_LIBRARY", err)
return
}
SharedLibrary = library

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
@@ -21,7 +22,9 @@ var SharedCountryManager = NewCountryManager()
func init() {
events.On(events.EventLoaded, func() {
go SharedCountryManager.Start()
goman.New(func() {
SharedCountryManager.Start()
})
})
}
@@ -46,13 +49,13 @@ func (this *CountryManager) Start() {
// 从缓存中读取
err := this.load()
if err != nil {
remotelogs.Error("COUNTRY_MANAGER", err.Error())
remotelogs.ErrorObject("COUNTRY_MANAGER", err)
}
// 第一次更新
err = this.loop()
if err != nil {
remotelogs.Error("COUNTRY_MANAGER", err.Error())
remotelogs.ErrorObject("COUNTRY_MANAGER", err)
}
// 定时更新
@@ -60,10 +63,10 @@ func (this *CountryManager) Start() {
events.On(events.EventQuit, func() {
ticker.Stop()
})
for range ticker.C {
for ticker.Next() {
err := this.loop()
if err != nil {
remotelogs.Error("COUNTRY_MANAGER", err.Error())
remotelogs.ErrorObject("COUNTRY_MANAGER", err)
}
}
}

View File

@@ -3,9 +3,12 @@ package iplibrary
import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/Tea"
"sync"
"time"
@@ -16,15 +19,15 @@ var IPListUpdateNotify = make(chan bool, 1)
func init() {
events.On(events.EventLoaded, func() {
go SharedIPListManager.Start()
goman.New(func() {
SharedIPListManager.Start()
})
})
}
// IPListManager IP名单管理
type IPListManager struct {
// 缓存文件
// 每行一个数据id|from|to|expiredAt
cacheFile string
db *IPListDB
version int64
pageSize int64
@@ -35,22 +38,24 @@ type IPListManager struct {
func NewIPListManager() *IPListManager {
return &IPListManager{
cacheFile: Tea.Root + "/configs/ip_list.cache",
pageSize: 1000,
listMap: map[int64]*IPList{},
pageSize: 500,
listMap: map[int64]*IPList{},
}
}
func (this *IPListManager) Start() {
// TODO 从缓存当中读取数据
this.init()
// 第一次读取
err := this.loop()
if err != nil {
remotelogs.Error("IP_LIST_MANAGER", err.Error())
remotelogs.ErrorObject("IP_LIST_MANAGER", err)
}
ticker := time.NewTicker(60 * time.Second)
if Tea.IsTesting() {
ticker = time.NewTicker(10 * time.Second)
}
events.On(events.EventQuit, func() {
ticker.Stop()
})
@@ -64,7 +69,7 @@ func (this *IPListManager) Start() {
if err != nil {
countErrors++
remotelogs.Error("IP_LIST_MANAGER", err.Error())
remotelogs.ErrorObject("IP_LIST_MANAGER", err)
// 连续错误小于3次的我们立即重试
if countErrors <= 3 {
@@ -79,6 +84,31 @@ func (this *IPListManager) Start() {
}
}
func (this *IPListManager) init() {
// 从数据库中当中读取数据
db, err := NewIPListDB()
if err != nil {
remotelogs.Error("IP_LIST_MANAGER", "create ip list local database failed: "+err.Error())
} else {
this.db = db
var offset int64 = 0
var size int64 = 1000
for {
items, err := db.ReadItems(offset, size)
if err != nil {
remotelogs.Error("IP_LIST_MANAGER", "read ip list from local database failed: "+err.Error())
} else {
if len(items) == 0 {
break
}
this.processItems(items, false)
}
offset += int64(len(items))
}
}
}
func (this *IPListManager) loop() error {
for {
hasNext, err := this.fetch()
@@ -88,10 +118,9 @@ func (this *IPListManager) loop() error {
if !hasNext {
break
}
time.Sleep(1 * time.Second)
}
// TODO 写入到缓存当中
return nil
}
@@ -111,22 +140,69 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
if len(items) == 0 {
return false, nil
}
// 保存到本地数据库
if this.db != nil {
for _, item := range items {
err = this.db.AddItem(item)
if err != nil {
remotelogs.Error("IP_LIST_MANAGER", "insert item to local database failed: "+err.Error())
}
}
}
this.processItems(items, true)
return true, nil
}
func (this *IPListManager) FindList(listId int64) *IPList {
this.locker.Lock()
var changedLists = map[*IPList]bool{}
list, _ := this.listMap[listId]
this.locker.Unlock()
return list
}
func (this *IPListManager) processItems(items []*pb.IPItem, shouldExecute bool) {
this.locker.Lock()
var changedLists = map[*IPList]zero.Zero{}
for _, item := range items {
list, ok := this.listMap[item.ListId]
if !ok {
var list *IPList
// TODO 实现节点专有List
if item.ServerId > 0 { // 服务专有List
switch item.ListType {
case "black":
list = SharedServerListManager.FindBlackList(item.ServerId, true)
case "white":
list = SharedServerListManager.FindWhiteList(item.ServerId, true)
}
} else if item.IsGlobal { // 全局List
switch item.ListType {
case "black":
list = GlobalBlackIPList
case "white":
list = GlobalWhiteIPList
}
} else { // 其他List
list = this.listMap[item.ListId]
}
if list == nil {
list = NewIPList()
this.listMap[item.ListId] = list
}
changedLists[list] = true
changedLists[list] = zero.New()
if item.IsDeleted {
list.Delete(item.Id)
// 从WAF名单中删除
waf.SharedIPBlackList.RemoveIP(item.IpFrom, item.ServerId)
// 操作事件
SharedActionManager.DeleteItem(item.ListType, item)
if shouldExecute {
SharedActionManager.DeleteItem(item.ListType, item)
}
continue
}
@@ -141,8 +217,10 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
})
// 事件操作
SharedActionManager.DeleteItem(item.ListType, item)
SharedActionManager.AddItem(item.ListType, item)
if shouldExecute {
SharedActionManager.DeleteItem(item.ListType, item)
SharedActionManager.AddItem(item.ListType, item)
}
}
for changedList := range changedLists {
@@ -151,13 +229,4 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
this.locker.Unlock()
this.version = items[len(items)-1].Version
return true, nil
}
func (this *IPListManager) FindList(listId int64) *IPList {
this.locker.Lock()
list, _ := this.listMap[listId]
this.locker.Unlock()
return list
}

View File

@@ -1,10 +1,36 @@
package iplibrary
import "testing"
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/logs"
"testing"
"time"
)
func TestIPListManager_init(t *testing.T) {
manager := NewIPListManager()
manager.init()
t.Log(manager.listMap)
t.Log(SharedServerListManager.blackMap)
logs.PrintAsJSON(GlobalBlackIPList.sortedItems, t)
}
func TestIPListManager_check(t *testing.T) {
manager := NewIPListManager()
manager.init()
var before = time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
t.Log(SharedServerListManager.FindBlackList(23, true).Contains(utils.IP2Long("127.0.0.2")))
t.Log(GlobalBlackIPList.Contains(utils.IP2Long("127.0.0.6")))
}
func TestIPListManager_loop(t *testing.T) {
manager := NewIPListManager()
manager.pageSize = 2
manager.Start()
manager.pageSize = 10
err := manager.loop()
if err != nil {
t.Fatal(err)

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
@@ -25,7 +26,9 @@ var SharedProvinceManager = NewProvinceManager()
func init() {
events.On(events.EventLoaded, func() {
go SharedProvinceManager.Start()
goman.New(func() {
SharedProvinceManager.Start()
})
})
}
@@ -50,13 +53,13 @@ func (this *ProvinceManager) Start() {
// 从缓存中读取
err := this.load()
if err != nil {
remotelogs.Error("PROVINCE_MANAGER", err.Error())
remotelogs.ErrorObject("PROVINCE_MANAGER", err)
}
// 第一次更新
err = this.loop()
if err != nil {
remotelogs.Error("PROVINCE_MANAGER", err.Error())
remotelogs.ErrorObject("PROVINCE_MANAGER", err)
}
// 定时更新
@@ -64,10 +67,10 @@ func (this *ProvinceManager) Start() {
events.On(events.EventQuit, func() {
ticker.Stop()
})
for range ticker.C {
for ticker.Next() {
err := this.loop()
if err != nil {
remotelogs.Error("PROVINCE_MANAGER", err.Error())
remotelogs.ErrorObject("PROVINCE_MANAGER", err)
}
}
}

View File

@@ -0,0 +1,61 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package iplibrary
import "sync"
var SharedServerListManager = NewServerListManager()
// ServerListManager 服务相关名单
type ServerListManager struct {
whiteMap map[int64]*IPList // serverId => *List
blackMap map[int64]*IPList // serverId => *List
locker sync.RWMutex
}
func NewServerListManager() *ServerListManager {
return &ServerListManager{
whiteMap: map[int64]*IPList{},
blackMap: map[int64]*IPList{},
}
}
func (this *ServerListManager) FindWhiteList(serverId int64, autoCreate bool) *IPList {
this.locker.RLock()
list, ok := this.whiteMap[serverId]
this.locker.RUnlock()
if ok {
return list
}
if autoCreate {
list = NewIPList()
this.locker.Lock()
this.whiteMap[serverId] = list
this.locker.Unlock()
return list
}
return nil
}
func (this *ServerListManager) FindBlackList(serverId int64, autoCreate bool) *IPList {
this.locker.RLock()
list, ok := this.blackMap[serverId]
this.locker.RUnlock()
if ok {
return list
}
if autoCreate {
list = NewIPList()
this.locker.Lock()
this.blackMap[serverId] = list
this.locker.Unlock()
return list
}
return nil
}

View File

@@ -7,6 +7,7 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/iwind/TeaGo/Tea"
@@ -34,14 +35,14 @@ func NewUpdater() *Updater {
func (this *Updater) Start() {
// 这里不需要太频繁检查更新因为通常不需要更新IP库
ticker := time.NewTicker(1 * time.Hour)
go func() {
goman.New(func() {
for range ticker.C {
err := this.loop()
if err != nil {
remotelogs.Error("IP_LIBRARY", err.Error())
remotelogs.ErrorObject("IP_LIBRARY", err)
}
}
}()
})
}
// 单次任务

View File

@@ -7,9 +7,12 @@ import (
"encoding/json"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/trackers"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/Tea"
_ "github.com/mattn/go-sqlite3"
"os"
@@ -50,8 +53,8 @@ type Task struct {
selectTopStmt *sql.Stmt
sumStmt *sql.Stmt
serverIdMap map[int64]bool // 所有的服务Ids
timeMap map[string]bool // time => bool
serverIdMap map[int64]zero.Zero // 所有的服务Ids
timeMap map[string]zero.Zero // time => bool
serverIdMapLocker sync.Mutex
statsMap map[string]*Stat
@@ -63,8 +66,8 @@ type Task struct {
func NewTask(item *serverconfigs.MetricItemConfig) *Task {
return &Task{
item: item,
serverIdMap: map[int64]bool{},
timeMap: map[string]bool{},
serverIdMap: map[int64]zero.Zero{},
timeMap: map[string]zero.Zero{},
statsMap: map[string]*Stat{},
}
}
@@ -162,8 +165,10 @@ ON "` + this.statTableName + `" (
func (this *Task) Start() error {
// 读取数据
this.statsTicker = utils.NewTicker(1 * time.Minute)
go func() {
goman.New(func() {
for this.statsTicker.Next() {
var tr = trackers.Begin("[METRIC]DUMP_STATS_TO_LOCAL_DATABASE")
this.statsLocker.Lock()
var statsMap = this.statsMap
this.statsMap = map[string]*Stat{}
@@ -175,30 +180,36 @@ func (this *Task) Start() error {
remotelogs.Error("METRIC", "insert stat failed: "+err.Error())
}
}
tr.End()
}
}()
})
// 清理
this.cleanTicker = utils.NewTicker(24 * time.Hour)
go func() {
goman.New(func() {
for this.cleanTicker.Next() {
var tr = trackers.Begin("[METRIC]CLEAN_EXPIRED")
err := this.CleanExpired()
tr.End()
if err != nil {
remotelogs.Error("METRIC", "clean expired stats failed: "+err.Error())
}
}
}()
})
// 上传
this.uploadTicker = utils.NewTicker(this.item.UploadDuration())
go func() {
goman.New(func() {
for this.uploadTicker.Next() {
var tr = trackers.Begin("[METRIC]UPLOAD_STATS")
err := this.Upload(1 * time.Second)
tr.End()
if err != nil {
remotelogs.Error("METRIC", "upload stats failed: "+err.Error())
}
}
}()
})
return nil
}
@@ -284,8 +295,8 @@ func (this *Task) InsertStat(stat *Stat) error {
}
this.serverIdMapLocker.Lock()
this.serverIdMap[stat.ServerId] = true
this.timeMap[stat.Time] = true
this.serverIdMap[stat.ServerId] = zero.New()
this.timeMap[stat.Time] = zero.New()
this.serverIdMapLocker.Unlock()
keyData, err := json.Marshal(stat.Keys)
@@ -337,14 +348,14 @@ func (this *Task) Upload(pauseDuration time.Duration) error {
for serverId := range this.serverIdMap {
serverIds = append(serverIds, serverId)
}
this.serverIdMap = map[int64]bool{} // 清空数据
this.serverIdMap = map[int64]zero.Zero{} // 清空数据
// 时间
var times = []string{}
for t := range this.timeMap {
times = append(times, t)
}
this.timeMap = map[string]bool{} // 清空数据
this.timeMap = map[string]zero.Zero{} // 清空数据
this.serverIdMapLocker.Unlock()
@@ -461,7 +472,7 @@ func (this *Task) loadServerIdMap() error {
return err
}
this.serverIdMapLocker.Lock()
this.serverIdMap[serverId] = true
this.serverIdMap[serverId] = zero.New()
this.serverIdMapLocker.Unlock()
}
}
@@ -482,7 +493,7 @@ func (this *Task) loadServerIdMap() error {
return err
}
this.serverIdMapLocker.Lock()
this.timeMap[timeString] = true
this.timeMap[timeString] = zero.New()
this.serverIdMapLocker.Unlock()
}
}

View File

@@ -6,6 +6,7 @@ import (
"encoding/json"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/iwind/TeaGo/maps"
@@ -16,7 +17,9 @@ var SharedValueQueue = NewValueQueue()
func init() {
events.On(events.EventLoaded, func() {
go SharedValueQueue.Start()
goman.New(func() {
SharedValueQueue.Start()
})
})
}
@@ -36,7 +39,7 @@ func (this *ValueQueue) Start() {
// 这里单次循环就行因为Loop里已经使用了Range通道
err := this.Loop()
if err != nil {
remotelogs.Error("MONITOR_QUEUE", err.Error())
remotelogs.ErrorObject("MONITOR_QUEUE", err)
}
}
@@ -72,7 +75,11 @@ func (this *ValueQueue) Loop() error {
CreatedAt: value.CreatedAt,
})
if err != nil {
remotelogs.Error("MONITOR", err.Error())
if rpc.IsConnError(err) {
remotelogs.Warn("MONITOR", err.Error())
} else {
remotelogs.Error("MONITOR", err.Error())
}
continue
}
}

View File

@@ -0,0 +1,28 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package monitor
import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/logs"
"google.golang.org/grpc/status"
"testing"
)
func TestValueQueue_RPC(t *testing.T) {
rpcClient, err := rpc.SharedRPC()
if err != nil {
t.Fatal(err)
}
_, err = rpcClient.NodeValueRPC().CreateNodeValue(rpcClient.Context(), &pb.CreateNodeValueRequest{})
if err != nil {
statusErr, ok:= status.FromError(err)
if ok {
logs.Println(statusErr.Code())
}
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -9,16 +9,19 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/configs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/Tea"
"io"
"net"
"net/http"
"net/url"
"os/exec"
"strconv"
"strings"
@@ -28,6 +31,9 @@ import (
type APIStream struct {
stream pb.NodeService_NodeStreamClient
isQuiting bool
cancelFunc context.CancelFunc
}
func NewAPIStream() *APIStream {
@@ -35,17 +41,19 @@ func NewAPIStream() *APIStream {
}
func (this *APIStream) Start() {
isQuiting := false
events.On(events.EventQuit, func() {
isQuiting = true
this.isQuiting = true
if this.cancelFunc != nil {
this.cancelFunc()
}
})
for {
if isQuiting {
if this.isQuiting {
return
}
err := this.loop()
if err != nil {
remotelogs.Error("API_STREAM", err.Error())
remotelogs.Warn("API_STREAM", err.Error())
time.Sleep(10 * time.Second)
continue
}
@@ -58,19 +66,17 @@ func (this *APIStream) loop() error {
if err != nil {
return errors.Wrap(err)
}
isQuiting := false
ctx, cancelFunc := context.WithCancel(rpcClient.Context())
nodeStream, err := rpcClient.NodeRPC().NodeStream(ctx)
events.On(events.EventQuit, func() {
isQuiting = true
remotelogs.Println("API_STREAM", "quiting")
if nodeStream != nil {
cancelFunc()
}
})
ctx, cancelFunc := context.WithCancel(rpcClient.Context())
this.cancelFunc = cancelFunc
defer func() {
cancelFunc()
}()
nodeStream, err := rpcClient.NodeRPC().NodeStream(ctx)
if err != nil {
if isQuiting {
if this.isQuiting {
return nil
}
return errors.Wrap(err)
@@ -78,14 +84,14 @@ func (this *APIStream) loop() error {
this.stream = nodeStream
for {
if isQuiting {
logs.Println("API_STREAM", "quit")
if this.isQuiting {
remotelogs.Println("API_STREAM", "quit")
break
}
message, err := nodeStream.Recv()
if err != nil {
if isQuiting {
if this.isQuiting {
remotelogs.Println("API_STREAM", "quit")
return nil
}
@@ -112,6 +118,8 @@ func (this *APIStream) loop() error {
err = this.handleNewNodeTask(message)
case messageconfigs.MessageCodeCheckSystemdService: // 检查Systemd服务
err = this.handleCheckSystemdService(message)
case messageconfigs.MessageCodeChangeAPINode: // 修改API节点地址
err = this.handleChangeAPINode(message)
default:
err = this.handleUnknownMessage(message)
}
@@ -135,15 +143,11 @@ func (this *APIStream) handleConnectedAPINode(message *pb.NodeStreamMessage) err
return errors.Wrap(err)
}
rpcClient, err := rpc.SharedRPC()
_, err = rpc.SharedRPC()
if err != nil {
return errors.Wrap(err)
}
_, err = rpcClient.NodeRPC().UpdateNodeConnectedAPINodes(rpcClient.Context(), &pb.UpdateNodeConnectedAPINodesRequest{ApiNodeIds: []int64{msg.APINodeId}})
if err != nil {
return errors.Wrap(err)
}
remotelogs.Println("API_STREAM", "connected to api node '"+strconv.FormatInt(msg.APINodeId, 10)+"'")
// 重新读取配置
@@ -235,7 +239,7 @@ func (this *APIStream) handleReadCache(message *pb.NodeStreamMessage) error {
}()
}
reader, err := storage.OpenReader(msg.Key)
reader, err := storage.OpenReader(msg.Key, false)
if err != nil {
if err == caches.ErrNotFound {
this.replyFail(message.RequestId, "key not found")
@@ -571,6 +575,65 @@ func (this *APIStream) handleCheckSystemdService(message *pb.NodeStreamMessage)
return nil
}
// 修改API地址
func (this *APIStream) handleChangeAPINode(message *pb.NodeStreamMessage) error {
config, err := configs.LoadAPIConfig()
if err != nil {
this.replyFail(message.RequestId, "read config error: "+err.Error())
return nil
}
var messageData = &messageconfigs.ChangeAPINodeMessage{}
err = json.Unmarshal(message.DataJSON, messageData)
if err != nil {
this.replyFail(message.RequestId, "unmarshal message failed: "+err.Error())
return nil
}
_, err = url.Parse(messageData.Addr)
if err != nil {
this.replyFail(message.RequestId, "invalid new api node address: '"+messageData.Addr+"'")
return nil
}
config.RPC.Endpoints = []string{messageData.Addr}
// 保存到文件
err = config.WriteFile(Tea.ConfigFile("api.yaml"))
if err != nil {
this.replyFail(message.RequestId, "save config file failed: "+err.Error())
return nil
}
this.replyOk(message.RequestId, "")
goman.New(func() {
// 延后生效防止变更前的API无法读取到状态
time.Sleep(1 * time.Second)
rpcClient, err := rpc.SharedRPC()
if err != nil {
remotelogs.Error("API_STREAM", "change rpc endpoint to '"+
messageData.Addr+"' failed: "+err.Error())
return
}
rpcClient.Close()
err = rpcClient.UpdateConfig(config)
if err != nil {
remotelogs.Error("API_STREAM", "change rpc endpoint to '"+
messageData.Addr+"' failed: "+err.Error())
return
}
remotelogs.Println("API_STREAM", "change rpc endpoint to '"+
messageData.Addr+"' successfully")
})
return nil
}
// 处理未知消息
func (this *APIStream) handleUnknownMessage(message *pb.NodeStreamMessage) error {
this.replyFail(message.RequestId, "unknown message code '"+message.Code+"'")

View File

@@ -4,66 +4,53 @@ package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/monitor"
"github.com/iwind/TeaGo/maps"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/ratelimit"
"net"
"sync"
"sync/atomic"
"time"
)
// 流量统计
var inTrafficBytes = uint64(0)
var outTrafficBytes = uint64(0)
// 发送监控流量
func init() {
events.On(events.EventStart, func() {
ticker := time.NewTicker(1 * time.Minute)
go func() {
for range ticker.C {
// 加入到数据队列中
if inTrafficBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficIn, maps.Map{
"total": inTrafficBytes,
})
}
if outTrafficBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficOut, maps.Map{
"total": outTrafficBytes,
})
}
// 重置数据
atomic.StoreUint64(&inTrafficBytes, 0)
atomic.StoreUint64(&outTrafficBytes, 0)
}
}()
})
}
// ClientConn 客户端连接
type ClientConn struct {
rawConn net.Conn
isClosed bool
once sync.Once
globalLimiter *ratelimit.Counter
isTLS bool
hasRead bool
BaseClientConn
}
func NewClientConn(conn net.Conn, quickClose bool) net.Conn {
func NewClientConn(conn net.Conn, isTLS bool, quickClose bool, globalLimiter *ratelimit.Counter) net.Conn {
if quickClose {
// TCP
tcpConn, ok := conn.(*net.TCPConn)
if ok {
// TODO 可以设置此值
_ = tcpConn.SetLinger(3)
// TODO 可以在配置中设置此值
_ = tcpConn.SetLinger(nodeconfigs.DefaultTCPLinger)
}
}
return &ClientConn{rawConn: conn}
return &ClientConn{BaseClientConn: BaseClientConn{rawConn: conn}, isTLS: isTLS, globalLimiter: globalLimiter}
}
func (this *ClientConn) Read(b []byte) (n int, err error) {
if this.isTLS {
if !this.hasRead {
_ = this.rawConn.SetReadDeadline(time.Now().Add(5 * time.Second)) // TODO 握手超时时间可以设置
this.hasRead = true
defer func() {
_ = this.rawConn.SetReadDeadline(time.Time{})
}()
}
}
n, err = this.rawConn.Read(b)
if n > 0 {
atomic.AddUint64(&inTrafficBytes, uint64(n))
atomic.AddUint64(&teaconst.InTrafficBytes, uint64(n))
}
return
}
@@ -71,13 +58,24 @@ func (this *ClientConn) Read(b []byte) (n int, err error) {
func (this *ClientConn) Write(b []byte) (n int, err error) {
n, err = this.rawConn.Write(b)
if n > 0 {
atomic.AddUint64(&outTrafficBytes, uint64(n))
atomic.AddUint64(&teaconst.OutTrafficBytes, uint64(n))
}
return
}
func (this *ClientConn) Close() error {
this.isClosed = true
// 全局并发数限制
this.once.Do(func() {
if this.globalLimiter != nil {
this.globalLimiter.Release()
}
})
// 单个服务并发数限制
sharedClientConnLimiter.Remove(this.rawConn.RemoteAddr().String())
return this.rawConn.Close()
}
@@ -100,7 +98,3 @@ func (this *ClientConn) SetReadDeadline(t time.Time) error {
func (this *ClientConn) SetWriteDeadline(t time.Time) error {
return this.rawConn.SetWriteDeadline(t)
}
func (this *ClientConn) IsClosed() bool {
return this.isClosed
}

View File

@@ -0,0 +1,38 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import "net"
type BaseClientConn struct {
rawConn net.Conn
isBound bool
serverId int64
remoteAddr string
isClosed bool
}
func (this *BaseClientConn) IsClosed() bool {
return this.isClosed
}
// IsBound 是否已绑定服务
func (this *BaseClientConn) IsBound() bool {
return this.isBound
}
// Bind 绑定服务
func (this *BaseClientConn) Bind(serverId int64, remoteAddr string, maxConnsPerServer int, maxConnsPerIP int) bool {
if this.isBound {
return true
}
this.isBound = true
this.serverId = serverId
this.remoteAddr = remoteAddr
// 检查是否可以连接
return sharedClientConnLimiter.Add(this.rawConn.RemoteAddr().String(), serverId, remoteAddr, maxConnsPerServer, maxConnsPerIP)
}

View File

@@ -0,0 +1,14 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
type ClientConnInterface interface {
// IsClosed 是否已关闭
IsClosed() bool
// IsBound 是否已绑定服务
IsBound() bool
// Bind 绑定服务
Bind(serverId int64, remoteAddr string, maxConnsPerServer int, maxConnsPerIP int) bool
}

View File

@@ -0,0 +1,130 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/TeaOSLab/EdgeNode/internal/zero"
"sync"
)
var sharedClientConnLimiter = NewClientConnLimiter()
// ClientConnRemoteAddr 客户端地址定义
type ClientConnRemoteAddr struct {
remoteAddr string
serverId int64
}
// ClientConnLimiter 客户端连接数限制
type ClientConnLimiter struct {
remoteAddrMap map[string]*ClientConnRemoteAddr // raw remote addr => remoteAddr
ipConns map[string]map[string]zero.Zero // remoteAddr => { raw remote addr => Zero }
serverConns map[int64]map[string]zero.Zero // serverId => { remoteAddr => Zero }
locker sync.Mutex
}
func NewClientConnLimiter() *ClientConnLimiter {
return &ClientConnLimiter{
remoteAddrMap: map[string]*ClientConnRemoteAddr{},
ipConns: map[string]map[string]zero.Zero{},
serverConns: map[int64]map[string]zero.Zero{},
}
}
// Add 添加新连接
// 返回值为true的时候表示允许添加否则表示不允许添加
func (this *ClientConnLimiter) Add(rawRemoteAddr string, serverId int64, remoteAddr string, maxConnsPerServer int, maxConnsPerIP int) bool {
if (maxConnsPerServer <= 0 && maxConnsPerIP <= 0) || len(remoteAddr) == 0 || serverId <= 0 {
return true
}
this.locker.Lock()
defer this.locker.Unlock()
// 检查服务连接数
var serverMap = this.serverConns[serverId]
if maxConnsPerServer > 0 {
if serverMap == nil {
serverMap = map[string]zero.Zero{}
this.serverConns[serverId] = serverMap
}
if maxConnsPerServer <= len(serverMap) {
return false
}
}
// 检查IP连接数
var ipMap = this.ipConns[remoteAddr]
if maxConnsPerIP > 0 {
if ipMap == nil {
ipMap = map[string]zero.Zero{}
this.ipConns[remoteAddr] = ipMap
}
if maxConnsPerIP > 0 && maxConnsPerIP <= len(ipMap) {
return false
}
}
this.remoteAddrMap[rawRemoteAddr] = &ClientConnRemoteAddr{
remoteAddr: remoteAddr,
serverId: serverId,
}
if maxConnsPerServer > 0 {
serverMap[rawRemoteAddr] = zero.New()
}
if maxConnsPerIP > 0 {
ipMap[rawRemoteAddr] = zero.New()
}
return true
}
// Remove 删除连接
func (this *ClientConnLimiter) Remove(rawRemoteAddr string) {
this.locker.Lock()
defer this.locker.Unlock()
addr, ok := this.remoteAddrMap[rawRemoteAddr]
if !ok {
return
}
delete(this.remoteAddrMap, rawRemoteAddr)
delete(this.ipConns[addr.remoteAddr], rawRemoteAddr)
delete(this.serverConns[addr.serverId], rawRemoteAddr)
if len(this.ipConns[addr.remoteAddr]) == 0 {
delete(this.ipConns, addr.remoteAddr)
}
if len(this.serverConns[addr.serverId]) == 0 {
delete(this.serverConns, addr.serverId)
}
}
// Conns 获取连接信息
// 用于调试
func (this *ClientConnLimiter) Conns() (ipConns map[string][]string, serverConns map[int64][]string) {
this.locker.Lock()
defer this.locker.Unlock()
ipConns = map[string][]string{} // ip => [addr1, addr2, ...]
serverConns = map[int64][]string{} // serverId => [addr1, addr2, ...]
for ip, m := range this.ipConns {
for addr := range m {
ipConns[ip] = append(ipConns[ip], addr)
}
}
for serverId, m := range this.serverConns {
for addr := range m {
serverConns[serverId] = append(serverConns[serverId], addr)
}
}
return
}

View File

@@ -0,0 +1,38 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/iwind/TeaGo/logs"
"testing"
)
func TestClientConnLimiter_Add(t *testing.T) {
var limiter = NewClientConnLimiter()
{
b := limiter.Add("127.0.0.1:1234", 1, "192.168.1.100", 10, 5)
t.Log(b)
}
{
b := limiter.Add("127.0.0.1:1235", 1, "192.168.1.100", 10, 5)
t.Log(b)
}
{
b := limiter.Add("127.0.0.1:1236", 1, "192.168.1.100", 10, 5)
t.Log(b)
}
{
b := limiter.Add("127.0.0.1:1237", 1, "192.168.1.101", 10, 5)
t.Log(b)
}
{
b := limiter.Add("127.0.0.1:1238", 1, "192.168.1.100", 5, 5)
t.Log(b)
}
limiter.Remove("127.0.0.1:1238")
limiter.Remove("127.0.0.1:1239")
limiter.Remove("127.0.0.1:1237")
logs.PrintAsJSON(limiter.remoteAddrMap, t)
logs.PrintAsJSON(limiter.ipConns, t)
logs.PrintAsJSON(limiter.serverConns, t)
}

View File

@@ -0,0 +1,40 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/monitor"
"github.com/iwind/TeaGo/maps"
"sync/atomic"
"time"
)
// 发送监控流量
func init() {
events.On(events.EventStart, func() {
ticker := time.NewTicker(1 * time.Minute)
goman.New(func() {
for range ticker.C {
// 加入到数据队列中
if teaconst.InTrafficBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficIn, maps.Map{
"total": teaconst.InTrafficBytes,
})
}
if teaconst.OutTrafficBytes > 0 {
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemTrafficOut, maps.Map{
"total": teaconst.OutTrafficBytes,
})
}
// 重置数据
atomic.StoreUint64(&teaconst.InTrafficBytes, 0)
atomic.StoreUint64(&teaconst.OutTrafficBytes, 0)
}
})
})
}

View File

@@ -11,12 +11,10 @@ func isClientConnClosed(conn net.Conn) bool {
if conn == nil {
return true
}
clientConn, ok := conn.(*ClientConn)
clientConn, ok := conn.(ClientConnInterface)
if ok {
return clientConn.IsClosed()
}
// TODO 解决tls.Conn无法获取底层连接对象的问题
return false
return true
}

View File

@@ -3,46 +3,66 @@
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/firewallconfigs"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"github.com/TeaOSLab/EdgeNode/internal/ratelimit"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"net"
)
var sharedConnectionsLimiter = ratelimit.NewCounter(nodeconfigs.DefaultTCPMaxConnections)
// ClientListener 客户端网络监听
type ClientListener struct {
rawListener net.Listener
isTLS bool
quickClose bool
}
func NewClientListener(listener net.Listener, quickClose bool) net.Listener {
func NewClientListener1(listener net.Listener, quickClose bool) *ClientListener {
return &ClientListener{
rawListener: listener,
quickClose: quickClose,
}
}
func (this *ClientListener) SetIsTLS(isTLS bool) {
this.isTLS = isTLS
}
func (this *ClientListener) IsTLS() bool {
return this.isTLS
}
func (this *ClientListener) Accept() (net.Conn, error) {
// 限制并发连接数
var limiter = sharedConnectionsLimiter
limiter.Ack()
conn, err := this.rawListener.Accept()
if err != nil {
limiter.Release()
return nil, err
}
// 是否在WAF名单中
ip, _, err := net.SplitHostPort(conn.RemoteAddr().String())
if err == nil {
if !waf.SharedIPWhiteList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip) &&
waf.SharedIPBlackList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip) {
if !iplibrary.AllowIP(ip, 0) || (!waf.SharedIPWhiteList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip) &&
waf.SharedIPBlackList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, ip)) {
tcpConn, ok := conn.(*net.TCPConn)
if ok {
_ = tcpConn.SetLinger(0)
}
_ = conn.Close()
limiter.Release()
return this.Accept()
}
}
return NewClientConn(conn, this.quickClose), nil
return NewClientConn(conn, this.isTLS, this.quickClose, limiter), nil
}
func (this *ClientListener) Close() error {

View File

@@ -0,0 +1,57 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"crypto/tls"
"net"
"time"
)
// ClientTLSConn TLS连接封装
type ClientTLSConn struct {
BaseClientConn
}
func NewClientTLSConn(conn *tls.Conn) net.Conn {
return &ClientTLSConn{BaseClientConn{rawConn: conn}}
}
func (this *ClientTLSConn) Read(b []byte) (n int, err error) {
n, err = this.rawConn.Read(b)
return
}
func (this *ClientTLSConn) Write(b []byte) (n int, err error) {
n, err = this.rawConn.Write(b)
return
}
func (this *ClientTLSConn) Close() error {
this.isClosed = true
// 单个服务并发数限制
sharedClientConnLimiter.Remove(this.rawConn.RemoteAddr().String())
return this.rawConn.Close()
}
func (this *ClientTLSConn) LocalAddr() net.Addr {
return this.rawConn.LocalAddr()
}
func (this *ClientTLSConn) RemoteAddr() net.Addr {
return this.rawConn.RemoteAddr()
}
func (this *ClientTLSConn) SetDeadline(t time.Time) error {
return this.rawConn.SetDeadline(t)
}
func (this *ClientTLSConn) SetReadDeadline(t time.Time) error {
return this.rawConn.SetReadDeadline(t)
}
func (this *ClientTLSConn) SetWriteDeadline(t time.Time) error {
return this.rawConn.SetWriteDeadline(t)
}

View File

@@ -1,9 +1,12 @@
package nodes
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"strings"
"time"
)
@@ -12,17 +15,21 @@ var sharedHTTPAccessLogQueue = NewHTTPAccessLogQueue()
// HTTPAccessLogQueue HTTP访问日志队列
type HTTPAccessLogQueue struct {
queue chan *pb.HTTPAccessLog
rpcClient *rpc.RPCClient
}
// NewHTTPAccessLogQueue 获取新对象
func NewHTTPAccessLogQueue() *HTTPAccessLogQueue {
// 队列中最大的值,超出此数量的访问日志会被丢弃
// TODO 需要可以在界面中设置
maxSize := 10000
maxSize := 20000
queue := &HTTPAccessLogQueue{
queue: make(chan *pb.HTTPAccessLog, maxSize),
}
go queue.Start()
goman.New(func() {
queue.Start()
})
return queue
}
@@ -49,8 +56,9 @@ func (this *HTTPAccessLogQueue) Push(accessLog *pb.HTTPAccessLog) {
// 上传访问日志
func (this *HTTPAccessLogQueue) loop() error {
accessLogs := []*pb.HTTPAccessLog{}
count := 0
var accessLogs = []*pb.HTTPAccessLog{}
var count = 0
Loop:
for {
select {
@@ -59,7 +67,7 @@ Loop:
count++
// 每次只提交 N 条访问日志,防止网络拥堵
if count > 1000 {
if count > 2000 {
break Loop
}
default:
@@ -72,15 +80,65 @@ Loop:
}
// 发送到API
client, err := rpc.SharedRPC()
if err != nil {
return err
if this.rpcClient == nil {
client, err := rpc.SharedRPC()
if err != nil {
return err
}
this.rpcClient = client
}
_, err = client.HTTPAccessLogRPC().CreateHTTPAccessLogs(client.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: accessLogs})
_, err := this.rpcClient.HTTPAccessLogRPC().CreateHTTPAccessLogs(this.rpcClient.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: accessLogs})
if err != nil {
// 是否包含了invalid UTF-8
if strings.Contains(err.Error(), "string field contains invalid UTF-8") {
for _, accessLog := range accessLogs {
this.toValidUTF8(accessLog)
}
// 重新提交
_, err = this.rpcClient.HTTPAccessLogRPC().CreateHTTPAccessLogs(this.rpcClient.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: accessLogs})
return err
}
return err
}
return nil
}
func (this *HTTPAccessLogQueue) toValidUTF8(accessLog *pb.HTTPAccessLog) {
accessLog.RemoteUser = this.toValidUTF8string(accessLog.RemoteUser)
accessLog.RequestURI = this.toValidUTF8string(accessLog.RequestURI)
accessLog.RequestPath = this.toValidUTF8string(accessLog.RequestPath)
accessLog.RequestFilename = this.toValidUTF8string(accessLog.RequestFilename)
accessLog.RequestBody = bytes.ToValidUTF8(accessLog.RequestBody, []byte{})
for _, v := range accessLog.SentHeader {
for index, s := range v.Values {
v.Values[index] = this.toValidUTF8string(s)
}
}
accessLog.Referer = this.toValidUTF8string(accessLog.Referer)
accessLog.UserAgent = this.toValidUTF8string(accessLog.UserAgent)
accessLog.Request = this.toValidUTF8string(accessLog.Request)
accessLog.ContentType = this.toValidUTF8string(accessLog.ContentType)
for k, c := range accessLog.Cookie {
accessLog.Cookie[k] = this.toValidUTF8string(c)
}
accessLog.Args = this.toValidUTF8string(accessLog.Args)
accessLog.QueryString = this.toValidUTF8string(accessLog.QueryString)
for _, v := range accessLog.Header {
for index, s := range v.Values {
v.Values[index] = this.toValidUTF8string(s)
}
}
}
func (this *HTTPAccessLogQueue) toValidUTF8string(v string) string {
return strings.ToValidUTF8(v, "")
}

View File

@@ -0,0 +1,133 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
_ "github.com/iwind/TeaGo/bootstrap"
"google.golang.org/grpc/status"
"reflect"
"runtime"
"strconv"
"strings"
"testing"
"time"
)
func TestHTTPAccessLogQueue_Push(t *testing.T) {
// 发送到API
client, err := rpc.SharedRPC()
if err != nil {
t.Fatal(err)
}
var requestId = 1_000_000
var utf8Bytes = []byte{}
for i := 0; i < 254; i++ {
utf8Bytes = append(utf8Bytes, uint8(i))
}
//bytes = []byte("真不错")
var accessLog = &pb.HTTPAccessLog{
ServerId: 23,
RequestId: strconv.FormatInt(time.Now().Unix(), 10) + strconv.Itoa(requestId) + strconv.FormatInt(1, 10),
NodeId: 48,
Host: "www.hello.com",
RequestURI: string(utf8Bytes),
RequestPath: string(utf8Bytes),
Timestamp: time.Now().Unix(),
Cookie: map[string]string{"test": string(utf8Bytes)},
Header: map[string]*pb.Strings{
"test": {Values: []string{string(utf8Bytes)}},
},
}
new(HTTPAccessLogQueue).toValidUTF8(accessLog)
// logs.PrintAsJSON(accessLog)
//t.Log(strings.ToValidUTF8(string(utf8Bytes), ""))
_, err = client.HTTPAccessLogRPC().CreateHTTPAccessLogs(client.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: []*pb.HTTPAccessLog{
accessLog,
}})
if err != nil {
// 这里只是为了重现错误
t.Logf("%#v, %s", err, err.Error())
statusErr, ok := status.FromError(err)
if ok {
t.Logf("%#v", statusErr)
}
return
}
t.Log("ok")
}
func TestHTTPAccessLogQueue_Push2(t *testing.T) {
var utf8Bytes = []byte{}
for i := 0; i < 254; i++ {
utf8Bytes = append(utf8Bytes, uint8(i))
}
var accessLog = &pb.HTTPAccessLog{
ServerId: 23,
RequestId: strconv.FormatInt(time.Now().Unix(), 10) + strconv.Itoa(1) + strconv.FormatInt(1, 10),
NodeId: 48,
Host: "www.hello.com",
RequestURI: string(utf8Bytes),
RequestPath: string(utf8Bytes),
Timestamp: time.Now().Unix(),
}
var v = reflect.Indirect(reflect.ValueOf(accessLog))
var countFields = v.NumField()
for i := 0; i < countFields; i++ {
var field = v.Field(i)
if field.Kind() == reflect.String {
field.SetString(strings.ToValidUTF8(field.String(), ""))
}
}
client, err := rpc.SharedRPC()
if err != nil {
t.Fatal(err)
}
_, err = client.HTTPAccessLogRPC().CreateHTTPAccessLogs(client.Context(), &pb.CreateHTTPAccessLogsRequest{HttpAccessLogs: []*pb.HTTPAccessLog{
accessLog,
}})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func BenchmarkHTTPAccessLogQueue_ToValidUTF8(b *testing.B) {
runtime.GOMAXPROCS(1)
var utf8Bytes = []byte{}
for i := 0; i < 254; i++ {
utf8Bytes = append(utf8Bytes, uint8(i))
}
for i := 0; i < b.N; i++ {
_ = bytes.ToValidUTF8(utf8Bytes, nil)
}
}
func BenchmarkHTTPAccessLogQueue_ToValidUTF8String(b *testing.B) {
runtime.GOMAXPROCS(1)
var utf8Bytes = []byte{}
for i := 0; i < 254; i++ {
utf8Bytes = append(utf8Bytes, uint8(i))
}
var s = string(utf8Bytes)
for i := 0; i < b.N; i++ {
_ = strings.ToValidUTF8(s, "")
}
}

View File

@@ -5,6 +5,7 @@ import (
"crypto/tls"
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/pires/go-proxyproto"
"net"
@@ -33,7 +34,9 @@ func NewHTTPClientPool() *HTTPClientPool {
clientsMap: map[string]*HTTPClient{},
}
go pool.cleanClients()
goman.New(func() {
pool.cleanClients()
})
return pool
}
@@ -171,7 +174,7 @@ func (this *HTTPClientPool) Client(req *HTTPRequest, origin *serverconfigs.Origi
MaxConnsPerHost: maxConnections,
IdleConnTimeout: idleTimeout,
ExpectContinueTimeout: 1 * time.Second,
TLSHandshakeTimeout: 0, // 不限
TLSHandshakeTimeout: 10 * time.Second,
TLSClientConfig: tlsConfig,
Proxy: nil,
}

View File

@@ -1,6 +1,7 @@
package nodes
import (
"bytes"
"context"
"errors"
"fmt"
@@ -10,8 +11,10 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/metrics"
"github.com/TeaOSLab/EdgeNode/internal/stats"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
"io"
"io/ioutil"
"net"
"net/http"
"net/url"
@@ -25,17 +28,13 @@ import (
// 环境变量
var HOSTNAME, _ = os.Hostname()
// byte pool
var bytePool256b = utils.NewBytePool(20480, 256)
var bytePool1k = utils.NewBytePool(20480, 1024)
var bytePool32k = utils.NewBytePool(20480, 32*1024)
var bytePool128k = utils.NewBytePool(20480, 128*1024)
// errors
var errWritingToClient = errors.New("writing to client error")
// HTTPRequest HTTP请求
type HTTPRequest struct {
requestId string
// 外部参数
RawReq *http.Request
RawWriter http.ResponseWriter
@@ -64,11 +63,14 @@ type HTTPRequest struct {
rewriteRule *serverconfigs.HTTPRewriteRule // 匹配到的重写规则
rewriteReplace string // 重写规则的目标
rewriteIsExternalURL bool // 重写目标是否为外部URL
cacheRef *serverconfigs.HTTPCacheRef // 缓存设置
cacheKey string // 缓存使用的Key
isCached bool // 是否已经被缓存
isAttack bool // 是否是攻击请求
bodyData []byte // 读取的Body内容
cacheRef *serverconfigs.HTTPCacheRef // 缓存设置
cacheKey string // 缓存使用的Key
isCached bool // 是否已经被缓存
cacheCanTryStale bool // 是否可以尝试使用Stale缓存
isAttack bool // 是否是攻击请求
requestBodyData []byte // 读取的Body内容
// WAF相关
firewallPolicyId int64
@@ -86,25 +88,36 @@ type HTTPRequest struct {
// 初始化
func (this *HTTPRequest) init() {
this.writer = NewHTTPWriter(this, this.RawWriter)
this.web = &serverconfigs.HTTPWebConfig{IsOn: true}
this.web = &serverconfigs.HTTPWebConfig{
IsOn: true,
}
// this.uri = this.RawReq.URL.RequestURI()
// 之所以不使用RequestURI()是不想让URL中的Path被Encode
var urlPath = this.RawReq.URL.Path
if this.Server.Web != nil && this.Server.Web.MergeSlashes {
urlPath = utils.CleanPath(urlPath)
this.web.MergeSlashes = true
}
if len(this.RawReq.URL.RawQuery) > 0 {
this.uri = this.RawReq.URL.Path + "?" + this.RawReq.URL.RawQuery
this.uri = urlPath + "?" + this.RawReq.URL.RawQuery
} else {
this.uri = this.RawReq.URL.Path
this.uri = urlPath
}
this.rawURI = this.uri
this.varMapping = map[string]string{
// 缓存相关初始化
"cache.status": "BYPASS",
"cache.age": "0",
"cache.key": "",
"cache.policy.name": "",
"cache.policy.id": "0",
"cache.policy.type": "",
}
this.logAttrs = map[string]string{}
this.requestFromTime = time.Now()
this.requestId = httpRequestNextId()
}
// Do 执行请求
@@ -121,7 +134,7 @@ func (this *HTTPRequest) Do() {
// Web配置
err := this.configureWeb(this.Server.Web, true, 0)
if err != nil {
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, false)
this.doEnd()
return
}
@@ -137,9 +150,16 @@ func (this *HTTPRequest) Do() {
}
}
// 带宽限制
if this.Server.BandwidthLimit != nil && this.Server.BandwidthLimit.IsOn && !this.Server.BandwidthLimit.IsEmpty() && this.Server.BandwidthLimitStatus != nil && this.Server.BandwidthLimitStatus.IsValid() {
this.doBandwidthLimit()
// 套餐
if this.Server.UserPlan != nil && !this.Server.UserPlan.IsAvailable() {
this.doPlanExpires()
this.doEnd()
return
}
// 流量限制
if this.Server.TrafficLimit != nil && this.Server.TrafficLimit.IsOn && !this.Server.TrafficLimit.IsEmpty() && this.Server.TrafficLimitStatus != nil && this.Server.TrafficLimitStatus.IsValid() {
this.doTrafficLimit()
this.doEnd()
return
}
@@ -185,6 +205,28 @@ func (this *HTTPRequest) Do() {
// 开始调用
func (this *HTTPRequest) doBegin() {
// 处理request limit
if this.web.RequestLimit != nil &&
this.web.RequestLimit.IsOn {
if this.doRequestLimit() {
return
}
}
// 处理requestBody
if this.RawReq.ContentLength > 0 &&
this.web.AccessLogRef != nil &&
this.web.AccessLogRef.IsOn &&
this.web.AccessLogRef.ContainsField(serverconfigs.HTTPAccessLogFieldRequestBody) {
var err error
this.requestBodyData, err = ioutil.ReadAll(io.LimitReader(this.RawReq.Body, AccessLogMaxRequestBodySize))
if err != nil {
this.write50x(err, http.StatusBadGateway, false)
return
}
this.RawReq.Body = ioutil.NopCloser(io.MultiReader(bytes.NewBuffer(this.requestBodyData), this.RawReq.Body))
}
// 处理健康检查
var healthCheckKey = this.RawReq.Header.Get(serverconfigs.HealthCheckHeaderName)
if len(healthCheckKey) > 0 {
@@ -193,11 +235,6 @@ func (this *HTTPRequest) doBegin() {
}
}
// 统计
if this.web.StatRef != nil && this.web.StatRef.IsOn {
this.doStat()
}
// 跳转
if len(this.web.HostRedirects) > 0 {
if this.doHostRedirect() {
@@ -213,7 +250,7 @@ func (this *HTTPRequest) doBegin() {
// 缓存
if this.web.Cache != nil && this.web.Cache.IsOn {
if this.doCacheRead() {
if this.doCacheRead(false) {
return
}
}
@@ -262,14 +299,15 @@ func (this *HTTPRequest) doEnd() {
// 流量统计
// TODO 增加是否开启开关
// TODO 增加Header统计考虑从Conn中读取
if this.Server != nil {
if this.isCached {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, this.writer.sentBodyBytes, 1, 1, 0, 0)
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, this.writer.sentBodyBytes, 1, 1, 0, 0, this.Server.ShouldCheckTrafficLimit(), this.Server.PlanId())
} else {
if this.isAttack {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 1, this.writer.sentBodyBytes)
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 1, this.writer.sentBodyBytes, this.Server.ShouldCheckTrafficLimit(), this.Server.PlanId())
} else {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 0, 0)
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 0, 0, this.Server.ShouldCheckTrafficLimit(), this.Server.PlanId())
}
}
}
@@ -278,6 +316,12 @@ func (this *HTTPRequest) doEnd() {
if metrics.SharedManager.HasHTTPMetrics() {
this.doMetricsResponse()
}
// 统计
if this.web.StatRef != nil && this.web.StatRef.IsOn {
// 放到最后执行
this.doStat()
}
}
// RawURI 原始的请求URI
@@ -403,6 +447,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.Auth = web.Auth
}
// request limit
if web.RequestLimit != nil && (web.RequestLimit.IsPrior || isTop) {
this.web.RequestLimit = web.RequestLimit
}
// 重写规则
if len(web.RewriteRefs) > 0 {
for index, ref := range web.RewriteRefs {
@@ -471,6 +520,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
continue
}
if varMapping, isMatched := location.Match(rawPath, this.Format); isMatched {
// 检查专属域名
if len(location.Domains) > 0 && !configutils.MatchDomains(location.Domains, this.Host) {
continue
}
if len(varMapping) > 0 {
this.addVarMapping(varMapping)
}
@@ -540,6 +594,8 @@ func (this *HTTPRequest) Format(source string) string {
return strconv.Itoa(this.requestRemotePort())
case "remoteUser":
return this.requestRemoteUser()
case "requestId":
return this.requestId
case "requestURI", "requestUri":
return this.rawURI
case "requestURL":
@@ -611,6 +667,11 @@ func (this *HTTPRequest) Format(source string) string {
return this.requestString()
case "cookies":
return this.requestCookiesString()
case "isArgs":
if strings.Contains(this.uri, "?") {
return "?"
}
return ""
case "args", "queryString":
return this.requestQueryString()
case "headers":
@@ -907,7 +968,7 @@ func (this *HTTPRequest) requestRemotePort() int {
return 0
}
// 情趣的URI中的参数部分
// 获取的URI中的参数部分
func (this *HTTPRequest) requestQueryString() string {
uri, err := url.ParseRequestURI(this.uri)
if err != nil {
@@ -1081,44 +1142,53 @@ func (this *HTTPRequest) processRequestHeaders(reqHeader http.Header) {
}
}
// Add
for _, header := range this.web.RequestHeaderPolicy.AddHeaders {
if !header.IsOn {
continue
}
oldValues, _ := this.RawReq.Header[header.Name]
newHeaderValue := header.Value // 因为我们不能修改header所以在这里使用新变量
if header.HasVariables() {
newHeaderValue = this.Format(header.Value)
}
oldValues = append(oldValues, newHeaderValue)
reqHeader[header.Name] = oldValues
// 支持修改Host
if header.Name == "Host" && len(header.Value) > 0 {
this.RawReq.Host = newHeaderValue
}
}
// Set
for _, header := range this.web.RequestHeaderPolicy.SetHeaders {
if !header.IsOn {
continue
}
newHeaderValue := header.Value // 因为我们不能修改header所以在这里使用新变量
if header.HasVariables() {
newHeaderValue = this.Format(header.Value)
// 是否已删除
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
}
// 请求方法
if len(header.Methods) > 0 && !lists.ContainsString(header.Methods, this.RawReq.Method) {
continue
}
// 域名
if len(header.Domains) > 0 && !configutils.MatchDomains(header.Domains, this.Host) {
continue
}
var headerValue = header.Value
if header.ShouldReplace {
if len(headerValue) == 0 {
headerValue = reqHeader.Get(header.Name) // 原有值
} else if header.HasVariables() {
headerValue = this.Format(header.Value)
}
for _, v := range header.ReplaceValues {
headerValue = v.Replace(headerValue)
}
} else if header.HasVariables() {
headerValue = this.Format(header.Value)
}
reqHeader[header.Name] = []string{newHeaderValue}
// 支持修改Host
if header.Name == "Host" && len(header.Value) > 0 {
this.RawReq.Host = newHeaderValue
this.RawReq.Host = headerValue
} else {
if header.ShouldAppend {
reqHeader[header.Name] = append(reqHeader[header.Name], headerValue)
} else {
reqHeader[header.Name] = []string{headerValue}
}
}
}
// Replace
// TODO 需要实现
}
}
@@ -1143,7 +1213,6 @@ func (this *HTTPRequest) processResponseHeaders(statusCode int) {
// 删除/添加/替换Header
// TODO 实现AddTrailers
// TODO 实现ReplaceHeaders
if this.web.ResponseHeaderPolicy != nil && this.web.ResponseHeaderPolicy.IsOn {
// 删除某些Header
for name := range responseHeader {
@@ -1152,44 +1221,58 @@ func (this *HTTPRequest) processResponseHeaders(statusCode int) {
}
}
// Add
for _, header := range this.web.ResponseHeaderPolicy.AddHeaders {
if !header.IsOn {
continue
}
if header.Match(statusCode) {
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
}
oldValues, _ := responseHeader[header.Name]
if header.HasVariables() {
oldValues = append(oldValues, this.Format(header.Value))
} else {
oldValues = append(oldValues, header.Value)
}
responseHeader[header.Name] = oldValues
}
}
// Set
for _, header := range this.web.ResponseHeaderPolicy.SetHeaders {
if !header.IsOn {
continue
}
if header.Match(statusCode) {
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
// 是否已删除
if this.web.ResponseHeaderPolicy.ContainsDeletedHeader(header.Name) {
continue
}
// 状态码
if header.Status != nil && !header.Status.Match(statusCode) {
continue
}
// 请求方法
if len(header.Methods) > 0 && !lists.ContainsString(header.Methods, this.RawReq.Method) {
continue
}
// 域名
if len(header.Domains) > 0 && !configutils.MatchDomains(header.Domains, this.Host) {
continue
}
// 是否为跳转
if header.DisableRedirect && httpStatusIsRedirect(statusCode) {
continue
}
var headerValue = header.Value
if header.ShouldReplace {
if len(headerValue) == 0 {
headerValue = responseHeader.Get(header.Name) // 原有值
} else if header.HasVariables() {
headerValue = this.Format(header.Value)
}
if header.HasVariables() {
responseHeader[header.Name] = []string{this.Format(header.Value)}
} else {
responseHeader[header.Name] = []string{header.Value}
for _, v := range header.ReplaceValues {
headerValue = v.Replace(headerValue)
}
} else if header.HasVariables() {
headerValue = this.Format(header.Value)
}
if header.ShouldAppend {
responseHeader[header.Name] = append(responseHeader[header.Name], headerValue)
} else {
responseHeader[header.Name] = []string{headerValue}
}
}
// Replace
// TODO
}
// HSTS
@@ -1214,19 +1297,16 @@ func (this *HTTPRequest) addError(err error) {
// 计算合适的buffer size
func (this *HTTPRequest) bytePool(contentLength int64) *utils.BytePool {
if contentLength <= 0 {
return bytePool1k
}
if contentLength < 1024 { // 1K
return bytePool256b
if contentLength < 8192 { // 8K
return utils.BytePool1k
}
if contentLength < 32768 { // 32K
return bytePool1k
return utils.BytePool4k
}
if contentLength < 1048576 { // 1M
return bytePool32k
if contentLength < 131072 { // 128K
return utils.BytePool16k
}
return bytePool128k
return utils.BytePool32k
}
// 检查是否可以忽略错误
@@ -1263,3 +1343,34 @@ func (this *HTTPRequest) canIgnore(err error) bool {
return false
}
// 关闭当前连接
func (this *HTTPRequest) closeConn() {
requestConn := this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn == nil {
return
}
conn, ok := requestConn.(net.Conn)
if ok {
_ = conn.Close()
return
}
return
}
// 检查连接是否已关闭
func (this *HTTPRequest) isConnClosed() bool {
requestConn := this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn == nil {
return true
}
conn, ok := requestConn.(net.Conn)
if ok {
return isClientConnClosed(conn)
}
return true
}

View File

@@ -33,7 +33,7 @@ func (this *HTTPRequest) doAuth() (shouldStop bool) {
return writer.StatusCode(), nil
}, this.Format)
if err != nil {
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, false)
return
}
if b {

View File

@@ -5,8 +5,10 @@ import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"net/http"
"path/filepath"
"strconv"
@@ -15,7 +17,9 @@ import (
)
// 读取缓存
func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
func (this *HTTPRequest) doCacheRead(useStale bool) (shouldStop bool) {
this.cacheCanTryStale = false
cachePolicy := this.Server.HTTPCachePolicy
if cachePolicy == nil || !cachePolicy.IsOn {
return
@@ -80,6 +84,12 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
}
}
// 校验请求
if !this.cacheRef.MatchRequest(this.RawReq) {
this.cacheRef = nil
return
}
// 相关变量
this.varMapping["cache.policy.name"] = cachePolicy.Name
this.varMapping["cache.policy.id"] = strconv.FormatInt(cachePolicy.Id, 10)
@@ -103,6 +113,7 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
}
this.cacheKey = key
this.varMapping["cache.key"] = key
// 读取缓存
storage := caches.SharedManager.FindStorageWithPolicy(cachePolicy.Id)
@@ -113,12 +124,14 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
// 判断是否在Purge
if this.web.Cache.PurgeIsOn && strings.ToUpper(this.RawReq.Method) == "PURGE" && this.RawReq.Header.Get("X-Edge-Purge-Key") == this.web.Cache.PurgeKey {
this.varMapping["cache.status"] = "PURGE"
err := storage.Delete(key)
if err != nil {
remotelogs.Error("HTTP_REQUEST_CACHE", "purge failed: "+err.Error())
}
go func() {
goman.New(func() {
rpcClient, err := rpc.SharedRPC()
if err == nil {
for _, rpcServerService := range rpcClient.ServerRPCList() {
@@ -132,16 +145,11 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
}
}
}
}()
})
return true
}
buf := bytePool32k.Get()
defer func() {
bytePool32k.Put(buf)
}()
var reader caches.Reader
var err error
@@ -150,16 +158,20 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
this.web.WebP.IsOn &&
this.web.WebP.MatchRequest(filepath.Ext(this.requestPath()), this.Format) &&
this.web.WebP.MatchAccept(this.requestHeader("Accept")) {
reader, _ = storage.OpenReader(key + webpSuffix)
reader, _ = storage.OpenReader(key+webpSuffix, useStale)
}
// 检查正常的文件
if reader == nil {
reader, err = storage.OpenReader(key)
reader, err = storage.OpenReader(key, useStale)
if err != nil {
if err == caches.ErrNotFound {
// cache相关变量
this.varMapping["cache.status"] = "MISS"
if !useStale && this.web.Cache.Stale != nil && this.web.Cache.Stale.IsOn {
this.cacheCanTryStale = true
}
return
}
@@ -173,11 +185,23 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
_ = reader.Close()
}()
this.varMapping["cache.status"] = "HIT"
this.logAttrs["cache.status"] = "HIT"
if useStale {
this.varMapping["cache.status"] = "STALE"
this.logAttrs["cache.status"] = "STALE"
} else {
this.varMapping["cache.status"] = "HIT"
this.logAttrs["cache.status"] = "HIT"
}
// 准备Buffer
var pool = this.bytePool(reader.BodySize())
var buf = pool.Get()
defer func() {
pool.Put(buf)
}()
// 读取Header
headerBuf := []byte{}
var headerBuf = []byte{}
err = reader.ReadHeader(buf, func(n int) (goNext bool, err error) {
headerBuf = append(headerBuf, buf[:n]...)
for {
@@ -204,8 +228,19 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
return
}
// 设置cache.age变量
var age = strconv.FormatInt(reader.ExpiresAt()-utils.UnixTime(), 10)
this.varMapping["cache.age"] = age
if addStatusHeader {
this.writer.Header().Set("X-Cache", "HIT, "+refType+", "+reader.TypeName())
if useStale {
this.writer.Header().Set("X-Cache", "STALE, "+refType+", "+reader.TypeName())
} else {
this.writer.Header().Set("X-Cache", "HIT, "+refType+", "+reader.TypeName())
}
}
if this.web.Cache.AddAgeHeader {
this.writer.Header().Set("Age", age)
}
// ETag
@@ -223,7 +258,7 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
// 这里强制设置Last-Modified如果先前源站设置了Last-Modified将会被覆盖避免因为源站的Last-Modified导致源站返回304 Not Modified
var modifiedTime = ""
if lastModifiedAt > 0 {
modifiedTime = time.Unix(lastModifiedAt, 0).Format("Mon, 02 Jan 2006 15:04:05 GMT")
modifiedTime = time.Unix(utils.GMTUnixTime(lastModifiedAt), 0).Format("Mon, 02 Jan 2006 15:04:05") + " GMT"
respHeader.Set("Last-Modified", modifiedTime)
}
@@ -250,6 +285,7 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
}
this.processResponseHeaders(reader.Status())
this.addExpiresHeader(reader.ExpiresAt())
// 输出Body
if this.RawReq.Method == http.MethodHead {
@@ -327,6 +363,8 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
return true, nil
})
if err != nil {
this.varMapping["cache.status"] = "MISS"
if err == caches.ErrInvalidRange {
this.processResponseHeaders(http.StatusRequestedRangeNotSatisfiable)
this.writer.WriteHeader(http.StatusRequestedRangeNotSatisfiable)
@@ -387,6 +425,8 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
_, err = this.writer.WriteString("\r\n--" + boundary + "--\r\n")
if err != nil {
this.varMapping["cache.status"] = "MISS"
// 不提示写入客户端错误
return true
}
@@ -402,6 +442,8 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
return true, nil
})
if err != nil {
this.varMapping["cache.status"] = "MISS"
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_CACHE", "read from cache failed: "+err.Error())
}
@@ -417,3 +459,19 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
return true
}
// 设置Expires Header
func (this *HTTPRequest) addExpiresHeader(expiresAt int64) {
if this.cacheRef.ExpiresTime != nil && this.cacheRef.ExpiresTime.IsPrior && this.cacheRef.ExpiresTime.IsOn {
if this.cacheRef.ExpiresTime.Overwrite || len(this.writer.Header().Get("Expires")) == 0 {
if this.cacheRef.ExpiresTime.AutoCalculate {
this.writer.Header().Set("Expires", time.Unix(utils.GMTUnixTime(expiresAt), 0).Format("Mon, 2 Jan 2006 15:04:05")+" GMT")
} else if this.cacheRef.ExpiresTime.Duration != nil {
var duration = this.cacheRef.ExpiresTime.Duration.Duration()
if duration > 0 {
this.writer.Header().Set("Expires", utils.GMTTime(time.Now().Add(duration)).Format("Mon, 2 Jan 2006 15:04:05")+" GMT")
}
}
}
}
}

View File

@@ -1,6 +1,7 @@
package nodes
import (
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
"net/http"
)
@@ -11,22 +12,42 @@ func (this *HTTPRequest) write404() {
}
this.processResponseHeaders(http.StatusNotFound)
msg := "404 page not found: '" + this.RawURI() + "'"
this.writer.WriteHeader(http.StatusNotFound)
_, _ = this.writer.Write([]byte(msg))
_, _ = this.writer.Write([]byte("404 page not found: '" + this.requestFullURL() + "'" + " (Request Id: " + this.requestId + ")"))
}
func (this *HTTPRequest) write50x(err error, statusCode int) {
func (this *HTTPRequest) writeCode(code int) {
if this.doPage(code) {
return
}
this.processResponseHeaders(code)
this.writer.WriteHeader(code)
_, _ = this.writer.Write([]byte(types.String(code) + " " + http.StatusText(code) + ": '" + this.requestFullURL() + "'" + " (Request Id: " + this.requestId + ")"))
}
func (this *HTTPRequest) write50x(err error, statusCode int, canTryStale bool) {
if err != nil {
this.addError(err)
}
// 尝试从缓存中恢复
if canTryStale &&
this.cacheCanTryStale &&
this.web.Cache.Stale != nil &&
this.web.Cache.Stale.IsOn &&
(len(this.web.Cache.Stale.Status) == 0 || lists.ContainsInt(this.web.Cache.Stale.Status, statusCode)) {
ok := this.doCacheRead(true)
if ok {
return
}
}
// 显示自定义页面
if this.doPage(statusCode) {
return
}
this.processResponseHeaders(statusCode)
this.writer.WriteHeader(statusCode)
_, _ = this.writer.Write([]byte(types.String(statusCode) + " " + http.StatusText(statusCode)))
_, _ = this.writer.Write([]byte(types.String(statusCode) + " " + http.StatusText(statusCode) + " (Request Id: " + this.requestId + ")"))
}

View File

@@ -81,7 +81,7 @@ func (this *HTTPRequest) doFastcgi() (shouldStop bool) {
client, err := fcgi.SharedPool(fastcgi.Network(), fastcgi.RealAddress(), uint(poolSize)).Client()
if err != nil {
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, false)
return
}
@@ -159,13 +159,13 @@ func (this *HTTPRequest) doFastcgi() (shouldStop bool) {
resp, stderr, err := client.Call(fcgiReq)
if err != nil {
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, false)
return
}
if len(stderr) > 0 {
err := errors.New("Fastcgi Error: " + strings.TrimSpace(string(stderr)) + " script: " + maps.NewMap(params).GetString("SCRIPT_FILENAME"))
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, false)
return
}

View File

@@ -14,7 +14,7 @@ func (this *HTTPRequest) doHealthCheck(key string) (stop bool) {
this.RawReq.Header.Del(serverconfigs.HealthCheckHeaderName)
data, err := nodeutils.DecryptData(sharedNodeConfig.NodeId, sharedNodeConfig.Secret, key)
data, err := nodeutils.Base64DecodeMap(key)
if err != nil {
remotelogs.Error("HTTP_REQUEST_HEALTH_CHECK", "decode key failed: "+err.Error())
return

View File

@@ -1,6 +1,7 @@
package nodes
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"net/http"
"strconv"
"strings"
@@ -8,7 +9,11 @@ import (
// 主机地址快速跳转
func (this *HTTPRequest) doHostRedirect() (blocked bool) {
fullURL := this.requestScheme() + "://" + this.Host + this.RawReq.URL.Path
var urlPath = this.RawReq.URL.Path
if this.web.MergeSlashes {
urlPath = utils.CleanPath(urlPath)
}
fullURL := this.requestScheme() + "://" + this.Host + urlPath
for _, u := range this.web.HostRedirects {
if !u.IsOn {
continue
@@ -22,9 +27,17 @@ func (this *HTTPRequest) doHostRedirect() (blocked bool) {
if u.KeepRequestURI {
afterURL += this.RawReq.URL.RequestURI()
}
// 前后是否一致
if fullURL == afterURL {
return false
}
if u.Status <= 0 {
this.processResponseHeaders(http.StatusTemporaryRedirect)
http.Redirect(this.RawWriter, this.RawReq, afterURL, http.StatusTemporaryRedirect)
} else {
this.processResponseHeaders(u.Status)
http.Redirect(this.RawWriter, this.RawReq, afterURL, u.Status)
}
return true
@@ -55,17 +68,31 @@ func (this *HTTPRequest) doHostRedirect() (blocked bool) {
}
}
// 前后是否一致
if fullURL == afterURL {
return false
}
if u.Status <= 0 {
this.processResponseHeaders(http.StatusTemporaryRedirect)
http.Redirect(this.RawWriter, this.RawReq, afterURL, http.StatusTemporaryRedirect)
} else {
this.processResponseHeaders(u.Status)
http.Redirect(this.RawWriter, this.RawReq, afterURL, u.Status)
}
return true
} else { // 精准匹配
if fullURL == u.RealBeforeURL() {
// 前后是否一致
if fullURL == u.AfterURL {
return false
}
if u.Status <= 0 {
this.processResponseHeaders(http.StatusTemporaryRedirect)
http.Redirect(this.RawWriter, this.RawReq, u.AfterURL, http.StatusTemporaryRedirect)
} else {
this.processResponseHeaders(u.Status)
http.Redirect(this.RawWriter, this.RawReq, u.AfterURL, u.Status)
}
return true

View File

@@ -0,0 +1,32 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import "net/http"
func (this *HTTPRequest) doRequestLimit() (shouldStop bool) {
// 检查请求Body尺寸
// TODO 处理分片提交的内容
if this.web.RequestLimit.MaxBodyBytes() > 0 &&
this.RawReq.ContentLength > this.web.RequestLimit.MaxBodyBytes() {
this.writeCode(http.StatusRequestEntityTooLarge)
return true
}
// 设置连接相关参数
if this.web.RequestLimit.MaxConns > 0 || this.web.RequestLimit.MaxConnsPerIP > 0 {
requestConn := this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn != nil {
clientConn, ok := requestConn.(ClientConnInterface)
if ok && !clientConn.IsBound() {
if !clientConn.Bind(this.Server.Id, this.requestRemoteAddr(true), this.web.RequestLimit.MaxConns, this.web.RequestLimit.MaxConnsPerIP) {
this.writeCode(http.StatusTooManyRequests)
this.closeConn()
return true
}
}
}
}
return false
}

View File

@@ -3,13 +3,14 @@ package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"strconv"
"strings"
"sync/atomic"
"time"
)
var requestId int64 = 1_0000_0000_0000_0000
const (
// AccessLogMaxRequestBodySize 访问日志存储的请求内容最大尺寸 TODO 此值应该可以在访问日志页设置
AccessLogMaxRequestBodySize = 2 * 1024 * 1024
)
// 日志
func (this *HTTPRequest) log() {
@@ -36,6 +37,11 @@ func (this *HTTPRequest) log() {
return
}
// 是否记录499
if !ref.EnableClientClosed && this.writer.StatusCode() == 499 {
return
}
addr := this.RawReq.RemoteAddr
index := strings.LastIndex(addr, ":")
if index > 0 {
@@ -85,7 +91,7 @@ func (this *HTTPRequest) log() {
}
accessLog := &pb.HTTPAccessLog{
RequestId: strconv.FormatInt(this.requestFromTime.UnixNano(), 10) + strconv.FormatInt(atomic.AddInt64(&requestId, 1), 10) + sharedNodeConfig.PaddedId(),
RequestId: this.requestId,
NodeId: sharedNodeConfig.Id,
ServerId: this.Server.Id,
RemoteAddr: this.requestRemoteAddr(true),
@@ -139,6 +145,15 @@ func (this *HTTPRequest) log() {
accessLog.OriginAddress = this.originAddr
}
// 请求Body
if ref.ContainsField(serverconfigs.HTTPAccessLogFieldRequestBody) {
accessLog.RequestBody = this.requestBodyData
if len(accessLog.RequestBody) > AccessLogMaxRequestBodySize {
accessLog.RequestBody = accessLog.RequestBody[:AccessLogMaxRequestBodySize]
}
}
// TODO 记录匹配的 locationId和rewriteId
sharedHTTPAccessLogQueue.Push(accessLog)

View File

@@ -23,7 +23,11 @@ func (this *HTTPRequest) doPage(status int) (shouldStop bool) {
if page.Match(status) {
if len(page.BodyType) == 0 || page.BodyType == shared.BodyTypeURL {
if urlPrefixRegexp.MatchString(page.URL) {
this.doURL(http.MethodGet, page.URL, "", page.NewStatus, true)
var newStatus = page.NewStatus
if newStatus <= 0 {
newStatus = status
}
this.doURL(http.MethodGet, page.URL, "", newStatus, true)
return true
} else {
file := Tea.Root + Tea.DS + page.URL
@@ -64,11 +68,11 @@ func (this *HTTPRequest) doPage(status int) (shouldStop bool) {
this.writer.Prepare(stat.Size(), status)
this.writer.WriteHeader(status)
}
buf := bytePool1k.Get()
buf := utils.BytePool1k.Get()
_, err = utils.CopyWithFilter(this.writer, fp, buf, func(p []byte) []byte {
return []byte(this.Format(string(p)))
})
bytePool1k.Put(buf)
utils.BytePool1k.Put(buf)
if err != nil {
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_PAGE", "write to client failed: "+err.Error())
@@ -84,6 +88,12 @@ func (this *HTTPRequest) doPage(status int) (shouldStop bool) {
return true
} else if page.BodyType == shared.BodyTypeHTML {
// 这里需要实现设置Status因为在Format()中可以获取${status}等变量
if page.NewStatus > 0 {
this.writer.statusCode = page.NewStatus
} else {
this.writer.statusCode = status
}
var content = this.Format(page.Body)
// 修改状态码

View File

@@ -0,0 +1,19 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"net/http"
)
// 套餐过期
func (this *HTTPRequest) doPlanExpires() {
this.tags = append(this.tags, "plan")
var statusCode = http.StatusNotFound
this.processResponseHeaders(statusCode)
this.writer.WriteHeader(statusCode)
_, _ = this.writer.WriteString(this.Format(serverconfigs.DefaultPlanExpireNoticePageBody))
}

View File

@@ -42,6 +42,7 @@ func (this *HTTPRequest) doRedirectToHTTPS(redirectToHTTPSConfig *serverconfigs.
}
newURL := "https://" + host + this.RawReq.RequestURI
this.processResponseHeaders(statusCode)
http.Redirect(this.writer, this.RawReq, newURL, statusCode)
return true

View File

@@ -39,9 +39,9 @@ func (this *HTTPRequest) doReverseProxy() {
origin := this.reverseProxy.NextOrigin(requestCall)
requestCall.CallResponseCallbacks(this.writer)
if origin == nil {
err := errors.New(this.requestPath() + ": no available backends for reverse proxy")
remotelogs.Error("HTTP_REQUEST_REVERSE_PROXY", err.Error())
this.write50x(err, http.StatusBadGateway)
err := errors.New(this.requestFullURL() + ": no available origin sites for reverse proxy")
remotelogs.ServerError(this.Server.Id, "HTTP_REQUEST_REVERSE_PROXY", err.Error(), "", nil)
this.write50x(err, http.StatusBadGateway, true)
return
}
this.origin = origin // 设置全局变量是为了日志等处理
@@ -59,9 +59,9 @@ func (this *HTTPRequest) doReverseProxy() {
// 处理Scheme
if origin.Addr == nil {
err := errors.New(this.requestPath() + ": origin '" + strconv.FormatInt(origin.Id, 10) + "' does not has a address")
err := errors.New(this.requestFullURL() + ": origin '" + strconv.FormatInt(origin.Id, 10) + "' does not has a address")
remotelogs.Error("HTTP_REQUEST_REVERSE_PROXY", err.Error())
this.write50x(err, http.StatusBadGateway)
this.write50x(err, http.StatusBadGateway, true)
return
}
this.RawReq.URL.Scheme = origin.Addr.Protocol.Primary().Scheme()
@@ -117,7 +117,15 @@ func (this *HTTPRequest) doReverseProxy() {
}
this.RawReq.URL.Host = this.RawReq.Host
} else if this.reverseProxy.RequestHostType == serverconfigs.RequestHostTypeOrigin {
this.RawReq.Host = originAddr
// 源站主机名
var hostname = originAddr
if origin.Addr.Protocol.IsHTTPFamily() {
hostname = strings.TrimSuffix(hostname, ":80")
} else if origin.Addr.Protocol.IsHTTPSFamily() {
hostname = strings.TrimSuffix(hostname, ":443")
}
this.RawReq.Host = hostname
this.RawReq.URL.Host = this.RawReq.Host
} else {
this.RawReq.URL.Host = this.Host
@@ -148,7 +156,7 @@ func (this *HTTPRequest) doReverseProxy() {
client, err := SharedHTTPClientPool.Client(this, origin, originAddr, this.reverseProxy.ProxyProtocol)
if err != nil {
remotelogs.Error("HTTP_REQUEST_REVERSE_PROXY", err.Error())
this.write50x(err, http.StatusBadGateway)
this.write50x(err, http.StatusBadGateway, true)
return
}
@@ -167,18 +175,18 @@ func (this *HTTPRequest) doReverseProxy() {
SharedOriginStateManager.Fail(origin, this.reverseProxy, func() {
this.reverseProxy.ResetScheduling()
})
this.write50x(err, http.StatusBadGateway)
this.write50x(err, http.StatusBadGateway, true)
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.RawReq.URL.String()+"': "+err.Error())
} else if httpErr.Err != context.Canceled {
SharedOriginStateManager.Fail(origin, this.reverseProxy, func() {
this.reverseProxy.ResetScheduling()
})
if httpErr.Timeout() {
this.write50x(err, http.StatusGatewayTimeout)
this.write50x(err, http.StatusGatewayTimeout, true)
} else if httpErr.Temporary() {
this.write50x(err, http.StatusServiceUnavailable)
this.write50x(err, http.StatusServiceUnavailable, true)
} else {
this.write50x(err, http.StatusBadGateway)
this.write50x(err, http.StatusBadGateway, true)
}
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.RawReq.URL.String()+"': "+err.Error())
} else {
@@ -186,6 +194,12 @@ func (this *HTTPRequest) doReverseProxy() {
isClientError := false
if ok {
if httpErr.Err == context.Canceled {
// 如果是服务器端主动关闭,则无需提示
if this.isConnClosed() {
this.disableLog = true
return
}
isClientError = true
this.addError(errors.New(httpErr.Op + " " + httpErr.URL + ": client closed the connection"))
this.writer.WriteHeader(499) // 仿照nginx
@@ -193,7 +207,7 @@ func (this *HTTPRequest) doReverseProxy() {
}
if !isClientError {
this.write50x(err, http.StatusBadGateway)
this.write50x(err, http.StatusBadGateway, true)
}
}
if resp != nil && resp.Body != nil {
@@ -218,8 +232,6 @@ func (this *HTTPRequest) doReverseProxy() {
}
}
// TODO 清除源站错误次数
// 特殊页面
if len(this.web.Pages) > 0 && this.doPage(resp.StatusCode) {
err = resp.Body.Close()
@@ -281,7 +293,7 @@ func (this *HTTPRequest) doReverseProxy() {
closeErr := resp.Body.Close()
if closeErr != nil {
if !this.canIgnore(err) {
if !this.canIgnore(closeErr) {
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", closeErr.Error())
}
}
@@ -294,7 +306,7 @@ func (this *HTTPRequest) doReverseProxy() {
}
// 是否成功结束
if err == nil && closeErr == nil {
if (err == nil || err == io.EOF) && (closeErr == nil || closeErr == io.EOF) {
this.writer.SetOk()
}
}

View File

@@ -30,8 +30,10 @@ func (this *HTTPRequest) doRewrite() (shouldShop bool) {
// 跳转
if this.rewriteRule.Mode == serverconfigs.HTTPRewriteModeRedirect {
if this.rewriteRule.RedirectStatus > 0 {
this.processResponseHeaders(this.rewriteRule.RedirectStatus)
http.Redirect(this.writer, this.RawReq, this.rewriteReplace, this.rewriteRule.RedirectStatus)
} else {
this.processResponseHeaders(http.StatusTemporaryRedirect)
http.Redirect(this.writer, this.RawReq, this.rewriteReplace, http.StatusTemporaryRedirect)
}
return true

View File

@@ -2,6 +2,7 @@ package nodes
import (
"fmt"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/cespare/xxhash"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/logs"
@@ -17,23 +18,23 @@ import (
)
// 文本mime-type列表
var textMimeMap = map[string]bool{
"application/atom+xml": true,
"application/javascript": true,
"application/x-javascript": true,
"application/json": true,
"application/rss+xml": true,
"application/x-web-app-manifest+json": true,
"application/xhtml+xml": true,
"application/xml": true,
"image/svg+xml": true,
"text/css": true,
"text/plain": true,
"text/javascript": true,
"text/xml": true,
"text/html": true,
"text/xhtml": true,
"text/sgml": true,
var textMimeMap = map[string]zero.Zero{
"application/atom+xml": {},
"application/javascript": {},
"application/x-javascript": {},
"application/json": {},
"application/rss+xml": {},
"application/x-web-app-manifest+json": {},
"application/xhtml+xml": {},
"application/xml": {},
"image/svg+xml": {},
"text/css": {},
"text/plain": {},
"text/javascript": {},
"text/xml": {},
"text/html": {},
"text/xhtml": {},
"text/sgml": {},
}
// 调用本地静态资源
@@ -109,7 +110,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
}
return
} else {
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, true)
logs.Error(err)
return true
}
@@ -138,7 +139,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
}
return
} else {
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, true)
logs.Error(err)
return true
}
@@ -283,7 +284,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
reader, err := os.OpenFile(filePath, os.O_RDONLY, 0444)
if err != nil {
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, true)
return true
}

View File

@@ -64,11 +64,11 @@ func (this *HTTPRequest) doShutdown() {
this.processResponseHeaders(http.StatusOK)
this.writer.WriteHeader(http.StatusOK)
}
buf := bytePool1k.Get()
buf := utils.BytePool1k.Get()
_, err = utils.CopyWithFilter(this.writer, fp, buf, func(p []byte) []byte {
return []byte(this.Format(string(p)))
})
bytePool1k.Put(buf)
utils.BytePool1k.Put(buf)
if err != nil {
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_SHUTDOWN", "write to client failed: "+err.Error())

View File

@@ -1,6 +1,8 @@
package nodes
import "github.com/TeaOSLab/EdgeNode/internal/stats"
import (
"github.com/TeaOSLab/EdgeNode/internal/stats"
)
// 统计
func (this *HTTPRequest) doStat() {
@@ -9,6 +11,6 @@ func (this *HTTPRequest) doStat() {
}
// 内置的统计
stats.SharedHTTPRequestStatManager.AddRemoteAddr(this.Server.Id, this.requestRemoteAddr(true))
stats.SharedHTTPRequestStatManager.AddRemoteAddr(this.Server.Id, this.requestRemoteAddr(true), this.writer.SentBodyBytes(), this.isAttack)
stats.SharedHTTPRequestStatManager.AddUserAgent(this.Server.Id, this.requestHeader("User-Agent"))
}

View File

@@ -3,6 +3,7 @@ package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/iwind/TeaGo/assert"
"runtime"
"testing"
)
@@ -33,3 +34,17 @@ func TestHTTPRequest_RedirectToHTTPS(t *testing.T) {
a.IsBool(req.web.RedirectToHttps.IsOn == true)
}
}
func TestHTTPRequest_Memory(t *testing.T) {
var stat1 = &runtime.MemStats{}
runtime.ReadMemStats(stat1)
var requests = []*HTTPRequest{}
for i := 0; i < 1_000_000; i++ {
requests = append(requests, &HTTPRequest{})
}
var stat2 = &runtime.MemStats{}
runtime.ReadMemStats(stat2)
t.Log((stat2.HeapInuse-stat1.HeapInuse)/1024/1024, "MB,")
}

View File

@@ -6,9 +6,9 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
)
// 带宽限制
func (this *HTTPRequest) doBandwidthLimit() {
var config = this.Server.BandwidthLimit
// 流量限制
func (this *HTTPRequest) doTrafficLimit() {
var config = this.Server.TrafficLimit
this.tags = append(this.tags, "bandwidth")
@@ -17,8 +17,8 @@ func (this *HTTPRequest) doBandwidthLimit() {
this.writer.WriteHeader(statusCode)
if len(config.NoticePageBody) != 0 {
_, _ = this.writer.WriteString(config.NoticePageBody)
_, _ = this.writer.WriteString(this.Format(config.NoticePageBody))
} else {
_, _ = this.writer.WriteString(serverconfigs.DefaultBandwidthLimitNoticePageBody)
_, _ = this.writer.WriteString(this.Format(serverconfigs.DefaultTrafficLimitNoticePageBody))
}
}

View File

@@ -35,7 +35,7 @@ func (this *HTTPRequest) doURL(method string, url string, host string, statusCod
resp, err := client.Do(req)
if err != nil {
remotelogs.Error("HTTP_REQUEST_URL", req.URL.String()+": "+err.Error())
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, false)
return
}
defer func() {

View File

@@ -3,9 +3,13 @@ package nodes
import (
"crypto/rand"
"fmt"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
"net/http"
"strconv"
"strings"
"sync/atomic"
)
// 分解Range
@@ -125,3 +129,27 @@ func httpRequestGenBoundary() string {
}
return fmt.Sprintf("%x", buf[:])
}
// 判断状态是否为跳转
func httpStatusIsRedirect(statusCode int) bool {
return statusCode == http.StatusPermanentRedirect ||
statusCode == http.StatusTemporaryRedirect ||
statusCode == http.StatusMovedPermanently ||
statusCode == http.StatusSeeOther ||
statusCode == http.StatusFound
}
// 生成请求ID
var httpRequestTimestamp int64
var httpRequestId int32 = 1_000_000
func httpRequestNextId() string {
var unixTime = utils.UnixTimeMilli()
if unixTime > httpRequestTimestamp {
atomic.StoreInt32(&httpRequestId, 1_000_000)
httpRequestTimestamp = unixTime
}
// timestamp + requestId + nodeId
return strconv.FormatInt(unixTime, 10) + teaconst.NodeIdString + strconv.Itoa(int(atomic.AddInt32(&httpRequestId, 1)))
}

View File

@@ -1,8 +1,13 @@
package nodes
import (
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/zero"
"github.com/iwind/TeaGo/assert"
"runtime"
"sync"
"testing"
"time"
)
func TestHTTPRequest_httpRequestParseContentRange(t *testing.T) {
@@ -53,3 +58,60 @@ func TestHTTPRequest_httpRequestParseContentRange(t *testing.T) {
t.Log(set)
}
}
func TestHTTPRequest_httpRequestNextId(t *testing.T) {
teaconst.NodeId = 123
teaconst.NodeIdString = "123"
t.Log(httpRequestNextId())
t.Log(httpRequestNextId())
t.Log(httpRequestNextId())
time.Sleep(1 * time.Second)
t.Log(httpRequestNextId())
t.Log(httpRequestNextId())
time.Sleep(1 * time.Second)
t.Log(httpRequestNextId())
}
func TestHTTPRequest_httpRequestNextId_Concurrent(t *testing.T) {
var m = map[string]zero.Zero{}
var locker = sync.Mutex{}
var count = 4000
var wg = &sync.WaitGroup{}
wg.Add(count)
var countDuplicated = 0
for i := 0; i < count; i++ {
go func() {
defer wg.Done()
var requestId = httpRequestNextId()
locker.Lock()
_, ok := m[requestId]
if ok {
t.Log("duplicated:", requestId)
countDuplicated++
}
m[requestId] = zero.New()
locker.Unlock()
}()
}
wg.Wait()
t.Log("ok", countDuplicated, "duplicated")
var a = assert.NewAssertion(t)
a.IsTrue(countDuplicated == 0)
}
func BenchmarkHTTPRequest_httpRequestNextId(b *testing.B) {
runtime.GOMAXPROCS(1)
teaconst.NodeIdString = "123"
for i := 0; i < b.N; i++ {
_ = httpRequestNextId()
}
}

View File

@@ -7,32 +7,40 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/stats"
"github.com/TeaOSLab/EdgeNode/internal/waf"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
"io"
"io/ioutil"
"net"
"net/http"
)
// 调用WAF
func (this *HTTPRequest) doWAFRequest() (blocked bool) {
var remoteAddr = this.requestRemoteAddr(true)
// 检查是否为白名单直连
if !Tea.IsTesting() && sharedNodeConfig.IPIsAutoAllowed(remoteAddr) {
return
}
// 当前连接是否已关闭
var conn = this.RawReq.Context().Value(HTTPConnContextKey)
if conn != nil {
if isClientConnClosed(conn.(net.Conn)) {
this.disableLog = true
return true
}
if this.isConnClosed() {
this.disableLog = true
return true
}
// 是否在全局名单中
if !iplibrary.AllowIP(remoteAddr, this.Server.Id) {
this.disableLog = true
this.closeConn()
return true
}
// 检查是否在临时黑名单中
var remoteAddr = this.WAFRemoteIP()
if waf.SharedIPBlackList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeService, this.Server.Id, remoteAddr) || waf.SharedIPBlackList.Contains(waf.IPTypeAll, firewallconfigs.FirewallScopeGlobal, 0, remoteAddr) {
this.disableLog = true
if conn != nil {
_ = conn.(net.Conn).Close()
}
this.closeConn()
return true
}
@@ -181,17 +189,11 @@ func (this *HTTPRequest) checkWAFRequest(firewallPolicy *firewallconfigs.HTTPFir
return
}
w.OnAction(func(action waf.ActionInterface) (goNext bool) {
switch action.Code() {
case waf.ActionTag:
this.tags = action.(*waf.TagAction).Tags
}
return true
})
goNext, ruleGroup, ruleSet, err := w.MatchRequest(this, this.writer)
if err != nil {
remotelogs.Error("HTTP_REQUEST_WAF", this.rawURI+": "+err.Error())
if !this.canIgnore(err) {
remotelogs.Error("HTTP_REQUEST_WAF", this.rawURI+": "+err.Error())
}
return
}
@@ -245,17 +247,11 @@ func (this *HTTPRequest) checkWAFResponse(firewallPolicy *firewallconfigs.HTTPFi
return
}
w.OnAction(func(action waf.ActionInterface) (goNext bool) {
switch action.Code() {
case waf.ActionTag:
this.tags = action.(*waf.TagAction).Tags
}
return true
})
goNext, ruleGroup, ruleSet, err := w.MatchResponse(this, resp, this.writer)
if err != nil {
remotelogs.Error("HTTP_REQUEST_WAF", this.rawURI+": "+err.Error())
if !this.canIgnore(err) {
remotelogs.Error("HTTP_REQUEST_WAF", this.rawURI+": "+err.Error())
}
return
}
@@ -291,12 +287,12 @@ func (this *HTTPRequest) WAFRemoteIP() string {
// WAFGetCacheBody 获取缓存中的Body
func (this *HTTPRequest) WAFGetCacheBody() []byte {
return this.bodyData
return this.requestBodyData
}
// WAFSetCacheBody 设置Body
func (this *HTTPRequest) WAFSetCacheBody(body []byte) {
this.bodyData = body
this.requestBodyData = body
}
// WAFReadBody 读取Body
@@ -304,16 +300,14 @@ func (this *HTTPRequest) WAFReadBody(max int64) (data []byte, err error) {
if this.RawReq.ContentLength > 0 {
data, err = ioutil.ReadAll(io.LimitReader(this.RawReq.Body, max))
}
return
}
// WAFRestoreBody 恢复Body
func (this *HTTPRequest) WAFRestoreBody(data []byte) {
if len(data) > 0 {
rawReader := bytes.NewBuffer(data)
buf := make([]byte, 1024)
_, _ = io.CopyBuffer(rawReader, this.RawReq.Body, buf)
this.RawReq.Body = ioutil.NopCloser(rawReader)
this.RawReq.Body = ioutil.NopCloser(io.MultiReader(bytes.NewBuffer(data), this.RawReq.Body))
}
}
@@ -324,14 +318,22 @@ func (this *HTTPRequest) WAFServerId() int64 {
// WAFClose 关闭连接
func (this *HTTPRequest) WAFClose() {
requestConn := this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn == nil {
return
}
conn, ok := requestConn.(net.Conn)
if ok {
_ = conn.Close()
return
}
return
this.closeConn()
}
func (this *HTTPRequest) WAFOnAction(action interface{}) (goNext bool) {
if action == nil {
return true
}
instance, ok := action.(waf.ActionInterface)
if !ok {
return true
}
switch instance.Code() {
case waf.ActionTag:
this.tags = append(this.tags, action.(*waf.TagAction).Tags...)
}
return true
}

View File

@@ -2,6 +2,8 @@ package nodes
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
"net/http"
"net/url"
@@ -42,7 +44,7 @@ func (this *HTTPRequest) doWebsocket() {
// TODO 增加N次错误重试重试的时候需要尝试不同的源站
originConn, err := OriginConnect(this.origin, this.RawReq.RemoteAddr)
if err != nil {
this.write50x(err, http.StatusBadGateway)
this.write50x(err, http.StatusBadGateway, false)
return
}
defer func() {
@@ -51,21 +53,22 @@ func (this *HTTPRequest) doWebsocket() {
err = this.RawReq.Write(originConn)
if err != nil {
this.write50x(err, http.StatusBadGateway)
this.write50x(err, http.StatusBadGateway, false)
return
}
clientConn, _, err := this.writer.Hijack()
if err != nil || clientConn == nil {
this.write50x(err, http.StatusInternalServerError)
this.write50x(err, http.StatusInternalServerError, false)
return
}
defer func() {
_ = clientConn.Close()
}()
go func() {
buf := make([]byte, 4*1024) // TODO 使用内存池
goman.New(func() {
var buf = utils.BytePool4k.Get()
defer utils.BytePool4k.Put(buf)
for {
n, err := originConn.Read(buf)
if n > 0 {
@@ -81,6 +84,6 @@ func (this *HTTPRequest) doWebsocket() {
}
_ = clientConn.Close()
_ = originConn.Close()
}()
})
_, _ = io.Copy(originConn, clientConn)
}

View File

@@ -1,3 +1,5 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
@@ -76,22 +78,6 @@ func NewHTTPWriter(req *HTTPRequest, httpResponseWriter http.ResponseWriter) *HT
}
}
// Reset 重置
func (this *HTTPWriter) Reset(httpResponseWriter http.ResponseWriter) {
this.writer = httpResponseWriter
this.compressionConfig = nil
this.compressionWriter = nil
this.statusCode = 0
this.sentBodyBytes = 0
this.bodyCopying = false
this.body = nil
this.compressionBodyBuffer = nil
this.compressionBodyWriter = nil
}
// SetCompression 设置内容压缩配置
func (this *HTTPWriter) SetCompression(config *serverconfigs.HTTPCompressionConfig) {
this.compressionConfig = config
@@ -118,6 +104,14 @@ func (this *HTTPWriter) Prepare(size int64, status int) (delayHeaders bool) {
this.PrepareCompression(size)
}
// 是否限速写入
if this.req.web != nil &&
this.req.web.RequestLimit != nil &&
this.req.web.RequestLimit.IsOn &&
this.req.web.RequestLimit.OutBandwidthPerConnBytes() > 0 {
this.writer = NewHTTPRateWriter(this.writer, this.req.web.RequestLimit.OutBandwidthPerConnBytes())
}
return
}
@@ -421,10 +415,12 @@ func (this *HTTPWriter) Close() {
if this.isOk {
err := this.cacheWriter.Close()
if err == nil {
var expiredAt = this.cacheWriter.ExpiredAt()
this.cacheStorage.AddToList(&caches.Item{
Type: this.cacheWriter.ItemType(),
Key: this.cacheWriter.Key(),
ExpiredAt: this.cacheWriter.ExpiredAt(),
ExpiredAt: expiredAt,
StaleAt: expiredAt + int64(this.calculateStaleLife()),
HeaderSize: this.cacheWriter.HeaderSize(),
BodySize: this.cacheWriter.BodySize(),
Host: this.req.Host,
@@ -552,11 +548,6 @@ func (this *HTTPWriter) prepareCache(size int64) {
return
}
// 不支持Range
if len(this.Header().Get("Content-Range")) > 0 {
return
}
cachePolicy := this.req.Server.HTTPCachePolicy
if cachePolicy == nil || !cachePolicy.IsOn {
return
@@ -567,17 +558,40 @@ func (this *HTTPWriter) prepareCache(size int64) {
return
}
var addStatusHeader = this.req.web != nil && this.req.web.Cache != nil && this.req.web.Cache.AddStatusHeader
// 不支持Range
if len(this.Header().Get("Content-Range")) > 0 {
this.req.varMapping["cache.status"] = "BYPASS"
if addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, not supported Content-Range")
}
return
}
// 如果允许 ChunkedEncoding就无需尺寸的判断因为此时的 size 为 -1
if !cacheRef.AllowChunkedEncoding && size < 0 {
this.req.varMapping["cache.status"] = "BYPASS"
if addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, ChunkedEncoding")
}
return
}
if size >= 0 && ((cacheRef.MaxSizeBytes() > 0 && size > cacheRef.MaxSizeBytes()) ||
(cachePolicy.MaxSizeBytes() > 0 && size > cachePolicy.MaxSizeBytes()) || (cacheRef.MinSizeBytes() > size)) {
this.req.varMapping["cache.status"] = "BYPASS"
if addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, Content-Length")
}
return
}
// 检查状态
if len(cacheRef.Status) > 0 && !lists.ContainsInt(cacheRef.Status, this.StatusCode()) {
this.req.varMapping["cache.status"] = "BYPASS"
if addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, Status: "+types.String(this.StatusCode()))
}
return
}
@@ -588,6 +602,10 @@ func (this *HTTPWriter) prepareCache(size int64) {
values := strings.Split(cacheControl, ",")
for _, value := range values {
if cacheRef.ContainsCacheControl(strings.TrimSpace(value)) {
this.req.varMapping["cache.status"] = "BYPASS"
if addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, Cache-Control: "+cacheControl)
}
return
}
}
@@ -596,24 +614,59 @@ func (this *HTTPWriter) prepareCache(size int64) {
// Set-Cookie
if cacheRef.SkipResponseSetCookie && len(this.writer.Header().Get("Set-Cookie")) > 0 {
this.req.varMapping["cache.status"] = "BYPASS"
if addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, Set-Cookie")
}
return
}
// 校验其他条件
if cacheRef.Conds != nil && cacheRef.Conds.HasResponseConds() && !cacheRef.Conds.MatchResponse(this.req.Format) {
this.req.varMapping["cache.status"] = "BYPASS"
if addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, ResponseConds")
}
return
}
// 打开缓存写入
storage := caches.SharedManager.FindStorageWithPolicy(cachePolicy.Id)
if storage == nil {
this.req.varMapping["cache.status"] = "BYPASS"
if addStatusHeader {
this.Header().Set("X-Cache", "BYPASS, Storage")
}
return
}
this.req.varMapping["cache.status"] = "UPDATING"
if addStatusHeader {
this.Header().Set("X-Cache", "UPDATING")
}
this.cacheStorage = storage
life := cacheRef.LifeSeconds()
if life <= 60 { // 最小不能少于1分钟
if life <= 0 {
life = 60
}
// 支持源站设置的max-age
if this.req.web.Cache != nil && this.req.web.Cache.EnableCacheControlMaxAge {
var cacheControl = this.Header().Get("Cache-Control")
var pieces = strings.Split(cacheControl, ";")
for _, piece := range pieces {
var eqIndex = strings.Index(piece, "=")
if eqIndex > 0 && piece[:eqIndex] == "max-age" {
var maxAge = types.Int64(piece[eqIndex+1:])
if maxAge > 0 {
life = maxAge
}
}
}
}
expiredAt := utils.UnixTime() + life
var cacheKey = this.req.cacheKey
if this.webpIsEncoding {
@@ -641,3 +694,32 @@ func (this *HTTPWriter) prepareCache(size int64) {
}
}
}
// 计算stale时长
func (this *HTTPWriter) calculateStaleLife() int {
var staleLife = 600 // TODO 可以在缓存策略里设置此时间
var staleConfig = this.req.web.Cache.Stale
if staleConfig != nil && staleConfig.IsOn {
// 从Header中读取stale-if-error
var isDefinedInHeader = false
if staleConfig.SupportStaleIfErrorHeader {
var cacheControl = this.Header().Get("Cache-Control")
var pieces = strings.Split(cacheControl, ",")
for _, piece := range pieces {
var eqIndex = strings.Index(piece, "=")
if eqIndex > 0 && strings.TrimSpace(piece[:eqIndex]) == "stale-if-error" {
// 这里预示着如果stale-if-error=0可以关闭stale功能
staleLife = types.Int(strings.TrimSpace(piece[eqIndex+1:]))
isDefinedInHeader = true
break
}
}
}
// 自定义
if !isDefinedInHeader && staleConfig.Life != nil {
staleLife = types.Int(staleConfig.Life.Duration().Seconds())
}
}
return staleLife
}

View File

@@ -0,0 +1,102 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"bufio"
"github.com/iwind/TeaGo/types"
"net"
"net/http"
"time"
)
// HTTPRateWriter 限速写入
type HTTPRateWriter struct {
parentWriter http.ResponseWriter
rateBytes int
lastBytes int
timeCost time.Duration
}
func NewHTTPRateWriter(writer http.ResponseWriter, rateBytes int64) http.ResponseWriter {
return &HTTPRateWriter{
parentWriter: writer,
rateBytes: types.Int(rateBytes),
}
}
func (this *HTTPRateWriter) Header() http.Header {
return this.parentWriter.Header()
}
func (this *HTTPRateWriter) Write(data []byte) (int, error) {
if len(data) == 0 {
return 0, nil
}
var left = this.rateBytes - this.lastBytes
if left <= 0 {
if this.timeCost > 0 && this.timeCost < 1*time.Second {
time.Sleep(1*time.Second - this.timeCost)
}
this.lastBytes = 0
this.timeCost = 0
return this.Write(data)
}
var n = len(data)
// n <= left
if n <= left {
this.lastBytes += n
var before = time.Now()
defer func() {
this.timeCost += time.Since(before)
}()
return this.parentWriter.Write(data)
}
// n > left
var before = time.Now()
result, err := this.parentWriter.Write(data[:left])
this.timeCost += time.Since(before)
if err != nil {
return result, err
}
this.lastBytes += left
return this.Write(data[left:])
}
func (this *HTTPRateWriter) WriteHeader(statusCode int) {
this.parentWriter.WriteHeader(statusCode)
}
// Hijack Hijack
func (this *HTTPRateWriter) Hijack() (conn net.Conn, buf *bufio.ReadWriter, err error) {
if this.parentWriter == nil {
return
}
hijack, ok := this.parentWriter.(http.Hijacker)
if ok {
return hijack.Hijack()
}
return
}
// Flush Flush
func (this *HTTPRateWriter) Flush() {
if this.parentWriter == nil {
return
}
flusher, ok := this.parentWriter.(http.Flusher)
if ok {
flusher.Flush()
return
}
}

View File

@@ -5,6 +5,7 @@ import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/goman"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"net"
"sync"
@@ -55,11 +56,11 @@ func (this *Listener) listenTCP() error {
}
protocol := this.group.Protocol()
netListener, err := this.createTCPListener()
tcpListener, err := this.createTCPListener()
if err != nil {
return err
}
netListener = NewClientListener(netListener, protocol.IsHTTPFamily() || protocol.IsHTTPSFamily())
var netListener = NewClientListener1(tcpListener, protocol.IsHTTPFamily() || protocol.IsHTTPSFamily())
events.On(events.EventQuit, func() {
remotelogs.Println("LISTENER", "quit "+this.group.FullAddr())
_ = netListener.Close()
@@ -72,6 +73,7 @@ func (this *Listener) listenTCP() error {
Listener: netListener,
}
case serverconfigs.ProtocolHTTPS, serverconfigs.ProtocolHTTPS4, serverconfigs.ProtocolHTTPS6:
netListener.SetIsTLS(true)
this.listener = &HTTPListener{
BaseListener: BaseListener{Group: this.group},
Listener: netListener,
@@ -82,6 +84,7 @@ func (this *Listener) listenTCP() error {
Listener: netListener,
}
case serverconfigs.ProtocolTLS, serverconfigs.ProtocolTLS4, serverconfigs.ProtocolTLS6:
netListener.SetIsTLS(true)
this.listener = &TCPListener{
BaseListener: BaseListener{Group: this.group},
Listener: netListener,
@@ -97,7 +100,7 @@ func (this *Listener) listenTCP() error {
this.listener.Init()
go func() {
goman.New(func() {
err := this.listener.Serve()
if err != nil {
// 在这里屏蔽accept错误防止在优雅关闭的时候有多余的提示
@@ -109,7 +112,7 @@ func (this *Listener) listenTCP() error {
// 打印其他错误
remotelogs.Error("LISTENER", err.Error())
}
}()
})
return nil
}
@@ -129,12 +132,12 @@ func (this *Listener) listenUDP() error {
Listener: listener,
}
go func() {
goman.New(func() {
err := this.listener.Serve()
if err != nil {
remotelogs.Error("LISTENER", err.Error())
}
}()
})
return nil
}

Some files were not shown because too many files have changed in this diff Show More